Policy gradients from scratch

This project builds a policy gradient reinforcement learning agent from scratch in PyTorch. The algorithm is REINFORCE, written without a reinforcement learning library, so every part of it can be explained line by line.

The first environment is CartPole, where the agent learns to balance a pole on a moving cart. After training, the network is exported to a small JSON file and replayed in your browser on the demo page, so what you watch is the same set of weights that came out of the run.

A later phase moves the same algorithm to a small language model, to show reward hacking and what a KL penalty does about it. Training curves from every run, a dated experiment log and the full write-up are published here as the work progresses.