I did not start this project to build a production-ready autopilot. I started it because every RL tutorial ends with “…and the agent solved the task!” while my own runs ended with a pixel-car doing donuts in a digital meadow. I wanted to know why.

CarRacing-v3 is a toy environment, but it captures every headache of real-world continuous control: high-dimensional images, noisy rewards, and a control surface smooth enough to let you believe you’re making progress—right up to the moment the agent locks itself into a 200-frame circle.
Proximal Policy Optimization (PPO) is the algorithm we are all told to reach for first. If PPO cannot learn to stay on a gray road that is literally painted for it, what hope do we have on a real highway?
Those are the stakes. The blog that follows is a postmortem, not a victory lap.

flowchart illustrating a reinforcement learning pipeline for the CarRacing-v3 environment. It starts with raw image observations, followed by preprocessing to grayscale and resizing. Then, frames are stacked and input into a CNN encoder, which connects to a PPO policy network. The network produces actions (steer, throttle, brake) and receives reward signals, completing the loop.

The Reward Trap: Why the Agent Loops

The environment rewards the agent for visiting new green tiles (+1), but penalizes time (-0.1 per frame). This creates a perverse incentive: - A 12-tile loop gives +12 reward per lap. - After 40 frames, the time cost is -4. - Net gain: +8 per lap. - Once the loop is closed, no new tiles are found, so the reward turns negative. - But the policy stays in the loop because the gradient is flat—PPO refuses to explore actions that might break the loop.

The agent is rewarded for visiting new tiles, not for driving forward—like being paid for walking into a forest, not for reaching the exit

Environment & Setup – A 96×96 Playground That Hides a Trap

Key takeaway: the agent only sees pixels. No speedometer, no gyroscope.
Milestones:

• 20 k steps → 48 tiles (random wiggles)

• 300 k steps → 210 tiles (partial track)

• 1 M steps → 353 tiles, then a longer path still a donut

The road is empty, the tyres are virtual, and the embarrassment is real, I recorded the moment; watch the 1-M agent spinning in a loop

Click here on the link symbol:- CLICK HERE🔗 1M Timestamps car racing ppo

The PPO Pipeline: How I Fed Pixels to a Policy and Got a Merry-Go-Round

Stable-Baselines3 gives you a one-liner PPO("CnnPolicy", env, …), but the defaults are not your friend in visual control.
Here is the pipeline that reached 353 tiles before the donut phase.

Hyper-parameters That Mattered

n_steps          = 2048
batch_size       = 256
learning_rate    = 3e-4  (no schedule)
ent_coef         = 0.0   ← big change; entropy encouraged donuts
clip_range       = 0.2
gae_lambda       = 0.95
total_timesteps  = 1 000 000

Entropy regularization (default 0.01) literally paid the agent to spin—a wider action distribution samples tight circles more often. After I zeroed it, the loops became tighter but still present, proving the reward, not entropy, is the root.

Reward is a Liar: How +1 Per Tile Trained a Professional Figure-Skater

The environment’s reward looks sensible on paper: “Visit new tiles, get +1.”
In practice it creates a greedy, myopic scorer that cares exactly zero about where the car will be in 50 steps—perfect fodder for PPO’s expected-return maximizer.

What the Agent Learnt

• A 12-tile clockwise loop yields +12 every 40 frames

• Net return after step cost: +12 – 0.1 × 40 = +8 per lap

• Velocity is low ⇒ no wall penalty ⇒ risk-free income stream

Once the policy stumbles on this cycle, the PPO’s clipping objective keeps it there: the ratio
rₜ(θ) = π_θ(a|s) / π_θ_old(a|s)
hovers near 1 because no competing action looks significantly better inside the clip range. The optimizer quietly fine-tunes the spin instead of breaking it.

Human Eye vs Policy Eye

To us the tyres are drawing a perfect circle on grass; to the policy it is simply revisiting fresh green tiles at the edge of its 96×96 window.
Because the track is closed and symmetric, the convolutional features for “entering turn” and “exiting turn” collapse to the same embedding—the network literally can’t tell it’s looping.

Failed Quick-Fixes (ablation summary)

• Before loop: The agent gains +8 per lap.

• After loop: The agent loses -4 per lap.

But the policy still stays in the self loop because:

• The gradient is flat — PPO’s clipped surrogate objective refuses to explore actions that might break the loop.

• The local optimum is stable — the policy doesn’t update to escape.

Take-away

“More training” only amplifies whatever the reward signal praises.
If the praise is misaligned with human intent, PPO will deliver a higher-resolution failure.

The video still shows 353 tiles, but every frame after step 450 is déjà-vu.

Code & Reproducibility Snapshot

Want to re-create the 353-tile donut yourself?
Grab the exact notebook I ran on Google Colab (T4 GPU, ~35 min):

📦 Notebook: carselfdriving.ipynb
(All cells are self-contained; just “Run All” after mounting Drive.)

Key library pins (as of 2026-01-03):

gymnasium==0.29.1  
stable-baselines3==2.3.0  
box2d-py==2.3.5  
opencv-python-headless==4.9.0

Difficulty check: Intermediate

• Requires a free Colab account + 5 GB Drive space

• No local GPU needed; do not try on CPU unless you like 6-hour spins.

Video-First Evaluation: Why Your Eyeball is a Better Metric than episode_reward

Reward curves are seductive—they slope upward, you tweet the screenshot, and nobody notices the car is figure-skating.
I only realised the agent was looping after watching a 60-second clip. That embarrassment spawned a rule I now apply to every RL project:

Never trust a scalar you can’t see.

Reward
  ↑           *
  |         *   *
  |       *       *   ← global optimum (desired)
  |     *
  |   *        ← local optimum (looping)
  +----------------------------> Policy Space

My Cheap Evaluation Stack

1. VecVideoRecorder wrapped around a 4-env vector

2. Record every 50 k steps (≈ 3 min of video)

3. Push mp4 to Drive, watch at 2× speed on phone

4. Tag behaviours: straight, wiggle, spin, stall

One evening I had four videos open in split-screen—20 k, 300 k, 500 k, 1 M.
The 1-M policy looked smoother, tyres never left tarmac, steering wheel barely twitched… and yet the car never advanced more than 12 tiles before resetting the same GPS coordinates. The reward graph lied, my eyes didn’t.

Take-away for Practitioners

• Always render first, log second.

• Store short clips, not just .csv—disk is cheaper than retraining.

• If behaviour plateaus while reward climbs, you are overfitting the reward, not the task.

Seeing is believing; graphs are just gossip with error bars.

Look at the numbers while the tyres were still on tarmac:

The system was working exactly as advertised:
“Agent on road → new tile → reward goes up.”
No bug, no sparse signal, no cliff.

environment’s default reward for “any new pixel” doesn’t actually means “good driving.
On the lawn the car gone into a 12-tile circle, entering untouched pixels every 38 frames.
Tile counter kept climbing → reward still positive → algorithm logs another “success.”
By tile 349 the loop was closed; after that zero fresh tiles appear, only the −0.1 time-tax remains, so the total return slides downhill (353 → 280 reward count).
The feedback flips from “good job” to “you’re wasting time,” yet the policy stays glued to the circle. Why?

Because in the policy-space landscape that little ring is a local optimum—a nice, safe hill with smooth gradients and no walls.
The global optimum (full lap, 600+ tiles, finish line) is a higher peak, but it lies across a valley of rare actions: sharp braking, counter-steer, maybe a backwards flick to re-align.
PPO’s clipped surrogate objective refuses to leap that valley; it would rather fine-tune the steering angle that keeps the cosy 12-tile merry-go-round alive.

So the agent ignores the road not out of stupidity, but because the reward contract never mentions “road” at all—only “new tile new point.”
Grass tiles count just the same, and once the policy discovers they can be farmed infinitely, asphalt becomes irrelevant.
The decreasing return after 353 is the environment’s way of whispering “this isn’t the route,” yet the car keeps spinning—locally optimal, globally blind.

Why Car Prefers a 280-Tax Spin to a 600-Tile Triumph

Imagine you’re the policy network for a second.
Every 0.1 s you pick a steering angle; PPO hands you a report card that averages the next few seconds of reward.

After hundreds of thousands of practice laps you discover:

• Action “0.42-left” → safe loop, −0.1 per step, variance = 0

• Action “0.15-right” → car drifts toward unknown, −0.3 ± 20, variance = huge

The agent discovers a 12-tile loop that gives a high return (+12 per lap, minus 0.1×40 = -4, so +8 net). Once the loop is closed, it can no longer gain new tiles, so the reward per frame turns negative. But the policy still stays in the loop because the gradient is flat — PPO’s clipped surrogate objective refuses to explore actions that might break the loop, even though they would lead to higher long-term rewards.

Mathematically, the second option has higher expected loss and explosive variance, so the gradient shouts: “Don’t go there!”
The clip-range safety net (±20 %) then flattens the gradient to zero for every nearby action. Result: your steering weights stop updating even though the numerical return is sliding downhill.
You’re trapped in a flat valley of local optimum—safe, predictable, boring, and impossible to escape.

“But grass gave +1 at the start—why visit grass at all?”

The first grass pixel is just random exploration—a micro-wiggle that happens to land on a green tile the network has never seen.
That single lucky +1 nudges the probability of the wiggle upward.
Because the track edge is rotationally symmetric, the same micro-wiggle scores another +1 38 frames later.
After a few dozen reinforcements the probability mass collapses onto the repeating sequence.
From that moment the agent almost never samples a departure action again; grass started as an accident, became a habit, and turned into a life sentence.

🔚 Conclusion – Kill the Donut, Not the Experiment

This project started as a weekend hack and ended as a 1 000 000-step seminar on misaligned incentives. PPO did exactly what we asked it to do: maximise the sum of future rewards. The environment handed out those rewards for grazing fresh asphalt, not for arriving anywhere. Is it any surprise the agent became a professional figure-skater?

The takeaway is bigger than a toy car. In recommendation systems, trading bots, or real autonomous vehicles, the same law applies:

If the metric is sloppy, the optimiser will be precisely sloppy.

So render your agents, question your rewards, and publish the failures alongside the fireworks. The next time your policy starts spinning, don’t curse the algorithm—rewrite the contract.

Clone the notebook, swap in the progress-based reward, and let me know when your pixel-car finally drives past the donut shop and keeps on going.

Key takeaways

1. Watch before you tweet.
   A pretty reward graph is worth nothing if the behavior is stuck in a jig. Render 30 s of video every few checkpoints; it takes two minutes and saves days of denial.

2. A reward is a contract, not a suggestion.
   The agent will find the cheapest path to maximize the signal—even if that path looks absurd to you. Write the contract carefully or prepare to renegotiate.

3. Local optima are features, not bugs.
   PPO (and any policy-gradient method) is designed to climb whatever hill it lands on. If the tallest hill in your landscape is a donut, you’ll get a world-class donut.

4. Continuous control needs progress, not just collection.*
   “+1 when you see something new” works for grid-worlds; on a looping track it’s an invitation to tread-water. Tie the reward to forward displacement or time-to-goal and the loop disappears.

5. Fail in public, learn in public.
   I gained more from this 280-tile regression than from any benchmark I ever “solved.” Document the collapse, share the video, and the next researcher starts one step closer to the fix.

Keep these in mind and your next agent might still crash—but at least it will crash somewhere new.