In the previous post we built Deep Q Learning and saw it converge to near-perfect performance on CartPole. DQN works, but it has a known flaw baked into its update rule — one that causes it to systematically overestimate Q-values, and one that requires a single line of code to fix. That fix is Double DQN.

In the previous post we saw that Q Learning can outperform Hill Climb when the state space is naturally discrete and the optimal policy is non-linear. But we also saw its limits: on CartPole, a continuous environment, the discretization required to build a Q-table introduced enough noise that Hill Climb won anyway.

In the previous post we introduced the Carrot and Stick framework and walked through the Hill Climb algorithm on the CartPole environment. Hill Climb is simple and surprisingly effective, but it has a fundamental limitation: it treats the entire episode as a single experience. It only asks “did this episode go well?” and updates accordingly — it never asks “which specific decisions in the episode were good, and which were bad?”.

A while ago I went to a Meetup about Reinforcement Learning (RL), I got into a conversation with some one that sat next to me. He asked me several question about the subject - What is the difference between RL and supervised/unsupervised learning? What is the difference between several types of algorithms? When would you choose this framework over another one?

Decisions are hard, they have always been. And when you finally find something you like, there is always that thought, in the back of your head - “can I find something better?”.