Reinforcement Learning with Neural Networks: Mathematical Details

2025-04-14 · Source: StatQuest with Josh Starmer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Intermediate, long

Summary

This StatQuest details the mathematical underpinnings of training a neural network using reinforcement learning, specifically the policy gradients method. It focuses on optimizing a single bias parameter within a simple neural network designed to decide between two "fry shacks" based on hunger level. The process involves calculating the network's output probabilities, making a random decision, and then quantifying the difference between the ideal outcome and the network's output using cross-entropy. The core of the training involves calculating the derivative of the cross-entropy with respect to the bias using the chain rule, then multiplying this derivative by a reward (1.0 for a correct guess, -1.0 for an incorrect guess) to update its direction. This updated derivative is then fed into gradient descent to adjust the bias, iteratively refining the network's decision-making over multiple training cycles with varying hunger inputs until the bias converges.

Key takeaway

For Machine Learning Engineers implementing reinforcement learning with neural networks, understanding the detailed mathematical steps for policy gradients is crucial. Your ability to correctly calculate derivatives, especially using the chain rule for cross-entropy and sigmoid functions, and then apply rewards to steer parameter updates via gradient descent, directly impacts model convergence and performance. Ensure your reward function accurately reflects desired outcomes to effectively correct "wrong guesses" and guide the network towards optimal policies.

Key insights

Policy gradients train neural networks by adjusting parameters based on rewards that correct derivative directions.

Principles

• Cross-entropy quantifies probability differences.
• Chain rule decomposes complex derivatives.
• Rewards correct derivative direction.

Method

Calculate cross-entropy derivative with respect to bias using the chain rule, multiply by a reward (1.0 or -1.0) to correct direction, then update the bias via gradient descent.

In practice

• Use cross-entropy for probability-based loss.
• Apply chain rule for backpropagation.
• Implement reward scaling for derivative correction.

Topics

• Reinforcement Learning
• Neural Networks
• Policy Gradients
• Cross Entropy
• Gradient Descent

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Reinforcement Learning with Neural Networks: Essential Concepts — Reinforcement learning with policy gradients trains neural networks without ideal outputs by using guesses and reward signals.
• Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models — Entropy-Gradient Inversion, a negative correlation between token entropy and logit gradients, fingerprints LRM reasoning.
• How I Finally Understood Backpropagation by Deriving It by Hand — Deriving backpropagation by hand provides a fundamental understanding of neural network gradient computation.
• Policy Gradient in One Minute — Policy gradient algorithms iteratively optimize neural network policies by approximating gradients from sampled trajectories to maximize rewards.
• Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation — Gradient exchangeability is crucial for stable intra-group reinforcement learning to prevent drift and entropy collapse.
• Backpropagation is Just the Chain Rule — Backpropagation efficiently computes neural network gradients by applying the chain rule backward through layers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by StatQuest with Josh Starmer.