Set your Publisher ID in src/config/ads.ts to display live ads.
Reinforcement Learning & Policy Optimization
Markov Decision Processes, Q-Learning, and Policy Gradients
Reinforcement Learning (RL) trains agents to maximize cumulative rewards through trial-and-error interactions with an environment formalized as a Markov Decision Process (MDP).
The RL Loop and Bellman Equation
Agent and Environment: At time step t, the agent observes state S_t, takes action A_t, and receives numerical reward R_{t+1} while the environment transitions to state S_{t+1}.
Policy π(a|s) defines the probability distribution over actions given the current state.
The Bellman Optimality Equation states that the value of an action is the immediate reward plus discounted expected future returns: Q*(s, a) = R + γ * max_a' Q*(s', a'), where γ is the discount factor (0 <= γ < 1).
# Q-Learning update rule
learning_rate = 0.1
discount_gamma = 0.95
# Current estimated Q-value for (state, action)
current_q = 4.2
reward = 1.0
max_future_q = 6.0 # Best possible action from next state
# Temporal Difference (TD) target and error
td_target = reward + (discount_gamma * max_future_q)
td_error = td_target - current_q
new_q = current_q + (learning_rate * td_error)
print(f"Updated Q-value: {new_q:.4f}")RLHF (Reinforcement Learning from Human Feedback) uses human preferences to align Large Language Models with helpful and harmless behavior.
Set your Publisher ID in src/config/ads.ts to display live ads.
Test Yourself With Exercise
In the Bellman equation for Q-learning, what is the role of the discount factor gamma (γ)?