Slide 1

Slide 1: Title Slide

  • Title: Advanced Policy Gradient Methods: TRPO & PPO
  • Session: YBIGTA SUMMER SESSION
  • Presenter: DS 26 Jungmook Kang
  • Date: 2025.08.26

Slide 2 Slide 3 Slide 4 Slide 5

Slide 2-5: Why does Reinforcement Learning sometimes fail?

  • Main Question: “Why does reinforcement learning sometimes break down?”
  • Problem: Basic policy gradient methods (like A2C) have limitations. Making drastic changes to the policy—the agent’s decision-making criteria—can cause significant problems.
  • Key Issues:
    1. Unstable Learning: If an update step is too large and in the wrong direction, the agent’s performance can collapse, making recovery impossible. This is more damaging than in supervised learning because a bad policy affects the distribution of states and rewards the agent will see in the future.
    2. Changing Data Distribution: As the policy changes, the data (states, actions, rewards) it collects also changes. This non-stationarity of the input data makes learning difficult.
  • Example Analogy: A robot learning to walk. If it’s walking well and tries a new strategy by swinging its legs too wide, it will fall. Once it has fallen, it can no longer collect useful data about walking, and learning stops.

Slide 6 Slide 7 Slide 8

Slide 6-8: What is TRPO? “Let’s not change the policy too much!”

  • Concept: Trust Region Policy Optimization (TRPO).
  • Core Idea: When updating the policy, TRPO establishes a “trust region” or a “safe zone” to ensure the new policy does not stray too far from the old one.
  • Goal: “Maximize policy performance under the constraint that the difference (distance) between the old and new policies does not exceed a certain value, δ.” This prevents the destructive, large policy updates that can lead to performance collapse.

Slide 9 Slide 10 Slide 11

Slide 9-11: The Mathematical Expression of TRPO

This slide introduces the core optimization problem of TRPO. The goal is to maximize an objective function, subject to a constraint.

  • Objective Function (What to Maximize):
    • The expression $\mathbb{E}[\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}}(a|s)}\hat{A}]$ represents the expected advantage of the new policy $\pi_{\theta}$ relative to the old policy $\pi_{\theta_{old}}$.
    • This is a surrogate objective that uses importance sampling to estimate the performance of the new policy using data collected from the old one. The term $\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}}(a|s)}$ is the importance sampling ratio. The full objective function is:
$$ L_{\theta_{old}}^{IS}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{old}}(a_{t}\mid s_{t})}\hat{A}_{t}\right] $$
  • Constraint (The “Trust Region”):
    • The term $\mathbb{E}[D_{KL}(\pi_{\theta_{old}}||\pi_{\theta})]\le\delta$ is the constraint that defines the trust region.
    • $D_{KL}$ stands for Kullback-Leibler (KL) Divergence. It’s a way to measure the “distance” or difference between two probability distributions. Here, it measures how much the new policy $\pi_{\theta}$ has changed from the old policy $\pi_{\theta_{old}}$.
    • By keeping this value less than a small constant $\delta$, we ensure the policy update is small and stays within a “trusted” region where the performance estimate is reliable.

The full optimization problem solved by TRPO is:

$$ \underset{\theta}{\text{maximize}} \quad \hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\hat{A}_{t}\right] $$
$$ \text{subject to} \quad \hat{\mathbb{E}}_{t}\left[KL\big(\pi_{\theta_{old}}(\cdot|s_{t}),\,\pi_{\theta}(\cdot|s_{t})\big)\right]\le\delta $$

Slide 12

Slide 12: Limitations of TRPO

  • Problem: While TRPO’s core idea was groundbreaking for stabilizing RL, it wasn’t widely adopted in practice.
  • Major Drawbacks:
    1. High Computational Cost: It uses a complex, second-order optimization technique called the “conjugate gradient method” which is computationally heavy.
    2. Difficult to Implement: The algorithm is complex to code and debug.
    3. Poor Performance on Certain Tasks: It struggled with tasks involving complex visual inputs (requiring deep CNNs), such as the Atari benchmark. It is also hard to use with architectures that have multiple outputs (e.g., a policy and a value function).

Slide 13 Slide 14

Slide 13-14: PPO: The Practical Version of TRPO

  • Concept: Proximal Policy Optimization (PPO).
  • Core Idea: PPO inherits TRPO’s philosophy of “let’s change the policy a little at a time” but replaces the complex and strict trust region constraint with a simpler mechanism that is easier to implement and more efficient.
  • How it Improves on TRPO: Instead of using a hard constraint, PPO modifies the objective function to penalize policies that move too far away from the old policy. The most common technique for this is called Clipping.
  • Impact: PPO is now one of the most widely used policy optimization algorithms due to its simplicity, stability, and strong performance.

Slide 15 Slide 16 Slide 17 Slide 18 Slide 19

Slide 15-19: PPO’s Core Idea: Clipping

This section details the PPO-Clip objective function, which is the key to its success.

  • PPO-Clip Objective Function:
$$ L^{CLIP}(\theta)=\hat{\mathbb{E}}\left[\min\big(r_{t}(\theta)\hat{A}_{t},\ \operatorname{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big)\right] $$

  • Breakdown of Terms:

    1. $r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}$: This is the same probability ratio used in TRPO. It indicates whether an action is more or less likely under the new policy compared to the old one.
    2. $clip(r_{t}(\theta),1-\epsilon,1+\epsilon)$: This is the clipping mechanism. It forces the probability ratio $r_t(\theta)$ to stay within a small range, $[1-\epsilon, 1+\epsilon]$ (e.g., [0.8, 1.2]). This acts as a safety rail, preventing the ratio from becoming too large or too small.
    3. $min(…)$: The final objective takes the minimum of two values:
      • The original objective ($r_{t}(\theta)\hat{A}_{t}$)
      • The clipped objective ($clip(…)\hat{A}_{t}$)
    • By taking the minimum, PPO creates a pessimistic, lower-bound estimate of the policy’s performance. This effectively puts a cap on the benefit you can get from a single update, which discourages excessively large changes to the policy.

  • Analogy: It’s like setting a study plan. No matter how much you enjoy one subject, you cap its study time at “a maximum of 3 hours per day.” This prevents you from neglecting other subjects and keeps your overall learning balanced and stable.

Slide 20 Slide 21

Slide 20-21: The Power and Application of PPO

  • Why it’s popular: PPO is simple, powerful, and stable, making it the de-facto standard for reinforcement learning.
  • Where is PPO used?:
    • Robotics: Training stable and precise movements.
    • Game Playing: Achieving superhuman performance in complex games.
    • Large Language Models (LLMs): Fine-tuning models like ChatGPT to produce more helpful and safer responses (a process known as Reinforcement Learning from Human Feedback or RLHF).
  • Major Users: Almost all leading AI companies, including OpenAI, DeepMind, Meta, and NVIDIA, use PPO as a core algorithm.

Slide 22