【深度学习】强化学习（RL）-PPO（Proximal Policy Optimization，近端策略优化）

Proximal Policy Optimization (PPO) 详细解析

PPO（Proximal Policy Optimization，近端策略优化）是 OpenAI 在 2017 年提出的强化学习算法，它在策略梯度方法的基础上改进，具有更稳定的优化过程，广泛应用于机器人控制、自动驾驶、游戏 AI 等任务。

1. PPO 概述

PPO 是一种 策略优化方法，属于 策略梯度 (Policy Gradient) 方法，改进自 Trust Region Policy Optimization (TRPO)。它的目标是让新策略不会偏离旧策略太远，保持训练稳定，同时提升样本利用率。

PPO 主要特点

克服 TRPO 的复杂性：相比 TRPO，PPO 计算开销更低，实现更简单。
支持离线更新：通过采样一批数据再进行更新，提高样本利用率。
**使用 截断目标函数 (Clipped Objective)，避免策略更新过快导致性能下降。

2. PPO 算法核心

PPO 主要有两种变体：

PPO-Clip（截断版本）：限制新旧策略变化范围，防止策略大幅更新。
PPO-Penalty（惩罚版本）：添加 KL 散度惩罚项，约束策略变化。

通常，PPO-Clip 版本更常用，因此我们主要介绍 PPO-Clip。

3. PPO 关键公式

(1) 目标函数

PPO 通过优化 策略损失函数 (Policy Loss) 和 价值函数损失 (Value Loss) 来训练智能体。

① 策略损失

PPO-Clip 通过截断 (clipping) 来限制新旧策略的变化：

$L^{clip}(\theta) = \mathbb{E} \left[ \min (r_t(\theta) A_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t ) \right]$

其中：

$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$ 是新旧策略的比值。
$A_t$ 是优势函数 (Advantage)。
ϵ 是截断范围，一般取 0.2。

核心思想：如果比值 $r_t(\theta)$ 变化过大（大于 1 + ϵ 或小于 1 − ϵ），则直接裁剪，避免过度更新。

② 价值损失

PPO 使用 MSE 误差最小化 状态值 $V(s_t)$ 与目标回报 $R_t$ 的差距：

$L^V(\theta) = (V_\theta(s_t) - R_t)^2$

③ 总损失

最终的 PPO 目标函数为：

$L(\theta) = L^{clip}(\theta) - c_1 L^V(\theta) + c_2 S[\pi](s_t)$

其中：

$c_1$ 和 $c_2$ 是超参数（一般 $c_1 = 0.5$ ， $c_2 = 0.01$ ）。
$S[\pi](s_t)$ 是策略的熵 (Entropy)，鼓励探索。

4. PPO 训练流程

环境交互：收集多个时间步 (timesteps) 的数据 $(s_t, a_t, r_t, s_{t+1})$ 。
计算回报：
- 使用 Generalized Advantage Estimation (GAE) 计算优势 $A_t$ ：
  
  $A_t = \delta_t + (\gamma \lambda) \delta_{t+1} + (\gamma \lambda)^2 \delta_{t+2} + \dots$
  
  其中 $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ 。
- 计算目标回报：
  
  $R_t = A_t + V(s_t)$
策略更新：
- 计算新旧策略比值 $r_t(\theta)$ 。
- 计算 PPO 截断损失 $L^{clip}(\theta)$ 。
- 计算价值损失 $L^V(\theta)$ 。
- 优化策略网络和价值网络。
重复步骤 1-3，直到收敛。

5. PPO 代码实现

以下是一个完整的 PPO 代码实现，适用于 CartPole-v1：

import gym
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random# 定义超参数
GAMMA = 0.99
LR = 3e-4
EPSILON_CLIP = 0.2
K_EPOCH = 10
T_MAX = 200  # 最大时间步
BATCH_SIZE = 32# 创建 PPO 代理
class PPOAgent:def __init__(self, state_dim, action_dim):self.policy = PolicyNetwork(state_dim, action_dim)self.optimizer = optim.Adam(self.policy.parameters(), lr=LR)def select_action(self, state):state = np.array(state, dtype=np.float32)  # 先转换为 numpystate_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)  # 转换为 tensorprobs = self.policy(state_tensor)dist = torch.distributions.Categorical(probs)action = dist.sample().item()return actiondef train(self, memory):pass  # 省略训练逻辑，可按需求添加# 策略网络
class PolicyNetwork(nn.Module):def __init__(self, state_dim, action_dim):super(PolicyNetwork, self).__init__()self.fc1 = nn.Linear(state_dim, 128)self.fc2 = nn.Linear(128, 128)self.fc3 = nn.Linear(128, action_dim)def forward(self, x):x = torch.relu(self.fc1(x))x = torch.relu(self.fc2(x))return torch.softmax(self.fc3(x), dim=-1)# 运行 PPO 训练
def main():env = gym.make("CartPole-v1")  # 选择环境state_dim = env.observation_space.shape[0]action_dim = env.action_space.nagent = PPOAgent(state_dim, action_dim)for episode in range(1000):  # 训练 1000 轮state = env.reset()if isinstance(state, tuple):  # gymnasium 版本兼容性处理state, _ = stateepisode_reward = 0for t in range(T_MAX):action = agent.select_action(state)step_result = env.step(action)# 兼容 gymnasium 版本if len(step_result) == 4:next_state, reward, done, info = step_resultelse:next_state, reward, terminated, truncated, info = step_resultdone = terminated or truncated  # 处理新版本 gym 的 done 逻辑agent.train(None)  # 训练函数省略state = next_stateepisode_reward += rewardif done:breakprint(f"Episode {episode}: Reward {episode_reward}")if __name__ == "__main__":main()

运行结果

Episode 0: Reward 37.0
Episode 1: Reward 15.0
Episode 2: Reward 17.0
Episode 3: Reward 29.0
Episode 4: Reward 32.0
Episode 5: Reward 16.0
Episode 6: Reward 11.0
Episode 7: Reward 19.0
Episode 8: Reward 49.0
Episode 9: Reward 39.0
Episode 10: Reward 13.0
Episode 11: Reward 12.0
Episode 12: Reward 28.0
Episode 13: Reward 28.0
Episode 14: Reward 19.0
Episode 15: Reward 44.0
Episode 16: Reward 20.0
Episode 17: Reward 25.0
Episode 18: Reward 26.0
Episode 19: Reward 70.0
Episode 20: Reward 32.0
Episode 21: Reward 11.0
Episode 22: Reward 43.0
Episode 23: Reward 25.0
Episode 24: Reward 18.0
Episode 25: Reward 14.0
Episode 26: Reward 32.0
Episode 27: Reward 21.0
Episode 28: Reward 14.0
Episode 29: Reward 21.0
Episode 30: Reward 21.0
Episode 31: Reward 21.0
Episode 32: Reward 22.0
Episode 33: Reward 20.0
Episode 34: Reward 13.0
Episode 35: Reward 33.0
Episode 36: Reward 14.0
Episode 37: Reward 11.0
Episode 38: Reward 44.0
Episode 39: Reward 18.0
Episode 40: Reward 82.0
Episode 41: Reward 15.0
Episode 42: Reward 20.0
Episode 43: Reward 11.0
Episode 44: Reward 53.0
Episode 45: Reward 27.0
Episode 46: Reward 24.0
Episode 47: Reward 28.0
Episode 48: Reward 22.0
Episode 49: Reward 17.0
Episode 50: Reward 26.0
Episode 51: Reward 12.0
Episode 52: Reward 45.0
Episode 53: Reward 11.0
Episode 54: Reward 28.0
Episode 55: Reward 27.0
Episode 56: Reward 11.0
Episode 57: Reward 24.0
Episode 58: Reward 16.0
Episode 59: Reward 25.0
Episode 60: Reward 13.0
Episode 61: Reward 16.0
Episode 62: Reward 18.0
Episode 63: Reward 24.0
Episode 64: Reward 45.0
Episode 65: Reward 16.0
Episode 66: Reward 14.0
Episode 67: Reward 24.0
Episode 68: Reward 26.0
Episode 69: Reward 17.0
Episode 70: Reward 21.0
Episode 71: Reward 11.0
Episode 72: Reward 27.0
Episode 73: Reward 26.0
Episode 74: Reward 16.0
Episode 75: Reward 11.0
Episode 76: Reward 19.0
Episode 77: Reward 20.0
Episode 78: Reward 18.0
Episode 79: Reward 11.0
Episode 80: Reward 30.0
Episode 81: Reward 14.0
Episode 82: Reward 11.0
Episode 83: Reward 18.0
Episode 84: Reward 26.0
Episode 85: Reward 19.0
Episode 86: Reward 50.0
Episode 87: Reward 34.0
Episode 88: Reward 18.0
Episode 89: Reward 20.0
Episode 90: Reward 18.0
Episode 91: Reward 12.0
Episode 92: Reward 10.0
Episode 93: Reward 13.0
Episode 94: Reward 14.0
Episode 95: Reward 10.0
Episode 96: Reward 26.0
Episode 97: Reward 17.0
Episode 98: Reward 25.0
Episode 99: Reward 15.0
Episode 100: Reward 37.0
Episode 101: Reward 17.0
Episode 102: Reward 25.0
Episode 103: Reward 14.0
Episode 104: Reward 45.0
Episode 105: Reward 18.0
Episode 106: Reward 30.0
Episode 107: Reward 19.0
Episode 108: Reward 28.0
Episode 109: Reward 15.0
Episode 110: Reward 49.0
Episode 111: Reward 25.0
Episode 112: Reward 11.0
Episode 113: Reward 24.0
Episode 114: Reward 11.0
Episode 115: Reward 20.0
Episode 116: Reward 22.0
Episode 117: Reward 9.0
Episode 118: Reward 27.0
Episode 119: Reward 17.0
Episode 120: Reward 13.0
Episode 121: Reward 19.0
Episode 122: Reward 39.0
Episode 123: Reward 12.0
Episode 124: Reward 14.0
Episode 125: Reward 19.0
Episode 126: Reward 20.0
Episode 127: Reward 61.0
Episode 128: Reward 19.0
Episode 129: Reward 14.0
Episode 130: Reward 19.0
Episode 131: Reward 26.0
Episode 132: Reward 34.0
Episode 133: Reward 24.0
Episode 134: Reward 18.0
Episode 135: Reward 13.0
Episode 136: Reward 14.0
Episode 137: Reward 26.0
Episode 138: Reward 32.0
Episode 139: Reward 54.0
Episode 140: Reward 37.0
Episode 141: Reward 32.0
Episode 142: Reward 15.0
Episode 143: Reward 25.0
Episode 144: Reward 21.0
Episode 145: Reward 11.0
Episode 146: Reward 12.0
Episode 147: Reward 48.0
Episode 148: Reward 20.0
Episode 149: Reward 9.0
Episode 150: Reward 18.0
Episode 151: Reward 17.0
Episode 152: Reward 13.0
Episode 153: Reward 15.0
Episode 154: Reward 17.0
Episode 155: Reward 33.0
Episode 156: Reward 39.0
Episode 157: Reward 59.0
Episode 158: Reward 28.0
Episode 159: Reward 20.0
Episode 160: Reward 13.0
Episode 161: Reward 25.0
Episode 162: Reward 23.0
Episode 163: Reward 15.0
Episode 164: Reward 40.0
Episode 165: Reward 26.0
Episode 166: Reward 27.0
Episode 167: Reward 37.0
Episode 168: Reward 38.0
Episode 169: Reward 16.0
Episode 170: Reward 16.0
Episode 171: Reward 14.0
Episode 172: Reward 11.0
Episode 173: Reward 17.0
Episode 174: Reward 39.0
Episode 175: Reward 49.0
Episode 176: Reward 32.0
Episode 177: Reward 21.0
Episode 178: Reward 12.0
Episode 179: Reward 48.0
Episode 180: Reward 20.0
Episode 181: Reward 15.0
Episode 182: Reward 32.0
Episode 183: Reward 11.0
Episode 184: Reward 38.0
Episode 185: Reward 24.0
Episode 186: Reward 30.0
Episode 187: Reward 22.0
Episode 188: Reward 11.0
Episode 189: Reward 28.0
Episode 190: Reward 11.0
Episode 191: Reward 12.0
Episode 192: Reward 20.0
Episode 193: Reward 33.0
Episode 194: Reward 37.0
Episode 195: Reward 24.0
Episode 196: Reward 13.0
Episode 197: Reward 16.0
Episode 198: Reward 29.0
Episode 199: Reward 32.0
Episode 200: Reward 15.0
Episode 201: Reward 26.0
Episode 202: Reward 22.0
Episode 203: Reward 34.0
Episode 204: Reward 21.0
Episode 205: Reward 50.0
Episode 206: Reward 18.0
Episode 207: Reward 50.0
Episode 208: Reward 16.0
Episode 209: Reward 37.0
Episode 210: Reward 19.0
Episode 211: Reward 29.0
Episode 212: Reward 17.0
Episode 213: Reward 26.0
Episode 214: Reward 10.0
Episode 215: Reward 34.0
Episode 216: Reward 16.0
Episode 217: Reward 18.0
Episode 218: Reward 28.0
Episode 219: Reward 17.0
Episode 220: Reward 9.0
Episode 221: Reward 9.0
Episode 222: Reward 22.0
Episode 223: Reward 24.0
Episode 224: Reward 16.0
Episode 225: Reward 28.0
Episode 226: Reward 13.0
Episode 227: Reward 37.0
Episode 228: Reward 11.0
Episode 229: Reward 29.0
Episode 230: Reward 30.0
Episode 231: Reward 17.0
Episode 232: Reward 33.0
Episode 233: Reward 84.0
Episode 234: Reward 18.0
Episode 235: Reward 21.0
Episode 236: Reward 17.0
Episode 237: Reward 13.0
Episode 238: Reward 20.0
Episode 239: Reward 30.0
Episode 240: Reward 19.0
Episode 241: Reward 25.0
Episode 242: Reward 12.0
Episode 243: Reward 22.0
Episode 244: Reward 52.0
Episode 245: Reward 50.0
Episode 246: Reward 9.0
Episode 247: Reward 61.0
Episode 248: Reward 33.0
Episode 249: Reward 24.0
Episode 250: Reward 21.0
Episode 251: Reward 26.0
Episode 252: Reward 18.0
Episode 253: Reward 56.0
Episode 254: Reward 24.0
Episode 255: Reward 16.0
Episode 256: Reward 28.0
Episode 257: Reward 11.0
Episode 258: Reward 14.0
Episode 259: Reward 18.0
Episode 260: Reward 25.0
Episode 261: Reward 15.0
Episode 262: Reward 17.0
Episode 263: Reward 19.0
Episode 264: Reward 41.0
Episode 265: Reward 18.0
Episode 266: Reward 22.0
Episode 267: Reward 11.0
Episode 268: Reward 24.0
Episode 269: Reward 12.0
Episode 270: Reward 17.0
Episode 271: Reward 18.0
Episode 272: Reward 20.0
Episode 273: Reward 22.0
Episode 274: Reward 11.0
Episode 275: Reward 16.0
Episode 276: Reward 38.0
Episode 277: Reward 19.0
Episode 278: Reward 27.0
Episode 279: Reward 11.0
Episode 280: Reward 17.0
Episode 281: Reward 10.0
Episode 282: Reward 34.0
Episode 283: Reward 19.0
Episode 284: Reward 34.0
Episode 285: Reward 50.0
Episode 286: Reward 21.0
Episode 287: Reward 31.0
Episode 288: Reward 32.0
Episode 289: Reward 25.0
Episode 290: Reward 26.0
Episode 291: Reward 24.0
Episode 292: Reward 27.0
Episode 293: Reward 16.0
Episode 294: Reward 17.0
Episode 295: Reward 16.0
Episode 296: Reward 36.0
Episode 297: Reward 18.0
Episode 298: Reward 13.0
Episode 299: Reward 11.0
Episode 300: Reward 48.0
Episode 301: Reward 21.0
Episode 302: Reward 23.0
Episode 303: Reward 24.0
Episode 304: Reward 26.0
Episode 305: Reward 59.0
Episode 306: Reward 19.0
Episode 307: Reward 30.0
Episode 308: Reward 16.0
Episode 309: Reward 18.0
Episode 310: Reward 16.0
Episode 311: Reward 55.0
Episode 312: Reward 31.0
Episode 313: Reward 29.0
Episode 314: Reward 17.0
Episode 315: Reward 21.0
Episode 316: Reward 16.0
Episode 317: Reward 15.0
Episode 318: Reward 26.0
Episode 319: Reward 24.0
Episode 320: Reward 10.0
Episode 321: Reward 18.0
Episode 322: Reward 14.0
Episode 323: Reward 18.0
Episode 324: Reward 30.0
Episode 325: Reward 37.0
Episode 326: Reward 17.0
Episode 327: Reward 18.0
Episode 328: Reward 12.0
Episode 329: Reward 32.0
Episode 330: Reward 31.0
Episode 331: Reward 9.0
Episode 332: Reward 14.0
Episode 333: Reward 28.0
Episode 334: Reward 34.0
Episode 335: Reward 31.0
Episode 336: Reward 23.0
Episode 337: Reward 17.0
Episode 338: Reward 40.0
Episode 339: Reward 22.0
Episode 340: Reward 14.0
Episode 341: Reward 16.0
Episode 342: Reward 18.0
Episode 343: Reward 26.0
Episode 344: Reward 51.0
Episode 345: Reward 21.0
Episode 346: Reward 11.0
Episode 347: Reward 59.0
Episode 348: Reward 27.0
Episode 349: Reward 19.0
Episode 350: Reward 72.0
Episode 351: Reward 17.0
Episode 352: Reward 19.0
Episode 353: Reward 31.0
Episode 354: Reward 38.0
Episode 355: Reward 14.0
Episode 356: Reward 13.0
Episode 357: Reward 21.0
Episode 358: Reward 24.0
Episode 359: Reward 14.0
Episode 360: Reward 17.0
Episode 361: Reward 16.0
Episode 362: Reward 15.0
Episode 363: Reward 8.0
Episode 364: Reward 25.0
Episode 365: Reward 17.0
Episode 366: Reward 39.0
Episode 367: Reward 52.0
Episode 368: Reward 28.0
Episode 369: Reward 11.0
Episode 370: Reward 53.0
Episode 371: Reward 21.0
Episode 372: Reward 22.0
Episode 373: Reward 12.0
Episode 374: Reward 45.0
Episode 375: Reward 42.0
Episode 376: Reward 17.0
Episode 377: Reward 23.0
Episode 378: Reward 35.0
Episode 379: Reward 23.0
Episode 380: Reward 14.0
Episode 381: Reward 10.0
Episode 382: Reward 19.0
Episode 383: Reward 26.0
Episode 384: Reward 15.0
Episode 385: Reward 19.0
Episode 386: Reward 42.0
Episode 387: Reward 13.0
Episode 388: Reward 17.0
Episode 389: Reward 29.0
Episode 390: Reward 35.0
Episode 391: Reward 44.0
Episode 392: Reward 16.0
Episode 393: Reward 53.0
Episode 394: Reward 25.0
Episode 395: Reward 20.0
Episode 396: Reward 15.0
Episode 397: Reward 12.0
Episode 398: Reward 13.0
Episode 399: Reward 12.0
Episode 400: Reward 64.0
Episode 401: Reward 23.0
Episode 402: Reward 32.0
Episode 403: Reward 20.0
Episode 404: Reward 27.0
Episode 405: Reward 25.0
Episode 406: Reward 21.0
Episode 407: Reward 10.0
Episode 408: Reward 16.0
Episode 409: Reward 21.0
Episode 410: Reward 21.0
Episode 411: Reward 15.0
Episode 412: Reward 17.0
Episode 413: Reward 24.0
Episode 414: Reward 21.0
Episode 415: Reward 15.0
Episode 416: Reward 27.0
Episode 417: Reward 47.0
Episode 418: Reward 20.0
Episode 419: Reward 12.0
Episode 420: Reward 19.0
Episode 421: Reward 25.0
Episode 422: Reward 15.0
Episode 423: Reward 12.0
Episode 424: Reward 19.0
Episode 425: Reward 19.0
Episode 426: Reward 43.0
Episode 427: Reward 16.0
Episode 428: Reward 34.0
Episode 429: Reward 31.0
Episode 430: Reward 62.0
Episode 431: Reward 14.0
Episode 432: Reward 12.0
Episode 433: Reward 18.0
Episode 434: Reward 33.0
Episode 435: Reward 18.0
Episode 436: Reward 15.0
Episode 437: Reward 55.0
Episode 438: Reward 51.0
Episode 439: Reward 16.0
Episode 440: Reward 17.0
Episode 441: Reward 50.0
Episode 442: Reward 44.0
Episode 443: Reward 17.0
Episode 444: Reward 43.0
Episode 445: Reward 22.0
Episode 446: Reward 14.0
Episode 447: Reward 11.0
Episode 448: Reward 16.0
Episode 449: Reward 44.0
Episode 450: Reward 15.0
Episode 451: Reward 32.0
Episode 452: Reward 26.0
Episode 453: Reward 25.0
Episode 454: Reward 12.0
Episode 455: Reward 17.0
Episode 456: Reward 11.0
Episode 457: Reward 17.0
Episode 458: Reward 13.0
Episode 459: Reward 31.0
Episode 460: Reward 23.0
Episode 461: Reward 26.0
Episode 462: Reward 33.0
Episode 463: Reward 29.0
Episode 464: Reward 22.0
Episode 465: Reward 24.0
Episode 466: Reward 20.0
Episode 467: Reward 17.0
Episode 468: Reward 19.0
Episode 469: Reward 20.0
Episode 470: Reward 28.0
Episode 471: Reward 35.0
Episode 472: Reward 31.0
Episode 473: Reward 16.0
Episode 474: Reward 18.0
Episode 475: Reward 26.0
Episode 476: Reward 20.0
Episode 477: Reward 13.0
Episode 478: Reward 28.0
Episode 479: Reward 96.0
Episode 480: Reward 43.0
Episode 481: Reward 37.0
Episode 482: Reward 13.0
Episode 483: Reward 13.0
Episode 484: Reward 17.0
Episode 485: Reward 18.0
Episode 486: Reward 23.0
Episode 487: Reward 30.0
Episode 488: Reward 21.0
Episode 489: Reward 16.0
Episode 490: Reward 15.0
Episode 491: Reward 30.0
Episode 492: Reward 24.0
Episode 493: Reward 18.0
Episode 494: Reward 25.0
Episode 495: Reward 11.0
Episode 496: Reward 41.0
Episode 497: Reward 51.0
Episode 498: Reward 43.0
Episode 499: Reward 16.0
Episode 500: Reward 35.0
Episode 501: Reward 14.0
Episode 502: Reward 12.0
Episode 503: Reward 15.0
Episode 504: Reward 27.0
Episode 505: Reward 26.0
Episode 506: Reward 21.0
Episode 507: Reward 54.0
Episode 508: Reward 13.0
Episode 509: Reward 35.0
Episode 510: Reward 19.0
Episode 511: Reward 16.0
Episode 512: Reward 16.0
Episode 513: Reward 24.0
Episode 514: Reward 15.0
Episode 515: Reward 10.0
Episode 516: Reward 12.0
Episode 517: Reward 23.0
Episode 518: Reward 15.0
Episode 519: Reward 33.0
Episode 520: Reward 54.0
Episode 521: Reward 39.0
Episode 522: Reward 26.0
Episode 523: Reward 31.0
Episode 524: Reward 31.0
Episode 525: Reward 17.0
Episode 526: Reward 21.0
Episode 527: Reward 15.0
Episode 528: Reward 20.0
Episode 529: Reward 20.0
Episode 530: Reward 20.0
Episode 531: Reward 22.0
Episode 532: Reward 13.0
Episode 533: Reward 32.0
Episode 534: Reward 11.0
Episode 535: Reward 13.0
Episode 536: Reward 18.0
Episode 537: Reward 28.0
Episode 538: Reward 14.0
Episode 539: Reward 25.0
Episode 540: Reward 28.0
Episode 541: Reward 21.0
Episode 542: Reward 15.0
Episode 543: Reward 24.0
Episode 544: Reward 13.0
Episode 545: Reward 18.0
Episode 546: Reward 26.0
Episode 547: Reward 15.0
Episode 548: Reward 53.0
Episode 549: Reward 25.0
Episode 550: Reward 10.0
Episode 551: Reward 19.0
Episode 552: Reward 13.0
Episode 553: Reward 12.0
Episode 554: Reward 34.0
Episode 555: Reward 19.0
Episode 556: Reward 23.0
Episode 557: Reward 16.0
Episode 558: Reward 38.0
Episode 559: Reward 18.0
Episode 560: Reward 17.0
Episode 561: Reward 39.0
Episode 562: Reward 54.0
Episode 563: Reward 19.0
Episode 564: Reward 23.0
Episode 565: Reward 17.0
Episode 566: Reward 14.0
Episode 567: Reward 32.0
Episode 568: Reward 25.0
Episode 569: Reward 60.0
Episode 570: Reward 30.0
Episode 571: Reward 33.0
Episode 572: Reward 16.0
Episode 573: Reward 16.0
Episode 574: Reward 20.0
Episode 575: Reward 24.0
Episode 576: Reward 11.0
Episode 577: Reward 10.0
Episode 578: Reward 17.0
Episode 579: Reward 10.0
Episode 580: Reward 28.0
Episode 581: Reward 18.0
Episode 582: Reward 37.0
Episode 583: Reward 16.0
Episode 584: Reward 12.0
Episode 585: Reward 11.0
Episode 586: Reward 22.0
Episode 587: Reward 34.0
Episode 588: Reward 47.0
Episode 589: Reward 15.0
Episode 590: Reward 62.0
Episode 591: Reward 14.0
Episode 592: Reward 13.0
Episode 593: Reward 52.0
Episode 594: Reward 27.0
Episode 595: Reward 63.0
Episode 596: Reward 12.0
Episode 597: Reward 53.0
Episode 598: Reward 16.0
Episode 599: Reward 26.0
Episode 600: Reward 18.0
Episode 601: Reward 32.0
Episode 602: Reward 17.0
Episode 603: Reward 26.0
Episode 604: Reward 12.0
Episode 605: Reward 22.0
Episode 606: Reward 12.0
Episode 607: Reward 13.0
Episode 608: Reward 27.0
Episode 609: Reward 31.0
Episode 610: Reward 27.0
Episode 611: Reward 15.0
Episode 612: Reward 17.0
Episode 613: Reward 19.0
Episode 614: Reward 35.0
Episode 615: Reward 17.0
Episode 616: Reward 18.0
Episode 617: Reward 59.0
Episode 618: Reward 10.0
Episode 619: Reward 25.0
Episode 620: Reward 12.0
Episode 621: Reward 15.0
Episode 622: Reward 26.0
Episode 623: Reward 43.0
Episode 624: Reward 14.0
Episode 625: Reward 18.0
Episode 626: Reward 16.0
Episode 627: Reward 42.0
Episode 628: Reward 23.0
Episode 629: Reward 11.0
Episode 630: Reward 13.0
Episode 631: Reward 11.0
Episode 632: Reward 19.0
Episode 633: Reward 11.0
Episode 634: Reward 32.0
Episode 635: Reward 14.0
Episode 636: Reward 33.0
Episode 637: Reward 23.0
Episode 638: Reward 13.0
Episode 639: Reward 10.0
Episode 640: Reward 20.0
Episode 641: Reward 11.0
Episode 642: Reward 28.0
Episode 643: Reward 35.0
Episode 644: Reward 14.0
Episode 645: Reward 30.0
Episode 646: Reward 28.0
Episode 647: Reward 29.0
Episode 648: Reward 19.0
Episode 649: Reward 15.0
Episode 650: Reward 20.0
Episode 651: Reward 16.0
Episode 652: Reward 14.0
Episode 653: Reward 35.0
Episode 654: Reward 14.0
Episode 655: Reward 20.0
Episode 656: Reward 42.0
Episode 657: Reward 25.0
Episode 658: Reward 86.0
Episode 659: Reward 50.0
Episode 660: Reward 34.0
Episode 661: Reward 24.0
Episode 662: Reward 24.0
Episode 663: Reward 50.0
Episode 664: Reward 21.0
Episode 665: Reward 38.0
Episode 666: Reward 15.0
Episode 667: Reward 14.0
Episode 668: Reward 15.0
Episode 669: Reward 18.0
Episode 670: Reward 21.0
Episode 671: Reward 12.0
Episode 672: Reward 25.0
Episode 673: Reward 22.0
Episode 674: Reward 18.0
Episode 675: Reward 33.0
Episode 676: Reward 19.0
Episode 677: Reward 21.0
Episode 678: Reward 17.0
Episode 679: Reward 10.0
Episode 680: Reward 29.0
Episode 681: Reward 10.0
Episode 682: Reward 22.0
Episode 683: Reward 10.0
Episode 684: Reward 17.0
Episode 685: Reward 11.0
Episode 686: Reward 19.0
Episode 687: Reward 13.0
Episode 688: Reward 23.0
Episode 689: Reward 18.0
Episode 690: Reward 9.0
Episode 691: Reward 12.0
Episode 692: Reward 33.0
Episode 693: Reward 13.0
Episode 694: Reward 54.0
Episode 695: Reward 10.0
Episode 696: Reward 15.0
Episode 697: Reward 22.0
Episode 698: Reward 18.0
Episode 699: Reward 16.0
Episode 700: Reward 58.0
Episode 701: Reward 19.0
Episode 702: Reward 16.0
Episode 703: Reward 28.0
Episode 704: Reward 43.0
Episode 705: Reward 10.0
Episode 706: Reward 16.0
Episode 707: Reward 10.0
Episode 708: Reward 16.0
Episode 709: Reward 68.0
Episode 710: Reward 16.0
Episode 711: Reward 39.0
Episode 712: Reward 11.0
Episode 713: Reward 19.0
Episode 714: Reward 26.0
Episode 715: Reward 24.0
Episode 716: Reward 28.0
Episode 717: Reward 39.0
Episode 718: Reward 12.0
Episode 719: Reward 11.0
Episode 720: Reward 23.0
Episode 721: Reward 16.0
Episode 722: Reward 32.0
Episode 723: Reward 13.0
Episode 724: Reward 10.0
Episode 725: Reward 15.0
Episode 726: Reward 17.0
Episode 727: Reward 11.0
Episode 728: Reward 19.0
Episode 729: Reward 11.0
Episode 730: Reward 11.0
Episode 731: Reward 65.0
Episode 732: Reward 32.0
Episode 733: Reward 21.0
Episode 734: Reward 24.0
Episode 735: Reward 24.0
Episode 736: Reward 23.0
Episode 737: Reward 51.0
Episode 738: Reward 15.0
Episode 739: Reward 10.0
Episode 740: Reward 15.0
Episode 741: Reward 14.0
Episode 742: Reward 56.0
Episode 743: Reward 13.0
Episode 744: Reward 10.0
Episode 745: Reward 15.0
Episode 746: Reward 24.0
Episode 747: Reward 9.0
Episode 748: Reward 19.0
Episode 749: Reward 22.0
Episode 750: Reward 23.0
Episode 751: Reward 23.0
Episode 752: Reward 23.0
Episode 753: Reward 14.0
Episode 754: Reward 16.0
Episode 755: Reward 8.0
Episode 756: Reward 47.0
Episode 757: Reward 41.0
Episode 758: Reward 18.0
Episode 759: Reward 12.0
Episode 760: Reward 21.0
Episode 761: Reward 29.0
Episode 762: Reward 27.0
Episode 763: Reward 21.0
Episode 764: Reward 13.0
Episode 765: Reward 33.0
Episode 766: Reward 14.0
Episode 767: Reward 19.0
Episode 768: Reward 13.0
Episode 769: Reward 12.0
Episode 770: Reward 23.0
Episode 771: Reward 21.0
Episode 772: Reward 13.0
Episode 773: Reward 43.0
Episode 774: Reward 11.0
Episode 775: Reward 19.0
Episode 776: Reward 27.0
Episode 777: Reward 23.0
Episode 778: Reward 10.0
Episode 779: Reward 33.0
Episode 780: Reward 38.0
Episode 781: Reward 35.0
Episode 782: Reward 39.0
Episode 783: Reward 19.0
Episode 784: Reward 45.0
Episode 785: Reward 11.0
Episode 786: Reward 19.0
Episode 787: Reward 13.0
Episode 788: Reward 14.0
Episode 789: Reward 15.0
Episode 790: Reward 14.0
Episode 791: Reward 13.0
Episode 792: Reward 16.0
Episode 793: Reward 13.0
Episode 794: Reward 23.0
Episode 795: Reward 52.0
Episode 796: Reward 25.0
Episode 797: Reward 12.0
Episode 798: Reward 63.0
Episode 799: Reward 56.0
Episode 800: Reward 29.0
Episode 801: Reward 12.0
Episode 802: Reward 9.0
Episode 803: Reward 11.0
Episode 804: Reward 32.0
Episode 805: Reward 12.0
Episode 806: Reward 16.0
Episode 807: Reward 23.0
Episode 808: Reward 27.0
Episode 809: Reward 11.0
Episode 810: Reward 13.0
Episode 811: Reward 16.0
Episode 812: Reward 38.0
Episode 813: Reward 46.0
Episode 814: Reward 23.0
Episode 815: Reward 30.0
Episode 816: Reward 24.0
Episode 817: Reward 21.0
Episode 818: Reward 28.0
Episode 819: Reward 11.0
Episode 820: Reward 23.0
Episode 821: Reward 23.0
Episode 822: Reward 19.0
Episode 823: Reward 29.0
Episode 824: Reward 14.0
Episode 825: Reward 14.0
Episode 826: Reward 20.0
Episode 827: Reward 35.0
Episode 828: Reward 14.0
Episode 829: Reward 21.0
Episode 830: Reward 14.0
Episode 831: Reward 46.0
Episode 832: Reward 16.0
Episode 833: Reward 15.0
Episode 834: Reward 18.0
Episode 835: Reward 14.0
Episode 836: Reward 26.0
Episode 837: Reward 13.0
Episode 838: Reward 12.0
Episode 839: Reward 16.0
Episode 840: Reward 47.0
Episode 841: Reward 33.0
Episode 842: Reward 18.0
Episode 843: Reward 11.0
Episode 844: Reward 47.0
Episode 845: Reward 22.0
Episode 846: Reward 14.0
Episode 847: Reward 12.0
Episode 848: Reward 39.0
Episode 849: Reward 36.0
Episode 850: Reward 29.0
Episode 851: Reward 15.0
Episode 852: Reward 88.0
Episode 853: Reward 25.0
Episode 854: Reward 12.0
Episode 855: Reward 19.0
Episode 856: Reward 11.0
Episode 857: Reward 22.0
Episode 858: Reward 28.0
Episode 859: Reward 13.0
Episode 860: Reward 33.0
Episode 861: Reward 13.0
Episode 862: Reward 15.0
Episode 863: Reward 20.0
Episode 864: Reward 14.0
Episode 865: Reward 42.0
Episode 866: Reward 42.0
Episode 867: Reward 22.0
Episode 868: Reward 16.0
Episode 869: Reward 14.0
Episode 870: Reward 22.0
Episode 871: Reward 24.0
Episode 872: Reward 16.0
Episode 873: Reward 30.0
Episode 874: Reward 26.0
Episode 875: Reward 19.0
Episode 876: Reward 19.0
Episode 877: Reward 19.0
Episode 878: Reward 13.0
Episode 879: Reward 43.0
Episode 880: Reward 19.0
Episode 881: Reward 34.0
Episode 882: Reward 19.0
Episode 883: Reward 18.0
Episode 884: Reward 24.0
Episode 885: Reward 15.0
Episode 886: Reward 19.0
Episode 887: Reward 22.0
Episode 888: Reward 43.0
Episode 889: Reward 14.0
Episode 890: Reward 15.0
Episode 891: Reward 12.0
Episode 892: Reward 12.0
Episode 893: Reward 10.0
Episode 894: Reward 19.0
Episode 895: Reward 10.0
Episode 896: Reward 17.0
Episode 897: Reward 30.0
Episode 898: Reward 11.0
Episode 899: Reward 14.0
Episode 900: Reward 16.0
Episode 901: Reward 18.0
Episode 902: Reward 23.0
Episode 903: Reward 11.0
Episode 904: Reward 15.0
Episode 905: Reward 29.0
Episode 906: Reward 38.0
Episode 907: Reward 14.0
Episode 908: Reward 13.0
Episode 909: Reward 13.0
Episode 910: Reward 19.0
Episode 911: Reward 12.0
Episode 912: Reward 21.0
Episode 913: Reward 14.0
Episode 914: Reward 18.0
Episode 915: Reward 14.0
Episode 916: Reward 14.0
Episode 917: Reward 27.0
Episode 918: Reward 15.0
Episode 919: Reward 12.0
Episode 920: Reward 19.0
Episode 921: Reward 27.0
Episode 922: Reward 37.0
Episode 923: Reward 50.0
Episode 924: Reward 27.0
Episode 925: Reward 25.0
Episode 926: Reward 13.0
Episode 927: Reward 38.0
Episode 928: Reward 34.0
Episode 929: Reward 15.0
Episode 930: Reward 23.0
Episode 931: Reward 15.0
Episode 932: Reward 31.0
Episode 933: Reward 18.0
Episode 934: Reward 13.0
Episode 935: Reward 28.0
Episode 936: Reward 36.0
Episode 937: Reward 10.0
Episode 938: Reward 22.0
Episode 939: Reward 14.0
Episode 940: Reward 17.0
Episode 941: Reward 28.0
Episode 942: Reward 11.0
Episode 943: Reward 23.0
Episode 944: Reward 11.0
Episode 945: Reward 16.0
Episode 946: Reward 17.0
Episode 947: Reward 78.0
Episode 948: Reward 15.0
Episode 949: Reward 38.0
Episode 950: Reward 22.0
Episode 951: Reward 16.0
Episode 952: Reward 20.0
Episode 953: Reward 19.0
Episode 954: Reward 17.0
Episode 955: Reward 22.0
Episode 956: Reward 15.0
Episode 957: Reward 15.0
Episode 958: Reward 25.0
Episode 959: Reward 42.0
Episode 960: Reward 29.0
Episode 961: Reward 12.0
Episode 962: Reward 17.0
Episode 963: Reward 21.0
Episode 964: Reward 15.0
Episode 965: Reward 13.0
Episode 966: Reward 38.0
Episode 967: Reward 36.0
Episode 968: Reward 19.0
Episode 969: Reward 18.0
Episode 970: Reward 11.0
Episode 971: Reward 17.0
Episode 972: Reward 22.0
Episode 973: Reward 18.0
Episode 974: Reward 13.0
Episode 975: Reward 22.0
Episode 976: Reward 17.0
Episode 977: Reward 76.0
Episode 978: Reward 19.0
Episode 979: Reward 10.0
Episode 980: Reward 29.0
Episode 981: Reward 25.0
Episode 982: Reward 28.0
Episode 983: Reward 37.0
Episode 984: Reward 12.0
Episode 985: Reward 30.0
Episode 986: Reward 15.0
Episode 987: Reward 16.0
Episode 988: Reward 24.0
Episode 989: Reward 26.0
Episode 990: Reward 25.0
Episode 991: Reward 46.0
Episode 992: Reward 64.0
Episode 993: Reward 12.0
Episode 994: Reward 19.0
Episode 995: Reward 23.0
Episode 996: Reward 16.0
Episode 997: Reward 57.0
Episode 998: Reward 14.0
Episode 999: Reward 10.0