Scaling the reward down,
did not make it learn better.奖励缩小 7 倍，
学得并没有更好。
Harry Xin · Exploratory, not confirmatory探索性研究，非验证性20 runs · 6,000,000 episodes20 次运行 · 600 万局
Does shrinking every reward to one seventh make a poker agent learn more stably? Two NFSP set-ups, the original terminal reward and the same reward divided by 7, were trained on the same 10 random seeds for 300,000 episodes each. At the end of training, the scaled reward was easier to exploit on all 10 seeds (mean +10.0%), and worse over the whole run on 9 of 10. The original reward stays as the baseline. The extension was chosen after seeing earlier results, so it is reported as exploratory evidence only.把每次奖励缩小为原来的七分之一，能否让扑克 AI 学得更稳定？研究让两种设置（原始终局奖励，以及除以 7 的奖励）使用相同的 10 个随机种子，各训练 30 万局。训练结束时，缩放奖励在全部 10 个种子上都更容易被针对（平均高 10.0%）；按整个训练过程计算，10 组中有 9 组更差。因此继续以原始奖励为基线。这次延长实验是在看过前期结果后才决定的，只作为探索性证据报告。
More reward,
is not more learning.奖励的数字变大，
学习不一定变好。
Changing the size or shape of a reward also changes the optimisation problem the agent actually faces. The study asks whether reward scaling, or potential-based reward shaping (PBRS), helps an agent learn a poker strategy that is harder to exploit.改变奖励的大小或形式，也会改变智能体实际面对的优化问题。这项研究要回答：缩放奖励或势函数奖励塑形（PBRS），能否帮助智能体学到更不容易被针对的扑克策略。
- Leduc Hold’em
- A small poker game used in research: six cards, two betting rounds. Small enough to compute exact exploitability.研究中常用的小型扑克：六张牌、两轮下注。规模足够小，可以精确计算可利用度。
- NFSP
- Neural Fictitious Self-Play: two agents learn by playing each other.神经虚拟自我对弈：两个智能体通过互相对局来学习。
- Exploitability可利用度
- How much a best-responding opponent can win against the strategy. Lower is better.一个针对性最强的对手能从这个策略身上赢走多少。数值越低越好。
Two studies,
one environment.两项实验，
同一个环境。
| Mechanism screen机制实验 | Extended run延长实验 | |
|---|---|---|
| Conditions条件 | Terminal · scaled · scaled + PBRS · unscaled + PBRS终局 · 缩放 · 缩放 + PBRS · 未缩放 + PBRS | Terminal · reward ÷ 7原始奖励 · 奖励 ÷ 7 |
| Seeds随机种子 | 8 per condition每种条件 8 个 | 10, shared by both arms10 个，两组共用 |
| Runs运行次数 | 32 | 20 × 300,000 episodes20 次 × 30 万局 |
| Metric指标 | Exact exploitability: at the final checkpoint, and as the area under the curve over training精确可利用度：最终检查点的数值，以及训练全程的曲线下面积 | |
All ten seeds,
moved the same way.十个随机种子，
方向完全一致。
Each line joins one seed’s two runs. At 300,000 episodes the scaled reward is worse on every seed; the paired mean difference is 0.118 (bootstrap 95% interval 0.078 to 0.155; exact sign-flip p = 0.00195). Over the whole run, seed 57721 is the one exception.每条线连接同一个随机种子的两次运行。在 30 万局时，缩放奖励在每个种子上都更差；配对平均差为 0.118（bootstrap 95% 区间 0.078–0.155；精确符号翻转检验 p = 0.00195）。按训练全程计算，随机种子 57721 是唯一的例外。
Pick any seed, in Figure 1 or in Table 2, and it is highlighted in both.点选任意一个随机种子，图 1 和表 2 中的同一个种子会一起高亮。
| Seed种子 | Terminal原始 | ÷ 7÷ 7 | Difference差值 |
|---|---|---|---|
| 31415 | 1.166 | 1.310 | +0.145 |
| 27182 | 1.203 | 1.304 | +0.101 |
| 16180 | 1.108 | 1.331 | +0.223 |
| 57721 | 1.213 | 1.341 | +0.128 |
| 65537 | 1.151 | 1.317 | +0.167 |
| 99991 | 1.184 | 1.201 | +0.017 |
| 104729 | 1.196 | 1.297 | +0.101 |
| 13007 | 1.234 | 1.242 | +0.008 |
| 17011 | 1.156 | 1.326 | +0.170 |
| 23003 | 1.185 | 1.305 | +0.119 |
| Mean均值 | 1.180 | 1.298 | +0.118 |
A clear result,
with a clear boundary.结论清楚，
边界也要清楚。
Why it is only exploratory为什么只算探索性证据
The extended run was decided after the pilot results were seen, and it used seeds that had been held in reserve. The pilot itself promoted no reward scheme; scaling was picked for follow-up only because it had the lowest pilot AUC among the non-control arms and is the simplest change. A confirmation would need a new pre-registered plan and unused seeds.延长实验是在看过 Pilot 结果之后才决定的，而且用了原本预留的随机种子。Pilot 本身没有让任何奖励方案晋级；选择缩放奖励继续，只因为它是非对照组中 Pilot 全程指标最低、变换也最简单的一种。若要做验证，需要新的预先登记方案和从未用过的随机种子。
What follows接下来怎么做
- Keep保留The original reward stays as the baseline.继续以原始奖励作为基线。
- Drop不晋级Reward scaling showed no practical advantage.缩放奖励没有显示实际优势。
- Scope适用范围The result holds for this Leduc Hold’em, NFSP and training setup only.结论只适用于这次的 Leduc Hold’em、NFSP 与训练设置。
One laptop GPU,
reproducible runs.一张笔记本显卡，
可复现地训练。
The study runs on its own harness around the RLCard library, on a single RTX 4060 Laptop GPU.研究在 RLCard 库之上自建了一套实验框架，在一张 RTX 4060 笔记本显卡上运行。
- Four kinds of agent四类智能体DQN against a random agent, two-agent NFSP self-play, CFR, and RandomAgent as a reference.DQN（对随机对手）、双智能体 NFSP 自我对弈、CFR，以及作为参照的随机智能体。
- Resumable training断点续训An interrupted run writes a checkpoint and can resume from it; only self-generated checkpoints are loaded.训练中断时会写入检查点并可从中恢复；只加载本项目自己生成的检查点。
- Fair evaluation公平评估Seat-swapped round robins of 20,000 games, with bootstrap intervals, per-game raw data and a bilingual report.交换座位的 2 万局循环对战，附 bootstrap 区间、逐局原始数据和双语报告。
- A caution built in内置的提醒Beating a random agent only shows that a model exploits random play, not that it is near a Nash equilibrium.战胜随机对手只说明模型学会了利用随机策略，并不代表接近纳什均衡。
What the record,
says about the method.回头看，
方法上的不足。
The pilot did not promote any reward scheme. The extended run was decided only after seeing it, and it used seeds that had been held back. Both facts are written into the published report, and they are the reason this result can only be called exploratory, however consistent the ten seeds look.Pilot 没有让任何奖励方案晋级。延长实验是看过 Pilot 之后才决定的，用的还是原本预留的随机种子。这两点都写在已公开的报告里，也正是这个结果无论十个种子看上去多一致，都只能称为探索性证据的原因。
Done properly, a confirmation would start from a written plan, fixed before any data, and from seeds nobody has looked at.要做严格的验证，应当在看到任何数据之前写好方案，并使用没人看过的新随机种子。