archwall 把架构画全Every architecture, drawn in full

Dream-RSI：拿旧记录当模拟器，在梦里改策略Dream-RSI: dream on old runs to explore better

agentsself-improvementexplorationsimulator

Dream-RSI overview: 1 explore online: the exploration policy, which is code, decides where, how many and when to stop, a coding agent makes one try per pick, a fixed evaluator scores every try; 2 store the tree: every try and its real score go into a simulator pool that grows by one tree a round, and replay reruns nothing; 3 dream: a policy-development agent rewrites the policy code, each version is replayed on every tree at zero executions, and scores and traces guide the next rewrite, M versions with the current one included; 4 deploy the best: the best replay score wins, never worse on replay than the current policy, and the new policy runs the next round

AI 做研究要试很多轮。算力花得值不值，看探索：从哪分叉、几路并行、何时停。这套策略通常写死，想换就得整轮重跑。

Dream-RSI 来自谷歌，想法是：跑完的记录就是模拟器。代码未发布，以下据论文。

发现树。一个节点是一次尝试：工作区、程序、分数。选根开新分支，选叶子往下走一步。

Discovery tree: 1 one node is one try: the saved workspace after the try, the program and proposal the agent wrote, the evaluator's score and diagnostics, and each try starts from its parent's workspace; 2 how a tree grows: the root is the starting workspace, branches hang off it, each a chain of tries, and only the root and leaves can be picked; 3 what a pick does: the root opens a new branch, a leaf adds the next step, a refine or a repair, up to W picks a round, one per worker, and recorded nodes never change; 4 the pool: one more tree each round, which the next run reads for context while it grows a brand-new tree
发现树：每个节点是一次真实的尝试和它的分数

1. 在线探索。策略每轮最多选 W 个节点。编码智能体读完旧尝试再接着做，评估器打分。树进池子。

Step 1, one online run: 1 pick: the exploration policy, code that stays fixed for the run, picks a batch of the root or leaves, up to W, where the root opens a new branch and a leaf takes the next step; 2 try: a coding agent resumes the parent's workspace, reads every past try, wins and failures, then writes a new attempt, a program and a proposal, Gemini CLI in the tests, one per worker, and never kills unrelated processes; 3 score: a fixed evaluator gives a score and diagnostics, and the result becomes a new child node; 4 again or stop: the next round picks from the root and leaves, the run stops on an empty pick or after K1 rounds, and the finished tree joins the pool
在线探索：策略选点，智能体动手，评估器打分，树一轮轮长大

2. 回放。新策略换种走法走旧树，结果现成，不用重跑。分数看最好结果、尝试次数、并行程度。

Step 2, replay: 1 start: a recorded tree from the pool that never changes, and the policy starts fresh on every tree seeing only the root; 2 pick: the policy picks a batch of the root or leaves, up to W, using only what it has seen, and hidden scores stay hidden; 3 reveal: a picked leaf reveals its recorded next step, a picked root reveals the next unopened branch, nothing runs and results come off disk, then the policy picks again; 4 stop and score: replay stops on an empty pick, after K2 rounds or when the whole tree is seen, and the replay score counts the best result, the tries and the batching, charging for every node revealed and rewarding batches
回放：新策略在旧树上重走一遍，一次都不用真跑

3. 做梦。另一智能体把策略改 M 版，每版在所有树上回放，分数指导下一版。最高分上线。当前版也参赛，回放分不会变差。

Step 3, dream: 1 start: the current policy is version 0, and the pool of t trees stays fixed while dreaming; 2 score: each version is replayed on every tree at zero executions and gets an average score; 3 rewrite: a policy-development agent reads the traces and scores plus feedback from earlier versions and writes the next version of the policy code, which goes back to be scored; 4 pick: after M versions the best average score is deployed for the next online run, and since version 0 competes too, the pick is never worse on replay of these trees
做梦：改 M 版，每版回放打分，挑最好的上线

策略是代码，还定下一轮多宽多深。做梦时只看已揭开的部分。模型和评估器不变。

Policy and rules: 1 what the policy decides: which node next, how many at once (up to W), when to stop, and how wide and deep the next run's grid is; it is code, rewritten by an agent between rounds; 2 one knob, beta: high beta is wider and more patient, low beta makes fewer tries and stops early, fixed during a run with the default re-chosen between rounds; 3 rules while dreaming, set in the policy-development agent's prompt: it sees only what it revealed, copies no scores or node ids, reads traces only between rounds, and the agent edits only the policy file; 4 what never changes: the coding agent and model get no training, and the evaluator and tools stay the same; only the policy code changes
策略管四件事，做梦时只能看已揭开的部分

结果。八个任务，对照固定策略。Lasso 上调用少 1.7 倍，比 SimpleTES 少 162 倍。

Results: 1 the test: the same start policy, parallel refine, so round 1 is identical; fixed exploration never changes the policy, Dream-RSI rewrites it between rounds; same agent, evaluator and budget; cost is total agent calls; 2 Lasso path solver: Gemini-3.1 Pro over 5 rounds, 317 calls against 550, average runtime 2931 ms against 3587 ms on 6 held-out datasets, 162 times fewer calls than SimpleTES (51,200), faster than sklearn and glmnet on all 6; 3 math, 3 problems: sum-difference 1.145427, the best listed, circle packing 2.635983, tied best, autocorrelation competitive but not best, under 1,000 generations against 51,200 for SimpleTES; 4 GPU kernels, 4 KernelBench tasks with Gemini-3.1 Pro: VGG16 2.43 times and LayerNorm 1.79 times fewer generations for the same result, ConvDiv 2.09 times and ConvMax 1.44 times higher on the same budget
结果：三个领域八个任务，对照组是策略固定的同一套流程

学到的策略涨时省、停时多花。把历史写成提示，不如回放。

Learned behaviour on the ConvDiv kernel task: 1 scores climb: the round-best score rises and the policy spends less, from 110 to 50 tries a round; 2 progress stalls: the score flattens and the policy spends more with wider rounds, and the next score jumps follow those wider rounds; 3 history as tips: past runs turned into tips in the prompt did worse than no tips, for both the fixed policy and Dream-RSI, because tips narrow the search; 4 history as a world: replaying the history, as Dream-RSI does, beats the tips on the same budget
学到的策略会看情况省或花；回放历史胜过把历史写成提示

AI research runs many rounds of tries. Exploring decides whether that compute pays off: where to branch, how many lines at once, when to stop. That strategy is usually hand-written. Changing it means a whole new run.

Dream-RSI, from Google, has one idea: a finished run is already a simulator. The code isn't out yet; this is from the paper.

The tree. A node is one try: workspace, program, score. Picking the root opens a new branch; picking a leaf takes the next step.

Discovery tree: 1 one node is one try: the saved workspace after the try, the program and proposal the agent wrote, the evaluator's score and diagnostics, and each try starts from its parent's workspace; 2 how a tree grows: the root is the starting workspace, branches hang off it, each a chain of tries, and only the root and leaves can be picked; 3 what a pick does: the root opens a new branch, a leaf adds the next step, a refine or a repair, up to W picks a round, one per worker, and recorded nodes never change; 4 the pool: one more tree each round, which the next run reads for context while it grows a brand-new tree
The discovery tree: each node is one real try and its score

1. Explore online. Each round the policy picks up to W nodes. A coding agent reads the past tries, then continues from each. An evaluator scores it. The tree joins the pool.

Step 1, one online run: 1 pick: the exploration policy, code that stays fixed for the run, picks a batch of the root or leaves, up to W, where the root opens a new branch and a leaf takes the next step; 2 try: a coding agent resumes the parent's workspace, reads every past try, wins and failures, then writes a new attempt, a program and a proposal, Gemini CLI in the tests, one per worker, and never kills unrelated processes; 3 score: a fixed evaluator gives a score and diagnostics, and the result becomes a new child node; 4 again or stop: the next round picks from the root and leaves, the run stops on an empty pick or after K1 rounds, and the finished tree joins the pool
Explore online: the policy picks, the agent works, the evaluator scores, the tree grows

2. Replay. A new policy walks an old tree its own way. Results are saved, so nothing reruns. The score counts the best result, the tries spent and how parallel they ran.

Step 2, replay: 1 start: a recorded tree from the pool that never changes, and the policy starts fresh on every tree seeing only the root; 2 pick: the policy picks a batch of the root or leaves, up to W, using only what it has seen, and hidden scores stay hidden; 3 reveal: a picked leaf reveals its recorded next step, a picked root reveals the next unopened branch, nothing runs and results come off disk, then the policy picks again; 4 stop and score: replay stops on an empty pick, after K2 rounds or when the whole tree is seen, and the replay score counts the best result, the tries and the batching, charging for every node revealed and rewarding batches
Replay: a new policy walks an old tree without running anything

3. Dream. Another agent rewrites the policy M times. Each version is replayed on every tree, and its scores guide the next. The top scorer ships. The current one competes too, so the replay score never drops.

Step 3, dream: 1 start: the current policy is version 0, and the pool of t trees stays fixed while dreaming; 2 score: each version is replayed on every tree at zero executions and gets an average score; 3 rewrite: a policy-development agent reads the traces and scores plus feedback from earlier versions and writes the next version of the policy code, which goes back to be scored; 4 pick: after M versions the best average score is deployed for the next online run, and since version 0 competes too, the pick is never worse on replay of these trees
Dream: M versions, each scored by replay, the best one ships

The policy is code. It also sets how wide and deep the next run goes. While dreaming it only sees what it has revealed. The model and evaluator never change.

Policy and rules: 1 what the policy decides: which node next, how many at once (up to W), when to stop, and how wide and deep the next run's grid is; it is code, rewritten by an agent between rounds; 2 one knob, beta: high beta is wider and more patient, low beta makes fewer tries and stops early, fixed during a run with the default re-chosen between rounds; 3 rules while dreaming, set in the policy-development agent's prompt: it sees only what it revealed, copies no scores or node ids, reads traces only between rounds, and the agent edits only the policy file; 4 what never changes: the coding agent and model get no training, and the evaluator and tools stay the same; only the policy code changes
The policy decides four things, and while dreaming it sees only what it has revealed

Results. Eight tasks, against a fixed policy. On Lasso: 1.7× fewer calls, and 162× fewer than SimpleTES.

Results: 1 the test: the same start policy, parallel refine, so round 1 is identical; fixed exploration never changes the policy, Dream-RSI rewrites it between rounds; same agent, evaluator and budget; cost is total agent calls; 2 Lasso path solver: Gemini-3.1 Pro over 5 rounds, 317 calls against 550, average runtime 2931 ms against 3587 ms on 6 held-out datasets, 162 times fewer calls than SimpleTES (51,200), faster than sklearn and glmnet on all 6; 3 math, 3 problems: sum-difference 1.145427, the best listed, circle packing 2.635983, tied best, autocorrelation competitive but not best, under 1,000 generations against 51,200 for SimpleTES; 4 GPU kernels, 4 KernelBench tasks with Gemini-3.1 Pro: VGG16 2.43 times and LayerNorm 1.79 times fewer generations for the same result, ConvDiv 2.09 times and ConvMax 1.44 times higher on the same budget
Results: eight tasks in three areas, against the same loop with a fixed policy

The learned policy saves while scores climb and spends more when they stall. Turning history into prompt tips did worse than replay.

Learned behaviour on the ConvDiv kernel task: 1 scores climb: the round-best score rises and the policy spends less, from 110 to 50 tries a round; 2 progress stalls: the score flattens and the policy spends more with wider rounds, and the next score jumps follow those wider rounds; 3 history as tips: past runs turned into tips in the prompt did worse than no tips, for both the fixed policy and Dream-RSI, because tips narrow the search; 4 history as a world: replaying the history, as Dream-RSI does, beats the tips on the same budget
The learned policy saves or spends as needed; replaying history beats turning it into tips

来源Source: dream-rsi.com

← 全部架构← All architectures