World modelsLesson 4openfrom distillate
Dream-RSI: Replay Worlds for Recursive Self-Improvement
Dream-RSI turns completed discovery histories into replay simulators that test exploration policies before spending money on new online searches.
World models · agentic discovery · recursive self-improvement
Dream-RSI addresses a meta-level problem: not how to improve the solution generator directly, but how to improve the policy that decides which branches to explore, how much to parallelize, and when to stop.
The unusual “world model”
In Dreamer, a world model is a learned parametric simulator. Dream-RSI uses a different construction: a discovery tree records workspaces, decisions, candidate results, and scores. That finite tree becomes an empirical replay world.
Online policy → discovery agent → evaluated candidates → recorded discovery tree → replay simulator → policy improvement → deploy online again.
What is being improved?
The policy chooses a batch of eligible tree nodes. A batch controls which branches to continue and how many workers run in parallel. The paper evaluates policies using a quality–cost tradeoff: find high-scoring candidates with fewer agent calls, while avoiding excessive sequential rounds.
Root workspace branches into recorded candidate attempts. A replayed policy chooses which recorded branches to reveal, possibly skipping a known dead end.
Online phase versus dreaming phase
| Phase | What changes? | What it costs |
|---|---|---|
| Online exploration | The agent generates and evaluates new candidates; the tree grows. | Expensive agent calls and execution. |
| Offline dreaming | A candidate policy traverses already-recorded branches. | Mostly replay and scoring; no new candidate generation. |
| Redeployment | The best replay-scoring policy becomes the next online policy. | One new online rollout validates the improvement. |
Why this is recursive self-improvement
Each online round expands the history. That larger history creates more replay worlds. Those worlds provide more feedback for revising the exploration policy. The revised policy then determines what new evidence enters the next generation of the simulator pool.
history = []
policy = initial_policy
repeat:
tree = explore_online(policy)
history.append(tree)
candidates = revise_policy_by_replaying(policy, history)
policy = best_replay_scoring_candidate(candidates)
Dream-RSI versus Dreamer
Both systems improve behavior through cheap internal “dreaming,” but their simulators differ:
- Dreamer: learns a neural latent transition model that can generate new hypothetical states.
- Dream-RSI: replays a finite tree of already-observed discovery outcomes and changes how that tree is traversed.
This makes Dream-RSI closer to planning over an empirical search graph than to learning a conventional physics-like world model.
Retrieval check
What makes Dream-RSI’s replay simulator different from Dreamer’s world model?
Primary source
Zheng et al., Dream-RSI: Recursive Self-Improvement through Evolving Worlds. The paper was submitted in September 2026 and revised in October 2026; this lesson follows its v2 description.
Next: Ask me to unpack the replay objective, especially its quality–cost hypervolume and parallelism penalty.