World models: technical cheat sheet

Core loop

An environment emits an observation o_t. An encoder maps it to a latent state z_t; a dynamics model predicts the next latent state after action a_t:

z_t = e(o_t)

p(z_{t+1}, r_t, done_t | z_t, a_t)

A policy chooses actions, and a planner evaluates imagined rollouts under the learned dynamics.

Three model-based RL families

• Predictive latent model: reconstruct or predict observations and rewards; useful for imagination.
• Value/planning model: predict only quantities needed by search, such as reward, value, and policy (MuZero).
• Model-predictive control: repeatedly plan a short horizon, execute one action, observe the real result, then re-plan.

Why latent space?

Pixels contain details irrelevant to control. A good latent state preserves task-relevant information while discarding nuisance variation. The dangerous failure is aliasing: two observations map to similar latents even though they have different future consequences.

Main failure modes

• Compounding error: one-step prediction errors become bad long-horizon rollouts.
• Distribution shift: the policy visits states absent from the model's training data.
• Stochasticity collapse: a deterministic prediction averages incompatible futures.
• Reward hacking: the model predicts an easy-to-exploit reward artifact.