We think the main reason today’s models are good at completing tasks but bad at making decisions is that they never have to live with the consequences of their actions. The purpose of our simulated worlds is to understand longer-term decision making, as well as the outcomes it leads to, and this requires the other agents in the world to be as realistic as possible. We built Delos, our graph-based system for generating high-fidelity population distributions that track real people and places.
Delos takes a real county and produces a cast of personas that match it demographically and politically, along with the town they live in, and it validates the whole cast on the models that will play them before any simulation runs. We start from real places because everything that happens in a world grows out of its starting population, so a cast that is narrow or generic will produce a narrow world regardless of how good the engine is. Hand-written casts have the same problem from the other direction, since they drift toward whatever a model finds easy to play, and characters written in the same style end up sounding alike. Using real counties gives us diversity we can trust, and because the underlying data is fixed, it also gives us a standard to validate the cast against.
Our data is a mix of public records and datasets we license from third parties. Since this data describes real people, all of it is de-identified at ingestion and stripped of personal information before it reaches a model. No persona corresponds to a real individual and personas are statistical constructions that match populations in aggregate, with names generated from real naming patterns and speech taken from licensed recordings, transcribed as recorded.
Delos coordinates all of this work through a graph-based orchestration system. Research, sampling, calibration, authoring, and validation all run as parts of one connected graph, and each part can draw on the outputs of the others while it works. This allows each persona to be written with full knowledge of the county’s research. The population distributions come from a variety of quality public datasets, including census microdata and the Cooperative Election Study, and we calibrate the cast jointly so that its compositional demographics and turnout reproduce the county’s recorded results in backtests. A typical county dossier grows to hundreds of sourced claims before any persona is written.
Every stage is checked as it runs to improve casting fidelity. Various statistical gates examine composition, calibration, variance, and leakage, among other properties. Similarity checks compare every pair of personas so that no two read alike, and critic models review each persona against the underlying record. We then evaluate every persona on every model in the cast matrix. Though this is computationally expensive, it allows for very high fidelity casting. Our evaluations measure, via psychological and behavioral tests, how the persona behaves under persuasion, persona drift, etc. Any persona that does not pass our standards is re-cast to maintain the fidelity of our simulation.
The payoff is a world where conflict comes from people with real incentives, and where an agent’s decisions come back to it through neighbors and employers that behave like their real counterparts, which is the environment long-horizon planning and continual learning need to develop. Persona fidelity becomes a measured quantity for every model we run and licensed data keeps the cast accurate whenever the public record is thin.
Delos builds the world, Terrarium runs it, and we study what the agents inside do. The parts that have to be exact are deterministic, and the open-ended work is done by models, which means the whole pipeline improves every time models do, and a new world only requires pointing the same system at a new county. We think such systems are worth investing in as they materially improve with each model iteration, allowing our simulations to similarly scale in fidelity.