Science · Validation
Reality is our benchmark.
A synthetic population is not validated because it looks realistic.
It is validated when it can be confronted with reality, the gap can be measured, and its limitations can be documented.
Imagine All The People validates its simulations at several levels: population structure, individual responses, comparative decisions and, progressively, dynamics over time.
From population structure to collective behavior.
Synthetic population validity is not a single metric.
We distinguish four levels, each answering a different question.
These statuses describe the state of the evidence, not the availability of features.
Peer review of a generation method, published in Frontiers in Artificial Intelligence, does not automatically validate every behavioral simulation produced with it.
- Level 1 — Statistical reconstruction of populationsDocumented benchmark results
Does the synthetic population accurately represent the real population?
We measure the reproduction of statistical distributions and their dependencies: individual variables, pairwise relationships and multidimensional constraints.
The objective is not only to reproduce each variable independently, but to preserve the structures connecting them.
- Level 2 — Individual responsesEvaluation in progress
Do synthetic individuals answer like real populations?
The same question is asked to a real population and its synthetic counterpart.
The responses used to measure performance are not used to calibrate the expected outcome.
We then compare response distributions, differences across segments and the direction of observed effects.
- Level 3 — Decision comparisonProgressive validation
Does the system lead to the same decision signal as the real world?
Correctly reproducing a response distribution is not enough.
When several alternatives are tested, we evaluate whether the simulation reproduces their ranking, the direction of effects and differences across segments.
- Is A preferred to B?
- Which segment reacts the most?
- In which direction does the result move?
- Is the observed difference large enough to change the decision?
- Level 4 — Temporal dynamicsProspective validation in development
Do simulated dynamics withstand the test of time?
A society does not simply produce answers.
Individuals observe, react, influence one another and change.
Dynamic validation compares simulated trajectories with subsequently observed real-world developments: adoption, opposition, diffusion, polarization or behavioral change.
Run reality and simulation in parallel.
The principle is simple.
- Same question.
- Same target population.
- Same moment.
- Two environments.
On one side, a study conducted with real people.
On the other, the same study conducted on a synthetic population built independently of the field results.
Only then are the results compared.
Real-world fieldwork
OBSERVED- Real population
- Same protocol
- Real-world results
Synthetic population
SIMULATED- Synthetic population
- Same protocol
- Simulated results
Comparison
BACKTESTED- Distribution by distribution
- Segment by segment
- Question by question
- Decision by decision
We do not measure whether the simulation looks plausible.We measure the gap between simulation and reality.
From the past to the future.
- 01 — Backtesting
Replay what already happened.
We use a situation whose real-world outcome is known but hidden from the system at the time of simulation.
The simulated result is then compared with what actually happened.
QuestionWould the model have read the situation correctly?
- 02 — Parallel testing
Test reality and simulation simultaneously.
A real-world study and its synthetic equivalent are run in parallel.
No field result is used to adjust the simulation before comparison.
QuestionDo the two systems produce convergent results?
- 03 — Prospective validation
Produce the result before reality exists.
The simulation is run and its results are frozen.
When the real-world outcome becomes observable, the two trajectories are compared.
QuestionWhat could we actually have anticipated?
There is no magic “realism” score.
A high correlation on one question does not validate an entire model.
A statistically faithful population does not automatically guarantee behavioral validity.
Accurate individual responses do not automatically establish the validity of collective dynamics.
Each level must therefore be measured independently.
- Populationerror against reference distributions
- Responsesgap between real and synthetic responses
- Decisionsconsistency of scenario rankings
- Segmentsperformance differences across subpopulations
- Dynamicsgap between simulated and observed trajectories
- Robustnessstability of conclusions when assumptions change
EVIDENCE GRAMMAR
Every result tells you where it comes from.
At Imagine All The People, a number is never presented simply as a “result”. We distinguish between what was observed in the real world, produced by simulation, compared with reality, or used purely for illustration.
- OBSERVED
- result measured in the real world
- SIMULATED
- result from the simulation
- BACKTESTED
- simulation compared with reality
- ILLUSTRATIVE
- product demonstration
Measuring the distance between simulation and reality.
Published results on population construction apply to a defined protocol and reproducible measurements.
Validation of responses, decisions and dynamics is documented separately according to its maturity level.
A result always describes a specific protocol. It does not constitute a general guarantee of accuracy for every population, question or situation.
See the protocol and limitationsKnowing where simulation works. And where it does not.
We do not treat validation as an exercise designed to prove that the system always works.
It must also identify situations in which it performs less well.
Limitation
Some questions rely on genuinely lived experience that cannot necessarily be reconstructed accurately from a synthetic population.
Limitation
Performance demonstrated in one domain cannot automatically be transferred to another.
Limitation
Strong validity for static responses does not in itself establish dynamic validity.
Limitation
Long-term projections carry more uncertainty than short-horizon tests.
Validity is always tied to a protocol, a population, a question and a time horizon.
AI agents
The same validation levels.
The same validation levels apply when Imagine All The People is used through an AI agent or agentic system.
A successful technical integration shows that the agent can use the service. It does not, by itself, constitute additional validation of behavioral accuracy.
Evaluating an agentic system is a separate question.
Synthetic interlocutors can be used to build test scenarios. An agent’s quality should nevertheless be assessed against criteria suited to its mission and, where needed, compared with human or field results.
Our results must be open to verification.
- We publish the methods used to construct populations.
- We document our benchmarks.
- We distinguish published results from work still in progress.
- We preserve protocols that allow simulations to be reproduced.
- And we continuously seek to confront our results with new real-world data.
A simulation system that cannot be proven wrong cannot be scientifically validated.
Parallel testing
Run your next study twice.
Planning a real-world study?
Run Imagine All The People in parallel without sharing the field results with our system.
When the study is complete, compare the two.
Same brief.Same questions.Reality vs synthetic.
Run a parallel test