AI has made a striking transition from generating answers to taking action. Coding agents can inspect repositories, edit files, run commands, and debug their own work. Computer-use agents can navigate interfaces and complete multi-step tasks. Capability is advancing quickly. But capability alone does not answer a more human question: capable for whom?
Selected results from SWE-bench, METR, Turing Experiments, and MT-Bench/Chatbot Arena. These studies measure different constructs and should not be read as one leaderboard.1, 3, 4, 5
1. Capability is not preference
Ask a coding agent to implement a feature and two users may want opposite results. One wants explanatory comments; another considers them noise. One values explicit, readable functions; another prefers a compact and clever abstraction. Both can be competent engineers. Neither preference is universally correct.
The same problem appears everywhere. A financial assistant may need to be cautious for one person and opportunity-seeking for another. A health application may need detailed evidence for one user and a short, reassuring summary for another. A shopping assistant should not recommend the same product to a student on a tight budget and a professional optimizing for durability.
Today's personalization is often shallow: a remembered name, a preferred format, a few lines of custom instructions. The deeper challenge is behavioral. Can an agent represent what a person values, what they know, how they make decisions, what constraints they face, and how those factors should change its actions over time?
We believe this is a natural next step for AI: give agents personas. A persona here is a structured, inspectable representation of a user, one that goes beyond a decorative biography or a role-playing prompt. It conditions how an agent interprets a situation, weighs trade-offs, communicates, and acts.
A credible persona also needs boundaries. It should distinguish observed evidence from inference, attach provenance and confidence to attributes, and leave unsupported fields unknown. Conditioning an agent on that representation is the first step. Demonstrating that the agent behaves differently for the right reasons, and not because it is reproducing a stereotype, is the harder one.
2. The measurement gap: benchmark success is not user success
Offline benchmarks are essential because they are repeatable, inexpensive to rerun, and easy to compare. SWE-bench, for example, gives an agent a real GitHub issue and codebase and asks it to generate a resolving patch. SWE-bench Verified contains 500 engineer-confirmed solvable problems and uses a containerized harness for reproducibility.1 This is valuable capability evidence. But the harness observes a narrow slice of product value.
From executable correctness to lived usefulness
Conceptual comparison. “Partial” means the signal can be captured only when the method is explicitly designed or instrumented to do so.
Real software work exposes what the benchmark omits: prior repository knowledge, implicit quality standards, negotiation over requirements, interruptions, review practices, and subjective judgments about readability or trust. METR's early-2025 randomized study made the discrepancy concrete. Sixteen experienced open-source developers worked on 246 real issues from repositories they knew; AI access made completion 19% slower in that specific historical setting, while developers expected a 24% speedup and still perceived a 20% speedup afterward.2
That number should not be frozen into a claim about current tools. METR now explicitly labels it out of date. Its late-2025 follow-up covered 57 developers, 143 repositories, and more than 800 tasks, but wider AI adoption introduced severe selection and time-measurement problems. METR concluded that the experiment gave an unreliable signal of the current productivity effect.3
The lesson is methodological, not anti-agent: benchmark scores, controlled experiments, self-reports, and production outcomes answer different questions. A test suite can verify a patch. It cannot tell us whether an engineer enjoyed steering the agent, trusted the implementation, accepted its style, or would choose it again. Task completion is one part of satisfaction, and the only part existing benchmarks measure. It is necessary, but nowhere near sufficient. For a product, satisfaction is the outcome that decides whether people come back, and completion is just the floor of it.
3. Why simulation is becoming plausible
Human simulation is not a new ambition. Social science has long used agent-based models; HCI has long used personas; product research has long recruited representative participants. What changes with language-model agents is the possibility of combining natural-language identity, memory, interactive action, and scalable execution in one system.
Four empirical signals from human-simulation research
Turing Experiments reproduced three established findings, but the fourth revealed a “hyper-accuracy distortion,” showing that simulation can be systematically wrong.4
Strong LLM judges matched human preferences at roughly inter-human agreement, while retaining position, verbosity, self-enhancement, and reasoning biases.5
Silicon samples conditioned on real survey participants reproduced nuanced subgroup response patterns, motivating the testable idea of algorithmic fidelity.6
Automated experiments in negotiation, bail, interviews, and auctions recovered useful effect directions, while magnitude predictions remained unreliable.7
These results do not establish a general human emulator. They establish something narrower and more useful: under some conditions, models preserve enough structure in human responses to support controlled approximations, and those approximations have measurable failure modes. Argyle et al. call the target algorithmic fidelity: whether conditioning a model on human backstories reproduces patterns found across actual subpopulations, not merely whether an answer sounds believable.6
A simulation can be eloquent but distributionally wrong. It can reproduce a mean while erasing variance, imitate a demographic stereotype while missing within-group diversity, or predict the sign of an effect but not its magnitude. In automated social-science experiments, Manning et al. found that LLM-based simulations could recover useful causal structure in negotiation, bail, interview, and auction scenarios, yet direct model elicitation could not reliably predict effect magnitudes.7
4. Two targets: populations and individuals
Once agents can behave as different people rather than as one generic user, two research directions become possible. They operate at different scales, but share the same foundation.
Population simulation
Run diverse persona-conditioned agents as a synthetic population to study variation, disagreement, subgroup effects, and possible collective outcomes.
Individual simulation
Build a persistent model of one consenting user so a personal agent can anticipate preferences, follow habits, and act on that person's behalf.
Population simulation: rehearsing decisions before reality
Many consequential decisions require feedback from more than one kind of person. Product teams want to know which groups will struggle with a new workflow. Researchers want to test how different messages might be understood. Policymakers want to anticipate where a proposal may create confusion, resistance, or unequal effects. Market forecasters care not only about an average response, but about how responses interact across a population.
Today, that feedback is difficult to obtain early and repeatedly. Surveys and panels take time. Samples can miss small but important groups. Running a new study after every product or policy change is rarely practical. A population of persona-conditioned agents could act as a rehearsal layer: vary the population, run the same scenario many times, inspect where behavior diverges, and identify questions that deserve real-world study.
The goal is not an oracle that predicts society; human behavior is contextual, adaptive, and shaped by events no model can fully observe. What simulation offers is more modest and more useful: a map of plausible responses, subgroup differences, sensitivities, and failure modes. It should help us ask better questions before we make irreversible decisions.
Individual simulation: from assistant to representative
At the other end of the scale is an agent designed around one person. Imagine a digital twin that learns, with explicit consent and control, how a user writes, organizes work, evaluates options, and handles recurring decisions. It could draft code in the user's preferred style, triage messages according to their priorities, prepare a decision in the format they trust, or complete routine tasks without repeatedly asking for the same context.
This changes the role of a personal assistant. Instead of merely responding to instructions, it becomes a representative: an agent that can act in ways the user recognizes as their own. That promise also creates serious obligations. A personal simulation requires privacy, provenance, editable memory, uncertainty, revocation, and clear boundaries on when the agent may act. A system that claims to know a person must also know what it does not know.
The evidence bar for an individual simulation must be higher than for a population aggregate. Matching subgroup distributions does not prove that an agent can represent one person across unseen contexts. A digital twin should therefore be evaluated longitudinally against that user's own choices, corrections, and changing preferences, with explicit abstention whenever the evidence does not support a prediction.
| Target | Primary validity question | Primary risk |
|---|---|---|
| Population | Does the simulated distribution preserve real subgroup patterns and treatment effects? | False confidence, stereotype amplification, omitted populations |
| Individual | Does the agent predict this person's response across time and unseen contexts? | Privacy loss, identity capture, unauthorized delegation |
5. Why product evaluation needs simulated humans
The immediate value of human simulation may be evaluation. The product under test could be physical, a website or app, an AI model, or an autonomous agent. In every case, technical correctness is only part of the outcome. A product succeeds when real people can use it, trust it, and feel that it serves their goals.
Most evaluation today sits between two imperfect endpoints.
Offline benchmarks are fast and repeatable, but simplify reality. Consider a coding agent evaluated on a repository benchmark. The benchmark may verify whether tests pass, yet real usage is interactive: users clarify goals, reject an approach, change constraints, or stop when the agent becomes difficult to steer. A benchmark rarely captures whether the code is readable to that team, whether the agent communicated well, or whether the user felt confident accepting the result. Rule-based metrics measure what can be formalized; user satisfaction often depends on what cannot.
Online experiments are realistic, but expensive and risky. A/B testing requires an existing user base, enough traffic, careful instrumentation, and time. Human behavior is noisy, so subtle effects require scale. More importantly, the experiment exposes real people to an unfinished system. For high-stakes products, learning from failure may mean that users experience the failure first. Startups face an additional problem: a product with no users cannot run a meaningful user experiment.
Persona-conditioned simulation creates a layer between these endpoints. Before a live test, teams can let many simulated users interact with each version, collect full trajectories, compare outcomes across personas, and search for likely points of frustration. Unlike a static benchmark, the evaluation can be multi-turn and adaptive. Unlike an A/B test, it can run before launch, repeat after every change, and deliberately include users who are rare or absent from the current customer base.
Simulated users should not be presented as real public opinion or used to replace human validation. Their role is to expose assumptions, identify likely failures, compare alternatives under controlled conditions, and focus scarce human studies on the questions that matter most.
6. A three-layer evaluation stack
Rather than choosing between benchmarks, simulated users, and real users, we see them as three layers of one evaluation system:
- Benchmarks test defined capabilities cheaply and consistently.
- Persona simulations test interaction, diversity, preference, and behavior across controlled user populations.
- Human studies and live experiments provide calibration, discovery, and evidence about what people actually do.
Each layer catches failures the others miss. Benchmarks establish whether a system can work. Simulation asks for whom it works and under what conditions. Real users determine whether those simulated conclusions survive contact with reality.
What each layer can claim
| Layer | Best question | Scale / speed | Human realism | Claim supported |
|---|---|---|---|---|
| 1 · Offline benchmark | Can the system complete a defined task? | High / fast | Low–medium | Capability under benchmark conditions |
| 2 · Persona simulation | For whom might it work, and where could interaction fail? | High / fast | Modeled | Hypotheses, comparative risk, coverage |
| 3 · Human study / A/B | What do people actually do in context? | Limited / slow | Observed | Human response; causal lift if randomized |
For a coding agent, Layer 1 can run repository tests. Layer 2 can ask simulated developers with different expertise and style preferences to steer the agent through multi-turn work, then measure corrections, abandonment, trust, and acceptance. Layer 3 recruits real developers or runs a production experiment. The same structure applies to chatbots, websites, physical products represented through prototypes, and AI models themselves.
This middle layer is especially valuable for startups. A team without traffic cannot run a powered A/B test. A team with traffic may still not want to expose users to an obviously weak or unsafe variant. Simulation can pre-screen options and find failures before the live experiment. It cannot establish real conversion lift because simulated agents are not randomized visitors from the production population.
The economic difference can be large. In our separate transparent cost analysis, a hypothetical 10,000-response human concept survey costs at least $28,560 using public fair-pay and platform rates, while a two-pass text-model simulation was estimated at $3.60–$104.40 before engineering, calibration, and review. The ratio is not equivalence: the cheap run buys modeled evidence, not 10,000 human opinions.8
7. What would make a human simulation credible?
Scale is not validity. Running one biased agent ten thousand times produces a precise estimate of the wrong distribution. A credible system needs a validation ladder that tests the persona, the behavior, and the downstream claim separately.
Evidence required before deployment claims
Attach source, consent status, timestamp, and confidence to persona attributes; leave unsupported fields unknown.
Test whether relevant attributes alter behavior and irrelevant attributes do not; check consistency over long interactions.
Compare simulated and human marginals, correlations, subgroup gaps, variance, and tails, not only average accuracy.
Test whether treatment effects transfer when products, prompts, policies, or environments change.
Validate the exact operational claim and monitor drift after the model, population, or environment changes.
Several tests are especially important. Counterfactual sensitivity asks whether changing a relevant attribute changes behavior in the expected direction. Identity consistency asks whether an agent maintains preferences after many turns and environmental pressure. Model robustness asks whether findings survive different base models, prompts, temperatures, and judge models. Human calibration compares a stratified subset against people, including groups most likely to be misrepresented.
Evaluation also needs abstention. If a persona lacks evidence about a preference, the correct simulation may be “unknown,” not a plausible invention. If models disagree sharply, uncertainty should propagate into the result. If the claim concerns high-stakes health, employment, credit, or public policy, the threshold for human evidence must rise.
8. Failure modes are part of the research object
Giving an agent a persona does not guarantee human-like behavior. Models may ignore assigned traits, exaggerate stereotypes, become inconsistent over long interactions, or produce confident explanations unsupported by evidence. Population simulations can create an illusion of precision simply by generating many responses. Individual simulations can become invasive when convenience outruns consent.
Human simulation therefore needs more than larger models. It needs grounded persona data, explicit provenance, behavioral validation, uncertainty estimates, privacy controls, and continuous calibration against human evidence. We must test whether changing a relevant persona attribute changes behavior in the expected direction, whether the same persona remains coherent over time, and where the simulation stops being reliable.
The strongest argument against careless simulation is that models can sound convincing while compressing human diversity. The Turing Experiment result is instructive: three classic findings replicated, but the Wisdom of Crowds experiment exposed a hyper-accuracy distortion. The model did not fail randomly; it failed in a systematic direction.4
LLM-as-a-judge research shows the same duality. Strong judges achieved over 80% agreement with humans in MT-Bench and Chatbot Arena, but the study also documented position, verbosity, self-enhancement, and reasoning biases.5 A simulator and judge drawn from the same model family may share blind spots and reward each other's style.
Individual twins add governance risks beyond statistical validity. A model of a person can reveal sensitive traits, freeze an outdated identity, or be used after consent is withdrawn. The more useful the twin becomes, the more consequential unauthorized access becomes. Privacy must shape data minimization, storage, access control, retention, and action permissions.
9. The MatrAIx research agenda
We are not claiming that language models already simulate humanity. Our claim is narrower: human simulation has become empirically testable and operationally useful enough to deserve its own infrastructure and standards.
- Persona populations: typed, evidence-aware representations with compatibility constraints, provenance, uncertainty, and coverage measurement.
- Persona-conditioned agents: mechanisms that convert identity and context into stable, testable behavioral differences rather than cosmetic role-play.
- Interactive environments: survey, conversation, web, app, and multi-agent settings where behavior unfolds over time and produces observable trajectories.
- Evaluation and calibration: adherence, subgroup fidelity, treatment-effect transfer, uncertainty, drift, and comparison with held-out human evidence.
This is the motivation behind MatrAIx. We are building persona populations, agent environments, and evaluation tools to study how diverse simulated users behave before systems meet the real world. The ambition is the opposite of reducing people to profiles: it is to make human difference a first-class part of how AI systems are designed and evaluated.
References
- SWE-bench documentation; Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024.
- Becker et al., “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, METR, 2025. Historical result; METR states that it is now out of date.
- Becker et al., “We are Changing our Developer Productivity Experiment Design”, METR, 2026.
- Aher, Arriaga, and Kalai, “Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies”, ICML 2023.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, NeurIPS 2023.
- Argyle et al., “Out of One, Many: Using Language Models to Simulate Human Samples”, Political Analysis 31(3), 2023.
- Manning, Zhu, and Horton, “Automated Social Science: Language Models as Scientist and Subjects”, arXiv:2404.11794, 2024.
- MatrAIx, “The Price of a Synthetic User”, 2026. Illustrative scenarios with public price assumptions and explicit limitations.
Evidence note. This is a position paper, not a report of a completed MatrAIx validation study. Figures 1, 3, and 4 are conceptual frameworks proposed by the authors. Figure 2 summarizes independent studies with different tasks, populations, models, and outcome definitions; its values must not be treated as directly comparable performance scores.