Evaluating chat assistants with diverse simulated users.
An assistant can solve the task. Does it also serve the person who asked?
People who bring the same request to an assistant can need different kinds of help: plain explanations, the reasoning behind each question, a clear recommendation, or reassurance before deciding. Evaluations built around a single cooperative user can't show how reliably an assistant serves different people. PersonaChatBench holds the task fixed and varies only the user.
A source task with its native evaluator: a clinical case, a programming problem, a math problem, a financial question, a policy case, or a trip, prepared with the facts the user knows.
A simulated user follows a private persona card. The assistant never sees the card; it learns the user's needs only from what the user says.
Native task outcome, fulfillment of expressed needs, and simulator fidelity are each scored on their own and never combined into one number.
A need only counts once the user has voiced it. For every need, the judge asks:
Each domain has its own compact panel of personas, built around traits that matter for seeking help there: willingness to disclose in a medical consultation, explanation preferences in programming, trust and decision style in travel. Coding and Math profiles draw on MatrAIx attributes. On each card, task gap is the spread between the highest- and lowest-scoring user conditions in mean task score, with every model weighted equally; needs low/high are the lowest- and highest-scoring personas on expressed-needs satisfaction.
Task gaps are from the paper's Figure 3; Travel's reward is kept out of that comparison, so its card shows the pooled persona-minus-neutral change in choice reward (Table 22). Needs ranges are from each domain's appendix condition table and its own model cohort (Tables 7, 9, 15, 17, 22); for Math, the paper reports help-preference satisfaction for hint-first versus demonstration-first students. Appendix figures can differ slightly from the main-text numbers quoted under Findings.
Stage 1: each task is split into what the assistant sees, what the simulated user knows, and what only the evaluators see. Stage 2: the user, guided by a private persona card, talks with the assistant; in five domains every condition continues from the same opening question and first assistant reply. Stage 3: a native task evaluator, a persona-needs judge, and a simulator auditor each score the dialogue on their own.
Public task context for the assistant; authorized facts for the user, disclosed only through dialogue; hidden tests, reference answers, and rubrics kept for grading.
Persona and neutral users continue from a shared opening and first reply (Math generates both per persona). Users ask follow-ups, disclose facts, clarify preferences, or stop naturally before the turn limit.
Native task score with the source's own tests or rubric; needs scored 1, 0.5, or 0 and averaged on a 0–100 scale; a simulator audit of persona and fact fidelity.
| Role | Information used | Output |
|---|---|---|
| Tested assistant | Public task context, dialogue, permitted tools and policies | Replies, tool calls, or a final task artifact |
| User simulator | Public dialogue, authorized user facts, assigned persona content | A user message or a decision to stop |
| Native task evaluator | The artifact or conversation the source rubric requires, with its tests or references | Native task result |
| Persona-needs judge | Public dialogue and pre-recorded needs with their criteria | Per-need judgments and supporting evidence |
| Simulator auditor | Dialogue, authorized task facts, and assigned persona | Findings about simulator fidelity |
Figure 1 — The PersonaChatBench protocol and the information available to each role (after the paper's Section 3 and Table 4). The assistant never sees the persona card; the simulator never sees hidden answers, tests, rubrics, or judge feedback. LLM grading and simulator auditing primarily use GPT-6 Luna at medium reasoning effort. Across five domains and 20 assistants, the simulator-audit pass rate averages 95.70%.
Claude Opus 5.5 leads the five-domain task average at 80.53, followed by Fable 5.1 (79.57) and GPT-6 Astra (77.66). GPT-6 Astra leads the six-domain needs average at 91.27, and GPT models hold the first four needs positions. Astra ranks third on task and first on needs; Opus first and fifth; Fable second and eighth.
With the problem fixed and all 20 models weighted equally, the gap between the highest- and lowest-scoring user condition is:
Single models swing further: on 29 IMO-ProofBench tasks, GPT-5.5 scores 86.21 with the example-led explorer and 68.97 with the concise independent learner.
On matched cases, the receptive, informed contributor gets higher inquiry coverage than the private, time-conscious novice for all 20 assistants: 7.08 task points, with a 15.93-point gap in patient facts acquired. Responding is not eliciting: GPT-6 Sol scores 98.54 on needs but 41.00 on inquiry.
Of 4,594 persona conversations with passing code and a needs score, 2,014 leave at least one scored need partly or wholly unmet: 22.3% for the detailed typed professional, but 66.7% for the plain-language technologist and 64.9% for the examples-first learner.
Help-preference satisfaction averages 75.3 for hint-first students and 86.6 for demonstration-first students, and demonstrations score higher for all 20 tutors. The 11.3-point gap holds even when all six persona conversations reach correct mathematics.
Skeptical and contrarian clients are associated with a 9.0-percentage-point larger loss of previously earned final-position rubric items (95% interval [6.3, 12.4]). On one hedging question, 16 of the 17 initially rubric-preferred answers are withdrawn after the challenge; Astra's task score falls from 64.8 to 53.4 while all three expressed needs get full credit.
The self-doubting customer has the highest task mean (78.71) and the lowest needs mean (37.13): completing the workflow need not include checking the doubted fact against the record. Astra completes 94.70% of self-doubting conversations but 71.97% with customers who ask for reasons.
Needs satisfaction separates the skeptical comparer from the concise decider by 33.76 points. Postponing a choice can earn full needs credit and zero task reward; recording a rental car the user wanted kept provisional earns full task credit and zero needs credit. Kimi K3 asks for confirmation first and gets full credit on both.
The paper names three priorities for user adaptation:
Per-model scores (0–100) from the paper's domain appendices: Health inquiry coverage (Table 6), Coding native pass rate (Table 8), Finance expert-rubric percent (Table 16), Customer Service rule-based completion (Table 18), and persona-needs satisfaction for each, plus Travel. Bold marks the best value per column within each domain's main cohort. Models are listed in the paper's Appendix B.1 order.
† simulator and judges not all GPT-6 Luna at medium effort; ‡ score-only package without full dialogue evidence. Both are outside that domain's main cohort and are not counted for bold. Coverage differs across models; for example, GPT-6 Astra's Coding needs score rests on 24 of 600 planned persona episodes. Travel's choice reward is left out of the task view, as in the paper, because it mainly records whether the assistant commits. Math per-model scores and the cross-domain averages appear only in the paper's Table 1 and 2 graphics and are not reproduced here; the main-text Travel needs values also differ from Table 23 by up to 1.4 points.
@ARTICLE{2026arXiv261042060D,
author = {{Deng}, Zheyuan and {Li}, Taoran and {Zhang}, Jingwen and {Li}, Yuante and {Zhao}, Dachuan and {Zhang}, Fulei and {Zhang}, Yunkai and {Xu}, Zhikun and {Liao}, Yajie and {Zhang}, Haichao and {Hu}, Haodi and {Wang}, Cong and {Yang}, Chen and {Wu}, Siyang and {Cao}, Defu and {Hu}, Xu and {Li}, Haoming and {Wang}, Haixin and {George}, Robert and {Zhang}, Weizhi and {Wang}, Yuchu and {Peng}, Zhejian and {Li}, Yueying and {Jiang}, Wanting and {Liang}, Shushu and {Yu}, Zhiyuan and {Wang}, Yifan and {Huang}, Jintao and {Wang}, Yijun and {Hao}, Yuexing and {Li}, Xiaomin},
title = "{PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users}",
journal = {arXiv e-prints},
keywords = {Computation and Language},
year = 2026,
month = oct,
eid = {arXiv:2610.42060},
pages = {arXiv:2610.42060},
archivePrefix = {arXiv},
eprint = {2610.42060},
primaryClass = {cs.CL}
}
All numbers on this page are from the PersonaChatBench paper (arXiv:2610.42060).