PersonaScienceBench

The collaborator is part of the task.

Scientific work is a conversation. Researchers bring different expertise, preferences, and ways of explaining what they need, yet benchmarks usually hold the collaborator fixed. PersonaScienceBench holds the science fixed and varies only the researcher; the agent meets those differences through the conversation.

Beyond the native score, we ask three questions of every model:

Four researchers, one control. Hover to meet them.

The simulated researchers are drawn from the public Persona8B coreset released by MatrAIx (599,847 human-grounded and 400,000 synthetic records), chosen by a coverage-based strategy over communication, decision-making, interaction, tool-collaboration, and workflow attributes. Only the researcher simulator receives the profile; persona off omits it and keeps the same protocol. Numbers below are GPT-6 Astra's mean normalized score (0–100) under each setting: the domain-weighted average and physical sciences & engineering.

Profiles are selected contrasts, not a representative sample of researchers. Persona diversity does not, by itself, establish human realism.

The researcher holds the context; the actor learns it through conversation.

Stage 1: each source task is split into public inputs for the scientific actor and researcher-held context for the simulator. Stage 2: the researcher simulator, with persona off or one of four injected profiles, answers questions and gives feedback while the actor asks, proposes, and uses sandboxed tools where applicable. Stage 3: the submission is graded by the source benchmark's native evaluator. Actor and simulator share a model, and GPT-6 Astra with low reasoning effort performs all model-based evaluations.

How an interaction works. Scientific inputs are frozen; tools, review checkpoints, and termination follow each benchmark's protocol. Persona descriptions are never supplied to the actor.

Built around real scientific tasks

Five source benchmarks, forty tasks each, spread across four scientific domains.

The researcher changes the conversation — a final score alone misses it.

The first two results compare all 23 model configurations; the rest come from the paper's detailed persona analyses on GPT-6 Astra.

A closer look at the conversations

The saved interactions make collaborator-dependent changes visible beyond a single final score. Figures are from the paper's GPT-6 Astra analyses; selected trajectories illustrate variation and are not independent statistical evidence.

23 model configurations, four metrics.

Performance is each source benchmark's native score normalized to 0–100, averaged over questions within a benchmark with the five benchmarks weighted equally; Mean Persona Performance weights profiles 155, 001, 003, and 046 equally, and persona off is reported separately. Fact recovery is the share of annotated withheld facts recovered in eligible SciConvBench disambiguation episodes. Self-repetition is the share of six-token windows that reuse earlier actor wording; its threshold is exploratory, and lower is not automatically better. Bold marks the highest value per column (self-repetition is not ranked).

Means with 95% confidence intervals as reported in the paper's Table 3. Actor and simulator share a model, and GPT-6 Astra with low reasoning effort judges all models. Estimation procedures and coverage differ across rows (see the paper's Appendix B), so the ordering is descriptive and does not establish significance between models.

Cite PersonaScienceBench

@ARTICLE{2026arXiv261032106W,
       author = {{Wu}, Siyang and {Zhang}, Weizhi and {Yang}, Chen and {Fei}, Esther Xu and {Overwiening}, Jorin and {Pan}, Xu and {Tang}, Jing and {He}, Yixuan and {Mo}, Kelvin C. and {Li}, Peiwen and {Cao}, Defu and {Hu}, Xu and {Wang}, Haixin and {Li}, Taoran and {Li}, Haoming and {Deng}, Zheyuan and {Zhang}, Jingwen and {Zhang}, Pengsong and {Zhang}, Haichao and {Luo}, Guangze and {Lu}, Yi Daniel and {He}, Zihuai and {Qiu}, Chenxi and {Wang}, Dianzhuo and {Peng}, Zhejian and {Liang}, Shushu and {Lu}, Wenyi and {Wang}, Yifan and {Huang}, Jintao and {Wang}, Yijun and {Jiang}, Wanting and {Li}, Yueying and {George}, Robert and {Hu}, Haodi and {Wang}, Cong and {Li}, Xiaomin and {Hao}, Yuexing},
        title = "{PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers}",
      journal = {arXiv e-prints},
     keywords = {Artificial Intelligence},
         year = 2026,
        month = oct,
          eid = {arXiv:2610.32106},
        pages = {arXiv:2610.32106},
archivePrefix = {arXiv},
       eprint = {2610.32106},
 primaryClass = {cs.AI}
}

All numbers on this page are from the PersonaScienceBench paper (arXiv:2610.32106v2).