Evaluating coding agents in persona-driven, human-interactive sessions.
Can a coding agent finish the task and meet the requirements of the person asking?
Real coding sessions involve clarifying goals, reviewing changes, and asking for revisions. A patch can pass every test and still be wrong for the person who asked: an analyst who has no permission to create database indexes, a student who needs the change explained. PersonaCodeBench keeps each coding task and its native checks, and binds it to a persona whose requirements the agent must also satisfy.
A task with its native verifier, drawn from Terminal-bench 3, DeepSWE, SWE-bench-Live, SWE-Together, or SWE-bench Pro.
A simulated user turns role, technical background, goals, preferences, and working constraints into concrete requirements, disclosed progressively over a multi-turn session.
A completion verifier checks native correctness; a personalization verifier checks every mandatory requirement. Passing only one earns no joint credit.
Every session is graded on two separate questions, and only both together count:
The persona pool combines ten shared scenario templates with structured profiles: role, technical background, goals, behavioral preferences, and working constraints. Each task is bound to one persona; the Lite release has 100 pairs, 186 evolved requirements, and 35 distinct persona IDs. Numbers below are for the reference agent, GPT-6 Luna (medium effort): Joint pass by source, overall Joint pass, and the personalization grades that could be observed.
A persona is a static profile plus a set of evolved requirements, p = (s, R). Each requirement fixes its applicability, task-specific bindings, disclosure trigger, acceptance conditions, required evidence, and the feedback returned when it is Unmet. Requirement content and acceptance criteria are frozen before execution.
Plays the persona (GPT-6 Luna, medium effort). It cannot introduce grading conditions, grant permissions, or advance the session.
Tracks disclosed requirements, active obligations, authorization, dependencies, and remaining budget; validates every proposed user message.
Works in its source's own scaffold (Harbor 0.23.0, Docker environments) and answers the user.
At checkpoints, checks active requirements against their frozen contracts: deterministic rubrics for executable conditions, LLM judges for semantic ones.
Native task correctness, using the source benchmark's own checks.
Every mandatory evolved requirement must be accepted.
Joint pass = both verifiers pass, within the declared budget. Errors and unexecuted tasks count as zero.
The session loop, drawn from the paper's description of its Figure 2. The controller governs the disclosure of evolved requirements and uses the online requirement evaluator to check responses; Satisfied requirements let the session advance, Unmet ones return feedback to the coding agent for revision. A Satisfied verdict marks progress on the current obligation, not final acceptance.
How the benchmark is built. Four stages turn source tasks into task–persona pairs. Contrastive screening keeps tasks the reference agent passes natively but fails once the persona is added.
Build the candidate pool from the five source benchmarks.
Check task–persona compatibility, requirement grounding, that the original objective is preserved, and that disclosed requests agree with acceptance criteria. Unsupported or conflicting requirements are repaired or rejected.
Target tasks where GPT-6 Luna passes the native task but fails the persona-adapted one.
Full set of 231 pairs and a source-proportional Lite set of 100, with no extra model-performance filter.
Task–persona pairs by source benchmark, Full and Lite releases.
| Source | Full | Lite |
|---|---|---|
| Terminal-bench 3 | 52 | 23 |
| DeepSWE | 64 | 28 |
| SWE-bench-Live | 45 | 19 |
| SWE-Together | 9 | 4 |
| SWE-bench Pro | 61 | 26 |
| Total pairs | 231 | 100 |
| Evolved requirements | 422 | 186 |
| Distinct persona IDs | 66 | 35 |
Requirements per Lite task:
Share of Lite tasks with at least one requirement in each category. A task counts once per category, so shares need not sum to 100%.
GPT-6 Astra at max effort reaches 48% Joint pass on Lite, followed by GPT-6.1 Sol at xhigh (42%) and Claude Opus 5.5 at medium (40%). No evaluated configuration clears half of the task–persona pairs.
In the GPT-6 Luna ablation, dropping persona requirements raises Joint pass by 26 points and completion pass from 24.0% to 45.0%. Giving every requirement at the opening, instead of progressively, moves Joint pass only from 19.0% to 20.0% in a single run.
Across 1,962 non-provisional run–task records spanning models and reasoning settings, 179 of 874 complete native-test passes fail the recorded persona criteria.
The highest Terminal-bench 3 score is 17.4% (4 of 23 tasks); the best scores elsewhere are 80.8% on SWE-bench Pro, 57.9% on SWE-bench-Live, and 57.1% on DeepSWE. Strengths are source-specific: Claude Sonnet 5.5 (high) scores 65.4% on SWE-bench Pro and 0.0% on DeepSWE.
For GPT-6 Luna, founders reach 3/4 Joint passes while students reach 2/20, first-time contributors 3/19, and managers 2/18; designers and shop owners get none. The more persistent personalization failures arise when the agent must preserve execution history, keep a valid final workspace, or turn tests into evidence the user can verify.
Against an adjudicated reference from two software-engineering experts on 148 blinded requirement-level judgments, the online evaluator agrees 71.0% of the time (κ = 0.473) and 77.2% on the accept/reject decision (MCC 0.541). Satisfied verdicts reach 89.2% precision and 74.5% recall.
Pass rate (%) pooled over all 100 Lite tasks, with the rate on each source (task count under the name). Every non-pass outcome, including errors and unexecuted tasks, counts as zero. All settings follows the row order of the paper's Table 2; Best effort per model keeps each model's highest Joint pass (both rows on a tie), sorted by it. Bold marks the best value per column. † only counts the textual part. Kimi-K3 mixes original settings and high-effort repairs. GPT-6 Luna also serves as the user simulator and judge.
Effort effects are mixed: GPT-6.1 Sol rises from 31% at medium to 42% at xhigh, while Claude Opus 5.5 scores 40% at medium and 39% at high. In the paper's common-task analysis, only one of 33 within-model setting comparisons reaches nominal significance (p = 0.049), and none survives a Bonferroni correction.
@ARTICLE{2026arXiv261105739W,
author = {{Wang}, Haixin and {Hou}, Jianheng and {Shen}, Hongyu and {Fan}, Wenzhe and {Huang}, Wei-Chieh and {Zhu}, Qianyu and {Li}, Weichen and {Li}, Taoran and {Yao}, Yuhang and {Wang}, Ruohan and {Peng}, Zhejian and {Jiang}, Wanting and {Chen}, Yifan and {Liang}, Shushu and {Li}, Haoming and {Cao}, Defu and {Hu}, Xu and {Wu}, Siyang and {Yang}, Chen and {Hu}, Haodi and {Zhang}, Haichao and {Wang}, Cong and {Yu}, Zhiyuan and {Zhang}, Weizhi and {Wang}, Yifan and {Huang}, Jintao and {Sun}, Yizhou and {Wang}, Yijun and {Hao}, Yuexing and {Li}, Xiaomin},
title = "{PersonaCodeBench: Evaluating Coding Agents in Persona-driven Human-Interactive Sessions}",
journal = {arXiv e-prints},
keywords = {Artificial Intelligence},
year = 2026,
month = nov,
eid = {arXiv:2611.05739},
pages = {arXiv:2611.05739},
archivePrefix = {arXiv},
eprint = {2611.05739},
primaryClass = {cs.AI}
}
All numbers on this page are from the PersonaCodeBench paper (arXiv:2611.05739).