Real user studies remain essential, but recruiting participants and repeating studies after every product change is expensive and slow. PersonaEval explores a complementary layer: simulated users who can interact with different applications in repeatable, parallel runs.
Our system paper, accepted to the COLM 2026 Workshop on Social Simulation with LLMs, introduces a modular workflow that separates the simulated user from the application under test. A Persona API turns an existing persona profile into user behavior, while a Task API packages the interface, interaction protocol, environment, and evaluation form.
One workflow, three interaction settings
- Survey: personas respond to structured market-research instruments and explain their choices.
- Chatbot: personas pursue realistic goals over multi-turn conversations, then evaluate satisfaction, constraint following, and preference match.
- Web: personas navigate an ecommerce environment, compare products, complete sandbox checkout, and report on the experience.
Because the task adapter can change without rebuilding the persona side, the same population can evaluate different systems while interaction trajectories and outcomes stay in a common format.
What the first results show
We evaluated six application entries with 50 relevant Nemotron personas each. The results show application-level differences and meaningful variation across persona groups. Multi-turn chatbots produced wider outcome spreads as users revealed constraints and negotiated responses, while survey and web tasks produced more concentrated scores.
Human-judged persona alignment remained high across the three settings. Representative traces also show personas shaping goals, product constraints, critiques, and final ratings. These results are an initial demonstration, not a substitute for human calibration; future work will compare simulated outcomes directly with real-user data and expand to more application domains.