Simulated users are how you evaluate a product before real users are involved. But a product's "user" is not always a person typing prompts. For an Agent Development Kit (ADK), the SDK a person uses to build an LLM agent, the real user is an engineer: someone reading the API, wiring up tools, and fighting through errors until their agent runs.
So to evaluate that product, you simulate that user. This post is about a methodology we call LLM-as-a-Developer: a controlled simulation of the developer that measures, at ecosystem scale, both how usable a framework is and how good the agents you build with it turn out to be.
The core idea: simulate the user, where the user is a developer
Simulation-based evaluation has one defining virtue: it replaces an expensive, slow, biased human loop with a controlled, repeatable, scalable one. LLM-as-a-Judge did this for the evaluator. It simulates a human grader so you can score thousands of outputs without a human in the loop.
We apply the same move to a different bottleneck. Developers today face dozens of agent frameworks, LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, Google ADK, OpenAI Agents SDK, and a long tail beyond, with no principled way to answer the obvious question: which framework produces the best agents, and at what development cost? Answering it the traditional way means having experts hand-implement every benchmark against every framework's API, an O(N × M) effort that is slow, biased toward whichever framework the author already knows, and impossible at ecosystem scale.
Instead, we simulate the developer:
That is the entire methodology in one sentence. And because we run the identical simulated developer against every framework, same model, same tools, same budget, same starting point, we eliminate the single biggest confound in framework benchmarking: the human expert who already knows framework A better than framework B.
Why simulating the developer is the right unit of measurement
Simulating the developer produces two measurements from a single run that no survey or model-benchmark can produce together:
- The generation process → API usability. As the simulated developer works, we record everything: tokens consumed, turns needed, validation failures, where it gets stuck. That effort is a direct, objective proxy for how hard the framework is to learn. A clean API is cheap to onboard; a sprawling or under-documented one burns tokens and stalls.
- The execution results → framework effectiveness. The agent the simulation produces is a real program. We run it on real benchmarks and record resolution rate, cost, and latency. That tells us not just "is the framework usable" but "does what you build with it actually work."
Contrast the alternatives. Developer surveys capture what people say, not how frameworks behave. Agent benchmarks vary the model and hold the framework fixed. Hand-written comparisons reflect the author's familiarity and cannot scale. Simulating the developer is the only approach that is controlled, scalable, and dual-signal at once.
Anatomy of the simulated developer
The method only works if the simulation is faithful: it has to behave like a real developer onboarding to a framework, not a script. Three design choices make it faithful.
It works from the framework's own materials, nothing else. Inside a per-framework Docker container, the simulated developer gets exactly what a human would: the curated documentation (browsed incrementally, 20K characters per call) and the source code, mounted read-only and searchable. No hand-written hints, no per-framework special-casing. Information is restricted to each framework's own repository, so the comparison stays apples-to-apples.
It debugs like a developer, through a validate-and-repair loop rather than one-shot generation. The simulation writes an agent.py and runs it through a three-level validation pipeline of increasing fidelity:
- Level 1 — Static analysis: compile check, import verification, a "does this actually use the framework?" check (raw OpenAI/httpx fallbacks are rejected), plus 40+ anti-pattern detectors.
- Level 2 — Real LLM smoke test: execute the agent against a live LLM to confirm it can make a call, parse the response, and return a result.
- Level 3 — Real benchmark task: run one real task from the target benchmark and confirm the agent behaves as expected, proving it can run the full benchmark without crashing.
On any failure, the validator pattern-matches the error against 70+ known signatures and hands back a structured diagnostic, a root-cause diagnosis plus a concrete fix, exactly like a senior colleague pointing at the bug:
https://api.openai.com) instead of the proxy endpoint. The execution environment routes all LLM traffic through a local proxy at ADK_BASE_URL.FIX: Replace all hardcoded URLs with
os.environ["ADK_BASE_URL"].The simulated developer reads the hint and makes a targeted single-edit repair instead of blindly regenerating. This is what makes the effort signal meaningful: it reflects real debugging, not random retries.
Everything downstream is uniform. Every generated agent exposes one entry point, solve(prompt, workdir), so any benchmark can invoke any framework's agent identically. And all LLM traffic, whatever provider API a framework natively targets (OpenAI, Anthropic, Google), is routed through a single token proxy that translates protocols and captures per-call tokens, latency, and tool-call sequences. That is how one simulated developer can be measured consistently across 51 frameworks and 4 benchmarks.
agent.py. (3) Benchmark the generated agent through a token proxy and result evaluator.ADK Arena: the methodology as a running system
We packaged all of this into ADK Arena, a fully automated pipeline that takes a framework's GitHub repository as input and produces benchmark scores as output, with zero manually written agent code. It has three stages: environment setup (collect 51 ADKs, build isolated Docker images), agent generation (the simulated-developer loop above), and benchmarking against four established suites that span different agent skills: SWE-bench (code editing), τ²-bench (conversational tool use), MCP-Atlas (multi-tool orchestration), and Terminal-Bench (terminal interaction).
The 51 frameworks we tested
We ran the simulated developer against every one of these popular Python Agent Development Kits, spanning official vendor SDKs, research releases, and community projects:
Each was evaluated across four benchmarks (SWE-bench, τ²-bench, MCP-Atlas, Terminal-Bench). For the full per-framework generation effort and execution results, see the paper: arxiv.org/abs/2606.05548.
The evidence
The methodology is the point; the numbers are how we know it works. Running the simulated developer across all 51 frameworks (204 pairs, two developer models authoring agents, all executing on the same backbone) produced four results worth calling out.
Effort is a real usability signal. The simulated developer produces a validated agent for 57% of runs, and its per-agent effort varies 5.6× ($0.6 → $3.4) across frameworks. Clean APIs (LangGraph, OpenAI Agents, Haystack, AG2) are cheap to onboard; sprawling ones cost several times more. Effort tracks API complexity, though not as a clean pass/fail line, the cheapest runs are often frameworks where the developer gave up early.
The developer you simulate matters more than the model that runs the agent. On the same execution backbone, agents authored by the stronger developer model resolve ~2× as many tasks (41% vs 22% mean). A more careful simulated developer wires up the framework's full loop and registers tools; a lazier one emits a single model call and returns. That is itself a validation of the method: the measurement is sensitive to exactly the thing it should be, the quality of the developer.
A well-chosen framework can rival hand-built agents. The best single-benchmark generated agents are competitive with, and sometimes beat, production coding agents at a fraction of the cost. On MCP-Atlas the generated Agno agent resolves 68% of tasks at 38K tokens per task, beating Copilot's 58% at 327K (~9× fewer tokens). For reference, the five production agents we benchmarked (all on the same backbone):
| System | SWE-bench | τ²-bench | MCP-Atlas | Terminal | Avg |
|---|---|---|---|---|---|
| Copilot | 74% | 46% | 58% | 46% | 56.0% |
| OpenHands | 60% | 46% | 52% | 36% | 48.5% |
| OpenCode | 76% | 48% | 18% | 38% | 45.0% |
| Claude Code | 48% | 46% | 38% | 30% | 40.5% |
| Codex CLI | 60% | 46% | 36% | 16% | 39.5% |
Per-task averages over the 50-task subset per benchmark. The median generated framework resolves only 32%, so beating the frontier is a top-of-distribution effect, not a guarantee.
Docs, source, and prior knowledge are largely substitutable. When we ablate what the simulated developer can see, genuine native framework usage stays inside a narrow 28–40% band. Counterintuitively, Source only (40%) beats Docs only (28%): documentation's worked examples anchor the simulation toward copying quickstart snippets into thin wrappers, while raw source pushes it toward the real API. Even with no reference at all (33%), prior knowledge of popular frameworks carries a third of them. No single information source is a hard bottleneck.
The specific numbers will age as models and frameworks change; the method will not. "Simulate the user, hold the simulation constant, vary the thing under test, and read off effort and outcome" is a general recipe. When the user is a developer, you measure SDK usability and the quality of what gets built. When the user is an evaluator, you get LLM-as-a-Judge. Whenever a human-in-the-loop makes evaluation expensive, slow, or biased, simulating that human with a controlled agent turns it into something you can run on demand, at scale.
Read the paper & get the code
The full study, with per-framework generation effort, execution results, and the information-source ablation, is on arXiv. The complete pipeline is open source: the LLM-as-a-Developer simulation loop, per-framework Docker isolation, the three-level validate-and-repair pipeline, the token proxy, and all four benchmark adapters.
Paper — arxiv.org/abs/2606.05548 →
Code — github.com/jintao-h/ADK-Arena →
Note. ADK Arena is research from the MatrAIx community on simulating developers as users of agent frameworks. Reported figures are from the study's controlled runs and are meant to illustrate the methodology, not to rank any single framework definitively.