Member of Technical Staff, Personalization & Agent Evaluation
Apply now ↗About the role
Develop benchmarks and evaluation methods for personal AI agents. You will design tasks grounded in real user insights, analyze agent behavior across apps, and measure task completion, personalization, memory, proactivity, and safety.
Collaborate with MatrAIx researchers and engineers to define research questions, implement reliable systems, and turn experimental findings into improvements to our evaluation and training infrastructure.
What you'll do
- Design new evaluation tasks or metrics for personalized, long-horizon agent behavior
- Run evaluations across models and agent systems, and analyze their failure modes
- Study how reliable graders are, including LLM-as-judge agreement with human labels
- Communicate experimental results, implementation decisions, and research progress in regular team reviews
- Deliver documented, reproducible research and software that contributes to MatrAIx's products and technology
Required qualifications
- Research or engineering experience in computer science, machine learning, or a related field, demonstrated through academic work, open-source contributions, or industry projects.
- Strong understanding of LLM agents and evaluation methodology
- Solid statistics and experimental design
- Fluent in Python and ML frameworks
Preferred qualifications
- Research in personalized agents, user modeling, recommender systems or HCI
- Experience with long-horizon, browser or computer-use agent benchmarks
- Publications or open-source work in agent evaluation
Work authorization: We support CPT, OPT, STEM OPT and H1B.
Compensation & benefits
Annual base salary: $200,000–$500,000 USD. Compensation within this range depends on experience, qualifications, and responsibilities.
- Meals: Reimbursed up to $1,500 per month.
Questions? Email jobs@matraix.ai.
Apply now ↗