About the role
The evaluations team measures what models can actually do, before and after they ship. That includes capability benchmarks, agentic task suites, safety and misuse tests, and regression checks that run on every training checkpoint. The hard part is validity: a number is only useful if it tracks the thing you care about and is hard to game, leak or overfit.
Every team in Frontier AI relies on this work to know whether an idea helped. Post-Training and Reasoning use it to select checkpoints, Alignment uses it to gate releases, and the product teams around Clexerl Go use it to decide whether a model is ready for a real workflow. Evaluations here are treated as production software with owners and on-call.
Research engineers build both the harness and the tasks. You will work with domain experts to write problems, with infrastructure engineers to run them cheaply at scale, and with researchers who will, reasonably, try to argue your benchmark is measuring something else.
What you’ll do
- Build and maintain an evaluation platform that runs thousands of tasks against every checkpoint, with caching, retries and reproducible configuration.
- Design new benchmarks and agentic task suites, with held-out splits, contamination checks and documented scoring rules.
- Develop statistical tooling that reports confidence intervals, detects noisy benchmarks and measures how much variance comes from sampling versus the model.
- Study the reliability of model-graded evaluations, calibrate judges against human labels and track judge drift.
- Create dangerous-capability and misuse evaluations in collaboration with the Alignment team and external domain experts.
- Investigate surprising results, such as sudden score jumps, and determine whether they reflect real capability or an artefact.
What you’ll bring
- Strong software engineering skills, with experience building reliable data or evaluation pipelines that others depend on.
- Good grasp of experimental statistics: variance, multiple comparisons, bootstrap methods and power analysis.
- Familiarity with language model benchmarks and their known weaknesses, including saturation, leakage and prompt sensitivity.
- Experience designing tasks or tests that are hard to game, and the humility to expect that some will be gamed anyway.
- Fluency in Python, with comfort running jobs against large inference clusters and managing their cost.
- Written clarity: a benchmark report should say what was measured, what was not and how much to trust it.
Nice to have
- Experience with sandboxed environments for evaluating code-executing agents.
- Background in psychometrics, item response theory or survey methodology.
- Work on human evaluation programmes, including rater training and agreement analysis.
- Prior research on benchmark contamination detection.
Work you might take on
- 01
Build a contamination detector that flags benchmark items appearing in training data and reports how scores change after removal.
- 02
Create a long-horizon agent suite that scores partial progress and detects shortcuts in task completion.
- 03
Calibrate a model-based grader against human raters and publish its bias across answer length and style.
- 04
Set up nightly checkpoint evaluation with automatic regression alerts and variance-aware thresholds.
Compensation
$280K – $460K
The US base salary range for this role. Every offer also includes equity in Clexerl and, when it spins out, in the venture you help build. We tell you the reasoning behind your number.
Benefits
Health, fully covered
Medical, dental and vision for you and your dependants, premiums paid in full.
Two kinds of equity
Equity in Clexerl, plus a stake in the venture you help build when it spins out.
22 weeks parental leave
Fully paid, for every parent, however your family arrives.
Rest that sticks
Flexible time off with a four-week minimum we actually check on, plus a company-wide week in December.
Learning and compute
A yearly budget for courses, conferences and books, and compute credits for side projects.
401(k) with a 4% match
Matched from your first paycheque and vested immediately.
How we hire
Apply 5 minutes
Send links to what you have built and a few lines on why this role. No cover letter needed.
Intro call 30 minutes
A conversation with the hiring manager about the work, the team and what you want next.
Work sample 2–4 hours
A paid take-home or a pairing session on a real-shaped problem. You pick the format.
Studio day One day
Meet the team, go deep on your work and ours. Remote or in person.
Offer Within 5 days
A written offer with the full range, the equity and the reasoning behind both.
Opens an email with a short template. Links beat résumés.