About the role
This team keeps very large training runs alive, fast and legible. A run that spans thousands of accelerators for weeks fails in ways small experiments never show: silent data corruption, stragglers, network partitions, numerical drift. You build the systems that detect those failures, recover from them and tell researchers what the run is really learning.
Every point of model FLOPs utilisation recovered is compute that goes back into research, and every undetected fault is a week of wasted training. The same infrastructure underpins the models behind Clexerl Go's agents, so reliability here is felt directly in products. The team also shares tooling with the Parent company infrastructure group.
You sit between researchers and hardware. Most of the week is profiling, reading traces and rewriting a kernel or a sharding plan, then explaining to a scientist why their change cost eight percent of throughput. Post-mortems are written for every lost run.
What you’ll do
- Optimise training throughput across data, tensor, pipeline and expert parallelism, using FSDP-style sharding where it fits and custom schedules where it does not.
- Profile runs end to end, from dataloader stalls to collective communication, and remove the bottlenecks that matter most.
- Build fault detection and automatic recovery, including checkpoint validation, straggler eviction and bit-level checks for silent data corruption.
- Write and tune GPU kernels in Triton or CUDA for attention, normalisation and fused optimiser steps.
- Instrument runs with metrics that expose gradient statistics, activation outliers and loss divergence early, before a run is lost.
- Write post-mortems for failed or degraded runs and turn each one into a guard, a test or a runbook.
What you’ll bring
- Substantial experience with distributed training of large neural networks, including debugging hangs, deadlocks and nondeterminism across many nodes.
- Working knowledge of NCCL-style collectives, interconnect topology and how communication overlaps with compute.
- Solid GPU performance engineering: memory hierarchy, occupancy, roofline reasoning and reading profiler output.
- Strong Python and PyTorch or JAX skills, plus comfort in C++ or CUDA when the problem demands it.
- An instinct for numerics: mixed precision, loss scaling, accumulation order and why results differ between runs.
- Calm judgement under pressure when a long run starts to misbehave and the decision to stop it is yours.
Nice to have
- Experience with low-precision training in FP8 or block-scaled formats.
- Contributions to a distributed training framework or a compiler stack such as XLA or Triton.
- Background in cluster scheduling, job orchestration or fleet health monitoring.
- Experience designing deterministic, bitwise-reproducible training pipelines.
Work you might take on
- 01
Raise model FLOPs utilisation on a mixture-of-experts run by overlapping all-to-all communication with expert compute.
- 02
Build a checkpointing system that saves asynchronously, verifies integrity and restores a failed run in minutes rather than hours.
- 03
Develop a silent-data-corruption detector that compares gradient checksums across replicas and quarantines suspect hosts automatically.
- 04
Write a fused kernel for a new attention variant and prove it matches the reference implementation numerically.
Compensation
$320K – $520K
The US base salary range for this role. Every offer also includes equity in Clexerl and, when it spins out, in the venture you help build. We tell you the reasoning behind your number.
Benefits
Health, fully covered
Medical, dental and vision for you and your dependants, premiums paid in full.
Two kinds of equity
Equity in Clexerl, plus a stake in the venture you help build when it spins out.
22 weeks parental leave
Fully paid, for every parent, however your family arrives.
Rest that sticks
Flexible time off with a four-week minimum we actually check on, plus a company-wide week in December.
Learning and compute
A yearly budget for courses, conferences and books, and compute credits for side projects.
401(k) with a 4% match
Matched from your first paycheque and vested immediately.
How we hire
Apply 5 minutes
Send links to what you have built and a few lines on why this role. No cover letter needed.
Intro call 30 minutes
A conversation with the hiring manager about the work, the team and what you want next.
Work sample 2–4 hours
A paid take-home or a pairing session on a real-shaped problem. You pick the format.
Studio day One day
Meet the team, go deep on your work and ours. Remote or in person.
Offer Within 5 days
A written offer with the full range, the equity and the reasoning behind both.
Opens an email with a short template. Links beat résumés.