About the role
Cluster Infrastructure owns the physical and logical platform that large training and serving jobs run on. That means schedulers, high-performance storage, the network fabric, node provisioning and the health systems that decide which machines are fit to work.
Our clusters are expensive, and small inefficiencies compound across thousands of nodes. When a researcher waits for capacity, or a job dies because one link flaps, the cost is measured in lost experiments. The same fleet also hosts the inference workloads behind Clexerl Go, so fair, predictable sharing between training and serving is a core design problem.
We run the platform the way a good operations team would: everything declarative, every failure classified, every repair automated once it has happened twice.
What you’ll do
- Develop gang scheduling, preemption and quota policies so large jobs start promptly and smaller workloads fill the gaps.
- Build node health checks and automated remediation that detect failing GPUs, flaky links and memory errors before a job does.
- Operate and scale parallel file systems and object storage for checkpoints and training data, tuning for throughput and metadata load.
- Design and monitor the high-bandwidth fabric, including RDMA transport, congestion behaviour and topology-aware placement.
- Automate fleet provisioning, firmware and driver rollouts with staged deployment and safe rollback.
- Write runbooks and post-mortems, and turn recurring incidents into software.
What you’ll bring
- Experience operating large Linux fleets, ideally with accelerators, and a deep respect for the failure modes at scale.
- Working knowledge of Kubernetes or a comparable scheduler, including custom controllers and scheduling extensions.
- Understanding of datacentre networking: Ethernet and InfiniBand, RDMA, routing, and how to read a congestion problem.
- Proficiency in Go, Python or Rust for building infrastructure services and automation.
- Experience with storage systems under heavy parallel read and write load, including tail latency analysis.
- Calm judgement during incidents and the discipline to follow through afterwards.
Nice to have
- Experience with infrastructure-as-code and declarative fleet management at thousands of nodes.
- Knowledge of GPU diagnostics, ECC behaviour and silent data corruption detection.
- Background in capacity planning and chargeback or showback models for shared compute.
- Experience with bare-metal provisioning and out-of-band management.
Work you might take on
- 01
Build a topology-aware placement policy that keeps tightly coupled jobs inside a single network domain and reports fragmentation.
- 02
Create a burn-in and continuous-health pipeline that quarantines nodes showing correctable memory errors or degraded link speeds.
- 03
Redesign the checkpoint storage tier to absorb synchronised write bursts from thousands of ranks without stalling training.
- 04
Implement fair-share quotas with preemptible low-priority work so idle capacity is used and large jobs still start on time.
Compensation
$280K – $460K
The US base salary range for this role. Every offer also includes equity in Clexerl and, when it spins out, in the venture you help build. We tell you the reasoning behind your number.
Benefits
Health, fully covered
Medical, dental and vision for you and your dependants, premiums paid in full.
Two kinds of equity
Equity in Clexerl, plus a stake in the venture you help build when it spins out.
22 weeks parental leave
Fully paid, for every parent, however your family arrives.
Rest that sticks
Flexible time off with a four-week minimum we actually check on, plus a company-wide week in December.
Learning and compute
A yearly budget for courses, conferences and books, and compute credits for side projects.
401(k) with a 4% match
Matched from your first paycheque and vested immediately.
How we hire
Apply 5 minutes
Send links to what you have built and a few lines on why this role. No cover letter needed.
Intro call 30 minutes
A conversation with the hiring manager about the work, the team and what you want next.
Work sample 2–4 hours
A paid take-home or a pairing session on a real-shaped problem. You pick the format.
Studio day One day
Meet the team, go deep on your work and ours. Remote or in person.
Offer Within 5 days
A written offer with the full range, the equity and the reasoning behind both.
Opens an email with a short template. Links beat résumés.