Infrastructure Engineer (ML Systems)

About the role

You will own the GPU compute infrastructure that powers model training, evaluation, and serving systems. This role involves managing clusters, GPUs, and schedulers while overseeing capacity across cloud providers to ensure reliable production performance.

Responsibilities

• Running Kubernetes clusters for GPU training and inference
• Improving how GPU work is scheduled and placed
• Detecting, draining and repairing unhealthy GPU nodes
• Managing capacity with our cloud providers
• Debugging failures in GPUs, Linux, storage and networking

Required skills

• Kubernetes
• GPU workloads
• Linux
• Python
• GPU drivers
• Networking
• Storage

Nice to have

• SRE
• Slurm
• Kueue
• Karpenter
• SkyPilot
• AWS EKS
• SageMaker HyperPod
• EFA
• InfiniBand
• NCCL