Principal AI Evaluations Platform Engineer

About the role

Microsoft AI is seeking an experienced engineer to build and operate the evaluation platform supporting large-scale model training and development. This role involves building reliable evaluation infrastructure at scale and working closely with researchers and training teams across European offices to ensure evaluations are efficient and available. The successful candidate will combine strong systems engineering fundamentals with operational excellence to support the development of artificial intelligence systems.

Responsibilities

• Serve as a primary engineer for the evaluation platform during European hours, providing dedicated coverage for the evaluation stack supporting large-scale model training.
• Develop and extend core evaluation platform capabilities, including benchmark configurations, problem sets, graders, and evaluation runners.
• Improve evaluation throughput and scheduling efficiency.
• Build and maintain monitoring, alerting, and dashboards to proactively detect evaluation degradation.
• Partner closely with researchers and training teams across European offices to define evaluation requirements, coordinate launches, and unblock experiments.
• Contribute to technical design and code reviews across the evaluation platform stack, and maintain high-quality operational documentation and on-call runbooks.

Required skills

• Python
• C++
• Go
• Rust
• Distributed systems
• On-call rotation

Nice to have

• GPU-based workloads
• Distributed training
• Ray
• Kubernetes
• Slurm
• Datadog
• Prometheus
• Grafana
• LLM evaluation
• Reinforcement learning

Qualifications

• Bachelor's Degree in Computer Science, Engineering, Mathematics, or a related field
• Master's Degree in Computer Science, Engineering, Mathematics, or a related field
• Equivalent practical experience

About the Company

Microsoft's mission is to empower every person and every organization on the planet to achieve more. The company focuses on building systems with artificial intelligence across agents, applications, services, and infrastructure while fostering a culture of inclusion, respect, integrity, and accountability.