Applied Research Engineer, Evaluation

About the role

You will build and sharpen the evaluation stack used to measure the performance of frontier image and video models. This role involves designing evaluation taxonomies, benchmarking against competitors, and closing the loop with research teams to define target metrics. You will ensure that model improvements are verified and that evaluation systems remain robust against gaming and overfitting.

Responsibilities

• Building and maintaining the evaluation stack for model checkpoints, including automated benchmarking and LLM-as-judge pipelines.
• Designing evaluation breadth across prompt adherence, editing fidelity, aesthetics, text rendering, and safety.
• Evaluating the evaluation stack itself to ensure results are stable and human-automated agreement is high.
• Benchmarking models against frontier competitors to obtain meaningful performance perspectives.
• Identifying model output shortcomings not captured by aggregate numbers.
• Defining target metrics for RL and finetuning researchers and verifying checkpoint improvements.

Required skills

• Python
• PyTorch
• ML infrastructure
• Statistics
• LLM-as-judge
• Generative models

Nice to have

• Human evaluation
• Study design
• Reward models
• RLHF
• DPO
• Agentic systems

Qualifications

• Advanced degree (MS/PhD) in a relevant field
• Equivalent practical experience

About the Company

Black Forest Labs is a frontier research lab behind foundational technologies like Latent Diffusion, Stable Diffusion, and FLUX. Headquartered in Freiburg, Germany, with a presence in San Francisco, the company focuses on research excellence, open science, and expanding human creativity.