Senior/Staff AI Engineer, Quality & Evals

About the role

Cortea is seeking an engineer to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality, focusing on improving the reliability, performance, and cost-efficiency of agentic pipelines. You will design data architecture and write production code to ensure high-quality outcomes for audit-focused AI software.

Responsibilities

• Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.
• Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
• Analyze large volumes of agent traces and executions in columnar and analytical databases to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.
• Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging, and experiment management of audit agents.
• Team up with backend engineers to improve the speed and reliability of retrieval and reasoning agents.

Required skills

• Python
• Backend engineering
• LLM evaluation
• GCP
• ML pipeline evaluation
• Analytical databases
• System design
• Observability
• Monitoring

Nice to have

• ETL/ELT workflows
• Event-processing systems
• Temporal
• Distributed systems
• Audit domain knowledge
• Finance domain knowledge

Benefits

• Equity
• Coding tools budget
• Flexible vacation
• Team lunches
• Retreats

About the Company

Cortea is a Berlin-based startup transforming audits with AI-powered software and specialized agents. Backed by top-tier VCs, the company focuses on removing repetitive work from audit processes to allow experts to focus on judgment.