Senior AI Engineer, Agents

About the role

Bliro is seeking a Senior AI Engineer to own the quality, prompting, and evaluation of their AI chat and voice agents. You will join a small team of specialists to orchestrate and benchmark models, ensuring agents provide accurate and useful interactions for field sales teams. The role focuses on building evaluation pipelines and improving agent performance through data-driven insights.

Responsibilities

• Map chat and voice agents end to end and identify failure modes.
• Build evaluation sets and metrics to benchmark models on quality, latency, and cost.
• Establish a repeatable model for improving agent quality through automated testing.
• Partner with backend engineers to ship improvements based on real user conversations.
• Turn real conversations into new test cases and set the long-term roadmap for agent quality.

Required skills

• Python
• LLM agents
• Model benchmarking
• Eval pipelines
• LLM judges
• Statistics

Nice to have

• Agent frameworks
• Realtime voice agents
• Multilingual agents

Qualifications

• Degree in a technical major from a major technical university

Benefits

• Equity ownership
• 30 days of PTO
• Free meals and drinks
• Deutschlandticket
• Wellpass

About the Company

Bliro is an AI assistant for field sales that documents customer touchpoints. Backed by 468 Capital, Lea Partners, and Rockstart, the company is a leader in privacy-preserving transcription.