Staff / Senior Software Engineer (Agentic Search) - Crawler

About the role

Nebius is seeking a Senior Software Engineer to develop the content acquisition and crawling infrastructure for an agent-native search platform. You will design distributed systems to discover, fetch, and refresh content from the web at scale, ensuring high-quality data availability for AI systems. This role involves managing high-throughput data flows and optimizing infrastructure for reliability and performance.

Responsibilities

• Design, implement, and operate web-scale crawling systems for acquiring content from the internet
• Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
• Ensure reliable operation of crawling infrastructure under high-throughput conditions
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
• Monitor resource usage, bandwidth consumption, and infrastructure cost
• Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
• Enable safe experimentation with crawling strategies and content acquisition policies

Required skills

• Go
• C++
• Distributed systems
• Web protocols
• Crawling
• Scraping
• Content extraction
• Scalability
• Fault tolerance

Nice to have

• Streaming data pipelines
• Event-driven systems
• Kafka
• Pulsar
• NATS
• RabbitMQ
• Distributed schedulers
• Spark
• Flink
• Beam

Benefits

• Competitive compensation
• Career growth and learning opportunities
• Flexibility and ownership
• Collaborative and innovative culture
• Opportunity to work on impactful AI projects
• International environment

About the Company

Nebius is a full-stack AI cloud platform supporting developers and enterprises from data and model training through to production deployment. Listed on Nasdaq, the company has a global footprint with R&D hubs across Europe, the UK, North America, and Israel.