Python Engineer – Web Crawling & Data Extraction

About the role

Certivity builds and operates large-scale web crawling and extraction systems to transform unstructured web data into clean, structured output. You will own the end-to-end crawler lifecycle, including discovery, fetching, and data transformation. The role involves managing anti-bot defenses and ensuring high data accuracy within a cloud-based environment.

Responsibilities

• Build and scale crawlers to handle dynamic, large-scale websites.
• Implement strategies to bypass anti-bot measures such as TLS fingerprinting, proxies, and rate-limiting.
• Transform raw HTML and PDF content into clean, structured data using LLM-assisted extraction.
• Design and operate data pipelines for ingestion, deduplication, versioning, and enrichment.
• Deploy and maintain containerized applications in the cloud using CI/CD practices.

Required skills

• Python
• Web scraping
• Async programming
• Docker
• GCP
• Git
• CLI

Nice to have

• LLM-assisted extraction
• ML pipelines
• Go
• Java
• Spring Boot
• React
• Terraform
• Kubernetes

Qualifications

• 3+ years of professional software development experience

Benefits

• Flexible working hours
• 30 vacation days per year
• 30 days of workation per year
• EGYM Wellpass
• Udemy access

About the Company

Certivity builds and operates large-scale web crawling and extraction systems that turn unstructured, dynamic websites into clean, structured and correct data.