About the Role
You will build and own web-crawling systems, from distributed internet-scale collection through filtering, deduplication, and data retention decisions. You will develop the crawler, its scalable infrastructure, and pipelines that transform raw crawls into usable pretraining data. You will create specialized crawlers, improve reliability and efficiency at petabyte scale, assess how data changes affect model performance, and help set technical direction.
Requirements
•8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems
•Track record of owning crawler or data-acquisition infrastructure at internet scale
•Strong software engineering skills in Python, Go, or Rust
•Experience with distributed systems
•Working knowledge of practical and legal considerations for large-scale web data collection, including robots.txt, rate limiting, and licensing
Responsibilities
•Design and scale web crawler and ingestion infrastructure for pretraining data
•Build pipelines for large-scale extraction, deduplication, and data-quality filtering
•Build specialized crawlers for high-value or hard-to-reach data sources
•Work with the pretraining team to assess how crawled-data changes affect model performance
•Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale
•Set technical direction and bring other engineers up to speed
Benefits
•Health, dental, and vision benefits
•Unlimited PTO
•Paid parental leave
•Relocation support as needed
