About
Perplexity's crawling and storage system processes URLs through multiple stages, including discovery, crawling, parsing, and indexing. The company's search and answer systems rely on this pipeline.
Responsibilities
•Find and diagnose quality issues in the crawling pipeline
•Train small models to optimize specific aspects of the pipeline, such as parsing quality
•Build datasets for model training, including LLM-as-a-judge labeling pipelines
•Improve page selection algorithms for indexing
•Design and analyze experiments to validate improvements
Requirements
•4+ years of experience as a data analyst, ML engineer, or in a related role
•Strong coding skills and ability to write production-grade code at the level of a mid-level backend engineer
•Experience designing metrics from scratch
•Experience training ML models that have shipped to production with measurable metric improvements
Nice to have
•Direct experience working on web crawling or indexing pipelines
Conditions
No conditions specified in the posting.
