About
Pencil uses Generative AI in its SaaS product to create advertising content, including video and adaptations in multiple formats. Its mission is to make AI the default in advertising while supporting creative people, small businesses, and creative individuals.
Responsibilities
•Own the Evals & Quality charter, including its problem definition, success metrics, and scope
•Define core quality metrics, including first-pass-right, coverage, and regression detection, and improve them
•Own the judging methodology, creative-quality rubrics, and calibration between automated judges and human raters
•Design and run the human quality-control mechanism, including how reviews inform automated evaluation and scale
•Own the golden dataset, including curation, format and client-context coverage, and versioning
•Build evaluation loops that other teams can set up and use independently
•Own adevals.ai as a product and ensure it meets a high design standard
•Provide evidence for commercial work, including client RFP responses and creative enablement packages
•Partner with Agent Architects on client-specific quality standards using configuration and rubrics rather than custom evaluation code
•Run the full product lifecycle, from discovery through post-launch audits, with a written hypothesis before each launch and a review 14 days afterward
Requirements
•Have owned a quality, evaluation, or ML measurement product with its own roadmap and users
•Have shipped measurement in a subjective domain, defining quality and keeping signals reliable as they scale
•Have direct experience with golden datasets, rubric design, judge calibration, and inter-rater reliability, including addressing failures
•Derive product decisions from evidenced customer needs in a domain that had to be learned; starting with a solution is not a fit for the role
•Have owned a metric that influenced decisions beyond your team
•Have experience presenting or defending methodology to clients
•Make decisions and iterate quickly with imperfect information, addressing problems early
•Care deeply about users and teammates and design empathetic workflows
•Take responsibility for outcomes, spot problems early, and take initiative
•Reduce unnecessary complexity, create reusable patterns, and make complex tasks intuitive
•Enjoy finding unexpected ways to delight users, especially through polished, usable experiences
Conditions
•25 Days PTO plus public holidays; the company operates a Flexible Time Off scheme
•Health insurance / private medical cover
•Monthly stipend towards wellness, fitness, and learning and development
•Enhanced parental leave for parents through birth, adoption, or surrogacy
•Flexible working hours
•Remote work is available from anywhere in the employee's home country
•UK employees can access the Pencil office in The Shard, London
