About the Role
Join a highly technical AI research and engineering organisation building the data infrastructure behind next-generation AI models. As a Senior Data Platform Engineer, you'll design and operate distributed data-processing systems capable of handling petabyte-scale workloads, ensuring research teams can access reliable, reproducible datasets and run complex pipelines efficiently. This role combines distributed systems engineering, workflow orchestration and performance optimisation, with a focus on scalability, fault tolerance and cost efficiency across large-scale data and machine learning infrastructure.
Responsibilities
•Design, build and operate distributed data-processing infrastructure for petabyte-scale workloads
•Develop and optimise data pipelines, execution frameworks and workflow orchestration across diverse computational workloads, including GPU-accelerated stages
•Improve system performance, throughput and resource utilisation while managing infrastructure costs
•Build reliable, observable pipelines with robust failure recovery, restartability and monitoring
•Implement dataset versioning, lineage and reproducibility to ensure research datasets can be reliably traced and recreated
•Integrate data infrastructure with large-scale ML training workflows and establish engineering standards and best practices
Requirements
•PhD in Computer Science, Computer Engineering, Software Engineering or a related discipline
•At least 3 years of relevant full-time experience building or operating distributed data-processing systems or ML data infrastructure
•Strong experience with distributed processing frameworks such as Ray, Apache Spark or Flink, alongside workflow orchestration and large-scale pipeline execution
•Solid understanding of distributed computing, performance profiling, throughput optimisation and fault tolerance
•Experience with high-performance data formats such as Apache Arrow or Parquet, and dataset versioning, lineage or reproducibility
•Familiarity with workload orchestration and scheduling tools such as Kubernetes or Slurm, and containerisation technologies such as Docker or Enroot
•Strong software engineering skills and experience owning production systems where reliability, performance and cost matter
Conditions
Help build the infrastructure that enables cutting-edge AI research at scale. You'll tackle challenging distributed-systems problems, influence core architectural decisions and work closely with researchers and engineers to make large-scale data and ML workflows more efficient, reliable and reproducible.
