Description
We are seeking a highly skilled Data Engineer to drive the development and optimization of our data platform. This position is designed for a technical expert who excels at the intersection of data engineering, infrastructure automation, and business logic. The successful candidate will be responsible for building robust data lifecycles, ensuring the scalability of our cloud-native environment, and maintaining the integrity of our streaming and batch processing systems.
Responsibilities
•Design, implement, and scale end-to-end data pipelines using Python and SQL-based transformations within a cloud-native data lake environment.
•Manage and scale containerized data workloads and automated workflow orchestration (using tools like Apache Airflow) and Infrastructure as Code (using Terraform) or similar functional equivalents.
•Maintain and evolve real-time data replication services and stream processing architectures (utilizing Apache Kafka or similar distributed messaging systems) to ensure low-latency data availability.
•Establish and implement Data Governance frameworks and automated validation protocols to ensure the highest standards of data accuracy, compliance, and platform reliability.
•Partner with cross-functional stakeholders to interpret complex business requirements and translate them into performant data models for reporting and business intelligence.
•Provide technical insights into the continuous improvement of the data platform, ensuring the stack remains resilient and performant as data volumes and complexity increase.
Requirements
•A proven track record in a Data Engineering capacity, with extensive experience managing and scaling modern, distributed data platforms.
•Deep experience managing cloud-native data ecosystems, with a strong preference for candidates familiar with AWS or GCP environments.
•Proficiency in workflow orchestration and Infrastructure as Code (IaC); specific experience with Apache Airflow and Terraform is highly preferred, though experience with similar tools (e.g., Prefect, Dagster, Pulumi) will be considered.
•Demonstrated experience in configuring, maintaining, and troubleshooting distributed messaging and real-time data replication systems such as Apache Kafka or equivalent technologies (e.g., Amazon Kinesis, Google Pub/Sub).
•Solid understanding of Data Governance principles, including data lineage, metadata management, and automated quality checks.
•Exceptional Python programming capabilities and advanced SQL proficiency, with a focus on writing clean, modular, and highly testable code.
•Strong analytical aptitude with the ability to interface directly with business units, understand underlying data logic, and architect schemas that support complex organizational reporting and data product needs.
Conditions
No conditions stated in the posting.
