Задачи:
•Design a data lake to process and store structured and unstructured data from raw sources
•Develop data models to store data and provide access for business purposes
•Develop integrations with various systems to provide a unified view of key indicators for decision-making
•Develop ETL pipelines to extract, transform, and load data
•Develop and monitor data pipelines to ensure data quality and integrity
•Automate and optimize data transformation processes
•Collaborate with other teams to ensure data is available and accessible for analytics and decision-making
Требования:
•Degree in computer science, information science, engineering, mathematics, or a related technical discipline
•5+ Years of experience with SQL and NoSQL technologies
•Strong experience with Oracle and PostgreSQL databases
•Extensive ETL development experience and strong Python programming skills
•Advanced knowledge of PySpark for large-scale distributed data processing
•Proficiency in Airflow for scheduling, orchestrating, and monitoring complex ETL/ELT workflows
•Expertise in Kafka for event streaming and messaging pipelines
•Experience with MinIO (S3-compatible storage), Parquet, and Iceberg
•Familiarity with Lakehouse concepts
•Deep understanding of MPP architecture concepts
Будет плюсом:
•Familiarity with Impala for interactive SQL querying on Big Data
•Experience with Greenplum, Docker and Kubernetes
•CDC tools such as Debezium and Oracle GoldenGate
•Data visualization tools such as Tableau and PowerBI
•DataOps and DevOps principles
•Data modeling and database design
•Data engineering best practices including data security, data access control and data governance
•Grafana, GitLab
•Mentoring junior data engineers
•Collaboration with Data Scientists, Analysts, and DevOps
Условия: не указаны
