About the Role
We are looking for a Senior Data Engineer to join a team developing data ingestion processes for a data warehouse, a Big Data environment, the Google Cloud platform, and campaign systems. We build solutions that process large volumes of data in batch and near-real time. Our data is used to create data marts, analyses, reports, customer segments, and personalized offers and campaigns. If you want to build reliable data pipelines in a hybrid and cloud environment, we invite you to get in touch!
Responsibilities
•Designing, building, and developing ETL/ELT pipelines operating in batch and streaming modes.
•Creating data ingestion processes for BigQuery, data warehouses, a Data Lake, and a campaign platform.
•Integrating data from on-premise systems, the GCP cloud, databases, files, APIs, and event systems.
•Designing logical and physical data models and preparing data layers and subject-specific data marts.
•Creating and optimizing SQL queries in relational and columnar databases.
•Programming data processing and loading processes in Python and/or Rust.
•Building event flows using Kafka or Pub/Sub.
•Ensuring data quality, completeness, consistency, and timeliness.
•Optimizing process performance and processing costs in GCP.
•Implementing monitoring, logging, error handling, and processing retry mechanisms.
•Translating business requirements into technical concepts, data models, and ready-to-use solutions.
•Testing, supporting deployments, diagnosing incidents, and maintaining solutions.
•Creating technical documentation and developing solutions in line with the DevSecOps approach.
•Independence, accountability, and consistency in completing tasks.
•Attention to the quality, reliability, and maintainability of solutions.
•Good work organization and the ability to acquire knowledge quickly.
•The ability to collaborate effectively within a team.
•Experience working with business clients.
•Good communication skills and an analytical approach to problems.
Requirements
•Senior-level experience designing and building ETL/ELT processes and data pipelines.
•Good knowledge of SQL, including query optimization and working with large volumes of data.
•Practical knowledge of BigQuery: designing data structures, partitioning, clustering, and optimizing performance and costs.
•Experience programming in Python and/or Rust.
•Knowledge of batch and streaming data processing.
•Knowledge of event-driven architecture (Kafka or Pub/Sub).
•Experience designing logical and physical data models.
•Knowledge of relational databases, e.g. Oracle or PostgreSQL.
•Knowledge of Linux sufficient to work independently and diagnose problems.
•Ability to analyze business requirements and translate them into technical solutions.
Nice to Have
•Experience with Google Cloud services: BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage.
•Knowledge of Apache Spark (especially on Dataproc) and Apache Beam / Dataflow.
•Experience with Kubernetes, Docker, and CI/CD.
•Knowledge of Informatica PowerCenter or other ETL tools.
•Experience building a Data Lake, data warehouse, or data platform in a hybrid environment.
•Knowledge of campaign systems, customer segmentation, or marketing automation.
•Experience testing data quality, monitoring pipelines, and data observability.
•Knowledge of data security, access control, anonymization, and sensitive data processing.
•Knowledge of AutomateNOW.
Conditions
•B2B cooperation.
•Rate: 160 PLN/hour net.
•Cooperation period: over 12 months.
