Job Summary
Lead the architecture, design, and technical strategy for the company's open-source data platform. Make the end-to-end technical vision — from ingestion to analytics — while mentoring a team of data platform engineers. This is a hands-on leadership role responsible for design the blueprints and write critical-path code.
Job Duties
Responsibilities
Architecture & Strategy
•Define the overall data platform architecture on OpenShift, covering the full lifecycle: ingestion (Kafka/Redpanda), compute (Spark, Flink, Trino), storage (MinIO, Iceberg), catalog (Polaris/Gravitino), orchestration (Airflow/Dagster), and serving (StarRocks/ClickHouse)
•Design the Data Lakehouse architecture: open table formats (Iceberg, Delta Lake), catalog federation, multi-engine interoperability
•Architect the real-time data path: Kafka topic design, schema registry (Apicurio/Confluent), stream processing (Flink/Spark Structured Streaming)
•Design the analytical serving layer: OLAP engine selection (StarRocks/ClickHouse/Doris), materialized view strategy, query federation with Trino
•Design multi-tenancy, data security (Ranger/OpenPolicyAgent), RBAC, and governance (DataHub/Atlas) across all platform layers
•Set technical standards for Helm chart design, CI/CD pipelines, and GitOps (ArgoCD/Flux) workflows
•Own data lifecycle management: hot/warm/cold tiering across MinIO, compaction, retention, and GDPR/deletion compliance
Technical Leadership
•Lead a team of 3–6 data platform engineers; run code reviews, design reviews, and sprint planning
•Establish engineering best practices: testing, observability (OpenTelemetry, Prometheus, Grafana, Loki), incident response, runbooks
•Partner with data engineering, analytics, and business teams to translate their needs into platform capabilities
•Define platform SLOs/SLIs across freshness, latency, availability, and durability; drive the on-call rotation and incident post-mortems
Hands-on Engineering
•Build and maintain the core OpenShift infrastructure: operators, Helm charts, namespaces, RBAC, network policies
•Develop Spark and Flink job frameworks, tuning guides, workload scheduling (YuniKorn/Volcano)
•Implement the data mesh or data product model: domain ownership, self-serve data infrastructure, federated governance
•Performance-optimize across the stack: Kafka throughput, Spark shuffle, Iceberg compaction, Trino query planning, OLAP cold reads
Required Skills & Experience
•10+ years in data/platform engineering, with 3+ years as architect or tech lead
•Deep Kubernetes/OpenShift: operators, Helm, CRDs, admission webhooks, SCC, multi-tenancy at production scale
•Expert-level streaming: Kafka/Redpanda internals (partitioning, consumer groups, exactly-once semantics), schema registry, Kafka Connect
•Expert-level batch & streaming compute: Spark internals (Catalyst, Tungsten, AQE, shuffle, dynamic allocation), Flink or Spark Structured Streaming
•Strong open table format knowledge: Iceberg and/or Delta Lake — table format internals, metadata evolution, partitioning transforms, maintenance, catalog integration (Polaris, Gravitino, Nessie, or Unity Catalog)
•Hands-on with federated query engines: Trino (coordinator/worker architecture, connector ecosystem, fault-tolerant execution mode)
•Solid OLAP knowledge: StarRocks, ClickHouse, or Apache Doris — primary key models, materialized views, query optimization
•Production MinIO/S3-compatible storage at scale: erasure coding, multi-site replication, IAM, tiering
•Pipeline orchestration: Airflow (DAG design, sensors, dynamic task mapping) or Dagster
•Infrastructure-as-code (Terraform/Pulumi/Crossplane) and GitOps (ArgoCD/Flux)
•Fluent in at least one JVM language (Scala/Java) and Python
Nice to Have
•Experience deploying dbt (data build tool) with Trino or Spark adapters for data transformation
•Data governance hands-on: DataHub or Apache Atlas for lineage, glossary, and metadata search
•Data security: Apache Ranger or OpenPolicyAgent for fine-grained access control
•Open-source contributions to any of the above projects
Company Introduction
ITD SZ
HKEX Technology (Shenzhen) Limited is a wholly foreign-owned enterprise established on December 28, 2016 in the Qianhai Free Trade Zone of Shenzhen. As the technology subsidiary of HKEX, HKEX Technology (Shenzhen) Limited primarily provides the group and its affiliated companies with development, technical services, technical consulting, and technology transfer for computer software, computer hardware, information systems, cloud storage, cloud computing, the Internet of Things, and computer networks; economic information consulting, enterprise management consulting, business information consulting, commercial information consulting, information system design, integration, and operation and maintenance; database management, big data analysis; and system application management and maintenance, information technology support management, data processing and other information technology and business process outsourcing services provided through service outsourcing.
