Описание
The enterprise data platform team is building a Lakehouse that powers a web app, LLM agents, and ML models.
Задачи
•Build and maintain a Databricks Lakehouse using a medallion architecture
•Ingest data from Oracle systems and APIs into the bronze layer
•Model the silver layer with Data Vault and the gold layer with dimensional models
•Optimize Spark performance and Delta table layouts
•Implement governance, access control, and PII masking in Unity Catalog
•Replace manual scheduling processes and spreadsheets with a unified data platform
•Collaborate with the application team building on the gold layer for the web app, LLMs, and ML models
Требования
•Strong experience with Databricks and Lakehouse medallion architecture (bronze / silver / gold)
•Advanced PySpark skills for building transformations in notebooks
•Hands-on experience with Delta Lake
•Experience with Data Vault modeling (hubs, links, satellites)
•Experience with dimensional modeling (facts and dimensions)
•Experience ingesting data with Fivetran and custom API integrations
•Experience working with Oracle SQL databases as source systems
•Spark performance tuning, including skew handling and Liquid Clustering
•Experience deploying with Databricks Asset Bundles (DABs)
•Experience with Unity Catalog, group-based RBAC, and PII masking
Будет плюсом
•Great Expectations or similar data quality frameworks
•Handling unstructured files in Blob storage
•Supporting LLM/agent and ML use cases with curated data
Условия
•Performance-based bonus
