Lead Data Engineer
Inviting applications for the role of Principal Consultant - Databricks Lead Developer!
In this role, the Databricks Developer is responsible for solving real world cutting edge problems to meet both functional and non-functional requirements.
•Maintains close awareness of new and emerging technologies and their potential application for service offerings and products
•Work with architect and lead engineers for solutions to meet functional and non-functional requirements
•Demonstrated knowledge of relevant industry trends and standards
•Demonstrate strong analytical and technical problem-solving skills
•Must have experience in Data Engineering domain
Requirements
•Bachelor's Degree or equivalency (CS, CE, CIS, IS, MIS, or engineering discipline) or equivalent work experience
•Overall 8+ years of experience in IT
•Maintains close awareness of new and emerging technologies and their potential application for service offerings and products
•Work with architect and lead engineers for solutions to meet functional and non-functional requirements
•Demonstrated knowledge of relevant industry trends and standards
•Demonstrate strong analytical and technical problem-solving skills
•Must have excellent coding skills either Python, Spark/Scala, preferably Python & Spark
•Must have experience in Data Engineering domain
•Must have implemented at least 2 projects end-to-end in Databricks
•Must have at least experience on Databricks which consists of various components as below: AutoLoader, Delta Lake, Unity Catalog, Lakeflow Connect, DB API 2.0, Databricks workflows orchestration
•Must be well versed with the Databricks Lakehouse concept and its implementation in enterprise environments
•Must have good understanding to create complex data pipelines
•Must have good knowledge of Data structure & algorithms
•Must be strong in SQL and Spark-SQL
•Must have strong performance optimization skills to improve efficiency and reduce cost
•Must have worked on both Batch and streaming data pipelines
•Must have extensive knowledge of Spark and Hive data processing framework
•Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases
•Must be strong in writing unit test cases and integration tests
•Must have strong communication skills and have worked on a team of size 15+
•Must have great attitude towards learning new skills and upskilling existing skills
Preferred Qualifications
•Good to have Genie One, MosaicAI, DQX, and basic governance knowledge
•Good to have Databricks SQL Endpoint understanding
•Good to have CI/CD experience to build pipelines for Databricks jobs
•Good to have if worked on a migration project to build a Unified data platform
•Good to have knowledge of DBT
•Good to have knowledge of Docker and Kubernetes
Qualifications
Bachelors - Business Analytics, Bachelors - Computer Science, Bachelors - Statistics, Masters - Data Science
Certifications
Certified Data Management Professional - Riversand, Databricks Certified Associate Developer for Apache Spark 3.0 - Databricks Academy, Databricks Certified Associate Developer for Apache Spark - Databricks Academy, Microsoft Certified: Azure Data Engineer Associate - Microsoft, Oracle Database 12c Certified Implementation Specialist - Oracle
Required Skills
Agile Methodology, Banking Capital Markets, Client Relations, Cloud Computing, Collaboration Tools, Commercial Sales, Consumer Packaged Goods (CPG), Databricks Platform, Data Engineering, Data Literacy, Design Thinking, Electronic Hardware, Executive Presence, Inclusion, Manufacturing Equipment, People Leadership, Personal Effectiveness, Risk Management, Semiconductors, Snowflake (Platform), Sourcing and Procurement, Storytelling
Language
English (Required) - Proficient - C2
Job Type
Regular
Remote Type
Hybrid
Work Shift
Flex Time (India)
