Role Overview :

We are seeking a high-caliber Technical Lead to spearhead the design and development on our enterprise-scale Databricks data platform. As a hands-on lead, you will be the primary authority on our Azure-based Lakehouse, ensuring that data pipelines are not just functional, but scalable, secure, and cost-optimized. You must have a proven track record of leading end-to-end implementations-from initial discovery and schema design to production deployment and monitoring.

Core Requirements (Non-Negotiable) :

- 10 - 12+ years of experience in Data Engineering

- Azure Mastery : Minimum 3+ years of lead experience specifically within the Azure cloud ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault).

- Databricks Expertise : Deep hands-on experience with Databricks (Spark/PySpark), specifically implementing Medallion Architecture (Bronze, Silver, Gold layers).

- Proven Leadership : Must have led at least two full-lifecycle, end-to-end data platform implementations in a Tech Lead capacity.

- Proven Tech Lead experience on both Databricks and Azure

- Excellent understanding of Lakehouse architecture

- Strong SQL skills and data modeling expertise

Key Responsibilities :

- Architectural Design : Lead the design and evolution of our Delta Lakehouse architecture, ensuring it supports both high-volume and complex batch processing.

- End-to-End Implementation : Take full ownership of the data lifecycle : ingestion from disparate sources (APIs, RDBMS, NoSQL), complex transformations in PySpark, and final exposure via Gold tables or semantic layers.

- Technical Mentorship : Guide a team of 3-5 data engineers through code reviews, design workshops, and the establishment of "best-in-class" Python and SQL coding standards.

- Optimization : Conduct performance tuning of Spark clusters by exploring all available options like optimizing shuffle partitions, and managing "Small File" problems to reduce Azure consumption costs.

- Data Governance : Implement and manage Databricks Unity Catalog for centralized fine-grained access control, lineage tracking, and data discovery.

- Handle production issues, root-cause analysis, and long-term fixes.

Technical Skills & Toolset :

- Languages : Expert-level Python (PySpark) and Advanced SQL. Familiarity with .net framework or Java is a plus.

- Orchestration : Advanced use of Azure Data Factory (ADF) and Databricks Workflows/Jobs for complex dependency management.

- DevOps/DataOps : Strong proficiency in Azure DevOps or GitHub Actions, specifically building CI/CD pipelines for automated Databricks notebook deployment and Infrastructure as Code (Terraform/Bicep).

- Observability : Implementation of data quality frameworks (e.g., Great Expectations or Databricks Expectations) and monitoring via Azure Monitor/Log Analytics.

Preferred Qualifications :

- Certifications : Microsoft Certified : Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.

- AI/ML Integration : Experience supporting ML workloads by building feature stores and integrating with MLflow.

- Agile : Experience working in a high-velocity Scrum and Kanban environment.

Soft Skills :

- Strong ownership mindset and accountability

- Excellent communication with technical and non-technical stakeholders