About the Role :

We are seeking an experienced Senior Databricks Engineer to lead the design, development, and optimization of our modern Data & Analytics platform. This role will be responsible for building scalable Lakehouse architectures, developing high-performance data pipelines, implementing governance frameworks, and driving engineering excellence across the data ecosystem.

The ideal candidate will possess deep expertise in Databricks, Azure Data Services, PySpark, and modern Data Engineering practices, along with a strong focus on scalability, reliability, and performance.

Key Responsibilities :

Databricks Platform Engineering :

- Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, Unity Catalog, and Medallion Architecture.

- Develop and manage Delta Live Tables (DLT) pipelines for reliable and maintainable data transformation workflows.

- Implement and govern data assets through Unity Catalog, including access control, lineage, and security policies.

- Optimize Spark workloads and Delta tables using partitioning, Z-ordering, liquid clustering, and performance tuning techniques.

- Build and orchestrate complex workflows using Databricks Workflows.

Data Engineering & Integration :

- Design, develop, and maintain enterprise-scale ELT/ETL pipelines.

- Integrate data from APIs, databases, flat files, and streaming platforms into the Databricks Lakehouse.

- Build real-time and near real-time data pipelines using Structured Streaming and Delta Live Tables.

- Work extensively with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics, and Event Hubs.

DevOps & Automation :

- Develop clean, reusable, and scalable code using PySpark, Python, and SQL.

- Implement CI/CD pipelines using Azure DevOps for automated testing and deployments.

- Manage infrastructure using Terraform or Bicep.

- Package and deploy Databricks solutions using Databricks Asset Bundles (DABs).

Data Governance & Quality :

- Implement automated data quality frameworks using Great Expectations and DLT Expectations.

- Maintain end-to-end data lineage through Unity Catalog and Microsoft Purview.

- Create and maintain technical documentation, architecture diagrams, and operational runbooks.

Leadership & Collaboration :

- Mentor and guide junior engineers through code reviews and technical coaching.

- Collaborate with business stakeholders, analysts, and data modelers to deliver scalable analytics solutions.

- Establish engineering best practices, coding standards, governance frameworks, and SDLC processes.

Required Skills & Qualifications :

Mandatory Certifications :

- Databricks Certified Data Engineer Associate or Professional

- Microsoft Certified : Azure Data Engineer Associate

Technical Expertise :

- 10+ years of experience in Data Engineering.

- Strong hands-on experience with :

1. Databricks

2. Delta Lake

3. Unity Catalog

4. Delta Live Tables (DLT)

5. Databricks Workflows

- Advanced proficiency in :

1. PySpark

2. Python

3. SQL

- Experience with :

1. Azure Data Factory (ADF)

2. Azure Data Lake Storage Gen2 (ADLS)

3. Azure Synapse Analytics

4. Azure DevOps

5. Terraform/Bicep

- Strong understanding of :

1. Data Warehousing

2. Dimensional Modeling

3. Lakehouse Architecture

4. Medallion Architecture

- Experience with real-time data processing and Structured Streaming.

Preferred Skills :

- Microsoft Purview

- dbt

- MLflow

- Databricks Feature Store

- Databricks Asset Bundles (DABs)

- Azure Functions

- Azure Logic Apps

- Azure Event Hubs

- Data Mesh Architecture

- Data Product Thinking