Azure/Databricks Senior Data Engineer

We are currently hiring for a Lead Data Engineer for one of our clients (a leading Software Services/Product Company) based out of Mumbai.

We are looking for experienced Data Engineering professionals who are passionate about building modern, scalable, and cloud-native data platforms.

The ideal candidate should have strong expertise in Azure Databricks, Apache Spark, Azure Data Services, and enterprise-scale data engineering.

This role requires both hands-on technical leadership and the ability to mentor engineering teams while delivering high-quality data solutions.

This role offers above-market compensation along with standard benefits for the right candidate.

Key Responsibilities :

- Design, build, and maintain scalable end-to-end data pipelines for data ingestion, transformation, and serving.

- Develop and optimize large-scale data processing solutions using Azure Databricks and Apache Spark.

- Design and implement Delta Lake architectures following Bronze, Silver, and Gold data layers.

- Build cloud-native data solutions using Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory, Azure Key Vault, Event Grid, and API Management.

- Lead and execute enterprise data migration initiatives, including on-premises migrations, cross-workspace migrations, Unity Catalog migrations, and historical data loads.

- Implement CI/CD pipelines and automation for data engineering workflows and infrastructure.

- Develop Infrastructure as Code (IaC) solutions using Terraform and Databricks Asset Bundles.

- Implement data governance, security, and access management using Unity Catalog, RBAC, and Service Principals.

- Monitor production environments, troubleshoot incidents, and ensure platform reliability and performance.

- Lead technical discussions, mentor engineering teams, conduct code reviews, and drive engineering best practices across projects.

Required Skills/Primary Skills :

- 10+ years of experience in Data Engineering and Big Data technologies.

- Minimum 3+ years of hands-on experience with Azure Databricks and Apache Spark.

- Strong experience building end-to-end enterprise data pipelines.

- Deep understanding of Delta Lake Architecture including Bronze, Silver, and Gold data layers.

- Strong hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).

- Experience with Azure Data Factory (ADF) for orchestration and pipeline development.

- Hands-on experience with Azure Key Vault, Event Grid, and Azure API Management.

- Experience delivering large-scale cloud-based data platforms on Microsoft Azure.

- Strong experience with enterprise data migration projects, including historical data migration and platform modernization.

- Experience implementing CI/CD pipelines for data engineering workloads.

- Hands-on expertise with Terraform and Databricks Asset Bundles.

- Strong understanding of data governance using Unity Catalog, RBAC, and Service Principals.

- Experience supporting production environments, monitoring data platforms, and resolving production incidents.

Additional Skills :

- Experience leading and mentoring Data Engineering teams.

- Strong understanding of software engineering best practices, code reviews, and technical governance.

- Experience designing highly available, scalable, and secure cloud data platforms.

- Strong problem-solving and troubleshooting skills in distributed data processing environments.

- Experience working in Agile development environments.

- Excellent communication and stakeholder management skills.

- Ability to drive architecture decisions and establish engineering best practices across teams.

Notice Period :

- Immediate Joiners Preferred.