Azure Databricks Engineer

Azure Databricks Engineer

Location : Bangalore / Hyderabad / Pune

Experience : 5+ Years

Employment Type : Full-Time / Contract

Role Overview :

We are looking for a highly skilled Azure Databricks Engineer with strong expertise in modern data engineering, big data processing, and distributed systems. The ideal candidate should have hands-on experience in building scalable ETL/ELT pipelines, optimizing Spark workloads, and working with enterprise grade Databricks environments.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure Databricks.

- Build and optimize ETL/ELT workflows for large-scale data processing.

- Work extensively with Apache Spark (PySpark & Spark SQL) for distributed data engineering.

- Implement Delta Lake architecture and optimize performance using Liquid Clustering and skew handling techniques.

- Develop reusable frameworks and automation solutions using Python.

- Optimize SQL queries, data models, and analytical workloads.

- Utilize Databricks Genie for engineering productivity and workflow acceleration.

- Collaborate with DevOps teams to implement CI/CD and Git-based deployment workflows.

- Work with orchestration and scheduling tools for automated pipeline execution.

- Ensure data quality, scalability, security, and reliability across platforms.

Required Skills :

- 5+ years of experience in Data Engineering or related domains.

- Strong hands-on experience with Azure Databricks.

- Expertise in :

a. Databricks Notebooks

b. Delta Lake

c. Job Orchestration

d. Data Skewing

e. Liquid Clustering

- Strong knowledge of Apache Spark :

a. PySpark

b. Spark SQL

c. Spark Optimization Techniques

- Strong Python programming skills.

- Advanced SQL knowledge including :

a. Complex Queries

b. Performance Tuning

c. Analytical Functions

- Experience with Databricks Genie.

- Strong understanding of CI/CD pipelines and Git workflows.

- Experience in data modeling and ETL/ELT pipeline design.

- Knowledge of automation frameworks and scheduling tools.

- Good understanding of distributed systems and big data architecture.

Preferred Qualifications :

- Experience with cloud platforms like Microsoft Azure.

- Exposure to enterprise-scale data lake architecture.

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration abilities.