Role Description :

We are looking for an experienced Data Engineer with strong expertise in Azure Databricks, PySpark, Spark SQL, Delta Lake, and Airflow. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines, migrating legacy ETL workloads, and delivering enterprise-grade data engineering solutions.

Job Location :

Pune, India.

Experience :

7+ Years

Roles & Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.

- Migrate SQL Server stored procedures and legacy ETL processes into Databricks notebooks and workflows.

- Develop incremental and CDC-based data pipelines using Delta Lake.

- Build and optimize ETL/ELT workflows for structured and semi-structured data.

- Develop Spark SQL and Databricks SQL transformations for large datasets.

- Design and manage workflow orchestration using Apache Airflow or equivalent schedulers.

- Perform source-to-target data mapping, validation, reconciliation, and data quality checks.

- Optimize Spark jobs for performance, scalability, and cost efficiency.

- Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and Azure services.

- Collaborate with Business Analysts, Architects, QA, and Development teams throughout the project lifecycle.

- Participate in code reviews, deployment activities, production support, and performance tuning.

- Follow Agile development methodologies and CI/CD best practices.

Required Skills:

- Strong experience in Azure Databricks.

- Hands-on experience with PySpark, Spark SQL, and Databricks SQL.

- Expertise in Delta Lake, MERGE operations, CDC, and incremental data loads.

- Strong SQL and SQL Server knowledge.

- Experience with Apache Airflow or similar workflow orchestration tools.

- Experience with Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF).

- Strong understanding of Data Warehousing, ETL/ELT, and Lakehouse architecture.

- Experience with Git, CI/CD, and Agile methodologies.

- Excellent problem-solving, analytical, and communication skills.

Preferred Skills :

- Snowflake

- Kafka / Azure Event Hub

- Structured Streaming

- Unity Catalog

- Microsoft Purview

- Great Expectations / dbt

- Azure DevOps

- Databricks Certification

- Banking or Financial Domain Experience

Qualifications :

- Bachelor's degree in computer science, Information Technology, Engineering, or related field.

- 7 - 12 years of experience in Data Engineering.

- Minimum 4+ years of hands-on experience with Azure Databricks.