Work Timing : 12:00 noon 9:00 PM

Location : PAN India Office (Except Gurgaon, Coimbatore, Kolkata, Vijayawada)

Notice Period : Immediate or Serving NP (= < 21 Days)

Primary Skills :

Azure Data Factory (ADF), Azure Databricks, Pyspark, SQL ADLS, ADB, Azure Services / Key Vault, Git / Jenkins / Azure Devops, CICD.

Job Description :

We are seeking an experienced Azure Databricks Engineer to design, build, and optimize data engineering pipelines and analytics solutions on Microsoft Azure.

We are seeking an experienced Azure Databricks Engineer to design, build, and optimize data engineering pipelines and analytics solutions on Microsoft Azure.

The ideal candidate will have strong hands-on skills in Azure Data Factory (ADF), Azure Databricks, PySpark, SQL, ADLS, and experience with Azure DevOps /

Git / CI-CD, along with various Azure cloud services.

Key Responsibilities :

- Design, build, and maintain large-scale data ingestion, transformation, and ETL pipelines using ADF and Azure Databricks.

- Develop high-performance PySpark scripts & notebooks for data processing.

- Optimize cluster usage, performance tuning, parallel processing, and caching strategy in Databricks.

- Implement data workflows using Delta Lake, structured streaming, and batch processing.

- Utilize Azure Data Lake Storage (ADLS) for structured/semi-structured data storage.

Integrate pipelines across Azure services such as :

- Azure Key Vault

- Azure Event Hub

- Azure Service Principal-based authentication

- Azure SQL / Synapse Analytics

- Manage deployment, configuration, and versioning of Databricks notebooks and jobs.

- Ensure data accuracy, validation, and consistency across ETL pipelines.

- Implement best practices for data governance, cataloging, lineage, and security.

- Work with stakeholders to understand business rules and convert them into ETL logic.

- Use Git, Jenkins, or Azure DevOps for version control and CI/CD workflow automation.

- Build automated deployment pipelines for Databricks notebooks, clusters, and ADF components.

- Manage release cycles, code branching, tagging, and environment management.

- Work closely with data architects, analysts, BI teams, and business leads to deliver end-to-end cloud data solutions.

- Participate in design sessions, code reviews, sprint planning, and requirement analysis.

- Provide support for production pipelines and debugging issues in real-time.

Education :

- Bachelors or Masters degree in Computer Science, IT, Data Engineering, or a related field.