Description :

Title : Azure Data Engineer

Experience : 8+ Years

Work Location : Indore/Pune

Notice Period : Immediate to 15 Days Max only

Job Summary :

- Design and implement a scalable data warehouse on Azure Databricks using data and dimensional modeling techniques to support analytical and reporting requirements.

- Develop and optimize ETL/ELT pipelines using Python, Azure Databricks and PySpark for large-scale data processing, ensuring data quality, consistency, and integrity.

- Establish and implement best practices for data ingestion, transformation, and storage using the medallion architecture (Bronze, Silver, Gold).

- Architect and develop highly scalable data applications using Azure Databricks and distributed computing.

- Optimize Databricks clusters and ETL/ELT workflows for performance and scalability.

- Manage data storage solutions using Azure Data Lake Storage (ADLS) and Delta Lake while leveraging Unity Catalog for data governance, security, and access control.

- Develop and schedule Databricks notebooks and jobs for automated daily execution, implementing monitoring, alerting, and automated recovery processes for job failures.

- Identify and resolve bottlenecks in existing code and follow best coding practices to improve performance and maintainability.

- Use GitHub as version control tool to manage code and collaborate effectively with other developers; build and maintain CI/CD pipelines for deployment and testing using Azure DevOps and GitHub.

- Create comprehensive documentation for data architecture, ETL processes, and business logic.