Azure Data Engineer - ETL/PySpark

Job Description : Data Engineer

Position : Data Engineer

Experience : 8+ Years

Location : Indore / Pune

Notice Period : Immediate to 15 days

Role Summary :

Key Responsibilities :

- Pipeline Development Design, develop, and optimize ETL/ELT pipelines using Python, PySpark, and Databricks for large-scale data processing.

- Medallion Architecture Establish best practices for ingestion, transformation, and storage following the Bronze, Silver, Gold architecture.

- Scalable Data Applications Build and deploy distributed data applications on Azure Databricks for performance and reliability.

- Performance Optimization Optimize Databricks clusters, jobs, and ETL workflows for efficiency and scalability.

- Data Storage & Governance Manage Azure Data Lake Storage (ADLS) and Delta Lake with Unity Catalog for data governance, security, and compliance.

- Automation & Scheduling Develop and schedule Databricks notebooks/jobs, with monitoring, alerting, and automated recovery for failures.

- Code Optimization Troubleshoot, resolve bottlenecks, and follow best coding practices to improve performance and maintainability.

- Version Control & DevOps Use GitHub for version control and implement CI/CD pipelines for deployment/testing using Azure DevOps.

- Documentation Create comprehensive documentation for data architecture, pipelines, and business logic.

Required Skills :

- Proven experience in ETL/ELT pipeline development and large-scale data processing.

- Strong understanding of Medallion architecture (Bronze, Silver, Gold).

- Hands-on with ADLS, Delta Lake, and Unity Catalog.

- Strong knowledge of Python, Spark optimization, and distributed computing.

- Experience in cluster optimization and performance tuning.

- Proficiency in GitHub, CI/CD pipelines, and Azure DevOps.

- Excellent problem-solving skills with the ability to troubleshoot and optimize code.

- Strong communication and documentation skills.

Good to Have :

- Familiarity with BI/reporting tools.

- Exposure to data security, compliance, and governance frameworks.