Key Responsibilities:

- Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.

- Build robust ETL/ELT workflows for batch and real-time data processing.

- Develop data transformations using PySpark, Python, and SQL.

- Design and implement data lake/lakehouse and data warehouse solutions.

- Integrate data from multiple structured and unstructured sources.

- Optimize data pipelines, queries, and Spark workloads for performance and cost.

- Implement data quality, validation, monitoring, and error-handling mechanisms.

- Work with cloud-based data storage and Azure data services.

- Collaborate with architects, data scientists, analysts, and application teams.

- Follow engineering best practices around CI/CD, security, governance, and documentation.

Required Skills :

- 8 to 15 years of experience in data engineering.

- Strong hands-on experience with Azure Databricks.

- Strong hands-on experience with Azure Data Factory (ADF).

- Excellent PySpark, Python, and SQL skills.

- Strong understanding of ETL/ELT concepts and data pipeline architecture.

- Experience with Data Lake / Lakehouse architecture.

- Experience with data modeling and performance optimization.

- Good understanding of Microsoft Azure data services.

- Experience working in enterprise-scale data environments.