Roles & Responsibilities :

- Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration of enterprise-wide data from diverse sources.

- Build and optimize data engineering workflows using Databricks and PySpark.

- Write efficient, high-performance SQL for data transformation and analysis.

- Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing, data models, and pipelines.

- Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the development lifecycle.

- Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production environments with proper change control processes.

- Collaborate with cross-functional teams to translate business requirements into scalable data solutions.

- Ensure data quality, reliability, and performance across all pipelines and platforms.

Required Skillset :

- Demonstrated expertise in building and maintaining high-performance data platforms using PySpark, Azure Databricks, and Python.

- Proficient in orchestrating data workflows using Azure Data Factory and managing storage solutions within Azure Data Lake.

- Advanced capability in writing complex SQL queries and optimizing data models within Azure Synapse Analytics.

- Proven ability to integrate CI/CD practices into data engineering lifecycles to streamline development and deployment processes.

- Strong analytical and problem-solving skills with the ability to communicate technical concepts clearly to non-technical stakeholders.