Data Engineer - Azure Data Factory

Description :

- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services

- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark

- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation

- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes

- Create and maintain data models, schemas, and documentation to ensure data quality and consistency

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions

- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed

- Ensure data security, privacy, and compliance with industry standards and regulations

- Mentor junior team members and provide technical guidance on data engineering best practices

- Stay current with emerging technologies and industry trends in data engineering and cloud computing

- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables

- Design and implement data governance frameworks and data quality monitoring solutions

Experience :

- Overall 8+yrs of software knowledge

- 7+ years of relevant experience in Data Engineering

- Strong hands-on experience with Azure Data Engineering services

- 4+ yrs of Databricks and Lakeflow Declarative Pipelines

- Experience in building scalable data pipelines and ETL/ELT workflows

- Good understanding of data processing, transformation, and optimization techniques

- Experience working with large-scale structured and unstructured data

- Strong problem-solving and performance optimization skills in data platforms