Job Description:

Key Responsibilities:

- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services

- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark

- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation

- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes

- Create and maintain data models, schemas, and documentation to ensure data quality and consistency

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions

- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed

- Ensure data security, privacy, and compliance with industry standards and regulations

- Mentor junior team members and provide technical guidance on data engineering best practices

- Stay current with emerging technologies and industry trends in data engineering and cloud computing

- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables

- Design and implement data governance frameworks and data quality monitoring solutions

Required Qualifications:

- 8+ years of total professional experience in software development, data engineering, or related fields

- 7+ years of hands-on experience in data engineering roles with proven expertise in designing and implementing data solutions

- Extensive experience with Azure Data Factory (ADF) for building and orchestrating data pipelines

- Strong proficiency in Azure Databricks and Apache Spark for large-scale data processing

- Hands-on experience with Lake flow Declarative Pipelines and modern data pipeline architectures

- Advanced SQL skills with experience in query optimization and database design

- Proficient in PySpark and Python for data processing and transformation tasks

- Experience with Azure cloud services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics

- Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT processes

- Knowledge of data governance, data quality, and data security best practices

- Experience with version control systems (Git) and CI/CD pipelines

- Excellent problem-solving skills and attention to detail

- Strong communication skills and ability to work effectively in cross-functional teams