Job Description:

Key Responsibilities:

- Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks

- Develop and optimize ETL/ELT processes for large-scale distributed data processing

- Build and maintain data lake and lakehouse architectures on Azure cloud platform

- Write efficient SQL and PySpark code for data transformation and analysis

- Implement Lakeflow Declarative Pipelines for streamlined data workflows

- Perform performance tuning and optimization of data processing systems

- Manage version control using Git and implement CI/CD pipelines for data solutions

- Collaborate with data scientists, analysts, and stakeholders to understand data requirements

- Ensure data quality, security, and governance standards are met across all pipelines

- Mentor junior team members and contribute to technical best practices

Mandatory Requirements:

- Azure Data Factory (ADF): Strong hands-on experience in designing and managing data pipelines

- Azure Databricks: Proficiency in building and optimizing distributed data processing solutions

- Apache Spark: In-depth knowledge of Spark architecture and distributed computing concepts

- Lakeflow Declarative Pipelines: Practical experience with declarative pipeline development

- SQL: Expert-level proficiency in writing complex queries and data manipulation

- PySpark: Strong programming skills in PySpark for data transformation

- Azure Cloud Platform: Hands-on experience with Azure data services and cloud infrastructure

- ETL/ELT Concepts: Strong understanding of data integration patterns and methodologies

- Data Warehousing Principles: Knowledge of dimensional modeling, schema design, and best practices

- Distributed Data Processing: Experience working with large-scale data systems and handling big data challenges

- Performance Tuning & Optimization: Ability to identify bottlenecks and optimize data pipeline performance

- Data Lake & Lakehouse Architectures: Understanding of modern data architecture patterns and implementation

- Git & CI/CD: Experience with version control systems and continuous integration/deployment practices

Preferred Skills & Experience:

- Experience with Delta Lake and Unity Catalog for data governance

- Knowledge of streaming frameworks and real-time data processing

- Exposure to Azure Synapse Analytics or Microsoft Fabric

- Working experience in Agile/Scrum development environments

- Familiarity with data governance frameworks and security best practices

Qualifications:

- Bachelor's degree in Computer Science, Engineering, Mathematics, or related field (or equivalent professional experience)

- Minimum 5+ years of experience as a Data Engineer or similar role

- Proven track record of delivering production-grade data solutions

- Strong problem-solving and analytical skills

- Excellent communication and collaboration abilities

- Ability to work independently and as part of a team

What We Offer:

- Competitive salary and benefits package

- Opportunity to work with cutting-edge data technologies

- Professional development and training opportunities

- Collaborative and inclusive work environment

- Flexible working arrangements

- Career growth and advancement opportunities