Job Description :

- Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks

- Develop and optimize ETL/ELT processes for large-scale distributed data processing

- Build and maintain data lake and lakehouse architectures on Azure cloud platform

- Write efficient SQL and PySpark code for data transformation and analysis

- Implement Lakeflow Declarative Pipelines for streamlined data workflows

- Perform performance tuning and optimization of data processing systems

- Manage version control using Git and implement CI/CD pipelines for data solutions

- Collaborate with data scientists, analysts, and stakeholders to understand data requirements

- Ensure data quality, security, and governance standards are met across all pipelines

- Mentor junior team members and contribute to technical best practices

Required Skills & Experience :

Mandatory Requirements :

- Azure Data Factory (ADF) : Strong hands-on experience in designing and managing data pipelines

- Azure Databricks : Proficiency in building and optimizing distributed data processing solutions

- Apache Spark : In-depth knowledge of Spark architecture and distributed computing concepts

- Lakeflow Declarative Pipelines : Practical experience with declarative pipeline development

- SQL : Expert-level proficiency in writing complex queries and data manipulation

- PySpark : Strong programming skills in PySpark for data transformation

- Azure Cloud Platform : Hands-on experience with Azure data services and cloud infrastructure

- ETL/ELT Concepts : Strong understanding of data integration patterns and methodologies

- Data Warehousing Principles : Knowledge of dimensional modeling, schema design, and best practices

- Distributed Data Processing : Experience working with large-scale data systems and handling big data challenges

- Performance Tuning & Optimization : Ability to identify bottlenecks and optimize data pipeline performance

- Data Lake & Lakehouse Architectures : Understanding of modern data architecture patterns and implementation

- Git & CI/CD : Experience with version control systems and continuous integration/deployment practices

- Strong experience with Azure Data Factory (ADF) and Azure Databricks

- Hands-on expertise in Apache Spark and PySpark

- Experience with Lakeflow Declarative Pipelines

- Strong proficiency in SQL and data modeling

- Solid understanding of ETL/ELT processes and data warehousing concepts

- Experience building and managing Data Lake/Lakehouse architectures

- Hands-on experience with Azure cloud data services

- Proven experience processing and optimizing large-scale data workloads

- Strong knowledge of performance tuning for Spark/Databricks environments

- Experience with Git, version control, and CI/CD implementation

- Notice Period Immediate to 30days only