Azure Data Engineer - ETL Pipeline

Description :

Role Overview :

We are seeking a highly skilled Azure Data Engineer with extensive hands-on experience in designing, building, and maintaining large-scale data engineering solutions using Azure Databricks, Synapse, and ADF (Azure Data Factory).

The ideal candidate will possess strong expertise in ETL/ELT pipeline development, data modeling, and performance optimization, along with a deep understanding of Spark-based environments and Azure ecosystem.

The role demands technical leadership, problem-solving acumen, and the ability to collaborate effectively with cross-functional teams to deliver high-quality, scalable, and cost-efficient data solutions.

Key Roles & Responsibilities :

- Design, build, and maintain ETL/ELT pipelines for large, complex data sets ensuring performance, reliability, and scalability.

- Develop and manage data processing workflows using Azure Databricks, Synapse, ADF, and Airflow.

- Optimize data pipelines for cost, performance, and scalability while maintaining robust data quality and governance.

- Implement data transformations, feature engineering, and business logic for analytics and AI/ML pipelines.

- Manage and monitor Azure Data Lake Storage (ADLS), Delta Lake, and Azure SQL Database for structured and unstructured data.

- Collaborate with business stakeholders, data scientists, and analysts to understand data requirements and deliver actionable insights.

- Integrate CI/CD pipelines using GitHub and Azure DevOps for automated deployment and version control.

- Work closely with architecture teams to implement best practices around security, data lineage, and cloud cost management.

- Participate in Agile/Scrum ceremonies, contribute to sprint planning, and ensure timely delivery of project milestones.

- Provide technical mentorship and guidance to junior team members.

Mandatory Skills :

- 6 to 8 years of total IT experience with 6+ years in Data Engineering.

- Strong expertise in Azure Databricks (ADB), Delta Lake, Azure Synapse, ADF (Azure Data Factory), and Azure SQL DB.

- Proven experience in SparkCore, PySpark, SparkSQL, Scala, and Python.

- Hands-on experience with Azure Data Lake Storage (ADLS) and Azure Blob Storage.

- Skilled in ETL/ELT pipeline development within a Spark-based environment.

- Proficient in SQL (query optimization, data transformation, and performance tuning).

- Experience with CI/CD, GitHub, and Azure DevOps for version control and continuous integration.

- Strong knowledge of Azure Cloud Platform (compute, storage, and data services).

- Proven experience in requirements gathering, client engagement, and managing cross-functional teams.

Preferred Skills (Good to Have) :

- Experience with containerization technologies such as Docker and Kubernetes.

- Exposure to real-time data processing using Apache Kafka, Azure EventHub, IoT, Structured Streaming, or Stream Analytics.

- Experience working in Agile methodology and familiarity with DevOps best practices.

- Knowledge of data architecture design, data modeling, and metadata management.

- Strong analytical, problem-solving, and communication skills.

Certifications (Preferred) :

- Microsoft Certified : Azure Data Engineer Associate

- Databricks Certified Associate

- Azure Big Data Certification