JOB DESCRIPTION :

Role : Senior Data Engineer

Data & Analytics Engineering

About the Role :

We are looking for a Senior Data Engineer to join our growing Data & Analytics team. In this role, you will design, build, and maintain scalable data pipelines and platforms that power critical business decisions. You will be a key contributor in driving our cloud-first data strategy, leveraging Azure Databricks as a core technology and Azure Data Factory (ADF) as a good to have feature. You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.

Key Responsibilities :

Data Pipeline & Architecture :

- Design, develop, and maintain robust ETL/ELT pipelines using Azure Databricks and Azure Data Factory (ADF).

- Architect and implement scalable data lakehouse solutions on Azure using Delta Lake.

- Build and optimize data workflows across batch and streaming workloads (Spark Structured Streaming, Event Hubs).

- Define and enforce data modeling best practices (star schema, data vault, medallion architecture).

Databricks Engineering :

- Develop and optimize Spark-based data transformations using PySpark and Spark SQL in Databricks.

- Manage Databricks clusters, jobs, and workspace configurations for performance and cost efficiency.

- Implement Delta Live Tables (DLT) pipelines for declarative, auto-scaling data transformations.

- Leverage Unity Catalog for data governance, lineage tracking, and access control.

- Utilize Databricks Asset Bundles (DABs) and CI/CD practices for deployment automation.

Azure Data Factory (ADF) (Good To Have Feature) :

- Build, schedule, and monitor complex ADF pipelines with parameterized templates.

- Integrate ADF with Azure Key Vault, Linked Services, and Integration Runtimes (SHIR/Azure IR).

- Implement incremental load patterns, watermarking, and change data capture (CDC) strategies.

- Troubleshoot pipeline failures and optimize ADF pipeline performance and cost.

Data Quality & Governance :

- Implement data quality frameworks and validation checks across pipelines.

- Enforce data cataloging, lineage, and metadata management practices.

- Collaborate with data governance teams to ensure compliance with data policies and regulations (GDPR, HIPAA).

Collaboration & Leadership :

- Mentor junior data engineers and conduct code reviews.

- Work closely with data scientists and ML engineers to productionize machine learning models.

- Partner with DevOps/Cloud teams on infrastructure-as-code (Terraform/Bicep) for data platform provisioning.

- Document architecture decisions, pipeline designs, and operational runbooks.

Required Qualifications :

- 7+ years of hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark).

- Strong proficiency in SQL preferred and Python.

- Deep understanding of distributed computing principles and the Apache Spark ecosystem.

- Experience with Azure data services : ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka.

- Solid understanding of data warehousing concepts and dimensional modeling.

- Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions).

- Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep).

Preferred Qualifications :

- Databricks Certified Data Engineer Associate or Professional certification.

- Microsoft Certified : Azure Data Engineer Associate (DP-203).

- Experience with Delta Live Tables (DLT) and Databricks Workflows.

- Familiarity with Power BI or other BI/reporting tools.

- Experience with Scala or Java for Spark development.

Technology Stack :

Core Platforms :

- Azure Databricks (Preferred), Azure Data Factory (ADF) (Good to have)

Data Storage :

- Azure Data Lake Storage Gen2, Delta Lake, Azure SQL, Cosmos DB

Languages :

- Python (PySpark), SQL

Orchestration :

- ADF Triggers, Databricks Workflows, Apache Airflow

Monitoring :

- Azure Monitor, Log Analytics, Databricks Cluster Policies

Governance :

- Unity Catalog, Azure Purview, Azure Key Vault (Good to have)

BI / Reporting :

- Power BI, Tableau

Key Competencies :

- Strong analytical and problem-solving skills with attention to detail.

- Excellent communication skills - ability to translate complex technical concepts to non-technical stakeholders.

- Self-motivated and able to manage competing priorities in an agile environment.

- Collaborative team player with a growth mindset and eagerness to mentor others.

- Proactive in identifying performance bottlenecks and proposing architectural improvements.