Job Description :

What We Are Looking For :

- 7+ years in data engineering with a deep Azure focus.

- End-to-end ownership of Databricks and lakehouse architecture.

- Proven ETL modernization at enterprise scale.

- Native fluency across Azure Data Factory, Microsoft Fabric, Synapse, and ADLS Gen2.

- Production-level programming in PySpark, Scala, and Python.

Required Skills and Competencies :

- Azure Data Factory : pipeline authoring, parameterization, triggers, and integration runtimes.

- Azure Databricks : cluster management, notebook development, job orchestration, and Delta Live Tables.

- Azure Data Lake Storage Gen2 : hierarchical namespace, access tiers, and lifecycle policies.

- Azure Synapse Analytics : dedicated and serverless SQL pools, pipelines, and integration with Databricks.

- Delta Lake : ACID transactions, schema evolution, merge operations, and performance tuning.

- Microsoft Fabric : unified analytics across Lakehouse, Data Factory, and Real-Time Intelligence workloads.

Programming and Processing :

- Advanced PySpark and/or Scala for large-scale distributed data processing.

- Python for scripting, automation, and data transformation logic.

- SQL proficiency for complex analytics and data modelling.

- Strong grasp of distributed computing, shuffle optimization, and caching strategies.

Architecture Patterns :

- Lakehouse architecture and medallion design (Bronze, Silver, Gold).

- Modern data warehouse patterns and dimensional modelling.

- Event-driven architecture and real-time streaming pipelines.

DevOps and Collaboration :

- CI/CD for data pipelines : GitHub Actions, Azure DevOps, or equivalent.

- Infrastructure as Code : Terraform or Azure Bicep.

- Orchestration : ADF, Apache Airflow, or Databricks Workflows.

- Strong Git practices : branching strategies, PR reviews, and collaborative development.

Technology Stack :

1. Data Storage :

- ADLS Gen2, Azure Blob Storage, Azure SQL Database, Delta Lake

2. Data Processing :

- Databricks, Azure Synapse Analytics, Azure Data Factory, Apache Spark

3. Streaming :

- Azure Event Hubs, Azure Stream Analytics, Kafka, Spark Streaming

4. Analytics & BI :

- Databricks SQL, Azure Analysis Services, Power BI

5. ML / AI :

- Azure Machine Learning, MLflow, Databricks ML

6. Security :

- Azure Active Directory, Key Vault, RBAC, data encryption

7. Governance :

- Unity Catalog, Azure Purview, Data Lineage

8. DevOps :

- Azure DevOps, GitHub Actions, Terraform, Bicep

9. Compliance :

- GDPR, access controls, audit logging

Preferred / Good to Have :

- Data mesh principles and domain-oriented data ownership.

- Exposure to ML enablement on Azure (Azure Machine Learning, MLflow, Databricks ML).

- Familiarity with BI and analytics tooling such as Power BI and Databricks SQL.