Senior Engineer - Azure Databricks

Job Profile :

We are looking for a Senior Azure Data Engineer (Databricks) who can independently design, develop, and optimize cloud-based data solutions. The ideal candidate must be strong in Azure Databricks, Data Lakehouse architecture, ETL/ELT development, and data warehouse modelling. This role requires hands-on expertise in building and enhancing pipelines, migrating enterprise data (SAP/SQL Server) to cloud, and implementing scalable solutions using Azure services.

This is a development-heavy role (initially 100% development, later ~60% development) with ownership of end-to-end implementation projects.

Roles & Responsibilities :

- Work on Azure Databricks pipelinesenhance, modify, and optimize existing pipelines and develop new ones when required.

- Handle data migration from SAP/SQL Server and enterprise databases into Azure cloud environments (Lakehouse, Delta, Synapse).

- Configure, manage, and optimize Databricks clusters, compute, workspaces, and resources for performance and cost efficiency.

- Build scalable ETL/ELT processes using PySpark, SQL, Delta Lake, and medallion architecture across BronzeSilverGold layers.

- Work extensively on Azure Data Factory, orchestration pipelines, triggers, monitoring, and integration with Databricks.

- Design and maintain data warehouse & dimensional models (Facts, Dimensions, SCD, Star/Snowflake schema).

- Ensure strong data quality, validation, governance, and securityincluding Unity Catalog, RBAC, and Azure platform controls.

- Develop solutions independently, involving end-to-end development, not just supportinitially 100% development-focused.

Skill Set Required :

- Azure Databricks (Notebooks, Clusters, Delta Lake, Optimization, Resource Configuration)

- Azure Data Lake / Lakehouse Architecture

- Azure Data Factory (Pipelines, Orchestration, Integration, Monitoring)

- Azure SQL / SQL Server

- PySpark, SQL for transformation and performance tuning

- Data Warehouse Fundamentals

- Star & Snowflake schema

- Fact & Dimension modelling

- SCD, Data Marts

- Azure Cloud Services (Key Vault, Storage, Synapse basics, RBAC, security)

- ETL/ELT development (end-to-end implementation)