- Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.

- Build and optimize data architectures and data models (Lakehouse / medallion, dimensional) for efficient data storage and retrieval.

- Develop ETL/ELT processes to transform and load data from various sources into data warehouses and data lakes.

- Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL and Delta Lake, orchestrated with Databricks Workflows.

- Integrate data from enterprise source systems including SAP (ABAP/CDS extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.

- Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB) and Azure DevOps (Git repositories, YAML build and release pipelines), promoting code across dev, QA and production.

- Implement data quality, validation, freshness and reconciliation checks with pipeline observability.

- Ensure data integrity, quality, and security across all data systems.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

- Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.

- Document data processes, architectures, and data flow diagrams.

Preferred candidate profile :

- 7 - 8 years of hands-on data engineering experience building and running production data pipelines at scale.

- Strong expertise in Azure and Azure data services (ADLS Gen2, Azure Databricks, Azure DevOps).

- Deep hands-on experience with Databricks : PySpark, Spark SQL, Delta Lake, Lakehouse / medallion architecture and Databricks Workflows.

- CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure DevOps (Git, YAML build/release pipelines, multi-environment promotion). (Must-have)

- Strong Snowflake experience (data modeling, performance tuning, loading and optimization).

- Proficiency in SQL and Python.

- Experience integrating data from SAP and other enterprise ERP / source systems into a data lake or warehouse.

- Solid data modeling (dimensional, star/snowflake, Lakehouse) and ETL/ELT design.

- Building data-quality, validation, reconciliation and pipeline monitoring / observability.

The job is for:

May work from home