We are looking for an experienced Senior Data Engineer to drive the expansion of a high-availability data hub on Azure Databricks, working in the US time zone. In this role, you will design and implement robust data pipelines using PySpark and Structured Streaming, ensuring seamless integration of new data sources within a Medallion Architecture framework.

You will work closely with the team to manage Unity Catalog objects and optimize performance, building scalable solutions that power critical data ecosystems. This position is ideal for an expert in Azure and Databricks who thrives on handling large-scale data challenges.

This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance.

The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.

Key Responsibilities :

- Developing Databricks pipelines in PySpark / SQL

- Ingesting data from APIs

- Handling Spark Structure Streaming pipelines

- Designing and managing objects in Unity Catalog according to Medallion Architecture

Required Skills :

- Commercial experience of 5-8 years in Data Engineering with focus on large scale data pipelines.

- Proven track record using advanced solutions in Azure cloud and Databricks

- Knowledge of Spark Structured Streaming

- Proficiency in Python and SQL

- Ability to build pipelines using Github or similar tools

Nice to Have :

- Cloud or Data Engineering certifications or specialization certifications (e.g. Google Professional Machine Learning Engineer, Microsoft Certified: Azure AI Engineer Associate Exam AI-102, AWS Certified Machine Learning Specialty (MLS-C01), Databricks Certified Machine Learning).

- Contributions to open-source AI projects.

- Experience with Agile/Scrum methodologies.

What We Offer :

- Work with a senior team of 10+ years experienced ML/AI experts

- Hands-on experience with latest Generative AI and agentic technologies

- Global exposure working on projects for leading brands

- Technology-agnostic environment fostering innovation

- Competitive compensation package

- Opportunity to shape AI strategy in our growing India operations.