Experience:

- 5 to 10 years of experience in Data Engineering.

Key Skills Required:

- Strong hands-on experience with Databricks + Streaming.

- PySpark.

- SQL.

- Delta Lake.

- Unity Catalog.

- Databricks Workflows / Jobs orchestration.

- Experience with Azure Data Factory (ADF).

- Experience building Lakehouse architectures using Bronze, Silver & Gold layers.

- Good knowledge of Databricks Lakeflow:

1. Declarative Pipelines.

2. Lakeflow Connect.

3. CDC (Change Data Capture).

4. Data quality validations.

- Experience with AI/BI and Genie:

1. Semantic layers.

2. Certified datasets.

3. Self-service analytics solutions.

- Experience building AI Agents/Copilots using:

1. LLMs.

2. RAG (Retrieval-Augmented Generation).

3. Vector Search.

4. Prompt Engineering.

- Good understanding of Unity Catalog governance:

1. Access controls.

2. Data masking.

3. Audit logging.

4. Lineage.

- Experience with CI/CD, Git, Databricks Repos & Asset Bundles.

Expected Responsibilities:

- Design, develop, and support end-to-end data pipelines in Databricks.

- Build and maintain Delta tables under Unity Catalog.

- Develop Lakeflow pipelines with incremental processing and CDC.

- Enable AI/BI use cases through semantic models and Genie spaces.

- Integrate SharePoint and external data sources into the Lakehouse.

- Ensure data quality, governance, and operational reliability.

- Drive delivery independently and proactively resolve issues.