ABOUT THE ROLE :

We are looking for experienced Data Engineers to join a high-impact engagement building an AI-driven data platform from scratch. You will design accelerators, define self-service frameworks for a growing user base, and own end-to-end pipeline delivery from enterprise source systems. Requirements will evolve and you are expected to drive clarity, not wait for it.

WHAT YOU WILL DO :

- Design and build reusable data engineering accelerators and self-service frameworks on Databricks

- Build end-to-end pipelines ingesting from enterprise systems including Bullhorn, Beeline, and SAP Fieldglass

- Lead Unity Catalog adoption data discovery, lineage, access control, and sharing standards

- Build API/MCP integrations enabling programmatic access to the data platform

- Develop dbt models and semantic layer components for commercial and operational reporting

- Embed AI tooling into development workflows and data pipeline design

TECHNICAL REQUIREMENTS :

1. Databricks :

- Production experience with Spark, Delta Lake, Workflows, DLT, and Repos

- Unity Catalog implementation and governance

- Performance tuning : partitioning, Z-ordering, caching, cluster configuration, AQE

2. Python & SQL :

- Strong Python : OOP, testing (pytest), modular design, packaging

- Advanced SQL : window functions, CTEs, query optimisation at scale

- PySpark : DataFrame API, UDFs, Structured Streaming

3. Data Modelling & dbt :

- dbt Core or Cloud : models, macros, testing, documentation, semantic layer

- Medallion architecture (bronze/silver/gold) and lake house design principles

4. Cloud & Infrastructure :

- Hands-on with AWS, Azure, or GCP - IAM, storage, secrets management, networking

- CI/CD pipelines with Git-based workflows (GitHub Actions, Azure DevOps, or equivalent)

5. Data Security :

- Row/column-level security, dynamic data masking, audit logging

- Secrets and credential management in cloud-native environments

6. AI & Emerging Tooling :

- Practical experience with LLM APIs, prompt engineering, or AI-assisted development