About the Role:

We are seeking a highly capable, hands-on Lead Data Engineer (Manager) with deep expertise in Databricks to lead the design and delivery of scalable data solutions.

In this role, you will:

- Lead engineering teams across distributed environments

- Own end-to-end delivery of modern data solutions

- Work closely with business and technical stakeholders

- Drive innovation across data platforms

You will play a key role in enabling clients to unlock value from their data while ensuring scalability, reliability, and performance.

Key Responsibilities:

- Lead, manage, and grow a distributed Databricks engineering team

- Own end-to-end delivery of data engineering solutions (design - deployment - optimisation)

- Provide hands-on technical leadership in building scalable data pipelines and data products

- Design and implement Lakehouse architecture aligned with enterprise standards

- Translate business requirements into robust, scalable data solutions

- Establish and enforce best practices (code quality, testing, documentation, CI/CD)

- Ensure governance, security, and compliance across data platforms

- Act as escalation point for complex technical challenges

- Collaborate with architecture, DevOps, and platform teams

- Drive adoption of new Databricks features, tools, and AI capabilities

Skills & Experience:

Essential:

- Proven experience leading and managing distributed data engineering teams

- Strong hands-on expertise with Databricks (Spark PySpark/SQL)

- Experience building data platforms on Azure (ADLS, ADF, Databricks) or similar cloud ecosystems

- End-to-end data engineering expertise (ingestion, transformation, modelling, serving)

- Deep understanding of Lakehouse architecture (Delta Lake, Medallion - Bronze/Silver/Gold layers)

- Strong stakeholder management and business translation skills

- Experience establishing engineering standards and code review practices

- Expertise in CI/CD & DevOps (Azure DevOps, Git, Terraform)

- Excellent leadership, mentoring, and communication skills

- Relevant Databricks certifications

Desirable:

- Experience in onshore/offshore delivery models and vendor management

- Exposure to both Databricks and Snowflake

- Knowledge of Unity Catalog and data governance frameworks

- Experience with real-time/streaming data (Kafka, Structured Streaming)

- Exposure to AI/ML or GenAI workloads (feature engineering, RAG, model pipelines)

- Experience working in large, complex enterprise environments

- Experience in cost optimisation and performance tuning

- Exposure to Azure AI (OpenAI, Cognitive Services, AI Foundry)

- Experience contributing to a Data/AI Centre of Excellence

Behavioural Competencies:

- Collaborative: Builds strong cross-functional relationships

- Influential: Guides decisions and drives alignment

- Delivery-focused: Owns outcomes and drives execution

- Structured thinker: Creates clarity in complex environments

- Adaptable: Thrives in dynamic and evolving contexts

- Detail-oriented: Ensures accuracy and quality

- Proactive: Anticipates risks and mitigates early

- Continuous improvement mindset: Drives innovation and optimization

- Resilient: Performs effectively under pressure

Education & Certifications:

- Microsoft Azure Certifications

- Databricks Certified (Associate / Professional)

- SnowPro Data Engineer (Advanced)

- Microsoft Fabric Certifications (DP-600 / DP-700)