HamburgerMenu
hirist

Provana - Technical Lead - Databricks Data Platform

Provana
10 - 12 Years
Noida

Posted on: 17/11/2026

Job Description

Role Overview :

We are seeking a high-caliber Technical Lead to spearhead the design and development on our enterprise-scale Databricks data platform. As a hands-on lead, you will be the primary authority on our Azure-based Lakehouse, ensuring that data pipelines are not just functional, but scalable, secure, and cost-optimized. You must have a proven track record of leading end-to-end implementations - from initial discovery and schema design to production deployment and monitoring.

Core Requirements :

- 10 - 12 + years of experience in Data Engineering

- Azure Mastery: Minimum 3 + years of lead experience specifically within the Azure cloud ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault, Azure functions).

- Databricks Expertise: Deep hands-on experience with Databricks (Spark/PySpark), specifically implementing Medallion Architecture (Bronze, Silver, Gold layers).

- Conversant with streaming and batch data processing.

- Proven Leadership: Must have led at least two full-lifecycle, end-to-end data platform implementations in a Tech Lead capacity.

- Proven Tech Lead experience on both Databricks and Azure

- Excellent understanding of Lakehouse architecture

- Very strong SQL skills and data modeling expertise is a must.

- Strong experience with GenAI-assisted development. Should have used GitHub Copilot, Claude Code, Codex or equivalent; AI-assisted code generation, refactoring, testing, documentation and debugging using GenAI tools.

- Should be conversant with LLM APIs, RAG, embeddings, vector databases, structured outputs, model selection, latency and token-cost optimization.

- Should be familiar with agentic development and loop engineering.

Key Responsibilities :

- Architectural Design: Lead the design and evolution of our Delta Lakehouse architecture, ensuring it supports both high-volume and complex batch processing.

- End-to-End Implementation: Take full ownership of the data lifecycle: ingestion from disparate sources (APIs, RDBMS, NoSQL), complex transformations in PySpark, and final exposure via Gold tables or semantic layers.

- Technical Mentorship: Guide a team of 3 - 5 data engineers through code reviews, design workshops, and the establishment of "best-in-class" Python and SQL coding standards.

- Optimization: Conduct performance tuning of Spark clusters by exploring all available options like optimizing shuffle partitions and managing "Small File" problems to reduce Azure consumption costs.

- Data Governance: Implement and manage Databricks Unity Catalog for centralized fine-grained access control, lineage tracking, and data discovery.

- Handle production issues, root-cause analysis, and long-term fixes.

Technical Skills & Toolset :

- Languages: Expert-level Python (PySpark) and Advanced SQL. Familiarity with .net framework or Java is a plus.

- Orchestration: Advanced use of Azure Data Factory (ADF) and Databricks Workflows/Jobs for complex dependency management.

- DevOps/DataOps: Strong proficiency in Azure DevOps or GitHub Actions, specifically building CI/CD pipelines for automated Databricks notebook deployment and Infrastructure as Code (Terraform/Bicep).

- Observability: Implementation of data quality frameworks (e.g., Great Expectations or Databricks Expectations) and monitoring via Azure Monitor/Log Analytics.

Preferred Qualifications :

- Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.

- AI/ML Integration: Experience supporting ML workloads by building feature stores and integrating with MLflow.

- Agile: Experience working in a high-velocity Scrum and Kanban environment.

Soft Skills :

- Strong ownership mindset and accountability

- Excellent communication with technical and non-technical stakeholders

info-icon

Did you find something suspicious?

Similar jobs that you might be interested in

Loading chat...