HamburgerMenu
hirist

Senior Databricks Architect

Hunterz Research
10 - 15 Years
Multiple Locations

Posted on: 20/11/2026

Job Description

Senior Databricks Architect

Work Location: Bangalore - Pune - Noida - Hyderabad - Chennai

Notice period: Immediate and Max 15 to 20 Days

Experience: 10 - 15 years

Role Overview:

Senior platform and solution architect responsible for enterprise-scale Databricks lakehouse architecture, modernization strategies, governance, migration roadmaps, engineering standards, performance and cost guardrails, and AI-ready data foundations. The role converts business and non-functional requirements into implementable platform patterns and guides engineering teams through delivery.

Core Responsibilities:

Lakehouse & Platform Architecture:

- Define target-state lakehouse architecture, workspace and environment strategy, Unity Catalog topology, storage and compute patterns, governed sharing, lineage and domain boundaries.

- Establish implementation standards for Medallion architecture, Lakeflow Declarative Pipelines / DLT, Workflows, Auto Loader, Delta Lake, system tables and Databricks Asset Bundles.

- Create reference architectures, architecture decision records, reusable blueprints and non-functional requirement guardrails.

Spark, Data Engineering & Streaming:

- Set engineering standards for PySpark, Spark SQL, Delta Lake, Structured Streaming, CDC, SCD, incremental processing, data contracts and schema evolution.

- Guide execution-plan analysis, AQE, join strategy, shuffle and skew management, partitioning, file sizing, caching and Photon adoption.

- Drive idempotent, auditable and recoverable pipelines with clear batch-versus-streaming decision rationale.

Migration & Modernization:

- Lead legacy warehouse, Hadoop and ETL modernization assessments and define migration waves, conversion patterns and dependency plans.

- Own validation, reconciliation, parallel-run, cutover, rollback and decommissioning approaches.

- Create reusable migration playbooks and accelerators while controlling delivery risk and business continuity.

Governance, Security & Data Sharing:

- Design Unity Catalog operating patterns covering catalogs, schemas, external locations, service principals, RBAC/ABAC, masking, row/column controls and auditability.

- Define lineage, data quality, privacy, retention and compliance-ready implementation standards.

- Guide Delta Sharing, Lakehouse Federation and governed producer-consumer patterns where appropriate.

Platform Engineering, DataOps & Reliability:

- Define Git-based CI/CD, testing, environment promotion, infrastructure-as-code and release controls for Databricks assets.

- Establish SLIs/SLOs, observability, alerting, runbooks, reliability scorecards and production-readiness reviews using platform telemetry and system tables.

- Improve developer experience through templates, reusable frameworks, guardrails and self-service patterns.

Cost Optimization & FinOps:

- Establish cluster and compute policies, autoscaling, workload sizing, serverless decision guidelines, tagging and chargeback/showback controls.

- Create cost and performance baselines and drive workload-level optimization with measurable outcomes.

- Balance performance, reliability and cost rather than optimizing one dimension in isolation.

AI-ready Data Foundations:

- Guide governed data foundations for RAG and AI use cases, including data quality, lineage, access controls and retrieval readiness.

- Demonstrate working awareness of Vector Search, model serving/lifecycle, MLflow, context engineering and AI governance.

- Assess emerging capabilities without weakening core data engineering and platform-accountability expectations.

Consulting & Stakeholder Leadership:

- Lead discovery and architecture workshops; articulate options, trade-offs, risks, roadmap and estimates to technical and business stakeholders.

- Explain the latest project with clear business problem, architecture choice, Databricks fit, alternatives considered, challenges, outcomes and lessons learned.

- Mentor engineers and leads, review designs, and support proposals, solution narratives and customer-facing architecture discussions.

info-icon

Did you find something suspicious?

Similar jobs that you might be interested in

Loading chat...