Happiest Minds Technologies - Databricks Architect

Job Description :

We are looking for an experienced Databricks Architect to design and lead enterprise-scale Lakehouse platforms on Databricks. The ideal candidate will have strong expertise in Databricks architecture, Apache Spark, Delta Lake, Unity Catalog, data governance, and cloud-native data platforms. This role involves defining platform standards, driving modernization initiatives, optimizing performance and cost, and enabling secure, scalable, and governed data solutions.

Key Responsibilities :

- Design and own the target-state Lakehouse architecture, including Unity Catalog, metastores, workspace strategy, data sharing, and lineage.

- Define enterprise standards for Delta Live Tables (DLT), Databricks Workflows, streaming pipelines, CDC/SCD implementations, data contracts, and schema design.

- Establish platform governance, security, and compliance standards, including RBAC/ABAC, row/column-level security, auditing, and data privacy controls.

- Drive migration of legacy data warehouses and ETL platforms to Databricks using scalable migration frameworks and best practices.

- Optimize platform performance through Spark tuning, Delta Lake optimization, Photon adoption, cluster policies, autoscaling, and workload management.

- Implement cost governance and FinOps practices, including cluster optimization, resource utilization monitoring, and cost dashboards.

- Define DataOps practices, CI/CD pipelines, automated testing, observability, SLIs/SLOs, and platform reliability standards.

- Collaborate with data engineering, analytics, cloud, and business teams to deliver scalable and secure data solutions.

- Mentor engineering teams and establish architectural guidelines, reusable frameworks, and best practices.

Required Skills :

- 10-16 years of experience in Data Engineering, Big Data, or Cloud Data Platforms, with significant experience in Databricks architecture.

- Deep expertise in Apache Spark, Delta Lake, Databricks Lakehouse Platform, and performance optimization.

- Strong experience with Unity Catalog, data governance, lineage, and enterprise security models.

- Hands-on experience with Delta Live Tables (DLT), Databricks Workflows, streaming, CDC, and large-scale ETL pipelines.

- Experience designing enterprise data architectures on AWS, Azure, or GCP.

- Strong understanding of CI/CD, DataOps, monitoring, observability, and platform automation.

- Experience with migration from traditional data warehouses or legacy ETL platforms to Databricks.

- Excellent problem-solving, stakeholder management, and architecture documentation skills.

Good to Have :

- Experience with Delta Sharing, Lakehouse Federation, Databricks Marketplace, Model Serving, or Vector Search.

- Knowledge of AI/ML workloads on Databricks.

- Databricks or cloud platform certifications (AWS, Azure, or GCP).