Job Overview:

We are looking for an accomplished Databricks Solution Architect / Data Architect to lead the design, modernization, and implementation of enterprise-scale data platforms on Azure. In this senior role, you will define target-state data architecture, establish scalable Lakehouse design patterns, and guide the development of high-performance data pipelines using Databricks, Delta Lake, and related cloud-native services.

Key Responsibilities:

- Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, with strong focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems.

- Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers to support reporting, advanced analytics, ML, and business-critical decisioning use cases.

- Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables for batch and near real-time processing.

- Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management across Databricks solutions.

- Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and integration with enterprise governance services such as Purview.

- Optimize solution performance by tuning Spark workloads, cluster policies, partitioning strategy, file sizing, caching, and compute cost management for large-scale data processing.

- Collaborate with business stakeholders, product owners, analysts, architects, and downstream consumers to translate functional and non-functional requirements into scalable technical designs.

- Provide technical leadership to engineering teams by reviewing designs, guiding implementation, resolving architectural bottlenecks, and establishing best practices for Databricks-based delivery.

- Evaluate and recommend Databricks capabilities such as Photon, serverless compute, Lakehouse Federation, and streaming patterns to improve scalability, maintainability, and time to value.

- Ensure strong delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution including sprint planning, backlog refinement, and design reviews.

Candidate Profile:

- 10 - 15 years of experience in data engineering, cloud data platform design, or enterprise data architecture, with at least 5+ years of strong hands-on experience on Databricks and Azure.

- Bachelors degree in computer science, Information Technology, Engineering, or a related discipline; masters degree is preferred.

- Strong expertise in designing modern data architectures, including Lakehouse, medallion architecture, data modeling, data warehousing, and scalable ingestion and transformation frameworks.

- Deep technical proficiency in Databricks, PySpark, Python, SQL, Delta Lake, Databricks Workflows, Auto Loader, and Delta Live Tables.

- Strong experience with Azure cloud services such as Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure DevOps, and integration of Databricks with broader enterprise cloud ecosystems.

- Hands-on experience in defining architecture standards, reusable design patterns, CI/CD strategy, environment management, and delivery best practices for enterprise-scale data platforms.

- Experience with data governance, lineage, and security frameworks including Unity Catalog, role-based access controls, and integration with enterprise governance tools such as Azure Purview.

- Proven ability to optimize large-scale Spark and Databricks workloads, including performance tuning, cluster sizing, workload management, and cost optimization.

- Experience engaging with business stakeholders, product owners, architects, and engineering teams to convert business requirements into scalable solution designs and implementation roadmaps.

- Strong communication, leadership, and problem-solving skills with the ability to mentor teams, review technical designs, and drive architecture decisions across cross-functional programs.

- Preferred: Knowledge of insurance domain data models, regulatory considerations, and analytics use cases relevant to underwriting, claims, pricing, or risk functions.