Databricks Architect with AI

Role Overview:

We are looking for an experienced Databricks Data Architect to lead the design and implementation of enterprise-scale data platforms using Databricks Lakehouse Architecture. The candidate should have strong expertise in modern data engineering, data architecture, dimensional modeling, and scalable cloud data solutions.

Key Responsibilities:

- Design and implement enterprise-scale data platforms using Databricks Lakehouse Architecture.

- Define end-to-end data architecture covering ingestion, transformation, storage, governance, and consumption layers.

- Establish architecture standards, coding guidelines, reusable design patterns, and best practices.

- Lead technical decisions related to scalability, performance, reliability, and security.

- Design and build scalable ETL/ELT pipelines using PySpark on Databricks.

- Architect Bronze, Silver, and Gold data transformation frameworks.

- Design and manage Delta Lake storage architecture, including optimization and data lifecycle management.

- Implement Databricks Workflows/Lakeflow pipelines for orchestration and automation.

- Define data quality, validation, reconciliation, and monitoring frameworks.

- Oversee CI/CD and DevOps practices for data engineering projects.

- Design enterprise data models using dimensional modeling techniques.

- Develop Star Schemas, Fact Tables, Dimension Tables, and conformed dimensions.

- Define business metrics, KPI layers, semantic models, and curated Gold datasets.

- Provide technical leadership and mentorship to data engineers and developers.

- Conduct architecture and code reviews and ensure adherence to standards.

- Troubleshoot complex technical issues and guide teams in solution design.

- Support solution proposals, estimation, technical governance, and architectural roadmap activities.

- Drive adoption of modern data engineering and cloud-native best practices.

Technical Skills:

- Mandatory: Data Architecture, Databricks Lakehouse Architecture, Databricks Workflows / Lakeflow, Delta Lake, Unity Catalog, Databricks SQL, PySpark and Spark SQL, ETL/ELT design, Dimensional Data Modeling, Star Schema, Fact & Dimension Modeling, Slowly Changing Dimensions (SCD), Metric View, Python, SQL, Git, CI/CD frameworks, Performance tuning, Data quality frameworks, Batch and incremental processing.

- Cloud & Integration: Azure Databricks (preferred), ADLS Gen2, Azure Data Factory (ADF), Event-based and batch ingestion, Data governance, metadata management, and security.

- Good to Have: Data warehouse modernization, legacy EDW migration, Data Mesh, Medallion Architecture, BI platforms (Power BI, Tableau, Looker).

Candidate Profile:

- 15+ years of relevant experience in data architecture and enterprise data platforms.

- Strong hands-on experience with Databricks and modern data engineering practices.

- Proven experience designing scalable and high-performance data platforms.

- Strong expertise in data modeling, Lakehouse architecture, and cloud data solutions.

- Ability to define target-state architecture and technical roadmaps.

- Experience reviewing technical designs and code and guiding development teams.

- Strong communication skills with the ability to drive technical discussions with architects and business stakeholders.

- Ability to provide architectural governance across multiple workstreams.

- Strong leadership and mentoring capabilities.