Databricks Architect

Role Overview :

We are seeking an experienced Databricks Architect to lead the design and implementation of enterprise-scale data platforms using the Databricks Lakehouse architecture. The role requires strong expertise in data engineering, Apache Spark, cloud data platforms, data modeling, governance, and modern AI capabilities. The successful candidate will own technical architecture from requirements and solution design through implementation, deployment, and optimization.

Key Responsibilities :

1. Lakehouse Architecture :

- Design and implement scalable Databricks Lakehouse platforms for enterprise data workloads.

- Define architecture patterns for batch processing, streaming, ingestion, storage, transformation, and consumption.

- Build reusable frameworks that support high-volume and evolving data requirements.

- Translate business and technical requirements into practical data architecture solutions.

2. Data Engineering & Pipelines :

- Develop robust ETL/ELT pipelines using Apache Spark, PySpark, and SQL.

- Design incremental ingestion, metadata-driven processing, and reusable data ingestion frameworks.

- Implement and manage Delta Lake, Delta Live Tables, Auto Loader, Structured Streaming, and Databricks Workflows.

- Troubleshoot and optimize data pipelines for reliability, throughput, and scalability.

- Support both batch and near-real-time data processing requirements.

3. Data Warehousing & Modeling :

- Design enterprise data warehouse structures using 3NF and dimensional modeling techniques.

- Define appropriate data structures for analytical and operational reporting requirements.

- Develop data models that support large-scale workloads while maintaining performance and usability.

4. Governance & Data Quality :

- Establish frameworks for data governance, quality, security, and access management.

- Implement Unity Catalog for centralized data governance and controlled access.

- Define standards for data lineage, discoverability, quality checks, and secure data consumption.

5. AI & Machine Learning :

- Support data platforms that enable modern AI and machine learning use cases.

- Work with MLflow and contribute to model lifecycle and MLOps practices.

- Understand and support AI applications such as RAG and AI/BI from a data platform perspective.

- Collaborate with AI/ML teams to make enterprise data accessible and production-ready for intelligent applications.

6. Azure & Enterprise Integration :

- Design cloud-native data solutions on Microsoft Azure.

- Work with Azure data services and Azure DevOps-based engineering environments.

- Integrate the data platform with enterprise applications and systems such as ServiceNow.

- Ensure architecture aligns with enterprise security, scalability, and operational standards.

7. DevOps & Optimization :

- Establish CI/CD practices for data engineering and Databricks workloads.

- Use Databricks Asset Bundles, Terraform, or comparable IaC and deployment technologies where applicable.

- Build reusable deployment templates and engineering standards.

- Analyze Spark workloads and optimize jobs, queries, clusters, and data processing strategies.

8. Technical Leadership :

- Own architecture decisions and provide technical direction throughout project delivery.

- Review designs and implementations to ensure adherence to engineering standards.

- Mentor data engineering teams and promote best practices.

- Work directly with stakeholders to gather requirements, clarify technical needs, and communicate solution approaches.

- Support successful delivery of complex data initiatives for international clients.

Required Experience :

- 10+ years of overall software engineering experience.

- At least 5+ years of professional Data Engineering experience with hands-on Databricks exposure.

- 5+ years of strong experience across the Databricks ecosystem, including Delta Lake, Delta Live Tables, Auto Loader, Structured Streaming, Workflows, and Unity Catalog.

- Proven experience working at an architecture level, from solution design through implementation and production deployment.

- Strong programming and data-processing skills in Python and SQL.

- Deep knowledge of Apache Spark, including performance tuning and scalability techniques.

- Strong experience with enterprise data warehousing and advanced 3NF and dimensional data modeling.

- Experience designing solutions for both batch and real-time data environments.

- Working knowledge of AI services or modern AI tools.

- Strong stakeholder management and requirements-gathering skills with US or UK clients.

- Experience working in a B2B IT services or IT consulting organization.

Preferred Skills :

- Hands-on experience with Azure Databricks and Azure data services.

- Experience with MLflow, MLOps, RAG, or AI/BI use cases.

- Knowledge of CI/CD, Databricks Asset Bundles (DABs), Terraform, Infrastructure as Code, or similar deployment approaches.

- Experience integrating Databricks with ServiceNow or other enterprise platforms.

- Databricks certifications such as Data Engineer Associate/Professional, ML, or GenAI tracks.

- Azure or AWS cloud certifications.

Ideal Candidate :

- The ideal candidate is a hands-on Data Architect / Databricks Architect who can combine architecture thinking with strong engineering depth. You should be capable of making technical decisions, building scalable Lakehouse solutions, guiding engineering teams, and communicating effectively with business and international stakeholders.

- Note : The compensation for this position includes a 5% variable component.