We are looking for an experienced Databricks Administrator / Architect with strong expertise in managing enterprise-scale Databricks environments on AWS. The ideal candidate will have hands-on experience designing secure, multi-region Databricks infrastructure and implementing cloud networking, IAM and security best practices.

The role also involves building and managing the underlying infrastructure required for Generative AI and RAG workloads, including MLflow, Databricks Vector Search and GPU-optimised compute environments.

Key Responsibilities :

- Architect, deploy and maintain multi-region Databricks environments on AWS.

- Design and implement secure-by-default networking using :

1. Customer-managed VPCs

2. AWS PrivateLink

3. Transit Gateway

4. VPC endpoints

5. Secure network segmentation

- Design and manage IAM cross-account roles, permissions and access controls.

- Implement enterprise security, governance and compliance controls across Databricks environments.

- Manage Databricks workspaces, clusters, policies, users, groups and access permissions.

- Configure and optimise Databricks compute infrastructure based on workload requirements.

- Design infrastructure supporting Generative AI and Machine Learning workloads.

- Configure and manage MLflow for experiment tracking, model lifecycle management and model governance.

- Implement and manage Databricks Vector Search for RAG and enterprise AI applications.

- Design and configure GPU-optimised compute clusters for AI/ML model training and inference.

- Monitor infrastructure performance, availability, security and cost.

- Troubleshoot Databricks, AWS networking, connectivity and infrastructure-related issues.

- Work closely with Data Engineering, ML Engineering, Security and Cloud teams to ensure reliable and scalable platform architecture.

- Establish automation and infrastructure-as-code practices for consistent Databricks deployments.

Required Skills :

- 10 - 12 years of experience in Databricks administration, cloud infrastructure or platform engineering.

- Strong hands-on experience with Databricks on AWS.

- Experience designing and managing multi-region Databricks environments.

- Strong knowledge of AWS networking, including :

1. VPC

2. PrivateLink

3. Transit Gateway

4. VPC Endpoints

5. Network security and segmentation

- Strong understanding of AWS IAM and cross-account roles.

- Experience implementing secure-by-default cloud architectures.

- Hands-on experience with Databricks workspace and cluster administration.

- Experience with MLflow and model governance.

- Experience with Databricks Vector Search and RAG architectures.

- Experience configuring GPU-enabled compute clusters for AI/ML workloads.

- Strong understanding of cloud security, access management and governance.

Good to Have :

- Experience with Terraform or other Infrastructure-as-Code tools.

- Knowledge of Databricks Unity Catalog and data governance.

- Experience with CI/CD for Databricks deployments.

- Knowledge of AWS security services and monitoring tools.

- Exposure to Generative AI, LLM and RAG workloads.

- Experience optimising cloud infrastructure for performance and cost.