CirrusLabs - Senior Databricks Platform Engineer - Azure Cloud & Data Operations

Description :

The ideal candidate understands how Databricks workloads, APIs, pipelines, storage layers, and operational processes integrate within broader enterprise cloud ecosystems.

Role Overview :

The Senior Databricks Platform Engineer will support the design, operational enablement, modernization, and ongoing support of enterprise Databricks platforms and associated data processing environments.

This engineer will work closely with :

a. Cloud platform teams

b. DevOps organizations

c. Infrastructure engineering teams

d. Application modernization groups data and analytics stakeholders

e. The role requires strong awareness of how Databricks modernization activities impact :

f. Existing CI/CD pipelines

g. Automation frameworks

h. Operational governance

Enterprise cloud platform standards

Key Responsibilities :

Databricks Platform Engineering Operations

- Support architecture and operational engineering across enterprise Databricks environments

- Design and maintain scalable Lakehouse, Delta Lake, and Unity Catalog solutions

- Support administration of :

a. Databricks Workspaces

b. Jobs

c. Notebooks

d. Clusters

- Optimize Spark workloads, cluster configurations, and platform performance

- Support batch and streaming data processing workloads

- Assist with multi-environment platform management across Dev, QA, and Production

Platform Integration DevOps Alignment :

- Support integration of Databricks platforms with Azure-native ecosystems and enterprise applications

- Work across integrations involving :

a. APIs

b. Azure Storage Accounts

c. Data Lakes reporting and analytics platforms

- Collaborate with platform and DevOps teams to align Databricks activities with existing CI/CD and automation standards

- Support deployment workflows utilizing :

a. GitHub

b. Azure DevOps

c. Infrastructure as Code frameworks

- Help ensure modernization activities do not disrupt existing automated deployment pipelines and operational processes

Governance, Reliability Platform Support:

- Support governance, access management, monitoring, logging, and operational security practices

- Assist with implementation of :

a. RBAC

b. lineage

c. encryption

d. platform governance controls

- Participate in troubleshooting, operational support, and root cause analysis activities

- Contribute to documentation, operational standards, and platform best practices

Required Qualifications :

- 6+ years of experience within cloud, platform engineering, data engineering, or enterprise analytics ecosystems

- 3+ years of hands-on experience supporting enterprise Databricks environments

- Strong experience with :

a. Delta Lake

b. Unity Catalog

c. Spark workloads

d. Cluster operations

e. Databricks Workspaces

- Experience with Azure cloud services and cloud-native platform concepts

- Familiarity with :

a. Azure DevOps

b. GitHub workflows

c. CI/CD pipeline concepts

d. Deployment automation

- Experience supporting APIs, distributed application integrations, and enterprise data platforms

- Strong troubleshooting and operational support capabilities

- Strong communication and stakeholder collaboration skills

Preferred Qualifications :

- PySpark

- Python

- SQL

- Spark SQL

- Scala

- AKS/Kubernetes

- Terraform

- Bicep

- Infrastructure as Code practices

What This Role Is :

- A hands-on enterprise Databricks platform engineering role

- A modernization-focused engineering opportunity within a large Azure ecosystem

- A hybrid role spanning cloud integration, operational engineering, and DevOps-aware platform support

- A collaborative engineering role supporting scalable enterprise data and analytics platforms