Celebal Technologies - DevOps Architect - Azure Platform

Job Summary :

The DevOps Architect is a key member of our data and analytics team, responsible for the overall administration, configuration, and optimization of the Databricks Lakehouse Platform.

This role ensures the platform's stability, security, and performance, enabling data engineering, data science, and machine learning initiatives.

The administrator will work closely with various cross-functional teams to understand requirements, provide technical solutions, and maintain best practices for the Databricks environment.

Key Responsibilities :

- Create catalogs, schemas, access controls, and lineage configurations.

- Implement identity and access management using account groups, workspace-level permissions, and data-level governance.

- Monitor platform health, cluster utilization, job performance, and cost using Databricks admin tools and observability dashboards.

- Automate workspace onboarding, schema creation, user/group assignments, and external location setup using Terraform, APIs, or CLI.

- Integrate with Azure services like ADLS Gen2, Azure Key Vault, Azure Data Factory, and Azure Synapse.

- Support model serving, feature store, and MLflow lifecycle management for Data Science/ML teams.

- Manage secrets, tokens, and credentials securely using Databricks Secrets and integration with Azure Key Vault.

- Define and enforce tagging policies, data masking, and row-level access control using Unity Catalog and Attribute-Based Access Control (ABAC).

- Ensure compliance with enterprise policies, security standards, and audit requirements.

- Coordinate with Ops Architect, Cloud DevOps teams for network, authentication (e.

- Troubleshoot workspace, job, cluster, or permission issues for end users and data teams.

Preferred Qualifications :

- Experience with Apache Spark and data engineering concepts.

- Knowledge of monitoring tools (e., Splunk, Grafana, Cloud-native monitoring).

- Familiarity with data warehousing and data lake concepts.

- Experience with other big data technologies (eg, Hadoop, Kafka).

- Previous experience leading or mentoring junior administrators.