Databricks Administrator

Job Title : Databricks Administrator

Experience Required : 7 to 15 Years

Work Location : Bangalore / Chennai / Hyderabad/ Pune/ Gurgaon/ Jaipur

Role Overview :

We are seeking an experienced Databricks Administrator to manage, optimize, and maintain our Databricks platform across multiple environments.

The ideal candidate will have a strong background in data engineering infrastructure, cluster management, security, governance, and automation within Databricks and related Azure/AWS ecosystems.

Key Responsibilities :

Databricks Platform Administration :

- Manage and maintain Databricks workspaces, clusters, jobs, pools, and libraries.

- Monitor performance and optimize resource utilization across environments (dev, test, prod).

- Implement autoscaling, cluster policies, and cost control measures.

Security & Access Management :

- Configure and maintain user/group access via Azure AD or IAM integration.

- Implement fine-grained access control using Unity Catalog and role-based permissions.

- Ensure compliance with enterprise data security standards.

Automation & CI/CD :

- Automate cluster provisioning, job deployment, and workspace management using Terraform, Databricks CLI, or REST APIs.

- Integrate Databricks with CI/CD tools like Azure DevOps, Jenkins, or GitHub Actions.

Monitoring & Troubleshooting :

- Proactively monitor cluster health, job performance, and data pipeline stability.

- Diagnose and resolve system, configuration, and performance issues.

- Implement alerting, logging, and audit trail monitoring.

Integration & Collaboration :

- Work with Data Engineers, DevOps, and Cloud teams to ensure smooth data operations.

- Support integration with Snowflake, Kafka, Azure Data Lake, or AWS S3.

- Collaborate on optimizing Spark configurations and ETL performance.

Governance & Compliance :

- Implement best practices for workspace and notebook governance.

- Manage version control for notebooks and data pipelines.

- Enforce tagging, cost allocation, and audit standards.

Required Skills & Qualifications :

- Bachelors degree in Computer Science, Engineering, or related field.

- 7+ years of experience in Data Engineering or Platform Administration, with at least 3+ years in Databricks Administration.

- Strong understanding of Apache Spark architecture and performance tuning.

- Hands-on experience with Azure Databricks or AWS Databricks environments.

- Proficiency with Terraform, Databricks CLI, REST APIs, and CI/CD pipelines.

- Expertise in cluster configuration, job scheduling, and resource optimization.

- Familiarity with Unity Catalog, Delta Lake, DBFS, and Lakehouse architecture.

- Experience integrating Databricks with ADLS, S3, Snowflake, Synapse, or Redshift.

- Strong problem-solving and scripting skills (Python, Shell, or PowerShell).

- Excellent communication and documentation abilities