Azure Databricks Platform Engineer - Insurance Data Infrastructure

Location : Pune / Bangalore/ Mumbai/ Chennai/ Hyderabad/ Gurugram

Band : M3/M4 (7 to 14 years)

Key Responsibilities :

- Azure Databricks Platform Engineering - Design, deploy, and manage secure, scalable, and high-performance Databricks environments for insurance data processing and analytics.

- Cloud Integration - Collaborate with architects, engineers, and security teams to establish infrastructure standards and ensure connectivity with legacy systems, cloud-native data sources, and third-party platforms.

- Infrastructure as Code (IaC) - Implement automated provisioning and configuration of Azure and Databricks resources using Terraform, supporting enterprise DevOps practices.

- Automation & CI/CD - Build and maintain GitHub Actions-based automation pipelines for environment deployment, monitoring, and incident response to ensure consistency, traceability, and efficiency.

- Performance & Scalability - Monitor platform health, optimize cluster utilization, and conduct tuning to ensure reliability, scalability, and cost efficiency for mission-critical workloads.

- Security & Compliance - Enforce enterprise and regulatory standards (RBAC, Key Vault, Managed Identities, Encryption, PDPO, GDPR, insurance compliance).

- Data Governance - Lead integration of Informatica tools for metadata management, cataloguing, lineage tracking, and compliance across the platform.

- Documentation & Governance - Prepare infrastructure architecture diagrams, network topology, security configurations, and runbooks for governance, audits, and operational readiness.

- Incident Management - Troubleshoot platform and infrastructure issues, perform RCA, and drive continuous improvements to enhance reliability.

- Innovation & Best Practices - Stay up to date with Azure, Databricks, and DevOps advancements, proactively recommending enhancements for platform efficiency and business value.

Requirements :

- Bachelor's degree in Computer Science, IT, or related field (Master's preferred).

- 7-14 years of experience in Azure cloud engineering, with at least 5+ years specialized in Databricks platform management.

- Proven expertise in provisioning and managing Databricks clusters & workspaces for processing structured, semi-structured, and unstructured data in insurance/banking domains.

- Hands-on experience integrating Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 with Databricks for automated ETL/ELT pipelines.

- Strong proficiency in Infrastructure as Code (Terraform) for automated deployments and resource management.

- Experience deploying and integrating Informatica solutions for metadata, lineage, and governance in large-scale data ecosystems.

- Deep knowledge of cloud security, identity & access management, encryption, and compliance frameworks in regulated industries (insurance/financial services preferred).

- Practical expertise in GitHub Actions or similar CI/CD tools for platform automation and DevOps workflows.

- Proven ability in troubleshooting, performance optimization, and incident management for high-availability data platforms.

- Excellent documentation, collaboration, and stakeholder communication skills, with experience supporting both technical and business users.