Epergne Solutions - Platform Engineer - Databricks

Platform Engineer (Databricks - Multi-Cloud Focus)

Location : Noida, Gurgaon, Hyderabad, Bangalore, Pune, India

We're looking for a talented Platform Engineer to join our team and play a key role in managing and supporting our Databricks environments across both AWS and Azure.

If you're passionate about platform configuration, security, automation, and building robust CI/CD pipelines in a multi-cloud setting, this role is for you!

What You'll Do :

- Manage Databricks Platform : Configure and manage Databricks workspaces, ensuring optimal performance and resource utilization.

- Data Governance with Unity Catalog : Implement and oversee Databricks Unity Catalog to effectively organize, govern, and manage data assets, ensuring data quality and discoverability.

- Access Control & Security : Set up and manage Role-Based Access Control (RBAC) to define granular user and group permissions within Databricks.

- Integrate with Azure Active Directory (AD) groups for streamlined identity management.

- Automation & API Integration : Leverage Databricks APIs and Service Principals to automate routine tasks, build custom integrations, and ensure secure programmatic access to the platform.

- Networking & Cloud Connectivity : Configure and troubleshoot network connectivity (NCC) for Databricks on both AWS and Azure, ensuring seamless data flow and secure communication.

- System Monitoring : Monitor Databricks systems and integrated cloud services for performance, availability, and potential issues, proactively addressing any anomalies.

- Scripting & Automation : Write robust automation scripts using Python and PowerShell to streamline operational tasks and enhance platform efficiency.

- Infrastructure as Code (IaC) : Utilize Terraform to define, provision, and manage infrastructure components for Databricks environments, ensuring consistency and repeatability.

- DevOps & CI/CD : Build, maintain, and optimize deployment pipelines using Azure DevOps to facilitate continuous integration and continuous deployment of platform configurations and applications.

- Testing : Develop and implement unit tests and integration tests using Pytest to ensure the quality, reliability, and stability of platform code and configurations.

Core Skills & Qualifications :

- Proven experience as a Platform Engineer or similar role, with a strong focus on Databricks environments.

- Hands-on expertise with Databricks Unity Catalog.

- Solid understanding and practical experience with access controls (RBAC), Databricks APIs, and Service Principals.

- Experience integrating with Azure Active Directory (AD) groups.

- Proficiency in configuring and troubleshooting network connectivity (NCC) on both AWS and Azure.

- Strong scripting skills in Python and PowerShell.

- In-depth experience with Terraform for Infrastructure as Code.

- Demonstrated ability to build and manage deployment pipelines using Azure DevOps.

- Experience in writing unit tests and integration tests using Pytest.

- Familiarity with monitoring tools and best practices for cloud platforms.

- Excellent problem-solving skills and the ability to diagnose complex technical issues.

- Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams