Experience : 6- 8 YRS

Shift Timing : 8:00 AM - 5:00 PM UK

Key Responsibilities :

1. Model Migration & Optimization :

- Support migration of existing ML models from GCP to Azure Databricks. Understand existing model architecture and replicate/optimize it in Azure Databricks.

- Work closely with the Data Science team to operationalize migrated models and further optimize to reduce compute cost and increase test coverage.

- Knowledge of Databricks infrastructure and Terraform.

2. MLOps & Workflow Orchestration :

- Set up robust CI/CD pipelines using GitHub Actions for ML model deployments in Databricks.

- Implement and manage MLflow for tracking, versioning, and managing model lifecycle.

- Build efficient and scalable Data & ML pipelines using Databricks and PySpark.

3. Cloud & Data Movement Support :

- Collaborate with the Data Engineering team regarding data movement between GCP and Azure Databricks.

- In the future, take over parts of cross-cloud data movement from the DE team and build self-service automation for data flows.

- Build pipelines where outputs from Azure Databricks must be transferred back to GCP.

4. Architecture & Best Practices :

- Provide architectural inputs and workflow optimization guidance during and after migration.

- Ensure scalable, cost-efficient, and reliable model execution in Databricks.

- Improve testing, monitoring, and performance tuning for migrated and future ML models.

Must Have :

- 6- 8 years of experience in Data Engineering, ML Engineering, or MLOps roles.

- Strong hands-on expertise in Databricks and deep understanding of how it works under the hood.

- Proficiency in PySpark: writing scalable jobs, understanding execution plans, and optimization techniques.

- Experience building CI/CD pipelines using GitHub Actions.

- Experience with MLflow for tracking and operationalizing ML models.

- Knowledge of integrating workflows between GCP and Azure ecosystems.

- Strong debugging, optimization, and cost-efficiency mindset.

Good to Have :

- Experience with cross-cloud data movement patterns.

- Familiarity with Data Science model structures and ability to collaborate closely with Data Science teams.

- Exposure to model monitoring and alerts in a distributed/cloud environment.