Job Role : Data Engineer - Databricks, & MLflow (Remote)

Senior Data Engineer

About the Role :

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, MLflow, and Python. This role focuses on building scalable data pipelines, managing ML workflows, and developing production-grade data solutions. The ideal candidate will have hands-on experience with Databricks platform, MLflow for experiment tracking, and Python for end-to-end data engineering and ML development.

Position Details :

- Job ID : G-24482

- Experience : 5+ years

- Location : Permanent Remote/WFH

- Shift : 2 :00 PM to 11 :00 PM

- Employment Type : Full-time

Mandatory Skills :

- Python - Advanced programming expertise

- Databricks - Data and ML platform

- MLflow - ML lifecycle management

Critical Requirements :

- Shift : 2 :00 PM to 11 :00 PM (fixed)

- Strict Background Verification (BGV) required

Key Responsibilities :

Data Engineering on Databricks :

- Build and maintain scalable data pipelines on Databricks platform

- Develop data processing workflows using PySpark on Databricks

- Implement Delta Lake for data management and optimization

- Manage Databricks clusters and workspaces

- Create and optimize Databricks notebooks

- Automate workflows using Databricks Jobs

- Design data architectures for analytics and ML

MLflow Implementation :

- Manage ML lifecycle using MLflow

- Track experiments, parameters, and metrics

- Implement model registry and versioning

- Develop MLflow pipelines and projects

- Deploy models using MLflow serving

- Monitor model performance in production

- Integrate MLflow with Databricks

Python Development :

- Develop production-grade Python code for data engineering

- Build ETL/ELT pipelines using Python

- Implement data processing scripts with Pandas, NumPy

- Develop APIs and integrations for data systems

- Work with PySpark for distributed data processing

- Automate data workflows and processes

Data Pipeline Development :

- Design end-to-end data pipelines from ingestion to consumption

- Build batch and streaming data pipelines

- Implement data quality checks and validation

- Optimize pipeline performance and efficiency

- Ensure data governance and security

- Monitor and troubleshoot data workflows