Description :

Key Responsibilities :

- Design and develop scalable data pipelines using Databricks (Spark, PySpark, SQL)

- Build and optimize ETL/ELT workflows for large-scale data processing

- Work with Delta Lake for data management, performance tuning, and reliability

- Collaborate with data scientists to productionize ML models on Databricks

- Implement ML pipelines including data preparation, feature engineering, and model deployment

- Ensure data quality, governance, and security best practices

- Integrate Databricks solutions with cloud platforms (AWS/Azure/GCP)

- Monitor, troubleshoot, and optimize performance of data workflows

- Contribute to CI/CD pipelines and automation for data and ML workflows

Required Skills & Qualifications :

- 6 to 10 years of experience in Data Engineering / Big Data ecosystem

- Strong hands-on experience with Databricks, Apache Spark, and PySpark

- Proficiency in SQL and Python

- Experience with Delta Lake, Unity Catalog (preferred)

- Strong understanding of data modeling and data warehousing concepts

- Experience working on cloud platforms (AWS / Azure / GCP)

- Exposure to Machine Learning workflows (training, deployment, monitoring)

- Familiarity with ML libraries like scikit-learn, MLflow, TensorFlow, or similar

- Experience with workflow orchestration tools (Airflow, ADF, etc.)

- Strong problem-solving and debugging skills

Good to Have :

- Experience with MLflow for experiment tracking and model management

- Knowledge of MLOps practices

- Experience in real-time/streaming pipelines (Kafka, Structured Streaming)

- Exposure to Data Governance & Security frameworks

- Certification in Databricks or cloud platforms

Soft Skills :

- Strong communication and stakeholder management skills

- Ability to work in agile, fast-paced environments

- Collaborative mindset with a focus on continuous learning