Impetus - Lead Software Engineer - Big Data Ecosystem

Description :

Exp: 8 to 12 yrs.

Key Responsibilities :

- Design and develop scalable data pipelines using Databricks (Spark, PySpark, SQL).

- Build and optimize ETL/ELT workflows for large-scale data processing.

- Work with Delta Lake for data management, performance tuning, and reliability.

- Collaborate with data scientists to productionize ML models on Databricks.

- Implement ML pipelines including data preparation, feature engineering, and model deployment.

- Ensure data quality, governance, and security best practices.

- Integrate Databricks solutions with cloud platforms (AWS/Azure/GCP).

- Monitor, troubleshoot, and optimize performance of data workflows.

- Contribute to CI/CD pipelines and automation for data and ML workflows.

Required Skills & Qualifications :

- 8 to 12 years of experience in Data Engineering / Big Data ecosystem.

- Strong hands-on experience with Databricks, Apache Spark, and PySpark.

- Proficiency in SQL and Python.

- Experience with Delta Lake, Unity Catalog (preferred).

- Strong understanding of data modeling and data warehousing concepts.

- Experience working on cloud platforms (AWS / Azure / GCP).

- Exposure to Machine Learning workflows (training, deployment, monitoring).

- Familiarity with ML libraries like scikit-learn, MLflow, TensorFlow, or similar.

- Experience with workflow orchestration tools (Airflow, ADF, etc.

- Strong problem-solving and debugging skills.

Good to Have :

- Experience with MLflow for experiment tracking and model management.

- Knowledge of MLOps practices.

- Experience in real-time/streaming pipelines (Kafka, Structured Streaming).

- Exposure to Data Governance & Security frameworks.

- Certification in Databricks or cloud platforms.

Soft Skills :

- Ability to work in agile, fast-paced environments.

- Collaborative mindset with a focus on continuous learning.