Description :

Responsibilities :

Data Engineering & Architecture :

- Design and implement scalable and optimized data pipelines on Databricks using Delta Lake, PySpark, and SQL.

- Develop ETL/ELT frameworks for batch and streaming data processing.

- Ensure data quality, governance, and observability using Unity Catalog, Great Expectations, or custom validations.

- Optimize Spark jobs for performance, cost, and scalability.

Cloud & Infrastructure (Azure/AWS/GCP) :

- Deploy and manage Databricks clusters, workspaces, and Jobs.

- Work with Terraform or ARM templates for infrastructure automation.

- Integrate cloud-native services like Azure Data Factory, AWS Glue, or GCP Cloud Composer.

MLOps & CI/CD Automation :

- Implement CI/CD pipelines for Databricks notebooks, workflows, and ML models.

- Work with MLflow for model tracking and lifecycle management.

- Automate data pipelines using Azure DevOps, GitHub Actions, or Jenkins.

Leadership & Collaboration :

- Lead a team of data engineers, ensuring best practices and code quality.

- Collaborate with data scientists, analysts, and business teams to understand requirements.

- Conduct performance reviews, technical mentoring, and upskilling sessions.

Skills :

- Strong hands-on experience in Databricks, Apache Spark (PySpark/Scala), and Delta Lake.

- Expertise in SQL, ETL/ELT pipelines, and data modeling.

- Experience with Azure, AWS, or GCP cloud platforms.

- Knowledge of MLOps, MLflow, and CI/CD best practices.

- Experience in workflow orchestration using Databricks Workflows, Airflow, or Prefect.

- Understanding of cost optimization, cluster tuning, and performance monitoring in Databricks.

- Communication and leadership experience, with experience initiating and driving projects.

- Strong stakeholder management and mentoring skills.

- Experience with data sets, Hadoop, and data modernisation tools.

- Development experience in at least one object-oriented language (Python, Java, etc.).