Numerator - Senior Data Engineer - Python/PySpark

About the Role :

Key Responsibilities :

- Design, build, and maintain scalable data ingestion and transformation pipelines using PySpark and Python.

- Implement and manage data processing workloads on Databricks, ensuring optimal performance and cost efficiency.

- Monitor data pipeline health, troubleshoot issues, and implement proactive reliability enhancements.

- Document data models, pipeline designs, and operational procedures for knowledge sharing and compliance.

Requirements :

- 7 - 10 years of experience in data engineering roles

- Experience in Pyspark and Python

- Experience working with Microsoft Azure Cloud

- Experience in Agile methodologies (Scrum/Kanban)

- Experience with PySpark, Databricks

- Experience working with Devops pipeline, preferably Azure DevOps

Preferred Qualifications :

- Bachelor's or master's degree in computer science, Information Technology, Data Science, or a related field.

- Experience working in a support focused role

- Knowledge/experience in AI ML techniques

- Certification in relevant Data Engineering discipline or related fields.