Data Engineer - Python/ETL

Job Description :

We are seeking a highly skilled and motivated Data Engineer to join our dynamic technology team. The ideal candidate will have deep expertise in data engineering tools and platforms, particularly Apache Airflow, PySpark, and Python, with hands-on experience in Cloudera Data Platform (CDP). A strong understanding of DevOps practices and exposure to AI/ML and Generative AI use cases is highly desirable.

Key Responsibilities :

- Develop and optimize ETL workflows on Cloudera Data Platform (CDP).

- Implement data quality checks, monitoring, and alerting mechanisms.

- Ensure data security, governance, and compliance across all data pipelines.

- Work closely with cross-functional teams to understand data requirements and deliver solutions.

- Troubleshoot and resolve issues in production data pipelines.

- Contribute to the architecture and design of the data platform.

- Collaborate with engineering teams and analysts to work on AI/ML and Gen AI use cases.

- Automate deployment and monitoring of data workflows using DevOps tools and practices.

- Stay updated with the latest trends in data engineering, AI/ML, and Gen AI technologies.

Skills & Qualifications :

- Expertise in Cloudera Data Platform (CDP).

- In-depth knowledge of ETL workflows, data pipelines, and automation tools.

- Solid understanding of DevOps practices for automation and deployment.

- Experience with AI/ML and Generative AI use cases is a plus.

- Strong problem-solving skills and the ability to troubleshoot data pipeline issues.

- Excellent communication skills and the ability to collaborate effectively with cross-functional teams.

- Knowledge of data security, governance, and compliance.