We are looking for a skilled Spark Engineer to join our data engineering team.

The ideal candidate will have strong expertise in big data technologies, particularly Apache Spark, and will be responsible for building scalable data pipelines, processing large datasets, and supporting data-driven decision-making.

Key Responsibilities :

- Design, develop, and optimize scalable data processing pipelines using Apache Spark.

- Work extensively with Hive and SQL for data querying, transformation, and analysis.

- Develop and maintain data workflows using Python.

- Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.

- Manage and process large volumes of structured and unstructured data.

- Work on on-premises CDP (Cloudera Data Platform) environments for data engineering tasks.

- Ensure data quality, reliability, and performance optimization across pipelines.

- Troubleshoot and resolve performance bottlenecks in data processing jobs.

- Participate in code reviews and enforce best practices in data engineering.

Key Requirements :

- Minimum 5 years of experience in data engineering or big data technologies.

- Strong hands-on experience with Apache Spark and Hive.

- Proficiency in SQL for complex queries and data manipulation.

- Strong programming skills in Python.

- Experience working with or understanding of Cloudera Data Platform (CDP) - on-premises setup.

- Working knowledge of Java and Apache Flink.

- Good understanding of distributed computing and big data architecture.

Preferred Qualifications :

- Experience with Spark Streaming or real-time data processing.

- Familiarity with Apache Iceberg for data lake management.

- Experience with workflow orchestration tools like Apache Airflow.

- Hands-on experience with Apache NiFi for data ingestion and flow management.

- Understanding of data lake and data warehouse concepts