Senior Data Engineer - Hive/Spark

Description :

Role Summary :

We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kubernetes-based data platform. The role requires strong hands-on development skills in data engineering, with the ability to deliver high-quality pipelines under guidance from internal leads.

Key Responsibilities :

- Develop and optimize data pipelines using Spark 3.5 and Python/Scala.

- Migrate existing Hive, Spark, and Control-M jobs to Airflow and DBT-based workflows.

- Integrate data pipelines with messaging systems (Kafka, Solace) and object stores (S3, MinIO).

- Troubleshoot and optimize distributed jobs running in Kubernetes environments.

- Collaborate closely with internal leads and architects to implement best practices.

- Design and implement migration/acceleration framework to automate end to end migration.

- Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.

- Work with various data applications to enable and support the migration process.

- Deliver assigned migration tasks within agreed timelines.

Required Skills :

- 6- 9 years of hands-on data engineering experience.

- Strong expertise in Apache Spark (batch + streaming) and Hive.

- Proficiency in Python, Scala, or Java. Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).

- Experience working with Kafka, Solace, and object stores (S3, MinIO). Exposure to Docker/Kubernetes for deployment.

- Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).