IBM - Spark Data Engineer/Developer - Python/Scala

Description :

Spark Data Engineer / Developer (DevOps & OpenShift)

About the Role :

We are seeking a highly skilled and experienced Spark Data Engineer / Developer to join our dynamic team. This role is critical for building, optimizing, and supporting our cutting-edge data platform, leveraging Apache Spark, Apache Iceberg, and a robust DevOps approach within an OpenShift environment. The ideal candidate will be adept at both developing high-performance data solutions and ensuring their stability and reliability in a production setting.

Key Responsibilities :

- Design, develop, and optimize scalable and resilient data processing applications using Apache Spark (batch, streaming, and real-time).

- Implement and manage data pipelines, ensuring data quality, consistency, and performance.

- Perform Spark job performance tuning and optimization to handle large-scale datasets efficiently.

- Manage and automate the deployment of Spark applications within OpenShift clusters, utilizing Docker and Kubernetes.

- Establish and maintain CI/CD pipelines for automated testing, deployment, and release management of Spark workloads.

Required Skills & Experience :

- 9+ years of hands-on experience in data engineering or software development roles.

- Expert-level proficiency in Apache Spark (Spark Core, Spark SQL, Spark Streaming).

- Strong programming skills in Scala, Python (PySpark), or Java.

- Significant experience with OpenShift, including deploying, managing, and automating containerized applications within the platform.

- Solid understanding of Docker and Kubernetes for containerization and orchestration.

- Proven experience implementing and maintaining CI/CD pipelines using tools like Jenkins,