Spark Rockstar Wanted!

Imagine a world where big data is not just a buzzword, but a playground where you get to build innovative solutions that make a real impact. We're on the hunt for a Senior Big Data Engineer who's passionate about Spark and has the skills to prove it. If you're a Spark aficionado with a knack for designing and developing scalable data pipelines, we want to hear from you!

Responsibilities :

- Lead the charge in building and maintaining our Spark-based data ecosystem, with a focus on batch and streaming processing

- Collaborate with our team of data wizards to design and implement ETL workflows that migrate seamlessly to Spark

- Whip up Python, Scala, and Java code like a pro, and use your expertise to integrate Spark with Hadoop, HDFS, YARN, and Kafka

- Streamline our workflows using Airflow or other workflow engines, and make Jupyter notebooks do your bidding

- Bonus points if you can conjure up workflows with NiFi, Flink, and Kudu streaming!

Qualifications :

- 5+ years of experience in Spark development, with a strong portfolio of projects that showcase your expertise

- Hands-on experience with Spark, Hadoop, HDFS, YARN, Kafka, and Airflow (or other workflow engines)

- Programming prowess in Python, Scala, and Java

- Experience with ETL implementation and migration to Spark

- Good to have : Flink and Kudu streaming, automation of workflows CI/CD, Nifi streaming and transformation, and Informatica workflow migration