Senior Big Data Engineer - Scala/Python

Senior Data Engineer

Responsibilities :

- Skilled and experienced Sr. Data Engineer with proven experience in Spark, Kafka, Hadoop, Data Management.

Skills :

- Proficiency in programming languages Scala, Java, Python.

- Experience with design and development of ELT workflows and data pipelines for high-volume data systems e.g. Datastage, Informatica, Talend, etc.

- Experience in Data Engineering and implementation of Real-time data processing pipelines using Kafka, Kafka Streams and Spark Streams

- Good analytical skills with excellent knowledge of SQL, data management and relational databases like PostgreSQL, Oracle, MySQL.

- Experience working with Big Data Tools/Technologies and DataLakes (Hadoop,Map/Reduce,Spark,Hive)

- Knowledge/Experience working with In-memory/NoSQL databases - Redis, HBase, Cassandra, CouchBase, CouchDB

- Knowledge/experience with Apache Nifi/Airflow for Data Pipelines Orchestrations.

- Proven expertise with linux administration and unix scripting for the automation of tasks, Containerzation[Kubernetes,Docker,Openshift].

- Experience of building CI/CD tools (Jenkins).

- AI and Machine Learning knowledge, Database Administration, Big Data Storage, Data Analytics and Feature Engineering, ELK Stack, FLink

- Familiar with cloud platforms like AWS, Google Cloud, or Microsoft Azur.