Data Engineer - Spark/SQL

Job Description

We are seeking a skilled Ops Data Engineer to maintain robust data infrastructure and pipelines that support our operational analytics and business intelligence needs.

Candidates will bridge the gap between data engineering and operations, ensuring reliable, scalable, and efficient data systems that enable data-driven decision making across the organization.

Primary Skills :

- Hands-on experience with Apache Kafka, Apache Flink, and other relevant streaming technologies

- Databases : Strong knowledge of relational databases (Oracle, MySQL) and NoSQL systems

- Proficiency with Version Control Git, CI/CD practices and collaborative development workflows

- Good understanding of orchestrating complex computational workflows and data processing pipelines - UC4, Apache Airflow or equivalent.

- Agile development methodologies

Nice to have :

- Proficiency in programming language

- Python/Java or equivalent

Key Responsibilities :

- Design, develop, and maintain data pipelines and infrastructure to support operational analytics and BI needs

- Implement and optimize streaming data processing using technologies like Apache Kafka and Apache Flink

- Develop and maintain complex SQL queries and data transformations using Spark SQL

- Manage and optimize data storage across relational databases (Oracle, MySQL) and NoSQL systems

- Utilize version control systems like Git, and apply CI/CD best practices for deployment and collaboration

- Orchestrate and automate complex workflows and data processing pipelines using tools such as UC4, Apache Airflow, or

equivalents

- Collaborate with cross-functional teams in an Agile development environment to deliver high-quality data solutions