ACL Digital - Data Engineer - Big Data

Description :

- Spark

- HDFS

- Hive

- Spark SQL

- Kafka/Flink

Job Description :

Key Responsibilities :

- Design, develop, and maintain robust data pipelines in Hadoop and related ecosystems, ensuring data reliability, scalability, and performance.

- Implement data ETL processes for batch and streaming analytics requirements.

- Optimize and troubleshoot distributed systems for ingestion, storage, and processing.

- Collaborate with data engineers, analysts, and platform engineers to align solutions with business needs.

- Ensure data security, integrity, and compliance throughout the infrastructure.

- Maintain documentation and contribute to architecture reviews.

- Participate in incident response and operational excellence initiatives for the data warehouse.

- Continuously learn mindset and apply new Hadoop ecosystem tools and data technologies.

Required Skills and Experience :

- Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.

- Proven ability to design and implement automated data pipelines and materialized views.

- Proficiency in Python, Unix or similar languages.

- Good understanding of SQL oracle, SQL server or similar languages.

- Ops & CI/CD : Monitoring (Prometheus/Grafana), logging, pipelines (Jenkins/GitHub Actions).

- Core Engineering : Data structures/algorithms, testing (JUnit/pytest), Git, clean code.

- 5+ years of directly applicable experience

- BS in Computer Science, Engineering, or equivalent experience.