Data Engineer - Python/Spark

Main Responsibilities :

- Design, build, and maintain scalable ETL/ELT data pipelines to ingest and process large volumes of structured and unstructured data.

- Work with cloud-native services to manage, orchestrate, and monitor data workflows primarily on AWS (Glue, Lambda, Step Functions).

- Ensure data quality, integrity, and security across the pipeline lifecycle.

- Collaborate with data scientists, analysts, and business stakeholders to understand data

requirements and deliver clean, accessible datasets.

- Optimize performance of Spark-based jobs and Python scripts for faster processing.

- Document data pipelines, architecture, and transformation logic for internal use.

- Troubleshoot issues and implement long-term improvements in the data pipelines.

Technical Skills :

- AWS Glue, AWS Lambda, and AWS Step Functions

- SQL advanced querying and optimization techniques

- PySpark building and optimizing large-scale distributed data processing

Requirements :

- 6+ years of experience in data engineering, ETL development, or a similar role

- Hands-on experience with big data technologies such as Hadoop, Apache Spark, or Kafka

- Familiarity with cloud platforms like AWS, Azure, or Google Cloud

- Version control tools (e.g., Git), CI/CD pipelines, and workflow orchestration tools (e.g., Airflow) are a plus

Soft Skills :

- Excellent verbal and written communication skills

- Team player with a collaborative attitude and the ability to work independently

- Attention to detail and a commitment to data accuracy and security