Senior AWS Data Engineer - Python/Spark

We are looking for an experienced Senior Data Engineer with a strong foundation in Python, SQL, and Spark, and hands-on expertise in AWS, Databricks. In this role, you will build and maintain scalable data pipelines and architecture to support analytics, data science, and business intelligence initiatives. You'll work closely with cross-functional teams to drive data reliability, quality, and performance.

Responsibilities :

- Design, develop, and optimize scalable data pipelines using Databricks in AWS, such as Glue, S3 Lambda, EMR, Databricks notebooks, workflows, and jobs.

- Building a data lake in AWS Databricks.

- Build and maintain robust ETL/ELT workflows using Python and SQL to handle structured and semi-structured data.

- Develop distributed data processing solutions using Apache Spark or PySpark.

- Partner with data scientists and analysts to provide high-quality, accessible, and well-structured data.

- Ensure data quality, governance, security, and compliance across pipelines and data stores.

- Monitor, troubleshoot, and improve the performance of data systems and pipelines.

- Participate in code reviews and help establish engineering best practices.

- Mentor junior data engineers and support their technical development.

Requirements :

- Bachelor's or master's degree in computer science, Engineering, or a related field.

- 5+ years of hands-on experience in data engineering, with at least 2 years working with AWS Databricks.

- Strong programming skills in Python for data processing and automation.

- Advanced proficiency in SQL for querying and transforming large datasets.

- Deep experience with Apache Spark/PySpark in a distributed computing environment.

- Solid understanding of data modelling, warehousing, and performance optimization techniques.

- Proficiency with AWS services such as Glue, S3 Lambda, and EMR.

- Experience with version control, Gi, t or Code commit.

- Experience in any workflow orchestration, like Airflow, AWS Step function, s is a plus.