Senior Data Engineer - Python/Apache Spark

Technical Stack :

- AWS EMR

- AWS ECS

- Step Functions

- Apache Spark

- PostgreSQL

- DynamoDB

- Glue

- Apache Iceberg

About the Role :

Key Responsibilities :

- Design and implement endtoend data pipelines using Python and Apache Spark, ensuring high performance and reliability.

- Build and maintain scalable data storage solutions such as DynamoDB, Aurora, PostgreSQL, and Redshift, optimizing for low latency and cost efficiency.

- Architect and deploy data lakehouse structures on AWS, leveraging services like S3, Glue, and Athena to enable unified analytics.

- Develop serverless workflows with AWS Step Functions, Lambda, and EventBridge to automate data ingestion, transformation, and orchestration.

- Participate in Agile ceremonies, contribute to sprint planning, and deliver incremental value in short iterations.

Requirements :

- Proficiency in Python and Spark programming.

- Experience building highly scalable, low-latency data systems (e.g., DynamoDB, Aurora, PostgreSQL, Redshift).

- Proven experience designing and implementing data lakehouse architectures on AWS.

- Hands-on experience with AWS services such as Step Functions, Lambda, and EventBridge.

- Experience in Agile development.