Data Engineer - Python/PySpark

Data Engineer

We are seeking a highly skilled Data Engineer with deep experience in the AWS ecosystem and a strong understanding of big data processing, data mesh architecture, and infrastructure automation. This role will play a critical part in implementing scalable data pipelines, enforcing data quality, and enabling analytics at scale.

Key Responsibilities :

- Design and develop scalable data pipelines using AWS services including EMR, Glue, S3, Athena, EC2, and Serverless computing.

- Build efficient PySpark/Spark data processing applications to handle large-scale datasets.

- Implement data quality checks and data validation processes using AWS Glue and other relevant tools.

- Work with Lake Formation to manage and govern data lakes.

- Contribute to data mesh solutions, enabling decentralized data ownership and domain-oriented architecture.

- Utilize Terraform or other IaC tools to provision and manage AWS infrastructure.

- Integrate and manage Neptune DB for graph-based use cases.

- Operate within a predefined framework, ensuring alignment with architectural and governance standards.

- Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers.

Required Technical Skills :

- AWS Stack : S3, EMR, Glue, Athena, EC2, Lambda (Serverless)

- Big Data : Spark, PySpark

- Data Lakes & Governance : AWS Lake Formation

- Data Mesh Concepts : Experience or exposure preferred

- Data Quality : Writing and automating quality checks

- Infrastructure as Code (IaC) : Terraform (Strongly preferred)

- Graph Database : Neptune DB

Preferred Qualifications :

- Experience working in a data mesh or domain-driven data architecture

- Familiarity with CI/CD pipelines and basic DevOps practices

- Strong problem-solving skills with a proactive and analytical mindset

- Excellent communication and stakeholder collaboration abilities