Description :

We are seeking a Big Data QA Engineer with strong experience in validating and testing large-scale data processing systems.

The ideal candidate will design, execute, and maintain automated and manual test suites focused on batch workflows, data integrity, and transformation validation.

You will work closely with data engineers, developers, and product teams to ensure high-quality, reliable, and scalable data pipelines.

Key Responsibilities :

- Design, develop, and execute automated and manual batch test cases for big data pipelines.

- Validate data integrity, schema accuracy, and transformation logic across multiple stages.

- Monitor and troubleshoot batch job failures, performance issues, and data mismatches.

- Maintain and enhance a Python-based test automation framework.

- Reproduce data issues, analyze system performance, and collaborate with developers to implement fixes.

- Analyze logs and metrics to determine root cause of failures.

- Work closely with cross-functional teams to improve testing processes and ensure high-quality deliveries.

Required Qualifications :

- Minimum 4 years of experience in Big Data testing, batch processing, and data validation.

- Strong proficiency in Python for automation and scripting.

- Hands-on experience with AWS services such as S3, Lambda, EMR, and Athena.

- Experience working with big data formats: JSON, Parquet, CSV.

- Practical knowledge of Pandas and PySpark for data analysis and validation.

- Experience with version control tools (Git) and CI/CD pipelines (Jenkins, GitHub Actions).

- Strong understanding of testing methodologies: regression, integration, system testing.

- Ability to analyze logs/metrics and solve data quality issues.

- Experience working in Agile/Scrum environments.

- Excellent communication and collaboration skills.

Preferred Qualifications :

- Bachelors degree in Computer Science, Information Technology, or equivalent.

- Experience with large-scale distributed data systems and cloud-based architectures.

- Strong analytical mindset with attention to detail.

Summary :

This role is ideal for someone who is highly skilled in Python automation, AWS big data services, and batch data validation, with a passion for ensuring the accuracy and reliability of data pipelines.

Skills: aws,python,big data,testing.