Job Description :

Hiring : Automation QA Engineer Big Data Testing

Location : Pune

Experience : 13 to 18 Years

About the Role :

We are looking for a highly skilled Automation QA Engineer with strong expertise in Big Data Testing to join our growing data engineering and quality assurance team. The ideal candidate will have hands-on experience validating large-scale data platforms, testing complex ETL pipelines, and developing automated data validation frameworks using Python and PySpark.

This role requires a strong understanding of distributed data processing systems, cloud-based data platforms, and automation testing methodologies to ensure data accuracy, reliability, and platform performance across enterprise-scale environments.

Key Responsibilities :

- Design, develop, and execute test strategies for Big Data applications, ETL processes, and distributed data platforms.

- Perform end-to-end validation of large-scale data pipelines using Python, PySpark, and Spark SQL.

- Develop and maintain automated testing frameworks and reusable test scripts for data validation and quality assurance.

- Validate data transformations, aggregations, and business rules across complex ETL workflows.

- Execute data reconciliation, data integrity, and data quality testing across multiple data sources and platforms.

- Work with large structured and unstructured datasets to ensure consistency, completeness, and accuracy.

- Collaborate with Data Engineers, Developers, Business Analysts, and Product Teams to understand requirements and define test scenarios.

- Identify, document, and track defects while working closely with development teams for timely resolution.

- Perform API, database, and backend testing for data-intensive applications.

- Leverage Spark SQL to validate large datasets and perform complex data comparisons.

- Participate in test planning, estimation, automation strategy discussions, and release readiness activities.

- Integrate automated test suites into CI/CD pipelines to support continuous testing and faster release cycles.

- Monitor test execution results and provide quality metrics and reporting to stakeholders.

- Ensure platform stability, data reliability, and adherence to quality standards throughout the development lifecycle.

Required Skills & Experience :

- 13 to 18 years of experience in Automation Testing, Data Testing, ETL Testing, or Big Data Quality Assurance.

- Strong hands-on experience with Python for test automation and data validation.

- Expertise in PySpark and Spark SQL for testing large-scale distributed data environments.

- Solid experience validating ETL pipelines, data migrations, and data transformation processes.

- Strong understanding of Big Data technologies and distributed processing frameworks.

- Experience working with AWS cloud services including S3, EMR, Glue, Lambda, and related data services.

- Hands-on experience writing complex SQL queries and performing database testing.

- Strong understanding of software testing methodologies, defect management, and test lifecycle processes.

- Experience working in Agile and Scrum-based development environments.

- Excellent analytical, debugging, and problem-solving skills.

- Strong communication and stakeholder collaboration abilities.

Good to Have :

- Experience with Databricks and Databricks Notebooks.

- Exposure to Delta Lake and modern data lake architectures.

- Experience with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI, or Azure DevOps.

- Knowledge of automation testing frameworks and test orchestration tools.

- Understanding of data quality frameworks, data governance, and data observability practices.

- Exposure to cloud-native testing and validation methodologies.

- Experience with API testing and performance testing tools.

Preferred Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

- Certifications in AWS, Big Data technologies, Data Engineering, or Quality Assurance will be an added advantage.

Ideal Candidate :

The ideal candidate will possess strong automation testing expertise combined with deep knowledge of Big Data ecosystems. You should be comfortable working with large-scale data platforms, building robust automation frameworks, and ensuring data accuracy and reliability in cloud-based environments while collaborating effectively with cross-functional teams.