Atyeti - ETL Test Automation Engineer - Data Validation

Job Description :

We are looking for an experienced ETL Test Automation Engineer with 6-10 years of experience in data testing, ETL validation, and test automation.

The ideal candidate will have strong expertise in SQL, Python, and automation frameworks such as PyTest or Robot Framework, along with hands-on experience in designing and implementing automated testing solutions for complex data pipelines.

The candidate will be responsible for ensuring data accuracy, completeness, consistency, and integrity across data ingestion, transformation, and loading processes.

This role requires strong analytical and problem-solving skills, experience in automation and regression testing, and the ability to collaborate with data engineers, developers, business analysts, and QA teams.

Experience in AI/GenAI testing, PySpark, Databricks, Snowflake, and cloud-based data platforms will be an added advantage.

Key Responsibilities :

- Design, develop, and execute comprehensive test strategies for ETL processes, data pipelines, and data warehouse solutions.

- Perform end-to-end ETL testing to validate data extraction, transformation, and loading across multiple source and target systems.

- Validate data accuracy, completeness, consistency, integrity, and quality across various data layers.

- Develop and execute complex SQL queries to perform source-to-target data validation, reconciliation, and data transformation testing.

- Perform data migration testing, database testing, and backend testing to ensure accurate data movement across systems.

- Identify, analyze, and troubleshoot data discrepancies, transformation errors, and pipeline failures.

- Validate business rules, data mappings, aggregations, joins, and complex transformation logic.

- Perform negative testing, boundary testing, and exception handling validation for ETL workflows.

- Design, develop, and maintain automated test scripts using Python and frameworks such as PyTest or Robot Framework.

- Build scalable and reusable automation frameworks for ETL testing, data validation, and regression testing.

- Automate repetitive manual testing activities to improve test coverage, execution efficiency, and reliability.

- Develop automated test cases for data ingestion, transformation, reconciliation, and data quality validation.

- Integrate automated test suites into CI/CD pipelines to enable continuous testing and faster feedback.

- Implement test data management strategies and reusable automation utilities.

- Maintain and enhance existing automation scripts, frameworks, and test libraries.

- Identify opportunities to improve automation coverage and optimize test execution time.

- Write and optimize complex SQL queries involving joins, subqueries, CTEs, window functions, aggregations, and stored procedures.

- Perform database validation, backend testing, and data reconciliation across relational and analytical databases.

- Validate data transformations, business logic, referential integrity, and data consistency.

- Perform large-volume data validation and identify performance bottlenecks in data processing workflows.

- Analyze database structures, schemas, and data models to develop effective test scenarios.

- Validate incremental data loads, full data loads, and change data capture (CDC) processes, wherever applicable.

- Develop and maintain automated regression test suites for ETL workflows and data pipelines.

- Execute functional, integration, system, and end-to-end testing to ensure the reliability of data solutions.

- Perform impact analysis for changes in data models, transformation logic, source systems, and business requirements.

- Validate data pipeline stability following code changes, enhancements, and platform upgrades.

- Support performance and scalability testing of ETL workflows, identifying slow-running queries and inefficient transformations.

- Analyze test results, document defects, track resolutions, and perform retesting to ensure quality delivery.

- Use Python for test automation, data comparison, validation, reconciliation, and test data generation.

- Develop reusable Python utilities for data profiling, data quality checks, and automated test execution.

- Work with Python libraries such as Pandas and NumPy for data manipulation, comparison, and validation.

- Build automated validation scripts to compare datasets across multiple databases, files, and data platforms.

- Develop scripts to automate test reporting, defect analysis, and execution monitoring.

- Leverage PyTest fixtures, parameterization, assertions, and test organization to build maintainable automation suites.

- Support testing and validation of AI/ML and Generative AI-based applications, where applicable.

- Develop automated test scenarios to validate AI-generated outputs for accuracy, consistency, completeness, and adherence to expected formats.

- Perform data validation for AI/ML training datasets, data pipelines, and model input/output workflows.

- Support testing of LLM-based applications, including prompt validation, response consistency, and data integrity.

- Contribute to the development of automated testing approaches for AI-driven data processing solutions.

- Work with big data technologies such as PySpark for distributed data processing and large-scale data validation.

- Perform data testing and validation on platforms such as Databricks and Snowflake.

- Validate data pipelines, transformations, and processing workflows in cloud-based data environments.

- Support testing of cloud data warehouses, data lakes, and lakehouse architectures.

- Work with cloud-based ETL and data integration solutions, ensuring data accuracy, reliability, and scalability.

- Collaborate with data engineering teams to troubleshoot issues across distributed data processing environments.

- Collaborate with data engineers, software developers, business analysts, product teams, and QA professionals to understand business and technical requirements.

- Review functional specifications, technical design documents, data mapping documents, and transformation rules.

- Prepare detailed test plans, test scenarios, test cases, automation scripts, and test execution reports.

- Track and manage defects using defect management tools and ensure timely resolution.

- Participate in Agile ceremonies, sprint planning, daily stand-ups, retrospectives, and release activities.

- Provide regular updates on testing progress, risks, automation coverage, and quality metrics.

Mandatory Skills :

- 6-10 years of experience in ETL testing, data validation, database testing, and test automation.

- Strong hands-on experience in ETL testing, data warehouse testing, and end-to-end data pipeline validation.

- Excellent SQL skills, including complex queries, joins, CTEs, subqueries, window functions, and data reconciliation.

- Strong programming experience in Python for test automation and data validation.

- Hands-on experience with PyTest and/or Robot Framework.

- Experience in developing and maintaining test automation frameworks.

- Strong understanding of functional testing, integration testing, regression testing, and backend testing.

- Experience in data quality testing, source-to-target validation, and data integrity checks.

- Knowledge of Agile methodologies and defect management processes.

- Strong analytical, debugging, and problem-solving skills.

- Experience in AI/ML or Generative AI application testing.

- Knowledge of LLM testing, prompt validation, and AI output validation.

- Hands-on experience with PySpark and distributed data processing.

- Exposure to Databricks, Snowflake, and modern data warehouse architectures.

- Experience with cloud platforms such as AWS, Azure, or GCP.

- Knowledge of CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.

- Familiarity with data orchestration tools such as Apache Airflow.

- Experience with big data testing, data lake validation, and lakehouse architectures.

- Knowledge of data quality frameworks and monitoring solutions.