- Develop comprehensive test strategies, test plans, and test cases for AWS Glue inbound and outbound file processing data pipelines.

- Test data ingestion from sources such as :

1. Amazon S3

2. CSV, Excel, XML, fixed-length, variable-length, and delimited files

- Validate AWS Glue inbound and outbound file processing jobs developed using Python and PySpark.

- Validate Glue Workflows, triggers, job dependencies, schedules, retries, and failure handling.

- Test EventBridge-based scheduling and event-driven pipeline execution.

- Test StepFunction / Glue Workflows based file processing orchestration.

- Create comprehensive test cases for file and field level data validation/data quality checks for each of the file processing interface.

- Validate data transformations, joins, aggregations, filtering, sorting, lookups, enrichment, and business rules.

- Perform source-to-target data validation and reconciliation.

- Validate record counts, control totals, financial totals, hash totals, file counts, and data completeness.

- Test full-load, incremental-load, delta-load, and change-data-capture processes.

- Verify data types, field lengths, precision, scale, formats, null handling, default values, and mandatory fields.

- Test duplicate detection, referential integrity, orphan records, invalid records, and rejected-record processing.

- Test pipeline/interface restartability, rerun capability, idempotency, checkpointing, and recovery from failures.

- Validate error handling, exception management, dead-letter processing, and alert notifications.

- Test data movement between S3, Oracle, RDS, and other target systems.

- Validate CloudWatch logs, metrics, alarms, execution status, and operational audit information.

- Perform performance, volume, scalability, and batch-window testing for high-volume data pipelines.

- Identify data quality issues and work with developers to determine root causes.

- Create SQL queries and validation scripts for data comparison and reconciliation.

- Develop reusable test automation frameworks for data validation and regression testing.

- Validate security controls, including IAM permissions, encryption, access restrictions, and sensitive-data handling.

- Support system integration testing, regression testing, user acceptance testing, and production validation.

- Participate in Agile ceremonies, backlog refinement, defect triage, sprint planning, and release activities.

- Maintain test evidence, execution reports, defect reports, traceability matrices, and quality dashboards.

Required Technical Skills :

AWS Services :

- Familiarity with :

1. AWS Glue Jobs

2. AWS Glue Workflows

3. Amazon S3

4. Amazon CloudWatch

5. AWS IAM

6. Amazon EventBridge

7. AWS Lambda

8. Amazon SNS

9. Amazon SQS

10. AWS Secrets Manager

11. AWS Step Functions

12. AWS KMS

- Ability to review AWS Console configurations, job parameters, logs, metrics, and execution history.

SQL and Database Skills :

- Advanced SQL skills, including :

1. Joins

2. Subqueries

3. Common table expressions

4. Window functions

5. Aggregations

6. Data comparison queries

7. Duplicate and exception identification

- Experience with Oracle, or similar databases.

- Ability to validate large datasets efficiently using SQL.

Programming and Automation :

- Working knowledge test automation and data validation.

- Experience with AWS Glue ETL/Pipeline test automation.

- Experience developing automated checks for :

1. File availability

2. File naming and file counts

3. Schema validation

4. Record counts

5. Data completeness

6. Data accuracy

7. Duplicate records

8. Control totals

9. Source-to-target reconciliation

- Familiarity with pytest, unittest, Robot Framework, or similar testing frameworks