Capco - Senior ETL Test Automation Engineer - Kafka

Job Title : Sr. ETL Test Automation Engineer

Experience : 510 Years

Job Summary :

We are seeking a highly skilled Sr. ETL Test Automation Engineer with expertise in testing large-scale data integration, pipelines, and data warehouse ecosystems. The role requires advanced knowledge of ETL testing, automation frameworks, Kafka, and AWS data platforms, along with strong problem-solving skills to ensure data accuracy, consistency, and reliability across complex workflows.

Key Responsibilities :

Test Automation Design & Execution :

- Design and implement automated testing strategies for validating ETL pipelines, streaming data flows, and batch processes.

- Build and maintain automation frameworks to test data ingestion, transformation, storage, and delivery across heterogeneous platforms.

ETL/Data Validation :

- Perform end-to-end validation of data pipelines between DB2 source systems and downstream targets (e.g., S3, Redshift, Glue).

- Develop SQL queries for validating large datasets, transformations, and aggregations.

- Conduct data quality checks (duplicate detection, null handling, reconciliation, referential integrity).

Kafka Streaming Validation :

- Test and validate Kafka topics for schema evolution, offset tracking, and producer/consumer data consistency.

- Ensure real-time streaming data adheres to defined contracts.

Automation & CI/CD :

- Develop reusable automated test scripts using frameworks like Pytest, JUnit, or TestNG with Python/Java.

- Integrate test suites with CI/CD pipelines (Jenkins, GitHub Actions) to support continuous quality.

- Collaborate with DevOps teams to ensure scalable automated test execution.

Test Management & Reporting :

- Define test coverage and quality metrics in collaboration with developers and business analysts.

- Perform functional, regression, performance, and integration testing.

- Analyze test failures, perform root cause analysis, and document defects with reproducible steps.

Collaboration & Stakeholder Engagement :

- Work closely with ETL developers, data engineers, QA engineers, and product owners.

- Provide clear reporting on data quality and testing outcomes to technical and business stakeholders.

Required Skills & Experience

- 6+ years of QA/Test Engineering experience with strong ETL & Data Warehouse testing expertise.

Proven hands-on experience with :

- Apache Kafka (message validation, schema registry, consumer offsets).

- AWS ecosystem (S3, Redshift, Glue, Lambda, EMR).

- DB2 or other relational databases for large-scale data validation.

- Strong SQL scripting skills (complex queries, joins, data reconciliation).

- Proficiency in test automation tools/frameworks (Pytest, JUnit, TestNG, Selenium, or Python scripting).

- Familiarity with CI/CD tools (Jenkins, GitHub Actions) and version control (Git).

- Solid understanding of data quality, lineage, and metadata validation.

- Strong analytical, debugging, and communication skills.

Nice to Have :

- Experience with cloud-native data lakes and ETL orchestration.

- Exposure to Docker/Kubernetes for creating test environments.

- Experience in API testing (REST/microservices) supporting data workflows.

- Familiarity with big data tools (Spark, Hive, Hadoop).