About the Role:

We are looking for a Senior Automation Test Engineer to own the end-to-end testing strategy across UI, APIs, and CI/CD pipelines. The role involves building reliable automation frameworks, evaluating AI-generated responses, and establishing trusted pass/fail signals to ensure release quality.

Key Responsibilities:

- Define and implement end-to-end testing strategies covering unit, API, UI, contract, smoke, and release-gate testing.

- Develop and maintain robust automation frameworks using Playwright or equivalent tools, along with API testing frameworks.

- Design AI response testing using golden datasets, prompt/playbook regression tests, structured-output validation, and grounding/citation assertions.

- Build evaluation harnesses for LLM and AI workflows, including scoring, thresholds, failure triage, and regression gates.

- Implement retrieval-quality checks wherever required by the product, without owning the data platform.

- Integrate automated test suites into CI/CD pipelines with a focus on reducing flaky tests and providing clear quality dashboards.

- Mentor team members on risk-based test coverage and maintainable test fixtures for tenant-based or multi-environment data.

Required Skills & Experience:

- B.Tech degree.

- 10+ years of experience in QA / SDET, including ownership of end-to-end product testing strategy at a senior level.

- Strong programming skills in TypeScript, C#, or Python for test automation and evaluation frameworks.

- Hands-on experience evaluating LLM / AI outputs using golden datasets, evaluation rubrics, and regression gates.

- Experience with CI/CD tools such as Azure DevOps or GitHub Actions.

- Strong experience in API and UI automation at scale.

- Proven ability to improve flaky test suites and establish reliable release-quality signals.

Nice to Have:

- Experience testing RAG or hybrid search quality.

- Exposure to OWASP security smoke testing and performance smoke testing within CI/CD pipelines.

- Experience working with ephemeral and seeded test environments.