Posted on: 17/11/2026
Hiring Update: Senior Automation QA Engineer
Experience: 5+ Years
Location: Bangalore
Mode of Work: Hybrid
Summary:
We are looking for a Senior Automation QA Engineer to define and execute test strategies for AI agent features in the Agent Development Squad. This role goes beyond standard API and UI automation. You must understand the unique challenges of testing non-deterministic LLM outputs, multi-step agent graphs, and audit-critical financial workflows. Pytest and Playwright are the primary tooling standards. LLM and agent testing awareness is required.
What You'll Do:
- Design and implement test automation strategies for backend APIs, agent orchestration logic, and Angular-based agent UIs.
- Build and maintain Pytest-based test suites for Python backend services and agent workflow validation.
- Implement Playwright automation for Angular-based UI, covering interaction flows, approval queues, and streaming response handling.
- Validate multi-step agent workflows by asserting on intermediate states, tool call outputs, and handoff results.
- Use Langfuse traces to validate agent reasoning chains, confidence scores, and execution paths.
- Build and maintain LLM evaluation harnesses covering prompt regression testing, RAG output validation, and model version change detection.
- Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates.
- Verify audit log integrity, including immutable log structure, correlation ID chains, and SOX-compliant event emission.
Who You Are:
Test Automation:
- 5+ years of QA automation experience for API and web-based applications.
- Pytest as the primary test framework for Python backend and agent logic validation.
- Playwright for UI automation targeting Angular-based interfaces.
- REST API testing using Postman, HTTPX, or equivalent tools.
- Python for test scripting.
- SQL/NoSQL for test data setup, teardown, and database state validation.
- Azure DevOps CI/CD pipeline integration.
LLM / Agent Testing Awareness:
- Understands that LLM outputs are non-deterministic and knows how to adapt assertion patterns.
- Familiarity with multi-step agent graph testing, including asserting on intermediate states, tool calls, and handoff outputs.
- Understanding of Langfuse traces and their use for agent validation.
- Awareness of model version regression risks and quality degradation detection.
Nice to Have:
- Hands-on Langfuse experience covering trace validation, eval pipeline setup, and confidence score assertion.
- Experience with LLM evaluation frameworks such as Ragas or equivalent.
- SOX/GDPR audit trail validation.
- Azure Monitor or Application Insights for observability-assisted debugging.
What You'll Learn & Gain:
- A front-row seat to defining what quality means for AI agent systems.
- Hands-on experience building LLM evaluation harnesses and agent validation frameworks.
- Deep familiarity with Langfuse, agent trace analysis, and confidence-based quality gates.
- Collaboration with a cross-functional squad of backend, frontend, and AI/ML engineers.
The job is for:
Did you find something suspicious?
Posted by
Recruiter
Last Active: NA as recruiter has posted this job through third party tool.
Posted in
Quality Assurance
Functional Area
QA & Testing
Job Code
1683901