AI-powered systems require extensive testing beyond traditional software validation.

Forecasting accuracy, conversational AI behavior, model responses, and orchestration workflows must be rigorously validated to ensure reliability, performance, and user trust.

This role is critical in ensuring the quality, scalability, and robustness of AI applications, including Bedrock Agent workflows, forecasting systems, conversational interfaces, and API integrations.

Primary Responsibilities :

- Build and maintain automated test suites for AWS Bedrock Agent Actions (Action Groups), model invocation workflows, and API integrations

- Validate forecasting model accuracy against historical datasets and expected business outcomes

- Test conversational AI flows, edge cases, fallback scenarios, and error handling mechanisms

- Conduct performance, scalability, and load testing across backend services and AI systems

- Create and maintain regression test suites for model updates and platform enhancements

- Develop automated API validation and integration testing pipelines

- Collaborate with engineering and AI teams to identify defects, improve quality, and strengthen test coverage

- Validate security, reliability, and resiliency of distributed AI-powered applications

- Monitor testing metrics and ensure continuous quality improvements across releases

Required Technical Skills :

- Hands-on experience with test automation frameworks such as DeepEval, Playwright, Pytest, and Jest

- Strong understanding of ML/AI model evaluation metrics and validation techniques

- Experience with API testing tools such as Postman and REST Assured

- Performance and load testing experience using JMeter, Locust, or similar tools

- Strong understanding of regression testing, integration testing, and end-to-end testing methodologies

- Experience testing REST APIs, asynchronous workflows, and distributed systems

- Knowledge of conversational AI testing, prompt validation, and response quality assessment

- Familiarity with CI/CD pipelines and automated quality gates

- Strong analytical, debugging, and problem-solving skills

Preferred Qualifications :

- Experience testing AI assistants, copilots, or conversational AI platforms

- Familiarity with AWS Bedrock, LLM orchestration workflows, and GenAI systems

- Understanding of forecasting or analytics-driven applications

- Knowledge of cloud-native and event-driven architectures

- Exposure to observability, tracing, and AI evaluation tooling

- Ability to work closely with cross-functional engineering and AI teams

Ideal Candidate Profile :

- Strong QA/Test engineering experience with AI/GenAI systems

- Passion for quality engineering and automation

- Experience validating scalable AI-powered applications and workflows

- Strong attention to detail and analytical mindset

- Comfortable working in agile, fast-paced engineering environments