Job Summary:

We are looking for an AI QA Engineer with hands-on experience in Generative AI, RAG, LLM applications, and AI test automation. The ideal candidate should have strong QA fundamentals along with practical experience using AI automation and evaluation tools such as Promptfoo, TruLens, DeepEval, and LangSmith.

The candidate will be responsible for validating the functionality, quality, performance, security, reliability, and accuracy of AI/GenAI applications.

Key Responsibilities:

- Design and execute end-to-end QA strategies for AI/GenAI applications.

- Test AI applications involving LLMs, RAG pipelines, LangChain/LangGraph and GenAI workflows.

- Build and maintain automated test frameworks for AI/LLM-based applications.

- Use AI evaluation and testing tools such as Promptfoo, TruLens, DeepEval, and LangSmith.

- Develop test cases for prompts, LLM responses, RAG pipelines, agents, and AI workflows.

- Evaluate AI model performance using relevant AI/LLM evaluation metrics, including accuracy, relevance, faithfulness, groundedness, toxicity, hallucination, and response quality.

- Perform functional, non-functional, regression, integration, and performance testing.

- Conduct performance testing of AI systems, including latency, throughput, scalability, and response-time analysis.

- Support security testing of AI applications, including prompt injection, data leakage, unsafe outputs, and other AI-specific security risks.

- Identify, reproduce, document, and track functional and AI-specific defects/issues.

- Establish effective problem identification, root-cause analysis, defect reporting, and escalation mechanisms.

- Validate AI outputs against defined business and technical expectations.

- Develop automated evaluation pipelines and regression test suites for continuous validation of AI systems.

- Prepare detailed QA validation reports, test execution reports, defect reports, and quality assessment reports.

- Collaborate with AI/ML Engineers, Developers, Product Managers, and Business Teams to resolve issues and improve application quality.

Required Skills:

- 5+ years of experience in QA / Test Automation with relevant experience in AI/GenAI testing.

- Hands-on understanding of Generative AI, LLMs, RAG, and AI application workflows.

- Experience with LangChain / LangGraph or similar AI orchestration frameworks.

- Hands-on experience with one or more AI testing/evaluation tools: Promptfoo, TruLens, DeepEval, LangSmith.

- Strong understanding of AI/LLM evaluation metrics.

- Experience in Functional, Non-Functional, Regression, Integration, and Performance Testing.

- Experience in test automation and scripting/programming, preferably Python.

- Good understanding of API testing and automation.

- Experience in defect tracking, root-cause analysis, and QA reporting mechanisms.

- Strong understanding of the end-to-end QA lifecycle, from test planning and test-case creation through execution, defect management, validation, and reporting.