Role Overview :

We are looking for a QA Engineer (Software Test Engineer - Level 2/3) to ensure the quality, reliability, and performance of LLM-powered applications. You will work closely with AI/ML engineers and product teams to design testing strategies, evaluate model outputs, and build robust validation frameworks for generative AI systems.

This role is ideal for someone excited about AI quality, prompt evaluation, and LLM testing frameworks.

Mandatory Skills :

- Experience in testing LLM (Large Language Model) applications

- Hands-on exposure to DeepEval (or similar LLM evaluation tools)

- Knowledge of LLM evaluation frameworks (RAG evaluation, hallucination detection, prompt testing, etc.)

- Understanding of software testing concepts (test cases, test plans, regression testing)

- Familiarity with API testing and automation basics

Key Responsibilities :

- Design and execute test strategies for LLM-based features

- Validate model outputs for correctness, relevance, hallucination, bias, and safety

- Use DeepEval or similar frameworks to automate LLM evaluation

- Build test datasets, benchmarks, and evaluation pipelines

- Perform prompt testing and optimization validation

- Collaborate with AI/ML teams to improve model performance based on QA findings

- Report, track, and prioritize defects in AI workflows

- Ensure product quality across RAG systems, chatbots, and generative AI use cases

Good to Have :

- Basic knowledge of Python

- Experience with API testing tools (Postman, REST Assured, etc.)

- Understanding of ML concepts and model evaluation metrics

- Exposure to automation testing frameworks

- Knowledge of data annotation or NLP tasks

Qualifications :

- Bachelor's degree in Computer Science, IT, or related field

- 0- 4 years of experience in QA / Software Testing / AI testing