We are seeking an innovative and detail-oriented Agentic AI Test Engineer to join our AI Engineering team. The ideal candidate will have strong expertise in AI/ML testing, LLM validation, autonomous AI agents, and software quality engineering. You will be responsible for validating the performance, reliability, safety, and accuracy of AI agents, multi-agent systems, and GenAI-powered applications across mobile, web, and enterprise environments. This role requires a strong blend of AI testing (70%) and software automation/quality engineering (30%) to ensure robust and scalable AI-driven products.

The candidate will have responsibilities across the following functions:

Agentic AI and GenAI Testing (70%) :

- Design and execute testing strategies for Agentic AI systems, autonomous workflows, and LLM-powered applications.

- Validate AI agents' reasoning, planning, memory, tool usage, and decision-making capabilities.

- Perform functional, regression, performance, and behavioural testing for AI agents.

- Develop test scenarios for prompt engineering, response validation, hallucination detection, and output consistency.

- Evaluate AI model performance using metrics such as accuracy, reliability, latency, relevance, and safety.

- Conduct multi-agent orchestration testing and agent-to-agent interaction validation.

- Test integrations with LLMs (GPT, Claude, Gemini, Llama, etc.), APIs, vector databases, and retrieval systems (RAG).

- Ensure compliance with AI governance, ethical AI, bias testing, security, and privacy standards.

- Identify risks related to hallucination, prompt injection, data leakage, adversarial attacks, and model drift.

- Create automated validation frameworks for AI-generated outputs.

QA and Automation Testing (30%):

- Develop and maintain automation test scripts using frameworks such as Selenium, Playwright, Appium, Cypress, or API automation tools.

- Perform functional, API, integration, and end-to-end testing.

- Work with CI/CD pipelines for automated quality validation.

- Conduct defect tracking, root cause analysis, and collaborate with engineering teams for issue resolution.

- Create and maintain test plans, test cases, and quality documentation.

Requirements:

- Bachelor's/Master's degree in Computer Science, Engineering, AI, or related field.

- 4+ years of QA/Test Automation experience with exposure to AI/GenAI testing.

- Hands-on experience in LLM testing, Prompt Testing, RAG validation, and Agentic AI systems.

- Strong understanding of AI evaluation frameworks, model behaviour analysis, and autonomous systems testing.

- Experience with Python, Java, or JavaScript for automation and testing.

- Knowledge of Prompt Engineering, LangChain, LangGraph, CrewAI, AutoGen, MCP, OpenAI APIs, or AI orchestration frameworks.

- Experience with API testing tools such as Postman or Rest Assured.

- Familiarity with cloud platforms (AWS, Azure, GCP) is a plus.

- Strong analytical, debugging, and problem-solving skills.

Good to Have:

- Experience in AI Safety Testing / Responsible AI.

- Exposure to Mobile Testing (Android/iOS) using Appium.

- Understanding of MLOps, vector databases, embeddings, and RAG pipelines.

- Experience with performance testing tools such as JMeter or Locust.