AI Tester - Automation Testing

Job Title : AI Tester / GenAI-Tester

Experience : 610 Years

Location : Bangalore

Work Mode : Work From Office 4 Days Mandatory

Employment Type : Full-Time

Industry : Banking / Technology

Job Summary :

We are looking for an experienced AI-Native Quality Engineer / AI Tester to redefine quality assurance for AI-powered digital experiences. The role involves validating AI/ML models, testing LLM-based systems, automating AI testing frameworks, and ensuring the reliability, accuracy, fairness, and performance of AI agents and intelligent applications.

The ideal candidate should possess strong expertise in AI testing methodologies, prompt engineering, automation frameworks, API testing, CI/CD integration, and AI observability tools.

This role will work closely with Data Scientists, AI Researchers, Developers, Product Teams, and DevOps teams to ensure high-quality AI-driven solutions aligned with enterprise standards.

Key Responsibilities :

- Design and execute test cases for AI/ML models including NLP, recommendation systems, classification models, and Generative AI systems

- Validate model predictions and outputs for accuracy, consistency, bias detection, and expected behavior

- Perform regression testing after model retraining, tuning, or dataset updates

- Evaluate LLM outputs for hallucinations, correctness, relevance, toxicity, and business rule compliance

- Identify edge cases, response drift, and prompt vulnerabilities in AI systems

- Conduct adversarial and red-team testing using prompt injection and perturbed input scenarios

AI/ML Model Testing :

- Design and execute test cases for AI/ML models including NLP, recommendation systems, classification models, and Generative AI systems

- Validate model predictions and outputs for accuracy, consistency, bias detection, and expected behavior

- Perform regression testing after model retraining, tuning, or dataset updates

- Evaluate LLM outputs for hallucinations, correctness, relevance, toxicity, and business rule compliance

- Identify edge cases, response drift, and prompt vulnerabilities in AI systems

- Conduct adversarial and red-team testing using prompt injection and perturbed input scenarios

Prompt Engineering & LLM Evaluation :

- Develop and maintain prompt test libraries for LLM-based applications

- Define LLM evaluation strategies and benchmark frameworks

- Work with AI tools such as Claude, Gemini, UiPath, and OpenAI-based systems

- Create precise prompts to improve test generation and synthetic data quality

- Evaluate model responses for factual accuracy, tone, context alignment, and reliability

Test Automation & AI Tooling :

- Build and maintain AI testing automation frameworks using Python or JavaScript

- Develop self-healing automation solutions for AI workflows

- Automate API and model testing using PyTest, Playwright, Cypress, JUnit, and REST API frameworks

- Integrate AI testing into CI/CD pipelines and DevOps workflows

- Track quality metrics, defect trends, model performance benchmarks, and evaluation reports

AI Frameworks & Data Validation :

- Work with Agentic AI frameworks such as LangChain, LangGraph, RAG pipelines, and Vector Databases

- Validate training datasets and input data for completeness, balance, and bias reduction

- Ensure AI model compliance with governance, security, and enterprise standards

- Monitor AI systems using observability tools such as Datadog

Collaboration & Documentation :

- Collaborate with Data Scientists, AI Researchers, Product Teams, and Developers throughout the SDLC lifecycle

- Prepare detailed test plans, evaluation reports, defect summaries, and documentation

- Contribute to AI QA best practices, governance standards, and quality frameworks

- Participate in Agile ceremonies and release activities

Mandatory Skills :

- AI Testing AI/ML Model Testing

- Generative AI LLM Testing & Prompt Engineering

- Automation Playwright, PyTest, Cypress, JUnit

- API Testing REST API Testing

- AI Frameworks LangChain, LangGraph

- AI Architecture RAG Pipelines, Vector Databases

- Programming Python or JavaScript

- AI Evaluation Tools Promptfoo, DeepEval, OpenAI Evals

- DevOps CI/CD, GitHub Actions

- Cloud & Infrastructure Azure, Kubernetes

- Methodologies Agile, SDLC

- Monitoring Datadog

- Tools Git, Jira

- Testing Expertise Functional, Regression & Automation Testing

Preferred Skills :

- GCP Exposure

- SQL Knowledge

- AI Observability & Monitoring

- Banking / Financial Services Domain

- AI Governance & Compliance

- UiPath AI Automation

- Performance & Security Testing

- AI Ethics & Bias Testing

Qualification :

- B.E. / B.Tech in Computer Science, IT, Data Science, or related discipline

- MCA / M.Tech or equivalent qualification preferred

- 15 years of full-time education required

- AI Testing Certifications will be an added advantage