Description :

Job Title : AI Tester / AI QA Engineer.

Location : Hyderabad.

Job Description :

We are seeking an experienced AI Tester / AI QA Engineer to lead quality assurance efforts for AI-driven applications, including GenAI, LLM-based systems, chatbots, and cloud-native platforms.

The ideal candidate will combine strong QA fundamentals, automation expertise, and hands-on AI/LLM testing experience to ensure reliability, fairness, scalability, and production readiness of intelligent systems.

This role plays a key part in validating AI behaviour, model outputs, data pipelines, APIs, and UI workflows, while driving AI-assisted test automation and governance.

Key Responsibilities :

AI & GenAI Testing :

- Test LLM-based applications including chatbots, conversational agents, and AI-powered workflows.

- Validate prompt-response accuracy, context retention, multi-turn conversations, and fallback handling.

- Perform hallucination testing, response consistency checks, bias & fairness validation, and confidence threshold testing.

- Design and execute AI model evaluation frameworks covering accuracy, precision/recall (where applicable), latency, safety, and reliability.

- Validate RAG (Retrieval-Augmented Generation) pipelines, including data retrieval accuracy and grounding quality.

AI-Driven Test Automation :

- Build and maintain AI-powered test generation frameworks using tools such as AWS Bedrock,

OpenAI, Azure OpenAI, LangChain, and Hugging Face.

- Integrate AI-assisted coding tools (GitHub Copilot, Microsoft Copilot) to accelerate test development and improve coverage.

- Automate AI test execution with Python, PyTest, Selenium, Playwright, Cypress, and CI/CD pipelines.

- Implement human-in-the-loop validation for AI-generated test cases.

Automation, API & UI Testing :

- Develop and maintain robust test automation frameworks using Selenium, Playwright, Cypress, Appium, and WebDriverIO.

- Perform API testing using Postman, Rest Assured, Swagger/OpenAPI, and schema validation.

- Validate modern SPA and cloud-native UIs with role-based workflows and responsive behaviors.

- Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI/CD, Azure DevOps).

Data, Cloud & Pipeline Validation :

- Test event-driven data pipelines on AWS and Azure (S3, Lambda, Glue, EMR, Kinesis, DynamoDB, Athena).

- Validate Big Data and streaming systems (Hadoop, Spark, Kafka) for data completeness, consistency, and accuracy.

- Perform ETL and data quality testing using SQL, reconciliation checks, and profiling tools.

- Validate microservices deployed on AWS EKS and serverless architectures.

Observability, Performance & Governance :

- Monitor AI and application behavior using CloudWatch, Grafana, ELK/OpenSearch, Splunk, AppDynamics.

- Execute performance testing for AI services, APIs, and streaming workloads using JMeter.

- Build dashboards tracking AI KPIs such as response accuracy, hallucination rate, latency, and user satisfaction.

- Contribute to Responsible AI governance, documentation, and QA best practices.

Leadership & Collaboration :

- Lead QA strategy, planning, and execution across Agile/Scrum and SAFe teams.

- Mentor QA engineers and support adoption of AI testing practices.

- Collaborate with Product, Engineering, Data Science, and DevOps teams to align quality with business goals.

Required Skills & Qualifications :

Technical Skills :

- 8+ years of experience in Software QA / Automation, with hands-on AI/GenAI testing experience.

- Strong programming skills in Python, Java, or JavaScript.

- Experience testing LLMs, chatbots, conversational AI, and AI-driven applications.

- Hands-on with AWS Bedrock, OpenAI, Azure OpenAI, Hugging Face, LangChain, RAG.

- Expertise in test automation frameworks (Selenium, Playwright, Cypress, PyTest).

- Strong API testing and SQL/data validation skills.

- Experience with CI/CD pipelines, Docker, and cloud platforms (AWS/Azure).

Nice to Have :

- Big Data & streaming validation (Hadoop, Spark, Kafka).

- Performance engineering with JMeter.

- Observability and monitoring experience.

- Exposure to AI governance, fairness testing, and compliance.