Job Description :

- Location : Mumbai / Bangalore, India

- Experience Required : 36 Years

- Job Type : Full-time (14 Days/Month Work from Office)

- Work Timings : 12:30 PM 9:30 PM

- Interview Process : 3 Virtual Rounds + 1 In-Person Round

Must haves :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 3 - 6 years of professional QA experience, with at least 1 -2 years in AI/ML quality assurance.

- Strong proficiency in Python for test automation and data analysis.

- Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith).

- Hands-on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms.

- Solid understanding of the ML lifecycle- training, validation, deployment, and monitoring phases

Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, or a related field.

- 3 to 6 years of professional QA experience.

- Minimum 12 years of hands-on experience in AI/ML Quality Assurance.

- Strong proficiency in Python for test automation and data analysis.

- Familiarity with LLM evaluation frameworks such as RAGAS, DeepEval, Promptfoo, and LangSmith.

- Experience with testing tools including Pytest, Selenium, Postman, or similar platforms.

- Solid understanding of the Machine Learning lifecycle, including training, validation, deployment, and monitoring.

- Knowledge of data quality tools and pipeline testing frameworks such as Great Expectations and dbt tests.

Skills Required :

- Expertise in AI/ML model testing and validation.

- Strong understanding of LLM behavior, hallucination detection, bias evaluation, and model drift monitoring.

- Experience testing RAG (Retrieval-Augmented Generation) applications, chatbots, recommendation systems, and AI-powered solutions.

- Strong analytical and problem-solving abilities.

- Ability to create and maintain evaluation datasets, ground truth datasets, and adversarial test cases.

- Experience with data pipeline testing and data integrity validation.

- Excellent documentation and communication skills.

- Ability to collaborate effectively with Data Science, Engineering, and Product teams.

- Soft skills: Analytical & Inquisitive Mindset, Red-Team Thinking, Attention to Detail, Communication, Collaboration, Problem-Solving.

Role Overview :

We are looking for a skilled QA Engineer AI Initiatives to ensure the quality, reliability, fairness, and performance of AI/ML-powered products and systems. The ideal candidate should possess strong expertise in AI quality assurance, model evaluation, automation testing, and data validation.

This role provides an opportunity to work closely with Data Scientists, ML Engineers, and Product teams to deliver robust, ethical, and high-performing AI solutions while addressing AI-specific challenges such as hallucinations, bias, model drift, and output consistency.

Key Responsibilities :

- Design and execute comprehensive testing strategies for AI/ML models, LLM-based applications, and data pipelines.

- Develop and maintain automated testing frameworks for model validation, regression testing, and performance benchmarking.

- Evaluate AI model outputs for accuracy, consistency, relevance, hallucinations, and bias.

- Test RAG pipelines, chatbots, recommendation systems, and other AI-powered features.

- Collaborate with Data Scientists and ML Engineers to define quality benchmarks and acceptance criteria.

- Build and maintain evaluation datasets, ground truth datasets, and adversarial testing scenarios.

- Monitor production AI systems for model drift, performance degradation, and anomalous behavior.

- Validate data quality, feature stores, and data pipelines feeding AI systems.

- Document defects, edge cases, and AI-specific failure patterns with actionable recommendations.

- Conduct fairness assessments, bias audits, and explainability checks to ensure ethical AI deployment.

- Support continuous improvement of AI testing methodologies, tools, and frameworks.

Nice to have :

- Experience with prompt engineering and red-teaming LLMs

- Familiarity with MLOps platforms such as MLflow, SageMaker, or Vertex AI

- Knowledge of vector databases and embedding quality evaluation

- Understanding of AI safety, responsible AI principles, and fairness frameworks

- Experience with A/B testing and shadow deployment strategies

- Knowledge of CI/CD pipelines and DevOps practices in ML environments