Description :

- Design and execute comprehensive test strategies specifically for AI/ML models, LLM-based applications, and data pipelines

- Develop automated test frameworks for model validation, regression testing, and performance benchmarking

- Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs and use cases

- Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-driven features

- Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds for AI systems

- Build and maintain evaluation datasets, ground truth sets, and adversarial test cases for comprehensive model validation

- Monitor models in production for drift, degradation, and anomalous behavior; implement monitoring solutions as needed

- Validate data quality, data pipelines, and feature stores that feed AI systems to ensure data integrity

- Document defects, edge cases, and failure patterns specific to AI behavior with actionable insights

- Ensure AI systems meet ethical, fairness, and compliance standards through bias audits and explainability checks

Required Skills & Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field

- 3 to 6 years of professional QA experience, with at least 1 - 2 years in AI/ML quality assurance

- Strong proficiency in Python for test automation and data analysis

- Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)

- Hands-on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms

- Solid understanding of the ML lifecycle training, validation, deployment, and monitoring phases

- Knowledge of data quality tools and pipeline testing (e.g., Great Expectations, dbt tests)

- Strong analytical and inquisitive mindset with the ability to challenge model outputs critically

- Excellent documentation and communication skills with the ability to articulate complex technical concepts

- Collaborative approach and ability to work effectively with data science, engineering, and product teams