Job Title : QA Engineer - AI Initiatives

Location : India - Bangalore, Mumbai

Job Type : Full-time

Experience Required : 3 - 6 Years

Role Overview :

EisnerAmper is seeking a talented QA Engineer | AI Initiatives to join our growing team. In this role, you will be responsible for ensuring the quality, reliability, fairness, and performance of AI/ML-powered products and systems. Unlike traditional QA, this position requires a deep understanding of non-deterministic model behavior, data quality, and AI-specific failure modes such as hallucinations, bias, and model drift. You will be at the forefront of AI quality assurance, collaborating with data scientists, ML engineers, and product teams to deliver robust, ethical, and high-performing AI solutions.

Key Responsibilities :

- Design and execute comprehensive test strategies specifically for AI/ML models, LLM-based applications, and data pipelines

- Develop automated test frameworks for model validation, regression testing, and performance benchmarking

- Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs and use cases

- Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-driven features

- Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds for AI systems

- Build and maintain evaluation datasets, ground truth sets, and adversarial test cases for comprehensive model validation

- Monitor models in production for drift, degradation, and anomalous behavior; implement monitoring solutions as needed

- Validate data quality, data pipelines, and feature stores that feed AI systems to ensure data integrity

- Document defects, edge cases, and failure patterns specific to AI behavior with actionable insights

- Ensure AI systems meet ethical, fairness, and compliance standards through bias audits and explainability checks

Required Skills & Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field

- 3 - 6 years of professional QA experience, with at least 1 - 2 years in AI/ML quality assurance

- Strong proficiency in Python for test automation and data analysis

- Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)

- Hands-on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms

- Solid understanding of the ML lifecycle - training, validation, deployment, and monitoring phases

- Knowledge of data quality tools and pipeline testing (e.g., Great Expectations, dbt tests)

- Strong analytical and inquisitive mindset with the ability to challenge model outputs critically

- Excellent documentation and communication skills with the ability to articulate complex technical concepts

- Collaborative approach and ability to work effectively with data science, engineering, and product teams

Nice to Have :

- Experience with prompt engineering and red-teaming LLMs

- Familiarity with MLOps platforms such as MLflow, SageMaker, or Vertex AI

- Knowledge of vector databases and embedding quality evaluation

- Understanding of AI safety, responsible AI principles, and fairness frameworks

- Experience with A/B testing and shadow deployment strategies

- Knowledge of CI/CD pipelines and DevOps practices in ML environments

Testing Area What QA Validates :

- Model Accuracy Output correctness against ground truth and expected outcomes

- Hallucination Testing Factual consistency and source grounding in generated content

- Bias & Fairness Equitable outputs across demographics and user segments

- Robustness Behavior under adversarial or edge-case inputs and scenarios

- Latency & Throughput Response times and performance under production load conditions

- Drift Detection Model performance degradation and changes over time in production

- Data Pipeline QA Completeness and accuracy of training and inference data

- Prompt Regression Consistency of outputs after prompt or model updates

Soft Skills & Competencies :

- Analytical and Inquisitive Mindset : Comfortable challenging model outputs and thinking critically about AI behavior

- Red-Team Thinking : Ability to think like both a user and an adversary to identify vulnerabilities and edge cases

- Attention to Detail : High attention to detail with a quality-first attitude and commitment to excellence

- Communication : Strong documentation and communication skills with the ability to present findings to both technical and non-technical stakeholders

- Collaboration : Collaborative approach and ability to work effectively across cross-functional teams

- Problem-Solving : Creative problem-solving skills with the ability to develop innovative testing solutions

What We Offer :

- Opportunity to work on cutting-edge AI and machine learning initiatives

- Collaborative environment with talented data scientists, engineers, and product teams

- Professional development and continuous learning opportunities in the AI/ML space

- Competitive compensation and comprehensive benefits package

- Flexible work arrangements and a supportive company culture