Job Role : Test Engineer - AI

Experience : 6+ Years

Location : Bangalore

Job Summary :

We are looking for an experienced Test Engineer AI to design, develop, and execute comprehensive testing strategies for AI/ML-based applications and platforms. The ideal candidate should have strong expertise in software testing, test automation, AI/ML testing, API testing, and performance testing, with the ability to validate AI models and AI-powered applications for quality, reliability, accuracy, and scalability.

Key Responsibilities :

- Design and implement comprehensive test strategies, test plans, test scenarios, and test cases for AI/ML applications.

- Perform functional, regression, integration, system, API, and end-to-end testing.

- Develop and maintain automated test frameworks using Python and relevant automation tools.

- Test AI/ML models, LLM applications, RAG pipelines, chatbots, and AI-powered APIs.

- Validate model outputs for accuracy, consistency, relevance, bias, hallucinations, and robustness.

- Design testing approaches for LLM prompts, embeddings, vector databases, RAG workflows, and AI agents.

- Perform API testing using tools such as Postman, REST Assured, or PyTest.

- Integrate automated tests into CI/CD pipelines.

- Conduct performance, scalability, and reliability testing of AI applications and services.

- Identify defects, perform root-cause analysis, and work closely with developers, data scientists, and ML engineers to resolve issues.

- Develop test automation utilities, frameworks, and reusable components.

- Monitor test results, analyze failures, and provide detailed quality reports.

- Ensure testing aligns with security, reliability, data quality, and responsible AI requirements.

Required Skills :

- 6+ years of experience in Software Testing / Test Automation.

- Strong programming experience in Python.

- Hands-on experience with AI/ML testing or testing AI-powered applications.

- Strong knowledge of Selenium, Playwright, PyTest, or equivalent automation frameworks.

- Experience with API testing and RESTful services.

- Knowledge of LLMs, Generative AI, RAG, Prompt Engineering, Embeddings, and Vector Databases.

- Experience testing AI model responses, hallucinations, accuracy, relevance, and consistency.

- Strong understanding of SQL and database testing.

- Experience with Git, Docker, and CI/CD tools such as Jenkins, GitLab CI, or GitHub Actions.

- Familiarity with cloud platforms such as AWS, Azure, or GCP.

- Strong knowledge of SDLC, STLC, Agile/Scrum, defect lifecycle, and test methodologies.

Good to Have :

- Experience with LangChain, LlamaIndex, Hugging Face, OpenAI APIs, or similar AI frameworks.

- Knowledge of AI evaluation frameworks and metrics such as BLEU, ROUGE, Recall@K, Precision@K, MRR, and faithfulness.

- Experience with MLflow, Kubernetes, or cloud-based AI/ML platforms.

- Knowledge of AI security, prompt injection, data privacy, and responsible AI testing.

- Experience with performance testing tools such as JMeter or Locust.

Education :

- Bachelors/Masters degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies :

- Strong analytical and problem-solving skills.

- Excellent debugging and root-cause analysis abilities.

- Strong attention to detail.

- Good communication and collaboration skills.

- Ability to work independently in a fast-paced Agile environment.

- Strong understanding of quality engineering principles.