Qentelli - AI/LLM Quality Assurance Engineer

About the Role:

We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands-on experience in AI/ML or LLM-based applications. You will be responsible for validating the performance, accuracy, and robustness of next-generation AI systems, including orchestration-driven workflows and autonomous agents.

Key Responsibilities:

- Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.

- Perform LLM evaluation using metrics such as:

1. Exact match / Soft match

2. BLEU / ROUGE

3. BERTScore

4. Semantic similarity using embeddings

- Develop and maintain prompt testing frameworks and validate output consistency across use cases.

- Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).

- Build automated test pipelines for LLM-driven workflows and orchestration frameworks.

- Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.

- Identify defects, performance issues, and edge cases in AI systems.

- Integrate testing processes into CI/CD pipelines.

Mandatory Skills Core AI/LLM Testing:

- Hands-on experience in Agentic & Multi-Agent Testing

- Expertise in LLM evaluation frameworks and methodologies

- Strong understanding of:

1. Prompt engineering validation

2. Guardrails and safety mechanisms

3. Output evaluation using embeddings and NLP metrics

Software QA Experience:

- 3+ years in Software QA/Testing

- Minimum 2+ years of experience in AI/ML or LLM-based systems

Programming:

- Proficiency in Python OR TypeScript/JavaScript

Orchestration Frameworks - Experience with at least one:

- LangChain / LangGraph

- LlamaIndex

- DSPy

- OpenAI Assistants / Actions

- Or similar agentic orchestration frameworks

Tools & Platforms:

- CI/CD: GitLab, Jenkins

- Monitoring: Grafana

- Test Management & Tracking: Jira, X-Ray

- Exposure to AI Quality Engineering practices