Spektra Systems - Senior AI Quality Assurance Engineer - Automation Testing

Description :

- AI-Specific Testing & Validation

- Design and execute AI-specific test strategies, including :

a. Prompt robustness testing

b. Hallucination detection

c. Output consistency checks

d. Edge-case and adversarial testing

- Validate RAG pipelines :

a. Retrieval accuracy

b. Context relevance

c. Response grounding

- Test multi-agent workflows and tool integrations

- Automation & Tooling :

a. Build automated AI test harnesses

b. Create evaluation pipelines for AI responses

c. Define regression tests for prompt and agent changes

d. Integrate AI testing into CI/CD pipelines

- Collaboration & Quality Ownership :

a. Work closely with AI developers during design, not just after implementation

b. Help define acceptance criteria for AI features

c. Monitor AI behavior in production and flag drift or degradation

d. Document AI failure modes and mitigations

Requirements :

Must Have :

- Strong API testing and automation skills

- Understanding of AI/LLM behavior and limitations

- Experience testing non-deterministic systems

- Strong analytical and problem-solving mindset

Nice to Have :

- Experience testing AI agents or RAG systems

- Experience writing Python-based test automation

- Familiarity with LLM APIs and prompt engineering

- Experience with observability tools for AI systems