QA Automation Engineer - AI & Application Testing

Experience : 3 + Years

Location : Remote

Employment Type : Full-time

About LegacyLeap :

LegacyLeap is a fast-growing technology startup building AI-powered solutions for enterprise application modernization.

We're working at the intersection of GenAI, Agentic AI, data intelligence, and software engineering to solve complex real-world problems around legacy enterprise systems.

This is an opportunity to join an early-stage, engineering-driven team where you won't just execute predefined tasks - you'll have the opportunity to build, experiment, challenge existing approaches, and directly influence the quality of the product.

If you enjoy working on challenging engineering problems and want to grow with a startup that is building something meaningful with AI, we'd love to hear from you.

About the Role :

We are looking for a QA Automation Engineer with strong core QA fundamentals, hands-on coding skills in Python or Java, and experience in automation, API testing and CI/CD.

The ideal candidate has exposure to AI/GenAI/LLM or Agentic AI application testing and brings a strong QA mindset - someone who naturally looks for gaps, edge cases and opportunities to improve the product and testing process.

What You'll Do :

- Design and develop scalable test automation frameworks for web, mobile, API and modern applications.

- Perform functional, integration, regression and end-to-end testing.

- Build automation using Python or Java and frameworks such as PyTest, Playwright, Selenium, TestNG or similar.

- Integrate automated testing into CI/CD pipelines.

- Test AI/GenAI/LLM-powered applications and Agentic AI workflows.

- Evaluate AI outputs for correctness, relevance, consistency and quality.

- Identify AI-specific risks including hallucinations, prompt injection, jailbreaks and unexpected behaviour.

- Perform API and backend/data validation.

- Work closely with Product and Engineering teams to identify quality risks early.

- Continuously improve test coverage, automation, processes and overall product quality.

- Quickly adapt to new technologies, products and business domains.

Must-Have Skills :

- 3 + years of QA / Software Testing / Automation experience

- Strong understanding of Core QA concepts

- Good programming skills in Python or Java

- Strong problem-solving and logical reasoning skills

- Hands-on automation experience with PyTest, Playwright, Selenium, TestNG or similar

- API testing/automation experience

- CI/CD experience

- SQL/backend validation

- Experience with AI/GenAI/LLM application testing is highly preferred

- Experience with AI evaluation / LLM evaluation / RAG / AI Agent testing is a strong plus

Good to Have :

- Mobile application testing

- Linux / Unix and shell scripting

- RAGAS, DeepEval, LangSmith or similar AI evaluation tools

- Conversational AI / chatbot testing

- Agentic AI testing

- Experience across multiple projects, products or domains

- Product-based company experience

Who Will Thrive Here?

Someone who doesn't simply ask: "What should I test?" - but asks: "What could go wrong? What are we missing? How can we make this better?"

We value strong engineering fundamentals, curiosity, ownership, adaptability and a continuous-improvement mindset.

Why Join LegacyLeap?

- Growing startup - Be part of the journey from an early stage.

- Work with AI - Get hands-on exposure to GenAI, Agentic AI and intelligent enterprise systems.

- Build, don't just maintain - Your work will directly shape products and engineering practices.

- High ownership - Opportunity to take responsibility and make an impact.

- Learn & grow - Work alongside experienced engineers on challenging real-world problems.

- Remote - Work with a distributed, collaborative team.

If you're excited about QA, automation and the future of AI-powered software, come build it with us.