Job Title : Gen AI Test Consultant

Location : Pune

Experience Required : 10+ Years

Job Summary :

We are seeking an experienced and visionary Gen AI Test Consultant to drive the design, development, and implementation of advanced test automation frameworks for our GenAI-driven products and platforms. The ideal candidate will possess deep expertise in software testing, automation strategies, and AI/ML/GenAI technologies, with a proven track record of leading high-performing QA teams in complex, fast-paced environments. The candidate should understand the STLC landscape of the client and should be able to prescribe GenAI based solutions to fast track the clients test deliverables.

Key Responsibilities :

Leadership & Strategy :

- Provide GenAI based solutions to the clients STLC issues.

- Lead and mentor a team of GenAI test automation engineers, fostering a culture of innovation, collaboration, and continuous improvement.

- Define and execute the overall AI/ML/GenAI test automation strategy for AI/ML/GenAI products, ensuring alignment with business objectives and engineering best practices.

- Collaborate with client and account stakeholders, product managers, data scientists, developers, and DevOps teams to ensure high-quality releases.

Test Automation Development :

- Architecture, Design, develop, and maintain robust and reusable test automation frameworks in any of the following : Java-Selenium, Python-Selenium, Java-Appium, JavaScript -Cypress.

- Develop, execute, and maintain automated test scripts for functional, regression, integration, and performance testing.

- Integrate automated tests into CI/CD pipelines to support continuous testing and delivery.

GenAI Testing :

- Design GenAI/Agentic AI based solutions, can involve multiple tools, agents, MCPs.

- Understand business requirements/user stories and create test deliverables using GenAI.

- Create reusable prompt libraries utilizing different LLMs GPT, Mistral, Llama, Claude, Gemini.

- Understanding RAG (Retrieval-Augmented Generation) based architecture and test GenAI based RAG applications.

- Expertise on performing LLM validation using frameworks like EVALS, RAGAS for correctness, hallucination detection, reliability, and performance benchmarking.

Quality Assurance & Process Improvement :

- Analyze test results, identify defects, and collaborate with development teams for resolution.

- Continuously improve test automation processes, tools, and methodologies to enhance test coverage and efficiency.

- Contribute to the definition and tracking of key quality metrics and provide insights to stakeholders.

Coding & Technology :

- Should be able to script in Java/Selenium/Python/Javascript/C#.

- Should be able to write code in Python and worked with Python libraries for AI/ML (pandas, numpy, scikitlearn, streamlit, etc)

Documentation & Reporting :

- Prepare and maintain detailed test plans, test cases, and test documentation.

- Report test progress, results, and risks to project stakeholders in a clear and timely manner.

Stakeholder Management :

- Communicate test plans, progress, risks, and results to technical and non-technical stakeholders.

- Provide technical guidance and training to QA and development teams on AI/ML/GenAI testing best practices.

Good to Have :

- Exposure to responsible AI / model safety practices.

- Experience with data annotation or evaluation dataset creation.

Required Skills & Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, or a related field.

- 10+ years of experience in software testing and automation, with at least 3 years in a leadership role.

- Strong experience in designing and implementing test automation frameworks using tools such as Selenium, Appium, Cypress, TestNG, JUnit, or similar. Should have good experience on BDD, Cucumber.

- Experience on GenAI in Cloud hyperscalers AWS Bedrock, Google Vertex AI, Azure OpenAI.

- Experience in testing RAG based GenAI applications. Should have worked with any of the Vector Databases Pinecone, ChromaDB, FAISS, etc.

- Proficiency in programming languages such as Python/Java/JavaScript/C#.

- Good understanding of SDLC, STLC, agile methodologies, and DevOps practices.

- Strong analytical, problem-solving, and communication skills.

Preferred Qualifications :

- Relevant certifications (ISTQB, AI/ML, cloud technologies).

- Experience working in regulated industries (finance, healthcare, etc.).

- Contributions to open-source test automation or AI/GenAI communities.