Description:

We are looking for an experienced QA AI Architect to design and lead the architecture of AI-driven Quality Engineering systems for next-generation AI-powered products. This role will focus on building scalable frameworks for testing and validating AI systems built on Large Language Models (LLMs), embeddings, vector databases, and modern AI pipelines. The architect will work closely with product, engineering, and AI teams to design intelligent QA solutions including AI-assisted test generation, intelligent automation, document understanding, and AI-driven validation systems. The ideal candidate combines deep QA architecture experience with strong AI/ML system knowledge, and is passionate about redefining quality engineering for AI products.

Responsibilities:

AI Architecture & LLM Systems:

- Architect and build AI-powered quality engineering platforms using LLMs and SLMs.

- Evaluate and integrate open-source models such as Llama and Mistral.

- Design prompt engineering workflows and Retrieval-Augmented Generation (RAG) pipelines.

- Architect systems for LLM evaluation, guardrails, and validation frameworks.

- Optimize architectures for token efficiency, cost, and performance.

Embeddings & Vector Search:

- Design and implement embedding pipelines for semantic retrieval.

- Implement vectorization strategies for documents, test artifacts, and product knowledge bases.

- Architect vector storage using technologies such as:

1. Pinecone

2. Weaviate

3. Milvus

4. FAISS

- Optimize chunking and indexing strategies for large knowledge repositories.

Model Optimization & Fine-Tuning:

- Implement model fine-tuning pipelines using LoRA / PEFT techniques.

- Evaluate model performance, reliability, and hallucination mitigation.

- Manage training datasets and model evaluation metrics.

Backend & System Architecture:

- Design microservice-based AI system architectures.

- Build scalable backend services and APIs for model inference and evaluation.

- Develop data pipelines supporting AI workflows and large-scale inference systems.

MLOps & AI Lifecycle:

- Build MLOps pipelines for model deployment, monitoring, and retraining.

- Implement CI/CD pipelines for AI systems.

- Enable model observability, performance monitoring, and version control.

OCR & Document Intelligence:

- Design systems for OCR-based document processing.

- Integrate OCR pipelines with embedding and AI workflows.

- Build solutions for document understanding and QA automation.

Required Skills:

AI & LLM Systems:

- Strong understanding of LLM and SLM architectures.

- Experience working with open-source LLMs.

- Prompt engineering and RAG architectures.

- Model evaluation frameworks and hallucination control.

- Token optimization strategies.

Embeddings & Vector Databases:

- Experience working with embedding models and semantic search.

- Hands-on experience with vector databases such as:

1. FAISS

2. Pinecone

3. Weaviate

4. Milvus

- Infuse AI with:

1. Automated Test Failure analysis.

2. Crawl Product/Test code to figure out whether issue is probable Product/Test/Env issue.

Programming & AI Frameworks:

- Strong Python programming skills.

- Experience with AI frameworks such as:

1. LangChain

2. LlamaIndex

- Backend development using FastAPI / Flask / Node.js.

Backend & Cloud Architecture:

- Microservices architecture and REST APIs.

- Containerization using Docker.

- Experience working on AWS / Azure / GCP.

MLOps:

- Model deployment pipelines.

- CI/CD for machine learning systems.

- Monitoring and observability of AI models.

OCR & Document Processing:

- Experience with OCR frameworks such as:

1. Tesseract OCR

2. PaddleOCR

System Foundations:

- Strong understanding of system design, operating systems, networking, and distributed systems.

Qualifications:

Nice to Have:

- Experience building AI agents or autonomous systems.

- Architect testing frameworks for agentic AI workflows and multi-agent orchestration.

- Knowledge of LLM evaluation frameworks.

- Understanding of AI safety and guardrails.

- Experience architecting AI-powered SaaS products.

What Makes This Role Exciting:

You will play a key role in shaping the future of AI-powered Quality Engineering, building intelligent platforms that enable automated test generation, AI-driven validation, and scalable QA systems for AI products.