Core Responsibilities :

- Engineering the "Ground Truth" : Curate and maintain "Golden Datasets" definitive sets of prompts and ideal responsesthat serve as the benchmark for every model update we ship.

- Building the Eval Pipeline : Implement automated evaluation frameworks using tools like Ragas or DeepEval to quantify performance metrics such as Faithfulness, Relevancy, and Context Precision.

- Red Teaming & Jailbreaking : Proactively act as an adversary to bypass guardrails, inject malicious prompts, and force hallucinations to ensure our safety layers are bulletproof.

- RAG Debugging : Diagnose the "Retrieval" side of our systems. You will determine if a bad answer was caused by the vector database fetching the wrong data or the LLM failing to interpret the right data.

- LLM-as-a-Judge : Design and tune "Judge" models (using high-reasoning models like GPT-4o or Claude 3.5) to programmatically grade the outputs of our production models.

- Guardrail Governance : Implement and test real-time guardrails to intercept PII leakage, toxic content, or off-topic conversations before they reach the user.

Technical Skills & Qualifications :

- Python Native : Deep proficiency in Python. You should be comfortable building test suites that interact with asynchronous AI APIs and handle complex data structures.

- AI Orchestration : Working knowledge of LangChain, LangGraph, or Haystack. You understand how "chains" and "agents" behave and, more importantly, how they fail.

- The Eval Stack : Hands-on experience with (or a strong desire to master) LangSmith for tracing, debugging, and evaluating LLM workflows.

- Statistical Thinking : You understand that "accuracy" in GenAI is a moving target. You are comfortable using semantic similarity, cosine distance, and BLEU/ROUGE scores, while understanding their specific limitations.

- Data Driven : Experience working with vector databases and an understanding of how embeddings work.