HamburgerMenu
hirist

Senior Engineer - AI Observability & LLM Evaluation

The Judge Group
5 - 9 Years
Multiple Locations

Posted on: 16/11/2026

Job Description

Role Overview :

We are seeking a Senior AI Observability & LLM Evaluation Engineer with hands-on experience validating and monitoring LLM applications, GenAI solutions, RAG systems, and AI agents in production environments. The ideal candidate will have strong expertise in Langfuse, LLM evaluation frameworks, prompt testing, and AI quality observability.


This role is responsible for ensuring the quality, reliability, performance, safety, and behavioral correctness of AI systems across the full lifecycle - from development through production - using structured testing and continuous observability.

Key Responsibilities :

AI Quality Engineering (Testing, Evaluation & Regression) :

- Design and execute end-to-end testing strategies for LLM applications, RAG systems, and AI agents.

- Build and maintain golden datasets, evaluation sets, and edge-case test scenarios covering hallucination, relevance, factuality, bias, toxicity, consistency, and instruction adherence.

- Perform regression testing across changes in prompts, models, system instructions, tools, retrieval configurations, and application logic.

- Compare model/prompt versions using quantitative evaluation metrics and baseline benchmarking.

- Validate and calibrate LLM-as-a-Judge evaluation systems for accuracy and consistency.

AI Observability & Production Monitoring (Langfuse) :

- Use Langfuse to implement end-to-end observability for LLM applications, including tracing, evaluation, and production monitoring.

- Analyze traces, spans, prompts, responses, latency, token usage, errors, and tool executions to assess system behavior.

- Define and maintain evaluation workflows, dashboards, metrics, and alerting thresholds in Langfuse or equivalent tools.

- Monitor production AI systems for quality degradation, hallucinations, performance drift, and behavioral anomalies.

- Establish structured processes for triaging and investigating AI quality issues in production.

AI Agent & Workflow Validation :

- Design testing strategies for multi-step AI agents and agentic workflows.

- Validate end-to-end agent behavior including tool selection, execution accuracy, reasoning paths, memory/context handling, and orchestration logic.

- Test agent performance under normal, edge-case, adversarial, and failure scenarios.

- Ensure agents consistently deliver correct and business-aligned outcomes with appropriate guardrails.

Production Quality Governance :

- Define and maintain AI quality metrics, baselines, thresholds, and alerting strategies across applications.

- Conduct trend analysis and anomaly detection to identify degradation in model or system performance.

- Support periodic AI system quality reviews and production health assessments.

Requirements :

- Hands-on experience with Langfuse or similar LLM observability platforms.

- Strong understanding of: LLM evaluation and benchmarking, LLM-as-a-Judge systems, Prompt and RAG evaluation, AI observability and production monitoring, Regression testing for AI systems, AI agent evaluation.

- Experience defining evaluation metrics, baselines, and quality thresholds.

- Strong skills in trace analysis, telemetry, and production debugging of AI systems.

- Experience with API testing tools (e.g., Postman) and/or Python automation frameworks.

- Strong SQL and data analysis capabilities.

- Experience integrating testing into CI/CD pipelines.

- Familiarity with cloud-based AI/ML platforms and LLM APIs.

Preferred Experience :

- Deep experience with Langfuse evaluation workflows, datasets, and production monitoring.

- Familiarity with evaluation frameworks such as RAGAS, DeepEval, Azure AI Evaluation, OpenAI Evals, or similar.

- Experience testing AI agents and multi-agent systems.

- Understanding of model drift, data drift, and AI performance degradation patterns.

- Experience building enterprise AI quality and observability frameworks.

- Knowledge of responsible AI, governance, privacy, and security standards.

- Experience in regulated or high-compliance environments.

The job is for:

May work from home
info-icon

Did you find something suspicious?

Similar jobs that you might be interested in

Loading chat...