Uniphore - AI Quality Manager

Job Description :

We are looking for an experienced AI Quality Manager to lead the evaluation, monitoring, and continuous improvement of our AI and machine learning systems. In this role you will define quality standards for AI outputs, build robust testing and benchmarking frameworks, partner cross-functionally with engineering and product teams, and ensure our models meet both internal performance benchmarks and external regulatory requirements.

This is a highly cross-functional leadership role sitting at the intersection of AI safety, product quality, and operational excellence.

Key Responsibilities :

- Define and own quality metrics and acceptance criteria for AI systems across all product lines

- Establish and enforce AI quality policies aligned with internal standards and emerging regulatory frameworks

- Maintain a living AI quality handbook accessible to engineering, product, and legal stakeholders

Quality standards & governance :

- Define and own quality metrics and acceptance criteria for AI systems across all product lines

- Establish and enforce AI quality policies aligned with internal standards and emerging regulatory frameworks

- Maintain a living AI quality handbook accessible to engineering, product, and legal stakeholders

Testing & evaluation :

- Design and execute structured evaluation pipelines for LLM and ML model releases, including regression suites, edge case testing, and output quality audits

- Build automated monitoring systems to detect model drift, hallucination rates, bias signals, and output degradation in production

- Own the incident response process for AI quality failures, from root-cause analysis to post-mortem and remediation

Cross-functional leadership :

- Partner with ML engineers, data scientists, and product managers to embed quality checkpoints into the model development lifecycle

- Lead a team of AI quality analysts and evaluation engineers, setting goals, conducting performance reviews, and mentoring career growth

- Present quality reports and risk assessments to senior leadership and external auditors

Data & tooling :

- Oversee the build-out and maintenance of internal evaluation datasets, annotation pipelines, and benchmarking infrastructure

- Evaluate and procure third-party testing tools, model cards, and audit services as needed

Required Qualifications :

- 7+ years of experience in quality assurance, with at least 3 years focused on AI/ML systems or data-driven products

- Deep understanding of ASR (WER, keyword hit rate) and LLM evaluation methods (RLHF, human preference evaluation, automated LLM-as-judge techniques)

- Hands-on experience with Python and ML frameworks sufficient to review model code and evaluation scripts

- Proven track record owning a quality program from inception through production including writing standards, building tooling, and managing teams

- Strong written and verbal communication skills; ability to translate complex AI risk concepts for non-technical stakeholders

- Experience with statistical analysis for model performance measurement (precision/recall, F1, BLEU, ROUGE, win-rate, etc.)

Preferred Qualifications :

- Experience designing structured human evaluation programs or preference studies for generative AI outputs

- Experience managing annotation vendors or running large-scale human evaluation programs

- Prior work in a high-growth AI product company or research lab

Role : QA Team Manager

Industry Type : Software Product

Department : Engineering - Software & QA

Employment Type : Full Time, Permanent

Role Category : Quality Assurance and Testing

Education : UG: B.Tech / B.E. in Artificial Intelligence And Data Science

Key Skills : ASR, LLM, Python