Job Description :

We are looking for an experienced Principal AI/ML Engineer to join the Media Products development team, responsible for building and maintaining products in audio/video editing and media workflow management.

The role focuses on developing and evaluating LLM-powered AI agents, ensuring their performance, reliability, and observability in production environments.

Total Experience : 9 - 12 years

Key Responsibilities :

- Design and implement evaluation frameworks to measure the performance, quality, and reliability of LLM-powered applications and AI agents.

- Define metrics and strategies to monitor model and agent behavior across the platform.

- Develop and optimize AI agent architectures, prompts, and tool/function-calling workflows.

- Implement tracing, monitoring, and observability solutions using OpenTelemetry, Logfire, or similar tools.

- Build and maintain AI/LLM evaluation pipelines to assess system performance in production.

- Analyze model performance, identify issues, and improve the quality and reliability of AI-powered solutions.

- Optimize model selection and usage by balancing performance, quality, and cost at scale.

- Collaborate with cross-functional teams to integrate AI capabilities into media editing and workflow management products.

Required Skills :

- Strong hands-on experience with LLMs and AI/ML systems.

- Proficiency in Python for AI development and implementation.

- Proven experience evaluating and monitoring LLM-based applications in production.

- Strong understanding of AI agent architectures, prompt engineering, and tool/function calling.

- Hands-on experience with OpenTelemetry, Logfire, or similar tracing and observability tools.

- Experience designing evaluation methodologies and metrics for AI/LLM systems.

- Ability to optimize AI solutions for quality, reliability, and cost efficiency at scale.