Predigle - Senior Data Scientist - Generative AI

Job Title : Senior Data Scientist / Data Analyst GenAI & ML Evaluation

Experience : 5+ Years

About the Role :

We are looking for highly analytical and technically skilled Data Scientists and Data Analysts to lead the evaluation of cutting-edge Generative AI (GenAI) and Machine Learning systems. You will play a crucial role in validating, experimenting, and continuously improving our AI

capabilities through rigorous testing frameworks.

What Youll Do :

- Design and execute A/B testing and experimentation strategies for ML and GenAI features

- Develop evaluation frameworks using both automated (LLM as a judge/jury) and manual

approaches

- Analyze outputs of LLMs and provide insights into model behavior, accuracy, and reliability

- Use traditional ML techniques and statistical methods to validate hypotheses

- Collaborate with product, engineering, and research teams to assess system improvements

- Build dashboards and reports to communicate results to technical and non-technical

stakeholders

What Were Looking For :

- Hands-on experience with A/B testing, experimental design, and ML model evaluation

- Proficient in Python, SQL, and data visualization tools (Tableau, Power BI, etc.)

- Exposure to LLMs and GenAI frameworks (e.g., OpenAI, Hugging Face, LangChain)

- Ability to implement and analyze both quantitative and qualitative evaluation metrics

- Strong communication skills and an experimental, outcome-driven mindset

Good to Have :

- Prior work in GenAI system testing, user simulations, or annotation processes

- Familiarity with prompt engineering, retrieval-augmented generation (RAG), or vector search