Title : Sr. SW Engineer C++ (LLM Evaluation & Repository Validation)

Experience : 3 - 8 yrs

Location : remote

Employment type : Contractual

Must Have :

Software development

C++

Roles and resposibilities :

1. LLM evaluation and code analysis

Analyzing and triaging GitHub issues :

- Reviewing and prioritizing issues related to popular open-source libraries to understand LLM performance in addressing them.

Assessing LLM performance in code-related tasks :

- Modifying and running existing codebases to evaluate how LLMs perform in bug-fixing scenarios and code generation tasks.

Evaluating unit test coverage and quality :

- Assessing the quality and coverage of unit tests within repositories to inform LLM training and refinement.

Collaborating with researchers to design challenging LLM tasks :

- Working with AI researchers to design and identify open-source repositories and issues that are particularly difficult for LLMs to handle.

Reviewing and validating AI-generated code/solutions :

- Ensuring LLM-generated code adheres to quality standards and best practices.

2. Repository validation and development environment management

Setting up and configuring code repositories :

- Including Dockerization and environment setup to create consistent and reproducible evaluation environments.

Developing and maintaining software solutions in C++ :

- Designing, developing, and enhancing software systems related to the evaluation and validation process.

Automating development environments :

- Streamlining the setup and configuration of development environments for efficient testing and evaluation.

Maintaining and improving existing software :

- Focusing on performance optimization and reliability for the evaluation infrastructure.

3. Collaboration, leadership, and continuous improvement

Collaborating with cross-functional teams :

- Working closely with researchers, product managers, and other engineers to define, design, and implement evaluation strategies.

Leading and mentoring junior engineers :

- Guiding and supporting the growth of less experienced team members.

Conducting code reviews and providing feedback :

- Ensuring adherence to coding standards and best practices within the team.

Participating in the product development lifecycle :

- Actively engaging in the various stages of developing and deploying AI-assisted software solutions.

Staying updated with AI/LLM advancements :

- Continuously learning about new research, tools, and best practices related to LLM evaluation and software development.