Role Overview :

We are looking for an LLM Trainer Code Security to review and evaluate how AI coding agents behave on two closely related surfaces : software-engineering trajectories (an agent reading, writing, debugging and patching real code) and security trajectories (an agent finding, analysing, exploiting or fixing real vulnerabilities).

This is a high-judgment review role. You will read what an agent actually did across a multi-turn run, decide whether the outcome was genuinely correct and genuinely secure, and write up the reasoning so a researcher can act on it directly. Your verdicts become training signal and safety evidence.

Agents routinely reach the right-looking answer for the wrong reason patching a symptom, editing the test harness, overfitting to a proof-of-concept, or disabling the feature that carried the bug. Catching that requires someone who can analyse the vulnerability.

Key Responsibilities :

1. Review software-engineering trajectories :

- Read multi-turn agent runs on real repositories and assess whether the reasoning, tool use, and final patch were sound.

- Evaluate the diff itself : does it fix the actual root cause, is it complete, does it introduce regressions or side effects.

- Identify where an agent reached a passing result illegitimately modifying tests or the harness, special-casing the checked input, or removing the failing behavior rather than fixing it.

- Label failure modes precisely : incorrect reasoning, incomplete execution, silent scope creep, unsafe file or command operations.

2. Review security trajectories :

- Assess vulnerability-discovery runs did the agent locate the true vulnerable code path, or a plausible-looking but incorrect one.

- Judge fix quality against the vulnerability, not against the reference patch : would a different, correctly-written fix also be accepted? Flag verifiers and graders that only accept one implementation.

- Check that a recorded pass is real rather than an artifact proof-of-concept overfitting, a suppressed rather than resolved crash, or a fix applied outside the intended source.

- Reason about common vulnerability classes (OWASP Top 10, CWE/SANS Top 25) well enough to tell a genuine remediation from a superficial one.

3. Fix what you find :

- Repair defective tasks rather than only rejecting them correct an accept-set pointing at the wrong location, tighten one that admits spurious passes, or re-author a verifier coupled to a single implementation.

- Write or amend security regression tests so they genuinely fail on the vulnerable build and pass on the fixed one.

- Know the line between what you should repair yourself and what has to go back upstream and never quietly edit the ground truth to make a task pass.

- Turn a recurring defect into a rule others can apply, so the same class stops arriving.

4. Deliver usable verdicts :

- Produce consistent, well-evidenced evaluations that researchers can use directly for training data and safety assessment.

- Cite the specific line, command, or episode that justifies the call never a general impression.

- Surface systematic problems in tasks, graders and rubrics, not just per-task verdicts.

Required Experience :

- 13 years hands-on in one or more of : Application or product security, VAPT, secure code review, or vulnerability research; Software engineering with real code-review responsibility; LLM evaluation, annotation, or red teaming for coding or tool-calling agents.

- Strong security fundamentals and demonstrable hands-on work matter more here than years or pedigree.

- You can read code and judge a patch. Comfortable in at least one of Python, JavaScript/TypeScript, Go, Java, C/C++, Ruby or PHP, and able to reason about a diff in a language you don't write daily.

- Working knowledge of vulnerability classes injection, path traversal, deserialization, memory safety, authentication and access-control flaws, and how each is properly fixed.

- Able to read long agent trajectories 10+ turn reasoning and tool-call chains without losing the thread.

- Able to fix, not just flag. You can write or repair a test that fails on the vulnerable build and passes on the fixed one, and correct a broken verifier or accept-set without weakening what it checks.

- Clear written reasoning. Your annotations are the deliverable; they must stand up to a researcher disagreeing with them.

- Comfortable with Git, containers/Docker, reading test output and stack traces, and structured labeling formats (JSON/JSONL).