Role & Responsibilities :

You are the technical lead for a strategic project on LLM training. You own the complete projectfrom data mixture strategy to the final model evaluationensuring the project meets the client's domain-specific performance requirements within strict compute and data-privacy constraints.

Key Responsibilities :

- Lead the Training Roadmap : Define the stages of the LLM lifecycle : Continuous Pre-training (CPT) on domain-specific data, SFT, and alignment (DPO/PPO/RLHF).

- Architect Distributed Training Systems : Design the infrastructure for multi-node, multi-GPU training. You will make the call on using the right infrastructure.

- Data Mixture & Curriculum Design : Own the data strategy. You will decide the fine tuning between general-purpose data and domain-specific corpora to prevent catastrophic forgetting.

- Compute Budget & Optimization : Manage infrastructure optimize training efficiency (TFLOPS/GPU) and implement checkpointing strategies to ensure zero loss of progress during hardware failures.

- Evaluation Framework Design : Build a comprehensive, sovereign evaluation suite (beyond standard benchmarks) that measures the model's performance on the clients specific data.

- Direct Customer Delivery : Act as the primary technical point of contact for the customers team, translating their requirements into training approach and solution design.

Required Technical Skillset :

1. Distributed Training & Infrastructure :

- 3D Parallelism Mastery : Hands-on implementation of Data, Tensor, and Pipeline parallelism for models exceeding 70B parameters.

- Backend Optimization : Expert tuning to resolve bottlenecks in multi-node GPU clusters.

- Memory & Fault Tolerance : Mastery of ZeRO (Stages 13) and activation checkpointing; building automated, elastic training workflows for rapid failure recovery.

2. LLM Training Tricks & Stability :

- Convergence Engineering : Expertise in mitigating loss spikes, adaptive gradient clipping, and sophisticated LR schedules.

- Numerical Precision : Hands-on experience with Mixed Precision (BF16/FP16) and FP8 workflows, including dynamic loss scaling management.

- Advanced Optimization : Leveraging Grokking techniques and monitoring gradient health to adjust curriculum mid-run.

- Data Strategy : Implementing curriculum learning and domain-specific mixture ratios to prevent catastrophic forgetting during Continuous Pre-training (CPT).

3. Efficiency & Quantization Tricks :

- Advanced PEFT : Implementation of QLoRA, DoRA, and Rank-Stabilized LoRA to ensure fine-tuning performance matches full-parameter training.

- Inference-Aware Training : Utilizing Grouped Query Attention (GQA) and Multi-Query Attention (MQA) during training to optimize models for high-throughput deployment.

Required Qualifications :

- Experience : 15+ years in technology, with dedicated experience in design and development of transformer based AI/ML-based systems.

- GCP Mastery : Deep expertise in designing AI/ML solutions on GCP, specifically leveraging the Vertex AI ecosystem. A Google Cloud Professional Machine Learning Engineer certification is highly desirable.

- LLM Expertise : Hands-on experience architecting and delivering large-scale, production-grade AI systems, Proven experience in building and productionizing applications using Large Language Models (LLMs), including expertise in vector databases and model fine-tuning.