Staff MLOps Learning Engineer

We are looking for an MLOps/GenAIOps Engineer to own the end-to-end machine learning operations lifecycle for our US based client. This role is 100% REMOTE.

This role focuses on automating model training, evaluation, deployment, and monitoring workflows while partnering closely with our Full Stack AI/ML Engineers to ensure seamless production deployment of foundation models and classical ML systems. You'll build the infrastructure that enables rapid, safe iteration on AI/ML solutions that impact patient care.

Essential Duties & Responsibilities:

AWS SageMaker Operations & Orchestration:

- Design and implement SageMaker Pipelines with complex DAG authoring for end-to-end ML workflows including data preprocessing, training, evaluation, and deployment.

- Configure and optimize SageMaker Processing Jobs, Training Jobs (PyTorch/HuggingFace containers), and Managed Spot Instances for cost-effective model training.

- Manage SageMaker Model Registry with versioning, lineage tracking, and approval workflows for model governance.

- Implement sophisticated endpoint deployment strategies including blue/green traffic routing, canary deployments, and A/B testing configurations.

- Configure and maintain SageMaker Model Monitor for continuous data quality monitoring, model quality assessment, bias detection, and feature drift alerting.

GenAI Model Lifecycle Management:

- Build operational frameworks for Amazon Bedrock model deployments including provisioned throughput management, guardrail configuration, and usage monitoring.

- Implement automated evaluation pipelines for foundation model outputs with quality gates and human-in-the-loop review workflows.

- Design prompt versioning systems integrated with model registry for reproducible GenAI deployments.

- Monitor GenAI model performance including latency, cost per inference, hallucination rates, and guardrail trigger metrics.

Model Lifecycle Automation:

- Develop automated retraining trigger systems based on data drift, performance degradation, scheduled intervals, or manual triggers.

- Implement evaluation gates as code with configurable thresholds for metrics (F1, precision, recall, AUC, etc.) before model promotion.

- Build automated model promotion workflows from dev - staging - prod with approval gates and rollback capabilities.

- Design rollback automation with traffic shifting strategies to quickly revert to previous model versions upon performance degradation.

Training Data Engineering:

- Build AWS Glue ETL pipelines for training corpus assembly, transformation, and feature engineering at scale.

- Design dataset versioning systems using S3 + manifest files ensuring reproducibility and lineage tracking across model training runs.

- Establish reproducibility patterns including seed management, environment pinning, and deterministic data splitting.

Container Engineering & Registry Management:

- Build optimized Docker containers for training and inference workloads with multi-stage builds, layer caching, and security scanning.

- Manage Amazon ECR repositories with lifecycle policies, image scanning, and vulnerability remediation workflows.

- Implement bring-your-own-container (BYOC) patterns for SageMaker supporting custom frameworks and dependencies.

- Optimize container startup times and resource utilization for cost-effective inference.

ML Observability & Monitoring:

- Design comprehensive CloudWatch dashboards for model health including accuracy, latency, throughput, error rates, and drift metrics.

- Implement custom CloudWatch metrics for business-specific KPIs (e.g., clinical accuracy per condition, false positive rates for critical alerts).

- Build alerting systems for accuracy regression, data drift, concept drift, and model staleness with appropriate escalation paths.

- Create observability frameworks that integrate with AWS X-Ray for end-to-end request tracing from API call through model inference.

CI/CD for ML Artifacts:

- Design and implement GitHub Actions or AWS CodePipeline workflows for automated ML artifact testing, validation, and deployment.

- Build multi-stage promotion pipelines (dev - staging - prod) with automated testing gates and manual approval checkpoints.

- Implement artifact versioning and lineage tracking for models, datasets, feature transformations, and deployment configurations.

- Create integration testing frameworks for ML APIs including performance benchmarking and regression testing.

Infrastructure & Compliance:

- Collaborate with Full Stack AI/ML Engineers to define infrastructure requirements and translate them into production-grade AWS CDK constructs.

- Implement HIPAA-compliant ML pipelines with appropriate encryption, access controls, audit logging, and data residency requirements.

- Design multi-tenant ML infrastructure with tenant isolation, resource quotas, and cost allocation.

- Ensure reproducibility and auditability of all ML experiments and deployments for regulatory compliance.

Qualifications:

Technical Skills:

- AWS SageMaker Expertise: Deep hands-on experience with SageMaker Pipelines, Processing/Training Jobs, Model Registry, Endpoints, Model Monitor, and Managed Spot training.

- AWS Bedrock & GenAI Operations: Production experience managing Amazon Bedrock deployments including Claude family models, Titan Embeddings v2, and Bedrock Guardrails (PHI/PII detection); expertise in provisioned throughput vs on-demand capacity management, model evaluation automation, prompt versioning and registry integration, inference monitoring (latency, cost, token usage), and guardrail performance tracking; ability to build automated quality gates for foundation model outputs.

- AWS ML Services: Proficiency with AWS Glue ETL, Bedrock, S3, ECR, CloudWatch, and Step Functions.

- Container Technologies: Strong Docker skills including multi-stage builds, optimization techniques, and container orchestration.

- Python Programming: Proficient in Python for pipeline orchestration, data processing, and automation scripts (boto3, sagemaker SDK, pandas/polars).

- CI/CD Tools: Experience with GitHub Actions, AWS CodePipeline, or similar CI/CD platforms for ML workflows.

- Infrastructure as Code: Familiarity with AWS CDK, CloudFormation, or Terraform for ML infrastructure provisioning.

- ML Fundamentals: Solid understanding of ML training concepts, evaluation metrics, hyperparameter tuning, and model validation techniques.

- Observability: Experience building custom metrics, dashboards, and alerting systems for production ML systems.

Experience:

- 12+ years of software/data engineering experience with at least 5 years focused on MLOps or production ML systems.

- Proven track record of building automated ML pipelines and deployment workflows at scale.

- Experience with model monitoring, drift detection, and automated retraining systems.

- Healthcare or regulated industry experience strongly preferred.

- Demonstrated ability to operationalize both classical ML and foundation model systems.

Soft Skills:

- Strong collaboration skills to work closely with Full Stack AI/ML Engineers, data scientists, and platform teams.

- Systems thinking with ability to design reliable, fault-tolerant ML infrastructure.

- Excellent troubleshooting and debugging skills for complex distributed systems.

- Commitment to automation, reproducibility, and engineering excellence.

- Strong documentation skills for runbooks, architecture decisions, and operational procedures.

Preferred Qualifications:

- Experience with Amazon Bedrock operational patterns and GenAI model monitoring.

- Familiarity with PyTorch, HuggingFace Transformers, and deep learning training workflows.

- Knowledge of FHIR, HL7 v2, and healthcare data standards.

- Experience with cost optimization strategies for ML workloads (Spot instances, autoscaling, model optimization).

- Contributions to open-source MLOps tools or frameworks.

- AWS certifications (Machine Learning Specialty, Solutions Architect, or DevOps Engineer).