Senior Data Engineer

Humyn Labs & KGen - Full-time

Multimodal Pipelines

AWS Data Lake

GPU Validation

AI Training Data

Humyn Labs converts human action across sound, sight, movement, and touch into high-quality multi-modal data signals for physical AI. Operating across 20+ countries in India, Southeast Asia, Latin America, and the Middle East: the real-world environments where physical AI deploys, not the labs where it is built.

Role Overview:

We are looking for a Senior Data Engineer to own, extend, and harden the multimodal data infrastructure that powers our AI training data supply chain. You will work directly with the Head of Data, building and maintaining production-grade pipelines that move real-world audio, video, and egocentric capture data from ingestion through AWS-scale storage, into GPU-backed technical validation, and finally to delivery-ready datasets for frontier AI lab customers. This role sits at the intersection of data engineering and ML infrastructure you will own the full pipeline: ingestion, lake management, quality validation, and compute handoff.

What You Will Work On:

1. Multimodal Ingestion & Preprocessing:

- Build and maintain pipelines that ingest real-world audio, egocentric video, and image data at scale covering format normalization, chunking, metadata extraction, and landing into S3-based storage with consistent schema and partitioning.

2. AWS Data Lake:

- Manage and extend the KGen data lake: Athena query optimization, Glue crawlers and cataloguing, Apache Hudi table management, Lake Formation column-level permissions, and S3 lifecycle policies at TBPB scale.

3. GPU Validation Pipeline Handoff:

- Design and maintain the data layer that feeds GPU-backed technical validation workers own sharding strategies, manifest generation, throughput optimization, and I/O design so validation compute is never the bottleneck. Understand how data format choices (Parquet, WebDataset, sharded archives) affect GPU-side loading performance.

4. Airflow DAG Management:

- Author, debug, and monitor Airflow DAGs for scheduled processing, GPU job orchestration, and pipeline coordination across ingestion, validation, and delivery stages.

5. QC and Annotation Tooling:

- Support the FastAPI-backed audio QC portal used by annotation workers; extend data validation and quality-check scripts across egocentric video and audio datasets.

6. Universal Data Schema (UDS):

- Contribute to and enforce the Universal Data Schema for audio, image, and code modalities in the Humyn Labs dataset marketplace covering schema evolution, versioning, and partition strategies.

7. Infrastructure and Access Management:

- Maintain AWS IAM, Lake Formation, and S3 bucket policies; manage data engineer access controls; handle cross-region data movement and vendor data sharing infrastructure.

8. ETL and Third-Party Integrations:

- Build and maintain ingestion pipelines from APIs (Twitch, gaming analytics, Google Forms) into DynamoDB and PostgreSQL where required by business context.

You Should Have:

- 4+ years in a data engineering role with end-to-end pipeline ownership.

- Strong Python async patterns, subprocess management, API clients, data processing at scale.

- Hands-on AWS Athena, Glue, S3, DynamoDB, Lake Formation; production-grade, not just familiarity.

- Apache Hudi or Delta Lake schema evolution, partition strategies, upsert patterns.

- SQL proficiency able to write and optimise complex analytical queries.

- Experience with Airflow or an equivalent workflow orchestrator.

- Demonstrated experience with large-scale media data pipelines audio/video format conversion, metadata extraction, chunking, egocentric or multimodal datasets (hard requirement, not a plus).

- Understanding of how data format and I/O design affects downstream GPU compute workloads WebDataset, sharded Parquet, tfrecord, or equivalent.

Nice to Have:

- Direct experience designing data handoff for GPU clusters (AWS Batch GPU instances, Ray, or SLURM).

- Familiarity with ML training data formats and dataset standards used by AI labs (Hugging Face datasets, WebDataset, dataset cards).

- Experience with rclone, large-scale file transfer, or cloud-to-cloud sync pipelines.

- Exposure to data lineage or provenance tooling (OpenLineage, DataHub, or custom metadata schemas).