Introduction :

Our team builds a data platform that reliably collects, processes, and retrieves large-scale IoT data based on AWS. We are seeking a senior engineer who can design unobstructed flow even among billions of data records, push data storage performance to the limit, and technically lead service growth.

Key Responsibilities :

- High-availability data pipeline design : AWS MSK (Kafka), end-to-end data streaming architecture design and operation using Logstash.

- Storage optimization and sharding strategies : Optimizing large-scale search performance through OpenSearch's index design, sharding, and lifecycle (ISM) management.

- Data Modeling and Partitioning : Table partitioning and data modeling in PostgreSQL/MySQL, architecture guide for efficient querying.

- Performance Tuning and Guidance : Leading SQL optimization across services and spreading technical best practices to backend development teams.

- Release Management : Be the gatekeeper during key database / data pipeline release activities, create review checklist, review deployment details and validates details before deployment to ensure zero-error deployments without roll back needs.

- Infrastructure automation : DB schema versioning (such as Flyway) and deployment process automation systems.

Requirements :

- Over 7 years of experience in data engineering or related backend development.

- Extensive experience in operating infrastructure handling large-scale traffic in AWS cloud environments.

- Strong capabilities in building real-time data pipelines and troubleshooting using Kafka.

- Experience in OpenSearch (Elasticsearch) cluster design and sharding optimization is essential.

- Deep understanding of the internal structure of RDBMS (PostgreSQL/MySQL) and ability to tune complex queries.

Preferred Qualifications :

- Seamless technical collaboration capabilities : Communication skills to discuss infrastructure optimization with DevOps engineers and provide efficient data modeling and query guides to backend developers.

- Global collaboration skills : Able to share technical documents with engineers across the globe and conduct work via video conferences during collaborative work sessions.

- IoT Domain Experience : Experience in handling large-scale device messages and building data pipelines.

- Ensuring data stability : Experience in designing logic to prevent data loss during failures and ensure consistency verification.

- Nice to have - IaC Proficiency : Experience coding and automated operations of data infrastructure using Terraform, Ansible, and others.

- 4 years of bachelors degree or higher in Computer Science or equivalent discipline.