Posted on: 19/11/2026
1. Mandatory Technical Skills - Must Have:
A. Apache Airflow - Self-Managed (Critical Requirement):
- Hands-on experience installing, configuring, deploying, and operating Apache Airflow.
- Experience managing DAGs, scheduling, dependencies, retries, monitoring, troubleshooting, and production failures.
- Understanding of Airflow infrastructure, workers, schedulers, executors, and operational maintenance.
- Experience running Airflow independently, not exclusively through managed platforms such as AWS MWAA or Astronomer.
- Important: Candidates who have only created or maintained DAGs within an existing managed Airflow environment are not an exact match.
B. AWS Data Engineering:
- Hands-on experience building and maintaining data pipelines using Amazon S3: Data storage, ingestion, and organization.
- AWS Glue: ETL jobs, Glue Data Catalog, and data transformations.
- Amazon Athena: SQL-based querying, validation, and analysis of data stored in S3.
- Candidates should demonstrate how these services were used together in actual production pipelines.
C. SQL & Python - Strong Hands-On Expertise:
- Advanced SQL skills, including joins, CTEs, window functions, aggregations, and query optimization.
- Strong Python programming skills for building ETL/ELT pipelines, transformations, automation, and data validation.
- Experience troubleshooting data issues and improving pipeline reliability.
D. Data Quality & Reconciliation - Critical Requirement:
- Must have experience ensuring the accuracy and integrity of data processed through pipelines.
- Relevant experience includes: 1. Source-to-target data reconciliation. 2. Record count and aggregate validation. 3. Identifying missing, duplicate, inconsistent, or incorrect records. 4. Investigating data mismatches and transformation errors. 5. Implementing automated data quality checks. 6. Ensuring business metrics and reporting outputs are accurate.
- Important: Candidates focused only on pipeline execution, scheduling, or infrastructure monitoring without data validation experience are not a strong match.
E. dbt (Data Build Tool):
- Hands-on experience developing SQL transformation models using dbt.
- Experience with dbt tests, dependencies, documentation, and data transformation workflows.
- Ability to demonstrate dbt usage in actual projects.
F. Terraform - Infrastructure as Code:
- Practical experience using Terraform to provision and manage cloud infrastructure.
- Experience maintaining Terraform configurations and deploying AWS resources.
- Understanding of infrastructure version control and repeatable deployments.
3. Preferred / Good-to-Have Experience:
- Experience working with healthcare, health-tech, or personal/sensitive data.
- Familiarity with data privacy, security, and compliance requirements.
- Experience handling regulated or confidential datasets.
- Experience troubleshooting and maintaining production data pipelines.
Did you find something suspicious?
Posted by
Posted in
Data Engineering
Functional Area
Data Engineering
Job Code
1684713