Data Engineer - SQL/Hive/Spark

Role : Data Engineer / Analytics Engineer - SQL, dbt, Hive, Spark, Airflow

Experience Required : 3 to 6 years of hands-on experience in SQL-based data engineering, data modeling, and ETL/ELT development.

Project Overview :

The candidate will work on the WMS / ERP analytics data platform, supporting reporting, reconciliation, and operational dashboards. The project involves building and maintaining data models using dbt, Hive, Spark SQL, Airflow, and Metabase.

The candidate will be responsible for designing, developing, troubleshooting, and optimizing data models similar to existing warehouse analytics models used for inventory movement, material receipt, move order, return movement, reconciliation, and reporting use cases.

Key Responsibilities :

The candidate will be responsible for :

- Designing and developing dbt models using SQL, Hive SQL, and Spark SQL.

- Building data pipelines for warehouse, ERP, WMS, inventory, and reconciliation reporting.

- Creating and maintaining staging, intermediate, and final reporting models.

- Troubleshooting production issues related to :

i. dbt model failures

ii. Hive/Spark SQL errors

iii. Airflow job failures

iv. Data mismatch issues

v. Incremental load issues

vi. Performance degradation

vii. Schema or datatype mismatches

Mandatory Skills :

The candidate must have hands-on experience in :

- Strong SQL

- Hive SQL

- Spark SQL

- dbt model development

- Airflow DAG monitoring / troubleshooting

- Data warehouse concepts

- ETL / ELT pipeline development

- Data reconciliation and validation

- Performance tuning of SQL queries

- Complex joins, CTEs, window functions, aggregations, CASE logic

- Incremental data loading concepts

Good to Have Skills :

Experience in the following will be an added advantage :

- Metabase reporting/dashboarding

- Apache Hudi / Iceberg / ORC file formats

- S3-based data lake environment

- PySpark scripting for data comparison and validation

- Jenkins / Git-based deployment process

- ERP/WMS/inventory domain knowledge

- Experience with Kyuubi or Spark SQL execution environments