Data Engineer

Experience: 6+ years in data engineering, ETL/ELT, analytics engineering, data warehouse development, or big data pipelines; minimum 4 years on GCP cloud data platforms preferred.

What we're looking for:

- Strong SQL and Python.

- Data modelling, warehousing concepts, partitions, clustering, indexing, optimization.

- ETL/ELT pipeline design and production support.

- Batch and streaming pipeline fundamentals.

- Orchestration using Airflow, Composer, Control-M, ADF, dbt, or equivalent.

- Big data processing using Spark, PySpark, Beam, Dataflow, Dataproc, Databricks, EMR, or equivalent.

- Cloud storage and data lake concepts.

- Data quality, reconciliation, schema validation, lineage, monitoring, and alerting.

- CI/CD for data pipelines and code versioning.

- Ability to explain scale scenarios, not just tool definitions.

Technical Skills:

- Python and advanced SQL.

- Data quality checks, reconciliation, alerting, retries, and backfills.

- Production pipeline monitoring and troubleshooting.

GCP Skills:

- Mandatory: BigQuery: datasets, tables, views, partitioning, clustering, SQL optimisation, cost awareness.

- Option 1: Dataflow / Apache Beam or equivalent streaming/batch framework.

- Option 2: Dataproc / PySpark or strong Spark experience.

- Pub/Sub or equivalent event streaming.

- Cloud Composer / Airflow DAG development.

Nice to have:

- Dataform, dbt, CI/CD for data.

- BigQuery ML.

- Dataflow templates.

- Looker / LookML basics.

- Terraform for data infrastructure.