Job Title : Data Engineer - BigQuery & Cloud Composer (GCP)

Location : Gurugram

About the Role :

Were looking for a hands-on Data Engineer with strong expertise in Google BigQuery and Cloud Composer (Airflow on GCP) to build, orchestrate, and optimize robust data pipelines. The candidate will work across file integrations (XLSX/CSV/JSON), GCS-based ingestion/export, BigQuery SQL & Stored Procedures, and Python development for Composer DAGs, including API integrations and custom operators. The ideal candidate is equally comfortable designing scalable data models and writing production-grade orchestration code with reliability, lineage, and cost-efficiency in mind.

Key Responsibilities :

Pipeline Design & Orchestration :

- Design, develop, and manage Cloud Composer (Airflow) DAGs for batch and near-real-time workloads with robust scheduling, retries, SLAs, and dependency management.

- Build custom Python operators/sensors/hooks for APIs (REST/JSON), file processing, and event-driven workflows.

BigQuery Engineering:

- Develop ELT/ETL transformations using standard SQL, Stored Procedures, User-Defined Functions (UDFs), and views/materialized views.

- Implement performance best practices: partitioning, clustering, pruning, query optimization, storage optimization, and cost controls.

File & GCS Integrations :

- Ingest and process CSV/JSON/XLSX files from/to Google Cloud Storage (GCS) with schema management, validations, and error handling.

- Implement data export/import pipelines between BigQuery, GCS, and external systems; manage lifecycle policies and storage tiers.

API & Custom Code Development :

- Build Python-based integrations for third-party and internal APIs (OAuth2/API keys), pagination, throttling, and incremental loads.

- Create reusable libraries/utilities for common data processing tasks (parsing, normalization, deduplication, enrichment).

DevOps & Delivery :

- Use CI/CD (Cloud Build/GitHub Actions/GitLab CI) for DAGs and SQL assets, environment promotion, and versioning.

- Collaborate with Data Analysts/Scientists and Product teams; document pipelines, runbooks, and data contracts.

Required Qualifications :

- 3- 7 years of experience in Data Engineering with GCP, focusing on BigQuery and Cloud Composer (Airflow).

- Strong SQL skills and experience with BigQuery Stored Procedures, UDFs, and performance tuning (partitioning, clustering, join strategies).

- Proficiency in Python for data processing (pandas, pyarrow, openpyxl/xlrd) and Composer development (operators, sensors, hooks, XComs).

- Hands-on with GCS : reading/writing large datasets, lifecycle configuration, permissions (IAM), and storage cost optimization.

- Experience integrating files (XLSX/CSV/JSON) and REST APIs (auth, pagination, error/retry patterns).

- Solid understanding of Airflow DAG design : idempotency, backfills, scheduling, SLAs, retries, task isolation, and parallelism.

- Knowledge of testing (unit/integration), data quality checks, and CI/CD for data pipelines.

Nice-to-Have :

- Understanding about SAP S4, Ariba, Fieldglass, Siebel and Kenan data

- ETL Tool (SAP Data Services/Striim) development experience

- Infra-as-code (Terraform) for Composer/BigQuery/GCS provisioning.

- Performance/cost optimization at scale (slot management, reservations, storage tiers).

- Experience with SCD, CDC, and incremental processing patterns.

- Experience with on-prem to GCP migrations and cross-region strategies.