GCP Data Engineer - Python / PySpark

Experience : 5.5 - 9 Years

Openings : 8

Role Overview :

We are looking for experienced Data Engineers with strong expertise in Python or PySpark, SQL, GCP cloud services, data processing, data warehousing, and data pipelines. The ideal candidate should have hands-on experience with BigQuery, Dataflow or Dataproc, Pub/Sub, Apache Kafka, Airflow/Cloud Composer, and Terraform.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines.

- Develop data processing solutions using Python and/or PySpark.

- Develop complex SQL queries and optimize data processing workloads.

- Work with GCP data services including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Functions.

- Build and maintain data ingestion and transformation pipelines.

- Develop and manage workflows using Airflow, Astronomer, or Cloud Composer.

- Work with Apache Kafka for real-time data processing and streaming.

- Implement infrastructure automation using Terraform.

- Monitor data pipelines and troubleshoot production issues.

- Develop monitoring dashboards and ensure pipeline reliability.

- Collaborate with Data Scientists, Software Engineers, business stakeholders, and other technical teams.

Mandatory Technical Skills :

- SQL - minimum 4 years

- Python or PySpark - minimum 4 years

- GCP - minimum 3 years

- BigQuery

- Dataflow or Dataproc

- Pub/Sub

- Scheduled Queries

- Cloud Functions

- Monitoring Tools / Dashboards

- Apache Kafka

- Terraform scripting - minimum 2 years

- Airflow / Astronomer / Cloud Composer - minimum 2 years

Good to Have :

- Data Fusion, Tekton Pipeline, CI/CD Pipeline, GCP Secret Manager

- Data/ML solutions and advanced analytics exposure

- Prompt Engineering, Generative AI, LLM, Agentic AI, LangGraph, CrewAI, OpenAI SDK, Google ADK

Soft Skills :

- Strong written and verbal communication skills

- Strong presentation and stakeholder management skills

- Excellent analytical and problem-solving abilities

- Strong collaboration and interpersonal skills