Role : GCP Data Engineer

Experience : Not specified

Location : Navi Mumbai / Pune

Employment Type : Full-time, Permanent

Industry : IT Services & Consulting

Department : Engineering - Software & QA

Role Category : Software Development

About the Role :

We are looking for a skilled GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT frameworks on Google Cloud Platform (GCP).

The role will involve working with large-scale campaign and clickstream datasets, building reliable data processing workflows, optimizing BigQuery performance and costs, and implementing automation, monitoring, and alerting for critical data pipelines.

The ideal candidate should have strong hands-on expertise in GCP, BigQuery, SQL, Python/PySpark, and Airflow/Composer, along with an understanding of CI/CD and Git-based development workflows.

Key Responsibilities :

- Design, develop, and implement scalable data pipelines on GCP using services such as BigQuery, Cloud Storage, and Airflow/Composer.

- Build robust ETL/ELT pipelines to process and transform large volumes of campaign, clickstream, and other business data.

- Develop efficient data ingestion and transformation workflows to support analytics and downstream applications.

- Write optimized SQL queries in BigQuery for large-scale data processing and analytics.

- Optimize BigQuery query performance and processing costs through effective data modelling, partitioning, clustering, query optimization, and other best practices.

- Develop reusable and scalable data engineering frameworks for batch and scheduled data processing.

- Automate data workflows and pipeline execution using Airflow/Cloud Composer.

- Implement appropriate monitoring, logging, alerting, and failure-handling mechanisms for critical data pipelines.

- Identify and troubleshoot pipeline failures, data quality issues, performance bottlenecks, and integration-related problems.

- Build and maintain reliable data processing workflows with appropriate data validation and quality checks.

- Collaborate with analytics, product, engineering, and business teams to understand data requirements and deliver scalable solutions.

- Follow best practices for version control, code quality, testing, deployment, and documentation.

- Support CI/CD processes for data engineering workflows using tools such as Jenkins or equivalent platforms.

- Work with Git-based development workflows for managing and deploying data engineering code.

- Continuously identify opportunities to improve pipeline reliability, scalability, performance, and operational efficiency.

Required Skills & Experience :

- Strong hands-on experience in Data Engineering and building scalable data pipelines.

- Strong proficiency in SQL, preferably with extensive experience in Google BigQuery.

- Strong programming skills in Python and/or PySpark.

- Hands-on experience with Google Cloud Platform (GCP).

- Strong experience with BigQuery for data storage, transformation, querying, and optimization.

- Experience working with Google Cloud Storage (GCS).

- Experience developing ETL/ELT frameworks for large-scale datasets.

- Hands-on experience with Apache Airflow or Google Cloud Composer for workflow orchestration.

- Understanding of data pipeline monitoring, logging, alerting, and troubleshooting.

- Familiarity with CI/CD practices and tools such as Jenkins.

- Good understanding of Git and version-control workflows.

- Strong analytical and problem-solving skills.

Good to Have :

- Experience working with campaign, marketing, customer, or clickstream data.

- Experience handling large-volume datasets on cloud platforms.

- Knowledge of BigQuery partitioning and clustering.

- Experience with data quality frameworks and validation processes.

- Exposure to other GCP services used in data engineering.

- Experience working in Agile development environments