Job Description :

We are looking for a highly skilled GCP Data Engineer with strong expertise in designing, developing, and optimizing large-scale data processing systems on Google Cloud Platform (GCP).

The ideal candidate should have hands-on experience building batch and streaming data pipelines using BigQuery, Dataflow, Cloud Composer/Airflow, Dataproc, Apache Beam, and Pub/Sub.

The role requires strong Python and SQL programming skills along with experience in data warehousing, distributed data processing, and event-driven architectures.

Mandatory Skills :

Strong hands-on experience in :

- BigQuery

- Cloud Composer / Apache Airflow

- Dataflow

- Dataproc

- Apache Beam

Experience building :

- End-to-end batch pipelines

- Real-time/streaming data pipelines

Strong expertise in :

- Python

- SQL

Experience with :

- Pub/Sub

- Event-driven architectures

- Kafka integration or migration use cases

Strong understanding of :

- Data warehouse concepts

- Slowly Changing Dimensions (SCD)

- Partitioning & clustering strategies.

- Experience working with large-scale distributed data processing systems.

- Strong debugging, optimization, and performance tuning capabilities.

Key Responsibilities :

- Design, develop, and maintain scalable batch and real-time data pipelines on Google Cloud Platform (GCP).

- Build and optimize ETL/ELT workflows using Dataflow, Apache Beam, Dataproc, and Cloud Composer/Airflow.

- Develop robust streaming pipelines using Pub/Sub and Kafka integration patterns.

- Implement data ingestion, transformation, validation, and orchestration frameworks.

- Design and optimize BigQuery data warehouse structures including partitioning, clustering, and SCD implementations.

- Develop reusable and scalable data processing frameworks using Python and SQL.

- Monitor, troubleshoot, and optimize pipeline performance, reliability, and cost efficiency.

- Collaborate with data scientists, analysts, architects, and business teams for data integration and analytics requirements.

- Ensure data quality, governance, security, and compliance standards are maintained.

- Support CI/CD implementation and automation for data engineering workflows.

- Participate in architecture discussions and recommend best practices for cloud-native data solutions.