Job Description :

We are looking for an experienced GCP Data Engineer with strong expertise in Google Cloud Platform (GCP), Dataproc, PySpark, BigQuery, and Java. The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing solutions and cloud-based data platforms that support analytics, reporting, and business intelligence initiatives.

The candidate should have hands-on experience in building large-scale data pipelines, processing structured and unstructured data, optimizing data workflows, and working with distributed computing frameworks in a cloud environment.

Key Responsibilities :

- Design, develop, and maintain scalable data engineering solutions on Google Cloud Platform (GCP).

- Build and optimize batch and real-time data processing pipelines using PySpark and Dataproc.

- Develop efficient ETL/ELT processes to ingest, transform, validate, and load large volumes of data from multiple sources.

- Create and manage data lakes and data warehouse solutions using BigQuery.

- Write high-performance PySpark jobs for distributed data processing and analytics.

- Develop and maintain Java-based applications and services that integrate with data platforms and cloud services.

- Optimize BigQuery datasets, tables, partitions, clustering, and SQL queries for performance and cost efficiency.

- Implement data quality checks, monitoring, validation, and error-handling mechanisms.

- Collaborate with business analysts, data scientists, architects, and stakeholders to understand data requirements and deliver scalable solutions.

- Ensure data security, governance, and compliance standards are followed across cloud environments.

- Troubleshoot performance bottlenecks and production issues in data pipelines and cloud infrastructure.

- Participate in code reviews, architecture discussions, and technical design sessions.

- Support CI/CD implementation and deployment automation for data engineering workloads.

- Create technical documentation, operational runbooks, and best practices for data platforms.

Required Technical Skills :

Google Cloud Platform (GCP) :

- Strong hands-on experience with Google Cloud Platform services.

- Experience working with :

i. Dataproc

ii. BigQuery

iii. Cloud Storage (GCS)

iv. Pub/Sub

v. Cloud Composer

vi. Dataflow (preferred)

vii. IAM and Security Management

viii. Monitoring and Logging Tools

Dataproc :

- Experience creating and managing Dataproc clusters.

- Knowledge of cluster sizing, scaling, performance tuning, and job scheduling.

- Experience running Spark workloads on Dataproc.

- Understanding of cluster monitoring and troubleshooting.

PySpark :

- Strong expertise in PySpark development.

- Experience building scalable distributed data processing applications.

- Knowledge of Spark SQL, DataFrames, RDDs, and Spark optimization techniques.

- Experience handling large datasets efficiently.

- Understanding of partitioning, caching, joins, and performance tuning.

BigQuery :

- Hands-on experience with BigQuery data warehouse solutions.

- Strong SQL development and query optimization skills.

- Experience designing schemas, partitioned tables, and clustered tables.

- Knowledge of data loading, transformation, and reporting processes.

- Understanding of BigQuery performance optimization and cost management.

Java Development :

- Strong programming experience in Java.

- Good understanding of Core Java concepts, OOP principles, Collections Framework, and Exception Handling.

- Experience developing backend services, APIs, and data-processing applications using Java.

- Ability to integrate Java applications with cloud-based services and data platforms.

Data Engineering Concepts :

- Strong understanding of ETL and ELT methodologies.

- Data modeling and database design.

- Data quality management and validation.

- Data governance and security best practices.

- Batch and streaming data processing concepts.

Database Skills :

- Strong SQL programming skills.

- Experience working with relational databases such as :

i. PostgreSQL

ii. MySQL

iii. Oracle

- Ability to write complex queries and optimize database performance.

Required Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

- Typically 5+ years of experience in Data Engineering or Big Data technologies.

- Strong analytical and problem-solving skills.

- Excellent communication and stakeholder management skills.

- Ability to work in Agile development environments.