Job Summary:

We are seeking a skilled Data Engineer with strong expertise in GCP, PySpark, and Scala to design, develop, and optimize scalable data pipelines and ETL workflows. The ideal candidate will have hands-on experience working with modern data platforms, orchestration tools, and cloud-native data processing frameworks to support large-scale data engineering initiatives.

Key Responsibilities:

- Design, develop, deploy, monitor, and optimize ETL workflows for batch data processing.

- Build and maintain scalable data pipelines using Scala, PySpark, Airflow, BigQuery, Dataproc, and GCP services.

- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) with robust data quality checks and data summarization processes.

- Ensure high-performance and reliable data processing through continuous optimization and tuning.

- Develop observability frameworks, monitoring solutions, and operational runbooks to support production systems.

- Manage workflow orchestration using Apache Airflow, including dependency management, scheduling, and idempotent execution.

- Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.

- Troubleshoot and resolve data pipeline issues while ensuring data integrity and availability.

- Follow best practices for data engineering, coding standards, and operational excellence.

- Contribute to data governance initiatives and ensure compliance with organizational data management standards.

Required Skills:

- 6+ years of experience in Data Engineering.

- Strong hands-on experience with Google Cloud Platform (GCP).

- Proficiency in PySpark and Scala for large-scale data processing.

- Experience with BigQuery, Dataproc, and Apache Airflow.

- Solid understanding of ETL development, data warehousing, and batch processing frameworks.

- Experience implementing Medallion Architecture and data quality frameworks.

- Knowledge of performance optimization, monitoring, observability, and operational support processes.