Data Engineer - Google Cloud Platform

Description :

Roles and Responsibilities :

- Design, develop, and own scalable batch and near-real-time data pipelines on GCP, including large-scale streaming and batch processing

- Build and optimize ETL/ELT workflows using Cloud Composer and Dataflow for structured and semi-structured data

- Create and manage robust data models to support analytics and reporting use cases

- Optimize BigQuery performance and cost efficiency

- Implement and enforce data quality, governance, security, and compliance standards

- Develop data validation frameworks and quality checks

- Lead data architecture decisions aligned with business and enterprise strategy

- Collaborate with architects, analysts, data scientists, and product teams to ensure data readiness

- Provide production support, troubleshoot issues, and perform root cause analysis

- Drive performance optimization and continuous improvement initiatives

Key Skills :

- Cloud Platform : Google Cloud Platform (GCP) BigQuery, Cloud Composer (Airflow), Dataflow (Apache Beam), Cloud Storage

- Programming : Python, PySpark, Advanced SQL, Java/Scala (preferred)

- Data Engineering : ETL/ELT pipeline development, batch & near real-time processing, metadata-driven frameworks

- Streaming Technologies : Pub/Sub, Kafka

- Data Warehousing : BigQuery (Snowflake experience transferable)

- Workflow Orchestration : Cloud Composer (Airflow)

- DevOps & CI/CD : Git, CI/CD pipelines

- Containers (Optional) : Docker, Kubernetes

- BI & Reporting (Awareness) : Looker, Power BI

- Data Governance : Data quality, security, compliance, and validation frameworks

- Core Competencies : System design, problem-solving, performance optimization