Principal Data Platform Engineer - Google Cloud Platform

Description :

We are a fast-growing, technology-driven organization building a next-generation data platform to power analytics, reporting, and intelligent decision-making at scale.

We are looking for a Principal Data Platform Engineer to lead the design and development of a modern Lakehouse architecture and establish best practices across data engineering, platform design, and distributed systems.

This is a high-impact role requiring deep technical expertise, strong architectural thinking, and the ability to build scalable data systems from the ground up.

Tech Stack & Environment :

- Architecture : Lakehouse (Medallion: Bronze / Silver / Gold)

- Processing : Apache Spark (Expert level)

- Storage/Table Format : Delta Lake (Required), Apache Iceberg (Strong Plus)

- Transformation : dbt (Expert level)

- Orchestration : Airflow, Cosmos

- Infrastructure : Cloud-native (GCP preferred), Databricks / similar platforms

- Engineering Practices: Microservices, Event-driven architecture, CI/CD, IaC (Terraform)

Core Responsibilities :

1. Data Engineering & Distributed Systems :

- Design and build scalable, fault-tolerant ETL/ELT pipelines

- Develop idempotent and self-healing data workflows

- Handle schema evolution, late-arriving data, and data consistency challenges

- Deeply optimize Spark workloads (partitioning, shuffling, memory management)

2. Platform Architecture & System Design :

- Architect and implement a modern Lakehouse platform

- Design systems for high availability (99.9%+) and horizontal scalability

- Build reusable frameworks, libraries, and templates for engineering teams

- Apply SOLID principles, design patterns, and engineering best practices

3. Data Modeling & Transformation :

- Implement scalable data models using :

- Dimensional Modeling (Kimball)

- Data Vault 2.0

- One Big Table (OBT) patterns

- Lead advanced dbt implementations:

- Macros, packages, custom tests

- Dbt Mesh and modular architecture

- CI/CD-enabled dbt workflows

4. Integration & APIs :

- Build and integrate microservices-based data systems

- Design and consume APIs (REST/gRPC)

- Work with messaging systems like Kafka / Pub-Sub

5. Cloud & Data Platform Engineering

- Build and manage cloud-native data platforms (GCP )

Deep understanding of :

- IAM, networking (VPCs), object storage

- Serverless and distributed compute

- Lead large-scale data migrations (on-prem / legacy warehouses ? Lakehouse)

Required Skills & Experience :

- Strong experience in Apache Spark (internals + optimization)

- Expertise in Delta Lake (mandatory) and Iceberg (preferred)

- Advanced proficiency in: Python / Scala / Java

- Deep understanding of: Distributed data systems, Data pipeline design patterns

- Hands-on experience with dbt (advanced usage)

- Proven experience in designing and building data platforms from scratch

- Strong problem-solving, leadership, and stakeholder communication skills

Good to Have :

- Experience with GCP ecosystem (BigQuery, Dataproc, Cloud Composer)

- Exposure to MLOps (feature stores, model pipelines)

- Experience with data observability & quality tools: