Principal Data Platform Engineer

Description :

Role : Principal Data Platform Engineer

Experience : 9 to 15 years (strictly relevant experience only)

Location : Bangalore (Hybrid Work from office on Tuesday & Friday)

Notice Period : Immediate Joiners only

Please find the detailed Job Description below.

Principal Data Platform Engineer

The Stack & Environment :

- Architecture : Lakehouse (Medallion : Bronze/Silver/Gold)

- Compute : Apache Spark (Expert level)

- Storage/Table Format : Delta Lake (Required), Iceberg (Strong Plus)

- Transformation : dbt (Expert level)

- Orchestration : Airflow, Cosmos

- Infrastructure : Cloud-native (GCP preferred) + Databricks/Commercial tooling

- Patterns : Microservices, Event-driven, CI/CD, IaC (Terraform)

Core Technical Requirements :

1. Data Engineering & Spark Internals :

- Deep Spark : You must understand RDDs, DataFrames, Spark SQL, and internals (Shuffle, Partitioning, Memory Management, Catalyst Optimizer).

- Pipeline Mastery : Building idempotent, self-healing ELT/ETL pipelines.

- Experience with Schema Evolution and handling late-arriving data.

- Lakehouse ACID : Expert knowledge of transaction logs, time travel, and file compaction in Delta/Iceberg.

2. Software Architecture & Design :

- Engineering First : This isn't just "SQL and scripts." You apply SOLID principles, design patterns, and write production-grade Python/Scala/Java.

- Integration : Experience building and consuming Microservices.

- Knowledge of API design (REST/gRPC) and message brokers (Kafka/PubSub).

- System Design : Experience building a platform from scratch. You know how to design for 99.9% availability and horizontal scalability.

3. Data Modeling & dbt :

- Modeling : Expert in dimensional modeling (Kimball), Data Vault 2.0, or OBT (One Big Table) for high-performance analytics.

- dbt Power User : Advanced dbt usage (Macros, Packages, Custom Tests, dbt Mesh). You treat dbt projects like software repositories (version control, PR reviews, CI).

4. Cloud & Platform :

- Cloud Native : Deep understanding of IAM, VPCs, Object Storage, and serverless compute.

- Migrations : Proven track record of moving petabyte-scale data from legacy systems (On-prem, Redshift, Snowflake) to a Lakehouse without data loss.

Key Deliverables (First 6-12 Months) :

- Platform Zero : Evaluate, select, and deploy the foundational Lakehouse infrastructure.

- Core Frameworks : Build the reusable libraries/templates for the rest of the engineering team to build pipelines.

- Legacy Decommission : Design the technical map to migrate all high-priority finance/business data to the new stack.

- Performance Baseline : Optimize Spark/Cloud costs by at least 20% through better resource management