Happiest Minds Technologies - Lead Data Engineer - Medallion Architecture

Role Overview :

Engineering lead who owns design & delivery end-to-end; raises bar on performance, DataOps, and governance; mentors C3s.

Core Responsibilities :

- Design high-throughput batch & streaming (Structured Streaming/Autoloader) to Medallion targets.

- Implement advanced Delta : MERGE/SCD2, schema evolution/enforcement, OPTIMIZE/Z-ORDER/VACUUM, compaction.

- Engineer observability : metrics, alerts, SLIs/SLOs, error budgets; on-call runbooks.

- Implement Unity Catalog patterns (RBAC/ABAC), row/column-level security; Key Vault integration; PII/PHI handling.

- Own cost & performance : Photon, autoscaling, cluster policies/pools; job SLAs.

- Lead CI/CD (Repos/CLI/ADO) with automated unit/data tests and promotion gates; design reviews and reusable frameworks.

Must-have Skills (Screening Focus) :

- Deep PySpark/Spark SQL/Delta; DLT + Workflows orchestration.

- MERGE/SCD2; schema evolution/enforcement; time travel.

- Performance/cost levers : OPTIMIZE/Z-ORDER, file sizing, Photon, autoscaling; cluster policies/pools.

- Unity Catalog RBAC/ABAC; secrets; PII/PHI controls.

- CI/CD with tests (unit/data expectations) and environment promotion; on-call readiness.

Nice-to-have (Prefer, not Mandatory) :

- Kafka/Event Hubs streaming patterns; Synapse integration.

- Delta Sharing, Lakehouse Federation; dbt for transformations.

90-day Outcomes (Examples) :

- Reduce a priority jobs runtime 25% and cost/GB 20% within 90 days.

- Stand up CI/CD + tests + SLOs for one DLT pipeline with alerting; publish a reusable SCD2/DQ module.