Job Summary

As a Staff Data Engineer, you will be responsible for designing and operating large-scale batch and real-time data platforms handling terabytes to petabytes of data across multiple business and streaming systems.

You will work on distributed compute platforms, real-time streaming pipelines, analytical query engines, lakehouse architectures, orchestration frameworks, and AI-powered data workflows.

You will collaborate closely with Product, Analytics, AdTech, ML, Platform Engineering, and Business teams to deliver highly reliable, scalable, and efficient data systems.

This is a deeply technical individual contributor role focused on execution, optimization, platform engineering, and solving large-scale distributed data challenges.

Key Responsibilities :

Data Platform Engineering :

- Design, build, and maintain scalable batch and real-time data pipelines processing TBs/PBs of data daily

- Build and optimize distributed data processing systems using Apache Spark on AWS EMR

- Develop and operate real-time streaming pipelines using Kafka and AWS Kinesis

- Design and evolve modern lakehouse architectures using Apache Iceberg, S3, Glue Catalog, and Trino

- Build reliable, scalable, and high-performance analytical platforms on ClickHouse

Streaming & Real-Time Data Systems :

- Build low-latency streaming pipelines for clickstream, playback analytics, ad events, operational telemetry, and real-time dashboards

- Optimize ingestion throughput, partitioning strategies, schema evolution, and streaming reliability

- Improve scalability, fault tolerance, and operational visibility across streaming systems

Query Engine & Lakehouse Optimization :

- Optimize distributed query performance across Trino, ClickHouse, and Iceberg

- Improve partitioning, compaction, indexing, caching, and storage optimization strategies

- Design efficient metadata management and data governance workflows using AWS Glue Catalog

Workflow Orchestration & Automation :

- Design and maintain orchestration workflows using Apache Airflow

- Build resilient DAG architectures with strong dependency management, retry handling, observability, and failure recovery

- Improve deployment automation, CI/CD, testing, and operational reliability for data systems

AI-Powered Data Systems :

- Build Agentic Data Pipelines capable of intelligent orchestration, anomaly detection, self-healing workflows, and automated operational remediation

- Develop AI-powered data products and metadata intelligence systems

- Work on LLM orchestration, RAG pipelines, metadata enrichment, and semantic search capabilities over data platforms

Data Governance & Engineering Standards :

- Drive schema governance, data modelling standards, lineage, and metadata management practices

- Improve observability, monitoring, data quality, and operational excellence across the data ecosystem

- Collaborate closely with analytics, product, ML, and business teams to deliver reliable and well-documented data products

Required Skills & Experience :

Must Have :

- 8+ years of hands-on experience in Data Engineering or Distributed Data Systems

- Deep expertise in Apache Spark including:

1. Spark internals

2. Shuffle optimization

3. Memory management

4. Adaptive Query Execution (AQE)

5. Partitioning strategies

6. Performance tuning

- Strong production experience with:

1. ClickHouse

2. Apache Iceberg

3. Trino

- Strong AWS Analytics ecosystem experience:

1. AWS EMR

2. Kinesis Data Streams / Firehose

3. AWS Glue Catalog

4. AWS Glue ETL

5. Amazon S3 lifecycle management

- Hands-on experience building and operating real-time data pipelines

- Strong understanding of Kafka fundamentals and streaming architectures

- Strong experience with Apache Airflow :

1. DAG design

2. Dependency management

3. Failure handling

4. Scheduling optimization

- Experience building or integrating AI agents for data workflows:

1. Prompt engineering

2. LLM orchestration

3. LangChain / LlamaIndex or similar frameworks

4. RAG pipelines over metadata systems

- Strong programming skills in :

1. Python

2. Scala

- Strong understanding of:

1. Data modelling

2. Distributed systems

3. Data governance

4. Query optimization

5. Data reliability engineering

Good to Have :

- Experience with dbt or similar transformation frameworks

- Familiarity with data quality and validation frameworks

- Exposure to Kubernetes and containerized data platforms

- Experience with vector databases and embedding pipelines for semantic search

- Experience contributing to open-source data engineering tools

- Experience with Infrastructure-as-Code tools such as Terraform

- Exposure to MLOps, feature stores, or AI infrastructure platforms

- Experience working on OTT, AdTech, streaming analytics, or large-scale consumer internet platforms

Why Join SonyLIV :

- Work on one of Indias largest OTT and live sports streaming platforms

- Solve complex large-scale data engineering challenges involving real-time and batch systems

- Build next-generation lakehouse and streaming data platforms at scale

- Work on high-volume clickstream, playback analytics, adtech, and operational telemetry systems

- Opportunity to build AI-powered data workflows and modern distributed analytics infrastructure

- Work with cutting-edge technologies across streaming, analytics, observability, AI, and cloud-native data systems