The Market Innovation team is responsible for building scalable and resilient data platforms that support real-time analytics and business insights across NPCI's payment ecosystem. The role involves designing and developing high-volume streaming data pipelines using modern big data technologies while ensuring 24x7 availability, reliability, and performance.

About NPCI :

The National Payments Corporation of India (NPCI) is a pivotal institution in India's digital payments ecosystem, established by the Reserve Bank of India (RBI) and the Indian Banks Association (IBA). NPCI is dedicated to building world-class digital payment infrastructure through innovative and efficient retail payment platforms.

Key Responsibilities :

- Design and develop real-time data pipelines using Apache Kafka and stream processing frameworks (Spark Structured Streaming / Apache Flink).

- Ensure 24x7 data availability with fault-tolerant, highly reliable systems.

- Implement ingestion, transformation, and load (ITL) patterns for data lakes/lakehouses (S3/MinIO, HDFS).

- Work with table formats like Iceberg, Hudi, or Paimon for ACID transactions and schema evolution.

- Optimize SQL queries on Trino/Hive for large-scale analytics.

- Develop orchestration workflows using DBT, Dagster, or Airflow for data transformations.

- Write efficient code in Python, Scala, and Java for data processing and automation.

Tech Stack :

- Streaming : Apache Kafka, Apache Flink, Spark Structured Streaming

- Programming : Python, Scala, Java

- Data Platform : Iceberg, Hudi, Paimon, Hive, Trino

- Databases : MongoDB, Cassandra, Redis

- Orchestration : Airflow, DBT, Dagster

- Cloud & DevOps : CI/CD, Linux, Git

Requirements :

- Graduation in Computer Science/IT (preferably BE/B.Tech) or equivalent.

- 3 - 6 years of experience in data engineering.

- Strong analytical and problem-solving skills.

- Ability to work in 24x7 environments.