Apache Flink & Kafka Developer

Description :

Key Responsibilities :

- Apache Flink (Advanced - Mandatory)

- Design, develop, and optimize large-scale stateful stream-processing applications using Apache Flink.

- Implement stream processing semantics including event time, processing time, and ingestion time.

- Configure and manage watermarks, late data handling, and windowing strategies.

- Set up and manage checkpointing, savepoints, and Flink state backends (e.g., RocksDB).

- Tune Flink jobs for performance, scalability, and fault tolerance.

- Deploy and manage Flink clusters using YARN or Kubernetes environments.

Kafka Development & Administration (Mandatory) :

- Design and manage high-throughput Kafka-based messaging systems.

- Define and optimize partitioning strategies, producer/consumer configs, and message delivery guarantees.

- Manage and configure Kafka Connect, Schema Registry, and related ecosystem components.

- Perform Kafka broker tuning, cluster maintenance, upgrades, and optimizations.

- Monitor and manage consumer lag, offsets, cluster health, and throughput.

DevOps & Observability :

- Work with DevOps teams to support deployment and scaling of Flink and Kafka clusters on Kubernetes, YARN, or cloud platforms.

- Implement monitoring and observability using Grafana, Prometheus, and other open-source tools.

- Troubleshoot performance issues across streaming pipelines, broker clusters, and deployment environments.

Required Skills & Qualifications :

- 6+ years of hands-on experience in Flink and Kafka design, development, and administration.

- Strong expertise in real-time data pipelines, event-driven architectures, and stateful stream processing.

- In-depth understanding of distributed systems, partitioning, offset management, and fault tolerance.

- Experience with ci/cd, containerization, and orchestration (Docker, Kubernetes).

- Strong programming skills in Java/Scala/Python for Flink development.

- Experience integrating Flink/Kafka systems with data lakes, microservices, or analytics platforms.

- Strong debugging, performance tuning, and problem-solving abilities.

- Excellent communication and collaboration skills.

Good to Have :

- Experience with cloud-native streaming (AWS MSK, GCP Pub/Sub, Azure Event Hubs).

- Knowledge of Flink SQL, Kafka Streams, or Pulsar.

- Exposure to schema evolution, Protobuf/Avro, and data governance