Job Title : Kafka Administrator (Merchant Ecommerce platform)

Location : Noida Sector 62

Employment Type : Full-time

Role Summary :

We are looking for an experienced Kafka Administrator to manage, maintain, and optimize our distributed, multi-cluster Kafka infrastructure deployed in an on-premise environment. This role demands deep knowledge of Kafka internals, Zookeeper administration, performance tuning, and operational excellence in high-throughput, low-latency production systems. Additional exposure to API gateway operations (Kong) and observability tooling is a plus.

Key Responsibilities :

Kafka & Zookeeper Administration :

- Manage multiple Kafka clusters with high-availability Zookeeper setups.

- Perform end-to-end operational support including deployment, configuration, and health monitoring of Kafka brokers and Zookeeper nodes.

- Conduct capacity planning, partition strategy optimization, and topic lifecycle management.

- Implement and manage backup and disaster recovery processes with defined RPO/RTO targets.

- Enforce security configurations, including TLS encryption, authentication (SASL, mTLS), and ACL management.

- Optimize Kafka producer and consumer performance to meet low-latency, high-throughput requirements.

- Plan and execute Kafka and Zookeeper upgrades and patching with minimal/zero downtime.

- Integrate Kafka with monitoring platforms like Prometheus, Grafana, or similar tools.

- Define and enforce log retention and archival policies in line with compliance requirements.

Monitoring & Logging :

- Integrate Kafka metrics and logs with centralized observability and logging tools.

- Create dashboards and alerts to monitor Kafka consumer lag, partition health, and broker performance.

- Collaborate with DevOps/SRE teams to ensure visibility into Kafka services.

Security & Compliance :

- Apply CIS benchmarks and perform automated security scans across Kafka nodes.

- Manage secret and certificate rotation using tools like Vault or AWS ACM, as applicable.

- Support regular vulnerability assessments and ensure timely remediation.

Additional (Nice-to-Have) Responsibilities :

- Collaborate on Kong API Gateway support, especially in areas such as mTLS, monitoring, and certificate management.

- Participate in cross-functional discussions around observability, alerting, and compliance strategies.

Infrastructure Snapshot :

- Kafka Deployments :

- o1 Cluster : 5 Kafka + 5 Zookeeper nodes.

- o5 Clusters : 3 Kafka + 3 Zookeeper nodes each.

- API Gateway (Kong) : AWS EKS with mTLS support (if experience applicable).

Required Skills & Experience :

- 3+ years of hands-on Kafka administration experience in production environments.

- Strong understanding of Kafka internals (broker behavior, ISR, partitions, replication, etc.)

- Proficient in Zookeeper management and configuration.

- Experience with Kafka performance tuning and troubleshooting.

- Familiar with TLS/mTLS, ACLs, SASL, and other Kafka security mechanisms.

- Proficient with monitoring and logging tools (e.g., Prometheus, Grafana, ELK).

- Scripting skills (e.g., Bash, Python) for operational automation.

Preferred Qualifications :

- Experience with API gateways (Kong or equivalent).

- Exposure to Kubernetes-based environments (EKS preferred).

- Familiarity with compliance standards and security hardening practices.

- Experience with IaC tools (e.g., Terraform, Ansible).

What We Offer :

- A mission-critical role in managing large-scale real-time data infrastructure.

- Flexible work environment and opportunities for growth.

- Supportive team and access to modern observability and automation tools.