Virtusa - Kafka Administrator/Platform Engineer

Description :

Role Overview :

We are seeking an experienced Kafka Administrator / Platform Engineer responsible for the installation, configuration, administration, and operational support of Confluent Kafka clusters across multiple environments. The role involves managing large-scale distributed messaging systems, ensuring high availability, performance optimization, automation, and disaster recovery of Kafka-based data platforms.

The candidate will work closely with DevOps, Infrastructure, Data Engineering, and Application teams to support enterprise-grade data pipelines and streaming applications.

Key Responsibilities :

Kafka Cluster Administration :

- Perform end-to-end administration of Confluent Kafka clusters, including installation, configuration, and upgrades.

- Manage Kafka clusters across Development, UAT, and Production environments.

- Perform cluster migration activities, including data migration, environment replication, and configuration alignment.

- Add/remove nodes from production clusters and perform cluster rebalancing and partition reassignment.

- Manage Kafka brokers, topics, partitions, replication factors, and retention policies.

- Configure and manage Confluent platform components including :

- Kafka Brokers

- Zookeeper / KRaft

- Schema Registry

- Kafka Connect

- Control Center

- REST Proxy

- ksqlDB

Cluster Monitoring & Support :

- Perform day-to-day Kafka monitoring activities as part of the platform support team.

- Use monitoring tools such as :

- Confluent Control Center

- Cloudera Manager

- Prometheus

- Grafana

- Splunk

- Perform performance monitoring, troubleshooting, and root cause analysis for Kafka cluster issues.

- Investigate Kafka Streams and streaming application performance issues.

- Analyze logs and metrics to identify bottlenecks and improve system reliability.

- Integrate monitoring with AppDynamics for Kafka log analysis and metrics visibility.

Infrastructure & Platform Management :

- Deploy and manage Kafka clusters on both On-Premises and Cloud environments.

- Work with infrastructure teams to ensure network, storage, and compute prerequisites for Kafka deployments.

- Manage Confluent Kafka license configuration and compliance across UAT and Production environments.

- Support Hybrid infrastructure setups including :

- On-Prem Hadoop

- Spark clusters

- Azure Databricks

- Enterprise data pipeline environments.

DevOps & Automation :

- Implement Infrastructure as Code (IaC) practices using:

- Terraform

- Concourse

- Ansible

- Develop and maintain automation scripts for Kafka cluster management and health checks.

- Build proactive monitoring and alerting systems to detect issues before impacting production.

- Setup and maintain CI/CD pipelines for Kafka infrastructure and streaming deployments.

Containerization & Orchestration :

- Deploy and manage Kafka workloads using container technologies including:

- Docker

- Kubernetes

- Support container orchestration strategies for scalable and resilient Kafka infrastructure.

- Manage container-based Kafka deployments in enterprise environments.

Disaster Recovery & High Availability :

- Design and implement Kafka Disaster Recovery (DR) strategies.

- Configure multi-cluster replication and failover mechanisms.

- Conduct regular DR failover drills to ensure cluster resilience.

- Ensure business continuity through high availability architecture and replication strategies.

Cluster Maintenance & Operations :

- Perform routine cluster maintenance activities across all environments.

- Maintain environment parity between Non-Production and Production environments.

- Implement operational standards and best practices for Kafka platform governance.

- Perform capacity planning and scaling activities based on usage patterns.

Data Pipeline Support :

- Support enterprise data pipelines built on Kafka integrating with :

- Hadoop

- Spark

- Azure Databricks

- Data Lake platforms

- Work with data engineering teams to ensure reliable message ingestion and processing.