Description :

Kafka Consultant

Experience : 812 Years Shift: Coverage until 11:00 AM PST (IST Evening/Night Overlap)

Role Summary :

We are seeking a high-caliber Kafka Consultant to architect, automate, and govern our enterprise-scale streaming infrastructure. This is a technical leadership role designed for an expert who bridges the gap between Apache Kafka internals and modern DevOps automation.

You will be responsible for the end-to-end lifecycle of Confluent Platform (7.x) clusters, utilizing Ansible (cp-ansible) and Kubernetes to build resilient, CI/CD-driven pipelines.

Your expertise will extend beyond simple cluster management into the orchestration of Active-Active DR setups, Cluster Linking, and Schema Registry management.

Responsibilities :

- Cluster Orchestration & CI/CD: Build, scale, and manage enterprise Kafka clusters using automated CI/CD tools and Kubernetes container orchestration.

- End-to-End Automation: Utilize Ansible and custom scripting to automate repetitive tasks, including topic lifecycle management, connector deployment, and RBAC provisioning.

- Disaster Recovery Architecture: Design and implement robust DR and Active-Active cluster configurations using Cluster Linking and Schema Linking for global data consistency.

- Confluent Platform Governance: Lead the administration of Confluent Platform 7.x, governing Schema Registry, KSQL, and security protocols.

- Connector Ecosystem Management: Architect and configure a diverse range of source/sink connectors, including MQ, JDBC, Splunk, BQ, and Blob storage.

- Performance Engineering: Troubleshoot deep-tier producer/consumer issues, providing developers with precise tuning parameters for throughput and latency optimization.

- Legacy Integration Support: Manage technical touchpoints between Kafka and legacy systems such as IBM MQ, IBM IIB, MIM, and Stonebranch.

- Production Support & RCA: Serve as a primary technical escalation point for high-priority production incidents, driving root cause analysis (RCA) and permanent fix implementation.

- Vendor Management: Interface with technology vendors to resolve complex platform-level bugs and influence product roadmaps.

- Operational Health: Participate in the on-call ROTA, performing proactive health checks and maintaining 99.99% uptime for production messaging streams.

Technical Requirements :

- Kafka Mastery: Expert-level knowledge of Kafka Brokers, partitions, replication factors, and log compaction.

- Confluent Stack: Hands-on experience with Confluent Platform 7.x, including Control Center, Schema Registry, and KSQLDB.

- Automation & Configuration: Deep proficiency in Ansible (specifically cp-ansible) and Linux/Shell for environment automation.

- Containerization: Strong experience managing Kafka workloads on Kubernetes (K8s) and Docker.

- Integration Tooling: Working knowledge of IBM MQ, IIB, and enterprise scheduling tools like Stonebranch.

- Messaging Protocols: Solid understanding of Avro/Protobuf serialization and Schema evolution patterns.

Preferred Skills :

- Pipeline Architect: Proven track record of creating end-to-end automation pipelines that reduce manual intervention by >80%.

- Developer Consultant: Ability to provide architectural "suggestions" to development teams to improve application resilience and messaging semantics.

- Troubleshooting Acumen: Expert at analyzing thread dumps, network latency, and disk I/O to pinpoint Kafka performance degradation.

- Strategic Risk Management: Experience in proactive monitoring and alerting setup using Prometheus/Grafana for streaming health.

- Communication Excellence: Ability to lead technical calls during high-priority incidents and articulate RCAs to senior management.