Kafka Engineer

Description :

Role and Responsibilities :

- Design, build, and manage Kafka clusters using Confluent Platform and Kafka Cloud services (AWS MSK, Confluent Cloud).

- Develop and maintain Kafka topics, schemas (Avro/Protobuf), and connectors for data ingestion and processing pipelines.

- Monitor and ensure the reliability, scalability, and security of Kafka infrastructure.

- Collaborate with application and data engineering teams to integrate Kafka with other AWS-based services (e.g., Lambda, S3, EC2, Redshift).

- Implement and manage Kafka Connect, Kafka Streams, and ksqlDB where applicable.

- Optimize Kafka performance, troubleshoot issues, and manage incident response.

Required Skills :

- Strong knowledge of Kafka internals (brokers, zookeepers, partitions, replication, offsets).

- Experience with Kafka Connect, Schema Registry, REST Proxy, and Kafka security.

- Hands-on experience with AWS (EC2, IAM, CloudWatch, S3, Lambda, VPC, Load balancers).

- Proficiency in scripting and automation using Terraform, Ansible, or similar tools.

- Familiarity with DevOps practices and tools (CI/CD pipelines, monitoring tools like Prometheus/Grafana, Splunk, Datadog etc).

- Experience with containerization (Docker, Kubernetes) is a plus.

Nice to Have :

- AWS Certified

- CICD (AWS Code Pipeline, Harness)

- Containers ( Docker, Kubernetes)