Job Summary :

We are seeking a highly skilled Technical Architect with deep expertise in real-time data streaming platforms, especially within the Apache Kafka ecosystem, to lead the architecture and implementation of scalable, secure, and high-throughput event-driven systems. This role involves end-to-end solution design and optimization of streaming data pipelines using technologies like Kafka, Kafka Connect, Spark Streaming, Flink, Beam, and cloud-native tools such as Google Cloud Pub/Sub.

You will work with cross-functional teams, including data engineers, DevOps, product managers, and cloud architects, to define robust and high-performance streaming architectures that power data-intensive applications and analytics.

Key Responsibilities :

Kafka Architecture & Management :

- Design and architect scalable, fault-tolerant, and high-throughput streaming systems using Apache Kafka.

- Build and manage Kafka clusters for high availability, scalability, disaster recovery, and performance.

- Optimize cluster configuration parameters, partitioning strategies, consumer group behavior, and replication factors.

Streaming Pipeline Development :

- Architect and integrate Kafka Streams, Kafka Connect, Apache Flink, Beam, and Spark Streaming to enable complex real-time data processing pipelines.

- Design end-to-end pipelines that support ingestion, transformation, enrichment, validation, and delivery of streaming data across systems.

Platform Integration & Ecosystem :

- Implement and manage integrations with Schema Registry, Kafka Connect connectors, GCP Pub/Sub, and third-party systems.

- Drive seamless data interoperability across operational databases, data lakes, cloud services, and analytics platforms.

Security, Monitoring, and Compliance :

- Enforce security best practices for Kafka, including authentication, authorization (ACLs), TLS encryption, and audit logging.

- Implement observability solutions using monitoring tools (e.g., Prometheus, Grafana, Confluent Control Center, etc.) to track cluster health, latency, throughput, lag, and performance.

- Ensure compliance with enterprise policies and governance standards.

Collaboration & Leadership :

- Translate business and technical requirements into scalable Kafka-based architectural solutions.

- Collaborate with development and DevOps teams to automate Kafka deployments using infrastructure-as-code (e.g., Terraform, Ansible, Helm).

- Serve as a subject matter expert, mentor, and advisor for engineering teams on Kafka and stream processing best practices.

- Stay current with industry trends, contribute to architectural roadmaps, and evaluate emerging technologies in the real-time data space.

Required Technical Skills :

- Apache Kafka (core) architecture, broker configuration, topic management, partitioning, replication, consumer groups, offset management.

- Kafka Connect development and management of source/sink connectors (custom and Confluent-provided).

- Schema Registry schema design (Avro/Protobuf), evolution, compatibility settings.

- Kafka Streams, Apache Flink, Beam, Spark Streaming hands-on experience in real-time stream processing.

- Cloud Platforms especially GCP (Google Cloud Pub/Sub, Dataflow, BigQuery); exposure to AWS or Azure is a plus.

- Strong understanding of distributed systems, event-driven architecture, message serialization formats (Avro, Protobuf, JSON), and data consistency models.

- Hands-on with monitoring, logging, and alerting tools for real-time systems.

- Proficient in Java or Scala; Python is a bonus.

- Experience with containerization and orchestration tools like Docker, Kubernetes, and Helm.