Senior Big Data Engineer

Sr. Big Data Engineer

Location : Pune

Experience : 8+ years overall; 6+ years relevant

Mode : Hybrid

Role Overview :

We are seeking a talented Sr. Big Data Engineer to design, develop, and support a highly scalable, distributed SaaS-based Security Risk Prioritization product. You will lead the design and evolution of our data platform and pipelines, providing technical leadership to a team of engineers and architects.

Key Responsibilities :

- Provide technical leadership on data platform design, roadmaps, and architecture.

- Design and implement scalable architecture for Big Data and Microservices environments.

- Drive technology explorations, leveraging knowledge of internal and industry prior art.

- Ensure quality architecture and design of systems, focusing on performance, scalability, and security.

- Mentor and provide technical guidance to other engineers.

Required Skills & Technologies :

Mandatory : Kafka, Scala, Spark

.
- Big Data & Data Streaming : Spark, Kafka, Hadoop, Presto, Airflow, Data lakes, lambda architecture, kappa, and kappa ++ architectures with flink data streaming.

- Databases & Caching : RDBMS, NoSQL, Oracle, Cassandra, Redis.

- Search Solutions : Solar, Elastic.

- ML & Automation : Experience with ML models engineering and related deployment, scripting, and automation.

- Architecture : In-depth experience with messaging queues and caching components.

Other Skills : Strong troubleshooting and performance benchmarking skills for Big Data technologies.

- 8+ years of total experience, with 6+ years relevant.

- 2+ years in designing Big Data solutions with Spark.

- 3+ years with Kafka and performance testing for large infrastructure.