Job title : Lead Spark+Scala+Kafka engineer

Location : Bangalore

Years of experience : 8 to 15 Years

Joining time : Immediate to 30 Days

Key Responsibilities :

- Develop batch and real-time pipelines using Spark, Scala, and Kafka / Pub/Sub.

- Ingest, transform, and process large volumes of data.

- Work with GCP services such as: BigQuery, Cloud Storage, Cloud compressor, Dataflow/Dataproc

- Ensure data quality, performance, and reliability.

- Understand and manage downstream impacts of data loads (reporting, compliance, data quality).

- Follow best practices for scalable and maintainable data solutions.

- Collaborate with teams to deliver business-ready data platforms.

Required Skills & Experience :

- 8 to 15 years of experience in IT or professional services experience in IT delivery or large-scale IT analytics projects

- Mandatory programming experience in Scala

- Solid experience with Apache Spark.

- Experience with Kafka or Google Pub/Sub for streaming data.

- Experience with ETL and data integration tools (Dataflow, Spark, Dataprep, Python).

- Expertise in at least two of these technologies: Relational Databases, Analytical Databases, NoSQL databases.

- Implement data pipelines to automate the ingestion, transformation, and augmentation of data sources, and provide best practices for pipeline operations.

- Capability to work in a rapidly changing business environment and to enable simplified user access to massive data by building scalable data solutions

- Advanced SQL writing and experience in data mining (SQL, ETL, data warehouse, etc.) and using databases in a business environment with complex datasets

- Ability to work independently and guide teams in a fast- paced environment.

- Certified in Google Professional Data Engineer/ Solution Architect is a major Advantage

Good to Have :

- Strong hands-on experience as a Data Engineer.

- Experience with Python or Java

- Exposure to Machine Learning on GCP

- Google Professional Data Engineer or Solution Architect certification