Data Engineer - Scala/Spark

Talentonlease

Bangalore

6 - 8 Years

6+ Reviews

ETL Python Spark Data Analytics Data Modeling Hadoop Scala

Posted on: 10/11/2025

Job Description

Description :

- Big Data Orchestration : Airflow, Spark on Kubernetes, Yarn, Oozie

- Big Data Processing : Hadoop, Kafka, Spark & Spark Structured Streaming

- Experience on SOLID & DRY principles with Good Software Architecture & Design implementation experience

- Advanced Scala experience (e. Functional Programming, using Case classes, Complex Data Structures & Algorithms)

- Proficient in developing automated frameworks for unit & integration testing

- Proficient with Kubernetes, Docker, Helm and related container technologies

- Proficient in deploying and managing Spark workloads on Kubernetes clusters

- Candidates should have hands on experience in Spark and Scala at least 5+ years with good data engineering concepts

- Design, develop, and maintain data pipelines and ETL workflows for large-scale distributed systems.

- Implement and manage Big Data orchestration solutions using Apache Airflow, Oozie, Spark on Kubernetes, and YARN.

- Work extensively on data processing frameworks including Hadoop, Kafka, Spark, and Spark Structured Streaming.

- Develop and optimize real-time and batch data processing systems ensuring scalability, reliability, and high performance.

- Apply SOLID and DRY software engineering principles to ensure maintainable and high-quality code.

- Architect and implement data solutions using best practices for software design and distributed system patterns.

- Write efficient, modular, and testable code using Scala (Functional Programming, Case Classes, Data Structures & Algorithms).

- Develop automated unit and integration testing frameworks to ensure code quality and data pipeline reliability.

- Manage Spark workloads on Kubernetes clusters and use Docker, Helm, and other container technologies for deployment automation.

- Collaborate with cross-functional teams including Data Scientists, DevOps, and Business Analysts to ensure data availability and consistency.

- Monitor, troubleshoot, and optimize data workflows for performance and cost efficiency.Proficient in Big Data orchestration tools such as Airflow, Spark on Kubernetes, Yarn, Oozie.

- Strong experience with Hadoop, Kafka, Spark Structured Streaming, and related Big Data ecosystems.

- Proficiency in Functional Programming and advanced Scala concepts including case classes, higher-order functions, and immutability.