Job Description - Senior Spark Engineer (Python)

Location : Bangalore/Remote

Employment Type : Full-Time

Experience Required : 10+ Years

About the Role :

We are looking for highly experienced Spark experts with deep expertise in Spark internals and advanced Python programming skills. This is a senior engineering role for professionals who have built and optimized large-scale distributed data processing systems and can work hands-on with complex Spark architectures and performance tuning.

This is an urgent hiring requirement.

Key Responsibilities :

- Design, develop, and optimize large-scale distributed data processing applications using Apache Spark

- Work extensively on Spark internals, execution engine optimization, memory management, partitioning, and performance tuning

- Develop scalable and production-grade data pipelines using Python and Spark

- Analyze and resolve performance bottlenecks in distributed systems

- Collaborate with engineering and architecture teams to build high-performance data platforms

- Review, refactor, and improve existing Spark codebases for scalability and efficiency

- Drive best practices around distributed computing, coding standards, and system reliability

Required Skills & Experience :

- 10+ years of overall software engineering experience

- Expert-level knowledge of Apache Spark internals

- Strong hands-on expertise in PySpark and advanced Python programming

- Deep understanding of distributed systems and big data processing

- Strong experience with Spark optimization, DAG execution, shuffling, caching, serialization, and cluster tuning

- Experience working with large-scale data processing environments

- Strong debugging, problem-solving, and system design skills

- Excellent communication and collaboration abilities

Mandatory Requirement :

- Candidates must be willing to share previously written Spark code for technical evaluation and review as part of the assessment process.

Preferred Qualifications :

- Experience with Hadoop ecosystem and cloud platforms

- Exposure to distributed computing frameworks and data engineering best practices

- Prior experience in performance-critical data platforms is highly preferred