L4 Senior Data Software Engineer - Spark/Python/Databricks

Description :

Role Overview :

We are seeking a Senior Data Software Engineer (L4) with deep experience building scalable data platforms using Apache Spark, Python, and Azure/AWS Databricks.

The ideal candidate will drive complex architecture decisions, lead development and optimization efforts, and mentor junior team members.

Key Responsibilities :

- Lead end-to-end solution design and development of Big Data platforms

- Build and optimize distributed systems using Spark and Hadoop at scale

- Architect and build high-throughput streaming systems using Spark Streaming or Storm

- Build and optimize data engineering workloads on Databricks (Azure preferred)

- Drive performance tuning of Spark applications and distributed workloads

- Implement ETL/ELT pipelines for batch and streaming data

- Integrate diverse structured/unstructured datasets from RDBMS, ERP, and NoSQL sources

- Provide technical leadership and mentoring to L2/L3 engineers

- Collaborate cross-functionally with data science, engineering, and product teams

- Implement best practices, automation, CI/CD, and governance for data engineering

Required Skills :

- Apache Spark - Expert level (core, SQL, streaming)

- Python - strong hands-on engineering capability

- Hadoop ecosystem : Hadoop v2, MapReduce, HDFS, Sqoop

- Streaming : Spark Streaming / Storm

- Messaging : Kafka / RabbitMQ

- SQL - advanced (stored procedures, complex joins, optimization)

- Hive / Impala querying

- NoSQL : HBase / Cassandra / MongoDB

- Databricks (Azure preferred / AWS acceptable)

- Strong knowledge of distributed computing fundamentals

- Experience working in Agile