Job Description - Data Engineering Lead / Senior Developer

Location : Pune (Candidates should currently be based in Pune and available for Face-to-Face interview)

Work Mode : Hybrid

Notice Period : Immediate Joiners Preferred

Job Summary :

We are looking for experienced Data Engineering professionals with strong expertise in AWS, Scala, Redshift, Apache Spark, and SQL. The ideal candidate should have hands-on experience building scalable data pipelines and working with large-scale distributed data systems.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Apache Spark

- Write efficient, reusable, and optimized code using Scala and/or Python (PySpark)

- Process and manage large-scale structured and unstructured datasets

- Perform complex data transformations, aggregations, and joins across multiple data sources

- Optimize Spark jobs for performance and resource utilization

- Work with distributed storage systems such as S3 and HDFS

- Debug and troubleshoot production data issues

- Collaborate with data engineers, analysts, and business stakeholders

- Ensure data quality, consistency, and reliability across systems

- Support CI/CD implementation and deployment processes

Required Skills :

- Strong hands-on experience with Apache Spark

- Expertise in Scala and/or Python (PySpark)

- Strong SQL skills

- Experience with AWS services such as S3, EMR, Glue, Kinesis, Firehose, Hive, and Redshift

- Good understanding of Spark internals - RDD, DataFrame, Dataset APIs

- Experience with data formats such as Parquet, Avro, and JSON

- Knowledge of distributed systems and big data concepts

- Experience in Spark performance tuning and optimization

- Familiarity with CI/CD pipelines

- Exposure to Spark Streaming

- Experience with workflow orchestration tools like Apache Airflow

Preferred Candidate Profile :

- Strong problem-solving and analytical skills

- Excellent communication and stakeholder management abilities

- Ability to work in a fast-paced hybrid environment

- Experience handling enterprise-scale data engineering projects