Senior Big Data Engineer - Python

Description :

- We are looking for a skilled Python Developer with strong Big Data experience for our client company to design, develop, and optimize data-intensive applications. The ideal candidate will work on large-scale data processing, build reliable data pipelines, and collaborate with cross-functional teams to deliver scalable and high-performance solutions.

Key Responsibilities :

- Design, develop, and maintain Python-based applications for large-scale data processing

- Build and optimize ETL/data pipelines using Big Data frameworks

- Work with structured and unstructured data from multiple sources

- Develop batch and real-time data processing solutions

- Ensure data quality, performance, and reliability of pipelines

- Collaborate with data engineers, analysts, and product teams

- Perform code reviews and follow best coding practices

- Troubleshoot and resolve performance and data issues

Required Skills & Qualifications :

- Strong proficiency in Python (data structures, OOP, performance tuning)

- Hands-on experience with Big Data tools/frameworks such as :

1. Apache Spark (PySpark preferred)

2. Hadoop, HDFS

3. Kafka (real-time data streaming)

- Experience with SQL & NoSQL databases (Hive, HBase, Cassandra, MongoDB, etc.)

- Strong knowledge of data modeling, ETL concepts, and data warehousing

- Experience working with Linux/Unix environments

- Familiarity with REST APIs and data integration patterns

- GCP Services like Composer, data proc, pub sub and big query good to have