Hadoop Engineer - Scala/Spark

Job Description :

We are seeking an experienced Hadoop Engineer with a strong background in Big Data technologies to join our data engineering team in Bangalore. The ideal candidate will have a deep understanding of the Hadoop ecosystem and hands-on experience designing, developing, and maintaining large-scale distributed data processing systems.

Key Responsibilities :

- Design, build, and maintain big data pipelines using Hadoop ecosystem components

- Develop and optimize MapReduce, Hive, Pig, and Spark jobs

- Work with cross-functional teams to gather requirements and deliver scalable data solutions

- Manage and monitor Hadoop cluster performance, security, and resource utilization

- Implement data ingestion processes using tools such as Sqoop, Flume, or Kafka

- Develop solutions to handle structured, semi-structured, and unstructured data

- Automate data pipeline workflows and monitor failures or inconsistencies

- Troubleshoot production issues and ensure high availability and reliability of big data infrastructure

Required Skills & Experience :

- Minimum 8 years of experience working with Hadoop and Big Data technologies

- Strong hands-on experience with components such as HDFS, Hive, Spark, YARN, Oozie, HBase, and Kafka

- Proficient in Java, Scala, or Python for data processing

- Solid understanding of SQL and data modeling concepts

- Experience in deploying and maintaining Hadoop clusters in production environments

- Familiarity with Linux shell scripting and system-level monitoring

- Knowledge of data governance, security, and compliance best practices

- Excellent problem-solving and communication skills

Preferred Qualifications :

- Experience with cloud-based big data platforms like AWS EMR, Azure HDInsight, or GCP

Dataproc

- Familiarity with CI/CD, Git, and DevOps practices in a data environment

- Bachelors or Masters degree in Computer Science, Engineering, or a related field