Big Data Engineer - Hadoop/Spark

Job title : Big Data Engineer

Location : Bangalore

Role Type : Full time

Shift : UK Shift (1:00 10:00 PM)

Experience : 7-10 years

Key Responsibilities :

- Design, develop, and maintain scalable data processing pipelines using Hadoop and Spark.

- Implement data integration and ETL processes to ingest and transform large datasets.

- Collaborate with data scientists, analysts, business partners and other stakeholders to understand data requirements and deliver solutions.

- Optimize and tune Hadoop and Spark jobs for performance and efficiency.

- Manage and maintain data storage solutions, ensuring data integrity and security.

- Utilize GitHub for version control and collaboration on code development.

- Work with CDP (Cloudera Data Platform) to manage and deploy data applications.

- Integrate and manage data solutions on Cloud Azure, and Snowflake ensuring seamless data flow and accessibility.

- Monitor and troubleshoot data processing workflows, resolving issues promptly.

- Stay updated with the latest industry trends and technologies in big data and cloud computing.

- Explore and develop skills in newer technologies according to big data technology roadmap.

- Integrate and manage data solutions on Cloud Azure and on-premises infrastructure, ensuring seamless data flow and accessibility.

Required Qualifications :

- Strong experience with Apache Spark, Hadoop ecosystem, Hive, Iceberg, HBase, Apache Kafka, Spark Streaming or Flink, and Oozie (Airflow/NiFi good to have).

- Proficient in Scala, Java, and/or Python with solid coding and debugging skills.

- Applied experience with Solr for indexing and search workloads.

- Familiarity with Git/GitHub, GitHub Actions, Azure DevOps; good to have cloud experience (Azure/AWS).

- Knowledge of SQL, data modeling (nice to have), and experience working in UNIX or UNIX like environments.

- Experience with SCRUM or similar Agile frameworks; strong problem-solving skills; able to produce clear technical documentation.

- Self-starter, effective individual contributor, and good to have Oil & Gas domain knowledge.