Big Data Engineer - Hadoop/Hive

Description :

- Utilize Hadoop, Spark, and SparkSQL technologies to build efficient and high-performance data pipelines.

- Leverage Python for data manipulation, automation, and integration tasks.

- Collaborate with cross-functional teams to gather data requirements and deliver optimized solutions.

- Participate in the design, development, and deployment of Big Data applications and frameworks.

- Ensure data quality, integrity, and security across big data environments.

- Stay current with the latest trends and advancements in Hadoop and Spark technologies.

- Optimize system performance and troubleshoot issues related to big data infrastructure.

Roles and Responsibilities :

- Design and develop scalable Big Data solutions using the Hadoop and Spark ecosystems.

- Implement and optimize SparkSQL queries for performance and efficient resource utilization.

- Develop Python scripts for data ingestion, transformation, and workflow automation.

- Monitor and maintain Hadoop clusters to ensure high availability and reliability.

- Collaborate with data engineers, analysts, and stakeholders to deliver actionable insights.

- Conduct code reviews and provide technical guidance to junior team members.

- Participate in capacity planning, system tuning, and performance benchmarking.

- Document technical specifications, processes, and best practices for big data projects.