KMCCorp - Big Data Engineer - Hadoop/Scala

Description :

JOB TITLE : Big Data Engineer

Location : Hyderabad

Experience : 5 to 10 Years

About the job :

Job Description :

The successful candidate will work on batch and real-time data pipelines, leveraging technologies such as Scala, Java, and Apache Spark. They will manage distributed systems and cloud resources, ensure security compliance using HashiCorp Vault, orchestrate workflows with Apache Airflow, and investigate production issues by analyzing logs and monitoring metrics.

Responsibilities :

- Develop and maintain batch and streaming data pipelines using Scala and Java.

- Write and execute shell scripts and Yarn commands for Spark job management.

- Manage and optimize big data environments using Apache Spark, EMR, Hadoop, and YARN.

- Utilize cloud storage and container platforms such as Amazon S3 and EKS.

- Implement and manage security features, including HashiCorp Vault and tokenization/encryption protocols.

- Schedule and orchestrate batch workflows using Apache Airflow.

- Conduct root cause analysis and handle production incidents.

- Analyze application, Spark executor, and Dynatrace logs.

- Monitor jobs and EMR clusters using Dynatrace metrics.

- Validate data and set up production alerts.

Requirements :

- Experience with batch and streaming processing.

- Proficiency in distributed systems management.

- Knowledge of cloud platforms (AWS S3, EKS).

- Security expertise (Vault, tokenization, encryption).

- Workflow orchestration (Airflow).

- Strong analytical and troubleshooting skills for production investigation

Desired Skills and Experience :

1. Proven experience as a Java Tech Lead with expertise in micro services architecture.

2. Strong proficiency in Spring Boot for API development and Java programming.

3. Extensive experience with Kafka for building scalable and event-driven systems.

4. Solid understanding of containerization and orchestration tools, such as Docker and Kubernetes.

5. Hands-on experience in implementing and maintaining CI/CD pipelines.

6. Excellent communication skills and ability to collaborate effectively with diverse teams.

7. Strong problem-solving skills and a proactive attitude towards challenges.

8. Familiarity with cloud platforms (e.g., AWS, Azure) for deploying and managing applications.

Education and Certifications :

1. Bachelors or Masters degree in Computer Science or a related field.

2. Relevant certifications in Java, Spring, or Kafka are a plus.