Education :

- Bachelor's or master's degree in Computer Science, Information Technology, Engineering, or a related field.

Experience and Technical Requirements :

- Strong experience in big data engineering, data processing, and distributed computing.

- Proficiency in Apache Spark, PySpark, and Hadoop ecosystem technologies.

- Experience developing scalable data pipelines for batch and real-time data processing.

- Strong programming skills in Python, Scala, Java, or similar languages.

- Hands-on experience with Hive, HDFS, Kafka, and SQL.

- Knowledge of ETL/ELT processes, data ingestion, transformation, and data integration.

- Experience working with structured, semi-structured, and unstructured datasets.

- Understanding of distributed systems, data partitioning, performance tuning, and optimization.

- Familiarity with cloud-based data platforms such as AWS, Azure, or GCP.

- Experience with Git, CI/CD pipelines, and workflow orchestration tools such as Airflow.

Job Responsibilities:

- Design, develop, and maintain scalable big data pipelines and processing frameworks.

- Build efficient batch and streaming data workflows to process large volumes of data.

- Develop data ingestion and transformation processes from multiple source systems.

- Optimize Spark jobs and distributed data processing workloads for performance and scalability.

- Implement data quality checks, validation, and error-handling mechanisms.

- Integrate data from various databases, applications, and external sources.

- Monitor pipeline performance and troubleshoot data processing issues.

- Collaborate with data architects, analysts, and engineering teams to support data platform requirements.

- Maintain technical documentation and follow data engineering best practices.

Required Skills :

- Apache Spark, PySpark, Hadoop, SQL, Python/Scala/Java, ETL, Big Data Processing

Supported Skills :

- Kafka, Hive, HDFS, Airflow, AWS/Azure/GCP, Data Pipeline Optimization, CI/CD