Description :

Position : Big Data Engineer

Experience : 6-8+ Years

Location : Hyderabad

Job Mode : Work From Office (WFO)

Job Summary :

We are seeking an experienced Big Data Engineer with strong expertise in the Big Data Hadoop Ecosystem and Python-based data engineering to design, develop, and maintain scalable, high-performance data platforms. The ideal candidate will work on large-scale distributed systems, build robust data pipelines, and support analytics and business intelligence needs, preferably within banking and financial services environments.

Key Responsibilities :

- Design, develop, and maintain end-to-end data pipelines using the Hadoop ecosystem, including HDFS, Hive, Spark, and Sqoop.

- Build and optimize large-scale distributed data processing systems to handle structured and semi-structured data.

- Develop ETL and ELT workflows using PySpark, SparkSQL, Python, and Scala to support data ingestion, transformation, and analytics.

- Implement and manage data ingestion frameworks using technologies such as Kafka and/or NiFi (where applicable).

- Write efficient SparkSQL queries and optimize jobs for performance, scalability, and reliability.

- Perform Spark performance tuning, including partitioning strategies, caching, memory optimization, and shuffle reduction.

- Monitor big data jobs and clusters, troubleshoot failures, conduct root cause analysis (RCA), and ensure SLA adherence.

- Participate in capacity planning, job scheduling, and performance monitoring of Hadoop/Spark clusters.

- Ensure data quality, consistency, security, and governance across the data platform.

- Collaborate with data scientists, analysts, business stakeholders, and cross-functional teams to translate business requirements into technical solutions.

- Support production environments, including incident management, job monitoring, and issue resolution.

- Participate in code reviews, promote best practices, and contribute to continuous improvement of data engineering standards.

- Mentor junior engineers and provide technical guidance where required.

- Stay updated with emerging trends and best practices in Big Data, Spark, Hadoop, Kafka, and cloud-based data platforms.

Required Skills & Qualifications :

Technical Skills :

- Strong experience with the Big Data Hadoop Ecosystem (HDFS, Hive, Spark).

- Hands-on experience with PySpark, SparkSQL, and Python.

- Working knowledge of Scala or Java for big data processing.

- Experience building scalable ETL pipelines in big data environments.

- Strong SQL skills for data analysis and reporting.

- Exposure to Kafka and/or NiFi for streaming or data ingestion is a plus.

- Experience in performance tuning and optimization of Spark jobs.

- Knowledge of job scheduling, orchestration, and monitoring tools.

- Familiarity with Linux/Unix environments.

Domain & Functional Experience :

- Preferred experience in Banking / Financial Services / Investment Banking domains.

- Understanding of transactional data, payments, lending, or financial reporting systems is an advantage.

- Experience working in production support or SLA-driven environments is a plus.

Soft Skills :

- Strong problem-solving and analytical skills.

- Good communication skills and ability to work with cross-functional teams.

- Ability to work independently in a fast-paced environment.

- Mentoring mindset and collaborative approach.

Nice to Have :

- Exposure to cloud-based big data platforms (Azure/AWS/GCP).

- Experience with CI/CD pipelines for data engineering workloads.

- Knowledge of data governance frameworks