Role: Hadoop Data Engineer

Experience: 8+ Years

Location: Bangalore

Work Mode: Hybrid

POSITION SUMMARY:

We are seeking a highly skilled and driven Senior Data Engineer to join our growing Data Engineering team. The ideal candidate will have strong expertise in building scalable data solutions, modern cloud-based data platforms, and distributed data processing systems.

ROLES AND RESPONSIBILITIES:

- Design, develop, and maintain data services that integrate seamlessly with LendingClub products and external vendor systems.

- Build and manage scalable data pipelines using modern technologies such as Hadoop, Spark, DBT, Trino, Airflow/Dagster, and cloud-native AWS services.

- Develop solutions on modern data platforms including Databricks and Snowflake.

- Identify opportunities to automate manual processes, optimize data delivery, improve system efficiency, and reduce cloud infrastructure costs.

- Implement and enhance frameworks for Data Quality, Observability, Governance, Lineage, and Monitoring.

- Support production environments, troubleshoot critical issues, and perform root cause analysis (RCA) to ensure platform reliability.

- Follow engineering best practices including Test-Driven Development (TDD), unit and integration testing, documentation, and knowledge-sharing through technical wikis.

REQUIRED QUALIFICATIONS:

- Strong hands-on experience with distributed data systems and big data technologies, including Hadoop, Spark, Hive, DBT, and Airflow/Dagster.

- Minimum 4+ years of production-grade Python development experience, with a focus on data engineering and pipeline development.

- Experience working with cloud platforms, preferably AWS.

- Hands-on experience with modern data platforms such as Databricks and/or Snowflake.

- Proficiency in Git, Jenkins, JIRA, Shell Scripting, and CI/CD practices.

- Solid understanding of Agile methodologies, source control management, automated testing, and Test-Driven Development.

- Experience collaborating with cross-functional teams in fast-paced, dynamic environments.

- Strong analytical thinking, problem-solving abilities, and communication skills.

- Proven track record of designing and delivering scalable, reliable, high-performance data pipelines and data platforms.

EDUCATION:

Bachelor's degree in computer science, Software Engineering, MIS or equivalent combination of education and experience.

KEY SKILLS:

Python, PySpark, Hadoop, Spark, Hive, Data Engineering, Big Data, SQL