The Analytics Data Engineer will be responsible for designing, developing, and maintaining robust data pipelines and infrastructure for processing large volumes of structured and unstructured data. The role involves working closely with data scientists, analysts, and business teams to enable advanced analytics and machine learning capabilities within the financial services domain.

Key Responsibilities :

- Design and implement data architecture solutions following best practices for building scalable and maintainable data lakes.

- Develop and optimize batch and streaming data pipelines using Spark and related big data technologies.

Build, manage, and optimize data ingestion and processing workflows on platforms like Data Lake, Delta Lake, and Hive.

- Write complex, efficient SQL queries to extract and manipulate data from large datasets stored in RDBMS and OLAP databases such as MySQL and Redshift.

- Collaborate with data scientists and analytics teams to support ML engineering requirements and productionize ML models.

- Ensure proper data lifecycle management by overseeing processes related to data collection, access, usage, storage, transfer, and deletion in compliance with internal policies and regulatory requirements.

- Work in an Agile environment, participate in sprint planning, stand-ups, and retrospectives, and collaborate cross-functionally to meet project goals.

- Monitor, troubleshoot, and improve data pipeline performance and reliability.

- Keep abreast of emerging trends in big data technologies, financial services analytics, and data engineering best practices.

Required Qualifications & Skills :

- Experience: 2-5 years in Data Engineering or ML Engineering, preferably within NBFCs, commercial banking, investment banking, or financial services.

- Strong proficiency in Python, Scala, or Java for large-scale data processing.

- Hands-on experience with Apache Spark (batch and streaming), Data Lake, Delta Lake, and Hive.

- Deep understanding of batch and streaming data processing architectures and best practices.

- Expertise in writing complex, optimized SQL queries on large datasets.

- Experience with RDBMS and OLAP databases such as MySQL and Amazon Redshift.

- Knowledge of data lifecycle management from data ingestion through secure storage and compliant deletion.

- Familiarity with Agile methodologies and working in iterative delivery cycles.

- Strong analytical, problem-solving, and communication skills.

Preferred Qualifications :

- Prior experience working within NBFCs or financial institutions.

- Exposure to cloud platforms (AWS, Azure, or GCP) and their data services.

- Understanding of data governance, security, and compliance standards applicable to financial data.