Job Summary :

As a Big Data Engineer, you will work with talented data scientists, business analysts, software developers and product engineers in a geographically decentralized team across multiple locations in APAC. You will bring in industry best-practices around data pipelines. You will perform ETL on large and complex datasets.

Key Responsibilities :

- To build a Big Data Platform for APAC region

- To develop data pipelines to load different kind of data (both structured and unstructured data) to HADOOP system

- To migrate existing datasets from Data warehouse to Data Lake.

- Build views in hive based on the business needs.

- Establish connection between various source systems from and to hive.

- Prepare and maintain the inventory, mappings, documentation, assets and data dictionary for the platform.

- Conduct training / support end user and technical team on how to use hue and establish connection to data lake for automation.

- To work closely with our BI Team and users to understand their needs and provide necessary data for their reporting and analysis.

- Perform Q and A, unit testing for all use cases by preparing proper test cases.

- Provided L2 support for the workflows built by Data Management team.

Required Skills and Experience :

- Experience in ETL / Data Engineering / Data Warehouse

- Solid experience in Python, PySpark and SQL or ETL tools

- Experience in handling huge volume of data is a must

- Solid experience in structured and unstructured data in traditional and Big Data environments - Oracle / SQL server / MongoDB / HIVE / PostgreSQL.

- Experience in Adapting governance principles and coding best practices is a must.

- Experience in Setting up connecting to application interaction, Hue Client is an advantage.

- Experience with Cloud deployment and CI/CD pipeline building is a plus

- Experience with indexing engines like Indexima is a plus

- Experience in Data Cataloging would be an advantage

- Experience in Data Governance Principle, Ranger Policy is a plus.

Preferred Qualifications :

- Bachelor or master's degree in computer science, big data or related areas

- 5+ years' experience in data engineering role

- Ability to take ownership, acting as an individual contributor and delivering within timelines.

- Clear communicator.

- Proactive and should have clear design skills.

- Experience in requirement gathering and testing.