Roles and Responsibilities:-

- Develop, Monitor and Maintain data pipeline .

- Create and maintain optimal data pipeline architecture

- Assemble large, complex data sets that meet functional / non-functional business requirements.

- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.

- Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.

- Incident diagnosis, routing, evaluation & resolution

- Analyze the root cause of incidents.

- Create incident closure report

Requirements :

- BE Degree in computer science or equivalent

- Min 3 years of Experience in Data management

- Experience with data modeling, data warehousing, and building ETL pipelines

- Hands on experience on Spark SQL and Spark streaming

- Hands on experience with Airflow or Luigi

- Comfortable working with python and shell scripts

- Good understanding of Hadoop Ecosystem

- Experience with Data warehouses like RedShift and Databases like Postgres, MariaDB .

- Ability to implement webhooks if required

Immediate joining, need candidate within 15 or 30 days.