Data Engineer - ETL/Apache Airflow

Job Description :

Roles and Responsibilities :

• Solicit business, functional, non-functional and technical requirements through interviewing and requirements gathering process.
• Analyze and document above requirements and data definitions, perform data analysis, assist in change management, training and testing efforts.
• Works with stakeholders to gather requirements on merging, de-duplicating, standardizing data.
• Develop, support, and refine new data pipelines, data models, business logic, data schemas as code, and analytics to product specifications.
• Prototype and optimize data type checks to ensure data uniformity prior to load.
• Develop, and refine batch processing data pipeline frameworks.
• Maintain, improve, and develop expertise in existing production data models, and algorithms.
• Learn and utilize business data domain knowledge and its correlation to underlying data sources.
• Define, document, and maintain a data dictionary including data definitions, data sources, business meaning and usage of information.
• Identify and validate opportunities to reuse existing data and algorithms.
• Collaborate on design and implementation of data standardization procedures.
• Share team responsibilities, such as contributing to development of data warehouses and productizing algorithms created by Data Science team members.
• Participate in on-call and weekly shift rotation.

• 4 - 6 years of experience building data pipelines and using ETL tools(Must-have).
• 2+ years of experience in ETL tools like Talend /Jaspersoft ETL tools(Must-have) .
  2+ years of experience in SQL programming language (Must-have) .
  
• Strong in writing stored procedures and sql queries(Must-have).
  
• 2+ years of experience in python programming (Must-have) .
  
• Sound knowledge of distributed systems and data processing with spark.
  
• Knowledge of any tool for scheduling and orchestration of data pipelines or workflows (preferred Airflow)(must to have)
  
• 1+ years experience developing modern, industry standard big data frameworks with AWS or other cloud services(Must-have).
  
• Experience with common GitHub developer practices and paradigms.
  
• Experience working with agile methodologies and cross-functional teams.
  
• Knowledge in building AWS data pipelines using python, S3 data lake(nice to have).
  
• Knowledge of redshift or any other columnar database is preferred.
  
• Experience with AWS services including S3, Redshift, EMR (nice to have)
  
• Knowledge of distributed systems as it pertains to data storage and computing
  
• Knowledge of specialty pharmaceutical and retail pharmacy is a plus.
  
• Good to have knowledge of the Data integration process.
  
• Ability to effectively communicate with both business and technical teams
  

• Excellent verbal, written communication.
• Flexible with the working timing.