Axis Securities - Data Engineer - Apache Airflow

Description :

Role & Responsibilities :

- Build, develop, and maintain efficient and high-performance data pipelines across both cloud and on premises environments.

- Ensure the accuracy, adequacy, and legitimacy of data.

- Prepare ETL pipelines to extract data from various sources and store it in a centralized location.

- Analyse, interpret, and present results through effective visualization and reports.

- Identify critical metrics.

- Implement and instil best practices for effective data management.

- Monitor the use of data systems and ensure the correctness, completeness, and availability of data services.

- Optimize data infrastructure and processes for cost efficiency on AWS cloud and on premises

environments.

- Utilize Apache Airflow, NiFi, or equivalent tools to build and manage data workflows and integrations.

- Implement best practices for data governance, security, and compliance.

- Monitor and troubleshoot data pipeline issues to ensure timely resolution.

- Stay current with industry trends and emerging technologies in data engineering and cloud computing.

Preferred candidate profile :

- Proven experience as a Data Engineering Lead or in a similar role.

- Extensive hands-on experience with ETL and ELT processes.

- Strong expertise in data integrity and quality assurance.

- Proficiency in optimizing AWS cloud services and on-premises infrastructure for cost and performance.

- Hands-on experience with Apache Airflow and NiFi.

- Strong programming skills in languages such as Python, Java, or Scala.

- Experience with SQL and NoSQL databases.

- Experience in building and maintaining a single source of truth.

- Familiarity with data warehousing solutions like Amazon Redshift, Snowflake, or BigQuery.

- Strong problem-solving skills and the ability to work under pressure.

- Hands-on experience with data visualization tools such as Tableau, Power BI, or Looker.

- Experience in financial services is a must.

Educational Qualifications :

- Bachelors degree in Computer Science, Information Technology, Data Science, or a related field.