Data Engineer - SQL/Python

Role : Data engineer.

Looking for imm to 30days joiners.

Skills : Data engineer, python, astronomer airflow or Apache airflow, data pipeline, etl process.

Additional Troubleshooting, SQL,- Web application or mobile application.

Skills Required :

- Design, develop, support, and maintain data pipelines using Python and Astronomer Airflow to support data flow into customer-facing digital applications.

- Collaborate with members of the delivery team, and other stakeholders to understand data requirements for digital products and implement solutions accordingly.

- Build ETL processes to extract, transform, and load data from various sources into data stores that feed into digital applications.

- Ensure the reliability, scalability, and performance of data pipelines, with a focus on minimizing latency and maximizing data quality.

- Monitor and maintain data pipelines, proactively identifying and resolving issues to ensure consistent data delivery to applications.

- Implement data validation and quality checks to ensure data accuracy and integrity within our digital platforms.

- Continuously optimize and enhance data workflows and processes to support evolving product and business needs.

- Document data engineering processes, including pipeline design, data flows, and operational procedures.

- Stay up-to-date with the latest trends and best practices in data engineering, particularly in relation to digital applications.

- Performing gap analysis to identify improvement opportunities.

- Engage in Agile ceremonies, primarily within the delivery team as well as the wider Genus IT Agile Release Train.

- Contribute to analysis, documentation and requirements gathering within the ABS Digital team.

- Participate in code reviews, providing constructive feedback to peers and ensuring high code quality.

- The above list is not exhaustive, and the post holder will undertake such other duties as required.

Qualifications and Experience :

- Bachelor's degree in Computer Science, Data Engineering, or a related field, or equivalent practical experience.

- 3+ years of experience as a Data Engineer, with a focus on supporting data pipelines for digital applications.

- Proficiency in Python programming and experience with Astronomer Airflow or Apache Airflow, including DAG creation, workflow management, and scheduling.

- Solid understanding of ETL processes and data integration techniques.

- Experience working with SQL and relational databases and data warehousing solutions.

- Familiarity with cloud platforms, specifically Microsoft Azure and data-related services.

- Understanding of data architecture principles and best practices for data management in customer-facing applications.

- Strong problem-solving skills, with the ability to troubleshoot and resolve data pipeline issues quickly.

- Experience working with data pipelines in a customer-facing digital environment, such as web or mobile applications.

- Knowledge of data governance, privacy, and security best practices.

- Experience with containerization and orchestration tools like Docker.

- Understanding of data modelling and API integration for digital applications.

- Strong problem-solving skills, with attention to detail and a commitment to delivering high-quality work