Data Engineer - Scala/Python

Essential Duties and Responsibilities :

- Data Lake Organization : Structure data lake assets using medallion architecture following the Domain-drive and Source-driven approach. (10%)

- Data Pipeline Design and Development : Develop, deploy and orchestrate data pipelines using Data factory, pyspark/ sql notebooks and ensure smooth data flow from various sources to storage and processing systems (20%)

- Design and Build Data Systems : Create and maintain Candescents data systems, databases, and data warehouses to store and manage large volumes of data (20%)

- Data Compliance and Security : Ensure that data systems comply with security standards and regulations, protecting sensitive information. (10%)

- Collaboration : Work closely with Data Management team (Data architects, data analysts, and other stakeholders) to understand data needs and implement approved data solutions (20%)

- Troubleshooting and Optimization : Troubleshoot data-related issues and continuously

optimize data systems for better performance. (15%)

Requirements :

- Extensive experience in building API-based data pipelines using Azure ecosystem.

- Ability to build, maintain and improve data architecture, data collection and data storage systems.

- Ability to build and orchestrate end-to-end pipelines using Microsoft Fabric stack (ADF/ Dataflows).

- Proficiency in ETL/ELT technologies with focus in Microsoft Fabric stack (ADF, Spark, SQL).

- Proficiency in building data warehouse model (Dimensional Models) utilizing Azure Synapse

and Azure Delta lakehouse.

- Extensive programming experience of data processing languages (SQL/T-SQL, Python or

Scala).

- Expertise in code management utilizing GitHub as primary repository.

- Experience with DevOps practices, configuration frameworks and CI/CD automation tooling.

- Collaborate with report developers/analysts and business teams to improve data models that

feed BI tools and increase data accessibility.