CloudAngles - Data Engineer - Python/Azure Data Factory

Description :

Job Summary :

We are seeking a highly skilled Data Engineer with expertise in leveraging Data Lake architecture and the Azure cloud platform to develop, deploy, and optimize data-driven solutions. You will play a pivotal role in transforming raw data into actionable insights, supporting strategic decision-making across the organization.

Key Responsibilities :

- Build and orchestrate data workflows with Azure Data Factory

- Design and implement solutions using Azure Databricks and Synapse Analytics

- Manage and maintain data storage solutions in Azure Data Lake Storage Gen2, leveraging cost-efficient architecture

- Implement and manage CI/CD pipelines using Azure Logic Apps, with integration of tools such as Azure DevOps, GitHub Actions, or Jenkins

- Model and maintain Medallion Architecture across Bronze, Silver, and gold layers based on business requirements

- Collaborate with data scientists, analysts, and business stakeholders to ensure reliable data availability

- Monitor and optimize performance and cost efficiency of data solutions across the ecosystem

Required Skills & Experience :

- Hands-on experience with Azure Data Factory, Azure Databricks, and Synapse Analytics

- Familiarity with Azure Logic Apps for CI/CD pipeline management

- Knowledge of CI/CD tools such as :

i. Azure DevOps Pipelines

ii. GitHub Actions

iii. Jenkins

- Expertise in managing Azure Data Lake Storage Gen2 environments with emphasis on security and cost optimization

- Deep understanding of Medallion Architecture principles :

i. Bronze Layer : Raw data ingestion

ii. Silver Layer : Cleansed, enriched, and business-ready data

iii. Gold Layer : Aggregated and analytics-ready data models

- Strong problem-solving and communication skills

- Experience on Cost optimization