Job Purpose :

We are seeking an experienced Data Engineer to architect, develop, and manage scalable, resilient, and enterprise-grade data solutions on the Azure cloud ecosystem. The ideal candidate will bring strong hands-on expertise in PySpark, Python, Azure Synapse, and at least one modern orchestration framework such as Dagster, Prefect, Kestra, or Mage AI.

In this role, you will contribute to the development and modernization of large-scale data platforms, ensuring high standards of data quality, performance, reliability, and maintainability. You will be responsible for building robust end-to-end data pipelines, enabling seamless data movement from ingestion through transformation to analytics-ready consumption layers.

Key Responsibilities :

1. Distributed Data Engineering (PySpark) :

- Design, develop, and maintain scalable distributed data pipelines using PySpark for high-volume enterprise datasets

- Build and optimize Spark-based transformations leveraging Spark DataFrames, Spark SQL, and performance tuning best practices

- Implement efficient partitioning, caching, and join strategies to improve processing performance and scalability

- Develop optimized read/write mechanisms for cloud-based object storage environments

- Identify and resolve performance bottlenecks related to data skew, shuffle operations, and memory utilization

- Ensure data pipelines consistently meet defined SLAs for performance, reliability, and data quality

2. Python Development for Data Platforms :

- Develop robust, production-grade Python applications and utilities for data processing and platform operations

- Follow strong software engineering principles including modular design, reusable components, and maintainable code practices

- Implement dependency management, logging frameworks, error handling, and automated testing strategies

- Create orchestration-independent pipeline components and reusable processing frameworks where applicable

3. Azure Synapse & Cloud Data Integration :

- Design and implement scalable data solutions using Azure Synapse Analytics

- Develop and manage Synapse Pipelines for orchestration and data movement across enterprise systems

- Build Spark-based transformation workflows using Synapse Notebooks

- Configure and manage Linked Services, Integration Runtimes, and cloud-based connectivity solutions

- Contribute to enterprise data modernization initiatives, including migration from legacy or on-premise ETL platforms to Azure-native architectures

- Optimize Azure workloads for scalability, reliability, operational efficiency, and cost management

- Integrate Synapse seamlessly with ADLS Gen2, SQL pools, and downstream analytics or BI platforms

4. Data Orchestration & Workflow Automation :

- Design, implement, and manage orchestration workflows using Dagster or similar modern orchestration frameworks such as Prefect, Kestra, or Mage AI

- Develop asset-driven and dependency-aware pipeline orchestration patterns

- Implement event-based and schedule-driven workflows using sensors, triggers, and automated scheduling mechanisms

- Monitor, troubleshoot, and enhance orchestration workflows to ensure operational stability and observability

- Support deployment and operational management of orchestration platforms in managed or cloud-hosted environments

5. Data Engineering Architecture & Modeling :

- Design and build scalable ETL/ELT pipelines aligned with business intelligence and analytics requirements

- Apply strong data modeling concepts including dimensional modeling, curated data layers, and semantic consistency

- Architect and maintain enterprise data lake environments using ADLS Gen2 and medallion architecture principles

- Develop and optimize SQL-based transformations, validations, and analytical datasets

- Establish controls for data quality, schema evolution, lineage tracking, and platform observability

6. Platform Ownership & Engineering Excellence :

- Own the complete lifecycle of data pipelines including development, deployment, monitoring, troubleshooting, and ongoing support

- Implement validation, reconciliation, and recovery mechanisms to ensure data integrity and operational resilience

- Collaborate with analytics, BI, and business teams to deliver reliable and consumption-ready datasets

- Contribute to engineering standards, reusable frameworks, governance practices, and continuous platform improvement initiatives