Description :

Experience : 7+ years | Lead / Senior Role

About the Role :

Were looking for a Lead Data Engineer with strong hands-on expertise in PySpark, real-time data processing, and the Microsoft Azure ecosystem. Youll architect and develop large-scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems.

This is a hands-on leadership role where youll design, optimize, and scale a modern data platform that drives analytics and operational insights enterprise wide.

Key Responsibilities :

- Design and implement large-scale data pipelines using PySpark, Azure Databricks, and Microsoft Fabric

- Handle and process massive volumes of real-time data from diverse, high-velocity sources

- Build and optimize real-time and batch ingestion frameworks for structured and semi-structured data

- Develop event-driven architectures using Azure Functions, Event Hub, and Data Factory

- Implement Medallion architecture (Bronze/Silver/Gold) and strong data validation frameworks

- Ensure pipeline scalability, performance tuning, and fault tolerance for high-throughput workloads

- Enforce data governance, lineage, and security across all data layers

- Build CI/CD pipelines and implement observability for data workflows

- Mentor a team of data engineers and drive engineering best practices

Required Skills :

- 7+ years in Data Engineering, including 3+ years in a lead or architecture role

- Proven expertise in PySpark and Python for large-scale data processing

- Hands-on experience processing massive real-time data streams in production environments

- Strong proficiency in Microsoft Azure, including :

- Azure Databricks, Microsoft Fabric, Data Factory, Event Hub / Service Bus

- Delta Lake on ADLS Gen2

- Advanced knowledge of SQL, data modeling, schema evolution, and data quality frameworks combined with design and architecture skills

- Experience building event-driven, serverless data architectures

- Strong understanding of data privacy, lineage, and governance principles

Nice to Have :

- Domain expertise in financial services

- Familiarity with Microsoft Purview, Azure Monitor, and Key Vault

- Pre-sales experience desirable