Role Overview :

We are looking for a Senior Azure Data Engineer with strong expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable, secure, and high-performance data platforms, leveraging modern Lakehouse architecture and real-time data processing capabilities.

Key Skills :

- Databricks : Notebooks (Python, SQL), SQL Editor, Unity Catalog, Delta Lake, Databricks Notification Alerts, Genie Spaces, and performance optimization.

- Azure Data Factory (ADF) for ETL/ELT orchestration.

- Advanced Python, PySpark, and SQL skills with strong understanding of Spark performance tuning.

- Azure Ecosystem : Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure SQL Database, and Azure Event Hub.

- Real-time data ingestion : Structured Streaming, Kafka, Event Hubs, and Auto Loader.

- Architecture : Spark Declarative Pipelines (SDP), Delta Live Tables (DLT), and Medallion Architecture.

- Governance : Data modeling, data warehousing, governance, lineage, and access control.

- DevOps : GitHub, GitHub Actions, Azure DevOps, and CI/CD practices.

- Familiarity with Databricks Asset Bundles (DAB) and Lakeflow Connect.

Responsibilities :

- Design and develop scalable data pipelines using Databricks and ADF.

- Build and maintain Delta Lake tables and Lakehouse solutions for analytics and reporting.

- Implement governance and security using Unity Catalog.

- Optimize Databricks jobs, Spark workloads, and SQL queries for performance and cost efficiency.

- Develop and support real-time and batch data processing solutions.

- Configure Databricks Notification Alerts, SQL dashboards, and Genie Spaces to improve monitoring, self-service analytics, and business user adoption.

- Monitor, troubleshoot, and improve data pipeline reliability and quality.

- Collaborate with stakeholders to deliver business-focused data solutions.

- Mentor junior engineers and promote data engineering best practices.

Preferred Qualifications :

- 5+ years of Azure Data Engineering experience.

- Experience with large-scale distributed data processing systems.

- Databricks and/or Azure Data Engineering certifications preferred.

- Strong communication, collaboration, and problem-solving skills.