Key Responsibilities :

Architecture & Design :

- Lead the design and implementation of end-to-end Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake.

- Design scalable and maintainable data solutions aligned with enterprise standards.

Pipeline Development & Optimization :

- Build scalable batch and real-time streaming pipelines using PySpark, SQL, and Delta Live Tables (DLT).

- Perform advanced performance tuning including Z-Ordering, Liquid Clustering, partitioning, and Spark UI debugging to reduce compute costs and improve latency.

Data Governance & Security :

- Implement and manage enterprise-grade security and data lineage using Unity Catalog.

- Establish data governance frameworks and best practices across the organization.

Orchestration & DevOps :

- Manage complex workflows using Databricks Workflows or Apache Airflow.

- Implement CI/CD using Databricks Asset Bundles (DABs) and Git integration.

Migration & Integration :

- Lead migrations from legacy systems (Hadoop, On-prem SQL, SAP HANA) to the Databricks Lakehouse.

Technical Leadership & Mentoring :

- Act as the primary point of contact for Databricks architecture decisions.

- Conduct code reviews and mentor junior staff.

- Participate in the development of standards, processes, and frameworks to enhance overall data integration capabilities.

- Continuously stay updated with the latest trends and technologies in the data integration and cloud data platform space.

Core Competencies Required :

Databricks & Platform Expertise :

- Expert-level knowledge of Databricks (Photon engine, Unity Catalog, Serverless SQL, Lakeflow).

- Deep understanding of Delta Lake ACID transactions and Medallion Architecture patterns.

Programming & Data Processing :

- Mastery of Python/PySpark and Advanced SQL.

- Deep understanding of Apache Spark internals and Structured Streaming.

- Scala knowledge is a plus.

Cloud & Infrastructure :

- Extensive experience with at least one major cloud platform (Azure, AWS, or GCP).

- Proficiency with cloud-native storage solutions (ADLS Gen2, S3, GCS).

- Proficiency with Terraform and infrastructure-as-code practices.

DevOps & CI/CD :

- Proficiency with GitHub Actions or Azure DevOps.

- Experience implementing CI/CD pipelines for data engineering workflows.

Data Engineering Specialization :

- Expertise in ELT (Extract, Load, Transform) patterns and reporting frameworks.

- Strong understanding of data ecosystem design and data governance.

- Experience with performance optimization and cost management in cloud data platforms.

Soft Skills :

- Strong communication skills with the ability to explain complex technical concepts to non-technical business leaders.