Roles and Responsibilities:

- Design, construct, and manage scalable Lakehouse architecture on ADLS Gen2 and Azure Databricks following medallion pattern (Bronze, Silver, Gold).

- Develop metadata-driven ETL and ELT ingestion pipelines using Azure Data Factory, Databricks Workflows, and Structured Streaming.

- Create vector-ready data products and embeddings via Azure AI Search to enable Retrieval-Augmented Generation (RAG) and LLM applications.

- Standardize open table formats on Delta Lake while optimizing performance using partitioning, Z-Ordering, liquid clustering, and file compaction.

- Implement automated data governance, cataloging, lineage tracking, and role-based access control using Unity Catalog and Microsoft Purview.

- Ensure strict adherence to regulatory standards, data privacy, and security frameworks including PCAOB, NIST AI RMF, and PII/PHI protection.

- Optimize platform compute, storage, and pipeline execution costs to balance scale, reliability, and cost-per-workload efficiency.

- Build CI/CD data pipelines, automated testing, and observability monitors for pipeline freshness, schema drift, and data quality checks.

Preferred Profile:

- Work Experience: Minimum 4+ years of hands-on experience in building production-grade data engineering platforms, pipeline architecture, and data lakes.

Technical Expertise:

- Core Technologies: Azure Databricks, Azure Data Factory, ADLS Gen2, Python, PySpark, SQL, Delta Lake.

- Governance & Security: Unity Catalog, Microsoft Purview, PII segregation.

- AI & Advanced Data: Azure AI Search, Vector Databases, Embeddings, RAG architecture.

- Bonus Skills: dbt, Delta Live Tables, Terraform, Kafka, Event Hubs, Apache Iceberg.

- Educational Qualification: Bachelors or Masters degree in Computer Science, Data Science, Engineering, or relevant technical field.

- Domain Preference: Professional services, consulting, fintech, or regulated industries handling complex audit/financial datasets.