Roles and Responsibilities :

- Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).

- Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.

- Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.

- Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.

- Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).

- Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.

- Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume, schema drift).

- Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.

Technical Expertise :

- Core Cloud/Data : Azure Databricks, ADLS Gen2, Azure Data Factory, Synapse.

- Languages & Querying : Strong proficiency in Python, PySpark, Spark SQL, and SQL.

- Storage Formats : Delta Lake, Parquet, Avro, JSON.

- AI Grounding : Vector Search, RAG Pipelines, Azure AI Search, Embeddings, Chunking.

- Governance & DevOps : Databricks Unity Catalog, Microsoft Purview, CI/CD (Azure DevOps/GitHub Actions), Infrastructure-as-Code (Terraform/Bicep).

Requirements :

- 4+ years of hands-on experience building production data pipelines and data platforms.

- Proven track record designing lakehouse architectures on Azure cloud infrastructure.

- Candidate must be willing to relocate near to Goregaon, Mumbai Location.