Lead / Senior Microsoft Fabric Data Engineer

Location : Bangalore (On-site, with flexibility for 12 remote days)

Job Type : Permanent

Experience Level : 8+ years in data engineering overall, with 2+ years in a leadership/architect capacity

About the Role :

We are seeking an experienced Lead Microsoft Fabric Data Engineer to drive our initiative to modernize our legacy BI platform onto Microsoft Fabric. This is a hands-on technical leadership role : you will architect the core Fabric solution yourself, set practical engineering standards, and guide a small team through a phased migration. We expect most candidates to bring strong, deep Azure Synapse/ADF/Databricks experience plus 1.52+ years of practical, hands-on Microsoft Fabric work.

Key Responsibilities :

1. Technical Leadership & Hands-On Architecture :

- Own the end-to-end technical design of the Fabric Lakehouse/Warehouse solution, and remain hands-on in build for the first 2-3 quarters.

- Set pragmatic engineering standards and coding conventions for the Fabric platform (naming, folder structure, notebook/pipeline patterns).

- Act as the team's go-to person for Fabric - evaluate new features/capacity options and recommend what's actually useful for our workloads.

- Mentor 2-4 mid-level data engineers day to day on Fabric and Spark/SQL best practices.

2. Lakehouse & Warehouse Design :

- Design the Lakehouse/Data Warehouse architecture in Fabric, including medallion (Bronze/Silver/Gold) layout.

- Define workspace and domain structure, full enterprise data-mesh patterns are a plus, not a prerequisite.

- Right-size Fabric capacity (F-SKU) and monitor/optimize compute and storage cost.

- Choose sensibly between Direct Lake, Import, and Direct Query per use case.

3. Legacy Platform Migration :

- Assess our current BI/DW stack and produce a practical, phased migration plan.

- Lead pilot migrations of 1-2 priority workloads, validate the approach, then scale to the rest.

- Prior experience migrating any legacy platform (SQL Server, Synapse, Teradata, Oracle, Cognos, etc.) to a modern cloud/Fabric target is valued.

- Define simple, trackable success metrics for migration progress and data quality.

4. Core Data Engineering :

- Build and review data pipelines using Fabric Data Factory, Dataflows Gen2, and Notebooks (PySpark/Spark SQL).

- Implement incremental loads, CDC, and SCD patterns.

- Where relevant, build real-time ingestion with Event streams and KQL Database - deep Real-Time Intelligence expertise is a plus, not a must-have.

- Apply core performance practices : partition design, file/table optimization, query tuning.

- Stand up basic automated data-quality checks (schema, null/volume checks).

5. Security & Governance :

- Implement workspace security, item-level permissions, and row-level security (RLS) for the workloads we own.

- Apply sensible data classification.

- Follow existing company security/compliance policy; flag gaps to the security/compliance team.

6. DevOps & Automation :

- Set up Git integration and a working branching strategy for Fabric development.

- Build CI/CD for Fabric deployments using Azure DevOps or GitHub Actions and the Fabric REST APIs.

- Script routine deployment/admin tasks in PowerShell or Python.

- Full infrastructure-as-code (Terraform/Bicep) and Kubernetes are nice-to-have.

7. Stakeholder Collaboration :

- Work directly with business stakeholders and Power BI developers/analysts to translate requirements into working pipelines and models.

- Report progress and blockers clearly to engineering leadership.

- Run lightweight design reviews with the team rather than formal architecture review boards.

8. Continuous Improvement :

- Keep the team current on new Fabric features as they roll out.

- Share learnings internally - a short write-up or demo after each major milestone is enough; formal thought-leadership/publishing is a bonus.

Required Skills & Experience :

Microsoft Fabric (Must-Have) :

- 1.5-2+ years of genuine hands-on Fabric experience (production or serious pilot work) - note this is close to the realistic ceiling given Fabric's Nov 2023 GA date.

- Working knowledge of Lakehouse, Warehouse, Data Factory (Fabric), Dataflows Gen2, and Notebooks.

- Practical understanding of OneLake, workspace management, and Fabric capacity basics.

- Comfortable choosing between Direct Lake, Import, and DirectQuery for a given scenario.

Azure & Data Platform :

- 8+ years in data engineering, including 4+ years hands-on with Azure data services (Synapse Analytics, ADF, ADLS, or Databricks).

- Strong PySpark, Spark SQL, Python, and T-SQL skills.

- Solid grasp of Delta Lake and Parquet; distributed-computing fundamentals.

- Most candidates will be Azure Synapse/ADF veterans who moved into Fabric over the last 1-2 years - that profile is exactly what we want.

Architecture & Design :

- Experience designing data warehouse/lakehouse solutions for a mid-to-large organization.

- Solid dimensional modelling skills; data vault or data mesh exposure is a plus.

- Basic FinOps awareness for cloud data cost control.

Migration Experience :

- Has led or been a key contributor on at least 1 legacy-to-cloud data platform migration (any source : on-prem SQL Server, Oracle, Teradata, Synapse, Cognos, Tableau, etc.).

- Comfortable doing lightweight current-state assessment and migration planning.

Security & Governance :

- Practical experience with RLS/OLS/CLS and workspace-level security in Power BI/Fabric or Synapse.

- General familiarity with data governance concepts (classification, lineage) - deep Purview administration experience is a plus.

DevOps & Automation :

- Experience with Azure DevOps or GitHub Actions for CI/CD.

- Working knowledge of Fabric REST APIs, PowerShell, or Python for automation.

Certifications :

- Microsoft Certified : Fabric Analytics Engineer Associate (DP-600).

- Microsoft Certified : Fabric Data Engineer Associate (DP-700).

- Microsoft Certified : Azure Solutions Architect Expert (AZ-305).

Preferred Qualifications :

- Experience in manufacturing, retail, or healthcare data environments.

- Power BI administration and Premium/Fabric capacity management experience.

- Exposure to AI/ML integration with Fabric (Azure ML, Copilot, OpenAI) a strong plus.

- Comfort working in agile/Scrum delivery.

Leadership & Soft Skills :

- Able to lead and mentor a small team (2-4 engineers) by example, staying hands-on rather than purely directive.

- Clear communicator who can explain technical trade-offs to both engineers and business stakeholders.

- Comfortable operating with a fair amount of ambiguity - this is a build-as-we-go modernization, not a fully scoped enterprise program.

- Pragmatic, delivery-focused - biased toward shipping a working pilot over designing the perfect long-term architecture upfront.

Key Deliverables :

- Current-state assessment and a phased migration plan for priority workloads.

- Core Lakehouse/Warehouse architecture in Fabric, agreed with the team.

- Workload migrated and validated end-to-end.

- Baseline engineering standards, Git/CI-CD setup, and initial security model in place.

Why Join Us? :

This is a chance to build a modern Fabric-based data platform largely from scratch, with real ownership over the architecture and hands-on influence from day one. You'll work on one of Microsoft's newest and fastest-evolving data platforms, grow into a broader platform-leadership role as the team scales, and help shape how the organization does data engineering going forward.