Job Title : Senior QA Engineer Microsoft Fabric (Data Engineering & Notebook Validation)

Role Summary :

We are seeking an experienced Senior QA Engineer (7+ years) with strong expertise in Microsoft Fabric, particularly within the Data Engineering and Data Analytics environment.

The ideal candidate will have hands-on experience in :

- Developing and testing notebooks in Fabric (PySpark / Spark SQL)

- Supporting and validating data migrations to Microsoft Fabric

- Designing automated data validation frameworks

- Ensuring data integrity across Lakehouse, pipelines, and Power BI environments

This role requires a technically strong QA professional who understands both backend data validation and automation within modern data platforms.

Key Responsibilities :

Microsoft Fabric & Data Engineering QA :

- Perform end-to-end validation of data pipelines within Microsoft Fabric.

- Validate data across OneLake, Lakehouse, and Fabric SQL endpoints.

- Test Bronze, Silver, and Gold data layers for completeness, accuracy, and transformation integrity.

- Collaborate with Data Engineers to validate transformation logic and incremental load processing.

Notebook Development & Validation :

- Develop and execute validation logic within Fabric notebooks using PySpark and Spark SQL.

- Perform schema validation, row count reconciliation, null checks, duplicate checks, and aggregation validation.

- Build reusable notebook-based validation frameworks.

- Log validation results into Delta tables for auditability and traceability.

Data Migration to Microsoft Fabric :

- Validate data migrations from legacy platforms (AWS, Azure, On-prem, etc.) to Microsoft Fabric.

- Perform before-and-after data comparison between legacy and Fabric environments.

- Validate transformation logic, metadata migration, and incremental load accuracy.

- Support report-level validation post-migration (Power BI comparisons).

QA Automation in Data Platforms :

- Design and implement automated validation frameworks for data pipelines.

- Integrate notebook-based validations into Fabric pipelines.

- Support CI/CD integration for automated QA gating.

- Develop automated checks for :

1. Data completeness

2. Schema consistency

3. Business rule validation

4. Incremental load verification

Collaboration & Reporting :

- Work closely with Data Engineers, BI Developers, and Business Analysts.

- Perform root cause analysis for data discrepancies.

- Document test cases, validation logic, and QA metrics.

- Provide QA sign-off for production releases.

Required Qualifications :

- 7+ years of experience in Data QA, ETL Testing, or BI/Data Engineering validation.

- Strong hands-on experience in Microsoft Fabric (Data Engineering environment).

- Proven experience writing and executing Fabric notebooks (PySpark / Spark SQL).

- Experience supporting data migration projects to Microsoft Fabric.

- Strong SQL skills for backend data validation.

- Experience validating Delta/Parquet data structures.

- Understanding of data quality principles :

1. Accuracy

2. Completeness

3. Consistency

4. Timeliness

- Experience designing or implementing QA automation in data platforms.

- Exposure to CI/CD pipelines and DevOps integration for data testing.

Preferred Qualifications :

- Experience validating Power BI reports post Fabric migration.

- Knowledge of OneLake architecture and Delta Lake format.

- Experience in Azure Data ecosystem.

- Exposure to Python scripting for data validation.

- Experience in regulated or enterprise data environments.

Key Technical Skills :

- Microsoft Fabric (Lakehouse, OneLake, Fabric SQL)

- PySpark

- Spark SQL

- Advanced SQL

- Delta Lake

- Data migration validation

- QA automation frameworks

- CI/CD integration

- Power BI validation (preferred)

What Success Looks Like :

- Successful validation of Fabric migrations with zero critical data defects.

- Automated notebook-based data validation framework in place.

- Reduced manual data testing effort through automation.

- High confidence in data accuracy across reporting and analytics layers.