Role Summary :

We are hiring a Senior Data Engineer to drive the development of scalable, high-performance data pipelines for a migration project to a modern Azure Native architecture. This role will be instrumental in building an AI-ready data platform using Microsoft Fabric (Fabric IQ) and enabling seamless integration with AI/ML workflows (Foundry IQ).

Key Responsibilities :

- Design and build robust, scalable data pipelines (batch + near real-time) using Microsoft Fabric (Data Pipelines, Lakehouse).

- Lead migration of existing pipelines (AKS/custom ETL) to Fabric-native architecture.

- Implement Medallion Architecture (Bronze , Silver , Gold) for structured and unstructured healthcare data.

- Develop and optimize data models and transformations for analytics and AI use cases.

- Build ingestion frameworks for APIs (FHIR, EHR systems) and Files (JSON, HL7, CSV).

- Ensure data quality, validation (FHIR/claims), and observability across pipelines.

- Enable AI/ML workflows by preparing datasets for RAG pipelines and feature engineering.

- Optimize pipeline performance, cost, and scalability.

- Collaborate with Architects, AI/ML engineers, and Product teams to translate requirements into data solutions.

- Mentor junior engineers and drive engineering best practices.

Technical Skills :

- Languages : Python, PySpark, SQL (advanced).

- Azure / Fabric : Microsoft Fabric (preferred) OR Azure Data Factory, Synapse, Data Lake.

- Data Architecture : Lakehouse, Medallion Architecture, Batch + streaming data processing.

- Data Handling : JSON, Parquet, CSV, API integrations.

- Performance Optimization : Partitioning, indexing, query tuning.

Nice to Have :

- Experience with healthcare data (FHIR, claims, EHR).

- Exposure to RAG pipelines / vector databases / AI data prep.

- Familiarity with event-driven architectures.

- Understanding of data security & governance in regulated environments.

What Were Looking For :

- Strong hands-on engineer with ownership mindset.

- Ability to operate in fast-paced, evolving architecture environment.

- Problem-solver with focus on performance, scalability, and data quality.

- Mentor and team player.