Job Description:

About the Role:

We are looking for a Senior Data Engineer to design, build, and operate enterprise-scale data and AI platforms on Microsoft Azure.

You will own the lakehouse and ETL/ELT layer end to end using Microsoft Fabric, Azure Databricks, and Azure Data Factory.

You will also package and ship data services, pipelines, and AI workloads as production-grade Docker containers.

The role suits someone who combines strong data engineering fundamentals with a DevOps mindset and is comfortable extending into Generative AI and RAG solutions.

Key Responsibilities:

- Design and deliver scalable, metadata-driven ETL/ELT pipelines using Azure Data Factory, Fabric Data Factory, Databricks Workflows, and Lakeflow Jobs.

- Build and maintain lakehouse platforms using Medallion Architecture (Bronze, Silver, Gold) on OneLake, ADLS Gen2, and Delta Lake.

- Develop high-quality PySpark, Spark SQL, Python, and T-SQL transformations covering incremental loads, CDC, data-quality rules, and business-rule validation.

- Optimize Spark and Delta Lake workloads using AQE, partitioning, join strategies, Z-ORDER, Liquid Clustering, data skipping, and file compaction.

- Containerize data applications, ingestion services, APIs, and pipeline components using Docker.

- Own the full container lifecycle: build, tag, scan, publish, deploy, and monitor.

- Author optimized, secure multi-stage Dockerfiles and docker-compose setups for local development, testing, and integration.

- Manage container images in Azure Container Registry (ACR) and deploy to Azure Kubernetes Service (AKS) or Azure Container Apps.

- Build CI/CD pipelines in Azure DevOps for automated container builds, testing, vulnerability scanning, and release.

- Use Fabric Deployment Pipelines and Databricks Asset Bundles for platform deployments.

- Implement data governance and security using Unity Catalog, Microsoft Entra ID, RBAC, Managed Identity, and Azure Key Vault.

- Build RAG pipelines and AI Agent workflows (document parsing, chunking, embeddings, vector search, model serving), and deploy agent and retrieval services as containerized microservices.

- Enable self-service analytics through SQL Endpoints, Power BI datasets, and natural-language interfaces such as Databricks Genie.

- Own production support: SLA monitoring, incident troubleshooting, root-cause analysis, and performance tuning.

- Mentor junior engineers, conduct code reviews, and contribute to engineering standards and best practices.

Required Skills & Experience:

Data Engineering Core:

- 8+ years in data engineering, ETL/ELT, and data warehousing.

- Strong hands-on experience with Azure Databricks (Delta Lake, Auto Loader, Unity Catalog, Lakeflow/DLT, Databricks Workflows).

- Strong hands-on experience with Microsoft Fabric (OneLake, Lakehouse, Data Warehouse, Fabric Data Factory, Notebooks, Dataflows Gen2, SQL Endpoint).

- Expert in Azure Data Factory, Azure Synapse Analytics, and ADLS Gen2.

- Advanced proficiency in Python, PySpark, Spark SQL, and SQL/T-SQL, including stored procedures, query tuning, and performance optimization.

- Proven experience with Spark and Delta Lake performance tuning at scale.

- Solid grounding in dimensional modelling and data-quality frameworks.

Docker & Containerization (Strong, Hands-on):

- 3+ years of production experience with Docker, covering image design, layering, caching, and size and security optimization.

- Proficient in writing multi-stage Dockerfiles, docker-compose, and container networking, volumes, and environment/secret management.

- Experience containerizing Python and PySpark applications, data ingestion services, REST/FastAPI services, and AI/RAG microservices.

- Hands-on experience with Azure Container Registry (ACR), plus deployment on AKS or Azure Container Apps.

- Working knowledge of Kubernetes fundamentals (pods, deployments, services, config maps, secrets, scaling).

- Container security practices: base-image hardening, non-root users, image scanning (e.g., Trivy or Microsoft Defender for Cloud), and dependency management.

- Ability to integrate container builds and deployments into CI/CD pipelines.

- Experience with Databricks custom containers or containerized job runtimes is a strong plus.

DevOps & Governance:

- Azure DevOps, Git, and CI/CD pipelines for data and container workloads.

- Infrastructure-as-code exposure (Terraform or Bicep), Databricks Asset Bundles, and Fabric Deployment Pipelines.

- Unity Catalog, Entra ID, RBAC, Managed Identity, and Key Vault.

Good to Have:

- Generative AI and Agentic AI: RAG, Azure OpenAI / Azure AI Foundry, Azure AI Search, Databricks Mosaic AI Vector Search and Model Serving, LangChain/LangGraph, MLflow, and Model Context Protocol (MCP).

- Power BI and dashboard development.

- Experience in BFSI or financial-services data platforms.

- Exposure to SSIS/SSRS and legacy-to-cloud migration.

- Observability tooling for containers and pipelines (Azure Monitor, Log Analytics, Prometheus/Grafana).

Certifications (Preferred):

- Microsoft Certified: Fabric Data Engineer Associate (DP-700)

- Any of: Azure Data Engineer Associate (DP-203), Databricks Data Engineer Professional, Azure Developer (AZ-204), Certified Kubernetes Application Developer (CKAD)

Education:

Bachelor's degree in Engineering, Computer Science, or a related field.

Soft Skills:

- Strong ownership and end-to-end accountability for production platforms

- Clear communication with business and technical stakeholders

- Structured problem-solving and troubleshooting

- Mentoring and collaborative working style