HamburgerMenu
hirist

Asrah - Senior Data & AI Engineer

Asrah Technologies
6 - 11 Years
Pune

Posted on: 16/11/2026

Job Description

Job Summary:

We are looking for a Senior Data Engineer to design, build, and operate our enterprise data platform on Azure and Databricks, with strong hands-on skills in GenAI/LLM application engineering. This role owns the full lifecycle of data pipelines and AI-powered applications from ingestion and orchestration to governance, API development, and deployment.

Key Responsibilities:

GenAI & LLM Application Engineering:

- Design and build RAG pipelines, including embedding-based and graph/metadata-grounded retrieval approaches.

- Orchestrate multi-step and multi-agent LLM workflows using LangGraph and LangChain.

- Build and integrate tools/agents using MCP for structured tool-calling and context management.

- Develop backend APIs for AI applications using FastAPI and Python Flask.

- Build and manage LLM/agent workflows using Azure AI Foundry, and implement semantic + vector search using Azure AI Search indexes.

- Implement prompt engineering, structured outputs, guardrails, and LLM evaluation/observability practices.

Data Engineering & Pipeline Development:

- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark, Spark SQL).

- Develop and maintain Databricks Lakeflow (Delta Live Tables) pipelines across bronze/silver/gold medallion layers.

- Use Databricks Autoloader for incremental, schema-evolving file ingestion into Delta Lake.

- Build reusable, config-driven ingestion frameworks supporting multiple heterogeneous source systems.

- Implement CDC, SCD Type 1/2, and incremental loading strategies.

- Package and deploy Databricks workflows using Databricks Asset Bundles (DABs).

Platform, Infrastructure & DevOps:

- Own infrastructure provisioning and deployment using Terraform for Databricks, ADF, and related Azure resources.

- Build and maintain Azure DevOps CI/CD pipelines for automated testing, deployment, and release management.

- Deploy and manage applications on Azure Container Apps, using Docker for containerization.

- Develop and support Databricks Apps for internal data tooling.

Governance, Security & Reliability:

- Implement and maintain data governance using Databricks Unity Catalog.

- Design and enforce security policies across managed tables.

- Build monitoring, alerting, and SLA dashboards to ensure pipeline reliability and data quality.

- Perform Spark/Databricks performance tuning, cluster sizing, partitioning, and cost optimization.

Collaboration:

- Partner with data architects, analysts, and business stakeholders to translate requirements into scalable data and AI solutions.

- Document architecture, pipelines, and operational runbooks.

- Mentor junior engineers on Databricks, Python development, and platform engineering best practices.

Preferred Candidate Profile:

- Strong hands-on experience with Azure Data Factory, Databricks (PySpark, Spark SQL, Delta Lake, Autoloader, Lakeflow, DABs), and Unity Catalog.

- Proficiency in Databricks Apps, Azure Container Apps, Docker, and Terraform.

- Strong Python development skills, including API development with FastAPI and Flask.

- Hands-on experience with LLM application engineering (RAG, LangGraph, LangChain, MCP).

- Experience with Azure AI Foundry, Azure AI Search, and vector databases.

- Experience with CI/CD using Azure DevOps and API integration.

info-icon

Did you find something suspicious?

Similar jobs that you might be interested in

Loading chat...