Job Summary :

We are looking for a high-impact Senior Databricks & Agentic AI Developer to design, build, and scale our next-generation Data & AI platform. This role blends enterprise-grade data engineering on Azure Databricks with production-ready Agentic AI system design, enabling autonomous AI agents to operate on reliable Medallion Architecture foundations (Bronze, Silver, Gold). You will bridge up-to-date data engineering with intelligent AI orchestration to deliver scalable, secure, and cost-efficient AI-driven systems.

Key Responsibilities :

1. Data Engineering (Azure Databricks) :

- Design and implement scalable batch & real-time pipelines using Structured Streaming.

- Build optimized transformations using PySpark & SQL.

- Implement and maintain Medallion Architecture (Bronze/Silver/Gold).

- Develop Delta Lake-based data models and CDC pipelines.

- Performance tuning & cost optimization of Databricks jobs.

- Cluster configuration, autoscaling & job orchestration.

- CI/CD pipeline setup for Databricks workflows.

- Data quality validation, observability & governance implementation.

- Manage Unity Catalog, data lineage & access control.

2. Agentic AI Development :

- Design and implement autonomous AI agents using LangChain, LangGraph, CrewAI.

- Develop multi-agent orchestration workflows.

- Implement RAG (Retrieval-Augmented Generation) pipelines.

- Integrate vector databases for semantic retrieval.

- Build reasoning workflows with structured planning & tool usage.

- Connect AI agents with enterprise data Lakehouse platforms.

- Implement memory, state management & long-running agent flows.

- Develop evaluation pipelines for LLM performance.

3. Productionization, Governance & Responsible AI :

- Deploy AI agents in production-grade cloud environments.

- Implement Responsible AI guardrails (bias, hallucination mitigation).

- Logging, monitoring & observability for AI workflows.

- Secure data access and ensure compliance alignment.

- Version control and CI/CD for data & AI systems.

- Implement audit trails & prompt version management.

Required Qualifications :

1. Experience :

- 5+ years in Data Engineering (Azure Databricks).

- 2+ years in Generative AI / Agentic AI development in production.

- Proven experience building enterprise-grade ETL/ELT systems.

2. Technical Skills - Programming :

- Expert-level Python.

- Strong PySpark.

- Advanced SQL.

3. Databricks & Data Engineering :

- Azure Databricks (Jobs, Workflows, Clusters).

- Delta Lake & Lakehouse architecture.

- Structured Streaming.

- Unity Catalog.

- Data modeling & optimization.

- Performance tuning & cost control.

4. Agentic AI & GenAI :

- LangChain / LangGraph / CrewAI (Mandatory).

- RAG architectures.

- Multi-agent systems.

- Prompt engineering.

- LLM orchestration patterns.

- Memory & stateful agent design.

5. Vector & AI Systems :

- Vector databases (e.g., Pinecone, Weaviate, FAISS, MongoDB Atlas Vector).

- Embedding pipelines.

- Semantic search architecture.

- LLM evaluation frameworks (Ragas, LangSmith, etc.).

6. Cloud & DevOps :

- Microsoft Azure (Data Lake, ADLS Gen2, Key Vault).

- Azure Functions / App Services (preferred).

- Git-based version control.

- CI/CD pipelines (Azure DevOps / GitHub Actions).

- Infrastructure-as-Code (Terraform preferred).

- Containerization (Docker preferred).

- Kubernetes knowledge (nice to have).

Preferred / Nice to Have :

- Experience building enterprise AI platforms.

- MLflow for model tracking.

- Azure OpenAI integration.

- Knowledge of Data Governance frameworks.

- Experience with MCP / agent protocol standards.

- Knowledge of observability tools (Prometheus, Grafana, Azure Monitor).

- Cost optimization strategies for GenAI workloads.

- Experience with microservices architecture.