Role : Senior Data Engineer - Azure, Databricks & GenAI

About the Role :

We are looking for a Senior Data Engineer to design, build, and operate an enterprise-scale data platform on Microsoft Azure and Databricks, with strong hands-on experience in Generative AI, LLMs, and RAG-based applications.

The role will own the end-to-end lifecycle of data pipelines and AI-powered applications, working closely with engineering, product, and business teams to build scalable, reliable, and production-ready data and AI solutions.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data processing solutions using Azure and Databricks.

- Build and optimize data engineering workflows using PySpark and Python.

- Develop and manage enterprise data platforms, ensuring scalability, reliability, security, and performance.

- Design and implement ETL/ELT pipelines for structured and unstructured data.

- Build GenAI/LLM-powered applications, including RAG (Retrieval-Augmented Generation) solutions.

- Work with LLMs, embeddings, vector databases, prompt engineering, and retrieval pipelines.

- Integrate LLM capabilities with enterprise data and business applications.

- Develop, deploy, monitor, and optimize AI-powered applications in production.

- Implement data quality, validation, monitoring, and governance practices across pipelines.

- Optimize Databricks workloads and Spark jobs for performance and cost efficiency.

- Collaborate with cross-functional teams to translate business requirements into scalable data and AI solutions.

- Troubleshoot production issues and ensure high availability and reliability of data and AI workloads.

- Stay updated with emerging technologies across Data Engineering, GenAI, LLMs, and Azure.

Required Skills & Experience :

- Strong hands-on experience in Data Engineering and building enterprise-scale data platforms.

- Strong proficiency in Python and PySpark.

- Hands-on experience with Azure data services and Databricks.

- Experience designing and developing ETL/ELT data pipelines.

- Strong understanding of data engineering concepts, distributed computing, and data architecture.

- Hands-on experience with Generative AI, LLMs, and RAG architectures.

- Experience working with LLM application development, embeddings, vector search, and retrieval pipelines.

- Understanding of data modeling, data quality, and data governance.

- Experience with cloud-based data platforms and production-grade engineering practices.

- Strong problem-solving and analytical skills.

- Ability to work independently and collaborate effectively with technical and business stakeholders.

Preferred Skills :

- Experience with Azure services such as Azure Data Factory, Azure Data Lake, Azure OpenAI, and Azure AI services.

- Experience with MLflow and Databricks-based AI/ML workflows.

- Experience with vector databases and semantic search.

- Knowledge of LLM frameworks and orchestration tools.

- Experience deploying AI applications using APIs and microservices.

- Understanding of CI/CD, DevOps, monitoring, and cloud security.

Education :

- B.Tech / B.E. in Computer Science, Information Technology, Engineering, or a related field.