Senior AI/ML Engineer - Databricks

Description :

Job Description : Senior Engineer - AI/ML (Databricks Certified)

We are seeking a highly skilled AI Engineer with proven experience in developing, deploying, and optimizing AI models using the Databricks AI platform. The ideal candidate will have a strong background in machine learning, distributed data processing, and will be a champion of Databricks platform.

Responsibilities :

- Generative AI : Experience of LLM-based solutions (LlamaIndex, LangChain, RAG pipelines, or similar frameworks). Ability to integrate GenAI and Agentic AI into business workflows.

- Design and implement end-to-end Generative AI solutions on Databricks, leveraging Unity Catalog, MLflow, Delta Lake, and Vector Search

- Design, build, and deploy large-scale AI/ML models using the Databricks environment.

- Implement data validation, lineage, and monitoring using Delta Live Tables and Unity Catalog.

- Leverage Databricks data engineering workflows for feature engineering, model training, and evaluation.

- Optimize training pipelines for efficiency, scalability, and accuracy.

- Integrate AI models into production systems using APIs and microservices.

- Build reusable ML pipelines using Databricks Repos, MLflow, and Feature Store.

- Implement robust testing, monitoring, and retraining protocols for deployed models.

- Ensure adherence to compliance, security, and performance standards.

- Stay updated on advancements in AI frameworks, distributed computing, and Databricks platform updates.

Required Skills and Qualifications :

- Bachelors or Masters degree in Computer Science, Data Science, or related field.

- Cerified Databricks Certified Generative AI Engineer

- Proven experience developing AI solutions on Databricks.

- Strong knowledge of Python, PySpark, MLflow, Spark and Databricks Notebooks.

- Strong understanding and knowlege of Databricks platform features such as Unity Catalog, DLT, MosaicAI, Data Assets Bundles, etc.

- Experience with Transformer-based models, generative AI, and Databricks pipelines.

- Proficiency in integrating AI models with cloud-native architectures (AWS, Azure, or GCP).

- Solid understanding of MLOps practices, Data Assets bundles (CI/CD), and containerization (Docker, Kubernetes) on Databricks

Platform.

- Familiarity with vector databases, embeddings, and retrieval-augmented generation (RAG).

- Strong problem-solving, analytical thinking, and communication skills.

Must have Skills :

- Databricks - 5 Years - Advanced

- Python - 8 Years

- Generative AI - 8 Years - Advanced

- Unity catalogue - 4 Years - Advanced

- ML flow - 4 Years - Advanced

- Delta lake - 5 Years - Advanced

- Vector search - 4 Years - Advanced

- PySpark - 5 Years - Advanced