We are looking for an experienced Senior Databricks AI/ML Engineer with expertise in Databricks, Apache Spark, Machine Learning, and Generative AI. The ideal candidate should have hands-on experience building scalable AI/ML solutions, modern data platforms, and advanced analytics pipelines on the Databricks Lakehouse platform. Experience with Teradata, MLflow, and LLM integration is highly desirable.

Key Responsibilities:

- Design, develop, and optimize AI/ML solutions using the Databricks Lakehouse Platform.

- Build scalable data pipelines using Apache Spark (PySpark/Spark SQL) and Delta Lake.

- Develop and deploy machine learning models using MLflow and Databricks Machine Learning.

- Build and integrate LLM-based and Generative AI solutions for enterprise use cases.

- Perform feature engineering, model training, evaluation, and deployment for predictive analytics and time-series use cases.

- Design and optimize SQL-based analytics and reporting solutions.

- Support Teradata integration, migration, and performance optimization where required.

- Monitor and optimize cluster performance, workloads, and resource utilization on Databricks.

- Develop reusable Python libraries and automation scripts for data engineering and ML workflows.

- Collaborate with Data Scientists, Data Engineers, Product teams, and business stakeholders to deliver scalable AI-driven solutions.

- Implement CI/CD, model governance, and MLOps best practices.

- Prepare technical documentation and ensure compliance with security and data governance standards.

Required Skills:

- 8-13 years of experience in Data Engineering, Machine Learning, or AI Engineering.

- Strong hands-on expertise with the Databricks Platform (Workspaces, Clusters, Jobs, Unity Catalog, Databricks SQL).

- Extensive experience with Apache Spark (PySpark/Spark SQL) and Delta Lake.

- Strong programming skills in Python, including Pandas, NumPy, and data visualization libraries.

- Advanced proficiency in SQL and large-scale data processing.

- Experience with MLflow, model lifecycle management, and MLOps practices.

- Experience integrating Large Language Models (LLMs) and Generative AI into enterprise applications.

- Strong knowledge of feature engineering, predictive modeling, and time-series analytics.

- Experience with Teradata Administration, DBQL, System Views, and performance tuning.

- Strong analytical, problem-solving, and stakeholder management skills.