Dashboard Design & Analytics Developer - LLM Models

Job Description :

We are seeking a highly skilled Dashboarding and Analytics Developer to join our JedAI team. In this role, you will be responsible for the visualization and development of Key Performance Indicators (KPIs) that are critical to monitoring and enhancing the performance of our generative AI systems.

You will develop and maintain comprehensive dashboards that provide real-time insights into the performance of LLMs, Retrieval Augmented Generation (RAG) systems, safety mechanisms, other generative AI features, billing, token consumption, and many more.

- Dashboard Development: Design, develop, and maintain interactive and user-friendly dashboards for monitoring AI system performance.

- KPI Identification: Collaborate with cross-functional teams to define and implement KPIs related to LLMs, RAG systems, safety protocols, and other AI features.

- Data Visualization: Create clear and insightful visualizations that communicate complex data trends and patterns eGectively to stakeholders.

- Performance Monitoring: Continuously monitor AI system metrics to identify anomalies, performance issues, and areas for improvement.

- Data Analysis: Analyze large and complex datasets to extract meaningful insights that support decision-making processes.

- Collaboration: Work closely with AI engineers, data scientists, and product managers to align dashboard functionalities with project goals.

- Innovation: Stay updated with the latest trends and technologies in data visualization and analytics to introduce innovative solutions.

- Documentation: Maintain thorough documentation of dashboard configurations, data sources, and visualization methodologies.

Details of work :

1. Performance Metrics :

- Latency and Throughput: Monitor the response times and the number of requests processed per unit time to ensure the system meets performance expectations.

- Resource Utilization: Track CPU, memory, disk I/O, and network bandwidth usage to identify bottlenecks or inefficiencies.

2. Model Performance and Drift Monitoring :

Accuracy Metrics : Keep track of model accuracy, precision, recall, F1 score, etc., to ensure the models are performing as expected.

Data and Concept Drift Detection : Monitor for changes in data distribution that could aEect model performance over time.

Feature Importance Tracking : Observe changes in feature importance to understand and explain model predictions.

3. Anomaly Detection :

Implement systems to detect unusual patterns or outliers in data inputs, user behavior, or system performance, which could indicate errors or security issues.

4. Security Monitoring :

- Access Logs : Maintain detailed logs of user access and actions for security auditing.

- Threat Detection : Use intrusion detection systems (IDS) to identify potential security threats.

- Compliance Monitoring : Ensure adherence to regulations like GDPR, HIPAA, or other industry-specific compliance requirements.

5. User Engagement and Feedback :

- Usage Analytics : Analyze how users interact with the system to improve user experience.

- Feedback Collection : Provide mechanisms for users to report issues or suggest improvements.

- Session Tracking : Monitor user sessions to understand behavior patterns and enhance personalization.

- Custom Dashboards : Create dashboards tailored to different stakeholders executives, developers, support teams.

- Scheduled Reports : Automate reporting on key metrics for regular review.

Some tools & skills preferred but does need to check all the boxes :

- Data Visualization Tools : Tableau, Power BI, Grafana, Splunk

- Programming Languages : Python, JQL, SPL

- Data Query Languages : SQL

- Cloud Platforms : AWS, Azure, GCP (Likely if Auto-Scaling is a key responsibility)

- Monitoring Tools : Prometheus, Datadog, New Relic, CloudWatch (AWS), Azure Monitor

- Version Control Systems : Git

- Ticketing Systems : Jira, Zendesk, ServiceNow

- Logging Tools : ELK Stack (Elasticsearch, Logstash, Kibana), Splunk

- CI/CD Tools : Jenkins, GitLab CI, CircleCI, GitHub Actions

- Shift hours - 2 PM to 11 PM IST working hours.

- Provide technical support for security-related issues and implement corrective actions.

- Work closely with functional, technical, and business teams to design and implement secure and scalable SAP solutions.

- Provide training and support to end-users on security policies and GRC usage.