Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Automated evaluation of LLM apps with the azure-ai-generative SDK | Python Data Science Day

Visual Studio Code
762K subscribers
<__slot-el>
<__slot-el>
668 views 1 year ago
Automated evaluation of LLM Apps with Azure ai-generative SDK with Pamela Fox. Chapters: 00:00 Automated evaluation of LLM apps with the azure-ai-generative SDK 00:55 Types of LLM apps 01:09 Prompt-only LLM app 03:14 Retrieval Augmented Generation (RAG) LLM app 07:05 RAG flow 08:10 Are the answers high quality? 11:21 LLM Ops for LLM Apps 12:46 Experimenting with quality factors 14:55 AI RAG Chat Evaluator: https://aka.ms/rag/eval 16:07 Ground truth data 18:32 Evaluation 25:29 Evaluation approach 25:57 Improving ground truth data sets 26:17 Next steps Resources: Slides: https://aka.ms/rag/eval/slides RAG Chat: https://aka.ms/ragchat RAG Eval: https://aka.ms/rag/eval Demo: https://github.com/Azure-Samples/azur... Prompt-only Repo: https://github.com/f/awesome-chatgpt-... Survey https://aka.ms/Python/DataScienceDay/... Python at Microsoft https://aka.ms/python Cloud Skills Challenge - through April 15, 2024 https://aka.ms/Python/DataScienceDay/CSC GitHub codespaces https://github.com/codespaces VS Code Release notes https://code.visualstudio.com/updates Featuring: Pamela Fox, Python Cloud Advocate, Microsoft (@pamelafox) ...more
...more
Automated evaluation of LLM apps with the azure-ai-generative SDK | Python Data Science Day
5Likes
668Views
2024Mar 28
Automated evaluation of LLM Apps with Azure ai-generative SDK with Pamela Fox. Chapters: 00:00 Automated evaluation of LLM apps with the azure-ai-generative SDK 00:55 Types of LLM apps 01:09 Prompt-only LLM app 03:14 Retrieval Augmented Generation (RAG) LLM app 07:05 RAG flow 08:10 Are the answers high quality? 11:21 LLM Ops for LLM Apps 12:46 Experimenting with quality factors 14:55 AI RAG Chat Evaluator: https://aka.ms/rag/eval 16:07 Ground truth data 18:32 Evaluation 25:29 Evaluation approach 25:57 Improving ground truth data sets 26:17 Next steps Resources: Slides: https://aka.ms/rag/eval/slides RAG Chat: https://aka.ms/ragchat RAG Eval: https://aka.ms/rag/eval Demo: https://github.com/Azure-Samples/azur... Prompt-only Repo: https://github.com/f/awesome-chatgpt-... Survey https://aka.ms/Python/DataScienceDay/... Python at Microsoft https://aka.ms/python Cloud Skills Challenge - through April 15, 2024 https://aka.ms/Python/DataScienceDay/CSC GitHub codespaces https://github.com/codespaces VS Code Release notes https://code.visualstudio.com/updates Featuring: Pamela Fox, Python Cloud Advocate, Microsoft (@pamelafox)

Follow along using the transcript.

Visual Studio Code

762K subscribers