Sign in to confirm you’re not a bot
This helps protect our community. Learn more

AI Factory Platform: AI Infrastructure as a service

Microsoft Reactor
126K subscribers
1.2K views Streamed 2 months ago #MicrosoftReactor #learnconnectbuild
As organizations move from AI experimentation to AI Operationalization, they are hit with several realizations around optimum Token utilization of the Azure Open AI Instances, how to scale and how many AOAI instances to maintain, calculation of chargeback for Azure AI services utilization, rate limiting, observability and monitoring. Also, for large organizations, while experimenting with AI Usecases, there is an overhead cost for creating the required infrastructure and ensuring its compliance with the internal security policies etc. The AI Factory Platform is a scalable and secure environment designed to support the development …...more
...more

Welcome & Housekeeping

0:00

Introducing Priyanka

1:02

What is an AI Factory Platform?

1:43

Challenges in Scaling GenAI Deployments

3:11

Azure OpenAI Regional Availability & PTU Management

6:02

Architecture: Current vs Target State

8:55

Centralized API Gateway & Observability

10:55

Token Tracking with Azure Functions

14:47

Load Balancing & Failover Strategies

18:45

Exponential Backoff & Retry Logic

22:52

Token Limit Policies & Circuit Breakers

24:45

Live Demo: API Management & Logging

27:00

Real-Time Token Usage Monitoring

33:00

Extending to Other Azure AI Services

39:00

Open Source Tools: AI Central & GPT Failover

42:00

Weighted Routing & Advanced Load Distribution

45:00

Chargeback Calculation Flow

48:20

Supported Services & Onboarding Policies

49:54

Final Q&A: AI Studio vs AI Factory

50:45

Wrap-Up & Resources

51:47
AI Factory Platform: AI Infrastructure as a service
As organizations move from AI experimentation to AI Operationalization, they are hit with several realizations around optimum Token utilization of the Azure Open AI Instances, how to scale and how many AOAI instances to maintain, calculation of chargeback for Azure AI services utilization, rate limiting, observability and monitoring. Also, for large organizations, while experimenting with AI Usecases, there is an overhead cost for creating the required infrastructure and ensuring its compliance with the internal security policies etc. The AI Factory Platform is a scalable and secure environment designed to support the development, deployment, and management of AI solutions across our client's organization. This platform enables application developers to request approved AI services, look at shared dashboards for utilization metrics, rate limits and application chargeback costs. There are also routing mechanisms implemented to ensure graceful failover from PTU to PAYG instances, retry with backoff of certain limited Azure Open AI deployments, priority-based routing and weight-based routing with the APIM policies. We also demonstrate how to handle instances scaling, load and traffic management via APIM for busy workloads and how to prevent throttling of Azure OPEN AI instances by chatty applications. We essentially build an AI control tower for organizations to easily and securely scale and manage their GEN AI workloads in different environments. Chapters: 00:00 - Welcome & Housekeeping 01:02 - Introducing Priyanka 01:43 - What is an AI Factory Platform? 03:11 - Challenges in Scaling GenAI Deployments 06:02 - Azure OpenAI Regional Availability & PTU Management 08:55 - Architecture: Current vs Target State 10:55 - Centralized API Gateway & Observability 14:47 - Token Tracking with Azure Functions 18:45 - Load Balancing & Failover Strategies 22:52 - Exponential Backoff & Retry Logic 24:45 - Token Limit Policies & Circuit Breakers 27:00 - Live Demo: API Management & Logging 33:00 - Real-Time Token Usage Monitoring 39:00 - Extending to Other Azure AI Services 42:00 - Open Source Tools: AI Central & GPT Failover 45:00 - Weighted Routing & Advanced Load Distribution 48:20 - Chargeback Calculation Flow 49:54 - Supported Services & Onboarding Policies 50:45 - Final Q&A: AI Studio vs AI Factory 51:47 - Wrap-Up & Resources #MicrosoftReactor #learnconnectbuild [eventID:25731]

Follow along using the transcript.

Microsoft Reactor

126K subscribers