Demystifying Azure APIM Monitoring for Azure Open AI Endpoints

This blog post stems from a discussion I had with my colleague Srini Padala here at Microsoft, where we delved into the details of setting up Azure monitoring for Azure API Management (APIM). The saying “the devil is in the details” certainly applies here. The goal is to simplify this process and help you get started quickly. Although we use the Azure portal in this blog post to demonstrate these concepts, you can automate the entire setup using Bicep, as illustrated here which is the preferred method for production.

In the realm of Azure APIM, monitoring is a pivotal aspect that ensures the smooth operation and management of APIs, especially when integrating with cutting-edge technologies like Azure Open AI. This blog post aims to shed light on the intricacies of setting up and utilizing the newly introduced Azure APIM “Emit Token Metric” policy while also touching on some high-level concepts which should help you shape your monitoring strategy within Azure APIM.

Enabling Application Insights

A critical step in the monitoring setup is enabling Application Insights within your APIM instance. This is done by navigating to the Application Insights section and ensuring it’s activated. Application Insights is instrumental in capturing and analyzing telemetry data, providing insights into the performance and usage of your APIs. Once its setup correctly it creates a logger within APIM which points to the associated application insights…