Issue: Few platform logs and metrics go missing when streaming them from Diagnostic Setting to Event Hub especially when Event Hub is Throttling.
Scenario: Using Diagnostic setting users route platform logs and metrics to Azure Event Hub (EH). These events are consumed by partner SIEM and Monitoring tools. At times users complain that events that occurred in the system or certain metrics\logs\traces has not reached the Event Hub.
Validation:
Explanation:
Let us start by understanding the 3 services participating in the Diagnostic setup.
Throughput units (standard tier) is pre-purchased units of capacity that control the throughput capacity of Event Hubs. A single throughput unit lets you:
Beyond the capacity of the purchased throughput units, ingress is throttled and a ServerBusyException is returned.
If the user indiscriminately sends diagnostic loads of logs\traces\metrics to EH without consideration to its Throughput capacity\units, the EH will throttle the ingress data, and throw ServerBusyExecption. The common modus operandi is to space out the ingress request and retry the operations.
Please read the support statement from Azure Monitoring - Diagnostic settings in Azure Monitor states:
Possibility of duplicated or dropped data: Every effort is made to ensure all log data is sent correctly to your destinations, however it's not possible guarantee 100% data transfer of logs between endpoints. Retries and other mechanisms are in place to work around these issues and attempt to ensure log data arrives at the endpoint. |
What are the Alternative Options to minimize data loss during EH Throttling?
The simple answer is to avoid throttling.
Also, Ref:
Processing units (Premium tier) or Capacity units (Dedicated)
Diagnostic settings in Azure Monitor - Azure Monitor
Automatically scale up throughput units in Azure Event Hubs - Azure Event Hubs
Frequently asked questions - Azure Event Hubs
How exactly does Event Hubs Throttling Work? (Old Article)
Tags: Diagnostic Setting, Missing, Lost, Event Hub, Throttling.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.