How to set offset for Event Hubs Trigger

[Calvin_Cai]

We understand that when retrieving data from Event Hubs, you can provide the offset to retrieve the data from offset.

The same mechanism applies when using the Function App Event Hubs Trigger, which stores the offset information in a storage account.

In this document, we will discuss how to check the offset information for Event Hubs and how to set or reset the value.

Prerequisite

• EventHubs with Standard Pricing tier
  • EventHub name : samples-workitems
  • Partition count : 4
  • Retention time : 168 hours

• Function App with EventHubs Trigger

ServiceBusExplorer

Before we connect to EventHubs from function app, we could use ServiceBusExplorer to test send/receive data from EventHubs.
You could download ServiceBusExplorer-4.1.112  version to do the test.

1. Navigate EventHubs instance and click "Shared access policies" -> "RootManageSharedAccessKey"

   
   

2. Open Service Bus Explorer we just downloaded, click "File" and select "Connect", select "Enter connection string" and paste the connection string.

   
   

3. Once you connected to EventHubs, you will see the entity name, here we already created "samples-workitems" for testing

   

4. Right click "samples-workitems" and click "Send Events" menu

   

5. Fill the Message Text and click "Start", now you sent a message to EventHubs

   
   

6. Right click a Consumer Groups, like "$Default", click "Create Consumer Group Listener"

   

7. Click Start button, and it will receive all the Events, you could see the Offset(it could be a large number) and SequenceNumber value
   And you could close the window and create listener again, you could receive the messages again.

   

Configure Function App to receive data from EventHubs

Here we use a PowerShell function to test receive data from EventHubs, please refer below sample file

• function.json

{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "eventHubMessages",
      "direction": "in",
      "eventHubName": "samples-workitems",
      "connection": "EVENTHUB_CONN",
      "cardinality": "many",
      "consumerGroup": "$Default"
    }
  ]
}

• run.ps1

param($eventHubMessages, $TriggerMetadata)

Write-Host "PowerShell eventhub trigger function called for message array: $eventHubMessages"

$json = ConvertTo-Json $eventHubMessages
Write-Host $json

$json = ConvertTo-Json $TriggerMetadata
Write-Host $json

$eventHubMessages | ForEach-Object { Write-Host "Processed message: $_" }

The Configuration EVENTHUB_CONN configred the connection string to EventHubs
After the code is deployed, function app should able to process data from EventHubs, you should able to see below messages

Check The offset information

1. Open function app from Azure Portal and navigate to Configuration Page, check the "AzureWebJobStorage" value and find storage account name
   
2. Navigate to the storage account instance and you would see "azure-webjob-eventhub" container
   
   
3. Navigate to <eventhub instance>/<eventhub entity>/<consume group>/checkpoint folder, and you would see partitions file
   
4. Click the partition file and you would see the Metadata stored "offset" and "sequencenumber" information
   
5. You could update the offset and sequencenumber information and save it, Or you could just delete the partition file
6. The function app will process the existing message accordingly (from the new offset, or from begining)

Refrences

• https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=p...