Background:
In most cases, the best authentication method for Azure services is by using a managed identity. However, there are scenarios where this may not be an option, and access keys or passwords are used. In such cases, it is important to rotate access keys and passwords regularly.
Automatic Secret rotation solution:
Based on the document (https://learn.microsoft.com/en-us/azure/key-vault/secrets/tutorial-rotation-dual?tabs=azure-cli), it proposes a solution to generate new access keys on storage account and update it in the key vault based on the secret expiry time. It uses Azure Event Grid to send the secret expiry event to an Azure Function App (aka the Azure Function App triggered by Azure Event Grid notifications) – which in turn generates the new access key and updates the secret in the Key vault to automate the periodic rotation of secret.
But it has a limitation as follows: The EventGrid System Topic cannot deliver events to the target which has “Public access disabled and only private endpoint enabled” (the block enclosed in red in the above screenshot)
To overcome this limitation a few tweaks to the architecture need to be made, which are presented in the following sections of this document. We introduce an Azure Event Hubs in the architecture with public access disabled but “allow trusted microsoft service to bypass” setting enabled.
Secret Rotation when private endpoints are enabled:
1. Configure event subscription on the Azure Key Vault.
Thirty days before the expiration date of a secret, Key Vault publishes the near expiry event to Event Grid.
2. Event Grid checks the event subscriptions and delivers the event to the Event Hub endpoint. The managed identity of the Event Grid should have “Azure Event Hub data sender” role assigned in the Event Hub. “Allow trusted Microsoft services to bypass this firewall” setting must be enabled on the Event hub namespace and public access disabled.
Event Hubs -> Networking blade:
3. The event hub triggers the function app.
4. The function app (created with event hub trigger) identifies the key and calls the storage account to regenerate it.
5. The function app adds the newly regenerated key to Azure Key Vault as the new version of the secret.
The function app managed identity should be assigned the following roles:
1. Key Vault secret officer role on the key vault (when using RBAC based access model)
2. Storage Account Key Operator Service Role assigned on the storage account to generate access keys.
The following link has the sample Function app code (PowerShell) which performs the secret rotation in the method "RoatateSecret" including RegenerateKey on storage account and AddSecretToKeyVault. (https://github.com/Azure-Samples/keyvault-rotation-storageaccountkey-powershell/blob/main/AKVStorage...)
But it needs a few tweaks as our function app is being triggered by Event hub and not event grid topic directly.
The last few lines must be modified in the following way:
$eventHubMessages | ConvertTo-Json | Write-Host
$eventHubMessages | ForEach-Object {
$secretName = $_.subject
$keyVaultName = $_.data.VaultName
Write-Host "Key Vault Name: $keyVaultName"
Write-Host "Secret Name: $secretName"
Write-Host "Rotation started."
RotateSecret $keyVAultName $secretName
Write-Host "Secret Rotated Successfully"
}
Reference:
1.Automate the rotation of a secret for resources that have two sets of authentication credentials
https://learn.microsoft.com/en-us/azure/key-vault/secrets/tutorial-rotation-dual?tabs=azure-cli
2.How to trigger Azure Function from Event Grid over virtual network
https://techcommunity.microsoft.com/t5/blogs/blogworkflowpage/blog-id/AppsonAzureBlog/article-id/171...
3.Azure services that support system topics in Azure Event Grid
https://learn.microsoft.com/en-us/azure/event-grid/system-topics
4.Keyvault-rotation-storageaccountkey-powershell