dapr
/
docs
mirror of https://github.com/dapr/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3103 from ItalyPaleAle/fix-3083 

Update Event Hub components docs
This commit is contained in:
Hannah Hunter 2023-02-02 14:25:54 -06:00 committed by GitHub
parent ed46f51ccd 5f8ee9e667
commit 319aebce14
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 133 additions and 97 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

77
daprdocs/content/en/reference/components-reference/supported-bindings/eventhubs.md
View File

@ -9,7 +9,7 @@ aliases:
## Component format ## Component format
To setup Azure Event Hubs binding create a component of type `bindings.azure.eventhubs`. See [this guide]({{< ref "howto-bindings.md#1-create-a-binding" >}}) on how to create and apply a binding configuration. To setup an Azure Event Hubs binding, create a component of type `bindings.azure.eventhubs`. See [this guide]({{< ref "howto-bindings.md#1-create-a-binding" >}}) on how to create and apply a binding configuration.
See [this](https://docs.microsoft.com/azure/event-hubs/event-hubs-dotnet-framework-getstarted-send) for instructions on how to set up an Event Hub. See [this](https://docs.microsoft.com/azure/event-hubs/event-hubs-dotnet-framework-getstarted-send) for instructions on how to set up an Event Hub.
@ -22,18 +22,39 @@ spec:
type: bindings.azure.eventhubs type: bindings.azure.eventhubs
version: v1 version: v1
metadata: metadata:
- name: connectionString # Azure EventHubs connection string # Hub name ("topic")
value: "Endpoint=sb://****" - name: eventHub
- name: consumerGroup # EventHubs consumer group value: "mytopic"
value: "group1" - name: consumerGroup
- name: storageAccountName # Azure Storage Account Name value: "myapp"
value: "accountName" # Either connectionString or eventHubNamespace is required
- name: storageAccountKey # Azure Storage Account Key # Use connectionString when *not* using Azure AD
value: "accountKey" - name: connectionString
- name: storageContainerName # Azure Storage Container Name value: "Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"
value: "containerName" # Use eventHubNamespace when using Azure AD
- name: partitionID # (Optional) PartitionID to send and receive events - name: eventHubNamespace
value: 0 value: "namespace"
- name: enableEntityManagement
value: "false"
# The following four properties are needed only if enableEntityManagement is set to true
- name: resourceGroupName
value: "test-rg"
- name: subscriptionID
value: "value of Azure subscription ID"
- name: partitionCount
value: "1"
- name: messageRetentionInDays
value: "3"
# Checkpoint store attributes
- name: storageAccountName
value: "myeventhubstorage"
- name: storageAccountKey
value: "112233445566778899"
- name: storageContainerName
value: "myeventhubstoragecontainer"
# Alternative to passing storageAccountKey
- name: storageConnectionString
value: "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<account-key>"
``` ```
{{% alert title="Warning" color="warning" %}} {{% alert title="Warning" color="warning" %}}
@ -42,25 +63,31 @@ The above example uses secrets as plain strings. It is recommended to use a secr
## Spec metadata fields ## Spec metadata fields
| Field | Required | Binding support | Details | Example | | Field | Required | Binding support | Details | Example |
|--------------------|:--------:|------------|-----|---------| |--------------------|:--------:|------------|-----|---------|
| connectionString | Y | Output | The [EventHubs connection string](https://docs.microsoft.com/azure/event-hubs/authorize-access-shared-access-signature). Note that this is the EventHub itself and not the EventHubs namespace. Make sure to use the child EventHub shared access policy connection string | `"Endpoint=sb://****"` | | `eventHub` | Y* | Input/Output | The name of the Event Hubs hub ("topic"). Required if using Azure AD authentication or if the connection string doesn't contain an `EntityPath` value | `mytopic` |
| consumerGroup | Y | Output | The name of an [EventHubs Consumer Group](https://docs.microsoft.com/azure/event-hubs/event-hubs-features#consumer-groups) to listen on | `"group1"` | | `connectionString` | Y* | Input/Output | Connection string for the Event Hub or the Event Hub namespace.<br>* Mutally exclusive with `eventHubNamespace` field.<br>* Required when not using [Azure AD Authentication]({{< ref "authenticating-azure.md" >}}) | `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"` or `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key}"`
| storageAccountName | Y | Output | The name of the account of the Azure Storage account to persist checkpoints data on | `"accountName"` | | `eventHubNamespace` | Y* | Input/Output | The Event Hub Namespace name.<br>* Mutally exclusive with `connectionString` field.<br>* Required when using [Azure AD Authentication]({{< ref "authenticating-azure.md" >}}) | `"namespace"`
| storageAccountKey | Y* | Output | The account key for the Azure Storage account to persist checkpoints data on. ***Not required if using AAD authentication.** | `"accountKey"` | | `enableEntityManagement` | N | Input/Output | Boolean value to allow management of the EventHub namespace and storage account. Default: `false` | `"true", "false"`
| storageContainerName | Y | Output | The name of the container in the Azure Storage account to persist checkpoints data on | `"containerName"` | | `resourceGroupName` | N | Input/Output | Name of the resource group the Event Hub namespace is part of. Required when entity management is enabled | `"test-rg"`
| partitionID | N | Output | ID of the partition to send and receive events | `0` | | `subscriptionID` | N | Input/Output | Azure subscription ID value. Required when entity management is enabled | `"azure subscription id"`
| eventHub | N | Output | The name of the EventHubs hub. **Required if using AAD authentication.** | `eventHubsNamespace-hubName` | | `partitionCount` | N | Input/Output | Number of partitions for the new Event Hub namespace. Used only when entity management is enabled. Default: `"1"` | `"2"`
| eventHubNamespace | N | Output | The name of the EventHubs namespace. **Required if using AAD authentication.** | `eventHubsNamespace` | | `messageRetentionInDays` | N | Input/Output | Number of days to retain messages for in the newly created Event Hub namespace. Used only when entity management is enabled. Default: `"1"` | `"90"`
| `consumerGroup` | Y | Input | The name of the [Event Hubs Consumer Group](https://docs.microsoft.com/azure/event-hubs/event-hubs-features#consumer-groups) to listen on | `"group1"` |
| `storageAccountName` | Y | Input | Storage account name to use for the checkpoint store. |`"myeventhubstorage"`
| `storageAccountKey` | Y* | Input | Storage account key for the checkpoint store account.<br>* When using Azure AD, it's possible to omit this if the service principal has access to the storage account too. | `"112233445566778899"`
| `storageConnectionString` | Y* | Input | Connection string for the checkpoint store, alternative to specifying `storageAccountKey` | `"DefaultEndpointsProtocol=https;AccountName=myeventhubstorage;AccountKey=<account-key>"`
| `storageContainerName` | Y | Input | Storage container name for the storage account name. | `"myeventhubstoragecontainer"`
### Azure Active Directory (AAD) authentication ### Azure Active Directory (AAD) authentication
The Azure Event Hubs pubsub component supports authentication using all Azure Active Directory mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of AAD authentication mechanism, see the [docs for authenticating to Azure]({{< ref authenticating-azure.md >}}).
The Azure Event Hubs pub/sub component supports authentication using all Azure Active Directory mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of AAD authentication mechanism, see the [docs for authenticating to Azure]({{< ref authenticating-azure.md >}}).
## Binding support ## Binding support
This component supports **output binding** with the following operations: This component supports **output binding** with the following operations:
- `create` - `create`: publishes a new message to Azure Event Hubs
## Input Binding to Azure IoT Hub Events ## Input Binding to Azure IoT Hub Events
@ -79,7 +106,7 @@ The device-to-cloud events created by Azure IoT Hub devices will contain additio
For example, the headers of a HTTP `Read()` response would contain: For example, the headers of a HTTP `Read()` response would contain:
```nodejs ```js
{ {
'user-agent': 'fasthttp', 'user-agent': 'fasthttp',
'host': '127.0.0.1:3000', 'host': '127.0.0.1:3000',

153
daprdocs/content/en/reference/components-reference/supported-pubsub/setup-azure-eventhubs.md
View File

@ -8,7 +8,8 @@ aliases:
--- ---
## Component format ## Component format
To setup Azure Event Hubs pubsub create a component of type `pubsub.azure.eventhubs`. See [this guide]({{< ref "howto-publish-subscribe.md#step-1-setup-the-pubsub-component" >}}) on how to create and apply a pubsub configuration.
To setup an Azure Event Hubs pub/sub, create a component of type `pubsub.azure.eventhubs`. See [this guide]({{< ref "howto-publish-subscribe.md#step-1-setup-the-pubsub-component" >}}) on how to create and apply a pub/sub configuration.
Apart from the configuration metadata fields shown below, Azure Event Hubs also supports [Azure Authentication]({{< ref "authenticating-azure.md" >}}) mechanisms. Apart from the configuration metadata fields shown below, Azure Event Hubs also supports [Azure Authentication]({{< ref "authenticating-azure.md" >}}) mechanisms.
```yaml ```yaml
@ -20,29 +21,34 @@ spec:
type: pubsub.azure.eventhubs type: pubsub.azure.eventhubs
version: v1 version: v1
metadata: metadata:
- name: connectionString # Either connectionString or eventHubNamespace. Should not be used when # Either connectionString or eventHubNamespace is required
# Azure Authentication mechanism is used. # Use connectionString when *not* using Azure AD
value: "Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}" - name: connectionString
- name: eventHubNamespace # Either connectionString or eventHubNamespace. Should be used when value: "Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"
# Azure Authentication mechanism is used. # Use eventHubNamespace when using Azure AD
value: "namespace" - name: eventHubNamespace
- name: enableEntityManagement value: "namespace"
value: "false" - name: enableEntityManagement
## The following four properties are needed only if enableEntityManagement is set to true value: "false"
- name: resourceGroupName # The following four properties are needed only if enableEntityManagement is set to true
value: "test-rg" - name: resourceGroupName
- name: subscriptionID value: "test-rg"
value: "value of Azure subscription ID" - name: subscriptionID
- name: partitionCount value: "value of Azure subscription ID"
value: "1" - name: partitionCount
- name: messageRetentionInDays value: "1"
## Subscriber attributes - name: messageRetentionInDays
- name: storageAccountName value: "3"
value: "myeventhubstorage" # Checkpoint store attributes
- name: storageAccountKey - name: storageAccountName
value: "112233445566778899" value: "myeventhubstorage"
- name: storageContainerName - name: storageAccountKey
value: "myeventhubstoragecontainer" value: "112233445566778899"
- name: storageContainerName
value: "myeventhubstoragecontainer"
# Alternative to passing storageAccountKey
- name: storageConnectionString
value: "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<account-key>"
``` ```
{{% alert title="Warning" color="warning" %}} {{% alert title="Warning" color="warning" %}}
@ -53,21 +59,24 @@ The above example uses secrets as plain strings. It is recommended to use a secr
| Field | Required | Details | Example | | Field | Required | Details | Example |
|--------------------|:--------:|---------|---------| |--------------------|:--------:|---------|---------|
| connectionString | Y* | Connection-string for the Event Hub or the Event Hub namespace. *Mutally exclusive with `eventHubNamespace` field. *Not to be used when [Azure Authentication]({{< ref "authenticating-azure.md" >}}) is used | `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"` or `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key}"` | `connectionString` | Y* | Connection string for the Event Hub or the Event Hub namespace.<br>* Mutally exclusive with `eventHubNamespace` field.<br>* Required when not using [Azure AD Authentication]({{< ref "authenticating-azure.md" >}}) | `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"` or `"Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key}"`
| eventHubNamespace | N* | The Event Hub Namespace name. *Mutally exclusive with `connectionString` field. *To be used when [Azure Authentication]({{< ref "authenticating-azure.md" >}}) is used | `"namespace"` | `eventHubNamespace` | Y* | The Event Hub Namespace name.<br>* Mutally exclusive with `connectionString` field.<br>* Required when using [Azure AD Authentication]({{< ref "authenticating-azure.md" >}}) | `"namespace"`
| storageAccountName | Y | Storage account name to use for the EventProcessorHost |`"myeventhubstorage"` | `storageAccountName` | Y | Storage account name to use for the checkpoint store. |`"myeventhubstorage"`
| storageAccountKey | Y* | Storage account key to use for the EventProcessorHost. Can be `secretKeyRef` to use a secret reference. *Omit if using [Azure Authentication]({{< ref "authenticating-azure.md" >}}) and AAD authentication to the storage account is preferred. | `"112233445566778899"` | `storageAccountKey` | Y* | Storage account key for the checkpoint store account.<br>* When using Azure AD, it's possible to omit this if the service principal has access to the storage account too. | `"112233445566778899"`
| storageContainerName | Y | Storage container name for the storage account name. | `"myeventhubstoragecontainer"` | `storageConnectionString` | Y* | Connection string for the checkpoint store, alternative to specifying `storageAccountKey` | `"DefaultEndpointsProtocol=https;AccountName=myeventhubstorage;AccountKey=<account-key>"`
| enableEntityManagement | N | Boolean value to allow management of EventHub namespace. Default: `false` | `"true", "false"` | `storageContainerName` | Y | Storage container name for the storage account name. | `"myeventhubstoragecontainer"`
| resourceGroupName | N | Name of the resource group the event hub namespace is a part of. Needed when entity management is enabled | `"test-rg"` | `enableEntityManagement` | N | Boolean value to allow management of the EventHub namespace and storage account. Default: `false` | `"true", "false"`
| subscriptionID | N | Azure subscription ID value. Needed when entity management is enabled | `"azure subscription id"` | `resourceGroupName` | N | Name of the resource group the Event Hub namespace is part of. Required when entity management is enabled | `"test-rg"`
| partitionCount | N | Number of partitions for the new event hub. Only used when entity management is enabled. Default: `"1"` | `"2"` | `subscriptionID` | N | Azure subscription ID value. Required when entity management is enabled | `"azure subscription id"`
| messageRetentionInDays | N | Number of days to retain messages for in the newly created event hub. Used only when entity management is enabled. Default: `"1"` | `"90"` | `partitionCount` | N | Number of partitions for the new Event Hub namespace. Used only when entity management is enabled. Default: `"1"` | `"2"`
| `messageRetentionInDays` | N | Number of days to retain messages for in the newly created Event Hub namespace. Used only when entity management is enabled. Default: `"1"` | `"90"`
### Azure Active Directory (AAD) authentication ### Azure Active Directory (AAD) authentication
The Azure Event Hubs pubsub component supports authentication using all Azure Active Directory mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of AAD authentication mechanism, see the [docs for authenticating to Azure]({{< ref authenticating-azure.md >}}).
The Azure Event Hubs pub/sub component supports authentication using all Azure Active Directory mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of AAD authentication mechanism, see the [docs for authenticating to Azure]({{< ref authenticating-azure.md >}}).
#### Example Configuration #### Example Configuration
```yaml ```yaml
apiVersion: dapr.io/v1alpha1 apiVersion: dapr.io/v1alpha1
kind: Component kind: Component
@ -77,32 +86,31 @@ spec:
type: pubsub.azure.eventhubs type: pubsub.azure.eventhubs
version: v1 version: v1
metadata: metadata:
# Azure Authentication Used # Azure Authentication Used
- name: azureTenantId - name: azureTenantId
value: "***" value: "***"
- name: azureClientId - name: azureClientId
value: "***" value: "***"
- name: azureClientSecret - name: azureClientSecret
value: "***" value: "***"
- name: eventHubNamespace - name: eventHubNamespace
value: "namespace" value: "namespace"
- name: enableEntityManagement - name: enableEntityManagement
value: "false" value: "false"
## The following four properties are needed only if enableEntityManagement is set to true # The following four properties are needed only if enableEntityManagement is set to true
- name: resourceGroupName - name: resourceGroupName
value: "test-rg" value: "test-rg"
- name: subscriptionID - name: subscriptionID
value: "value of Azure subscription ID" value: "value of Azure subscription ID"
- name: partitionCount - name: partitionCount
value: "1" value: "1"
- name: messageRetentionInDays - name: messageRetentionInDays
## Subscriber attributes # Checkpoint store attributes
- name: storageAccountName # In this case, we're using Azure AD to access the storage account too
value: "myeventhubstorage" - name: storageAccountName
- name: storageAccountKey value: "myeventhubstorage"
value: "112233445566778899" - name: storageContainerName
- name: storageContainerName value: "myeventhubstoragecontainer"
value: "myeventhubstoragecontainer"
``` ```
## Sending multiple messages ## Sending multiple messages
@ -115,27 +123,27 @@ Azure Event Hubs natively supports sending multiple messages in a single operati
## Create an Azure Event Hub ## Create an Azure Event Hub
Follow the instructions [here](https://docs.microsoft.com/azure/event-hubs/event-hubs-create) on setting up Azure Event Hubs. Follow the instructions on the [documentation](https://docs.microsoft.com/azure/event-hubs/event-hubs-create) to set up Azure Event Hubs.
Since this implementation uses the Event Processor Host, you will also need an [Azure Storage Account](https://docs.microsoft.com/azure/storage/common/storage-account-create?tabs=azure-portal). Follow the instructions [here](https://docs.microsoft.com/azure/storage/common/storage-account-keys-manage) to manage the storage account access keys.
See [here](https://docs.microsoft.com/azure/event-hubs/authorize-access-shared-access-signature) on how to get the Event Hubs connection string. Note this is not the Event Hubs namespace. Because this component uses Azure Storage as checkpoint store, you will also need an [Azure Storage Account](https://docs.microsoft.com/azure/storage/common/storage-account-create?tabs=azure-portal). Follow the instructions on the [documentation](https://docs.microsoft.com/azure/storage/common/storage-account-keys-manage) to manage the storage account access keys.
See the [documentation](https://docs.microsoft.com/azure/event-hubs/authorize-access-shared-access-signature) on how to get the Event Hubs connection string (note this is not for the Event Hubs namespace).
### Create consumer groups for each subscriber ### Create consumer groups for each subscriber
For every Dapr app that wants to subscribe to events, create an Event Hubs consumer group with the name of the `dapr id`. For every Dapr app that wants to subscribe to events, create an Event Hubs consumer group with the name of the Dapr app ID. For example, a Dapr app running on Kubernetes with `dapr.io/app-id: "myapp"` will need an Event Hubs consumer group named `myapp`.
For example, a Dapr app running on Kubernetes with `dapr.io/app-id: "myapp"` will need an Event Hubs consumer group named `myapp`.
Note: Dapr passes the name of the Consumer group to the EventHub and so this is not supplied in the metadata. Note: Dapr passes the name of the consumer group to the Event Hub, so this is not supplied in the metadata.
## Entity Management ## Entity Management
When entity management is enabled in configuration, as long as the application has the right role and permissions to manipulate the Event Hub namespace, creation of Event Hubs and consumer groups can be done on the fly. When entity management is enabled in the metadata, as long as the application has the right role and permissions to manipulate the Event Hub namespace, Dapr can automatically create the Event Hub and consumer group for you.
The Evet Hub name is the `topic` field in the incoming request to publish or subscribe to, while the consumer group name is the name of the `dapr app` which subscribes to a given Event Hub. For example, a Dapr app running on Kubernetes with name `dapr.io/app-id: "myapp"` requires an Event Hubs consumer group named `myapp`. The Evet Hub name is the `topic` field in the incoming request to publish or subscribe to, while the consumer group name is the name of the Dapr app which subscribes to a given Event Hub. For example, a Dapr app running on Kubernetes with name `dapr.io/app-id: "myapp"` requires an Event Hubs consumer group named `myapp`.
Entity management is only possible when using [Azure Authentication]({{< ref "authenticating-azure.md" >}}) mechanisms and not via `connectionString`. Entity management is only possible when using [Azure AD Authentication]({{< ref "authenticating-azure.md" >}}) and not using a connection string.
Note: Dapr passes the name of the Consumer group to the EventHub and this is not supplied in the metadata. > Dapr passes the name of the consumer group to the Event Hub, so this is not supplied in the metadata.
## Subscribing to Azure IoT Hub Events ## Subscribing to Azure IoT Hub Events
@ -154,7 +162,7 @@ The device-to-cloud events created by Azure IoT Hub devices will contain additio
For example, the headers of a delivered HTTP subscription message would contain: For example, the headers of a delivered HTTP subscription message would contain:
```nodejs ```js
{ {
'user-agent': 'fasthttp', 'user-agent': 'fasthttp',
'host': '127.0.0.1:3000', 'host': '127.0.0.1:3000',
@ -174,6 +182,7 @@ For example, the headers of a delivered HTTP subscription message would contain:
``` ```
## Related links ## Related links
- [Basic schema for a Dapr component]({{< ref component-schema >}}) - [Basic schema for a Dapr component]({{< ref component-schema >}})
- Read [this guide]({{< ref "howto-publish-subscribe.md#step-2-publish-a-topic" >}}) for instructions on configuring pub/sub components - Read [this guide]({{< ref "howto-publish-subscribe.md#step-2-publish-a-topic" >}}) for instructions on configuring pub/sub components
- [Pub/Sub building block]({{< ref pubsub >}}) - [Pub/Sub building block]({{< ref pubsub >}})