diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md new file mode 100644 index 000000000..e7ebc89ef --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -0,0 +1,163 @@ +--- +type: docs +title: "Send and receive messages in bulk" +linkTitle: "Send and receive messages in bulk" +weight: 7100 +description: "Learn how to use the bulk publish and subscribe APIs in Dapr." +--- + +{{% alert title="alpha" color="warning" %}} +The bulk publish and subscribe APIs are in **alpha** stage. +{{% /alert %}} + +With the bulk publish and subscribe APIs, you can send and receive multiple messages in a single request. + +## Native bulk publish and subscribe support + +When a pub/sub component supports the bulk publish API natively, Dapr also publishes messages to the underlying pub/sub component in bulk. + +Otherwise, Dapr falls back to sending messages one by one to the underlying pub/sub component. This is still more efficient than using the regular publish API, because applications can still send multiple messages in a single request to Dapr. + +## Supported components + +Refer to the [component reference]({{< ref supported-pubsub >}}) to see which components support the bulk publish API natively. + +## Publishing messages in bulk + +The bulk publish API allows you to publish multiple messages to a topic in a single request. If any of the messages fail to publish, the bulk publish operation returns a list of failed messages. Note, the bulk publish operation does not guarantee the order of messages. + +### Example + +{{< tabs Java Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} + +{{% codetab %}} + +```java +import io.dapr.client.DaprClientBuilder; +import io.dapr.client.DaprPreviewClient; +import io.dapr.client.domain.BulkPublishResponse; +import io.dapr.client.domain.BulkPublishResponseFailedEntry; +import java.util.ArrayList; +import java.util.List; + +class BulkPublisher { + public void publishMessages() { + try (DaprPreviewClient client = (new DaprClientBuilder()).buildPreviewClient()) { + // Create a list of messages to publish + List messages = new ArrayList<>(); + for (int i = 0; i < 10; i++) { + String message = String.format("This is message #%d", i); + messages.add(message); + } + + // Publish list of messages using the bulk publish API + BulkPublishResponse res = client.publishEvents(PUBSUB_NAME, TOPIC_NAME, "text/plain", messages).block(); + } + } +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```typescript + +import { DaprClient } from "@dapr/dapr"; + +const pubSubName = "my-pubsub-name"; +const topic = "topic-a"; + +async function start() { + const client = new DaprClient(); + + // Publish multiple messages to a topic. + await client.pubsub.publishBulk(pubSubName, topic, ["message 1", "message 2", "message 3"]); + + // Publish multiple messages to a topic with explicit bulk publish messages. + const bulkPublishMessages = [ + { + entryID: "entry-1", + contentType: "application/json", + event: { hello: "foo message 1" }, + }, + { + entryID: "entry-2", + contentType: "application/cloudevents+json", + event: { + specversion: "1.0", + source: "/some/source", + type: "example", + id: "1234", + data: "foo message 2", + datacontenttype: "text/plain" + }, + }, + { + entryID: "entry-3", + contentType: "text/plain", + event: "foo message 3", + }, + ]; + await client.pubsub.publishBulk(pubSubName, topic, bulkPublishMessages); +} + +start().catch((e) => { + console.error(e); + process.exit(1); +}); +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/my-pubsub-name/topic-a \ + -H 'Content-Type: application/json' \ + -d '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' +``` + +{{% /codetab %}} + +{{% codetab %}} + +```powershell +Invoke-RestMethod -Method Post -ContentType 'application/json' -Uri 'http://localhost:3500/v1.0-alpha1/publish/bulk/my-pubsub-name/topic-a' ` +-Body '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' +``` + +{{% /codetab %}} + +{{< /tabs >}} + +## Related links + +- List of [supported pub/sub components]({{< ref supported-pubsub >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index 2b90adda6..93bc84314 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -119,6 +119,10 @@ By default, all topic messages associated with an instance of a pub/sub componen Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). +### Send and receive messages in bulk + +Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that require a high throughput. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). + ## Try out pub/sub ### Quickstarts and tutorials diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index f4d8e47fa..3bdbe09aa 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -64,6 +64,90 @@ Parameter | Description > Additional metadata parameters are available based on each pubsub component. +## Publish multiple messages to a given topic + +This endpoint lets you publish multiple messages to consumers who are listening on a `topic`. + +### HTTP Request + +``` +POST http://localhost:/v1.0-alpha1/publish/bulk//[?] +``` + +The request body should contain a JSON array of entries with: +- Unique entry IDs +- The event to publish +- The content type of the event + +If the content type for an event is not `application/cloudevents+json`, it is auto-wrapped as a CloudEvent (unless `metadata.rawPayload` is set to `true`). + +Example: + +```bash +curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/pubsubName/deathStarStatus \ + -H 'Content-Type: application/json' \ + -d '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' +``` + +### Headers + +The `Content-Type` header should always be set to `application/json`. + +### URL Parameters + +|**Parameter**|**Description**| +|--|--| +|`daprPort`|The Dapr port| +|`pubsubname`|The name of pub/sub component| +|`topic`|The name of the topic| +|`metadata`|Query parameters for [metadata]({{< ref "pubsub_api.md#metadata" >}})| + +### Metadata + +Metadata can be sent via query parameters in the request's URL. It must be prefixed with `metadata.`, as shown in the table below. + +|**Parameter**|**Description**| +|--|--| +|`metadata.rawPayload`|Boolean to determine if Dapr should publish the messages without wrapping them as CloudEvent.| +|`metadata.maxBulkPubBytes`|Maximum bytes to publish in a bulk publish request.| + + +#### HTTP Response + +|**HTTP Status**|**Description**| +|--|--| +|204|All messages delivered| +|400|Pub/sub does not exist| +|403|Forbidden by access controls| +|500|At least one message failed to be delivered| + +The response body is a JSON containing a list of failed entries. Example: + +```json +{ + "failedEntries": [ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "error": "error message" + }, + ], + "errorCode": "ERR_PUBSUB_PUBLISH_MESSAGE" +} +``` + ## Optional Application (User Code) Routes ### Provide a route for Dapr to discover topic subscriptions diff --git a/daprdocs/data/components/pubsub/azure.yaml b/daprdocs/data/components/pubsub/azure.yaml index 3bb3c279a..e278829af 100644 --- a/daprdocs/data/components/pubsub/azure.yaml +++ b/daprdocs/data/components/pubsub/azure.yaml @@ -3,8 +3,14 @@ state: Stable version: v1 since: "1.8" + features: + bulkPublish: true + bulkSubscribe: false - component: Azure Service Bus link: setup-azure-servicebus state: Stable version: v1 since: "1.0" + features: + bulkPublish: true + bulkSubscribe: true diff --git a/daprdocs/data/components/pubsub/generic.yaml b/daprdocs/data/components/pubsub/generic.yaml index 58fde998f..f352dc202 100644 --- a/daprdocs/data/components/pubsub/generic.yaml +++ b/daprdocs/data/components/pubsub/generic.yaml @@ -13,6 +13,9 @@ state: Stable version: v1 since: "1.5" + features: + bulkPublish: true + bulkSubscribe: true - component: Redis Streams link: setup-redis-pubsub state: Stable diff --git a/daprdocs/layouts/partials/components/pubsub.html b/daprdocs/layouts/partials/components/pubsub.html index 244a6f623..b2ee91a1f 100644 --- a/daprdocs/layouts/partials/components/pubsub.html +++ b/daprdocs/layouts/partials/components/pubsub.html @@ -10,6 +10,8 @@ + + @@ -19,6 +21,8 @@ + +
ComponentBulk PublishBulk Subscribe Status Component version Since runtime version{{ .component }} {{ if .features.bulkPublish }}✅{{else}}{{ end }}{{ if .features.bulkSubscribe }}✅{{else}}{{ end }} {{ .state }} {{ .version }} {{ .since }}