From 12bbac91d47c5753e34e9dd84bda9dabeae02e66 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Wed, 14 Dec 2022 11:48:56 +0530 Subject: [PATCH 01/11] WIP publish docs Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 11 ++++ .../building-blocks/pubsub/pubsub-overview.md | 4 ++ .../content/en/reference/api/pubsub_api.md | 66 +++++++++++++++++++ 3 files changed, 81 insertions(+) create mode 100644 daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md new file mode 100644 index 000000000..3375dfc8c --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -0,0 +1,11 @@ +--- +type: docs +title: "Publishing & subscribing messages in bulk" +linkTitle: "Bulk messages" +weight: 2100 +description: "Learn how to send and receive multiple messages at once" +--- + +insert-introduction + +## Publishing messages in bulk \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index 2b90adda6..7dae7663d 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -119,6 +119,10 @@ By default, all topic messages associated with an instance of a pub/sub componen Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). +### Bulk messages + +Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that need to send or receive a large number of messages at once. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). + ## Try out pub/sub ### Quickstarts and tutorials diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index f4d8e47fa..69d7c813b 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -64,6 +64,72 @@ Parameter | Description > Additional metadata parameters are available based on each pubsub component. +## Publish multiple messages to a given topic + +This endpoint lets you publish multiple messages to consumers who are listening on a `topic`. + +### HTTP Request + +``` +POST http://localhost:/v1.0-alpha1/publish/bulk//[?] +``` + +The request body should contain a JSON array of entries with unique entry IDs. Example: + +```json +[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, +] +``` + +Just like the publish endpoint, the events are auto-wrapped as CloudEvents if `rawPayload` metadata is not set to true. The `contentType` field is optional and defaults to `text/plain`. + +### Headers + +The `Content-Type` header should be set to `application/json`. + +### Metadata + +Metadata can be sent via query parameters in the request's URL. If must be prefixed with `metadata.` as shown below. + +|**Parameter**|**Description**| +|--|--| +|`metadata.rawPayload`|Boolean to determine if Dapr should publish the messages without wrapping them as CloudEvent.| +|`metadata.maxBulkPubBytes`|Maximum bytes to publish in a bulk publish request.| + + +#### HTTP Response + +|**Code**|**Description**| +|--|--| +|204|All messages delivered| +|400|Pubsub does not exist| +|403|Forbidden by access controls| +|500|At least one message failed to be delivered| + +The response body is a JSON containing a list of failed messages. Example: + +```json +[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "error": "error message", + "status": "FAIL", + } +] +``` + ## Optional Application (User Code) Routes ### Provide a route for Dapr to discover topic subscriptions From e89864e2ad047a02e8ecdef65f55ce474d301b75 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Wed, 14 Dec 2022 11:53:48 +0530 Subject: [PATCH 02/11] Remove a statement Signed-off-by: Shubham Sharma --- daprdocs/content/en/reference/api/pubsub_api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index 69d7c813b..7b253198b 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -93,7 +93,7 @@ The request body should contain a JSON array of entries with unique entry IDs. E ] ``` -Just like the publish endpoint, the events are auto-wrapped as CloudEvents if `rawPayload` metadata is not set to true. The `contentType` field is optional and defaults to `text/plain`. +Just like the publish endpoint, the events are auto-wrapped as CloudEvents if `rawPayload` metadata is not set to true. ### Headers From 58eb6d6003ccafb13a18c1601373eb79c28d0f77 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Wed, 4 Jan 2023 16:34:01 +0530 Subject: [PATCH 03/11] Update API reference' Signed-off-by: Shubham Sharma --- .../content/en/reference/api/pubsub_api.md | 65 +++++++++++-------- 1 file changed, 39 insertions(+), 26 deletions(-) diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index 7b253198b..a7ca07ae0 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -74,30 +74,41 @@ This endpoint lets you publish multiple messages to consumers who are listening POST http://localhost:/v1.0-alpha1/publish/bulk//[?] ``` -The request body should contain a JSON array of entries with unique entry IDs. Example: +The request body should contain a JSON array of entries with unique entry IDs, the event to publish, and the content type of the event. If the content type for an event is not `application/cloudevents+json`, it is auto-wrapped as a CloudEvent (unless `metadata.rawPayload` is set to `true`, see below). -```json -[ - { - "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "event": "first", - "contentType": "text/plain" - }, - { - "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", - "event": { - "message": "second" +Example: + +```bash +curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/pubsubName/deathStarStatus \ + -H 'Content-Type: application/json' \ + -d '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" }, - "contentType": "application/json" - }, -] + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' ``` -Just like the publish endpoint, the events are auto-wrapped as CloudEvents if `rawPayload` metadata is not set to true. - ### Headers -The `Content-Type` header should be set to `application/json`. +The `Content-Type` header should always be set to `application/json`. + +### URL Parameters + +|**Parameter**|**Description**| +|--|--| +|`daprPort`|The Dapr port| +|`pubsubname`|The name of pubsub component| +|`topic`|The name of the topic| +|`metadata`|Query parameters for metadata as described below| ### Metadata @@ -111,23 +122,25 @@ Metadata can be sent via query parameters in the request's URL. If must be prefi #### HTTP Response -|**Code**|**Description**| +|**HTTP Status**|**Description**| |--|--| |204|All messages delivered| |400|Pubsub does not exist| |403|Forbidden by access controls| |500|At least one message failed to be delivered| -The response body is a JSON containing a list of failed messages. Example: +The response body is a JSON containing a list of failed entries. Example: ```json -[ +{ + "failedEntries": [ { - "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "error": "error message", - "status": "FAIL", - } -] + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "error": "error message" + }, + ], + "errorCode": "ERR_PUBSUB_PUBLISH_MESSAGE" +} ``` ## Optional Application (User Code) Routes From 6671a6e6bab6e61a00f1ef4e50087ab72241178f Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Thu, 5 Jan 2023 11:22:02 +0530 Subject: [PATCH 04/11] Add more docs Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 138 +++++++++++++++++- .../building-blocks/pubsub/pubsub-overview.md | 4 +- daprdocs/data/components/pubsub/azure.yaml | 6 + daprdocs/data/components/pubsub/generic.yaml | 3 + .../layouts/partials/components/pubsub.html | 4 + 5 files changed, 147 insertions(+), 8 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index 3375dfc8c..eddf845f6 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -1,11 +1,137 @@ --- type: docs -title: "Publishing & subscribing messages in bulk" -linkTitle: "Bulk messages" -weight: 2100 -description: "Learn how to send and receive multiple messages at once" +title: "Send and receive messages in bulk" +linkTitle: "Send and receive messages in bulk" +weight: 7100 +description: "Learn how to use the bulk publish and subscribe APIs in Dapr." --- -insert-introduction +{{% alert title="alpha" color="warning" %}} +The bulk publish and subscribe APIs are in **alpha** stage. +{{% /alert %}} -## Publishing messages in bulk \ No newline at end of file +With the bulk publish and subscribe APIs, you can send and receive multiple messages in a single request. + +## Native bulk publish and subscribe support + +When a pub/sub component supports the bulk publish API natively, Dapr also publishes messages to the underlying pub/sub component in bulk. + +Otherwise, Dapr falls back to sending messages one by one to the underlying pub/sub component. This is still more efficient than using the regular publish API, because applications can still send multiple messages in a single request to Dapr. + +## Supported components + +Refer [component reference]({{< ref supported-pubsub >}}) to see which components support the bulk publish API natively. + +## Publishing messages in bulk + +### Example + +{{< tabs Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} + +{{% codetab %}} + +```typescript + +import { DaprClient } from "@dapr/dapr"; + +const pubSubName = "my-pubsub-name"; +const topic = "topic-a"; + +async function start() { + const client = new DaprClient(); + + // Publish multiple messages to a topic. + await client.pubsub.publishBulk(pubSubName, topic, ["message 1", "message 2", "message 3"]); + + // Publish multiple messages to a topic with explicit bulk publish messages. + const bulkPublishMessages = [ + { + entryID: "entry-1", + contentType: "application/json", + event: { hello: "foo message 1" }, + }, + { + entryID: "entry-2", + contentType: "application/cloudevents+json", + event: { + specversion: "1.0", + source: "/some/source", + type: "example", + id: "1234", + data: "foo message 2", + datacontenttype: "text/plain" + }, + }, + { + entryID: "entry-3", + contentType: "text/plain", + event: "foo message 3", + }, + ]; + await client.pubsub.publishBulk(pubSubName, topic, bulkPublishMessages); +} + +start().catch((e) => { + console.error(e); + process.exit(1); +}); +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/my-pubsub-name/topic-a \ + -H 'Content-Type: application/json' \ + -d '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' +``` + +{{% /codetab %}} + +{{% codetab %}} + +```powershell +Invoke-RestMethod -Method Post -ContentType 'application/json' -Uri 'http://localhost:3500/v1.0-alpha1/publish/bulk/my-pubsub-name/topic-a' ` +-Body '[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second" + }, + "contentType": "application/json" + }, + ]' +``` + +{{% /codetab %}} + +{{< /tabs >}} +``` + +{{% /codetab %}} + +{{< /tabs >}} + +## Related links + +- List of [supported pub/sub components]({{< ref supported-pubsub >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index 7dae7663d..93bc84314 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -119,9 +119,9 @@ By default, all topic messages associated with an instance of a pub/sub componen Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). -### Bulk messages +### Send and receive messages in bulk -Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that need to send or receive a large number of messages at once. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). +Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that require a high throughput. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). ## Try out pub/sub diff --git a/daprdocs/data/components/pubsub/azure.yaml b/daprdocs/data/components/pubsub/azure.yaml index 3bb3c279a..e278829af 100644 --- a/daprdocs/data/components/pubsub/azure.yaml +++ b/daprdocs/data/components/pubsub/azure.yaml @@ -3,8 +3,14 @@ state: Stable version: v1 since: "1.8" + features: + bulkPublish: true + bulkSubscribe: false - component: Azure Service Bus link: setup-azure-servicebus state: Stable version: v1 since: "1.0" + features: + bulkPublish: true + bulkSubscribe: true diff --git a/daprdocs/data/components/pubsub/generic.yaml b/daprdocs/data/components/pubsub/generic.yaml index ef757e4b3..ab1bf5d01 100644 --- a/daprdocs/data/components/pubsub/generic.yaml +++ b/daprdocs/data/components/pubsub/generic.yaml @@ -13,6 +13,9 @@ state: Stable version: v1 since: "1.5" + features: + bulkPublish: true + bulkSubscribe: true - component: Redis Streams link: setup-redis-pubsub state: Stable diff --git a/daprdocs/layouts/partials/components/pubsub.html b/daprdocs/layouts/partials/components/pubsub.html index 244a6f623..b2ee91a1f 100644 --- a/daprdocs/layouts/partials/components/pubsub.html +++ b/daprdocs/layouts/partials/components/pubsub.html @@ -10,6 +10,8 @@ + + @@ -19,6 +21,8 @@ + + From 13d06288f438f7ba93d070d59cba25517a6c1f43 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Thu, 5 Jan 2023 11:23:09 +0530 Subject: [PATCH 05/11] Fix tag Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index eddf845f6..f49ed8a27 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -124,11 +124,6 @@ Invoke-RestMethod -Method Post -ContentType 'application/json' -Uri 'http://loca {{% /codetab %}} -{{< /tabs >}} -``` - -{{% /codetab %}} - {{< /tabs >}} ## Related links From 48b7ebce27453d51ceadd8d98a67c44c8c4d73b2 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Mon, 9 Jan 2023 16:44:39 +0530 Subject: [PATCH 06/11] Add Java example Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index f49ed8a27..86788fe33 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -26,7 +26,36 @@ Refer [component reference]({{< ref supported-pubsub >}}) to see which component ### Example -{{< tabs Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} +{{< tabs Java Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} + +{{% codetab %}} + +```java +import io.dapr.client.DaprClientBuilder; +import io.dapr.client.DaprPreviewClient; +import io.dapr.client.domain.BulkPublishResponse; +import io.dapr.client.domain.BulkPublishResponseFailedEntry; +import java.util.ArrayList; +import java.util.List; + +class BulkPublisher { + public void publishMessages() { + try (DaprPreviewClient client = (new DaprClientBuilder()).buildPreviewClient()) { + // Create a list of messages to publish + List messages = new ArrayList<>(); + for (int i = 0; i < 10; i++) { + String message = String.format("This is message #%d", i); + messages.add(message); + } + + // Publish list of messages using the bulk publish API + BulkPublishResponse res = client.publishEvents(PUBSUB_NAME, TOPIC_NAME, "text/plain", messages).block(); + } + } +} +``` + +{{% /codetab %}} {{% codetab %}} From b53f417dbdd1f99a0dca01a51cf8ce27dfe0838b Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Wed, 18 Jan 2023 11:19:18 +0530 Subject: [PATCH 07/11] Apply suggestions from code review Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 2 +- daprdocs/content/en/reference/api/pubsub_api.md | 15 ++++++++++----- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index 86788fe33..fd281015d 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -20,7 +20,7 @@ Otherwise, Dapr falls back to sending messages one by one to the underlying pub/ ## Supported components -Refer [component reference]({{< ref supported-pubsub >}}) to see which components support the bulk publish API natively. +Refer to the [component reference]({{< ref supported-pubsub >}}) to see which components support the bulk publish API natively. ## Publishing messages in bulk diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index a7ca07ae0..3bdbe09aa 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -74,7 +74,12 @@ This endpoint lets you publish multiple messages to consumers who are listening POST http://localhost:/v1.0-alpha1/publish/bulk//[?] ``` -The request body should contain a JSON array of entries with unique entry IDs, the event to publish, and the content type of the event. If the content type for an event is not `application/cloudevents+json`, it is auto-wrapped as a CloudEvent (unless `metadata.rawPayload` is set to `true`, see below). +The request body should contain a JSON array of entries with: +- Unique entry IDs +- The event to publish +- The content type of the event + +If the content type for an event is not `application/cloudevents+json`, it is auto-wrapped as a CloudEvent (unless `metadata.rawPayload` is set to `true`). Example: @@ -106,13 +111,13 @@ The `Content-Type` header should always be set to `application/json`. |**Parameter**|**Description**| |--|--| |`daprPort`|The Dapr port| -|`pubsubname`|The name of pubsub component| +|`pubsubname`|The name of pub/sub component| |`topic`|The name of the topic| -|`metadata`|Query parameters for metadata as described below| +|`metadata`|Query parameters for [metadata]({{< ref "pubsub_api.md#metadata" >}})| ### Metadata -Metadata can be sent via query parameters in the request's URL. If must be prefixed with `metadata.` as shown below. +Metadata can be sent via query parameters in the request's URL. It must be prefixed with `metadata.`, as shown in the table below. |**Parameter**|**Description**| |--|--| @@ -125,7 +130,7 @@ Metadata can be sent via query parameters in the request's URL. If must be prefi |**HTTP Status**|**Description**| |--|--| |204|All messages delivered| -|400|Pubsub does not exist| +|400|Pub/sub does not exist| |403|Forbidden by access controls| |500|At least one message failed to be delivered| From 51e8797170e6e99e2b532109f92dd3f6a78a3481 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Wed, 18 Jan 2023 12:16:25 +0530 Subject: [PATCH 08/11] Note about ordering Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index fd281015d..e7ebc89ef 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -24,6 +24,8 @@ Refer to the [component reference]({{< ref supported-pubsub >}}) to see which co ## Publishing messages in bulk +The bulk publish API allows you to publish multiple messages to a topic in a single request. If any of the messages fail to publish, the bulk publish operation returns a list of failed messages. Note, the bulk publish operation does not guarantee the order of messages. + ### Example {{< tabs Java Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} From 07ab3ebda9440f1ca9a9470eb3411cac2fd930e0 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Mon, 30 Jan 2023 10:59:37 +0530 Subject: [PATCH 09/11] Add explicit feature to comp data Signed-off-by: Shubham Sharma --- daprdocs/data/components/pubsub/aws.yaml | 3 +++ daprdocs/data/components/pubsub/gcp.yaml | 3 +++ daprdocs/data/components/pubsub/generic.yaml | 24 ++++++++++++++++++++ 3 files changed, 30 insertions(+) diff --git a/daprdocs/data/components/pubsub/aws.yaml b/daprdocs/data/components/pubsub/aws.yaml index fa5328ab9..07db7e27d 100644 --- a/daprdocs/data/components/pubsub/aws.yaml +++ b/daprdocs/data/components/pubsub/aws.yaml @@ -3,3 +3,6 @@ state: Beta version: v1 since: "1.6" + features: + bulkPublish: false + bulkSubscribe: false diff --git a/daprdocs/data/components/pubsub/gcp.yaml b/daprdocs/data/components/pubsub/gcp.yaml index 8fd547af9..815ced19b 100644 --- a/daprdocs/data/components/pubsub/gcp.yaml +++ b/daprdocs/data/components/pubsub/gcp.yaml @@ -3,3 +3,6 @@ state: Alpha version: v1 since: "1.0" + features: + bulkPublish: false + bulkSubscribe: false diff --git a/daprdocs/data/components/pubsub/generic.yaml b/daprdocs/data/components/pubsub/generic.yaml index ab1bf5d01..02035a7ec 100644 --- a/daprdocs/data/components/pubsub/generic.yaml +++ b/daprdocs/data/components/pubsub/generic.yaml @@ -3,11 +3,17 @@ state: Beta version: v1 since: "1.7" + features: + bulkPublish: false + bulkSubscribe: false - component: In Memory link: setup-inmemory state: Beta version: v1 since: "1.7" + features: + bulkPublish: false + bulkSubscribe: false - component: Apache Kafka link: setup-apache-kafka state: Stable @@ -21,28 +27,46 @@ state: Stable version: v1 since: "1.0" + features: + bulkPublish: false + bulkSubscribe: false - component: JetStream link: setup-jetstream state: Alpha version: v1 since: "1.4" + features: + bulkPublish: false + bulkSubscribe: false - component: Pulsar link: setup-pulsar state: Beta version: v1 since: "1.7" + features: + bulkPublish: false + bulkSubscribe: false - component: MQTT link: setup-mqtt state: Stable version: v1 since: "1.7" + features: + bulkPublish: false + bulkSubscribe: false - component: NATS Streaming link: setup-nats-streaming state: Beta version: v1 since: "1.0" + features: + bulkPublish: false + bulkSubscribe: false - component: RabbitMQ link: setup-rabbitmq state: Stable version: v1 since: "1.7" + features: + bulkPublish: false + bulkSubscribe: false From 73ebc9ac40f0842e30f350da98c4071da4ea9956 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Mon, 30 Jan 2023 12:14:01 +0530 Subject: [PATCH 10/11] Add more examples Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 135 ++++++++++++++++-- .../building-blocks/pubsub/pubsub-overview.md | 2 +- .../content/en/reference/api/pubsub_api.md | 10 +- 3 files changed, 133 insertions(+), 14 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index e7ebc89ef..afaff5430 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -1,7 +1,7 @@ --- type: docs -title: "Send and receive messages in bulk" -linkTitle: "Send and receive messages in bulk" +title: "Publish and subscribe to bulk messages" +linkTitle: "Publish and subscribe to bulk messages" weight: 7100 description: "Learn how to use the bulk publish and subscribe APIs in Dapr." --- @@ -10,7 +10,8 @@ description: "Learn how to use the bulk publish and subscribe APIs in Dapr." The bulk publish and subscribe APIs are in **alpha** stage. {{% /alert %}} -With the bulk publish and subscribe APIs, you can send and receive multiple messages in a single request. +With the bulk publish and subscribe APIs, you can publish and subscribe to multiple messages in a single request. + ## Native bulk publish and subscribe support @@ -28,7 +29,7 @@ The bulk publish API allows you to publish multiple messages to a topic in a sin ### Example -{{< tabs Java Javascript "HTTP API (Bash)" "HTTP API (PowerShell)" >}} +{{< tabs Java Javascript Dotnet Python Go "HTTP API (Bash)" "HTTP API (PowerShell)" >}} {{% codetab %}} @@ -41,6 +42,9 @@ import java.util.ArrayList; import java.util.List; class BulkPublisher { + private static final String PUBSUB_NAME = "my-pubsub-name"; + private static final String TOPIC_NAME = "topic-a"; + public void publishMessages() { try (DaprPreviewClient client = (new DaprClientBuilder()).buildPreviewClient()) { // Create a list of messages to publish @@ -112,19 +116,134 @@ start().catch((e) => { {{% codetab %}} +```csharp +using System; +using System.Collections.Generic; +using Dapr.Client; + +const string PubsubName = "my-pubsub-name"; +const string TopicName = "topic-a"; +IReadOnlyList BulkPublishData = new List() { + new { Id = "17", Amount = 10m }, + new { Id = "18", Amount = 20m }, + new { Id = "19", Amount = 30m } +}; + +using var client = new DaprClientBuilder().Build(); + +var res = await client.BulkPublishEventAsync(PubsubName, TopicName, BulkPublishData); +if (res == null) { + throw new Exception("null response from dapr"); +} +if (res.FailedEntries.Count > 0) +{ + Console.WriteLine("Some events failed to be published!"); + foreach (var failedEntry in res.FailedEntries) + { + Console.WriteLine("EntryId: " + failedEntry.Entry.EntryId + " Error message: " + + failedEntry.ErrorMessage); + } +} +else +{ + Console.WriteLine("Published all events!"); +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```python +import requests +import json + +base_url = "http://localhost:3500/v1.0-alpha1/publish/bulk/{}/{}" +pubsub_name = "my-pubsub-name" +topic_name = "topic-a" +payload = [ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first text message", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second JSON message" + }, + "contentType": "application/json" + } +] + +response = requests.post(base_url.format(pubsub_name, topic_name), json=payload) +print(response.status_code) +``` + +{{% /codetab %}} + +{{% codetab %}} + +```go +package main + +import ( + "fmt" + "strings" + "net/http" + "io/ioutil" +) + +const ( + pubsubName = "my-pubsub-name" + topicName = "topic-a" + baseUrl = "http://localhost:3500/v1.0-alpha1/publish/bulk/%s/%s" +) + +func main() { + url := fmt.Sprintf(baseUrl, pubsubName, topicName) + method := "POST" + payload := strings.NewReader(`[ + { + "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", + "event": "first text message", + "contentType": "text/plain" + }, + { + "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", + "event": { + "message": "second JSON message" + }, + "contentType": "application/json" + } +]`) + + client := &http.Client {} + req, _ := http.NewRequest(method, url, payload) + + req.Header.Add("Content-Type", "application/json") + res, err := client.Do(req) + // ... +} +``` + +{{% /codetab %}} + +{{% codetab %}} + ```bash curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/my-pubsub-name/topic-a \ -H 'Content-Type: application/json' \ -d '[ { "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "event": "first", + "event": "first text message", "contentType": "text/plain" }, { "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", "event": { - "message": "second" + "message": "second JSON message" }, "contentType": "application/json" }, @@ -140,13 +259,13 @@ Invoke-RestMethod -Method Post -ContentType 'application/json' -Uri 'http://loca -Body '[ { "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "event": "first", + "event": "first text message", "contentType": "text/plain" }, { "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", "event": { - "message": "second" + "message": "second JSON message" }, "contentType": "application/json" }, diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index 93bc84314..b7c999a94 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -119,7 +119,7 @@ By default, all topic messages associated with an instance of a pub/sub componen Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). -### Send and receive messages in bulk +### Publish and subscribe to bulk messages Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that require a high throughput. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index 3bdbe09aa..198f71f10 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -89,13 +89,13 @@ curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/pubsubName/deathStar -d '[ { "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "event": "first", + "event": "first text message", "contentType": "text/plain" }, { "entryId": "b1f40bd6-4af2-11ed-b878-0242ac120002", "event": { - "message": "second" + "message": "second JSON message" }, "contentType": "application/json" }, @@ -104,7 +104,7 @@ curl -X POST http://localhost:3500/v1.0-alpha1/publish/bulk/pubsubName/deathStar ### Headers -The `Content-Type` header should always be set to `application/json`. +The `Content-Type` header should always be set to `application/json` since the request body is a JSON array. ### URL Parameters @@ -134,14 +134,14 @@ Metadata can be sent via query parameters in the request's URL. It must be prefi |403|Forbidden by access controls| |500|At least one message failed to be delivered| -The response body is a JSON containing a list of failed entries. Example: +In case of a 500 status code, the response body will contain a JSON object containing a list of entries that failed to be delivered. For example from our request above, if the entry with event `"first text message"` failed to be delivered, the response would contain its entry ID and an error message from the underlying pub/sub component. ```json { "failedEntries": [ { "entryId": "ae6bf7c6-4af2-11ed-b878-0242ac120002", - "error": "error message" + "error": "some error message" }, ], "errorCode": "ERR_PUBSUB_PUBLISH_MESSAGE" From 864db6341f98a54607997cb8dc80113c2c82fb65 Mon Sep 17 00:00:00 2001 From: Shubham Sharma Date: Mon, 30 Jan 2023 13:15:15 +0530 Subject: [PATCH 11/11] Address comments Signed-off-by: Shubham Sharma --- .../building-blocks/pubsub/pubsub-bulk.md | 29 ++++++++++--------- .../building-blocks/pubsub/pubsub-overview.md | 2 +- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md index afaff5430..35c88dd86 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-bulk.md @@ -10,22 +10,15 @@ description: "Learn how to use the bulk publish and subscribe APIs in Dapr." The bulk publish and subscribe APIs are in **alpha** stage. {{% /alert %}} -With the bulk publish and subscribe APIs, you can publish and subscribe to multiple messages in a single request. - - -## Native bulk publish and subscribe support - -When a pub/sub component supports the bulk publish API natively, Dapr also publishes messages to the underlying pub/sub component in bulk. - -Otherwise, Dapr falls back to sending messages one by one to the underlying pub/sub component. This is still more efficient than using the regular publish API, because applications can still send multiple messages in a single request to Dapr. - -## Supported components - -Refer to the [component reference]({{< ref supported-pubsub >}}) to see which components support the bulk publish API natively. +With the bulk publish and subscribe APIs, you can publish and subscribe to multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests between the Dapr sidecar, the application, and the underlying pub/sub broker. ## Publishing messages in bulk -The bulk publish API allows you to publish multiple messages to a topic in a single request. If any of the messages fail to publish, the bulk publish operation returns a list of failed messages. Note, the bulk publish operation does not guarantee the order of messages. +### Restrictions when publishing messages in bulk + +The bulk publish API allows you to publish multiple messages to a topic in a single request. It is *non-transactional*, i.e., from a single bulk request, some messages can succeed and some can fail. If any of the messages fail to publish, the bulk publish operation returns a list of failed messages. + +The bulk publish operation also does not guarantee any ordering of messages. ### Example @@ -276,6 +269,16 @@ Invoke-RestMethod -Method Post -ContentType 'application/json' -Uri 'http://loca {{< /tabs >}} +## How components handle publishing and subscribing to bulk messages + +Some pub/sub brokers support sending and receiving multiple messages in a single request. When a component supports bulk publish or subscribe operations, Dapr runtime uses them to further optimize the communication between the Dapr sidecar and the underlying pub/sub broker. + +For components that do not have bulk publish or subscribe support, Dapr runtime uses the regular publish and subscribe APIs to send and receive messages one by one. This is still more efficient than directly using the regular publish or subscribe APIs, because applications can still send/receive multiple messages in a single request to/from Dapr. + +## Supported components + +Refer to the [component reference]({{< ref supported-pubsub >}}) to see which components support bulk publish and subscribe operations. + ## Related links - List of [supported pub/sub components]({{< ref supported-pubsub >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index b7c999a94..b0015ee2e 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -121,7 +121,7 @@ Dapr can set a timeout message on a per-message basis, meaning that if the messa ### Publish and subscribe to bulk messages -Dapr supports sending and receiving multiple messages in a single request. This is useful for applications that require a high throughput. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). +Dapr supports sending and receiving multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}). ## Try out pub/sub
ComponentBulk PublishBulk Subscribe Status Component version Since runtime version{{ .component }} {{ if .features.bulkPublish }}✅{{else}}{{ end }}{{ if .features.bulkSubscribe }}✅{{else}}{{ end }} {{ .state }} {{ .version }} {{ .since }}