From 27f399e87c667114d9c2952f724b65f330382537 Mon Sep 17 00:00:00 2001 From: Artur Souza Date: Wed, 13 Jan 2021 22:27:42 -0800 Subject: [PATCH 1/4] Adding documentation for PubSub Message TTL. --- .../pubsub/pubsub-message-ttl.md | 37 +++++++++++++++++++ .../content/en/reference/api/pubsub_api.md | 13 ++++++- 2 files changed, 49 insertions(+), 1 deletion(-) create mode 100644 daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md new file mode 100644 index 000000000..9b54a4149 --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -0,0 +1,37 @@ +--- +type: docs +title: "Message Time-To-Live" +linkTitle: "Message TTL" +weight: 6000 +description: "Use time-to-live in Pub/Sub messages." +--- + +## Introduction + +Dapr enables per message time-to-live for all Pub/Sub components. It means that applications can set time-to-live per message and subscribers will not receive those messages after expiration. + +In Kafka, time-to-live is configured in the topic via `retention.ms` as per [documentation](https://kafka.apache.org/documentation/#topicconfigs_retention.ms). With message time-to-live in Dapr, applications using Kafka can now set time-to-live per message too, since Dapr will handle the expiration. + +On the other hand, Azure Service Bus supports [entity level time-to-live]((https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-expiration)). It means that messages have a default time-to-live but can also be set with a shorter timespan at publishing time. Dapr will propagate the time-to-live metadata for the message and let Azure Service Bus handle expiration directly. This is referred as "native support" for message TTL. + +### Pub/Sub components with message TTL + +* All, some are implemented natively. + +### Subscribers outside Dapr + +If messages are consumed by subscribers without Dapr, expired messages are not automatically dropped. Still, subscribers can programatically drop expired messages by adding logic to handle the `expiration` attribute in the cloud event. + +> The `expiration` attribute follows the [RFC3339](https://tools.ietf.org/html/rfc3339) format. + +## Native support for message TTL + +When Message Time-To-Live has native support in the component, Dapr will simply forward the time-to-live configuration without adding extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component - like in Azure Service Bus, where expired messages are stored in the dead letter queue and not simply deleted. + +### Pub/Sub components with native support for message TTL + +* Azure Service Bus + +### Subscribers outside Dapr + +In this case, subscribers via Dapr or outside Dapr will not receive expired messages. No extra logic is needed. \ No newline at end of file diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index eedff06ad..3cbfd2e55 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -89,7 +89,7 @@ The following example illustrates this point, considering a subscription for top #### HTTP Request ``` -POST http://localhost:/ +POST http://localhost:/[?] ``` > Note, all URL parameters are case-sensitive. @@ -100,6 +100,17 @@ Parameter | Description --------- | ----------- appPort | the application port path | route path from the subscription configuration +metadata | query parameters for metadata as described below + +##### Metadata + +Metadata can be sent via query parameters in the request's URL. It must be prefixed with `metadata.` as shown below. + +Parameter | Description +--------- | ----------- +metadata.ttlInSeconds | the number of seconds for the message to expire as [described here]({{< ref pubsub-message-ttl.md >}}) + +> Additional metadata parameters can be available based on each pubsub component. #### Expected HTTP Response From d78b747361d65246b429ddbab67912da27b28a3d Mon Sep 17 00:00:00 2001 From: Aaron Crawfis Date: Fri, 15 Jan 2021 15:22:21 -0800 Subject: [PATCH 2/4] Update order --- .../pubsub/pubsub-message-ttl.md | 28 ++++++++----------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md index 9b54a4149..542142512 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -8,30 +8,24 @@ description: "Use time-to-live in Pub/Sub messages." ## Introduction -Dapr enables per message time-to-live for all Pub/Sub components. It means that applications can set time-to-live per message and subscribers will not receive those messages after expiration. +Dapr enables per-message time-to-live (TTL). This means that applications can set time-to-live per message, and subscribers will not receive those messages after expiration. -In Kafka, time-to-live is configured in the topic via `retention.ms` as per [documentation](https://kafka.apache.org/documentation/#topicconfigs_retention.ms). With message time-to-live in Dapr, applications using Kafka can now set time-to-live per message too, since Dapr will handle the expiration. +All Dapr [pub/sub components]({{< ref supported-pubsub >}}) are compatible with message TTL, as Dapr handles the TTL logic within the runtime. -On the other hand, Azure Service Bus supports [entity level time-to-live]((https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-expiration)). It means that messages have a default time-to-live but can also be set with a shorter timespan at publishing time. Dapr will propagate the time-to-live metadata for the message and let Azure Service Bus handle expiration directly. This is referred as "native support" for message TTL. +In some components, such as Kafka, time-to-live can be configured in the topic via `retention.ms` as per [documentation](https://kafka.apache.org/documentation/#topicconfigs_retention.ms). With message TTL in Dapr, applications using Kafka can now set time-to-live per message in addition to per topic. -### Pub/Sub components with message TTL +## Native message TTL support -* All, some are implemented natively. +When message time-to-live has native support in the component, Dapr will simply forward the time-to-live configuration without adding extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component - like in Azure Service Bus, where expired messages are stored in the dead letter queue and not simply deleted. -### Subscribers outside Dapr +### Components with native message TTL support -If messages are consumed by subscribers without Dapr, expired messages are not automatically dropped. Still, subscribers can programatically drop expired messages by adding logic to handle the `expiration` attribute in the cloud event. +#### Azure Service Bus -> The `expiration` attribute follows the [RFC3339](https://tools.ietf.org/html/rfc3339) format. +Azure Service Bus supports [entity level time-to-live]((https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-expiration)). It means that messages have a default time-to-live but can also be set with a shorter timespan at publishing time. Dapr will propagate the time-to-live metadata for the message and let Azure Service Bus handle expiration directly. -## Native support for message TTL +## Non-Dapr subscribers -When Message Time-To-Live has native support in the component, Dapr will simply forward the time-to-live configuration without adding extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component - like in Azure Service Bus, where expired messages are stored in the dead letter queue and not simply deleted. +If messages are consumed by subscribers without Dapr, expired messages are not automatically dropped, as expiration is handled by the Dapr runtime when a Dapr application receives a message. However, subscribers can programmatically drop expired messages by adding logic to handle the `expiration` attribute in the cloud event, which follows the [RFC3339](https://tools.ietf.org/html/rfc3339) format. -### Pub/Sub components with native support for message TTL - -* Azure Service Bus - -### Subscribers outside Dapr - -In this case, subscribers via Dapr or outside Dapr will not receive expired messages. No extra logic is needed. \ No newline at end of file +When non-Dapr subscribers use components such as Azure Service Bus, which natively handle message TTL, they will not receive expired messages. No extra logic is needed. \ No newline at end of file From 1223d5cd736e136f86e08921a49205199b53883b Mon Sep 17 00:00:00 2001 From: Artur Souza Date: Fri, 15 Jan 2021 15:51:37 -0800 Subject: [PATCH 3/4] Fix doc. --- .../pubsub/pubsub-message-ttl.md | 2 +- .../content/en/reference/api/pubsub_api.md | 28 +++++++++---------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md index 542142512..6ce9454e4 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -10,7 +10,7 @@ description: "Use time-to-live in Pub/Sub messages." Dapr enables per-message time-to-live (TTL). This means that applications can set time-to-live per message, and subscribers will not receive those messages after expiration. -All Dapr [pub/sub components]({{< ref supported-pubsub >}}) are compatible with message TTL, as Dapr handles the TTL logic within the runtime. +All Dapr [pub/sub components]({{< ref supported-pubsub >}}) are compatible with message TTL, as Dapr handles the TTL logic within the runtime. Simply set the `ttlInSeconds` metadata when publishing a message. In some components, such as Kafka, time-to-live can be configured in the topic via `retention.ms` as per [documentation](https://kafka.apache.org/documentation/#topicconfigs_retention.ms). With message TTL in Dapr, applications using Kafka can now set time-to-live per message in addition to per topic. diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index 3cbfd2e55..3a1e29626 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -14,7 +14,7 @@ Dapr guarantees at least once semantics for this endpoint. ### HTTP Request ``` -POST http://localhost:/v1.0/publish// +POST http://localhost:/v1.0/publish//[?] ``` ### HTTP Response codes @@ -31,8 +31,9 @@ Code | Description Parameter | Description --------- | ----------- daprPort | the Dapr port -pubsubname | the name of pubsub component. +pubsubname | the name of pubsub component topic | the name of the topic +metadata | query parameters for metadata as described below > Note, all URL parameters are case-sensitive. @@ -44,6 +45,16 @@ curl -X POST http://localhost:3500/v1.0/publish/pubsubName/deathStarStatus \ }' ``` +#### Metadata + +Metadata can be sent via query parameters in the request's URL. It must be prefixed with `metadata.` as shown below. + +Parameter | Description +--------- | ----------- +metadata.ttlInSeconds | the number of seconds for the message to expire as [described here]({{< ref pubsub-message-ttl.md >}}) + +> Additional metadata parameters are available based on each pubsub component. + ## Optional Application (User Code) Routes ### Provide a route for Dapr to discover topic subscriptions @@ -89,7 +100,7 @@ The following example illustrates this point, considering a subscription for top #### HTTP Request ``` -POST http://localhost:/[?] +POST http://localhost:/ ``` > Note, all URL parameters are case-sensitive. @@ -100,17 +111,6 @@ Parameter | Description --------- | ----------- appPort | the application port path | route path from the subscription configuration -metadata | query parameters for metadata as described below - -##### Metadata - -Metadata can be sent via query parameters in the request's URL. It must be prefixed with `metadata.` as shown below. - -Parameter | Description ---------- | ----------- -metadata.ttlInSeconds | the number of seconds for the message to expire as [described here]({{< ref pubsub-message-ttl.md >}}) - -> Additional metadata parameters can be available based on each pubsub component. #### Expected HTTP Response From 2e6ce4dfb36f3793d7c14fadfa0ec7adc4d8a59f Mon Sep 17 00:00:00 2001 From: Aaron Crawfis Date: Fri, 15 Jan 2021 16:16:31 -0800 Subject: [PATCH 4/4] Add example --- .../pubsub/pubsub-message-ttl.md | 45 ++++++++++++++++++- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md index 542142512..42af8b658 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -18,7 +18,7 @@ In some components, such as Kafka, time-to-live can be configured in the topic v When message time-to-live has native support in the component, Dapr will simply forward the time-to-live configuration without adding extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component - like in Azure Service Bus, where expired messages are stored in the dead letter queue and not simply deleted. -### Components with native message TTL support +### Supported components #### Azure Service Bus @@ -28,4 +28,45 @@ Azure Service Bus supports [entity level time-to-live]((https://docs.microsoft.c If messages are consumed by subscribers without Dapr, expired messages are not automatically dropped, as expiration is handled by the Dapr runtime when a Dapr application receives a message. However, subscribers can programmatically drop expired messages by adding logic to handle the `expiration` attribute in the cloud event, which follows the [RFC3339](https://tools.ietf.org/html/rfc3339) format. -When non-Dapr subscribers use components such as Azure Service Bus, which natively handle message TTL, they will not receive expired messages. No extra logic is needed. \ No newline at end of file +When non-Dapr subscribers use components such as Azure Service Bus, which natively handle message TTL, they will not receive expired messages. No extra logic is needed. + +## Example + +Message TTL can be set in the metadata as part of the publishing request: + +{{< tabs curl "Python SDK">}} + +{{% codetab %}} +```bash +curl -X "POST" http://localhost:3500/v1.0/publish/pubsub/TOPIC_A?metadata.ttlInSeconds=120 -H "Content-Type: application/json" -d '{"order-number": "345"}' +``` +{{% /codetab %}} + +{{% codetab %}} +```python +from dapr.clients import DaprClient + +with DaprClient() as d: + req_data = { + 'order-number': '345' + } + # Create a typed message with content type and body + resp = d.publish_event( + pubsub_name='pubsub', + topic='TOPIC_A', + data=json.dumps(req_data), + metadata=( + ('ttlInSeconds', '120') + ) + ) + # Print the request + print(req_data, flush=True) +``` +{{% /codetab %}} + +{{< /tabs >}} + +See [this guide]({{< ref pubsub_api.md >}}) for a full reference on the pub/sub API. + +## Related links +- [Pub/sub API reference]({{< ref pubsub_api.md >}}) \ No newline at end of file