[Multi-tenancy/namespace consumer group] Create howto and add to overview (#2864)

* initial draft of namespace consumer groups doc

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* updates per yaron

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* address marks comments in issue

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* updates per Mark

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* forgot namespace update

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* update bus to broker

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>
Co-authored-by: greenie-msft <56556602+greenie-msft@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
This commit is contained in:
Hannah Hunter 2022-10-07 11:37:16 -05:00 committed by GitHub
parent ce4600bd41
commit 47375831be
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 84 additions and 6 deletions

View File

@ -0,0 +1,74 @@
---
type: docs
title: "How to: Set up pub/sub namespace consumer groups"
linkTitle: "How to: Namespace consumer groups"
weight: 5000
description: "Learn how to use the metadata-based namespace consumer group in your component"
---
You've set up [Dapr's pub/sub API building block]({{< ref pubsub-overview >}}), and your applications are publishing and subscribing to topics smoothly, using a centralized message broker. What if you'd like to perform simple A/B testing, blue/green deployments, or even canary deployments for your applications? Even with using Dapr, this can prove difficult.
Dapr solves multi-tenancy at-scale with its pub/sub namespace consumer groups construct.
## Without namespace consumer groups
Let's say you have a Kubernetes cluster, with two applications (App1 and App2) deployed to the same namespace (namespace-a). App2 publishes to a topic called `order`, while App1 subscribes to the topic called `order`. This will create two consumer groups, named after your applications (App1 and App2).
<img src="/images/howto-namespace/basic-pubsub.png" width=1000 alt="Diagram showing basic pubsub process.">
In order to perform simple testing and deployments while using a centralized message broker, you create another namespace with two applications of the same `app-id`, App1 and App2.
Dapr creates consumer groups using the `app-id` of individual applications, so the consumer group names will remain App1 and App2.
<img src="/images/howto-namespace/without-namespace.png" width=1000 alt="Diagram showing complications around multi-tenancy without Dapr namespace consumer groups.">
To avoid this, you'd then need to have something "creep" into your code to change the `app-id`, depending on the namespace on which you're running. This workaround is cumbersome and a significant painpoint.
## With namespace consumer groups
Not only can Dapr allow you to change the behavior of a consumer group with a consumerID for your UUID and pod names, Dapr also provides a **namespace construct** that lives in the pub/sub component metadata. For example, using Redis as your message broker:
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.redis
version: v1
metadata:
- name: redisHost
value: localhost:6379
- name: redisPassword
value: ""
- name: consumerID
value: "{namespace}"
```
By configuring `consumerID` with the `{namespace}` value, you'll be able to use the same `app-id` with the same topics from different namespaces.
<img src="/images/howto-namespace/with-namespace.png" width=1000 alt="Diagram showing how namespace consumer groups help with multi-tenancy.">
In the diagram above, you have two namespaces, each with applications of the same `app-id`, publishing and subscribing to the same centralized message broker `orders`. This time, however, Dapr has created consumer group names prefixed with the namespace in which they're running.
Without you needing to change your code/`app-id`, the namespace consumer group allows you to:
- Add more namespaces
- Keep the same topics
- Keep the same `app-id` across namespaces
- Have your entire deployment pipeline remain intact
Simply include the `"{namespace}"` consumer group construct in your component metadata. You don't need to encode the namespace in the metadata. Dapr understands the namespace it is running in and completes the namespace value for you, like a dynamic metadata value injected by the runtime.
{{% alert title="Note" color="primary" %}}
If you add the namespace consumer group to your metadata afterwards, Dapr updates everything for you. This means that you can add namespace metadata value to existing pub/sub deployments.
{{% /alert %}}
## Demo
Watch [this video for an overview on pub/sub multi-tenancy](https://youtu.be/eK463jugo0c?t=1188):
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/eK463jugo0c?start=1188" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Next steps
- Learn more about configuring Pub/Sub components with multiple namespaces [pub/sub namespaces]({{< ref pubsub-namespaces >}}).

View File

@ -2,7 +2,7 @@
type: docs type: docs
title: "Message Time-to-Live (TTL)" title: "Message Time-to-Live (TTL)"
linkTitle: "Message TTL" linkTitle: "Message TTL"
weight: 6000 weight: 7000
description: "Use time-to-live in pub/sub messages." description: "Use time-to-live in pub/sub messages."
--- ---

View File

@ -32,7 +32,7 @@ When using pub/sub in Dapr:
1. The pub/sub building block makes calls into a Dapr pub/sub component that encapsulates a specific message broker. 1. The pub/sub building block makes calls into a Dapr pub/sub component that encapsulates a specific message broker.
1. To receive messages on a topic, Dapr subscribes to the pub/sub component on behalf of your service with a topic and delivers the messages to an endpoint on your service when they arrive. 1. To receive messages on a topic, Dapr subscribes to the pub/sub component on behalf of your service with a topic and delivers the messages to an endpoint on your service when they arrive.
In the diagram below, a "shipping" service and an "email" service have both subscribed to topics published by a "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component; for example: Redis Streams, NATS Streaming, Azure Service Bus, or GCP pub/sub. In the diagram below, a "shipping" service and an "email" service have both subscribed to topics published by a "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message broker component; for example: Redis Streams, NATS Streaming, Azure Service Bus, or GCP pub/sub.
<img src="/images/pubsub-overview-components.png" width=1000> <img src="/images/pubsub-overview-components.png" width=1000>
<br></br> <br></br>
@ -94,6 +94,10 @@ For more information on message routing, read [Dapr pub/sub API reference]({{< r
Sometimes, messages can't be processed because of a variety of possible issues, such as erroneous conditions within the producer or consumer application or an unexpected state change that causes an issue with your application code. Dapr allows developers to set dead letter topics to deal with messages that cannot be delivered to an application. This feature is available on all pub/sub components and prevents consumer applications from endlessly retrying a failed message. For more information, read about [dead letter topics]({{< ref "pubsub-deadletter.md">}}) Sometimes, messages can't be processed because of a variety of possible issues, such as erroneous conditions within the producer or consumer application or an unexpected state change that causes an issue with your application code. Dapr allows developers to set dead letter topics to deal with messages that cannot be delivered to an application. This feature is available on all pub/sub components and prevents consumer applications from endlessly retrying a failed message. For more information, read about [dead letter topics]({{< ref "pubsub-deadletter.md">}})
### Namespace consumer groups
Dapr solves multi-tenancy at-scale with [namespaces for consumer groups]({{< ref howto-namespace >}}). Simply include the `"{namespace}"` value in your component metadata for consumer groups to allow multiple namespaces with applications of the same `app-id` to publish and subscribe to the same message broker.
### At-least-once guarantee ### At-least-once guarantee
Dapr guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the pub/sub API, Dapr ensures the message is delivered *at least once* to every subscriber. Dapr guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the pub/sub API, Dapr ensures the message is delivered *at least once* to every subscriber.

View File

@ -2,7 +2,7 @@
type: docs type: docs
title: "Scope Pub/sub topic access" title: "Scope Pub/sub topic access"
linkTitle: "Scope topic access" linkTitle: "Scope topic access"
weight: 5000 weight: 6000
description: "Use scopes to limit pub/sub topics to specific applications" description: "Use scopes to limit pub/sub topics to specific applications"
--- ---

View File

@ -1,7 +1,7 @@
--- ---
type: docs type: docs
title: "Service invocation across namespaces" title: "How to: Service invocation across namespaces"
linkTitle: "Service invocation namespaces" linkTitle: "How to: Service invocation namespaces"
weight: 1000 weight: 1000
description: "Call between services deployed to different namespaces" description: "Call between services deployed to different namespaces"
--- ---

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

@ -1 +1 @@
Subproject commit be0277d815e5dd4e107ddffd1ca21cc4502806a5 Subproject commit 8bbfdbdcb536faaa338db9fd05bc73fd746c01b9