mirror of https://github.com/dapr/docs.git
Subscribe with StatefulSet
Signed-off-by: Artur Souza <asouza.pro@gmail.com>
This commit is contained in:
Artur Souza
parent
08671d3c95
commit
25355f1280
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How to: Subscribe to messages using StatefulSet"
|
||||
linkTitle: "How to: Subscribe with StatefulSet"
|
||||
weight: 2000
|
||||
description: "Learn how to subscribe with StatefulSet and scale horizontally with consistent consumer IDs"
|
||||
---
|
||||
|
||||
[StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) allows deployment of stateful applications on Kubernetes by keeping a sticky identity for each Pod, in contrast to Deployments - where Pods are ephemeral.
|
||||
|
||||
See an example of a StatefulSet with Dapr:
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: python-subscriber
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: python-subscriber # has to match .spec.template.metadata.labels
|
||||
serviceName: "python-subscriber"
|
||||
replicas: 3
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: python-subscriber # has to match .spec.selector.matchLabels
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "python-subscriber"
|
||||
dapr.io/app-port: "5001"
|
||||
spec:
|
||||
containers:
|
||||
- name: python-subscriber
|
||||
image: ghcr.io/dapr/samples/pubsub-python-subscriber:latest
|
||||
ports:
|
||||
- containerPort: 5001
|
||||
imagePullPolicy: Always
|
||||
```
|
||||
|
||||
When subscribing to a pub/sub topic via Dapr, the application can define the `consumerID`, which is used to determine the subscriber's position in the queue or topic. With StatefulSets, developers can take advantage of this sticky identity of Pods and have the `consumerID` to be unique per Pod - allowing each horizontal scale of the subscriber application. Dapr keeps track of the name of each Pod and it can be used when declaring components by using the `{podName}` marker.
|
||||
|
||||
On scaling the number of subscribers of a given topic, each component in Dapr has its unique settings to determine the behavior. Usually, there are two options for multiple consumers: broadcast or shared. In the broadcast configuration, each message published to the topic will be consumed by all subscribers. While in the shared option, a message is consumed by any subscriber (but not all).
|
||||
|
||||
Kafka isolates each subscriber by `consumerID` with its own position in the topic. When an instance restarts, it reuses the same `consumerID` and continues from its last known position, without skipping messages. The component below is an example on how a Kafka component can be used by multiple Pods:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: pubsub
|
||||
spec:
|
||||
type: pubsub.kafka
|
||||
version: v1
|
||||
metadata:
|
||||
- name: brokers
|
||||
value: my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
|
||||
- name: consumerID
|
||||
value: "{podName}"
|
||||
- name: authRequired
|
||||
value: "false"
|
||||
```
|
||||
|
||||
The MQTT3 protocol has shared topics, allowing multiple subscribers to "compete" for messages from the topic, meaning a message is only processed by one of them. For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: mqtt-pubsub
|
||||
spec:
|
||||
type: pubsub.mqtt3
|
||||
version: v1
|
||||
metadata:
|
||||
- name: consumerID
|
||||
value: "{podName}"
|
||||
- name: cleanSession
|
||||
value: "true"
|
||||
- name: url
|
||||
value: "tcp://admin:public@localhost:1883"
|
||||
- name: qos
|
||||
value: 1
|
||||
- name: retain
|
||||
value: "false"
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
- Try the [pub/sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub).
|
||||
- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}).
|
||||
- Review the list of [pub/sub components]({{< ref setup-pubsub >}}).
|
||||
- Read the [API reference]({{< ref pubsub_api.md >}}).
|
||||
|
|
@ -123,6 +123,10 @@ Dapr can set a timeout message on a per-message basis, meaning that if the messa
|
|||
|
||||
Dapr supports sending and receiving multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}).
|
||||
|
||||
### Scaling subscribers with StatefulSets
|
||||
|
||||
When running on Kubernetes, subscribers can have a sticky `consumerID` per instance when using StatefulSets in combination with the `{podName}` marker. See [how to horizontally scale subscribers with StatefulSets]({{< ref "howto-subscribe-statefulset.md" >}}).
|
||||
|
||||
## Try out pub/sub
|
||||
|
||||
### Quickstarts and tutorials
|
||||
|
|
|
|||
|
|
@ -92,7 +92,7 @@ Note that while the `caCert` and `clientCert` values may not be secrets, they ca
|
|||
|
||||
### Consuming a shared topic
|
||||
|
||||
When consuming a shared topic, each consumer must have a unique identifier. By default, the application ID is used to uniquely identify each consumer and publisher. In self-hosted mode, invoking each `dapr run` with a different application ID is sufficient to have them consume from the same shared topic. However, on Kubernetes, multiple instances of an application pod will share the same application ID, prohibiting all instances from consuming the same topic. To overcome this, configure the component's `consumerID` metadata with a `{uuid}` tag, which will give each instance a randomly generated `consumerID` value on start up. For example:
|
||||
When consuming a shared topic, each consumer must have a unique identifier. By default, the application ID is used to uniquely identify each consumer and publisher. In self-hosted mode, invoking each `dapr run` with a different application ID is sufficient to have them consume from the same shared topic. However, on Kubernetes, multiple instances of an application pod will share the same application ID, prohibiting all instances from consuming the same topic. To overcome this, configure the component's `consumerID` metadata with a `{uuid}` tag (which will give each instance a randomly generated value on start up) or `{podName}` (which will use the Pod's name on Kubernetes). For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
|
|
@ -121,6 +121,8 @@ The above example uses secrets as plain strings. It is recommended to use a secr
|
|||
|
||||
Note that in the case, the value of the consumer ID is random every time Dapr restarts, so you should set `cleanSession` to `true` as well.
|
||||
|
||||
It is recommended to use [StatefulSets]({{< ref "howto-subscribe-statefulset.md" >}}) with shared subscriptions.
|
||||
|
||||
## Create a MQTT3 broker
|
||||
|
||||
{{< tabs "Self-Hosted" "Kubernetes">}}
|
||||
|
|
|
|||
Loading…
Reference in New Issue