Merge pull request #3159 from artursouza/subscribe_statefulset

Subscribe with StatefulSet

daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-subscribe-statefulset.md

@@ -0,0 +1,94 @@
+---
+type: docs
+title: "How to: Subscribe to messages using StatefulSet"
+linkTitle: "How to: Subscribe with StatefulSet"
+weight: 2000
+description: "Learn how to subscribe with StatefulSet and scale horizontally with consistent consumer IDs"
+---
+
+Unlike Deployments, where Pods are ephemeral, [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) allows deployment of stateful applications on Kubernetes by keeping a sticky identity for each Pod.
+
+Below is an example of a StatefulSet with Dapr:
+```yaml
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: python-subscriber
+spec:
+  selector:
+    matchLabels:
+      app: python-subscriber  # has to match .spec.template.metadata.labels
+  serviceName: "python-subscriber"
+  replicas: 3
+  template:
+    metadata:
+      labels:
+        app: python-subscriber # has to match .spec.selector.matchLabels
+      annotations:
+        dapr.io/enabled: "true"
+        dapr.io/app-id: "python-subscriber"
+        dapr.io/app-port: "5001"
+    spec:
+      containers:
+      - name: python-subscriber
+        image: ghcr.io/dapr/samples/pubsub-python-subscriber:latest
+        ports:
+        - containerPort: 5001
+        imagePullPolicy: Always
+```
+
+When subscribing to a pub/sub topic via Dapr, the application can define the `consumerID`, which determines the subscriber's position in the queue or topic. With the StatefulSets sticky identity of Pods, you can have a unique `consumerID` per Pod, allowing each horizontal scale of the subscriber application. Dapr keeps track of the name of each Pod, which can be used when declaring components using the `{podName}` marker.
+
+On scaling the number of subscribers of a given topic, each Dapr component has unique settings that determine the behavior. Usually, there are two options for multiple consumers:
+
+ - Broadcast: each message published to the topic will be consumed by all subscribers.
+ - Shared: a message is consumed by any subscriber (but not all).
+
+Kafka isolates each subscriber by `consumerID` with its own position in the topic. When an instance restarts, it reuses the same `consumerID` and continues from its last known position, without skipping messages. The component below demonstrates how a Kafka component can be used by multiple Pods:
+
+```yaml
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: pubsub
+spec:
+  type: pubsub.kafka
+  version: v1
+  metadata:
+  - name: brokers
+    value: my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
+  - name: consumerID
+    value: "{podName}"
+  - name: authRequired
+    value: "false"
+```
+
+The MQTT3 protocol has shared topics, allowing multiple subscribers to "compete" for messages from the topic, meaning a message is only processed by one of them. For example:
+
+```yaml
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: mqtt-pubsub
+spec:
+  type: pubsub.mqtt3
+  version: v1
+  metadata:
+    - name: consumerID
+      value: "{podName}"
+    - name: cleanSession
+      value: "true"
+    - name: url
+      value: "tcp://admin:public@localhost:1883"
+    - name: qos
+      value: 1
+    - name: retain
+      value: "false"
+```
+
+## Next steps
+
+- Try the [pub/sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub).
+- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}).
+- Review the list of [pub/sub components]({{< ref setup-pubsub >}}).
+- Read the [API reference]({{< ref pubsub_api.md >}}).

daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md

@@ -123,6 +123,10 @@ Dapr can set a timeout message on a per-message basis, meaning that if the messa
 
 Dapr supports sending and receiving multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests. For more information, read [pub/sub bulk messages]({{< ref pubsub-bulk.md >}}).
 
+### Scaling subscribers with StatefulSets
+
+When running on Kubernetes, subscribers can have a sticky `consumerID` per instance when using StatefulSets in combination with the `{podName}` marker. See [how to horizontally scale subscribers with StatefulSets]({{< ref "howto-subscribe-statefulset.md" >}}).
+
 ## Try out pub/sub
 
 ### Quickstarts and tutorials

daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt3.md

@@ -92,7 +92,7 @@ Note that while the `caCert` and `clientCert` values may not be secrets, they ca
 
 ### Consuming a shared topic
 
-When consuming a shared topic, each consumer must have a unique identifier. By default, the application ID is used to uniquely identify each consumer and publisher. In self-hosted mode, invoking each `dapr run` with a different application ID is sufficient to have them consume from the same shared topic. However, on Kubernetes, multiple instances of an application pod will share the same application ID, prohibiting all instances from consuming the same topic. To overcome this, configure the component's `consumerID` metadata with a `{uuid}` tag, which will give each instance a randomly generated `consumerID` value on start up. For example:
+When consuming a shared topic, each consumer must have a unique identifier. By default, the application ID is used to uniquely identify each consumer and publisher. In self-hosted mode, invoking each `dapr run` with a different application ID is sufficient to have them consume from the same shared topic. However, on Kubernetes, multiple instances of an application pod will share the same application ID, prohibiting all instances from consuming the same topic. To overcome this, configure the component's `consumerID` metadata with a `{uuid}` tag (which will give each instance a randomly generated value on start up) or `{podName}` (which will use the Pod's name on Kubernetes). For example:
 
 ```yaml
 apiVersion: dapr.io/v1alpha1
@@ -121,6 +121,8 @@ The above example uses secrets as plain strings. It is recommended to use a secr
 
 Note that in the case, the value of the consumer ID is random every time Dapr restarts, so you should set `cleanSession` to `true` as well.
 
+It is recommended to use [StatefulSets]({{< ref "howto-subscribe-statefulset.md" >}}) with shared subscriptions.
+
 ## Create a MQTT3 broker
 
 {{< tabs "Self-Hosted" "Kubernetes">}}