dapr
/
docs
mirror of https://github.com/dapr/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Describing Kafka consumer behaviour when consuming from multiple topics (#3931) 

* Describing Kafka consumer behaviour when consuming from multiple topics

Signed-off-by: Simon Headley <headleysj@gmail.com>

* Further specifying the behaviour of Kafka consumer groups on topics.

Signed-off-by: Simon Headley <headleysj@gmail.com>

* Changing 'this will guarantee' to 'this guarantees' as per PR review

Signed-off-by: Simon Headley <headleysj@gmail.com>

* Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md

Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com>
Signed-off-by: Mark Fussell <markfussell@gmail.com>

---------

Signed-off-by: Simon Headley <headleysj@gmail.com>
Signed-off-by: Mark Fussell <markfussell@gmail.com>
Co-authored-by: Mark Fussell <markfussell@gmail.com>
Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com>
This commit is contained in:
Simon Headley 2024-01-12 00:38:57 +02:00 committed by GitHub
parent 946c2c8b61
commit 390de66da8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 38 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md
View File

@ -299,6 +299,44 @@ auth:
secretStore: <SECRET_STORE_NAME>
```
## Consuming from multiple topics
When consuming from multiple topics using a single pub/sub component, there is no guarantee about how the consumers in your consumer group are balanced across the topic partitions.
For instance, let's say you are subscribing to two topics with 10 partitions per topic and you have 20 replicas of your service consuming from the two topics. There is no guarantee that 10 will be assigned to the first topic and 10 to the second topic. Instead, the partitions could be divided unequally, with more than 10 assigned to the first topic and the rest assigned to the second topic.
This can result in idle consumers listening to the first topic and over-extended consumers on the second topic, or vice versa. This same behavior can be observed when using auto-scalers such as HPA or KEDA.
If you run into this particular issue, it is recommended that you configure a single pub/sub component per topic with uniquely defined consumer groups per component. This guarantees that all replicas of your service are fully allocated to the unique consumer group, where each consumer group targets one specific topic.
For example, you may define two Dapr components with the following configuration:
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub-topic-one
spec:
type: pubsub.kafka
version: v1
metadata:
- name: consumerGroup
value: "{appID}-topic-one"
```
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub-topic-two
spec:
type: pubsub.kafka
version: v1
metadata:
- name: consumerGroup
value: "{appID}-topic-two"
```
## Sending and receiving multiple messages
Apache Kafka component supports sending and receiving multiple messages in a single operation using the bulk Pub/sub API.