dapr
/
docs
mirror of https://github.com/dapr/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

keda how to updates (#793) 

* keda how to updates

* Update README.md

Co-authored-by: Mark Fussell <mfussell@microsoft.com>
This commit is contained in:
Mark Chmarny 2020-09-18 16:12:32 -07:00 committed by GitHub
parent 6e47e400e8
commit e3cd1bb21d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 90 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

133
howto/autoscale-with-keda/README.md
View File

@ -1,88 +1,135 @@
# Autoscaling a Dapr app with KEDA # Autoscaling Dapr application on Kubernetes using KEDA
Dapr is a programming model that's being installed and operated using a sidecar, and thus leaves autoscaling to the hosting layer, for example Kubernetes. Dapr, with its modular building-block approach, along with the 10+ different [pub/sub components](../../concepts/publish-subscribe-messaging), make it easy to write message processing applications. Since Dapr can run in many environments (e.g. VM, bare-metal, Cloud, or Edge) the autoscaling of Dapr applications is managed by the hosting later.
Many of Dapr's [bindings](../../concepts/bindings#supported-bindings-and-specs) overlap with those of [KEDA](https://github.com/kedacore/keda), an Event Driven Autoscaler for Kubernetes.
For apps that use these bindings, it is easy to configure a KEDA autoscaler. For Kubernetes, Dapr integrates with [KEDA](https://github.com/kedacore/keda), an event driven autoscaler for Kubernetes. Many of Dapr's pub/sub components overlap with the scalers provided by [KEDA](https://github.com/kedacore/keda) so it's easy to configure your Dapr deployment on Kubernetes to autoscale based on the back pressure using KEDA.
This how-to walks through the configuration of a scalable Dapr application along with the back pressure on Kafka topic, however you can apply this approach to [pub/sub components](../../concepts/publish-subscribe-messaging) offered by Dapr.
## Install KEDA ## Install KEDA
To install KEDA, follow the [Deploying KEDA](https://keda.sh/docs/latest/deploy/) instructions on the KEDA website. To install KEDA, follow the [Deploying KEDA](https://keda.sh/docs/latest/deploy/) instructions on the KEDA website.
## Create KEDA enabled Dapr binding ## Install Kafka (optional)
For this example, we'll be using Kafka. If you don't have access to a Kafka service, you can install it into your Kubernetes cluster for this example by using Helm:
You can install Kafka in your cluster by using Helm:
```bash ```bash
$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/
$ helm install my-kafka incubator/kafka helm repo update
kubectl create ns kafka
helm install kafka confluentinc/cp-helm-charts -n kafka \
--set cp-schema-registry.enabled=false \
--set cp-kafka-rest.enabled=false \
--set cp-kafka-connect.enabled=false
``` ```
Next, we'll create the Dapr Kafka binding for Kubernetes. To check on the status of the Kafka deployment:
Paste the following in a file named kafka.yaml: ```shell
kubectl rollout status deployment.apps/kafka-cp-control-center -n kafka
kubectl rollout status deployment.apps/kafka-cp-ksql-server -n kafka
kubectl rollout status statefulset.apps/kafka-cp-kafka -n kafka
kubectl rollout status statefulset.apps/kafka-cp-zookeeper -n kafka
```
When done, also deploy the Kafka client and wait until it's ready:
```shell
kubectl apply -n kafka -f deployment/kafka-client.yaml
kubectl wait -n kafka --for=condition=ready pod kafka-client --timeout=120s
```
Next, create the topic which is used in this example (for example `demo-topic`):
> The number of topic partitions is related to the maximum number of replicas KEDA creates for your deployments
```shell
kubectl -n kafka exec -it kafka-client -- kafka-topics \
--zookeeper kafka-cp-zookeeper-headless:2181 \
--topic demo-topic \
--create \
--partitions 10 \
--replication-factor 3 \
--if-not-exists
```
## Deploy a Dapr Pub/Sub component
Next, we'll deploy the Dapr Kafka pub/sub component for Kubernetes. Paste the following YAML into a file named `kafka-pubsub.yaml`:
```yaml ```yaml
apiVersion: dapr.io/v1alpha1 apiVersion: dapr.io/v1alpha1
kind: Component kind: Component
metadata: metadata:
name: kafkaevent name: autoscaling-pubsub
namespace: default
spec: spec:
type: bindings.kafka type: pubsub.kafka
metadata: metadata:
- name: brokers - name: brokers
value: "my-kafka:9092" value: kafka-cp-kafka.kafka.svc.cluster.local:9092
- name: topics - name: authRequired
value: "myTopic" value: "false"
- name: consumerGroup - name: consumerID
value: "group1" value: autoscaling-subscriber
``` ```
The following YAML defines a Kafka component that listens for the topic `myTopic`, with consumer group `group1` and that connects to a broker at `my-kafka:9092`. The above YAML defines the pub/sub component that your application subscribes to, the `demo-topic` we created above. If you used the Kafka Helm install instructions above you can leave the `brokers` value as is. Otherwise, change this to the connection string to your Kafka brokers.
Deploy the binding to the cluster: Also notice the `autoscaling-subscriber` value set for `consumerID` which is used later to make sure that KEDA and your deployment use the same [Kafka partition offset](http://cloudurable.com/blog/kafka-architecture-topics/index.html#:~:text=Kafka%20continually%20appended%20to%20partitions,fit%20on%20a%20single%20server.).
Now, deploy the component to the cluster:
```bash ```bash
$ kubectl apply -f kafka.yaml kubectl apply -f kafka-pubsub.yaml
``` ```
## Create the KEDA autoscaler for Kafka ## Deploy KEDA autoscaler for Kafka
Paste the following to a file named kafka_scaler.yaml, and put the name of your Deployment in the required places: Next, we will deploy the KEDA scaling object that monitors the lag on the specified Kafka topic and configures the Kubernetes Horizontal Pod Autoscaler (HPA) to scale your Dapr deployment in and out.
Paste the following into a file named `kafka_scaler.yaml`, and configure your Dapr deployment in the required place:
```yaml ```yaml
apiVersion: keda.k8s.io/v1alpha1 apiVersion: keda.sh/v1alpha1
kind: ScaledObject kind: ScaledObject
metadata: metadata:
name: kafka-scaler name: subscriber-scaler
namespace: default
labels:
deploymentName: <REPLACE-WITH-DEPLOYMENT-NAME>
spec: spec:
scaleTargetRef: scaleTargetRef:
deploymentName: <REPLACE-WITH-DEPLOYMENT-NAME> name: <REPLACE-WITH-DAPR-DEPLOYMENT-NAME>
pollingInterval: 15
minReplicaCount: 0
maxReplicaCount: 10
triggers: triggers:
- type: kafka - type: kafka
metadata: metadata:
type: kafkaTrigger topic: demo-topic
direction: in bootstrapServers: kafka-cp-kafka.kafka.svc.cluster.local:9092
name: event consumerGroup: autoscaling-subscriber
topic: myTopic lagThreshold: "5"
bootstrapServers: my-kafka:9092
consumerGroup: group1
dataType: binary
lagThreshold: '5'
``` ```
Deploy the KEDA scaler to Kubernetes: A few things to review here in the above file:
* `name` in the `scaleTargetRef` section in the `spec:` is the Dapr ID of your app defined in the Deployment (The value of the `dapr.io/id` annotation)
* `pollingInterval` is the frequency in seconds with which KEDA checks Kafka for current topic partition offset
* `minReplicaCount` is the minimum number of replicas KEDA creates for your deployment. (Note, if your application takes a long time to start it may be better to set that to `1` to ensure at least one replica of your deployment is always running. Otherwise, set that to `0` and KEDA creates the first replica for you)
* `maxReplicaCount` is the maximum number of replicas for your deployment. Given how [Kafka partition offset](http://cloudurable.com/blog/kafka-architecture-topics/index.html#:~:text=Kafka%20continually%20appended%20to%20partitions,fit%20on%20a%20single%20server.) works, you shouldn't set that value higher than the total number of topic partitions
* `topic` in the Kafka `metadata` section which should be set to the same topic to which your Dapr deployment subscribe (In this example `demo-topic`)
* Similarly the `bootstrapServers` should be set to the same broker connection string used in the `kafka-pubsub.yaml` file
* The `consumerGroup` should be set to the same value as the `consumerID` in the `kafka-pubsub.yaml` file
> Note: setting the connection string, topic, and consumer group to the *same* values for both the Dapr service subscription and the KEDA scaler configuration is critical to ensure the autoscaling works correctly.
Next, deploy the KEDA scaler to Kubernetes:
```bash ```bash
$ kubectl apply -f kafka_scaler.yaml kubectl apply -f kafka_scaler.yaml
``` ```
All done! All done!
You can now start publishing messages to your Kafka topic `myTopic` and watch the pods autoscale when the lag threshold is bigger than `5`, as defined in the KEDA scaler manifest. Now, that the `ScaledObject` KEDA object is configured, your deployment will scale based on the lag of the Kafka topic. More information on configuring KEDA for Kafka topics is available [here](https://keda.sh/docs/2.0/scalers/apache-kafka/).
You can now start publishing messages to your Kafka topic `demo-topic` and watch the pods autoscale when the lag threshold is higher than `5` topics, as we have defined in the KEDA scaler manifest. You can publish messages to the Kafka Dapr component by using the Dapr [Publish](https://github.com/dapr/CLI#publishsubscribe) CLI command