diff --git a/config/nav.yml b/config/nav.yml index 519aabb10..a9721a603 100644 --- a/config/nav.yml +++ b/config/nav.yml @@ -285,6 +285,7 @@ nav: - Configure Apache Kafka Channel defaults: eventing/configuration/kafka-channel-configuration.md - Configure event source defaults: eventing/configuration/sources-configuration.md - Configure Sugar Controller: eventing/configuration/sugar-configuration.md + - Configure KEDA Autoscaling of Knative Kafka Resources: eventing/configuration/keda-configuration.md # Eventing - observability - Observability: - Accessing CloudEvent traces: eventing/accessing-traces.md diff --git a/docs/eventing/brokers/broker-types/kafka-broker/README.md b/docs/eventing/brokers/broker-types/kafka-broker/README.md index 9fec6c036..b3670d299 100644 --- a/docs/eventing/brokers/broker-types/kafka-broker/README.md +++ b/docs/eventing/brokers/broker-types/kafka-broker/README.md @@ -465,6 +465,10 @@ All the configuration mechanisms that are available for the `Kafka` Broker class * A few more configmaps are propagated: `config-tracing` and `kafka-config-logging`. This means, tracing and logging are also not configurable per namespace. * Similarly, the data plane deployments are propagated from the `knative-eventing` namespace to the user namespace. This means that the data plane deployments are not configurable per namespace and will be identical to the ones in the `knative-eventing` namespace. +### Enabling and configuring autoscaling of triggers with KEDA + +To enable and configreu autoscaling of triggers referencing Kafka Brokers with KEDA, follow [the instructions here](../../../configuration/keda-configuration.md). + ## Additional information - To report a bug or request a feature, open an issue in the [eventing-kafka-broker repository](https://github.com/knative-extensions/eventing-kafka-broker). diff --git a/docs/eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md b/docs/eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md index ed67487b4..19ad9a878 100644 --- a/docs/eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md +++ b/docs/eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md @@ -52,9 +52,10 @@ The `brokers.topic.template` values determines the template used to generate the The `channels.topic.template` value determines the template used to generate the kafka topic names used by your channels. -* **Global Key:** `channels.topic.template` -* **Possible values:** Any valid [go text/template](https://pkg.go.dev/text/template) -* **Default:** `{% raw %}messaging-kafka.{{ .Namespace }}.{{ .Name }}{% endraw %}` +* **Global Key:** `controller-autoscaler-keda` +* **Possible values:** One of: `enabled`, `disabled` +* **Default:** `disabled` +* **Stability**: Alpha **Example:** @@ -66,5 +67,10 @@ The `channels.topic.template` value determines the template used to generate the name: config-kafka-features namespace: knative-eventing data: - channels.topic.template: {% raw %}"messaging-kafka.{{ .Namespace }}.{{ .Name }}"{% endraw %} + controller-autoscaler-keda: enabled ``` + +## Dispatcher autoscaling with KEDA + +The `controller-autoscaler-keda` value determines whether Knative Kafka dispatcher components will autoscale with KEDA. +For more information on this feature, please read [the documentation here](../../../configuration/keda-configuration.md) diff --git a/docs/eventing/configuration/kafka-channel-configuration.md b/docs/eventing/configuration/kafka-channel-configuration.md index e0ee35913..9e08fa969 100644 --- a/docs/eventing/configuration/kafka-channel-configuration.md +++ b/docs/eventing/configuration/kafka-channel-configuration.md @@ -62,3 +62,7 @@ To use Kafka Channels, you must: kubectl apply -f .yaml ``` Where `` is the name of the file you created in the previous step. + +## Enable/configure autoscaling of KafkaChannel dispatchers + +To enable autoscaling of the KafkaChannel dispatcher you can read [the instructions here](./keda-configuration.md) diff --git a/docs/eventing/configuration/keda-configuration.md b/docs/eventing/configuration/keda-configuration.md new file mode 100644 index 000000000..e53fd7fd5 --- /dev/null +++ b/docs/eventing/configuration/keda-configuration.md @@ -0,0 +1,67 @@ +# Configure KEDA Autoscaling of Knative Kafka Resources + +All of the Knative Kafka components which dispatch events (source, channel broker) support scaling the dispatcher data plane with KEDA. + +!!! important + This feature is Alpha + +## Enable Autoscaling of Kafka components + +!!! important + Note that enabling autoscaling will enable scaling for all KafkaSources, Triggers referencing Kafka Brokers, and Subscriptions referencing KafkaChannels + +To enable the feature, you need to modify the `config-kafka-features` configmap and set the `controller-autoscaler-keda` flag to `enabled`. +Note that in order for this feature to work, you must also install KEDA into your cluster. + +After your changes your configmap should look like: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-kafka-features + namespace: knative-eventing +data: + controller-autoscaler-keda: enabled + # other features and config options... + +``` + +## Configure Autoscaling for a Resource + +If you want to customize how KEDA scales a KafkaSource, trigger, or subscription you can set annotations on the resource: + +```yaml +apiVersion: .knative.dev/v1 +kind: +metadata: + annotations: + # The minimum number of replicas to scale down to + autoscaling.eventing.knative.dev/min-scale: "0" + # The maximum number of replicas to scale up to + autoscaling.eventing.knative.dev/max-scale: "50" + # The interval in seconds the autoscaler uses to poll metrics in order to inform its scaling decisions + autoscaling.eventing.knative.dev/polling-interval: "10" + # The period in seconds the autoscaler waits until it scales down + autoscaling.eventing.knative.dev/cooldown-period: "30" + # The lag that is used for scaling (1<->N) + autoscaling.eventing.knative.dev/lag-threshold: "100" + # The lag that is used for activation (0<->1) + autoscaling.eventing.knative.dev: "0" +spec: + # Spec fields for the resource... +``` + +## Disable Autoscaling for a Resource + +If you want to disable autoscaling for a KafkaSource, trigger, or subscription, you need to set an annotation on the resource: + +```yaml +apiVersion: .knative.dev/v1 +kind: +metadata: + annotations: + autoscaling.eventing.knative.dev/class: disabled +spec: + # Spec fields for the resource... +``` diff --git a/docs/eventing/sources/kafka-source/README.md b/docs/eventing/sources/kafka-source/README.md index d2631ab65..9fdd2278c 100644 --- a/docs/eventing/sources/kafka-source/README.md +++ b/docs/eventing/sources/kafka-source/README.md @@ -208,26 +208,7 @@ Alternatively, if you are using a GitOps approach, you can add the `consumers` k ### Automatic Scaling with KEDA -Kafka Sources and Brokers for Apache Kafka have (Alpha) support for serverless scaling with KEDA, including scale to zero. If you want Knative and KEDA to scale your Kafka source for you, -you must [install KEDA](https://keda.sh/docs/2.13/deploy/), and then enable the feature flag. - -To enable the feature flag, you need to create or modify the `config-kafka-features` configmap in the `knative-eventing` namespace. You can create the file as below: - -```yaml - - apiVersion: v1 - kind: Configmap - metadata: - name: config-kafka-features - namespace: knative-eventing - data: - controller-autoscaler-keda: "enabled" - -``` - -From there, apply the configmap into your cluster and assuming that KEDA is also installed your Kafka Sources will scale for you! For more information on other values you -can add to the `config-kafka-features` configmap, [read about the Kafka Broker features](../../brokers/broker-types/kafka-broker/configuring-kafka-features). - +You are able to autoscale the KafkaSource with KEDA. For information on how to enable and configure this feature, please read [the instructions here](../../configuration/keda-configuration.md). ### Verify