feat: added documentation for keda scaling of Kafka dispatcher components (#6061)

Signed-off-by: Calum Murray <cmurray@redhat.com>

config/nav.yml

@@ -285,6 +285,7 @@ nav:
           - Configure Apache Kafka Channel defaults: eventing/configuration/kafka-channel-configuration.md
           - Configure event source defaults: eventing/configuration/sources-configuration.md
           - Configure Sugar Controller: eventing/configuration/sugar-configuration.md
+          - Configure KEDA Autoscaling of Knative Kafka Resources: eventing/configuration/keda-configuration.md
       # Eventing - observability
       - Observability:
         - Accessing CloudEvent traces: eventing/accessing-traces.md

docs/eventing/brokers/broker-types/kafka-broker/README.md

@@ -465,6 +465,10 @@ All the configuration mechanisms that are available for the `Kafka` Broker class
 * A few more configmaps are propagated: `config-tracing` and `kafka-config-logging`. This means, tracing and logging are also not configurable per namespace.
 * Similarly, the data plane deployments are propagated from the `knative-eventing` namespace to the user namespace. This means that the data plane deployments are not configurable per namespace and will be identical to the ones in the `knative-eventing` namespace.
 
+### Enabling and configuring autoscaling of triggers with KEDA
+
+To enable and configreu autoscaling of triggers referencing Kafka Brokers with KEDA, follow [the instructions here](../../../configuration/keda-configuration.md).
+
 ## Additional information
 
 - To report a bug or request a feature, open an issue in the [eventing-kafka-broker repository](https://github.com/knative-extensions/eventing-kafka-broker).

docs/eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md

@@ -52,9 +52,10 @@ The `brokers.topic.template` values determines the template used to generate the
 
 The `channels.topic.template` value determines the template used to generate the kafka topic names used by your channels.
 
-* **Global Key:** `channels.topic.template`
-* **Possible values:** Any valid [go text/template](https://pkg.go.dev/text/template)
-* **Default:** `{% raw %}messaging-kafka.{{ .Namespace }}.{{ .Name }}{% endraw %}`
+* **Global Key:** `controller-autoscaler-keda`
+* **Possible values:** One of: `enabled`, `disabled`
+* **Default:** `disabled`
+* **Stability**: Alpha
 
 **Example:**
 
@@ -66,5 +67,10 @@ The `channels.topic.template` value determines the template used to generate the
       name: config-kafka-features
       namespace: knative-eventing
     data:
-      channels.topic.template: {% raw %}"messaging-kafka.{{ .Namespace }}.{{ .Name }}"{% endraw %}
+      controller-autoscaler-keda: enabled
     ```
+
+## Dispatcher autoscaling with KEDA
+
+The `controller-autoscaler-keda` value determines whether Knative Kafka dispatcher components will autoscale with KEDA.
+For more information on this feature, please read [the documentation here](../../../configuration/keda-configuration.md)

docs/eventing/configuration/kafka-channel-configuration.md

@@ -62,3 +62,7 @@ To use Kafka Channels, you must:
     kubectl apply -f <filename>.yaml
     ```
     Where `<filename>` is the name of the file you created in the previous step.
+
+## Enable/configure autoscaling of KafkaChannel dispatchers
+
+To enable autoscaling of the KafkaChannel dispatcher you can read [the instructions here](./keda-configuration.md)

docs/eventing/configuration/keda-configuration.md

@@ -0,0 +1,67 @@
+# Configure KEDA Autoscaling of Knative Kafka Resources
+
+All of the Knative Kafka components which dispatch events (source, channel broker) support scaling the dispatcher data plane with KEDA.
+
+!!! important
+    This feature is Alpha
+
+## Enable Autoscaling of Kafka components
+
+!!! important
+    Note that enabling autoscaling will enable scaling for all KafkaSources, Triggers referencing Kafka Brokers, and Subscriptions referencing KafkaChannels
+
+To enable the feature, you need to modify the `config-kafka-features` configmap and set the `controller-autoscaler-keda` flag to `enabled`.
+Note that in order for this feature to work, you must also install KEDA into your cluster.
+
+After your changes your configmap should look like:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-kafka-features
+  namespace: knative-eventing
+data:
+  controller-autoscaler-keda: enabled
+  # other features and config options...
+
+```
+
+## Configure Autoscaling for a Resource
+
+If you want to customize how KEDA scales a KafkaSource, trigger, or subscription you can set annotations on the resource:
+
+```yaml
+apiVersion: <eventing|messaging|sources>.knative.dev/v1
+kind: <KafkaSource|Subscription|Trigger>
+metadata:
+  annotations:
+    # The minimum number of replicas to scale down to
+    autoscaling.eventing.knative.dev/min-scale: "0"
+    # The maximum number of replicas to scale up to
+    autoscaling.eventing.knative.dev/max-scale: "50"
+    # The interval in seconds the autoscaler uses to poll metrics in order to inform its scaling decisions
+    autoscaling.eventing.knative.dev/polling-interval: "10"
+    # The period in seconds the autoscaler waits until it scales down
+    autoscaling.eventing.knative.dev/cooldown-period: "30"
+    # The lag that is used for scaling (1<->N)
+    autoscaling.eventing.knative.dev/lag-threshold: "100"
+    # The lag that is used for activation (0<->1)
+    autoscaling.eventing.knative.dev: "0"
+spec:
+  # Spec fields for the resource...
+```
+
+## Disable Autoscaling for a Resource
+
+If you want to disable autoscaling for a KafkaSource, trigger, or subscription, you need to set an annotation on the resource:
+
+```yaml
+apiVersion: <eventing|messaging|sources>.knative.dev/v1
+kind: <KafkaSource|Subscription|Trigger>
+metadata:
+  annotations:
+    autoscaling.eventing.knative.dev/class: disabled
+spec:
+  # Spec fields for the resource...
+```

docs/eventing/sources/kafka-source/README.md

@@ -208,26 +208,7 @@ Alternatively, if you are using a GitOps approach, you can add the `consumers` k
 
 ### Automatic Scaling with KEDA
 
-Kafka Sources and Brokers for Apache Kafka have (Alpha) support for serverless scaling with KEDA, including scale to zero. If you want Knative and KEDA to scale your Kafka source for you,
-you must [install KEDA](https://keda.sh/docs/2.13/deploy/), and then enable the feature flag.
-
-To enable the feature flag, you need to create or modify the `config-kafka-features` configmap in the `knative-eventing` namespace. You can create the file as below:
-
-```yaml
-
-    apiVersion: v1
-    kind: Configmap
-    metadata:
-      name: config-kafka-features
-      namespace: knative-eventing
-    data:
-      controller-autoscaler-keda: "enabled"
-
-```
-
-From there, apply the configmap into your cluster and assuming that KEDA is also installed your Kafka Sources will scale for you! For more information on other values you
-can add to the `config-kafka-features` configmap, [read about the Kafka Broker features](../../brokers/broker-types/kafka-broker/configuring-kafka-features).
-
+You are able to autoscale the KafkaSource with KEDA. For information on how to enable and configure this feature, please read [the instructions here](../../configuration/keda-configuration.md).
 
 ### Verify