mirror of https://github.com/knative/docs.git
Add Apache Kafka Broker (#2740)
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
This commit is contained in:
Pierangelo Di Pilato
GitHub
parent
b4bd42c38a
commit
80c67ed2ba
|
|
@ -6,3 +6,7 @@ implementations provided by Knative Eventing.
|
|||
|
||||
## GCP Broker
|
||||
The GCP Broker is optimized for running in GCP. For more details, refer to the [doc](https://github.com/google/knative-gcp/blob/master/docs/install/install-gcp-broker.md).
|
||||
|
||||
## Apache Kafka Broker
|
||||
|
||||
For information about the Apache Kafka broker, see [link](../kafka-broker.md).
|
||||
|
|
|
|||
|
|
@ -0,0 +1,212 @@
|
|||
---
|
||||
title: "Apache Kafka Broker"
|
||||
weight: 30
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
The Apache Kafka Broker is a native Broker implementation, that reduces
|
||||
network hops, supports any Kafka version, and has a better integration
|
||||
with Apache Kafka for the Knative Broker and Trigger model.
|
||||
|
||||
Notable features are:
|
||||
|
||||
- Control plane High Availability
|
||||
- Horizontally scalable data plane
|
||||
- [Extensively configurable](#Kafka-Producer-and-Consumer-configurations)
|
||||
- Ordered delivery of events based on [CloudEvents partitioning extension](https://github.com/cloudevents/spec/blob/master/extensions/partitioning.md)
|
||||
- Support any Kafka version, see [compatibility matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
[Knative Eventing installation](./../../install/any-kubernetes-cluster.md#installing-the-eventing-component).
|
||||
|
||||
## Installation
|
||||
|
||||
1. Install the Kafka broker by entering the following command:
|
||||
|
||||
```bash
|
||||
kubectl apply --filename {{< artifact org="knative-sandbox" repo="eventing-kafka-broker" file="eventing-kafka-broker.yaml" >}}
|
||||
```
|
||||
|
||||
2. Verify that `kafka-broker-controller`, `kafka-broker-receiver` and `kafka-broker-dispatcher` are running,
|
||||
by entering the following command:
|
||||
|
||||
```bash
|
||||
kubectl get deployments.apps -n knative-eventing
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```bash
|
||||
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
eventing-controller 1/1 1 1 10s
|
||||
eventing-webhook 1/1 1 1 9s
|
||||
kafka-broker-controller 1/1 1 1 3s
|
||||
kafka-broker-dispatcher 1/1 1 1 4s
|
||||
kafka-broker-receiver 1/1 1 1 5s
|
||||
```
|
||||
|
||||
## Create a Kafka Broker
|
||||
|
||||
A Kafka Broker object looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: eventing.knative.dev/v1
|
||||
kind: Broker
|
||||
metadata:
|
||||
annotations:
|
||||
# case-sensitive
|
||||
eventing.knative.dev/broker.class: Kafka
|
||||
name: default
|
||||
namespace: default
|
||||
spec:
|
||||
# Configuration specific to this broker.
|
||||
config:
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
name: kafka-broker-config
|
||||
namespace: knative-eventing
|
||||
# Optional dead letter sink, you can specify either:
|
||||
# - deadLetterSink.ref, which is a reference to a Callable
|
||||
# - deadLetterSink.uri, which is an absolute URI to a Callable (It can potentially be out of the Kubernetes cluster)
|
||||
delivery:
|
||||
deadLetterSink:
|
||||
ref:
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Service
|
||||
name: dlq-service
|
||||
```
|
||||
|
||||
`spec.config` should reference any `ConfigMap` that looks like the following:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kafka-broker-config
|
||||
namespace: knative-eventing
|
||||
data:
|
||||
# Number of topic partitions
|
||||
default.topic.partitions: "10"
|
||||
# Replication factor of topic messages.
|
||||
default.topic.replication.factor: "1"
|
||||
# A comma separated list of bootstrap servers. (It can be in or out the k8s cluster)
|
||||
bootstrap.servers: "my-cluster-kafka-bootstrap.kafka:9092"
|
||||
```
|
||||
|
||||
The above `ConfigMap` is installed in the cluster. You can edit
|
||||
the configuration or create a new one with the same values
|
||||
depending on your needs.
|
||||
|
||||
## Set as default broker implementation
|
||||
|
||||
To set the Kafka broker as the default implementation for all brokers in the Knative deployment,
|
||||
you can apply global settings by modifying the `config-br-defaults` ConfigMap in the `knative-eventing` namespace.
|
||||
|
||||
This allows you to avoid configuring individual or per-namespace settings for each broker,
|
||||
such as `metadata.annotations.eventing.knative.dev/broker.class` or `spec.config`.
|
||||
|
||||
<<<<<<< HEAD
|
||||
The following YAML is an example of a config-br-defaults ConfigMap using Kafka broker as the default implementation.
|
||||
=======
|
||||
This allows you to avoid configuring individual or per-namespace settings for each broker, such as `metadata.annotations.eventing.knative.dev/broker.class` or
|
||||
`spec.config`.
|
||||
>>>>>>> 26ab7c8d... lint
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: config-br-defaults
|
||||
namespace: knative-eventing
|
||||
data:
|
||||
default-br-config: |
|
||||
clusterDefault:
|
||||
brokerClass: Kafka
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
name: kafka-broker-config
|
||||
namespace: knative-eventing
|
||||
namespaceDefaults:
|
||||
namespace1:
|
||||
brokerClass: Kafka
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
name: kafka-broker-config
|
||||
namespace: knative-eventing
|
||||
namespace2:
|
||||
brokerClass: Kafka
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
name: kafka-broker-config
|
||||
namespace: knative-eventing
|
||||
```
|
||||
|
||||
## Kafka Producer and Consumer configurations
|
||||
|
||||
Knative exposes all available Kafka producer and consumer configurations that can be modified to suit your workloads.
|
||||
|
||||
You can change these configurations by modifying the `config-kafka-broker-data-plane` `ConfigMap` in
|
||||
the `knative-eventing` namespace.
|
||||
|
||||
Documentation for the settings available in this `ConfigMap` is available on the
|
||||
[Apache Kafka website](https://kafka.apache.org/documentation/),
|
||||
in particular, [Producer configurations](https://kafka.apache.org/documentation/#producerconfigs)
|
||||
and [Consumer configurations](https://kafka.apache.org/documentation/#consumerconfigs).
|
||||
|
||||
## Enable debug logging for data plane components
|
||||
|
||||
The following YAML shows the default logging configuration for data plane components, that is created during the
|
||||
installation step:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kafka-broker-config-logging
|
||||
namespace: knative-eventing
|
||||
data:
|
||||
config.xml: |
|
||||
<configuration>
|
||||
<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
|
||||
</appender>
|
||||
<root level="INFO">
|
||||
<appender-ref ref="jsonConsoleAppender"/>
|
||||
</root>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
To change the logging level to `DEBUG`, you need to:
|
||||
|
||||
1. Apply the following `kafka-broker-config-logging` `ConfigMap` or replace `level="INFO"` with `level="DEBUG"` to the
|
||||
`ConfigMap` `kafka-broker-config-logging`:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kafka-broker-config-logging
|
||||
namespace: knative-eventing
|
||||
data:
|
||||
config.xml: |
|
||||
<configuration>
|
||||
<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
|
||||
</appender>
|
||||
<root level="DEBUG">
|
||||
<appender-ref ref="jsonConsoleAppender"/>
|
||||
</root>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
2. Restart the `kafka-broker-receiver` and the `kafka-broker-dispatcher`, by entering the following commands:
|
||||
|
||||
```bash
|
||||
kubectl rollout restart deployment -n knative-eventing kafka-broker-receiver
|
||||
kubectl rollout restart deployment -n knative-eventing kafka-broker-dispatcher
|
||||
```
|
||||
|
||||
### Additional information
|
||||
|
||||
- To report bugs or add feature requests, open an issue in the [eventing-kafka-broker repository](https://github.com/knative-sandbox/eventing-kafka-broker).
|
||||
|
|
@ -605,8 +605,19 @@ kubectl apply --filename {{< artifact repo="eventing" file="in-memory-channel.ya
|
|||
1. Install a Broker (eventing) layer:
|
||||
|
||||
<!-- This indentation is important for things to render properly. -->
|
||||
|
||||
{{< tabs name="eventing_brokers" default="MT-Channel-based" >}}
|
||||
{{% tab name="Apache Kafka Broker" %}}
|
||||
{{< feature-state version="v0.17" state="alpha" >}}
|
||||
The following command installs the Apache Kafka broker, and runs event routing in a system namespace,
|
||||
`knative-eventing`, by default.
|
||||
|
||||
```bash
|
||||
kubectl apply --filename {{< artifact org="knative-sandbox" repo="eventing-kafka-broker" file="eventing-kafka-broker.yaml" >}}
|
||||
```
|
||||
|
||||
For more information, see the [Kafka Broker](./../eventing/broker/kafka-broker.md) documentation.
|
||||
{{< /tab >}}
|
||||
|
||||
{{% tab name="MT-Channel-based" %}}
|
||||
{{< feature-state version="v0.16" state="stable" >}}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue