From d88633360688c6e70f34785705b391526de3919d Mon Sep 17 00:00:00 2001 From: Matthias Wessendorf Date: Thu, 6 Feb 2020 17:02:31 +0100 Subject: [PATCH] Adding better Kafka install doc (#2156) * :lipstick: Adding better Kafka install hints, including a script, that installs _latest_ of Strimzi, as very small cluster Signed-off-by: Matthias Wessendorf * Adding some steps for pure/plain yaml, but still giving folks the script, if they just one step, instead of some yaml steps ... Co-Authored-By: Ali Ok * go with ephemeral instead of PVC Co-authored-by: Ali Ok --- docs/eventing/samples/kafka/README.md | 73 +++++++++++++++++++++- docs/eventing/samples/kafka/kafka.yaml | 25 ++++++++ docs/eventing/samples/kafka/kafka_setup.sh | 27 ++++++++ 3 files changed, 123 insertions(+), 2 deletions(-) create mode 100644 docs/eventing/samples/kafka/kafka.yaml create mode 100755 docs/eventing/samples/kafka/kafka_setup.sh diff --git a/docs/eventing/samples/kafka/README.md b/docs/eventing/samples/kafka/README.md index 6cf187b3e..087883ac0 100644 --- a/docs/eventing/samples/kafka/README.md +++ b/docs/eventing/samples/kafka/README.md @@ -9,9 +9,78 @@ All examples require: - Knative Serving v0.9+ - An Apache Kafka cluster -If you want to run the Apache Kafka cluster on Kubernetes, the simplest option is to install it by using Strimzi. Check out the [Quickstart](https://strimzi.io/quickstarts/) guides for both Minikube and Openshift. You can also install Kafka on the host. +### Setting up Apache Kafka -## Examples +If you want to run the Apache Kafka cluster on Kubernetes, the simplest option is to install it by using [Strimzi](https://strimzi.io). + +1. Create a namespace for your Apache Kafka installation, like `kafka`: + ```shell + kubectl create namespace kafka + ``` +1. Install the Strimzi operator, like: + ```shell + curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.2/strimzi-cluster-operator-0.16.2.yaml" \ + | sed 's/namespace: .*/namespace: kafka/' \ + | kubectl -n kafka apply -f - + ``` +1. Describe the size of your Apache Kafka installation, like: + ```yaml + apiVersion: kafka.strimzi.io/v1beta1 + kind: Kafka + metadata: + name: my-cluster + spec: + kafka: + version: 2.4.0 + replicas: 1 + listeners: + plain: {} + tls: {} + config: + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + log.message.format.version: "2.4" + storage: + type: ephemeral + zookeeper: + replicas: 3 + storage: + type: ephemeral + entityOperator: + topicOperator: {} + userOperator: {} + ``` +1. Deploy the Apache Kafka cluster + ``` + $ kubectl apply -n kafka -f kafka.yaml + ``` + +This will install a small, non-production, cluster of Apache Kafka. To verify your installation, +check if the pods for Strimzi are all up, in the `kafka` namespace: + + ```shell + $ kubectl get pods -n kafka + NAME READY STATUS RESTARTS AGE + my-cluster-entity-operator-65995cf856-ld2zp 3/3 Running 0 102s + my-cluster-kafka-0 2/2 Running 0 2m8s + my-cluster-zookeeper-0 2/2 Running 0 2m39s + my-cluster-zookeeper-1 2/2 Running 0 2m49s + my-cluster-zookeeper-2 2/2 Running 0 2m59s + strimzi-cluster-operator-77555d4b69-sbrt4 1/1 Running 0 3m14s + ``` + +> NOTE: For production ready installs check [Strimzi](https://strimzi.io). + +### Installation script + +If you want to install the latest version of Strimzi, in just one step, we have a [script](./kafka_setup.sh) for your convenience, which does exactly the same steps that are listed above: + +```shell +$ ./kafka_setup.sh +``` + +## Examples of Apache Kafka and Knative A number of different examples, showing the `KafkaSource` and the `KafkaChannel` can be found here: diff --git a/docs/eventing/samples/kafka/kafka.yaml b/docs/eventing/samples/kafka/kafka.yaml new file mode 100644 index 000000000..9188d7609 --- /dev/null +++ b/docs/eventing/samples/kafka/kafka.yaml @@ -0,0 +1,25 @@ +apiVersion: kafka.strimzi.io/v1beta1 +kind: Kafka +metadata: + name: my-cluster +spec: + kafka: + version: 2.4.0 + replicas: 1 + listeners: + plain: {} + tls: {} + config: + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + log.message.format.version: "2.4" + storage: + type: ephemeral + zookeeper: + replicas: 3 + storage: + type: ephemeral + entityOperator: + topicOperator: {} + userOperator: {} diff --git a/docs/eventing/samples/kafka/kafka_setup.sh b/docs/eventing/samples/kafka/kafka_setup.sh new file mode 100755 index 000000000..0d0fa51bb --- /dev/null +++ b/docs/eventing/samples/kafka/kafka_setup.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +set -e +# Turn colors in this script off by setting the NO_COLOR variable in your +# environment to any value: +# +# $ NO_COLOR=1 test.sh +NO_COLOR=${NO_COLOR:-""} +if [ -z "$NO_COLOR" ]; then + header=$'\e[1;33m' + reset=$'\e[0m' +else + header='' + reset='' +fi +strimzi_version=`curl https://github.com/strimzi/strimzi-kafka-operator/releases/latest | awk -F 'tag/' '{print $2}' | awk -F '"' '{print $1}' 2>/dev/null` +function header_text { + echo "$header$*$reset" +} +header_text "Using Strimzi Version: ${strimzi_version}" +header_text "Strimzi install" +kubectl create namespace kafka +curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/${strimzi_version}/strimzi-cluster-operator-${strimzi_version}.yaml" \ + | sed 's/namespace: .*/namespace: kafka/' \ + | kubectl -n kafka apply -f - +header_text "Applying Strimzi Cluster file" +kubectl -n kafka apply -f "https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/${strimzi_version}/examples/kafka/kafka-ephemeral-single.yaml" +