From 9f86f17c43bf29e09443a1e21983dc6cbdfcec8d Mon Sep 17 00:00:00 2001 From: Ignacio Cano Date: Fri, 10 May 2019 13:45:38 -0700 Subject: [PATCH] Registry documentation (#1292) * Adding bitbucket source documentation. * Adding README example for bitbucket source. * Updating example. * Missed to generate the README * removing stuff, and updating link to bitbucket source in sources.yaml * autogenerating README * removing bitbucket one here. * update * moving design from eventing * update * some updates to Registry * cosmetic * next steps and updates * updated next steps * event registry * updates after review * events * back to types * apostrophe --- docs/eventing/README.md | 12 ++ docs/eventing/event-registry.md | 260 ++++++++++++++++++++++++++++++++ 2 files changed, 272 insertions(+) create mode 100644 docs/eventing/event-registry.md diff --git a/docs/eventing/README.md b/docs/eventing/README.md index 08897db5e..7813e9508 100644 --- a/docs/eventing/README.md +++ b/docs/eventing/README.md @@ -53,6 +53,18 @@ Addressable. You can create as many Triggers as necessary. ![Broker Trigger Diagram](./images/broker-trigger-overview.svg) +### Event registry + +As of v0.6, Knative Eventing defines an EventType object to make it easier for +consumers to discover the types of events they can consume from the different +Brokers. + +The registry consists of a collection of event types. The event types stored in +the registry contain (all) the required information for a consumer to create a +Trigger without resorting to some other out-of-band mechanism. + +To learn how to use the registry, see the [Event Registry documentation](./event-registry.md). + ### Event channels and subscriptions Knative Eventing also defines an event forwarding and persistence layer, called diff --git a/docs/eventing/event-registry.md b/docs/eventing/event-registry.md new file mode 100644 index 000000000..4e5cdbd86 --- /dev/null +++ b/docs/eventing/event-registry.md @@ -0,0 +1,260 @@ +--- +title: "Event registry" +weight: 20 +type: "docs" +--- + +## Overview + +The Event Registry maintains a catalog of the event types that can be consumed from the system. +It introduces a new [EventType](../reference/eventing/eventing.md) CRD in order to persist the event type's information +in the cluster's data store. + +## Before you begin + +1. Read about the [Broker and Trigger objects](./broker-trigger.md). +1. Be familiar with the [CloudEvents spec](https://github.com/cloudevents/spec/blob/master/spec.md), + particularly the [Context Attributes](https://github.com/cloudevents/spec/blob/master/spec.md#context-attributes) + section. +1. Be familiar with the [Sources](./sources/README.md). + +## Discovering events with the registry + +Using the registry, you can discover the different types of events you can consume +from the Brokers' event meshes. The registry is designed for use with the Broker/Trigger model and aims to help +you create Triggers. + +To see the event types available to *subscribe* to, enter the following command: + +`kubectl get eventtypes -n ` + +Below, we show an example output of executing the above command using the `default` namespace in a testing cluster. +We will address the question of how this registry was populated in a later section. + +``` +NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON +dev.knative.source.github.push-34cnb dev.knative.source.github.push https://github.com/knative/eventing default True +dev.knative.source.github.push-44svn dev.knative.source.github.push https://github.com/knative/serving default True +dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request https://github.com/knative/eventing default True +dev.knative.source.github.pullrequest-97shf dev.knative.source.github.pull_request https://github.com/knative/serving default True +dev.knative.kafka.event-cjvcr dev.knative.kafka.event news default True +dev.knative.kafka.event-tdt48 dev.knative.kafka.event knative-demo default True +google.pubsub.topic.publish-hrxhh google.pubsub.topic.publish //pubsub.googleapis.com/knative/topics/testing dev False BrokerIsNotReady +``` + +We can see that there are seven different EventTypes in the registry of the `default` namespace. +Let's pick the first one and see how the EventType yaml looks like: + +`kubectl get eventtype dev.knative.source.github.push-34cnb -o yaml` + +Omitting irrelevant fields: + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + name: dev.knative.source.github.push-34cnb + namespace: default + generateName: dev.knative.source.github.push- +spec: + type: dev.knative.source.github.push + source: https://github.com/knative/eventing + schema: + description: + broker: default +status: + conditions: + - status: "True" + type: BrokerExists + - status: "True" + type: BrokerReady + - status: "True" + type: Ready +``` + +From an Event Consumer standpoint, the fields that matter the most are the `spec` fields as well as the `status`. +The `name` is advisory (i.e., non-authoritative), and we typically generate it (`generateName`) to avoid naming collisions +(e.g., two EventTypes listening to pull requests on two different Github repositories). +As `name` nor `generateName` are needed for consumers to create Triggers, we defer their discussion for later on. + +Regarding `status`, its main purpose it to tell Event Consumers (or Cluster Configurators) whether the EventType is ready +for consumption or not. That *readiness* is based on the `broker` being ready. We can see from the example output that +the PubSub EventType is not ready, as its `dev` Broker isn't. + +Let's talk in more details about the `spec` fields: + +- `type`: is authoritative. This refers to the CloudEvent type as it enters into the eventing mesh. It is mandatory. +Event Consumers can (and in most cases would) create Triggers filtering on this attribute. + +- `source`: Refers to the CloudEvent source as it enters into the eventing mesh. It is mandatory. +Event Consumers can (and in most cases would) create Triggers filtering on this attribute. + +- `schema`: is a valid URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. + +- `description`: is a string describing what the EventType is about. It is optional. + +- `broker` refers to the Broker that can provide the EventType. It is mandatory. + + +## Subscribing to events + +Now that you know what events can be consumed from the Brokers' event meshes, they you can create +Triggers to subscribe to particular events. +Here are a few example Triggers that subscribe to events using exact matching on `type` and/or `source`, +based on the above registry output: + +1. Subscribes to GitHub push requests from any `source`. + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: push-trigger + namespace: default + spec: + broker: default + filter: + sourceAndType: + type: dev.knative.source.github.push + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: push-service + ``` + + As per the registry output above, only two sources exist + for that particular type of event (knative's eventing and serving repositories). + If later on new sources are registered for GitHub pushes, this trigger will be able to consume them. + +1. Subscribes to GitHub pull requests from *knative's eventing* repository. + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: gh-knative-eventing-pull-trigger + namespace: default + spec: + broker: default + filter: + sourceAndType: + type: dev.knative.source.github.pull_request + source: https://github.com/knative/eventing + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: gh-knative-eventing-pull-service + ``` + +1. Subscribes to Kafka messages sent to the *knative-demo* topic + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: kafka-knative-demo-trigger + namespace: default + spec: + broker: default + filter: + sourceAndType: + type: dev.knative.kafka.event + source: knative-demo + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: kafka-knative-demo-service + ``` + +1. Subscribes to PubSub messages from GCP's *knative* project sent to the *testing* topic + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: gcp-pubsub-knative-testing-trigger + namespace: default + spec: + broker: dev + filter: + sourceAndType: + source: //pubsub.googleapis.com/knative/topics/testing + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: gcp-pubsub-knative-testing-service + ``` + + Note that events won't be able to be consumed by this Trigger's subscriber until the Broker becomes ready. + + +## Populating the registry + +Now that we know how to discover events using the registry and how we can leverage that information to subscribe to +events of interest, let's move on to the next topic: How do we actually populate the registry in the first place? + +- Manual Registration + + In order to populate the registry, a Cluster Configurator can manually register the EventTypes. + This means that the configurator can simply apply EventTypes yaml files, just as with any other Kubernetes resource: + + `kubectl apply -f ` + +- Automatic Registration + + As Manual Registration might be tedious and error-prone, we also support automatic registration of EventTypes. + Herein, the creation of the EventTypes is done upon instantiation of an Event Source. + We currently support automatic Registration of EventTypes for the following Event Sources: + + - CronJobSource + - ApiServerSource + - GithubSource + - GcpPubSubSource + - KafkaSource + - AwsSqsSource + + Let's look at an example, in particular, the KafkaSource sample we used to populate the registry in our testing + cluster. Below is what the yaml looks like. + + ```yaml + apiVersion: sources.eventing.knative.dev/v1alpha1 + kind: KafkaSource + metadata: + name: kafka-source-sample + namespace: default + spec: + consumerGroup: knative-group + bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092 + topics: knative-demo,news + sink: + apiVersion: eventing.knative.dev/v1alpha1 + kind: Broker + name: default + ``` + + If you are interested in more information regarding configuration options of a KafkaSource, please refer to the + [KafKaSource example](https://github.com/knative/eventing-sources/tree/master/contrib/kafka/samples). + + For this discussion, the relevant information from the yaml above are the `sink` and the `topics`. + We observe that the `sink` is of kind `Broker`. We currently only support automatic creation of EventTypes for Sources + instances that point to Brokers. + Regarding `topics`, this is what we use for the EventTypes `source` field, which is equal to the CloudEvent source + attribute. + + When you `kubectl apply` this yaml, the KafkaSource `kafka-source-sample` will be instantiated, + and two EventTypes will be added to the registry (as there are two topics). You can see that in the registry + example output from the previous sections. + +## What's next + +We suggest the reader to experiment in her own cluster with the different Event Sources listed above. The following +links might help to get you started. + +1. [Installing Knative](./install/README.md) in case you haven't already done so. +1. [Getting started with eventing](./eventing) in case you haven't read it. +1. [Knative code samples](./samples/) is a useful resource to better understand some of the Event Sources (remember to +point them to a Broker if you want automatic registration of EventTypes in the registry).