diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-message.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-message.md deleted file mode 100644 index 77d1b9a40..000000000 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-message.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -type: docs -title: "How-To: Publish message to a topic" -linkTitle: "How-To: Publish" -weight: 2000 -description: "Send messages to subscribes through topics" ---- - -Pub/Sub is a common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. -Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers. - -Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. -Dapr provides different implementation of the underlying system, and allows operators to bring in their preferred infrastructure, for example Redis Streams, Kafka, etc. - -## Setup the Pub/Sub component - -The first step is to setup the Pub/Sub component. -For this guide, we'll use Redis Streams, which is also installed by default on a local machine when running `dapr init`. - -*Note: When running Dapr locally, a pub/sub component YAML is automatically created for you locally. To override, create a `components` directory containing the file and use the flag `--components-path` with the `dapr run` CLI command.* - -```yaml -apiVersion: dapr.io/v1alpha1 -kind: Component -metadata: - name: pubsub-name - namespace: default -spec: - type: pubsub.redis - metadata: - - name: redisHost - value: localhost:6379 - - name: redisPassword - value: "" - - name: allowedTopics - value: "deathStartStatus" -``` - -Using the `allowedTopics` you can specify that only the `deathStartStatus` topic should be supported. - -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details in the yaml, and run `kubectl apply -f pubsub.yaml`. - -## Publish a topic - -To publish a message to a topic, invoke the following endpoint on a Dapr instance: - -```bash -curl -X POST http://localhost:3500/v1.0/publish/pubsub-name/deathStarStatus \ - -H "Content-Type: application/json" \ - -d '{ - "status": "completed" - }' -``` - -The above example publishes a JSON payload to a `deathStartStatus` topic. -Dapr wraps the user payload in a Cloud Events v1.0 compliant envelope. \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md new file mode 100644 index 000000000..902810309 --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md @@ -0,0 +1,225 @@ +--- +type: docs +title: "How-To: Publish message and subscribe to a topic" +linkTitle: "How-To: Publish & subscribe" +weight: 2000 +description: "Learn how to send messages to a topic with one service and subscribe to that topic in another service" +--- + +## Introduction + +Pub/Sub is a common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. +Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers. + +Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. +Dapr provides different implementation of the underlying system, and allows operators to bring in their preferred infrastructure, for example Redis Streams, Kafka, etc. + +## Step 1: Setup the Pub/Sub component + +The first step is to setup the Pub/Sub component: + +{{< tabs "Self-Hosted (CLI)" Kubernetes >}} + +{{% codetab %}} +Redis Streams is installed by default on a local machine when running `dapr init`. + +Verify by opening your components file under `%UserProfile%\.dapr\components\pubsub.yaml` on Windows or `~/.dapr/components/pubsub.yaml` on Linux/MacOS: +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: pubsub +spec: + type: pubsub.redis + metadata: + - name: redisHost + value: localhost:6379 + - name: redisPassword + value: "" +``` + +You can override this file with another Redis instance or another [pubsub component]({{< ref setup-pubsub >}}) by creating a `components` directory containing the file and using the flag `--components-path` with the `dapr run` CLI command. +{{% /codetab %}} + +{{% codetab %}} +To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of your [desired pubsub component]({{< ref setup-pubsub >}}) in the yaml below, save as `pubsub.yaml`, and run `kubectl apply -f pubsub.yaml`. + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: pubsub + namespace: default +spec: + type: pubsub.redis + metadata: + - name: redisHost + value: localhost:6379 + - name: redisPassword + value: "" +``` +{{% /codetab %}} + +{{< /tabs >}} + +## Step 2: Publish a topic + +To publish a message to a topic, invoke the following endpoint on a Dapr instance: + +{{< tabs "HTTP API (Bash)" "HTTP API (PowerShell)">}} + +{{% codetab %}} +Begin by ensuring a Dapr sidecar is running: +```bash +dapr --app-id myapp --port 3500 run +``` +Then publish a message to the `deathStarStatus` topic: +```bash +curl -X POST http://localhost:3500/v1.0/publish/pubsub/deathStarStatus -H "Content-Type: application/json" -d '{"status": "completed"}' +``` +{{% /codetab %}} + +{{% codetab %}} +Begin by ensuring a Dapr sidecar is running: +```bash +dapr --app-id myapp --port 3500 run +``` +Then publish a message to the `deathStarStatus` topic: +```powershell +Invoke-RestMethod -Method Post -ContentType 'application/json' -Body '{"status": "completed"}' -Uri 'http://localhost:3500/v1.0/publish/pubsub/deathStarStatus' +``` +{{% /codetab %}} + +{{< /tabs >}} + +Dapr automatically wraps the user payload in a Cloud Events v1.0 compliant envelope. + +## Step 3: Subscribe to topics + +Dapr allows two methods by which you can subscribe to topics: +- **Declaratively**, where subscriptions are are defined in an external file. +- **Programatically**, where subscriptions are defined in user code + +### Declarative subscriptions + +You can subscribe to a topic using the following Custom Resources Definition (CRD): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Subscription +metadata: + name: myevent-subscription +spec: + topic: deathStarStatus + route: /dsstatus + pubsubname: pubsub +scopes: +- app1 +- app2 +``` + +The example above shows an event subscription to topic `deathStarStatus`, for the pubsub component `pubsub`. +The `route` field tells Dapr to send all topic messages to the `/dsstatus` endpoint in the app. + +The `scopes` field enables this subscription for apps with IDs `app1` and `app2`. + +Set the component with: +{{< tabs "Self-Hosted (CLI)" Kubernetes>}} + +{{% codetab %}} +Place the CRD in your `./components` directory. When Dapr starts up, it will load subscriptions along with components. + +You can also override the default directory by pointing the Dapr CLI to a components path: + +```bash +dapr run --app-id myapp --components-path ./myComponents -- python3 myapp.py +``` + +*Note: By default, Dapr loads components from `$HOME/.dapr/components` on MacOS/Linux and `%USERPROFILE%\.dapr\components` on Windows. If you place the subscription in a custom components path, make sure the Pub/Sub component is present also.* +{{% /codetab %}} + +{{% codetab %}} +In Kubernetes, save the CRD to a file and apply it to the cluster: + +```bash +kubectl apply -f subscription.yaml +``` +{{% /codetab %}} + +{{< /tabs >}} + +#### Example + +After setting up the subscription above, download this javascript into a `app1.js` file: + +```javascript +const express = require('express') +const bodyParser = require('body-parser') +const app = express() +app.use(bodyParser.json()) + +const port = 3000 + +app.post('/dsstatus', (req, res) => { + res.sendStatus(200); +}); + +app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) +``` +Run this app with: + +```bash +dapr --app-id app1 --app-port 3000 run node app1.js +``` + +### Programmatic subscriptions + +To subscribe to topics, start a web server in the programming language of your choice and listen on the following `GET` endpoint: `/dapr/subscribe`. +The Dapr instance will call into your app at startup and expect a JSON response for the topic subscriptions with: +- `pubsubname`: Which pub/sub component Dapr should use +- `topic`: Which topic to subscribe to +- `route`: Which endpoint for Dapr to call on when a message comes to that topic + +#### Example + +*Note: The following example is written in Node.js, but can be in any programming language* + +```javascript +const express = require('express') +const bodyParser = require('body-parser') +const app = express() +app.use(bodyParser.json()) + +const port = 3000 + +app.get('/dapr/subscribe', (req, res) => { + res.json([ + { + pubsubname: "pubsub", + topic: "deathStarStatus", + route: "dsstatus" + } + ]); +}) + +app.post('/dsstatus', (req, res) => { + res.sendStatus(200); +}); + +app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) +``` + +The `/dsstatus` endpoint matches the `route` defined in the subscriptions and this is where Dapr will send all topic messages to. + +## Step 4: ACK-ing a message + +In order to tell Dapr that a message was processed successfully, return a `200 OK` response. If Dapr receives any other return status code than `200`, or if your app crashes, Dapr will attempt to redeliver the message following At-Least-Once semantics. + +#### Example + +*Note: The following example is written in Node.js, but can be in any programming language* +```javascript +app.post('/dsstatus', (req, res) => { + res.sendStatus(200); +}); +``` \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-subscribe-topic.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-subscribe-topic.md deleted file mode 100644 index 9b30fac13..000000000 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-subscribe-topic.md +++ /dev/null @@ -1,152 +0,0 @@ ---- -type: docs -title: "How-To: Subscribe to a topic" -linkTitle: "How-To: Subscribe" -weight: 3000 -description: "Consume messages from topics" ---- - -Pub/Sub is a very common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. -Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers. - -Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. -Dapr provides different implementation of the underlying system, and allows operators to bring in their preferred infrastructure, for example Redis Streams, Kafka, etc. - -Watch this [video](https://www.youtube.com/watch?v=NLWukkHEwGA&feature=youtu.be&t=1052) on how to consume messages from topics. - -## Setup the Pub Sub component - -The first step is to setup the Pub/Sub component. -For this guide, we'll use Redis Streams, which is also installed by default on a local machine when running `dapr init`. - -*Note: When running Dapr locally, a pub/sub component YAML is automatically created for you locally. To override, create a `components` directory containing the file and use the flag `--components-path` with the `dapr run` CLI command.* - -```yaml -apiVersion: dapr.io/v1alpha1 -kind: Component -metadata: - name: pubsub - namespace: default -spec: - type: pubsub.redis - metadata: - - name: redisHost - value: localhost:6379 - - name: redisPassword - value: "" -``` - -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details in the yaml, and run `kubectl apply -f pubsub.yaml`. - -## Subscribe to topics - -Dapr allows two methods by which you can subscribe to topics: programatically, where subscriptions are defined in user code and declaratively, where subscriptions are are defined in an external file. - -### Declarative subscriptions - -You can subscribe to a topic using the following Custom Resources Definition (CRD): - -```yaml -apiVersion: dapr.io/v1alpha1 -kind: Subscription -metadata: - name: myevent-subscription -spec: - topic: newOrder - route: /orders - pubsubname: kafka -scopes: -- app1 -- app2 -``` - -The example above shows an event subscription to topic `newOrder`, for the pubsub component `kafka`. -The `route` field tells Dapr to send all topic messages to the `/orders` endpoint in the app. - -The `scopes` field enables this subscription for apps with IDs `app1` and `app2`. - -An example of a node.js app that receives events from the subscription: - -```javascript -const express = require('express') -const bodyParser = require('body-parser') -const app = express() -app.use(bodyParser.json()) - -const port = 3000 - -app.post('/orders', (req, res) => { - res.sendStatus(200); -}); - -app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) -``` - -#### Subscribing on Kubernetes - -In Kubernetes, save the CRD to a file and apply it to the cluster: - -``` -kubectl apply -f subscription.yaml -``` - -#### Subscribing in Self Hosted - -When running Dapr in Self-hosted, either locally or on a VM, put the CRD in your `./components` directory. -When Dapr starts up, it will load subscriptions along with components. - -The following example shows how to point the Dapr CLI to a components path: - -``` -dapr run --app-id myapp --components-path ./myComponents -- python3 myapp.py -``` - -*Note: By default, Dapr loads components from $HOME/.dapr/components on MacOS/Linux and %USERPROFILE%\.dapr\components on Windows. If you place the subscription in a custom components path, make sure the Pub/Sub component is present also.* - -### Programmatic subscriptions - -To subscribe to topics, start a web server in the programming language of your choice and listen on the following `GET` endpoint: `/dapr/subscribe`. -The Dapr instance will call into your app, and expect a JSON response for the topic subscriptions. - -*Note: The following example is written in node, but can be in any programming language* - -
-const express = require('express') -const bodyParser = require('body-parser') -const app = express() -app.use(bodyParser.json()) - -const port = 3000 - -app.get('/dapr/subscribe', (req, res) => { - res.json([ - { - pubsubname: "pubsub", - topic: "newOrder", - route: "orders" - } - ]); -}) - -app.post('/orders', (req, res) => { - res.sendStatus(200); -}); - -app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) -- -In the payload returned to Dapr, `topic` tells Dapr which topic to subscribe to, `route` tells Dapr which endpoint to call on when a message comes to that topic, and `pubsubName` tells Dapr which pub/sub component it should use. In this example this is `pubsub` as this is the name of the component we outlined above. - -The `/orders` endpoint matches the `route` defined in the subscriptions and this is where Dapr will send all topic messages to. - -### ACK-ing a message - -In order to tell Dapr that a message was processed successfully, return a `200 OK` response: - -```javascript -res.status(200).send() -``` - -### Schedule a message for redelivery - -If Dapr receives any other return status code than `200`, or if your app crashes, Dapr will attempt to redeliver the message following At-Least-Once semantics. diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index d51b433ed..e165a895a 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -1,52 +1,50 @@ --- type: docs -title: "Pub/Sub overview" -linkTitle: "Pub/Sub overview" +title: "Publish/Subscribe overview" +linkTitle: "Overview" weight: 1000 description: "Overview of the Dapr Pub/Sub building block" --- +## Introduction + The [publish/subscribe pattern](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) allows your microservices to communicate with each other purely by sending messages. In this system, the **producer** of a message sends it to a **topic**, with no knowledge of what service will receive the message. A messages can even be sent if there's no consumer for it. Similarly, a **consumer** will receive messages from a topic without knowledge of what producer sent it. This pattern is especially useful when you need to decouple microservices from one another. Dapr provides a publish/subscribe API that provides at-least-once guarantees and integrates with various message brokers implementations. These implementations are pluggable, and developed outside of the Dapr runtime in [components-contrib](https://github.com/dapr/components-contrib/tree/master/pubsub). -## Publish/Subscribe API +## Features -The API for Publish/Subscribe can be found in the [spec repo](../../reference/api/pubsub_api.md). +### Publish/Subscribe API -## Behavior and guarantees +The API for Publish/Subscribe can be found in the [spec repo]({{< ref pubsub_api.md >}}). + +### At-Least-Once guarantee Dapr guarantees At-Least-Once semantics for message delivery. That means that when an application publishes a message to a topic using the Publish/Subscribe API, it can assume the message is delivered at least once to any subscriber when the response status code from that endpoint is `200`, or returns no error if using the gRPC client. +### Consumer groups and multiple instances + The burden of dealing with concepts like consumer groups and multiple instances inside consumer groups is all catered for by Dapr. -### App ID - -Dapr has the concept of an `id`. This is specified in Kubernetes using the `dapr.io/app-id` annotation and with the `app-id` flag using the Dapr CLI. Dapr requires an ID to be assigned to every application. - When multiple instances of the same application ID subscribe to a topic, Dapr will make sure to deliver the message to only one instance. If two different applications with different IDs subscribe to a topic, at least one instance in each application receives a copy of the same message. -## Cloud events +### Cloud events Dapr follows the [CloudEvents 1.0 Spec](https://github.com/cloudevents/spec/tree/v1.0) and wraps any payload sent to a topic inside a Cloud Events envelope. The following fields from the Cloud Events spec are implemented with Dapr: - -* `id` -* `source` -* `specversion` -* `type` -* `datacontenttype` (Optional) - +- `id` +- `source` +- `specversion` +- `type` +- `datacontenttype` (Optional) > Starting with Dapr v0.9, Dapr no longer wraps published content into CloudEvent if the published payload itself is already in CloudEvent format. The following example shows an XML content in CloudEvent v1.0 serialized as JSON: - - ```json { "specversion" : "1.0", @@ -59,3 +57,12 @@ The following example shows an XML content in CloudEvent v1.0 serialized as JSON "data" : "