From 3dd8edee51a7d9f0202182a214126e8b9f4117d8 Mon Sep 17 00:00:00 2001 From: joshvanl Date: Tue, 19 Dec 2023 16:40:55 +0000 Subject: [PATCH 1/9] Component Hot Reloading Adds documentation for Component Hot Reloading Part of https://github.com/dapr/dapr/issues/1172 Signed-off-by: joshvanl --- daprdocs/content/en/concepts/components-concept.md | 8 ++++++++ .../en/operations/support/support-preview-features.md | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index b7248d656..986682076 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -52,6 +52,14 @@ For example: For more information read [Pluggable components overview]({{< ref "pluggable-components-overview" >}}) +## Hot Reloading + +With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), Components are able to be "hot reloaded" at runtime. +This means that you can update component configuration without restarting the Dapr runtime. +Component reloading happens when a Component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +When a component is updated, the Component is first closed, and then reinitialized using the new configuration. +There will be a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. + ## Available component types The following are the component types provided by Dapr: diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index 9c9fff690..b62ea901e 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | - +| ** Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self Hosted mode.| N/A | [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From 4822030d6a6fa4f37390a67ecc1d9afceac56fb3 Mon Sep 17 00:00:00 2001 From: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Date: Thu, 4 Jan 2024 15:44:51 -0500 Subject: [PATCH 2/9] Submodules into 1.13 (#3927) * javascript submodule for 1.13 Signed-off-by: Hannah Hunter * go submodules into 1.13 Signed-off-by: Hannah Hunter * java submodules into 1.13 Signed-off-by: Hannah Hunter * dotnet submodules for 1.13 Signed-off-by: Hannah Hunter * python submodule Signed-off-by: Hannah Hunter --------- Signed-off-by: Hannah Hunter Co-authored-by: Mark Fussell --- sdkdocs/dotnet | 2 +- sdkdocs/go | 2 +- sdkdocs/java | 2 +- sdkdocs/js | 2 +- sdkdocs/python | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/sdkdocs/dotnet b/sdkdocs/dotnet index 99d874a2b..10ef81873 160000 --- a/sdkdocs/dotnet +++ b/sdkdocs/dotnet @@ -1 +1 @@ -Subproject commit 99d874a2b138af020df099a0fc0a09a7d0597fae +Subproject commit 10ef81873b3448fb136c73ad26a9fd2768954c2f diff --git a/sdkdocs/go b/sdkdocs/go index e16e0350a..04f7b595b 160000 --- a/sdkdocs/go +++ b/sdkdocs/go @@ -1 +1 @@ -Subproject commit e16e0350a52349b5a05138edc0b58e3be78ee753 +Subproject commit 04f7b595b6d19bbf1c42a3364992016c3ae3e40e diff --git a/sdkdocs/java b/sdkdocs/java index 5e45aa86b..6759f19f8 160000 --- a/sdkdocs/java +++ b/sdkdocs/java @@ -1 +1 @@ -Subproject commit 5e45aa86b81748bf1e6efdbf7f52c20645a12435 +Subproject commit 6759f19f8374c7c550c709b1fe8118ce738280a8 diff --git a/sdkdocs/js b/sdkdocs/js index df7eff281..6e89215f5 160000 --- a/sdkdocs/js +++ b/sdkdocs/js @@ -1 +1 @@ -Subproject commit df7eff281a5a1395a7967c658a5707e8dfb2b99e +Subproject commit 6e89215f5ca26f8f4d109424e2cad7792b9d8a28 diff --git a/sdkdocs/python b/sdkdocs/python index 6171b67db..c08e71494 160000 --- a/sdkdocs/python +++ b/sdkdocs/python @@ -1 +1 @@ -Subproject commit 6171b67db60d51704ed8425ae71dda9226bf1255 +Subproject commit c08e71494a644f9ff875941c669c6a1e1f3a3340 From 1a24364b3484e150ee7e4e47df9d8c4c78c3a920 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Tue, 9 Jan 2024 13:37:19 +0000 Subject: [PATCH 3/9] Apply suggestions from code review Co-authored-by: Mark Fussell Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index 986682076..48365186d 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -54,11 +54,11 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com ## Hot Reloading -With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), Components are able to be "hot reloaded" at runtime. +With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. -Component reloading happens when a Component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. When a component is updated, the Component is first closed, and then reinitialized using the new configuration. -There will be a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. +There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. ## Available component types From 7271911f32d0e83101f55f2e2fb9e4426f83a66f Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Tue, 9 Jan 2024 13:37:50 +0000 Subject: [PATCH 4/9] Update daprdocs/content/en/concepts/components-concept.md Co-authored-by: Mark Fussell Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index 48365186d..db5122e0a 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -57,7 +57,7 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. -When a component is updated, the Component is first closed, and then reinitialized using the new configuration. +When a component is updated, the component is first closed, and then reinitialized using the new configuration. There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. ## Available component types From d00938488cc1d624dcbacd50c7a655cb703f39b2 Mon Sep 17 00:00:00 2001 From: joshvanl Date: Tue, 9 Jan 2024 14:13:55 +0000 Subject: [PATCH 5/9] Update compoents-updates.md with hot reloading feature Signed-off-by: joshvanl --- .../en/operations/components/component-updates.md | 15 ++++++++++++--- .../support/support-preview-features.md | 2 +- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index 583eee28e..eb5c51ef4 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -6,18 +6,27 @@ weight: 300 description: "Updating deployed components used by applications" --- -When making an update to an existing deployed component used by an application, Dapr does not update the component automatically. The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. How this done depends on the hosting environment. +When making an update to an existing deployed component used by an application, Dapr does not update the component automatically unless the `HotReload` feature gate is enabled. +The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. +How this done depends on the hosting environment. + +{{% alert title="Note" color="primary" %}} +Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. +This is enabled by via the [`HotReload` feature gate]({{< ref "support-preview-features.md" >}}). +All component types are supported for hot reloading. +This feature is currently in preview. +{{% /alert %}} ## Kubernetes When running in Kubernetes, the process of updating a component involves two steps: 1. Applying the new component YAML to the desired namespace -2. Performing a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component +2. (Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}})), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component ## Self Hosted -When running in Self Hosted mode, the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. +Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. ## Further reading - [Components concept]({{< ref components-concept.md >}}) diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index b62ea901e..c960c2e5f 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | -| ** Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self Hosted mode.| N/A | [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | +| **Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From afec1f1d58780f9ecc2a996b716a52849d86ecb5 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Thu, 11 Jan 2024 11:52:36 +0000 Subject: [PATCH 6/9] Apply suggestions from code review Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 4 ++-- .../content/en/operations/components/component-updates.md | 8 ++++---- .../en/operations/support/support-preview-features.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index db5122e0a..cb2f9e7ec 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -56,9 +56,9 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. -Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +Component reloading occurs when a component resource is created, updated, or deleted, either in the Kubernetes API or in self-hosted mode when a file is changed in the `resources` directory. When a component is updated, the component is first closed, and then reinitialized using the new configuration. -There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. +The component is unavailable for a short period of time during reload and reinitialization. ## Available component types diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index eb5c51ef4..7f6dd5078 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -8,7 +8,7 @@ description: "Updating deployed components used by applications" When making an update to an existing deployed component used by an application, Dapr does not update the component automatically unless the `HotReload` feature gate is enabled. The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. -How this done depends on the hosting environment. +How this is done depends on the hosting environment. {{% alert title="Note" color="primary" %}} Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. @@ -21,12 +21,12 @@ This feature is currently in preview. When running in Kubernetes, the process of updating a component involves two steps: -1. Applying the new component YAML to the desired namespace -2. (Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}})), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component +1. Apply the new component YAML to the desired namespace +1. Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component ## Self Hosted -Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. +Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping and restarting the `daprd` process to pick up the latest component. ## Further reading - [Components concept]({{< ref components-concept.md >}}) diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index c960c2e5f..cd7fa4991 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | -| **Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | +| **Component Hot Reloading** | Allows for Dapr-loaded components to be "hot reloaded". A component spec is reloaded when it is created/updated/deleted in Kubernetes or on file when running in self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From be597e9b356713dcbbf7f26d668411bbf383fb20 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Thu, 11 Jan 2024 11:52:51 +0000 Subject: [PATCH 7/9] Update daprdocs/content/en/operations/components/component-updates.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/operations/components/component-updates.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index 7f6dd5078..fb8a9313c 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -11,7 +11,7 @@ The Dapr sidecar needs to be restarted in order to pick up the latest version of How this is done depends on the hosting environment. {{% alert title="Note" color="primary" %}} -Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. +Dapr can be made to "hot reload" components, where updates are picked up automatically without needing a restart. This is enabled by via the [`HotReload` feature gate]({{< ref "support-preview-features.md" >}}). All component types are supported for hot reloading. This feature is currently in preview. From ae4e13754b46c86f4aff55419f3fab06b2be2eea Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Fri, 12 Jan 2024 11:25:17 -0800 Subject: [PATCH 8/9] Feature/add-kafka-pubsub-schema-registry (#3946) * Added doc for kafka pubsub Avro schema registry support Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update setup-apache-kafka.md Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Mark Fussell Signed-off-by: Patrick Assuied --------- Signed-off-by: Patrick Assuied Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Co-authored-by: Mark Fussell --- .../supported-bindings/kafka.md | 15 +++ .../supported-pubsub/setup-apache-kafka.md | 112 ++++++++++++++++++ 2 files changed, 127 insertions(+) diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md b/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md index 43f030799..510e73c2d 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md @@ -49,6 +49,16 @@ spec: value: "2.0.0" - name: direction value: "input, output" + - name: schemaRegistryURL # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry URL. + value: http://localhost:8081 + - name: schemaRegistryAPIKey # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry API Key. + value: XYAXXAZ + - name: schemaRegistryAPISecret # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. + value: "ABCDEFGMEADFF" + - name: schemaCachingEnabled # Optional. When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. + value: true + - name: schemaLatestVersionCacheTTL # Optional. When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. + value: 5m ``` ## Spec metadata fields @@ -75,6 +85,11 @@ spec: | `version` | N | Input/Output | Kafka cluster version. Defaults to 2.0.0. Please note that this needs to be mandatorily set to `1.0.0` for EventHubs with Kafka. | `"1.0.0"` | | `direction` | N | Input/Output | The direction of the binding. | `"input"`, `"output"`, `"input, output"` | | `oidcExtensions` | N | Input/Output | String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token | `{"cluster":"kafka","poolid":"kafkapool"}` | +| `schemaRegistryURL` | N | Required when using Schema Registry Avro serialization/deserialization. The Schema Registry URL. | `http://localhost:8081` | +| `schemaRegistryAPIKey` | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Key. | `XYAXXAZ` | +| `schemaRegistryAPISecret` | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. | `ABCDEFGMEADFF` | +| `schemaCachingEnabled` | N | When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. Default is `true` | `true` | +| `schemaLatestVersionCacheTTL` | N | When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. Default is 5 min | `5m` | #### Note The metadata `version` must be set to `1.0.0` when using Azure EventHubs with Kafka. diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 05ff83520..7c39f01af 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -49,6 +49,17 @@ spec: value: 2.0.0 - name: disableTls # Optional. Disable TLS. This is not safe for production!! You should read the `Mutual TLS` section for how to use TLS. value: "true" + - name: schemaRegistryURL # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry URL. + value: http://localhost:8081 + - name: schemaRegistryAPIKey # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry API Key. + value: XYAXXAZ + - name: schemaRegistryAPISecret # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. + value: "ABCDEFGMEADFF" + - name: schemaCachingEnabled # Optional. When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. + value: true + - name: schemaLatestVersionCacheTTL # Optional. When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. + value: 5m + ``` > For details on using `secretKeyRef`, see the guide on [how to reference secrets in components]({{< ref component-secrets.md >}}). @@ -81,6 +92,11 @@ spec: | oidcClientSecret | N | The OAuth2 client secret that has been provisioned in the identity provider: Required when `authType` is set to `oidc` | `"KeFg23!"` | | oidcScopes | N | Comma-delimited list of OAuth2/OIDC scopes to request with the access token. Recommended when `authType` is set to `oidc`. Defaults to `"openid"` | `"openid,kafka-prod"` | | oidcExtensions | N | Input/Output | String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token | `{"cluster":"kafka","poolid":"kafkapool"}` | +| schemaRegistryURL | N | Required when using Schema Registry Avro serialization/deserialization. The Schema Registry URL. | `http://localhost:8081` | +| schemaRegistryAPIKey | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Key. | `XYAXXAZ` | +| schemaRegistryAPISecret | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. | `ABCDEFGMEADFF` | +| schemaCachingEnabled | N | When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. Default is `true` | `true` | +| schemaLatestVersionCacheTTL | N | When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. Default is 5 min | `5m` | The `secretKeyRef` above is referencing a [kubernetes secrets store]({{< ref kubernetes-secret-store.md >}}) to access the tls information. Visit [here]({{< ref setup-secret-store.md >}}) to learn more about how to configure a secret store component. @@ -348,6 +364,102 @@ curl -X POST http://localhost:3500/v1.0/publish/myKafka/myTopic?metadata.correla }' ``` +## Avro Schema Registry serialization/deserialization +You can configure pub/sub to publish or consume data encoded using [Avro binary serialization](https://avro.apache.org/docs/), leveraging an [Apache Schema Registry](https://developer.confluent.io/courses/apache-kafka/schema-registry/) (for example, [Confluent Schema Registry](https://developer.confluent.io/courses/apache-kafka/schema-registry/), [Apicurio](https://www.apicur.io/registry/)). + +### Configuration + +{{% alert title="Important" color="warning" %}} +Currently, only message value serialization/deserialization is supported. Since cloud events are not supported, the `rawPayload=true` metadata must be passed. +{{% /alert %}} + +When configuring the Kafka pub/sub component metadata, you must define: +- The schema registry URL +- The API key/secret, if applicable + +Schema subjects are automatically derived from topic names, using the standard naming convention. For example, for a topic named `my-topic`, the schema subject will be `my-topic-value`. +When interacting with the message payload within the service, it is in JSON format. The payload is transparently serialized/deserialized within the Dapr component. +Date/Datetime fields must be passed as their [Epoch Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) equivalent (rather than typical Iso8601). For example: +- `2024-01-10T04:36:05.986Z` should be passed as `1704861365986` (the number of milliseconds since Jan 1st, 1970) +- `2024-01-10` should be passed as `19732` (the number of days since Jan 1st, 1970) + +### Publishing Avro messages +In order to indicate to the Kafka pub/sub component that the message should be using Avro serialization, the `valueSchemaType` metadata must be set to `Avro`. + +{{< tabs curl "Python SDK">}} + +{{% codetab %}} +```bash +curl -X "POST" http://localhost:3500/v1.0/publish/pubsub/my-topic?metadata.rawPayload=true&metadata.valueSchemaType=Avro -H "Content-Type: application/json" -d '{"order_number": "345", "created_date": 1704861365986}' +``` +{{% /codetab %}} + +{{% codetab %}} +```python +from dapr.clients import DaprClient + +with DaprClient() as d: + req_data = { + 'order_number': '345', + 'created_date': 1704861365986 + } + # Create a typed message with content type and body + resp = d.publish_event( + pubsub_name='pubsub', + topic_name='my-topic', + data=json.dumps(req_data), + publish_metadata={'rawPayload': 'true', 'valueSchemaType': 'Avro'} + ) + # Print the request + print(req_data, flush=True) +``` +{{% /codetab %}} + +{{< /tabs >}} + + +### Subscribing to Avro topics +In order to indicate to the Kafka pub/sub component that the message should be deserialized using Avro, the `valueSchemaType` metadata must be set to `Avro` in the subscription metadata. + +{{< tabs "Python (FastAPI)" >}} + +{{% codetab %}} + +```python +from fastapi import APIRouter, Body, Response, status +import json +import sys + +app = FastAPI() + +router = APIRouter() + + +@router.get('/dapr/subscribe') +def subscribe(): + subscriptions = [{'pubsubname': 'pubsub', + 'topic': 'my-topic', + 'route': 'my_topic_subscriber', + 'metadata': { + 'rawPayload': 'true', + 'valueSchemaType': 'Avro', + } }] + return subscriptions + +@router.post('/my_topic_subscriber') +def my_topic_subscriber(event_data=Body()): + print(event_data, flush=True) + return Response(status_code=status.HTTP_200_OK) +``` + +app.include_router(router) + +{{% /codetab %}} + +{{< /tabs >}} + + + ## Create a Kafka instance {{< tabs "Self-Hosted" "Kubernetes">}} From b0dafe0897e4ae9332475f781de9f25d54f840be Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Fri, 12 Jan 2024 13:03:21 -0800 Subject: [PATCH 9/9] fix formatting (#3950) Signed-off-by: Patrick Assuied --- .../supported-pubsub/setup-apache-kafka.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 7c39f01af..ad1fa1e3b 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -450,10 +450,11 @@ def subscribe(): def my_topic_subscriber(event_data=Body()): print(event_data, flush=True) return Response(status_code=status.HTTP_200_OK) -``` app.include_router(router) +``` + {{% /codetab %}} {{< /tabs >}}