Document actor reminder partitioning

daprdocs/content/en/developing-applications/building-blocks/actors/actor-reminders-partitions.md

@@ -0,0 +1,96 @@
+---
+type: docs
+title: "How-to: Enable multiple partitions for actor reminders in Dapr"
+linkTitle: "How-To: partition actor reminders"
+weight: 30
+description: Scale actor reminders storage via partitioning
+---
+
+{{% alert title="Preview feature" color="warning" %}}
+Actor reminders partitioning is currently in [preview]({{< ref preview-features.md >}}).
+{{% /alert %}}
+
+## Actor reminders
+An Actor reminder enables scheduled events to be triggered on a given virtual actor. Actor reminders are persisted and continue to be triggered after sidecar restarts. Prior to Dapr runtime version 1.3, reminders were persisted on a single record in the actor state store:
+
+| Key      | Value |
+| ----------- | ----------- |
+| `actors\|\|<actor type>` | `[ <reminder 1>, <reminder 2>, ... , <reminder n> ]` |
+
+Applications that register many reminders can experience the following issues:
+
+* Low throughput on reminders registration and deregistration
+* Limit on total number of reminders registered based on the single record size limit on the state store
+
+Since version 1.3, applications can now enable partitioning of actor reminders in the state store. As data is distributed in multiple keys in the state store. First, there is a metadata record in `actors\|\|<actor type>\|\|metadata` that is used to store persisted configuration for a given actor type. Then, there are multiple records that stores subsets of the reminders for the same actor type.
+
+| Key      | Value |
+| ----------- | ----------- |
+| `actors\|\|<actor type>\|\|metadata` | `{ "id": <actor metadata identifier>, "actorRemindersMetadata": { "partitionCount": <number of partitions for reminders> } }` |
+| `actors\|\|<actor type>\|\|<actor metadata identifier>\|\|reminders\|\|1` | `[ <reminder 1-1>, <reminder 1-2>, ... , <reminder 1-n> ]` |
+| `actors\|\|<actor type>\|\|<actor metadata identifier>\|\|reminders\|\|2` | `[ <reminder 1-1>, <reminder 1-2>, ... , <reminder 1-m> ]` |
+| ... | ... |
+
+If the number of partitions is not enough, it can be changed and Dapr's sidecar will automatically redistribute the reminders's set.
+
+## Enabling actor reminders partitioning
+Actor reminders partitioning is currently in preview, so enabling it is a two step process.
+
+### Preview feature configuration
+Before using reminders partitioning, actor type metadata must be enabled in Dapr. For more information on preview configurations, see [the full guide on opting into preview features in Dapr]({{< ref preview-features.md >}}). Below is an example of the configuration:
+
+```yaml
+apiVersion: dapr.io/v1alpha1
+kind: Configuration
+metadata:
+  name: myconfig
+spec:
+  features:
+    - name: Actor.TypeMetadata
+      enabled: true
+```
+
+### Actor runtime configuration
+Once actor type metadata is enabled as an opt-in preview feature, the actor runtime must also provide the appropriate configuration to partition actor reminders. This is done by the actor's endpoint for `GET /dapr/config`, similar to other actor configuration elements. Here is a snipet of an actor written in Golang providing the configuration:
+
+```go
+type daprConfig struct {
+	Entities                   []string `json:"entities,omitempty"`
+	ActorIdleTimeout           string   `json:"actorIdleTimeout,omitempty"`
+	ActorScanInterval          string   `json:"actorScanInterval,omitempty"`
+	DrainOngoingCallTimeout    string   `json:"drainOngoingCallTimeout,omitempty"`
+	DrainRebalancedActors      bool     `json:"drainRebalancedActors,omitempty"`
+	RemindersStoragePartitions int      `json:"remindersStoragePartitions,omitempty"`
+}
+
+var daprConfigResponse = daprConfig{
+	[]string{defaultActorType},
+	actorIdleTimeout,
+	actorScanInterval,
+	drainOngoingCallTimeout,
+	drainRebalancedActors,
+	7,
+}
+
+func configHandler(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(daprConfigResponse)
+}
+```
+
+The following, is an example of a valid configuration for reminder partitioning:
+
+```json
+{
+	"entities": [ "MyActorType", "AnotherActorType" ],
+	"remindersStoragePartitions": 7
+}
+```
+
+### Handling configuration changes
+For production scenarios, there are some points to be considered before enabling this feature:
+
+* Enabling actor type metadata can only be reverted if the number of partitions remains zero, otherwise the reminders' set will be reverted to an previous state.
+* Number of partitions can only be increased and not decreased. This allows Dapr to automatically redistribute the data on a rolling restart where one or more partition configurations might be active.
+* This feature is in preview, so the Dapr team is collecting feedback on its correctness and performance prior to enabling it by default.

daprdocs/content/en/developing-applications/building-blocks/actors/actors-overview.md

@@ -102,3 +102,6 @@ The following example illustrates the above concepts. Consider an actor type tha
 
 <img src="/images/actors_background_concurrency.png" width=600>
 
+#### Scaling reminders
+If your application expects a high volume of reminders registered or unregistered, consider [actor reminders partitioning]({{<ref actor-reminders-partitions.md>}})
+

daprdocs/content/en/developing-applications/building-blocks/actors/howto-actors.md

@@ -139,6 +139,8 @@ You can configure the Dapr Actors runtime configuration to modify the default ru
 - `drainOngoingCallTimeout` - The duration when in the process of draining rebalanced actors. This specifies the timeout for the current active actor method to finish. If there is no current actor method call, this is ignored. **Default: 60 seconds**
 - `drainRebalancedActors` - If true, Dapr will wait for `drainOngoingCallTimeout` duration to allow a current actor call to complete before trying to deactivate an actor. **Default: true**
 - `reentrancy` (ActorReentrancyConfig) - Configure the reentrancy behavior for an actor. If not provided, reentrancy is diabled. **Default: disabled**
+**Default: 0**
+- `remindersStoragePartitions` - Configure the number of partitions for actor's reminders. If not provided, all reminders are saved as a single record in actor's state store. **Default: 0**
 
 {{< tabs Java Dotnet Python >}}
 
@@ -152,6 +154,7 @@ ActorRuntime.getInstance().getConfig().setActorScanInterval(Duration.ofSeconds(3
 ActorRuntime.getInstance().getConfig().setDrainOngoingCallTimeout(Duration.ofSeconds(60));
 ActorRuntime.getInstance().getConfig().setDrainBalancedActors(true);
 ActorRuntime.getInstance().getConfig().setActorReentrancyConfig(false, null);
+ActorRuntime.getInstance().getConfig().setRemindersStoragePartitions(7);
 ```
 
 See [this example](https://github.com/dapr/java-sdk/blob/master/examples/src/main/java/io/dapr/examples/actors/DemoActorService.java)
@@ -173,6 +176,7 @@ public void ConfigureServices(IServiceCollection services)
         options.ActorScanInterval = TimeSpan.FromSeconds(30);
         options.DrainOngoingCallTimeout = TimeSpan.FromSeconds(60);
         options.DrainRebalancedActors = true;
+        options.RemindersStoragePartitions = 7;
         // reentrancy not implemented in the .NET SDK at this time
     });
 
@@ -194,7 +198,8 @@ ActorRuntime.set_actor_config(
         actor_scan_interval=timedelta(seconds=30),
         drain_ongoing_call_timeout=timedelta(minutes=1),
         drain_rebalanced_actors=True,
-        reentrancy=ActorReentrancyConfig(enabled=False)
+        reentrancy=ActorReentrancyConfig(enabled=False),
+        remindersStoragePartitions=7
     )
 )
 ```

daprdocs/content/en/operations/configuration/preview-features.md

@@ -13,7 +13,8 @@ Currently, preview features are enabled on a per application basis when running
 
 ### Current preview features
 Below is a list of existing preview features:
-- [Actor Reentrancy]({{<ref actor-reentrancy.md>}})
+- [Actor reentrancy]({{<ref actor-reentrancy.md>}})
+- [Actor type metadata with reminders partitioning]({{<ref actor-reminders-partitions.md>}})
 
 ## Configuration properties
 The `features` section under the `Configuration` spec contains the following properties: