mirror of https://github.com/dapr/docs.git
Adds docs on using Scheduler with Ephemeral storage (#4306)
Signed-off-by: joshvanl <me@joshvanl.dev>
This commit is contained in:
parent
914863f13d
commit
9bb20a0df7
|
|
@ -16,11 +16,6 @@ Jobs in Dapr consist of:
|
||||||
|
|
||||||
[See example scenarios.]({{< ref "#scenarios" >}})
|
[See example scenarios.]({{< ref "#scenarios" >}})
|
||||||
|
|
||||||
{{% alert title="Warning" color="warning" %}}
|
|
||||||
By default, job data is not resilient to [Scheduler]({{< ref scheduler.md >}}) service restarts.
|
|
||||||
A persistent volume must be provided to Scheduler to ensure job data is not lost in either [Kubernetes]({{< ref kubernetes-persisting-scheduler.md >}}) or [Self-hosted]({{< ref self-hosted-persisting-scheduler.md >}}) mode.
|
|
||||||
{{% /alert %}}
|
|
||||||
|
|
||||||
<img src="/images/scheduler/scheduler-architecture.png" alt="Diagram showing the Scheduler control plane service and the jobs API">
|
<img src="/images/scheduler/scheduler-architecture.png" alt="Diagram showing the Scheduler control plane service and the jobs API">
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
|
||||||
|
|
@ -6,12 +6,15 @@ weight: 50000
|
||||||
description: "Configure Scheduler to persist its database to make it resilient to restarts"
|
description: "Configure Scheduler to persist its database to make it resilient to restarts"
|
||||||
---
|
---
|
||||||
|
|
||||||
The [Scheduler]({{< ref scheduler.md >}}) service is responsible for writing jobs to its embedded database and scheduling them for execution.
|
The [Scheduler]({{< ref scheduler.md >}}) service is responsible for writing jobs to its embedded Etcd database and scheduling them for execution.
|
||||||
By default, the Scheduler service database writes this data to an in-memory ephemeral tempfs volume, meaning that **this data is not persisted across restarts**. Job data will be lost during these events.
|
By default, the Scheduler service database writes this data to a Persistent Volume Claim of 1Gb of size using the cluster's default [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/). This means that there is no additional parameter required to run the scheduler service reliably on most Kubernetes deployments, although you will need additional configuration in some deployments or for a production environment.
|
||||||
|
|
||||||
To make the Scheduler data resilient to restarts, a persistent volume must be mounted to the Scheduler `StatefulSet`.
|
## Production Setup
|
||||||
This persistent volume is backed by a real disk that is provided by the hosted Cloud Provider or Kubernetes infrastructure platform.
|
|
||||||
Disk size is determined by how many jobs are expected to be persisted at once; however, 64Gb should be more than sufficient for most use cases.
|
In case your Kubernetes deployment does not have a default storage class or you are configuring a production cluster, defining a storage class is required.
|
||||||
|
|
||||||
|
A persistent volume is backed by a real disk that is provided by the hosted Cloud Provider or Kubernetes infrastructure platform.
|
||||||
|
Disk size is determined by how many jobs are expected to be persisted at once; however, 64Gb should be more than sufficient for most production scenarios.
|
||||||
Some Kubernetes providers recommend using a [CSI driver](https://kubernetes.io/docs/concepts/storage/volumes/#csi) to provision the underlying disks.
|
Some Kubernetes providers recommend using a [CSI driver](https://kubernetes.io/docs/concepts/storage/volumes/#csi) to provision the underlying disks.
|
||||||
Below are a list of useful links to the relevant documentation for creating a persistent disk for the major cloud providers:
|
Below are a list of useful links to the relevant documentation for creating a persistent disk for the major cloud providers:
|
||||||
- [Google Cloud Persistent Disk](https://cloud.google.com/compute/docs/disks)
|
- [Google Cloud Persistent Disk](https://cloud.google.com/compute/docs/disks)
|
||||||
|
|
@ -23,7 +26,7 @@ Below are a list of useful links to the relevant documentation for creating a pe
|
||||||
- [Alibaba Cloud Disk Storage](https://www.alibabacloud.com/help/ack/ack-managed-and-ack-dedicated/user-guide/create-a-pvc)
|
- [Alibaba Cloud Disk Storage](https://www.alibabacloud.com/help/ack/ack-managed-and-ack-dedicated/user-guide/create-a-pvc)
|
||||||
|
|
||||||
|
|
||||||
Once the persistent volume class is available, you can install Dapr using the following command, with Scheduler configured to use the persistent volume class (replace `my-storage-class` with the name of the storage class):
|
Once the storage class is available, you can install Dapr using the following command, with Scheduler configured to use the storage class (replace `my-storage-class` with the name of the storage class):
|
||||||
|
|
||||||
{{% alert title="Note" color="primary" %}}
|
{{% alert title="Note" color="primary" %}}
|
||||||
If Dapr is already installed, the control plane needs to be completely [uninstalled]({{< ref dapr-uninstall.md >}}) in order for the Scheduler `StatefulSet` to be recreated with the new persistent volume.
|
If Dapr is already installed, the control plane needs to be completely [uninstalled]({{< ref dapr-uninstall.md >}}) in order for the Scheduler `StatefulSet` to be recreated with the new persistent volume.
|
||||||
|
|
@ -53,3 +56,37 @@ helm upgrade --install dapr dapr/dapr \
|
||||||
|
|
||||||
{{% /codetab %}}
|
{{% /codetab %}}
|
||||||
{{< /tabs >}}
|
{{< /tabs >}}
|
||||||
|
|
||||||
|
## Ephemeral Storage
|
||||||
|
|
||||||
|
Scheduler can be optionally made to use Ephemeral storage, which is in-memory storage which is **not** resilient to restarts, i.e. all Job data will be lost after a Scheduler restart.
|
||||||
|
This is useful for deployments where storage is not available or required, or for testing purposes.
|
||||||
|
|
||||||
|
{{% alert title="Note" color="primary" %}}
|
||||||
|
If Dapr is already installed, the control plane needs to be completely [uninstalled]({{< ref dapr-uninstall.md >}}) in order for the Scheduler `StatefulSet` to be recreated without the persistent volume.
|
||||||
|
{{% /alert %}}
|
||||||
|
|
||||||
|
{{< tabs "Dapr CLI" "Helm" >}}
|
||||||
|
<!-- Dapr CLI -->
|
||||||
|
{{% codetab %}}
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dapr init -k --set dapr_scheduler.cluster.inMemoryStorage=true
|
||||||
|
```
|
||||||
|
|
||||||
|
{{% /codetab %}}
|
||||||
|
|
||||||
|
<!-- Helm -->
|
||||||
|
{{% codetab %}}
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm upgrade --install dapr dapr/dapr \
|
||||||
|
--version={{% dapr-latest-version short="true" %}} \
|
||||||
|
--namespace dapr-system \
|
||||||
|
--create-namespace \
|
||||||
|
--set dapr_scheduler.cluster.inMemoryStorage=true \
|
||||||
|
--wait
|
||||||
|
```
|
||||||
|
|
||||||
|
{{% /codetab %}}
|
||||||
|
{{< /tabs >}}
|
||||||
|
|
|
||||||
|
|
@ -10,11 +10,6 @@ weight: 1300
|
||||||
The jobs API is currently in alpha.
|
The jobs API is currently in alpha.
|
||||||
{{% /alert %}}
|
{{% /alert %}}
|
||||||
|
|
||||||
{{% alert title="Warning" color="warning" %}}
|
|
||||||
By default, job data is not resilient to [Scheduler]({{< ref scheduler.md >}}) service restarts.
|
|
||||||
A persistent volume must be provided to Scheduler to ensure job data is not lost in either [Kubernetes]({{< ref kubernetes-persisting-scheduler.md >}}) or [Self-Hosted]({{< ref self-hosted-persisting-scheduler.md >}}) mode.
|
|
||||||
{{% /alert %}}
|
|
||||||
|
|
||||||
With the jobs API, you can schedule jobs and tasks in the future.
|
With the jobs API, you can schedule jobs and tasks in the future.
|
||||||
|
|
||||||
> The HTTP APIs are intended for development and testing only. For production scenarios, the use of the SDKs is strongly
|
> The HTTP APIs are intended for development and testing only. For production scenarios, the use of the SDKs is strongly
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue