mirror of https://github.com/dapr/docs.git
Flesh out fan-out/fan-in description
Signed-off-by: Chris Gillum <cgillum@microsoft.com>
This commit is contained in:
parent
a83ab1e65f
commit
8aec0476a7
|
|
@ -40,7 +40,7 @@ The Dapr Workflow engine is internally powered by Dapr's actor runtime. The foll
|
||||||
|
|
||||||
Essentially, to use the Dapr Workflow building block, you write workflow code in your application using the Dapr Workflow SDK, which internally connects to the sidecar using a gRPC stream. This will register the workflow and any workflow activities, or tasks that workflows can schedule.
|
Essentially, to use the Dapr Workflow building block, you write workflow code in your application using the Dapr Workflow SDK, which internally connects to the sidecar using a gRPC stream. This will register the workflow and any workflow activities, or tasks that workflows can schedule.
|
||||||
|
|
||||||
Notice that the engine itself is embedded directly into the sidecar and implemented using the `durabletask-go` framework library. This framework allows you to swap out different storage providers, including a storage provider created specifically for Dapr that leverages internal actors behind the scenes. Since Dapr Workflows use actors, you can store workflow state in variety of Dapr-supported state stores, like Redis, CosmosDB, etc.
|
The engine itself is embedded directly into the sidecar and implemented using the [`durabletask-go`](https://github.com/microsoft/durabletask-go) framework library. This framework allows you to swap out different storage providers, including a storage provider created specifically for Dapr that leverages internal actors behind the scenes. Since Dapr Workflows use actors, you can store workflow state in variety of Dapr-supported state stores, like Redis, CosmosDB, etc.
|
||||||
|
|
||||||
For more information about the architecture of Dapr Workflow, see the [workflow architecture]({{< ref workflow-architecture >}}) article.
|
For more information about the architecture of Dapr Workflow, see the [workflow architecture]({{< ref workflow-architecture >}}) article.
|
||||||
|
|
||||||
|
|
@ -70,6 +70,7 @@ In some cases, the steps of the workflow may need to be orchestrated across mult
|
||||||
<img src="/images/workflow-overview/workflows-chaining.png" width=800 alt="Diagram showing how the task chaining workflow pattern works">
|
<img src="/images/workflow-overview/workflows-chaining.png" width=800 alt="Diagram showing how the task chaining workflow pattern works">
|
||||||
|
|
||||||
While the pattern is simple, there are many complexities hidden in the implementation. For example:
|
While the pattern is simple, there are many complexities hidden in the implementation. For example:
|
||||||
|
|
||||||
- What happens if one of the microservices are unavailable for an extended period of time?
|
- What happens if one of the microservices are unavailable for an extended period of time?
|
||||||
- Can failed steps be automatically retried?
|
- Can failed steps be automatically retried?
|
||||||
- If not, how do you facilitate the rollback of previously completed steps, if applicable?
|
- If not, how do you facilitate the rollback of previously completed steps, if applicable?
|
||||||
|
|
@ -111,7 +112,7 @@ catch (TaskFailedException)
|
||||||
{{< /tabs >}}
|
{{< /tabs >}}
|
||||||
|
|
||||||
{{% alert title="Note" color="primary" %}}
|
{{% alert title="Note" color="primary" %}}
|
||||||
In the example above, `"Step1"`, `"Step2"`, `"MyCompensation"`, etc. represent workflow activities, which are essentially other functions in your code that actually implement the steps of the workflow. For brevity, these implementations are left out of this example.
|
In the example above, `"Step1"`, `"Step2"`, `"MyCompensation"`, etc. represent workflow activities, which are functions in your code that actually implement the steps of the workflow. For brevity, these activity implementations are left out of this example.
|
||||||
{{% /alert %}}
|
{{% /alert %}}
|
||||||
|
|
||||||
As you can see, the workflow is expressed as a simple series of statements in the programming language of your choice. This allows any engineer in the organization to quickly understand the end-to-end flow without necessarily needing to understand the end-to-end system architecture.
|
As you can see, the workflow is expressed as a simple series of statements in the programming language of your choice. This allows any engineer in the organization to quickly understand the end-to-end flow without necessarily needing to understand the end-to-end system architecture.
|
||||||
|
|
@ -120,23 +121,59 @@ Behind the scenes, the Dapr Workflow runtime:
|
||||||
|
|
||||||
- Takes care of executing the workflow and ensuring that it runs to completion.
|
- Takes care of executing the workflow and ensuring that it runs to completion.
|
||||||
- Saves progress automatically.
|
- Saves progress automatically.
|
||||||
- Nudges the worfklow app to effectively resume from the last completed step if the workflow process itself fails for any reason.
|
- Automatically resumes the workflow from the last completed step if the workflow process itself fails for any reason.
|
||||||
- Enables error handling to be expressed naturally in your target programming language, allowing you to implement compensation logic easily.
|
- Enables error handling to be expressed naturally in your target programming language, allowing you to implement compensation logic easily.
|
||||||
- Provides built-in retry configuration primitives to simplify the process of configuring complex retry policies for individual steps in the workflow.
|
- Provides built-in retry configuration primitives to simplify the process of configuring complex retry policies for individual steps in the workflow.
|
||||||
|
|
||||||
### Fan out/fan in
|
### Fan-out/fan-in
|
||||||
|
|
||||||
In the fan out/fan in design pattern, you execute multiple tasks simultaneously across multiple workers and wait for them to recombine.
|
In the fan-out/fan-in design pattern, you execute multiple tasks simultaneously across potentially multiple workers, wait for them to finish, and perform some aggregation on the result.
|
||||||
|
|
||||||
The fan out part of the pattern involves distributing the input data to multiple workers, each of which processes a portion of the data in parallel.
|
|
||||||
|
|
||||||
The fan in part of the pattern involves recombining the results from the workers into a single output.
|
|
||||||
|
|
||||||
<img src="/images/workflow-overview/workflows-fanin-fanout.png" width=800 alt="Diagram showing how the fan out/fan in workflow pattern works">
|
<img src="/images/workflow-overview/workflows-fanin-fanout.png" width=800 alt="Diagram showing how the fan out/fan in workflow pattern works">
|
||||||
|
|
||||||
This pattern can be implemented in a variety of ways, such as using message queues, channels, or async/await. The Dapr Workflows extension handles this pattern with relatively simple code:
|
In addition to the challenges mentioned in the previous pattern, there are several important questions to consider when implementing the fan-out/fan-in pattern manually:
|
||||||
|
|
||||||
TODO: CODE EXAMPLE
|
- How do you control the degree of parallelism?
|
||||||
|
- How do you know when to trigger subsequent aggregation steps?
|
||||||
|
- What if the number of parallel steps is dynamic?
|
||||||
|
|
||||||
|
Dapr Workflows provides a way to express the fan-out/fan-in pattern as a simple function, as shown in the following example:
|
||||||
|
|
||||||
|
{{< tabs ".NET" >}}
|
||||||
|
|
||||||
|
{{% codetab %}}
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
// Get a list of N work items to process in parallel.
|
||||||
|
object[] workBatch = await context.CallActivityAsync<object[]>("GetWorkBatch", null);
|
||||||
|
|
||||||
|
// Schedule the parallel tasks, but don't wait for them to complete yet.
|
||||||
|
var parallelTasks = new List<Task<int>>(workBatch.Length);
|
||||||
|
for (int i = 0; i < workBatch.Length; i++)
|
||||||
|
{
|
||||||
|
Task<int> task = context.CallActivityAsync<int>("ProcessWorkItem", workBatch[i]);
|
||||||
|
parallelTasks.Add(task);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Everything is scheduled. Wait here until all parallel tasks have completed.
|
||||||
|
await Task.WhenAll(parallelTasks);
|
||||||
|
|
||||||
|
// Aggregate all N outputs and publish the result.
|
||||||
|
int sum = parallelTasks.Sum(t => t.Result);
|
||||||
|
await context.CallActivityAsync("PostResults", sum);
|
||||||
|
```
|
||||||
|
|
||||||
|
{{% /codetab %}}
|
||||||
|
|
||||||
|
{{< /tabs >}}
|
||||||
|
|
||||||
|
The key takeaways from this example are:
|
||||||
|
|
||||||
|
- The fan-out/fan-in pattern can be expressed as a simple function using ordinary programming constructs
|
||||||
|
- The number of parallel tasks can be static or dynamic
|
||||||
|
- The workflow itself is capable of aggregating the results of parallel executions
|
||||||
|
|
||||||
|
While not shown in the example, it's possible to go further and limit the degree of concurrency using simple, language-specific constructs. Furthermore, the execution of the workflow is durable. If a workflow starts 100 parallel task executions and 40 complete but then the process crashes, the workflow will restart itself automatically and schedule only the remaining 60 tasks.
|
||||||
|
|
||||||
### Async HTTP APIs
|
### Async HTTP APIs
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue