mirror of https://github.com/dapr/docs.git
feat: conversation api toolcalling support documentation (#4769)
* docs: add tool call support in conversation api docs Signed-off-by: Samantha Coyle <sam@diagrid.io> * fix: address feedback Signed-off-by: Samantha Coyle <sam@diagrid.io> --------- Signed-off-by: Samantha Coyle <sam@diagrid.io> Signed-off-by: Marc Duiker <marcduiker@users.noreply.github.com> Co-authored-by: Mark Fussell <markfussell@gmail.com> Co-authored-by: Marc Duiker <marcduiker@users.noreply.github.com>
This commit is contained in:
Sam
GitHub
parent
ef0e7d5e6a
commit
62f930f01f
|
|
@ -14,7 +14,12 @@ Dapr's conversation API reduces the complexity of securely and reliably interact
|
||||||
|
|
||||||
<img src="/images/conversation-overview.png" width=800 alt="Diagram showing the flow of a user's app communicating with Dapr's LLM components.">
|
<img src="/images/conversation-overview.png" width=800 alt="Diagram showing the flow of a user's app communicating with Dapr's LLM components.">
|
||||||
|
|
||||||
In additon to enabling critical performance and security functionality (like [prompt caching]({{% ref "#prompt-caching" %}}) and [PII scrubbing]({{% ref "#personally-identifiable-information-pii-obfuscation" %}})), you can also pair the conversation API with Dapr functionalities, like:
|
In addition to enabling critical performance and security functionality (like [prompt caching]({{% ref "#prompt-caching" %}}) and [PII scrubbing]({{% ref "#personally-identifiable-information-pii-obfuscation" %}})), the conversation API also provides:
|
||||||
|
|
||||||
|
- **Tool calling capabilities** that allow LLMs to interact with external functions and APIs, enabling more sophisticated AI applications
|
||||||
|
- **OpenAI-compatible interface** for seamless integration with existing AI workflows and tools
|
||||||
|
|
||||||
|
You can also pair the conversation API with Dapr functionalities, like:
|
||||||
- Resiliency circuit breakers and retries to circumvent limit and token errors, or
|
- Resiliency circuit breakers and retries to circumvent limit and token errors, or
|
||||||
- Middleware to authenticate requests coming to and from the LLM
|
- Middleware to authenticate requests coming to and from the LLM
|
||||||
|
|
||||||
|
|
@ -45,6 +50,17 @@ The PII scrubber obfuscates the following user information:
|
||||||
- SHA-256 hex
|
- SHA-256 hex
|
||||||
- MD5 hex
|
- MD5 hex
|
||||||
|
|
||||||
|
### Tool calling support
|
||||||
|
|
||||||
|
The conversation API supports advanced tool calling capabilities that allow LLMs to interact with external functions and APIs. This enables you to build sophisticated AI applications that can:
|
||||||
|
|
||||||
|
- Execute custom functions based on user requests
|
||||||
|
- Integrate with external services and databases
|
||||||
|
- Provide dynamic, context-aware responses
|
||||||
|
- Create multi-step workflows and automation
|
||||||
|
|
||||||
|
Tool calling follows [OpenAI's function calling format](https://platform.openai.com/docs/guides/function-calling), making it easy to integrate with existing AI development workflows and tools.
|
||||||
|
|
||||||
## Demo
|
## Demo
|
||||||
|
|
||||||
Watch the demo presented during [Diagrid's Dapr v1.15 celebration](https://www.diagrid.io/videos/dapr-1-15-deep-dive) to see how the conversation API works using the .NET SDK.
|
Watch the demo presented during [Diagrid's Dapr v1.15 celebration](https://www.diagrid.io/videos/dapr-1-15-deep-dive) to see how the conversation API works using the .NET SDK.
|
||||||
|
|
|
||||||
|
|
@ -17,8 +17,7 @@ description: "List of current alpha and beta APIs"
|
||||||
| Cryptography | [Crypto proto](https://github.com/dapr/dapr/blob/5aba3c9aa4ea9b3f388df125f9c66495b43c5c9e/dapr/proto/runtime/v1/dapr.proto#L118) | `v1.0-alpha1/crypto` | The cryptography API enables you to perform **high level** cryptography operations for encrypting and decrypting messages. | [Cryptography API]({{% ref "cryptography-overview.md" %}}) | v1.11 |
|
| Cryptography | [Crypto proto](https://github.com/dapr/dapr/blob/5aba3c9aa4ea9b3f388df125f9c66495b43c5c9e/dapr/proto/runtime/v1/dapr.proto#L118) | `v1.0-alpha1/crypto` | The cryptography API enables you to perform **high level** cryptography operations for encrypting and decrypting messages. | [Cryptography API]({{% ref "cryptography-overview.md" %}}) | v1.11 |
|
||||||
| Jobs | [Jobs proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto#L212-219) | `v1.0-alpha1/jobs` | The jobs API enables you to schedule and orchestrate jobs. | [Jobs API]({{% ref "jobs-overview.md" %}}) | v1.14 |
|
| Jobs | [Jobs proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto#L212-219) | `v1.0-alpha1/jobs` | The jobs API enables you to schedule and orchestrate jobs. | [Jobs API]({{% ref "jobs-overview.md" %}}) | v1.14 |
|
||||||
| Streaming Subscription | [Streaming Subscription proto](https://github.com/dapr/dapr/blob/310c83140b2f0c3cb7d2bef19624df88af3e8e0a/dapr/proto/runtime/v1/dapr.proto#L454) | N/A | Subscription is defined in the application code. Streaming subscriptions are dynamic, meaning they allow for adding or removing subscriptions at runtime. | [Streaming Subscription API]({{% ref "subscription-methods/#streaming-subscriptions" %}}) | v1.14 |
|
| Streaming Subscription | [Streaming Subscription proto](https://github.com/dapr/dapr/blob/310c83140b2f0c3cb7d2bef19624df88af3e8e0a/dapr/proto/runtime/v1/dapr.proto#L454) | N/A | Subscription is defined in the application code. Streaming subscriptions are dynamic, meaning they allow for adding or removing subscriptions at runtime. | [Streaming Subscription API]({{% ref "subscription-methods/#streaming-subscriptions" %}}) | v1.14 |
|
||||||
| Conversation | [Conversation proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto#L221-222) | `v1.0-alpha1/conversation` | Converse between different large language models using the conversation API. | [Conversation API]({{% ref "conversation-overview.md" %}}) | v1.15 |
|
| Conversation | [Conversation proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto#L226) | `v1.0-alpha2/conversation` | Converse between different large language models using the conversation API. | [Conversation API]({{% ref "conversation-overview.md" %}}) | v1.15 |
|
||||||
|
|
||||||
|
|
||||||
## Beta APIs
|
## Beta APIs
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -10,14 +10,16 @@ weight: 1400
|
||||||
The conversation API is currently in [alpha]({{% ref "certification-lifecycle.md#certification-levels" %}}).
|
The conversation API is currently in [alpha]({{% ref "certification-lifecycle.md#certification-levels" %}}).
|
||||||
{{% /alert %}}
|
{{% /alert %}}
|
||||||
|
|
||||||
Dapr provides an API to interact with Large Language Models (LLMs) and enables critical performance and security functionality with features like prompt caching and PII data obfuscation.
|
Dapr provides an API to interact with Large Language Models (LLMs) and enables critical performance and security functionality with features like prompt caching, PII data obfuscation, and tool calling capabilities.
|
||||||
|
|
||||||
|
Tool calling follow's OpenAI's function calling format, making it easy to integrate with existing AI development workflows and tools.
|
||||||
|
|
||||||
## Converse
|
## Converse
|
||||||
|
|
||||||
This endpoint lets you converse with LLMs.
|
This endpoint lets you converse with LLMs using the Alpha2 version of the API, which provides enhanced tool calling support and alignment with OpenAI's interface.
|
||||||
|
|
||||||
```
|
```
|
||||||
POST http://localhost:<daprPort>/v1.0-alpha1/conversation/<llm-name>/converse
|
POST http://localhost:<daprPort>/v1.0-alpha2/conversation/<llm-name>/converse
|
||||||
```
|
```
|
||||||
|
|
||||||
### URL parameters
|
### URL parameters
|
||||||
|
|
@ -30,34 +32,117 @@ POST http://localhost:<daprPort>/v1.0-alpha1/conversation/<llm-name>/converse
|
||||||
|
|
||||||
| Field | Description |
|
| Field | Description |
|
||||||
| --------- | ----------- |
|
| --------- | ----------- |
|
||||||
|
| `name` | The name of the conversation component. Required |
|
||||||
|
| `contextId` | The ID of an existing chat (like in ChatGPT). Optional |
|
||||||
| `inputs` | Inputs for the conversation. Multiple inputs at one time are supported. Required |
|
| `inputs` | Inputs for the conversation. Multiple inputs at one time are supported. Required |
|
||||||
| `cacheTTL` | A time-to-live value for a prompt cache to expire. Uses Golang duration format. Optional |
|
| `parameters` | Parameters for all custom fields. Optional |
|
||||||
| `scrubPII` | A boolean value to enable obfuscation of sensitive information returning from the LLM. Set this value if all PII (across contents) in the request needs to be scrubbed. Optional |
|
|
||||||
| `temperature` | A float value to control the temperature of the model. Used to optimize for consistency and creativity. Optional |
|
|
||||||
| `metadata` | [Metadata](#metadata) passed to conversation components. Optional |
|
| `metadata` | [Metadata](#metadata) passed to conversation components. Optional |
|
||||||
|
| `scrubPii` | A boolean value to enable obfuscation of sensitive information returning from the LLM. Optional |
|
||||||
|
| `temperature` | A float value to control the temperature of the model. Used to optimize for consistency (0) or creativity (1). Optional |
|
||||||
|
| `tools` | Tools register the tools available to be used by the LLM during the conversation. Optional |
|
||||||
|
| `toolChoice` | Controls which (if any) tool is called by the model. Values: `auto`, `required`, or specific tool name. Defaults to `auto` if tools are present. Optional |
|
||||||
|
|
||||||
#### Input body
|
#### Input body
|
||||||
|
|
||||||
| Field | Description |
|
| Field | Description |
|
||||||
| --------- | ----------- |
|
| --------- | ----------- |
|
||||||
| `content` | The message content to send to the LLM. Required |
|
| `messages` | Array of conversation messages. Required |
|
||||||
| `role` | The role for the LLM to assume. Possible values: 'user', 'tool', 'assistant' |
|
| `scrubPii` | A boolean value to enable obfuscation of sensitive information present in the content field. Optional |
|
||||||
| `scrubPII` | A boolean value to enable obfuscation of sensitive information present in the content field. Set this value if PII for this specific content needs to be scrubbed exclusively. Optional |
|
|
||||||
|
|
||||||
### Request content example
|
#### Message types
|
||||||
|
|
||||||
|
The API supports different message types:
|
||||||
|
|
||||||
|
- **`ofDeveloper`**: Developer role messages with optional name and content
|
||||||
|
- **`ofSystem`**: System role messages with optional name and content
|
||||||
|
- **`ofUser`**: User role messages with optional name and content
|
||||||
|
- **`ofAssistant`**: Assistant role messages with optional name, content, and tool calls
|
||||||
|
- **`ofTool`**: Tool role messages with tool ID, name, and content
|
||||||
|
|
||||||
|
#### Tool calling
|
||||||
|
|
||||||
|
Tools can be defined using the `tools` field with function definitions:
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
| --------- | ----------- |
|
||||||
|
| `function.name` | The name of the function to be called. Required |
|
||||||
|
| `function.description` | A description of what the function does. Optional |
|
||||||
|
| `function.parameters` | JSON Schema object describing the function parameters. Optional |
|
||||||
|
|
||||||
|
### Request content examples
|
||||||
|
|
||||||
|
#### Basic conversation
|
||||||
|
|
||||||
```json
|
```json
|
||||||
REQUEST = {
|
REQUEST = {
|
||||||
"inputs": [
|
"name": "openai",
|
||||||
{
|
"inputs": [{
|
||||||
"content": "What is Dapr?",
|
"messages": [{
|
||||||
"role": "user", // Optional
|
"of_user": {
|
||||||
"scrubPII": "true", // Optional. Will obfuscate any sensitive information found in the content field
|
"content": [{
|
||||||
|
"text": "What is Dapr?"
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}],
|
||||||
|
"parameters": {},
|
||||||
|
"metadata": {}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Conversation with tool calling
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "openai",
|
||||||
|
"inputs": [{
|
||||||
|
"messages": [{
|
||||||
|
"of_user": {
|
||||||
|
"content": [{
|
||||||
|
"text": "What is the weather like in San Francisco in celsius?"
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}],
|
||||||
|
"scrub_pii": false
|
||||||
|
}],
|
||||||
|
"parameters": {
|
||||||
|
"max_tokens": {
|
||||||
|
"@type": "type.googleapis.com/google.protobuf.Int64Value",
|
||||||
|
"value": "100"
|
||||||
},
|
},
|
||||||
],
|
"model": {
|
||||||
"cacheTTL": "10m", // Optional
|
"@type": "type.googleapis.com/google.protobuf.StringValue",
|
||||||
"scrubPII": "true", // Optional. Will obfuscate any sensitive information returning from the LLM
|
"value": "claude-3-5-sonnet-20240620"
|
||||||
"temperature": 0.5 // Optional. Optimizes for consistency (0) or creativity (1)
|
}
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"api_key": "test-key",
|
||||||
|
"version": "1.0"
|
||||||
|
},
|
||||||
|
"scrub_pii": false,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"tools": [{
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"description": "Get the current weather for a location",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"location": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The city and state, e.g. San Francisco, CA"
|
||||||
|
},
|
||||||
|
"unit": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": ["celsius", "fahrenheit"],
|
||||||
|
"description": "The temperature unit to use"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": ["location"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}],
|
||||||
|
"tool_choice": "auto"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -71,21 +156,62 @@ Code | Description
|
||||||
|
|
||||||
### Response content
|
### Response content
|
||||||
|
|
||||||
|
#### Basic conversation response
|
||||||
|
|
||||||
```json
|
```json
|
||||||
RESPONSE = {
|
RESPONSE = {
|
||||||
"outputs": {
|
"outputs": [{
|
||||||
{
|
"choices": [{
|
||||||
"result": "Dapr is distribution application runtime ...",
|
"finish_reason": "stop",
|
||||||
"parameters": {},
|
"index": 0,
|
||||||
},
|
"message": {
|
||||||
{
|
"content": "Dapr is a distributed application runtime that makes it easy for developers to build resilient, stateless and stateful applications that run on the cloud and edge.",
|
||||||
"result": "Dapr can help developers ...",
|
"tool_calls": []
|
||||||
"parameters": {},
|
}
|
||||||
}
|
}]
|
||||||
},
|
}]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Tool calling response
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"outputs": [{
|
||||||
|
"choices": [{
|
||||||
|
"finish_reason": "tool_calls",
|
||||||
|
"index": 0,
|
||||||
|
"message": {
|
||||||
|
"content": null,
|
||||||
|
"tool_calls": [{
|
||||||
|
"id": "call_123",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"celsius\"}"
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tool choice options
|
||||||
|
|
||||||
|
The `tool_choice` is an optional parameter that controls how the model can use available tools:
|
||||||
|
|
||||||
|
- **`auto`**: The model can pick between generating a message or calling one or more tools (default when tools are present)
|
||||||
|
- **`required`**: Requires one or more functions to be called
|
||||||
|
- **`{tool_name}`**: Forces the model to call a specific tool by name
|
||||||
|
|
||||||
|
## Legacy Alpha1 API
|
||||||
|
|
||||||
|
The previous Alpha1 version of the API is still supported for backward compatibility but is deprecated. For new implementations, use the Alpha2 version described above.
|
||||||
|
|
||||||
|
```
|
||||||
|
POST http://localhost:<daprPort>/v1.0-alpha1/conversation/<llm-name>/converse
|
||||||
|
```
|
||||||
|
|
||||||
## Next steps
|
## Next steps
|
||||||
|
|
||||||
- [Conversation API overview]({{% ref conversation-overview.md %}})
|
- [Conversation API overview]({{% ref conversation-overview.md %}})
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue