diff --git a/blog/config/nav.yml b/blog/config/nav.yml index f06e40f3f..218445664 100644 --- a/blog/config/nav.yml +++ b/blog/config/nav.yml @@ -47,6 +47,7 @@ nav: - releases/announcing-knative-v0-3-release.md - releases/announcing-knative-v0-2-release.md - Articles: + - articles/llm-agents-demo.md - articles/llm-agents-overview.md - articles/cross-namespace-event-links-feature.md - articles/aws_to_func_migration.md diff --git a/blog/docs/articles/llm-agents-demo.md b/blog/docs/articles/llm-agents-demo.md new file mode 100644 index 000000000..588e2e791 --- /dev/null +++ b/blog/docs/articles/llm-agents-demo.md @@ -0,0 +1,147 @@ +# LLM Agents with Knative: A Demo + +**Author: [Calum Murray](https://www.linkedin.com/in/calum-ra-murray/), Software Engineering Intern @ Red Hat** + +In the [previous blog post on LLM Agents](/blog/articles/llm-agents-overview) +we discussed what LLM Agents are, how LLMs call tools, and how Knative can +be used to automate the tool calling process for LLM Agents. In this blog post +we will be looking at a concrete example of how you can build an LLM Agent +application with Knative handling the tool calling and discovery, and explaining +what differentiates this approach from the other common approaches for building +LLM Agents. + +## An example LLM Agent using Knative + +To showcase the power of building LLM Agents with the tool discovery and +calling handled by Knative, we are going to build a simple chat application +that is capable of answering questions about the cost of resources being +used for a fictitious application we are running. Specifically, in our +application we are interested in accurately answering questions such as: + +> How has the trend in the average resource consumption (CPU, memory) +changed over the past few months for my application, and what impact +will this have on my costs? + +To begin, we will be deploying the Knative LLM Tool Provider demo into +a cluster we have access to following the instructions in the README in +[https://github.com/keventmesh/llm-tool-provider](https://github.com/keventmesh/llm-tool-provider). +Once we have deployed the chat app from this repository, we are able to access it by running: + +```sh +kubectl port-forward svc/chat-app-service 8080:8080 +``` + +If you access `http://localhost:8080` in your browser you will now be able +to see the chat application running, and ask it our simple question. When +I tried this while writing the blog post, the interaction was: + +Calum: +> How has the trend in the average resource consumption (CPU, memory) +changed over the past few months for my application, and what impact +will this have on my costs? + +LLM Agent: +> Could you provide me with the data on the average resource consumption +(CPU, GPU, memory) for your application over the past few months? + +If you try this you will likely notice that your LLM agent replies similarly, +and that the UI will indicate in the task list a task called “human”. This is +the LLM calling the one hand-coded tool that we have in our sample app: a tool +that asks the user (the human) for more information before continuing to answer +the question. + +From this interaction, we can see that the LLM isn’t capable of answering our +question yet: it has no information about the resource consumption or about +what the costs associated with that resource consumption are. To give the LLM +these capabilities we need to tell it about the two Knative Functions we +deployed into our cluster in the setup of the sample app: +`average-resource-consumption` and `resource-cost-calculator`. To do this +we will need to apply two EventTypes into our cluster which describe the +contracts expected by these Functions: + +```yaml +apiVersion: eventing.knative.dev/v1beta2 +kind: EventType +metadata: + name: average.resource.consumption +spec: + reference: + apiVersion: serving.knative.dev/v1 + kind: Service + name: average-resource-consumption + description: "This tells you the average resource consumption for recent months" + type: "average.resource.consumption" +--- +apiVersion: eventing.knative.dev/v1beta2 +kind: EventType +metadata: + name: resource.cost.calculator +spec: + reference: + apiVersion: serving.knative.dev/v1 + kind: Service + name: resource-cost-calculator + description: "This calculates the cost for a given resource kind and average usage over a month. Use this tool whenever there is a question about the cost of any cloud resources" + schemaData: '{"resourceKind":{"type":"string","description":"The type of resource you would like to calculate the cost of"},"usage":{"type":"float","description":"The average usage of the resource"},"unit":{"type":"string","description":"The unit of the resource usage"}}' + type: "resource.cost.calculator" +``` + +Both of these EventTypes refer to functions which were deployed in the install +script we ran earlier. One function returns some mock usage data for the past +few months, and the other is able to calculate the cost due to the usage of a +specific resource type for a specific month. + +The LLM Agents we built is programmed to fetch all the EventTypes in the cluster +whenever a new user session starts, and map those into tools it can use. After +applying both EventTypes and refreshing our brower, we can try asking our original +question again: + +Calum: +> How has the trend in the average resource consumption (CPU, memory) changed +over the past few months for my application, and what impact will this have on +my costs? + +LLM Agent: +> The trend in the average resource consumption for your application over the +past few months is as follows: + +> - March: CPU - 21.33 Cores, Memory - 8200 MiB +> - April: CPU - 27.56 Cores, Memory - 8117 MiB +> - May: CPU - 33.17 Cores, Memory - 9217 MiB +> - June: CPU - 34.51 Cores, Memory - 10117 MiB + +> To determine the impact on your costs, I will calculate the cost for CPU and +memory based on the average usage over a month for each resource. Let's proceed +with the cost calculation.The estimated impact on your costs based on the average +resource consumption for your application over the past few months is as follows: + +> - CPU cost: $448.28 +> - Memory cost: $161.87 + +> These costs are calculated based on the average monthly usage of CPU and memory +resources. If you have any more questions or need further assistance, feel free +to ask! + +Looking into the tools window, we can see that the LLM called both of the tools +we provided to it, and the answer is evidently higher quality and more accurate! + +What is truly powerful about what we just did was not that the LLM was able to +use tools to more accurately reply to the user, it was that the LLM was able to +use tools to more accurately reply to the user _without us re-deploying the LLM +application_. We have decoupled the development and deployment of the tools from +the development and deployment of the LLM agent! + +Now, we are able to declaratively bind these two components together through a +glue of EventTypes, but the people building our LLM agent do not need to worry +about what tools it will have access to or how it will call them, and the teams +making tools can give it access to their tools without learning the codebase for +the LLM agent. Furthermore, since all we did was provide metadata about a service +to the LLM we can now create EventTypes to describe existing services we have +already built, and the LLM agent will be able to successfully interact with those +systems with _no code changes_. + +This concept of automatic tool discovery by an LLM Agent through the use of +metadata is the central concept to what we are working to build. In the next +blog post we will discuss our vision for how this will evolve. If you are +interested in helping us build the future of LLM Agents in the cloud, message +us on the [CNCF slack instance](https://slack.cncf.io)!