diff --git a/.gitignore b/.gitignore index 9f11b75..6c10f5b 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,3 @@ .idea/ +model-runner +model-runner.sock \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..cae39b6 --- /dev/null +++ b/Makefile @@ -0,0 +1,30 @@ +# Project variables +APP_NAME := model-runner +GO_VERSION := 1.23.7 + +# Main targets +.PHONY: build run clean help + +# Default target +.DEFAULT_GOAL := help + +# Build the Go application +build: + CGO_ENABLED=0 go build -ldflags="-s -w" -o $(APP_NAME) ./main.go + +# Run the application locally +run: build + ./$(APP_NAME) + +# Clean build artifacts +clean: + rm -f $(APP_NAME) + rm -f model-runner.sock + +# Show help +help: + @echo "Available targets:" + @echo " build - Build the Go application" + @echo " run - Run the application locally" + @echo " clean - Clean build artifacts" + @echo " help - Show this help message" diff --git a/README.md b/README.md index cb7940b..81ebe6f 100644 --- a/README.md +++ b/README.md @@ -14,3 +14,101 @@ with [Model Distribution](https://github.com/docker/model-distribution) and the [Model CLI](https://github.com/docker/model-cli)). It includes a `main.go` that mimics its integration with Docker Desktop and allows the package to be run in a standalone mode. + +## Using the Makefile + +This project includes a Makefile to simplify common development tasks. It requires Docker Desktop >= 4.41.0 +The Makefile provides the following targets: + +- `build` - Build the Go application +- `run` - Run the application locally +- `clean` - Clean build artifacts +- `help` - Show available targets + +### Examples + +```sh +# Build the application +make build + +# Run the application locally +make run + +# Show all available targets +make help +``` + +## API Examples + +The Model Runner exposes a REST API over a Unix socket. You can interact with it using curl commands with the `--unix-socket` option. + +### Listing Models + +To list all available models: + +```sh +curl --unix-socket model-runner.sock localhost/models +``` + +### Creating a Model + +To create a new model: + +```sh +curl --unix-socket model-runner.sock localhost/models/create -X POST -d '{"from": "ai/smollm2"}' +``` + +### Getting Model Information + +To get information about a specific model: + +```sh +curl --unix-socket model-runner.sock localhost/models/ai/smollm2 +``` + +### Chatting with a Model + +To chat with a model, you can send a POST request to the model's chat endpoint: + +```sh +curl --unix-socket model-runner.sock localhost/engines/llama.cpp/v1/chat/completions -X POST -d '{ + "model": "ai/smollm2", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello, how are you?"} + ] +}' +``` + +The response will contain the model's reply: + +```json +{ + "id": "chat-12345", + "object": "chat.completion", + "created": 1682456789, + "model": "ai/smollm2", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "I'm doing well, thank you for asking! How can I assist you today?" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 24, + "completion_tokens": 16, + "total_tokens": 40 + } +} +``` + +### Deleting a model +To delete a model from the server, send a DELETE request to the model's endpoint: + +```sh +curl --unix-socket model-runner.sock localhost/models/ai/smollm2 -X DELETE +```