Also renames the Mixtral tokenizer to Mistral.
See gem at github.com/discourse/discourse_ai-tokenizers
Co-authored-by: Roman Rizzi <roman@discourse.org>
We have a flag to signal we are shortening the embeddings of a model.
Only used in Open AI's text-embedding-3-*, but we plan to use it for other services.
* Use AR model for embeddings features
* endpoints
* Embeddings CRUD UI
* Add presets. Hide a couple more settings
* system specs
* Seed embedding definition from old settings
* Generate search bit index on the fly. cleanup orphaned data
* support for seeded models
* Fix run test for new embedding
* fix selected model not set correctly
Rafael dos Santos Silva
Roman Rizzi
Kris