discourse-ai/tokenizers
Roman Rizzi 1d786fbaaf
FEATURE: Set endpoint credentials directly from LlmModel. (#625)
* FEATURE: Set endpoint credentials directly from LlmModel.

Drop Llama2Tokenizer since we no longer use it.

* Allow http for custom LLMs

---------

Co-authored-by: Rafael Silva <xfalcox@gmail.com>
2024-05-16 09:50:22 -03:00
..
Apache License
META_LLAMA_3_COMMUNITY_LICENSE.md FEATURE: Llama 3 tokenizer (#615) 2024-05-13 12:45:52 -03:00
MIT License
Meta-Llama-3-70B-Instruct.json FEATURE: Llama 3 tokenizer (#615) 2024-05-13 12:45:52 -03:00
README.md FEATURE: Llama 3 tokenizer (#615) 2024-05-13 12:45:52 -03:00
all-mpnet-base-v2.json
bert-base-uncased.json
bge-large-en.json
bge-m3.json FEATURE: Add BGE-M3 embeddings support (#569) 2024-04-10 17:24:01 -03:00
claude-v1-tokenization.json
mixtral.json
multilingual-e5-large.json

README.md

bert-base-uncased.json

Licensed under Apache License

claude-v1-tokenization.json

Licensed under MIT License

all-mpnet-base-v2.json

Licensed under Apache License

llama-2-70b-chat-hf

Licensed under LLAMA 2 COMMUNITY LICENSE AGREEMENT

multilingual-e5-large

Licensed under MIT License

bge-large-en

Licensed under MIT License

mixtral

Licensed under Apache 2.0 License

bge-m3

Licensed under MIT License

Meta-Llama-3-70B-Instruct

Licensed under META LLAMA 3 COMMUNITY LICENSE