Rafael dos Santos Silva a40e2d3156 FEATURE: Update OpenAI tokenizer to GPT-4o and later (#1467)		2025-06-26 15:26:09 -03:00
..
all_mpnet_base_v2_tokenizer.rb	DEV: port directory structure to Zeitwerk (#319)	2023-11-29 15:17:46 +11:00
anthropic_tokenizer.rb	DEV: port directory structure to Zeitwerk (#319)	2023-11-29 15:17:46 +11:00
basic_tokenizer.rb	FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)	2025-06-17 10:25:03 -03:00
bert_tokenizer.rb	DEV: port directory structure to Zeitwerk (#319)	2023-11-29 15:17:46 +11:00
bge_large_en_tokenizer.rb	DEV: port directory structure to Zeitwerk (#319)	2023-11-29 15:17:46 +11:00
bge_m3_tokenizer.rb	FEATURE: Add BGE-M3 embeddings support (#569)	2024-04-10 17:24:01 -03:00
gemini_tokenizer.rb	FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)	2025-06-17 10:25:03 -03:00
llama3_tokenizer.rb	FEATURE: Llama 3 tokenizer (#615)	2024-05-13 12:45:52 -03:00
mixtral_tokenizer.rb	Mixtral (#376)	2023-12-26 14:49:55 -03:00
multilingual_e5_large_tokenizer.rb	DEV: port directory structure to Zeitwerk (#319)	2023-11-29 15:17:46 +11:00
open_ai_tokenizer.rb	FEATURE: Update OpenAI tokenizer to GPT-4o and later (#1467)	2025-06-26 15:26:09 -03:00
qwen_tokenizer.rb	FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)	2025-06-17 10:25:03 -03:00