discourse-ai

Commit Graph

Author	SHA1	Message	Date
Rafael dos Santos Silva	3c9513e754	Refinements to embeddings and tokenizers (#61 ) * Refinements to embeddings and tokenizers * lint * Truncate with tokenizers for summary * fix	2023-05-15 15:10:42 -03:00
Sam	e76fc77189	fixes (#53 ) * Minor... use username suggester in case username already exists * FIX: ensure we truncate long prompts Previously we 1. Used raw length instead of token counts for counting length 2. We totally dropped a prompt if it was too long New implementation will truncate "raw" if it gets too long maintaining meaning.	2023-05-06 07:31:53 -03:00
Rafael dos Santos Silva	9783e3b025	FEATURE: Add a basic tokenizer API (#37 ) * FEATURE: Add a basic tokenizer API * Add tests * lint	2023-04-19 11:55:59 -03:00

Author

SHA1

Message

Date

Rafael dos Santos Silva

3c9513e754

Refinements to embeddings and tokenizers (#61 )

* Refinements to embeddings and tokenizers

* lint

* Truncate with tokenizers for summary

* fix

2023-05-15 15:10:42 -03:00

Sam

e76fc77189

fixes (#53 )

* Minor... use username suggester in case username already exists

* FIX: ensure we truncate long prompts

Previously we

1. Used raw length instead of token counts for counting length
2. We totally dropped a prompt if it was too long

New implementation will truncate "raw" if it gets too long maintaining
meaning.

2023-05-06 07:31:53 -03:00

Rafael dos Santos Silva

9783e3b025

FEATURE: Add a basic tokenizer API (#37 )

* FEATURE: Add a basic tokenizer API

* Add tests

* lint

2023-04-19 11:55:59 -03:00

3 Commits