vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,204 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

36 Commits

Author SHA1 Message Date
Simon Mo 02f0c7b220
[Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Nick Hill 2dbe8c0774
[Perf] API-server scaleout with many-to-many server-engine comms (#17546) 2025-05-30 08:17:00 -07:00
Feng XiaoLong 4fc1bf813a
[Bugfix] Migrate to REGEX Library to prevent catastrophic backtracking (#18454) 
Signed-off-by: Crucifixion-Fxl <xmufxl@gmail.com>
Co-authored-by: Crucifixion-Fxl <xmufxl@gmail.com>
 2025-05-23 16:16:26 -07:00
Aaron Pham 2fc9075b82
[V1] Structured Outputs + Thinking compatibility (#16577) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-05-14 15:45:24 -07:00
Chauncey dc1a821768
[Feature][V1] Support `tool_choice: required` when using Xgrammar as the `StructuredOutputBackend`. (#17845) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-05-12 23:01:31 -07:00
Russell Bryant ebab1ac37c
[CI] Make JSON output tests less likely to fail (#17859) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-05-12 22:31:54 +00:00
Cheng Kuan Yong Jason 08bf784078
[Bugfix] validate grammar and throw 400 error instead of crashing the engine when xgrammar validation fails (#17623) 
Signed-off-by: Jason Cheng <jasoncky96@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-05-12 09:06:10 +08:00
Russell Bryant 6930a41116
[V1] Add VLLM_ALLOW_INSECURE_SERIALIZATION env var (#17490) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-05-08 13:34:02 +08:00
Michael Goin 17b4d85f63
[CI][TPU] Skip structured outputs+spec decode tests on TPU (#17510) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-30 20:36:20 -07:00
Benjamin Chislett 34120f5acd
[V1][Feature] Enable Speculative Decoding with Structured Outputs (#14702) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
 2025-04-30 00:02:10 +00:00
Harry Mellor a6977dbd15
Simplify (and fix) passing of guided decoding backend options (#17008) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-29 19:02:23 +00:00
Michał Moskal 86d9fc29cb
implement Structural Tag with Guidance backend (#17333) 
Signed-off-by: Michal Moskal <michal@moskal.me>
 2025-04-29 02:21:32 +00:00
Russell Bryant f8acd01ff7
[V1] Add `structural_tag` support using xgrammar (#17085) 2025-04-26 14:06:37 +00:00
Sangyeon Cho 6aae216b4e
[Bugfix] remove fallback in guided_json (int range, patterns) (#16725) 
Signed-off-by: csy1204 <josang1204@gmail.com>
Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com>
 2025-04-25 06:54:43 +00:00
Michael Goin 14288d1332
Disable enforce_eager for V1 TPU sampler and structured output tests (#17016) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-24 02:50:09 -07:00
Travis Johnson 3cde34a4a4
[Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (#15949) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-04-23 18:34:41 +00:00
Chauncey acba33a0f1
[Bugfix] Fix the issue where llm.generate cannot be called repeatedly after setting GuidedDecodingParams (#16767) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-04-22 06:02:20 +00:00
leon-seidel e92d7085bf
[Feature][V1] Add xgrammar to support minLength, maxLength with test (#16516) 
Signed-off-by: Leon Seidel <leon.seidel@fau.de>
 2025-04-11 23:22:07 -07:00
Russell Bryant 14e53ed11f
[V1] Fix json_object support with xgrammar (#15488) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-02 02:00:08 -07:00
Roger Wang 7e3f7a4ee7
[CI] Disable flaky structure decoding test temporarily. (#15892) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-04-01 17:42:34 +00:00
Julien Denize 6909a76201
[Bugfix] Fix Mistral guided generation using xgrammar (#15704) 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
 2025-03-29 20:20:19 -07:00
Russell Bryant 7a7992085b
[CI] Speed up V1 structured output tests (#15718) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-28 21:10:45 -07:00
Russell Bryant 7329ff5468
[V1] Support disable_any_whtespace for guidance backend (#15584) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-28 23:46:45 +08:00
Chauncey 3b00ff9138
[Bugfix][v1] xgrammar structured output supports Enum. (#15594) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-03-28 06:14:53 -07:00
Russell Bryant a09ad90a72
[V1] guidance backend for structured output + `auto` fallback mode (#14779) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Loc Huynh <jc1da.3011@gmail.com>
Co-authored-by: Michal Moskal <michal@moskal.me>
 2025-03-24 21:02:33 -07:00
Russell Bryant eb63ea1e18
[V1] Add `disable-any-whitespace` option support for xgrammar (#15316) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-22 15:56:17 +00:00
Aaron Pham c0efdd655b
[Fix][Structured Output] using vocab_size to construct matcher (#14868) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-03-17 11:42:45 -04:00
Robert Shaw aecc780dba
[V1] Enable Entrypoints Tests (#14903) 2025-03-16 17:56:16 -07:00
Russell Bryant 46f98893dd
[V1] Fix model parameterization for structured output tests (#14833) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-14 20:55:18 +00:00
Nick Hill f5d3acd474
[BugFix][V1] Fix parallel sampling finishing/aborts (#14512) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-12 10:29:48 -07:00
Aaron Pham 77a318bd01
[V1][Core] Support MistralTokenizer for Structured Output (#14625) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2025-03-12 10:40:09 +08:00
Russell Bryant 4bf82d4b90
[V1] Add regex structured output support with xgrammar (#14590) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-11 23:03:44 +08:00
Aaron Pham 80e9afb5bc
[V1][Core] Support for Structured Outputs (#12388) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-07 07:19:11 -08:00
Harry Mellor cf069aa8aa
Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
afeldman-nm befc402d34
[V1] V1 engine implements parallel sampling (AsyncLLM and LLMEngine) (#10980) 
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-24 08:29:41 -08:00
afeldman-nm 0630d4537a
[V1] Logprobs and prompt logprobs support (#9880) 
This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.

Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>


Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-07 07:26:20 -08:00