vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,462 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

111 Commits

Author SHA1 Message Date
Michael Goin 7b1895e6ce
[CI Fix] Try fixing eagle e2e test OOM by reducing block allocation (#20213) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-06-29 10:31:37 +08:00
Lu Fang ee1531bc38
[Bugfix][2/n] Fix speculative decoding CI - Fix test_ngram_e2e_greedy_correctness (#19644) 2025-06-14 21:15:41 -07:00
Lu Fang 06be858828
[Bugfix] Fix the speculative decoding test by setting the target dtype (#19633) 2025-06-13 20:57:32 -07:00
Ning Xie 2f1c19b245
[CI] change spell checker from codespell to typos (#18711) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-11 19:57:10 -07:00
Simon Mo 02f0c7b220
[Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
wwl2755 9da1095daf
[Spec Decode][V0] Fix spec decode correctness test in V0 eagle/medusa (#18175) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-05-18 19:49:46 -07:00
Robert Shaw 856865008e
[CI] Disable Failing Tests (#18165) 2025-05-14 13:49:56 -07:00
Yong Hoon Shin 324a3119b0
Fix test_memory_usage_no_spec (#17754) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-05-07 00:10:33 -07:00
Huy Do b74d888c63
Fix more broken speculative decode tests (#17450) 
Signed-off-by: Huy Do <huydhn@gmail.com>
 2025-05-01 06:05:58 -07:00
Noah Yoshida 13cf6b6236
[BugFix] fix speculative decoding memory leak when speculation is disabled (#15506) 
Signed-off-by: Noah Yoshida <noahcy117@gmail.com>
 2025-04-30 23:28:17 -07:00
Huy Do 88fcf00dda
Fix some speculative decode tests with tl.dot (#17371) 
Signed-off-by: Huy Do <huydhn@gmail.com>
 2025-04-29 19:41:02 -07:00
Woosuk Kwon b411418ff0
[Chore] Remove Sampler from Model Code (#17084) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-24 02:49:33 -07:00
shangmingc 239b7befdd
[V1][Spec Decode] Remove deprecated spec decode config params (#15466) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-03-31 09:19:35 -07:00
shangmingc 50c9636d87
[V1][Usage] Refactor speculative decoding configuration and tests (#14434) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-03-22 19:28:10 -10:00
vllmellm 2bb0e1a799
[Bugfix][ROCm] running new process using spawn method for rocm in tests. (#14810) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-03-17 11:33:35 +00:00
Robert Shaw d4d93db2c5
[V1] V1 Enablement Oracle (#13726) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-03-14 22:02:20 -07:00
Harry Mellor cf069aa8aa
Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Lucia Fang f525c0be8b
[Model][Speculative Decoding] DeepSeek MTP spec decode (#12755) 
Signed-off-by: Lu Fang <fanglu@fb.com>
Co-authored-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-02-19 17:06:23 +08:00
shangmingc 46cdd59577
[Feature][Spec Decode] Simplify the use of Eagle Spec Decode (#12304) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-02-16 19:32:26 -08:00
Harry Mellor f2b20fe491
Consolidate Llama model usage in tests (#13094) 2025-02-13 22:18:03 -08:00
Thomas Parnell bb392af434
[Doc] Replace ibm-fms with ibm-ai-platform (#12709) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-02-04 07:05:04 +00:00
Russell Bryant e489ad7a21
[Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Nicolò Lucchesi 6116ca8cd7
[Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs` with ChunkedPrefill (#10132) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: wallashss <wallashss@ibm.com>
Co-authored-by: wallashss <wallashss@ibm.com>
 2025-01-27 13:38:35 -08:00
youkaichao 899136b857
[ci] fix broken distributed-tests-4-gpus (#11937) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-11 09:07:24 +08:00
jeongin601 1bf905ddaa
[Bugfix][SpecDecode] apply sampling parameters to target probabilities for consistency in rejection sampling. (#10198) 
Signed-off-by: jeongin601 <0200angela@gmail.com>
Signed-off-by: jeong_in.bae <jeong_in.bae@navercorp.com>
 2024-11-27 05:07:30 +00:00
Chendi.Xue 0a71900bc9
Remove hard-dependencies of Speculative decode to CUDA workers (#10587) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2024-11-26 17:57:11 -08:00
Murali Andoorveedu db66e018ea
[Bugfix] Fix for Spec model TP + Chunked Prefill (#10232) 
Signed-off-by: andoorve <37849411+andoorve@users.noreply.github.com>
Signed-off-by: Sourashis Roy <sroy@roblox.com>
Co-authored-by: Sourashis Roy <sroy@roblox.com>
 2024-11-26 09:11:16 -08:00
sroy745 f6778620a9
Disable spec-decode + chunked-prefill for draft models with tensor parallelism > 1 (#10136) 
Signed-off-by: Sourashis Roy <sroy@roblox.com>
 2024-11-08 15:56:18 +00:00
Nicolò Lucchesi 9d43afcc53
[Feature] [Spec decode]: Combine chunked prefill with speculative decoding (#9291) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2024-11-07 08:15:14 -08:00
Sungjae Lee 0c63c34f72
[Bugfix][SpecDecode] kv corruption with bonus tokens in spec decode (#9730) 
Co-authored-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2024-11-06 01:45:45 +00:00
youkaichao e893795443
[2/N] executor pass the complete config to worker/modelrunner (#9938) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2024-11-02 07:35:05 -07:00
wangshuai09 4e2d95e372
[Hardware][ROCM] using current_platform.is_rocm (#9642) 
Signed-off-by: wangshuai09 <391746016@qq.com>
 2024-10-28 04:07:00 +00:00
Cody Yu d11bf435a0
[MISC] Consolidate cleanup() and refactor offline_inference_with_prefix.py (#9510) 2024-10-18 14:30:55 -07:00
Kuntai Du 81ede99ca4
[Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
Lily Liu 89feb4c84d
[SpecDec] Remove Batch Expansion (2/3) (#9298) 2024-10-12 05:13:37 +00:00
sroy745 f3a507f1d3
[Core] Add an environment variable which needs to be set explicitly to allow BlockSpaceManagerV1 (#9149) 2024-10-10 14:17:17 +08:00
sroy745 91add85ec4
Fix failing spec decode test (#9054) 2024-10-03 23:07:29 +00:00
Lily Liu 1570203864
[Spec Decode] (1/2) Remove batch expansion (#8839) 2024-10-01 16:04:42 -07:00
Lily Liu bce324487a
[CI][SpecDecode] Fix spec decode tests, use flash attention backend for spec decode CI tests. (#8975) 2024-10-01 00:51:40 +00:00
Travis Johnson 01b6f9e1f0
[Core][Bugfix] Support prompt_logprobs returned with speculative decoding (#8047) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-09-24 17:29:56 -07:00
Lily Liu c6bd70d772
[SpecDec][Misc] Cleanup, remove bonus token logic. (#8701) 2024-09-22 12:34:14 -07:00
Cyrus Leung 0455c46ed4
[Core] Factor out common code in `SequenceData` and `Sequence` (#8675) 2024-09-21 02:30:39 +00:00
Lily Liu 775f00f81e
[Speculative Decoding] Test refactor (#8317) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-09-11 14:07:34 -07:00
Lily Liu e6a26ed037
[SpecDecode][Kernel] Flashinfer Rejection Sampling (#7244) 2024-09-01 21:23:29 -07:00
afeldman-nm 428dd1445e
[Core] Logprobs support in Multi-step (#7652) 2024-08-29 19:19:08 -07:00
Jonas M. Kübler f205c09854
[Bugfix] Unify rank computation across regular decoding and speculative decoding (#7899) 2024-08-28 22:18:13 -07:00
Nick Hill 1856aff4d6
[Spec Decoding] Streamline batch expansion tensor manipulation (#7851) 2024-08-25 15:45:14 -07:00
Travis Johnson cc0eaf12b1
[Bugfix] spec decode handle None entries in topk args in create_sequence_group_output (#7232) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-08-22 09:33:48 -04:00
Abhinav Goyal a3fce56b88
[Speculative Decoding] EAGLE Implementation with Top-1 proposer (#6830) 2024-08-22 02:42:24 -07:00
Abhinav Goyal 312f761232
[Speculative Decoding] Fixing hidden states handling in batch expansion (#7508) 2024-08-19 17:58:14 -07:00