vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,421 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

142 Commits

Author SHA1 Message Date
Maximilien de Bayser 214efc2c3c
Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Austin Veselka 1b886aa104
[Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLight/dse-qwen2-2b-mrl-v1 (#9944) 
Signed-off-by: FurtherAI <austin.veselka@lighton.ai>
Co-authored-by: FurtherAI <austin.veselka@lighton.ai>
 2024-11-13 08:28:13 +00:00
Joe Runde d58268c56a
[V1] Make v1 more testable (#9888) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-11-06 11:57:35 -08:00
Aaron Pham 21063c11c7
[CI/Build] drop support for Python 3.8 EOL (#8464) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2024-11-06 07:11:55 +00:00
Alex Brooks cc98f1e079
[CI/Build] VLM Test Consolidation (#9372) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2024-10-30 09:32:17 -07:00
kakao-kevin-us 6650e6a930
[Model] Add classification Task with Qwen2ForSequenceClassification (#9704) 
Signed-off-by: Kevin-Yang <ykcha9@gmail.com>
Co-authored-by: Kevin-Yang <ykcha9@gmail.com>
 2024-10-26 17:53:35 +00:00
Cyrus Leung c866e0079d
[CI/Build] Fix VLM test failures when using transformers v4.46 (#9666) 2024-10-25 01:40:40 +08:00
Isotr0py 3ff57ebfca
[Model] Initialize Florence-2 language backbone support (#9555) 2024-10-23 10:42:47 +00:00
Cyrus Leung 831540cf04
[Model] Support E5-V (#9576) 2024-10-23 11:35:29 +08:00
wangshuai09 3ddbe25502
[Hardware][CPU] using current_platform.is_cpu (#9536) 2024-10-22 00:50:43 -07:00
Cody Yu d11bf435a0
[MISC] Consolidate cleanup() and refactor offline_inference_with_prefix.py (#9510) 2024-10-18 14:30:55 -07:00
Cyrus Leung 051eaf6db3
[Model] Add user-configurable task for models that support both generation and embedding (#9424) 2024-10-18 11:31:58 -07:00
Cyrus Leung 7abba39ee6
[Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303) 2024-10-16 14:31:00 +08:00
Cyrus Leung 8c6de96ea1
[Model] Explicit interface for vLLM models and support OOT embedding models (#9108) 2024-10-07 06:10:35 +00:00
youkaichao 18b296fdb2
[core] remove beam search from the core (#9105) 2024-10-07 05:47:04 +00:00
Brendan Wong 168cab6bbf
[Frontend] API support for beam search (#9087) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-10-05 23:39:03 -07:00
Xin Yang 15986f598c
[Model] Support Gemma2 embedding model (#9004) 2024-10-05 06:57:05 +00:00
Roger Wang 26aa325f4f
[Core][VLM] Test registration for OOT multimodal models (#8717) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:38:25 -07:00
Cyrus Leung e1a3f5e831
[CI/Build] Update models tests & examples (#8874) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-28 09:54:35 -07:00
Nick Hill 4b377d6feb
[BugFix] Fix test breakages from transformers 4.45 upgrade (#8829) 2024-09-26 16:46:43 -07:00
bnellnm 300da09177
[Kernel] Fullgraph and opcheck tests (#8479) 2024-09-25 08:35:52 -06:00
Travis Johnson 01b6f9e1f0
[Core][Bugfix] Support prompt_logprobs returned with speculative decoding (#8047) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-09-24 17:29:56 -07:00
youkaichao 0250dd68c5
re-implement beam search on top of vllm core (#8726) 
Co-authored-by: Brendan Wong <bjwpokemon@gmail.com>
 2024-09-23 22:08:12 -07:00
afeldman-nm a8c1d161a7
[Core] *Prompt* logprobs support in Multi-step (#8199) 2024-09-18 08:38:43 -07:00
Aaron Pham 9d104b5beb
[CI/Build] Update Ruff version (#8469) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-18 11:00:56 +00:00
Cyrus Leung a84e598e21
[CI/Build] Reorganize models tests (#7820) 2024-09-13 10:20:06 -07:00
Patrick von Platen d31174a4e1
[Hotfix][Pixtral] Fix multiple images bugs (#8415) 2024-09-12 15:21:51 -07:00
Yangshen⚡Deng 6a512a00df
[model] Support for Llava-Next-Video model (#7559) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-10 22:21:36 -07:00
Cyrus Leung 2f707fcb35
[Model] Multi-input support for LLaVA (#8238) 2024-09-07 02:57:24 +00:00
chenqianfzh 4664ceaad6
support bitsandbytes 8-bit and FP4 quantized models (#7445) 2024-08-29 19:09:08 -04:00
zifeitong 5340a2dccf
[Model] Add multi-image input support for LLaVA-Next offline inference (#7230) 2024-08-28 07:09:02 +08:00
Peter Salas 1ca0d4f86b
[Model] Add UltravoxModel and UltravoxConfig (#7615) 2024-08-21 22:49:39 +00:00
Isotr0py aae6927be0
[VLM][Model] Add test for InternViT vision encoder (#7409) 2024-08-20 23:10:20 +08:00
youkaichao 16422ea76f
[misc][plugin] add plugin system implementation (#7426) 2024-08-13 16:24:17 -07:00
Cyrus Leung 7025b11d94
[Bugfix] Fix weight loading for Chameleon when TP>1 (#7410) 2024-08-13 05:33:41 +00:00
Cyrus Leung 7eb4a51c5f
[Core] Support serving encoder/decoder models (#7258) 2024-08-09 10:39:41 +08:00
afeldman-nm fd95e026e0
[Core] Subclass ModelRunner to support cross-attention & encoder sequences (towards eventual encoder/decoder model support) (#4942) 
Co-authored-by: Andrew Feldman <afeld2012@gmail.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-08-06 16:51:47 -04:00
Alphi 7b86e7c9cd
[Model] Add multi-image support for minicpmv (#7122) 
Co-authored-by: hezhihui <hzh7269@modelbest.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-05 09:23:17 +08:00
Michael Goin fb3db61688
[CI/Build] Remove sparseml requirement from testing (#7037) 2024-08-01 12:00:51 -07:00
Alphi 9e169a4c61
[Model] Adding support for MiniCPM-V (#4087) 2024-07-24 20:59:30 -07:00
Cyrus Leung 97234be0ec
[Misc] Manage HTTP connections in one place (#6600) 2024-07-22 21:32:02 -07:00
Cyrus Leung 9042d68362
[Misc] Consolidate and optimize logic for building padded tensors (#6541) 2024-07-20 04:17:24 +00:00
Antoni Baum 7bd82002ae
[Core] Allow specifying custom Executor (#6557) 2024-07-20 01:25:06 +00:00
Cyrus Leung 38ef94888a
[CI/Build] Remove "boardwalk" image asset (#6460) 2024-07-16 08:59:36 -07:00
Cyrus Leung d97011512e
[CI/Build] vLLM cache directory for images (#6444) 2024-07-15 23:12:25 -07:00
Cyrus Leung 9831aec49f
[Core] Dynamic image size support for VLMs (#5276) 
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: ywang96 <ywang@roblox.com>
Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-07-02 20:34:00 -07:00
Robert Shaw 4d26d806e1
Update conftest.py (#6076) 2024-07-02 20:14:22 +00:00
xwjiang2010 98d6682cd1
[VLM] Remove `image_input_type` from VLM config (#5852) 
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-02 07:57:09 +00:00
Alexander Matveev 3476ed0809
[Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602) 2024-07-01 20:10:37 -07:00
youkaichao 2be6955a3f
[ci][distributed] fix device count call 
[ci][distributed] fix some cuda init that makes it necessary to use spawn (#5991)
 2024-06-30 08:06:13 +00:00
Robert Shaw 8dbfcd35bf
[ CI/Build ] Added E2E Test For Compressed Tensors (#5839) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
 2024-06-29 21:12:58 +08:00
Cyrus Leung 6984c02a27
[CI/Build] Refactor image test assets (#5821) 2024-06-26 01:02:34 -07:00
Chang Su c35e4a3dd7
[BugFix] Fix test_phi3v.py (#5725) 2024-06-21 04:45:34 +00:00
Cyrus Leung 3730a1c832
[Misc] Improve conftest (#5681) 2024-06-19 19:09:21 -07:00
Isotr0py daef218b55
[Model] Initialize Phi-3-vision support (#4986) 2024-06-17 19:34:33 -07:00
Antoni Baum 50eed24d25
Add `cuda_device_count_stateless` (#5473) 2024-06-13 16:06:49 -07:00
youkaichao ea3890a5f0
[Core][Distributed] code deduplication in tp&pp with coordinator(#5293) 
[Core][Distributed] add coordinator to reduce code duplication in tp and pp (#5293)
 2024-06-12 17:27:08 -07:00
youkaichao 8ea5e44a43
[CI/Test] improve robustness of test (vllm_runner) (#5357) 
[CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) (#5357)
 2024-06-08 08:59:20 +00:00
youkaichao 9fb900f90c
[CI/Test] improve robustness of test (hf_runner) (#5347) 
[CI/Test] improve robustness of test by replacing del with context manager (hf_runner) (#5347)
 2024-06-07 22:31:32 -07:00
Antoni Baum ccdc490dda
[Core] Change LoRA embedding sharding to support loading methods (#5038) 2024-06-06 19:07:57 -07:00
Cyrus Leung 89c920785f
[CI/Build] Update vision tests (#5307) 2024-06-06 05:17:18 -05:00
Cyrus Leung 9ba093b4f4
[CI/Build] Simplify model loading for `HfRunner` (#5251) 2024-06-04 10:09:19 -07:00
Yuan cafb8e06c5
[CI/BUILD] enable intel queue for longer CPU tests (#4113) 2024-06-03 10:39:50 -07:00
Cyrus Leung 7a64d24aad
[Core] Support image processor (#4197) 2024-06-02 22:56:41 -07:00
Cyrus Leung dfbe60dc62
[Misc] Simplify code and fix type annotations in `conftest.py` (#5118) 2024-06-02 16:05:50 -07:00
Cyrus Leung 5ae5ed1e60
[Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Cyrus Leung e9cdd2b1e2
[CI/Build] Further decouple HuggingFace implementation from ours during tests (#4166) 2024-05-14 23:38:40 -07:00
SangBin Cho e7c46b9527
[Scheduler] Warning upon preemption and Swapping (#4647) 
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-05-13 23:50:44 +09:00
Chang Su e254497b66
[Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) 2024-05-11 11:30:37 -07:00
SangBin Cho f6a593093a
[CI] Make mistral tests pass (#4596) 2024-05-08 08:44:35 -07:00
SangBin Cho 0d62fe58db
[Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens is large & Add tests for preemption (#4451) 2024-05-01 19:24:13 -07:00
Nick Hill e46a60aa4c
[BugFix] Fix handling of stop strings and stop token ids (#3672) 2024-04-11 15:34:12 -07:00
youkaichao 63e7176f26
[Core][Refactor] move parallel_utils into vllm/distributed (#3950) 
[WIP][Core][Refactor] move vllm/model_executor/parallel_utils into vllm/distributed and vllm/device_communicators (#3950)
 2024-04-10 15:33:30 -07:00
Cade Daniel e5043a3e75
[Misc] Add pytest marker to opt-out of global test cleanup (#3863) 2024-04-04 21:54:16 -07:00
Cade Daniel eb69d68804
[Misc] [CI/Build] Speed up block manager CPU-only unit tests ~10x by opting-out of GPU cleanup (#3783) 2024-04-02 00:49:51 +00:00
SangBin Cho 26422e477b
[Test] Make model tests run again and remove --forked from pytest (#3631) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-03-28 21:06:40 -07:00
SangBin Cho b51c1cc9d2
[2/N] Chunked prefill data update (#3538) 2024-03-28 10:06:01 -07:00
xwjiang2010 64172a976c
[Feature] Add vision language model support. (#3042) 2024-03-25 14:16:30 -07:00
SangBin Cho 01bfb22b41
[CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
Antoni Baum fb96c1e98c
Asynchronous tokenization (#2879) 2024-03-15 23:37:01 +00:00
Robert Shaw c0c2335ce0
Integrate Marlin Kernels for Int4 GPTQ inference (#2497) 
Co-authored-by: Robert Shaw <114415538+rib-2@users.noreply.github.com>
Co-authored-by: alexm <alexm@neuralmagic.com>
 2024-03-01 12:47:51 -08:00
Harry Mellor ef978fe411
Port metrics from `aioprometheus` to `prometheus_client` (#2730) 2024-02-25 11:54:00 -08:00
Ronen Schaffer e433c115bc
Fix `vllm:prompt_tokens_total` metric calculation (#2869) 2024-02-18 23:55:41 -08:00
Zhuohan Li a61f0521b8
[Test] Add basic correctness test (#2908) 2024-02-18 16:44:50 -08:00
Zhuohan Li 4aaafdd289
[BUGFIX] Fix the path of test prompts (#2273) 2023-12-26 10:37:21 -08:00
Woosuk Kwon f1c8520146
[BugFix] Fix input positions for long context with sliding window (#2088) 2023-12-13 12:28:13 -08:00
Simon Mo 5ffc0d13a2
Migrate linter from `pylint` to `ruff` (#1665) 2023-11-20 11:58:01 -08:00
Zhuohan Li 9d9072a069
Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Zhuohan Li ba0bfd40e2
TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Antoni Baum c9927c1a6a
Use queue for finished requests (#957) 2023-09-05 19:27:23 -07:00
Zhuohan Li 002800f081
Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Woosuk Kwon 32b6816e55
Add tests for models (#922) 2023-09-01 11:19:43 +09:00