vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,152 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

612 Commits

Author SHA1 Message Date
Seth Kimmel 208cb34c81
[Doc]: Update tensorizer docs to include vllm[tensorizer] (#7889) 
Co-authored-by: Kaunil Dhruv <dhruv.kaunil@gmail.com>
 2024-10-22 15:43:25 -07:00
Yuan 32a1ee74a0
[Hardware][Intel CPU][DOC] Update docs for CPU backend (#6212) 
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Gubrud, Aaron D <aaron.d.gubrud@intel.com>
Co-authored-by: adgubrud <96072084+adgubrud@users.noreply.github.com>
 2024-10-22 10:38:04 -07:00
Isotr0py bb392ea2d2
[Model][VLM] Initialize support for Mono-InternVL model (#9528) 2024-10-22 16:01:46 +00:00
Rafael Vasquez f7db5f0fa9
[Doc] Use shell code-blocks and fix section headers (#9508) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-22 06:43:24 +00:00
youkaichao d621c43df7
[doc] fix format (#9562) 2024-10-21 13:54:57 -07:00
Dhia Eddine Rhaiem f6b97293aa
[Model] FalconMamba Support (#9325) 2024-10-21 12:50:16 -04:00
Michael Goin 3921a2f29e
[Model] Support Pixtral models in the HF Transformers format (#9036) 2024-10-18 13:29:56 -06:00
Cyrus Leung 051eaf6db3
[Model] Add user-configurable task for models that support both generation and embedding (#9424) 2024-10-18 11:31:58 -07:00
tomeras91 d2b1bf55ec
[Frontend][Feature] Add jamba tool parser (#9154) 2024-10-18 10:27:48 +00:00
Kuntai Du 81ede99ca4
[Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
Li, Jiang 5eda21e773
[Hardware][CPU] compressed-tensor INT8 W8A8 AZP support (#9344) 2024-10-17 12:21:04 -04:00
Junhao Li 5b8a1fde84
[Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft (#9396) 2024-10-16 16:40:24 +00:00
Roger Wang 59230ef32b
[Misc] Consolidate example usage of OpenAI client for multimodal models (#9412) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-16 11:20:51 +00:00
Cyrus Leung cee711fdbb
[Core] Rename input data types (#8688) 2024-10-16 10:49:37 +00:00
Cyrus Leung 7abba39ee6
[Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303) 2024-10-16 14:31:00 +08:00
Michael Goin 8e836d982a
[Doc] Fix code formatting in spec_decode.rst (#9348) 2024-10-14 21:29:11 -07:00
Tyler Michael Smith 169b530607
[Bugfix] Clean up some cruft in mamba.py (#9343) 2024-10-15 00:24:25 +00:00
Reza Salehi dfe43a2071
[Model] Molmo vLLM Integration (#9016) 
Co-authored-by: sanghol <sanghol@allenai.org>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-10-14 07:56:24 -07:00
Yunmeng 2b184ddd4f
[Misc][Installation] Improve source installation script and doc (#9309) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-10-12 09:36:40 -07:00
Wallas Henrique 8baf85e4e9
[Doc] Compatibility matrix for mutual exclusive features (#8512) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2024-10-11 11:18:50 -07:00
sixgod 6cf1167c1a
[Model] Add GLM-4v support and meet vllm==0.6.2 (#9242) 2024-10-11 17:36:13 +00:00
Tyler Michael Smith 7342a7d7f8
[Model] Support Mamba (#6484) 2024-10-11 15:40:06 +00:00
Cyrus Leung e808156f30
[Misc] Collect model support info in a single process per model (#9233) 2024-10-11 11:08:11 +00:00
omrishiv f990bab2a4
[Doc][Neuron] add note to neuron documentation about resolving triton issue (#9257) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-10-10 23:36:32 +00:00
Rafael Vasquez 055f3270d4
[Doc] Improve debugging documentation (#9204) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-10 10:48:51 -07:00
whyiug 04de9057ab
[Model] support input image embedding for minicpmv (#9237) 2024-10-10 15:00:47 +00:00
Li, Jiang ca77dd7a44
[Hardware][CPU] Support AWQ for CPU backend (#7515) 2024-10-09 10:28:08 -06:00
Jiangtao Hu dc4aea677a
[Doc] Fix VLM prompt placeholder sample bug (#9170) 2024-10-09 08:59:42 +00:00
Yuan Tang acce7630c1
Update link to KServe deployment guide (#9173) 2024-10-09 03:58:49 +00:00
Michael Goin 9ba0bd6aa6
Add `lm-eval` directly to requirements-test.txt (#9161) 2024-10-08 18:22:31 -07:00
Rafael Vasquez de24046fcd
[Doc] Improve contributing and installation documentation (#9132) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-08 20:22:08 +00:00
Sayak Paul 1874c6a1b0
[Doc] Update vlm.rst to include an example on videos (#9155) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-10-08 18:12:29 +00:00
TimWang 93cf74a8a7
[Doc]: Add deploying_with_k8s guide (#8451) 2024-10-07 13:31:45 -07:00
Cyrus Leung 151ef4efd2
[Model] Support NVLM-D and fix QK Norm in InternViT (#9045) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2024-10-07 11:55:12 +00:00
Cyrus Leung b22b798471
[Model] PP support for embedding models and update docs (#9090) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-10-06 16:35:27 +08:00
Cyrus Leung f22619fe96
[Misc] Remove user-facing error for removed VLM args (#9104) 2024-10-06 01:33:52 -07:00
Andy Dai 5df1834895
[Bugfix] Fix order of arguments matters in config.yaml (#8960) 2024-10-05 17:35:11 +00:00
Roger Wang 26aa325f4f
[Core][VLM] Test registration for OOT multimodal models (#8717) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:38:25 -07:00
Cyrus Leung 0e36fd4909
[Misc] Move registry to its own file (#9064) 2024-10-04 10:01:37 +00:00
Murali Andoorveedu 0f6d7a9a34
[Models] Add remaining model PP support (#7168) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Signed-off-by: Murali Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:56:58 +08:00
代君 3dbb215b38
[Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405) 2024-10-04 10:36:39 +08:00
Nick Hill 18c2e30c57
[Doc] Update Granite model docs (#9025) 2024-10-03 02:42:24 +00:00
Sergey Shlyapnikov f58d4fccc9
[OpenVINO] Enable GPU support for OpenVINO vLLM backend (#8192) 2024-10-02 17:50:01 -04:00
Cyrus Leung 4f341bd4bf
[Doc] Update list of supported models (#8987) 2024-10-02 00:35:39 +08:00
whyiug e01ab595d8
[Model] support input embeddings for qwen2vl (#8856) 2024-09-30 03:16:10 +00:00
youkaichao cc276443b5
[doc] organize installation doc and expose per-commit docker (#8931) 2024-09-28 17:48:41 -07:00
youkaichao d86f6b2afb
[misc] fix wheel name (#8919) 2024-09-27 22:10:44 -07:00
Cyrus Leung 3b00b9c26c
[Core] rename`PromptInputs` and `inputs` (#8876) 2024-09-26 20:35:15 -07:00
Maximilien de Bayser 344cd2b6f4
[Feature] Add support for Llama 3.1 and 3.2 tool use (#8343) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-09-26 17:01:42 -07:00
youkaichao 70de39f6b4
[misc][installation] build from source without compilation (#8818) 2024-09-26 13:19:04 -07:00
Roger Wang 4bb98f2190
[Misc] Update config loading for Qwen2-VL and remove Granite (#8837) 2024-09-26 07:45:30 -07:00
Roger Wang e2c6e0a829
[Doc] Update doc for Transformers 4.45 (#8817) 2024-09-25 13:29:48 -07:00
Chen Zhang 770ec6024f
[Model] Add support for the multi-modal Llama 3.2 model (#8811) 
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Chang Su <chang.s.su@oracle.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-25 13:29:32 -07:00
Simon Mo 4f1ba0844b
Revert "rename PromptInputs and inputs with backward compatibility (#8760) (#8810) 2024-09-25 10:36:26 -07:00
Cyrus Leung 28e1299e60
rename PromptInputs and inputs with backward compatibility (#8760) 2024-09-25 09:36:47 -07:00
Hongxia Yang 1c046447a6
[CI/Build][Bugfix][Doc][ROCm] CI fix and doc update after ROCm 6.2 upgrade (#8777) 2024-09-25 22:26:37 +08:00
Jee Jee Li 13f9f7a3d0
[[Misc]Upgrade bitsandbytes to the latest version 0.44.0 (#8768) 2024-09-24 17:08:55 -07:00
Simon Mo 3185fb0cca
Revert "[Core] Rename `PromptInputs` to `PromptType`, and `inputs` to `prompt`" (#8750) 2024-09-24 05:45:20 +00:00
Hongxia Yang 530821d00c
[Hardware][AMD] ROCm6.2 upgrade (#8674) 2024-09-23 18:52:39 -07:00
Daniele ee5f34b1c2
[CI/Build] use setuptools-scm to set __version__ (#4738) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-09-23 09:44:26 -07:00
Yan Ma d23679eb99
[Bugfix] fix docker build for xpu (#8652) 2024-09-22 22:54:18 -07:00
youkaichao d4a2ac8302
[build] enable existing pytorch (for GH200, aarch64, nightly) (#8713) 2024-09-22 12:47:54 -07:00
litianjian 5b59532760
[Model][VLM] Add LLaVA-Onevision model support (#8486) 
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-22 10:51:44 -07:00
Andy Dai 4dfdf43196
[Doc] Fix typo in AMD installation guide (#8689) 2024-09-21 00:24:12 -07:00
Cyrus Leung 0057894ef7
[Core] Rename `PromptInputs` and `inputs`(#8673) 2024-09-20 19:00:54 -07:00
omrishiv 7c8566aa4f
[Doc] neuron documentation update (#8671) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-09-20 15:04:37 -07:00
Niklas Muennighoff 3b63de9353
[Model] Add OLMoE (#7922) 2024-09-20 09:31:41 -07:00
Jiaxin Shan 260d40b5ea
[Core] Support Lora lineage and base model metadata management (#6315) 2024-09-20 06:20:56 +00:00
Isotr0py ea4647b7d7
[Doc] Add documentation for GGUF quantization (#8618) 2024-09-19 13:15:55 -06:00
Geun, Lim e18749ff09
[Model] Support Solar Model (#8386) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-09-18 11:04:00 -06:00
Alexander Matveev 7c7714d856
[Core][Bugfix][Perf] Introduce `MQLLMEngine` to avoid `asyncio` OH (#8157) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-09-18 13:56:58 +00:00
youkaichao fa0c114fad
[doc] improve installation doc (#8550) 
Co-authored-by: Andy Dai <76841985+Imss27@users.noreply.github.com>
 2024-09-17 16:24:06 -07:00
youkaichao 2759a43a26
[doc] update doc on testing and debugging (#8514) 2024-09-16 12:10:23 -07:00
ywfang 8a0cf1ddc3
[Model] support minicpm3 (#8297) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-14 14:50:26 +00:00
Isotr0py f57092c00b
[Doc] Add oneDNN installation to CPU backend documentation (#8467) 2024-09-13 18:06:30 +00:00
Cyrus Leung a84e598e21
[CI/Build] Reorganize models tests (#7820) 2024-09-13 10:20:06 -07:00
youkaichao cab69a15e4
[doc] recommend pip instead of conda (#8446) 2024-09-12 23:52:41 -07:00
Alex Brooks c6202daeed
[Model] Support multiple images for qwen-vl (#8247) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-12 10:10:54 -07:00
Patrick von Platen d394787e52
Pixtral (#8377) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-11 14:41:55 -07:00
Yang Fan 3b7fea770f
[Model][VLM] Add Qwen2-VL model support (#7905) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-11 09:31:19 -07:00
Yangshen⚡Deng 6a512a00df
[model] Support for Llava-Next-Video model (#7559) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-10 22:21:36 -07:00
Simon Mo a1d874224d
Add NVIDIA Meetup slides, announce AMD meetup, and add contact info (#8319) 2024-09-09 23:21:00 -07:00
Isotr0py e807125936
[Model][VLM] Support multi-images inputs for InternVL2 models (#8201) 2024-09-07 16:38:23 +08:00
Cyrus Leung 2f707fcb35
[Model] Multi-input support for LLaVA (#8238) 2024-09-07 02:57:24 +00:00
William Lin 12dd715807
[misc] [doc] [frontend] LLM torch profiler support (#7943) 2024-09-06 17:48:48 -07:00
Dipika Sikka 23f322297f
[Misc] Remove `SqueezeLLM` (#8220) 2024-09-06 16:29:03 -06:00
Jiaxin Shan db3bf7c991
[Core] Support load and unload LoRA in api server (#6566) 
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-09-05 18:10:33 -07:00
sroy745 2febcf2777
[Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (#7962) 2024-09-05 16:25:29 -04:00
Alex Brooks 9da25a88aa
[MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (#8029) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-05 12:48:10 +00:00
Cyrus Leung 288a938872
[Doc] Indicate more information about supported modalities (#8181) 2024-09-05 10:51:53 +00:00
Kyle Mistele e02ce498be
[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
Woosuk Kwon 61f4a93d14
[TPU][Bugfix] Use XLA rank for persistent cache path (#8137) 2024-09-03 18:35:33 -07:00
Wenxiang 1248e8506a
[Model] Adding support for MSFT Phi-3.5-MoE (#7729) 
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Zeqi Lin <zelin@microsoft.com>
Co-authored-by: Zeqi Lin <Zeqi.Lin@microsoft.com>
 2024-08-30 13:42:57 -06:00
Kaunil Dhruv 058344f89a
[Frontend]-config-cli-args (#7737) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Kaunil Dhruv <kaunil_dhruv@intuit.com>
 2024-08-30 08:21:02 -07:00
Yohan Na dc13e99348
[MODEL] add Exaone model support (#7819) 2024-08-29 23:34:20 -07:00
Stas Bekman 8c56e57def
[Doc] fix 404 link (#7966) 2024-08-28 13:54:23 -07:00
Woosuk Kwon eeffde1ac0
[TPU] Upgrade PyTorch XLA nightly (#7967) 2024-08-28 13:10:21 -07:00
Stas Bekman 98c12cffe5
[Doc] fix the autoAWQ example (#7937) 2024-08-28 12:12:32 +00:00
Peter Salas fab5f53e2d
[Core][VLM] Stack multimodal tensors to represent multiple images within each prompt (#7902) 2024-08-28 01:53:56 +00:00
Peter Salas 57792ed469
[Doc] Fix incorrect docs from #7615 (#7788) 2024-08-22 10:02:06 -07:00
zifeitong df1a21131d
[Model] Fix Phi-3.5-vision-instruct 'num_crops' issue (#7710) 2024-08-22 09:36:24 +08:00
Peter Salas 1ca0d4f86b
[Model] Add UltravoxModel and UltravoxConfig (#7615) 2024-08-21 22:49:39 +00:00
William Lin dd53c4b023
[misc] Add Torch profiler support (#7451) 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-21 15:39:26 -07:00
Roger Wang 4506641212
[Doc] Section for Multimodal Language Models (#7719) 2024-08-20 23:24:01 -07:00
Ilya Lavrenov 398521ad19
[OpenVINO] Updated documentation (#7687) 2024-08-20 07:33:56 -06:00
Michael Goin d4f0f17b02
[Doc] Update quantization supported hardware table (#7595) 2024-08-16 13:59:27 -07:00
Michael Goin b3f4e17935
[Doc] Add docs for llmcompressor INT8 and FP8 checkpoints (#7444) 2024-08-16 13:59:16 -07:00
Kameshwara Pavan Kumar Mantha 22b39e11f2
llama_index serving integration documentation (#6973) 
Co-authored-by: pavanmantha <pavan.mantha@thevaslabs.io>
 2024-08-14 15:38:37 -07:00
Cyrus Leung 3f674a49b5
[VLM][Core] Support profiling with multiple multi-modal inputs per prompt (#7126) 2024-08-14 17:55:42 +00:00
youkaichao 199adbb7cf
[doc] update test script to include cudagraph (#7501) 2024-08-13 21:52:58 -07:00
Cyrus Leung dd164d72f3
[Bugfix][Docs] Update list of mock imports (#7493) 2024-08-13 20:37:30 -07:00
Woosuk Kwon a08df8322e
[TPU] Support multi-host inference (#7457) 2024-08-13 16:31:20 -07:00
Peter Salas 00c3d68e45
[Frontend][Core] Add plumbing to support audio language models (#7446) 2024-08-13 17:39:33 +00:00
Woosuk Kwon e20233d361
Revert "[Doc] Update supported_hardware.rst (#7276)" (#7467) 2024-08-13 01:37:08 -07:00
jon-chuang a046f86397
[Core/Bugfix] Add FP8 K/V Scale and dtype conversion for prefix/prefill Triton Kernel (#7208) 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-12 22:47:41 +00:00
Roger Wang e6e42e4b17
[Core][VLM] Support image embeddings as input (#6613) 2024-08-12 16:16:06 +08:00
Simon Mo f020a6297e
[Docs] Update readme (#7316) 2024-08-11 17:13:37 -07:00
tomeras91 02b1988b9f
[Doc] building vLLM with VLLM_TARGET_DEVICE=empty (#7403) 2024-08-11 14:38:17 -07:00
Woosuk Kwon 90bab18f24
[TPU] Use mark_dynamic to reduce compilation time (#7340) 2024-08-10 18:12:22 -07:00
Simon Mo 5923532e15
Add Skywork AI as Sponsor (#7314) 2024-08-08 13:59:57 -07:00
Jee Jee Li 757ac70a64
[Model] Rename MiniCPMVQwen2 to MiniCPMV2.6 (#7273) 2024-08-08 14:02:41 +00:00
Michael Goin 6d94420246
[Doc] Update supported_hardware.rst (#7276) 2024-08-07 14:21:50 -07:00
Stas Bekman 0e12cd67a8
[Doc] add online speculative decoding example (#7243) 2024-08-07 09:58:02 -07:00
Ilya Lavrenov 80cbe10c59
[OpenVINO] migrate to latest dependencies versions (#7251) 2024-08-07 09:49:10 -07:00
Roger Wang 2385c8f374
[Doc] Mock new dependencies for documentation (#7245) 2024-08-07 06:43:03 +00:00
Thomas Parnell 789937af2e
[Doc] [SpecDecode] Update MLPSpeculator documentation (#7100) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-08-05 23:29:43 +00:00
Simon Mo 4db5176d97
bump version to v0.5.4 (#7139) 2024-08-05 14:39:48 -07:00
Jee Jee Li 179a6a36f2
[Model]Refactor MiniCPMV (#7020) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 08:12:41 +00:00
Yihuan Bu 654bc5ca49
Support for guided decoding for offline LLM (#6878) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 03:12:09 +00:00
Michael Goin b482b9a5b1
[CI/Build] Add support for Python 3.12 (#7035) 2024-08-02 13:51:22 -07:00
Murali Andoorveedu fc912e0886
[Models] Support Qwen model with PP (#6974) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
 2024-08-01 12:40:43 -07:00
Jee Jee Li 7ecee34321
[Kernel][RFC] Refactor the punica kernel based on Triton (#5036) 2024-07-31 17:12:24 -07:00
Alphi 2f4e108f75
[Bugfix] Clean up MiniCPM-V (#6939) 
Co-authored-by: hezhihui <hzh7269@modelbest.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-07-31 14:39:19 +00:00
Cyrus Leung f230cc2ca6
[Bugfix] Fix broadcasting logic for `multi_modal_kwargs` (#6836) 2024-07-31 10:38:45 +08:00
Ilya Lavrenov 5895b24677
[OpenVINO] Updated OpenVINO requirements and build docs (#6948) 2024-07-30 11:33:01 -07:00
Isotr0py 7cbd9ec7a9
[Model] Initialize support for InternVL2 series models (#6514) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-29 10:16:30 +00:00
Woosuk Kwon fad5576c58
[TPU] Reduce compilation time & Upgrade PyTorch XLA version (#6856) 2024-07-27 10:28:33 -07:00
Chenggang Wu f954d0715c
[Docs] Add RunLLM chat widget (#6857) 2024-07-27 09:24:46 -07:00
Cyrus Leung 1ad86acf17
[Model] Initial support for BLIP-2 (#5920) 
Co-authored-by: ywang96 <ywang@roblox.com>
 2024-07-27 11:53:07 +00:00
Roger Wang ecb33a28cb
[CI/Build][Doc] Update CI and Doc for VLM example changes (#6860) 2024-07-27 09:54:14 +00:00
Harry Mellor c53041ae3b
[Doc] Add missing mock import to docs `conf.py` (#6834) 2024-07-27 04:47:33 +00:00
omrishiv 3c3012398e
[Doc] add VLLM_TARGET_DEVICE=neuron to documentation for neuron (#6844) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-07-26 20:20:16 -07:00
Woosuk Kwon ced36cd89b
[ROCm] Upgrade PyTorch nightly version (#6845) 2024-07-26 20:16:13 -07:00
Zhanghao Wu 150a1ffbfd
[Doc] Update SkyPilot doc for wrong indents and instructions for update service (#4283) 2024-07-26 14:39:10 -07:00
Michael Goin 281977bd6e
[Doc] Add Nemotron to supported model docs (#6843) 2024-07-26 17:32:44 -04:00
Li, Jiang 3bbb4936dc
[Hardware] [Intel] Enable Multiprocessing and tensor parallel in CPU backend and update documentation (#6125) 2024-07-26 13:50:10 -07:00
youkaichao 85ad7e2d01
[doc][debugging] add known issues for hangs (#6816) 2024-07-25 21:48:05 -07:00
Woosuk Kwon b7215de2c5
[Docs] Publish 5th meetup slides (#6799) 2024-07-25 16:47:55 -07:00
youkaichao f3ff63c3f4
[doc][distributed] improve multinode serving doc (#6804) 2024-07-25 15:38:32 -07:00
Kuntai Du 6a1e25b151
[Doc] Add documentations for nightly benchmarks (#6412) 2024-07-25 11:57:16 -07:00
Alphi 9e169a4c61
[Model] Adding support for MiniCPM-V (#4087) 2024-07-24 20:59:30 -07:00
Hongxia Yang d88c458f44
[Doc][AMD][ROCm]Added tips to refer to mi300x tuning guide for mi300x users (#6754) 2024-07-24 14:32:57 -07:00
Woosuk Kwon ccc4a73257
[Docs][ROCm] Detailed instructions to build from source (#6680) 2024-07-24 01:07:23 -07:00
dongmao zhang 87525fab92
[bitsandbytes]: support read bnb pre-quantized model (#5753) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-07-23 23:45:09 +00:00
youkaichao 71950af726
[doc][distributed] fix doc argument order (#6691) 2024-07-23 08:55:33 -07:00
Woosuk Kwon cb1362a889
[Docs] Announce llama3.1 support (#6688) 2024-07-23 08:18:15 -07:00
Roger Wang 22fa2e35cb
[VLM][Model] Support image input for Chameleon (#6633) 2024-07-22 23:50:48 -07:00
youkaichao c051bfe4eb
[doc][distributed] doc for setting up multi-node environment (#6529) 
[doc][distributed] add more doc for setting up multi-node environment (#6529)
 2024-07-22 21:22:09 -07:00
Cyrus Leung 739b61a348
[Frontend] Refactor prompt processing (#4028) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-22 10:13:53 -07:00
Matt Wong 06d6c5fe9f
[Bugfix][CI/Build][Hardware][AMD] Fix AMD tests, add HF cache, update CK FA, add partially supported model notes (#6543) 2024-07-20 09:39:07 -07:00
Murali Andoorveedu 45ceb85a0c
[Docs] Update PP docs (#6598) 2024-07-19 16:38:21 -07:00
Simon Mo 30efe41532
[Docs] Update docs for wheel location (#6580) 2024-07-19 12:14:11 -07:00
milo157 a38524f338
[DOC] - Add docker image to Cerebrium Integration (#6510) 2024-07-17 10:22:53 -07:00
Cyrus Leung 5bf35a91e4
[Doc][CI/Build] Update docs and tests to use `vllm serve` (#6431) 2024-07-17 07:43:21 +00:00
Hongxia Yang 10383887e0
[ROCm] Cleanup Dockerfile and remove outdated patch (#6482) 2024-07-16 22:47:02 -07:00
Jiaxin Shan 94162beb9f
[Doc] Fix the lora adapter path in server startup script (#6230) 2024-07-16 10:11:04 -07:00
Woosuk Kwon c467dff24f
[Hardware][TPU] Support MoE with Pallas GMM kernel (#6457) 2024-07-16 09:56:28 -07:00
youkaichao 9f4ccec761
[doc][misc] remind to cancel debugging environment variables (#6481) 
[doc][misc] remind users to cancel debugging environment variables after debugging (#6481)
 2024-07-16 09:45:30 -07:00
Woosuk Kwon 3dee97b05f
[Docs] Add Google Cloud to sponsor list (#6450) 2024-07-15 11:58:10 -07:00
youkaichao 94b82e8c18
[doc][distributed] add suggestion for distributed inference (#6418) 2024-07-15 09:45:51 -07:00
youkaichao 22e79ee8f3
[doc][misc] doc update (#6439) 2024-07-14 23:33:25 -07:00
Robert Cohn 61e85dbad8
[Doc] xpu backend requires running setvars.sh (#6393) 2024-07-14 17:10:11 -07:00
Ethan Xu dbfe254eda
[Feature] vLLM CLI (#5090) 
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-07-14 15:36:43 -07:00
Yuan Tang 6ef3bf912c
Remove unnecessary trailing period in spec_decode.rst (#6405) 2024-07-14 07:58:09 +00:00
Isotr0py 540c0368b1
[Model] Initialize Fuyu-8B support (#3924) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-14 05:27:14 +00:00
Saliya Ekanayake a27f87da34
[Doc] Fix Typo in Doc (#6392) 
Co-authored-by: Saliya Ekanayake <esaliya@d-matrix.ai>
 2024-07-13 00:48:23 +00:00
Simon Mo d719ba24c5
Build some nightly wheels by default (#6380) 2024-07-12 13:56:59 -07:00
youkaichao 2d23b42d92
[doc] update pipeline parallel in readme (#6347) 2024-07-11 11:38:40 -07:00
Jie Fu (傅杰) 439c84581a
[Doc] Update description of vLLM support for CPUs (#6003) 2024-07-10 21:15:29 -07:00
Cyrus Leung 8a924d2248
[Doc] Guide for adding multi-modal plugins (#6205) 2024-07-10 14:55:34 +08:00
Murali Andoorveedu 673dd4cae9
[Docs] Docs update for Pipeline Parallel (#6222) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-07-09 16:24:58 -07:00
Roger Wang 6206dcb29e
[Model] Add PaliGemma (#5189) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-07-07 09:25:50 +08:00
Cyrus Leung 9389380015
[Doc] Move guide for multimodal model and other improvements (#6168) 2024-07-06 17:18:59 +08:00
Roger Wang 175c43eca4
[Doc] Reorganize Supported Models by Type (#6167) 2024-07-06 05:59:36 +00:00
Simon Mo 79d406e918
[Docs] Fix readthedocs for tag build (#6158) 2024-07-05 12:44:40 -07:00
Cyrus Leung ae96ef8fbd
[VLM] Calculate maximum number of multi-modal tokens by model (#6121) 2024-07-04 16:37:23 -07:00
youkaichao 27902d42be
[misc][doc] try to add warning for latest html (#5979) 2024-07-04 09:57:09 -07:00
youkaichao 966fe72141
[doc][misc] bump up py version in installation doc (#6119) 2024-07-03 15:52:04 -07:00
xwjiang2010 d9e98f42e4
[vlm] Remove vision language config. (#6089) 
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-03 22:14:16 +00:00
Michael Goin 47f0954af0
[Kernel] Expand FP8 support to Ampere GPUs using FP8 Marlin (#5975) 2024-07-03 17:38:00 +00:00
Roger Wang f1c78138aa
[Doc] Fix Mock Import (#6094) 2024-07-03 00:13:56 -07:00
Cyrus Leung 9831aec49f
[Core] Dynamic image size support for VLMs (#5276) 
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: ywang96 <ywang@roblox.com>
Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-07-02 20:34:00 -07:00
Mor Zusman 9d6a8daa87
[Model] Jamba support (#4115) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: Erez Schwartz <erezs@ai21.com>
Co-authored-by: Mor Zusman <morz@ai21.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
Co-authored-by: Tomer Asida <tomera@ai21.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
 2024-07-02 23:11:29 +00:00
xwjiang2010 98d6682cd1
[VLM] Remove `image_input_type` from VLM config (#5852) 
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-02 07:57:09 +00:00
Roger Wang 8e0817c262
[Bugfix][Doc] Fix Doc Formatting (#6048) 2024-07-01 15:09:11 -07:00
ning.zhang 83bdcb6ac3
add FAQ doc under 'serving' (#5946) 2024-07-01 14:11:36 -07:00
youkaichao 4050d646e5
[doc][misc] remove deprecated api server in doc (#6037) 2024-07-01 12:52:43 -04:00
Ilya Lavrenov 57f09a419c
[Hardware][Intel] OpenVINO vLLM backend (#5379) 2024-06-28 13:50:16 +00:00
Cyrus Leung 5cbe8d155c
[Core] Registry for processing model inputs (#5214) 
Co-authored-by: ywang96 <ywang@roblox.com>
 2024-06-28 12:09:56 +00:00
Woosuk Kwon 79c92c7c8a
[Model] Add Gemma 2 (#5908) 2024-06-27 13:33:56 -07:00
youkaichao 3fd02bda51
[doc][misc] add note for Kubernetes users (#5916) 2024-06-27 10:07:07 -07:00
Cyrus Leung 96354d6a29
[Model] Add base class for LoRA-supported models (#5018) 2024-06-27 16:03:04 +08:00
youkaichao 294104c3f9
[doc] update usage of env var to avoid conflict (#5873) 2024-06-26 17:57:12 -04:00
Roger Wang 3aa7b6cf66
[Misc][Doc] Add Example of using OpenAI Server with VLM (#5832) 2024-06-25 20:34:25 -07:00
Matt Wong dd793d1de5
[Hardware][AMD][CI/Build][Doc] Upgrade to ROCm 6.1, Dockerfile improvements, test fixes (#5422) 2024-06-25 15:56:15 -07:00
youkaichao c18ebfdd71
[doc][distributed] add both gloo and nccl tests (#5834) 2024-06-25 15:10:28 -04:00
Cyrus Leung f23871e9ee
[Doc] Add notice about breaking changes to VLMs (#5818) 2024-06-25 01:25:03 -07:00
Michael Goin 1744cc99ba
[Doc] Add Phi-3-medium to list of supported models (#5788) 2024-06-24 10:48:55 -07:00
Michael Goin e72dc6cb35
[Doc] Add "Suggest edit" button to doc pages (#5789) 2024-06-24 10:26:17 -07:00
youkaichao c246212952
[doc][faq] add warning to download models for every nodes (#5783) 2024-06-24 15:37:42 +08:00
Woosuk Kwon 8c00f9c15d
[Docs][TPU] Add installation tip for TPU (#5761) 2024-06-21 23:09:40 -07:00
Michael Goin 5b15bde539
[Doc] Documentation on supported hardware for quantization methods (#5745) 2024-06-21 12:44:29 -04:00
Roger Wang 1b2eaac316
[Bugfix][Doc] FIx Duplicate Explicit Target Name Errors (#5703) 2024-06-19 23:10:47 -07:00
Rafael Vasquez e83db9e7e3
[Doc] Update docker references (#5614) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-06-19 15:01:45 -07:00
milo157 2bd231a7b7
[Doc] Added cerebrium as Integration option (#5553) 2024-06-18 15:56:59 -07:00
Isotr0py daef218b55
[Model] Initialize Phi-3-vision support (#4986) 2024-06-17 19:34:33 -07:00
Kunshang Ji 728c4c8a06
[Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (#3814) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>
 2024-06-17 11:01:25 -07:00
youkaichao 845a3f26f9
[Doc] add debugging tips for crash and multi-node debugging (#5581) 2024-06-17 10:08:01 +08:00
Sanger Steel 6e2527a7cb
[Doc] Update documentation on Tensorizer (#5471) 2024-06-14 11:27:57 -07:00
Simon Mo cdab68dcdb
[Docs] Add ZhenFund as a Sponsor (#5548) 2024-06-14 11:17:21 -07:00
Cyrus Leung 0ce7b952f8
[Doc] Update LLaVA docs (#5437) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-06-13 11:22:07 -07:00
Woosuk Kwon a65634d3ae
[Docs] Add 4th meetup slides (#5509) 2024-06-13 10:18:26 -07:00
Li, Jiang 80aa7e91fc
[Hardware][Intel] Optimize CPU backend and add more performance tips (#4971) 
Co-authored-by: Jianan Gu <jianan.gu@intel.com>
 2024-06-13 09:33:14 -07:00
Cyrus Leung b8d4dfff9c
[Doc] Update debug docs (#5438) 2024-06-12 14:49:31 -07:00
Woosuk Kwon 1a8bfd92d5
[Hardware] Initial TPU integration (#5292) 2024-06-12 11:53:03 -07:00
youkaichao 8f89d72090
[Doc] add common case for long waiting time (#5430) 2024-06-11 11:12:13 -07:00
Nick Hill 99dac099ab
[Core][Doc] Default to multiprocessing for single-node distributed case (#5230) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2024-06-11 11:10:41 -07:00
Cade Daniel 89ec06c33b
[Docs] [Spec decode] Fix docs error in code example (#5427) 2024-06-11 10:31:56 -07:00
Kuntai Du 9fde251bf0
[Doc] Add an automatic prefix caching section in vllm documentation (#5324) 
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-06-11 10:24:59 -07:00
Cade Daniel 4c2ffb28ff
[Speculative decoding] Initial spec decode docs (#5400) 2024-06-11 10:15:40 -07:00
SangBin Cho 246598a6b1
[CI] docfix (#5410) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: ywang96 <ywang@roblox.com>
 2024-06-11 01:28:50 -07:00
Roger Wang 3c4cebf751
[Doc][Typo] Fixing Missing Comma (#5403) 2024-06-11 00:20:28 -07:00
youkaichao d8f31f2f8b
[Doc] add debugging tips (#5409) 2024-06-10 23:21:43 -07:00
Michael Goin 77c87beb06
[Doc] Add documentation for FP8 W8A8 (#5388) 2024-06-10 18:55:12 -06:00
Woosuk Kwon cb77ad836f
[Docs] Alphabetically sort sponsors (#5386) 2024-06-10 15:17:19 -05:00
Roger Wang 856c990041
[Docs] Add Docs on Limitations of VLM Support (#5383) 2024-06-10 09:53:50 -07:00
Cyrus Leung 6b29d6fe70
[Model] Initial support for LLaVA-NeXT (#4199) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-06-10 12:47:15 +00:00
Roger Wang 7a9cb294ae
[Frontend] Add OpenAI Vision API Support (#5237) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-06-07 11:23:32 -07:00
Simon Mo f270a39537
[Docs] Add Sequoia as sponsors (#5287) 2024-06-05 18:02:56 +00:00
Jie Fu (傅杰) 87d5abef75
[Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend (#5249) 2024-06-04 09:57:51 -07:00
Breno Faria f775a07e30
[FRONTEND] OpenAI `tools` support named functions (#5032) 2024-06-03 18:25:29 -05:00
Cyrus Leung 7a64d24aad
[Core] Support image processor (#4197) 2024-06-02 22:56:41 -07:00
Nick Hill 657579113f
[Doc] Add checkmark for GPTBigCodeForCausalLM LoRA support (#5171) 2024-05-31 17:20:19 -07:00
Chansung Park 429d89720e
add doc about serving option on dstack (#3074) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-30 10:11:07 -07:00
Cyrus Leung a9bcc7afb2
[Doc] Use intersphinx and update entrypoints docs (#5125) 2024-05-30 09:59:23 -07:00
youkaichao 4fbcb0f27e
[Doc][Build] update after removing vllm-nccl (#5103) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-05-29 23:51:18 +00:00
Cyrus Leung 5ae5ed1e60
[Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Simon Mo 290f4ada2b
[Docs] Add Dropbox as sponsors (#5089) 2024-05-28 10:29:09 -07:00
Eric Xihui Lin 8e192ff967
[Kernel][Backend][Model] Blocksparse flash attention kernel and Phi-3-Small model (#4799) 
Co-authored-by: beagleski <yunanzhang@microsoft.com>
Co-authored-by: bapatra <bapatra@microsoft.com>
Co-authored-by: Barun Patra <codedecde@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-05-24 22:00:52 -07:00
youkaichao 6a50f4cafa
[Doc] add ccache guide in doc (#5012) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-05-23 23:21:54 +00:00