vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,152 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

612 Commits

Author SHA1 Message Date
Yuan Tang 4800339c62
Add docs on serving with Llama Stack (#10183) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2024-11-11 11:28:55 -08:00
youkaichao f0f2e5638e
[doc] improve debugging code (#10206) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-10 17:49:40 -08:00
Shawn Du 20cf2f553c
[Misc] small fixes to function tracing file path (#9543) 
Signed-off-by: Shawn Du <shawnd200@outlook.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2024-11-10 15:21:06 -08:00
Yongzao bfb7d61a7c
[doc] Polish the integration with huggingface doc (#10195) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2024-11-10 10:22:04 -08:00
youkaichao 9fa4bdde9d
[ci][build] limit cmake version (#10188) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-09 16:27:26 -08:00
cjackal d88bff1b96
[Frontend] add `add_request_id` middleware (#9594) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2024-11-09 10:18:29 +00:00
youkaichao 8a4358ecb5
[doc] explaining the integration with huggingface (#10173) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-09 01:02:54 -08:00
Cyrus Leung 49d2a41a86
[Doc] Adjust RunLLM location (#10176) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-08 20:07:10 -08:00
Cyrus Leung e0191a95d8
[0/N] Rename `MultiModalInputs` to `MultiModalKwargs` (#10040) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-09 11:31:02 +08:00
Rafael Vasquez 6b30471586
[Misc] Improve Web UI (#10090) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-11-08 09:51:04 -08:00
Russell Bryant 3a7f15a398
[Doc] Move CONTRIBUTING to docs site (#9924) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-11-08 05:15:12 +00:00
whyiug 40d0e7411d
[Doc] Update FAQ links in spec_decode.rst (#9662) 
Signed-off-by: whyiug <whyiug@hotmail.com>
 2024-11-08 04:44:58 +00:00
litianjian 28b2877d30
Online video support for VLMs (#10020) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-07 20:25:59 +00:00
Maximilien de Bayser ae62fd17c0
[Frontend] Tool calling parser for Granite 3.0 models (#9027) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-11-07 07:09:02 -08:00
Rafael Vasquez d7263a1bb8
Doc: Improve benchmark documentation (#9927) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-11-06 23:50:35 -08:00
Cyrus Leung db7db4aab9
[Misc] Consolidate ModelConfig code related to HF config (#10104) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-07 06:00:21 +00:00
youkaichao e7b84c394d
[doc] add back Python 3.8 ABI (#10100) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-06 21:06:41 -08:00
Li, Jiang a4b3e0c1e9
[Hardware][CPU] Update torch 2.5 (#9911) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2024-11-07 04:43:08 +00:00
Russell Bryant 098f94de42
[CI/Build] Drop Python 3.8 support (#10038) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-06 14:31:01 +00:00
Eric 406d4cc480
[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration (#10022) 
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
 2024-11-06 14:13:15 +00:00
Jee Jee Li a5bba7d234
[Model] Add Idefics3 support (#9767) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: B-201 <Joy25810@foxmail.com>
Co-authored-by: B-201 <Joy25810@foxmail.com>
 2024-11-06 11:41:17 +00:00
Jee Jee Li 2003cc3513
[Model][LoRA]LoRA support added for LlamaEmbeddingModel (#10071) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-11-06 09:49:19 +00:00
Konrad Zawora a02a50e6e5
[Hardware][Intel-Gaudi] Add Intel Gaudi (HPU) inference backend (#6143) 
Signed-off-by: yuwenzho <yuwen.zhou@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Bob Zhu <bob.zhu@intel.com>
Signed-off-by: zehao-intel <zehao.huang@intel.com>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
Co-authored-by: Michal Adamczyk <madamczyk@habana.ai>
Co-authored-by: Marceli Fylcek <mfylcek@habana.ai>
Co-authored-by: Himangshu Lahkar <49579433+hlahkar@users.noreply.github.com>
Co-authored-by: Vivek Goel <vgoel@habana.ai>
Co-authored-by: yuwenzho <yuwen.zhou@intel.com>
Co-authored-by: Dominika Olszewska <dolszewska@habana.ai>
Co-authored-by: barak goldberg <149692267+bgoldberg-habana@users.noreply.github.com>
Co-authored-by: Michal Szutenberg <37601244+szutenberg@users.noreply.github.com>
Co-authored-by: Jan Kaniecki <jkaniecki@habana.ai>
Co-authored-by: Agata Dobrzyniewicz <160237065+adobrzyniewicz-habana@users.noreply.github.com>
Co-authored-by: Krzysztof Wisniewski <kwisniewski@habana.ai>
Co-authored-by: Dudi Lester <160421192+dudilester@users.noreply.github.com>
Co-authored-by: Ilia Taraban <tarabanil@gmail.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Michał Kuligowski <mkuligowski@habana.ai>
Co-authored-by: Jakub Maksymczuk <jmaksymczuk@habana.ai>
Co-authored-by: Tomasz Zielinski <85164140+tzielinski-habana@users.noreply.github.com>
Co-authored-by: Sun Choi <schoi@habana.ai>
Co-authored-by: Iryna Boiko <iboiko@habana.ai>
Co-authored-by: Bob Zhu <41610754+czhu15@users.noreply.github.com>
Co-authored-by: hlin99 <73271530+hlin99@users.noreply.github.com>
Co-authored-by: Zehao Huang <zehao.huang@intel.com>
Co-authored-by: Andrzej Kotłowski <Andrzej.Kotlowski@intel.com>
Co-authored-by: Yan Tomsinsky <73292515+Yantom1@users.noreply.github.com>
Co-authored-by: Nir David <ndavid@habana.ai>
Co-authored-by: Yu-Zhou <yu.zhou@intel.com>
Co-authored-by: Ruheena Suhani Shaik <rsshaik@habana.ai>
Co-authored-by: Karol Damaszke <kdamaszke@habana.ai>
Co-authored-by: Marcin Swiniarski <mswiniarski@habana.ai>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Jacek Czaja <jacek.czaja@intel.com>
Co-authored-by: Jacek Czaja <jczaja@habana.ai>
Co-authored-by: Yuan <yuan.zhou@outlook.com>
 2024-11-06 01:09:10 -08:00
Aaron Pham 21063c11c7
[CI/Build] drop support for Python 3.8 EOL (#8464) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2024-11-06 07:11:55 +00:00
Richard Liu cd34029e91
Refactor TPU requirements file and pin build dependencies (#10010) 
Signed-off-by: Richard Liu <ricliu@google.com>
 2024-11-05 16:48:44 +00:00
Roger Wang 6e056bcf04
[Doc] Update VLM doc about loading from local files (#9999) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2024-11-04 19:47:11 +00:00
shanshan wang 54597724f4
[Model] Add support for H2OVL-Mississippi models (#9747) 
Signed-off-by: Shanshan Wang <shanshan.wang@h2o.ai>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-11-04 00:15:36 +00:00
Michael Green 1d4cfe2be1
[Doc] Updated tpu-installation.rst with more details (#9926) 
Signed-off-by: Michael Green <mikegre@google.com>
 2024-11-02 10:06:45 -04:00
Nick Hill eed92f12fc
[Docs] Update Granite 3.0 models in supported models table (#9930) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-11-02 09:02:18 +00:00
Cyrus Leung ba0d892074
[Frontend] Use a proper chat template for VLM2Vec (#9912) 2024-11-01 14:09:07 +00:00
Cyrus Leung 06386a64dd
[Frontend] Chat-based Embeddings API (#9759) 2024-11-01 08:13:35 +00:00
Cyrus Leung d3aa2a8b2f
[Doc] Update multi-input support (#9906) 2024-11-01 07:34:49 +00:00
Yongzao 2b5bf20988
[torch.compile] Adding torch compile annotations to some models (#9876) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2024-11-01 00:25:47 -07:00
Joe Runde 031a7995f3
[Bugfix][Frontend] Reject guided decoding in multistep mode (#9892) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-11-01 01:09:46 +00:00
Jee Jee Li 5608e611c2
[Doc] Update Qwen documentation (#9869) 2024-10-31 08:54:18 +00:00
Guillaume Calmettes abbfb6134d
[Misc][OpenAI] deprecate max_tokens in favor of new max_completion_tokens field for chat completion endpoint (#9837) 2024-10-30 18:15:56 -07:00
youkaichao c2cd1a2142
[doc] update pp support (#9853) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-10-30 13:36:51 -07:00
Joe Runde 33d257735f
[Doc] link bug for multistep guided decoding (#9843) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-10-30 17:28:29 +00:00
Woosuk Kwon 211fe91aa8
[TPU] Correctly profile peak memory usage & Upgrade PyTorch XLA (#9438) 2024-10-30 09:41:38 +00:00
Yan Ma 04a3ae0aca
[Bugfix] Fix multi nodes TP+PP for XPU (#8884) 
Signed-off-by: YiSheng5 <syhm@mail.ustc.edu.cn>
Signed-off-by: yan ma <yan.ma@intel.com>
Co-authored-by: YiSheng5 <syhm@mail.ustc.edu.cn>
 2024-10-29 21:34:45 -07:00
Will Eaton 882a1ad0de
[Model] tool calling support for ibm-granite/granite-20b-functioncalling (#8339) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com>
 2024-10-29 15:07:37 -07:00
Russell Bryant c5d7fb9ddc
[Doc] fix third-party model example (#9771) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-10-28 19:39:21 -07:00
kakao-kevin-us 6650e6a930
[Model] Add classification Task with Qwen2ForSequenceClassification (#9704) 
Signed-off-by: Kevin-Yang <ykcha9@gmail.com>
Co-authored-by: Kevin-Yang <ykcha9@gmail.com>
 2024-10-26 17:53:35 +00:00
Rafael Vasquez 228cfbd03f
[Doc] Improve quickstart documentation (#9256) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-25 14:32:10 -07:00
Cyrus Leung b979143d5b
[Doc] Move additional tips/notes to the top (#9647) 2024-10-24 09:43:59 +00:00
Yongzao 8a02cd045a
[torch.compile] Adding torch compile annotations to some models (#9639) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2024-10-24 00:54:57 -07:00
Cyrus Leung 836e8ef6ee
[Bugfix] Fix PP for ChatGLM and Molmo (#9422) 2024-10-24 06:12:05 +00:00
Vinay R Damodaran 33bab41060
[Bugfix]: Make chat content text allow type content (#9358) 
Signed-off-by: Vinay Damodaran <vrdn@hey.com>
 2024-10-24 05:05:49 +00:00
Yunfei Chu fc6c274626
[Model] Add Qwen2-Audio model support (#9248) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-23 17:54:22 +00:00
Cyrus Leung 831540cf04
[Model] Support E5-V (#9576) 2024-10-23 11:35:29 +08:00
Seth Kimmel 208cb34c81
[Doc]: Update tensorizer docs to include vllm[tensorizer] (#7889) 
Co-authored-by: Kaunil Dhruv <dhruv.kaunil@gmail.com>
 2024-10-22 15:43:25 -07:00
Yuan 32a1ee74a0
[Hardware][Intel CPU][DOC] Update docs for CPU backend (#6212) 
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Gubrud, Aaron D <aaron.d.gubrud@intel.com>
Co-authored-by: adgubrud <96072084+adgubrud@users.noreply.github.com>
 2024-10-22 10:38:04 -07:00
Isotr0py bb392ea2d2
[Model][VLM] Initialize support for Mono-InternVL model (#9528) 2024-10-22 16:01:46 +00:00
Rafael Vasquez f7db5f0fa9
[Doc] Use shell code-blocks and fix section headers (#9508) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-22 06:43:24 +00:00
youkaichao d621c43df7
[doc] fix format (#9562) 2024-10-21 13:54:57 -07:00
Dhia Eddine Rhaiem f6b97293aa
[Model] FalconMamba Support (#9325) 2024-10-21 12:50:16 -04:00
Michael Goin 3921a2f29e
[Model] Support Pixtral models in the HF Transformers format (#9036) 2024-10-18 13:29:56 -06:00
Cyrus Leung 051eaf6db3
[Model] Add user-configurable task for models that support both generation and embedding (#9424) 2024-10-18 11:31:58 -07:00
tomeras91 d2b1bf55ec
[Frontend][Feature] Add jamba tool parser (#9154) 2024-10-18 10:27:48 +00:00
Kuntai Du 81ede99ca4
[Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
Li, Jiang 5eda21e773
[Hardware][CPU] compressed-tensor INT8 W8A8 AZP support (#9344) 2024-10-17 12:21:04 -04:00
Junhao Li 5b8a1fde84
[Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft (#9396) 2024-10-16 16:40:24 +00:00
Roger Wang 59230ef32b
[Misc] Consolidate example usage of OpenAI client for multimodal models (#9412) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-16 11:20:51 +00:00
Cyrus Leung cee711fdbb
[Core] Rename input data types (#8688) 2024-10-16 10:49:37 +00:00
Cyrus Leung 7abba39ee6
[Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303) 2024-10-16 14:31:00 +08:00
Michael Goin 8e836d982a
[Doc] Fix code formatting in spec_decode.rst (#9348) 2024-10-14 21:29:11 -07:00
Tyler Michael Smith 169b530607
[Bugfix] Clean up some cruft in mamba.py (#9343) 2024-10-15 00:24:25 +00:00
Reza Salehi dfe43a2071
[Model] Molmo vLLM Integration (#9016) 
Co-authored-by: sanghol <sanghol@allenai.org>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-10-14 07:56:24 -07:00
Yunmeng 2b184ddd4f
[Misc][Installation] Improve source installation script and doc (#9309) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-10-12 09:36:40 -07:00
Wallas Henrique 8baf85e4e9
[Doc] Compatibility matrix for mutual exclusive features (#8512) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2024-10-11 11:18:50 -07:00
sixgod 6cf1167c1a
[Model] Add GLM-4v support and meet vllm==0.6.2 (#9242) 2024-10-11 17:36:13 +00:00
Tyler Michael Smith 7342a7d7f8
[Model] Support Mamba (#6484) 2024-10-11 15:40:06 +00:00
Cyrus Leung e808156f30
[Misc] Collect model support info in a single process per model (#9233) 2024-10-11 11:08:11 +00:00
omrishiv f990bab2a4
[Doc][Neuron] add note to neuron documentation about resolving triton issue (#9257) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-10-10 23:36:32 +00:00
Rafael Vasquez 055f3270d4
[Doc] Improve debugging documentation (#9204) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-10 10:48:51 -07:00
whyiug 04de9057ab
[Model] support input image embedding for minicpmv (#9237) 2024-10-10 15:00:47 +00:00
Li, Jiang ca77dd7a44
[Hardware][CPU] Support AWQ for CPU backend (#7515) 2024-10-09 10:28:08 -06:00
Jiangtao Hu dc4aea677a
[Doc] Fix VLM prompt placeholder sample bug (#9170) 2024-10-09 08:59:42 +00:00
Yuan Tang acce7630c1
Update link to KServe deployment guide (#9173) 2024-10-09 03:58:49 +00:00
Michael Goin 9ba0bd6aa6
Add `lm-eval` directly to requirements-test.txt (#9161) 2024-10-08 18:22:31 -07:00
Rafael Vasquez de24046fcd
[Doc] Improve contributing and installation documentation (#9132) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-10-08 20:22:08 +00:00
Sayak Paul 1874c6a1b0
[Doc] Update vlm.rst to include an example on videos (#9155) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-10-08 18:12:29 +00:00
TimWang 93cf74a8a7
[Doc]: Add deploying_with_k8s guide (#8451) 2024-10-07 13:31:45 -07:00
Cyrus Leung 151ef4efd2
[Model] Support NVLM-D and fix QK Norm in InternViT (#9045) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2024-10-07 11:55:12 +00:00
Cyrus Leung b22b798471
[Model] PP support for embedding models and update docs (#9090) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-10-06 16:35:27 +08:00
Cyrus Leung f22619fe96
[Misc] Remove user-facing error for removed VLM args (#9104) 2024-10-06 01:33:52 -07:00
Andy Dai 5df1834895
[Bugfix] Fix order of arguments matters in config.yaml (#8960) 2024-10-05 17:35:11 +00:00
Roger Wang 26aa325f4f
[Core][VLM] Test registration for OOT multimodal models (#8717) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:38:25 -07:00
Cyrus Leung 0e36fd4909
[Misc] Move registry to its own file (#9064) 2024-10-04 10:01:37 +00:00
Murali Andoorveedu 0f6d7a9a34
[Models] Add remaining model PP support (#7168) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Signed-off-by: Murali Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:56:58 +08:00
代君 3dbb215b38
[Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405) 2024-10-04 10:36:39 +08:00
Nick Hill 18c2e30c57
[Doc] Update Granite model docs (#9025) 2024-10-03 02:42:24 +00:00
Sergey Shlyapnikov f58d4fccc9
[OpenVINO] Enable GPU support for OpenVINO vLLM backend (#8192) 2024-10-02 17:50:01 -04:00
Cyrus Leung 4f341bd4bf
[Doc] Update list of supported models (#8987) 2024-10-02 00:35:39 +08:00
whyiug e01ab595d8
[Model] support input embeddings for qwen2vl (#8856) 2024-09-30 03:16:10 +00:00
youkaichao cc276443b5
[doc] organize installation doc and expose per-commit docker (#8931) 2024-09-28 17:48:41 -07:00
youkaichao d86f6b2afb
[misc] fix wheel name (#8919) 2024-09-27 22:10:44 -07:00
Cyrus Leung 3b00b9c26c
[Core] rename`PromptInputs` and `inputs` (#8876) 2024-09-26 20:35:15 -07:00
Maximilien de Bayser 344cd2b6f4
[Feature] Add support for Llama 3.1 and 3.2 tool use (#8343) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-09-26 17:01:42 -07:00
youkaichao 70de39f6b4
[misc][installation] build from source without compilation (#8818) 2024-09-26 13:19:04 -07:00
Roger Wang 4bb98f2190
[Misc] Update config loading for Qwen2-VL and remove Granite (#8837) 2024-09-26 07:45:30 -07:00
Roger Wang e2c6e0a829
[Doc] Update doc for Transformers 4.45 (#8817) 2024-09-25 13:29:48 -07:00
Chen Zhang 770ec6024f
[Model] Add support for the multi-modal Llama 3.2 model (#8811) 
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Chang Su <chang.s.su@oracle.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-25 13:29:32 -07:00
Simon Mo 4f1ba0844b
Revert "rename PromptInputs and inputs with backward compatibility (#8760) (#8810) 2024-09-25 10:36:26 -07:00
Cyrus Leung 28e1299e60
rename PromptInputs and inputs with backward compatibility (#8760) 2024-09-25 09:36:47 -07:00
Hongxia Yang 1c046447a6
[CI/Build][Bugfix][Doc][ROCm] CI fix and doc update after ROCm 6.2 upgrade (#8777) 2024-09-25 22:26:37 +08:00
Jee Jee Li 13f9f7a3d0
[[Misc]Upgrade bitsandbytes to the latest version 0.44.0 (#8768) 2024-09-24 17:08:55 -07:00
Simon Mo 3185fb0cca
Revert "[Core] Rename `PromptInputs` to `PromptType`, and `inputs` to `prompt`" (#8750) 2024-09-24 05:45:20 +00:00
Hongxia Yang 530821d00c
[Hardware][AMD] ROCm6.2 upgrade (#8674) 2024-09-23 18:52:39 -07:00
Daniele ee5f34b1c2
[CI/Build] use setuptools-scm to set __version__ (#4738) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-09-23 09:44:26 -07:00
Yan Ma d23679eb99
[Bugfix] fix docker build for xpu (#8652) 2024-09-22 22:54:18 -07:00
youkaichao d4a2ac8302
[build] enable existing pytorch (for GH200, aarch64, nightly) (#8713) 2024-09-22 12:47:54 -07:00
litianjian 5b59532760
[Model][VLM] Add LLaVA-Onevision model support (#8486) 
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-22 10:51:44 -07:00
Andy Dai 4dfdf43196
[Doc] Fix typo in AMD installation guide (#8689) 2024-09-21 00:24:12 -07:00
Cyrus Leung 0057894ef7
[Core] Rename `PromptInputs` and `inputs`(#8673) 2024-09-20 19:00:54 -07:00
omrishiv 7c8566aa4f
[Doc] neuron documentation update (#8671) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-09-20 15:04:37 -07:00
Niklas Muennighoff 3b63de9353
[Model] Add OLMoE (#7922) 2024-09-20 09:31:41 -07:00
Jiaxin Shan 260d40b5ea
[Core] Support Lora lineage and base model metadata management (#6315) 2024-09-20 06:20:56 +00:00
Isotr0py ea4647b7d7
[Doc] Add documentation for GGUF quantization (#8618) 2024-09-19 13:15:55 -06:00
Geun, Lim e18749ff09
[Model] Support Solar Model (#8386) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-09-18 11:04:00 -06:00
Alexander Matveev 7c7714d856
[Core][Bugfix][Perf] Introduce `MQLLMEngine` to avoid `asyncio` OH (#8157) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-09-18 13:56:58 +00:00
youkaichao fa0c114fad
[doc] improve installation doc (#8550) 
Co-authored-by: Andy Dai <76841985+Imss27@users.noreply.github.com>
 2024-09-17 16:24:06 -07:00
youkaichao 2759a43a26
[doc] update doc on testing and debugging (#8514) 2024-09-16 12:10:23 -07:00
ywfang 8a0cf1ddc3
[Model] support minicpm3 (#8297) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-14 14:50:26 +00:00
Isotr0py f57092c00b
[Doc] Add oneDNN installation to CPU backend documentation (#8467) 2024-09-13 18:06:30 +00:00
Cyrus Leung a84e598e21
[CI/Build] Reorganize models tests (#7820) 2024-09-13 10:20:06 -07:00
youkaichao cab69a15e4
[doc] recommend pip instead of conda (#8446) 2024-09-12 23:52:41 -07:00
Alex Brooks c6202daeed
[Model] Support multiple images for qwen-vl (#8247) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-12 10:10:54 -07:00
Patrick von Platen d394787e52
Pixtral (#8377) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-11 14:41:55 -07:00
Yang Fan 3b7fea770f
[Model][VLM] Add Qwen2-VL model support (#7905) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-11 09:31:19 -07:00
Yangshen⚡Deng 6a512a00df
[model] Support for Llava-Next-Video model (#7559) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-10 22:21:36 -07:00
Simon Mo a1d874224d
Add NVIDIA Meetup slides, announce AMD meetup, and add contact info (#8319) 2024-09-09 23:21:00 -07:00
Isotr0py e807125936
[Model][VLM] Support multi-images inputs for InternVL2 models (#8201) 2024-09-07 16:38:23 +08:00
Cyrus Leung 2f707fcb35
[Model] Multi-input support for LLaVA (#8238) 2024-09-07 02:57:24 +00:00
William Lin 12dd715807
[misc] [doc] [frontend] LLM torch profiler support (#7943) 2024-09-06 17:48:48 -07:00
Dipika Sikka 23f322297f
[Misc] Remove `SqueezeLLM` (#8220) 2024-09-06 16:29:03 -06:00
Jiaxin Shan db3bf7c991
[Core] Support load and unload LoRA in api server (#6566) 
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-09-05 18:10:33 -07:00
sroy745 2febcf2777
[Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (#7962) 2024-09-05 16:25:29 -04:00
Alex Brooks 9da25a88aa
[MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (#8029) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-09-05 12:48:10 +00:00
Cyrus Leung 288a938872
[Doc] Indicate more information about supported modalities (#8181) 2024-09-05 10:51:53 +00:00
Kyle Mistele e02ce498be
[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
Woosuk Kwon 61f4a93d14
[TPU][Bugfix] Use XLA rank for persistent cache path (#8137) 2024-09-03 18:35:33 -07:00
Wenxiang 1248e8506a
[Model] Adding support for MSFT Phi-3.5-MoE (#7729) 
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Zeqi Lin <zelin@microsoft.com>
Co-authored-by: Zeqi Lin <Zeqi.Lin@microsoft.com>
 2024-08-30 13:42:57 -06:00
Kaunil Dhruv 058344f89a
[Frontend]-config-cli-args (#7737) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Kaunil Dhruv <kaunil_dhruv@intuit.com>
 2024-08-30 08:21:02 -07:00
Yohan Na dc13e99348
[MODEL] add Exaone model support (#7819) 2024-08-29 23:34:20 -07:00
Stas Bekman 8c56e57def
[Doc] fix 404 link (#7966) 2024-08-28 13:54:23 -07:00
Woosuk Kwon eeffde1ac0
[TPU] Upgrade PyTorch XLA nightly (#7967) 2024-08-28 13:10:21 -07:00
Stas Bekman 98c12cffe5
[Doc] fix the autoAWQ example (#7937) 2024-08-28 12:12:32 +00:00
Peter Salas fab5f53e2d
[Core][VLM] Stack multimodal tensors to represent multiple images within each prompt (#7902) 2024-08-28 01:53:56 +00:00
Peter Salas 57792ed469
[Doc] Fix incorrect docs from #7615 (#7788) 2024-08-22 10:02:06 -07:00
zifeitong df1a21131d
[Model] Fix Phi-3.5-vision-instruct 'num_crops' issue (#7710) 2024-08-22 09:36:24 +08:00
Peter Salas 1ca0d4f86b
[Model] Add UltravoxModel and UltravoxConfig (#7615) 2024-08-21 22:49:39 +00:00
William Lin dd53c4b023
[misc] Add Torch profiler support (#7451) 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-21 15:39:26 -07:00
Roger Wang 4506641212
[Doc] Section for Multimodal Language Models (#7719) 2024-08-20 23:24:01 -07:00
Ilya Lavrenov 398521ad19
[OpenVINO] Updated documentation (#7687) 2024-08-20 07:33:56 -06:00
Michael Goin d4f0f17b02
[Doc] Update quantization supported hardware table (#7595) 2024-08-16 13:59:27 -07:00
Michael Goin b3f4e17935
[Doc] Add docs for llmcompressor INT8 and FP8 checkpoints (#7444) 2024-08-16 13:59:16 -07:00
Kameshwara Pavan Kumar Mantha 22b39e11f2
llama_index serving integration documentation (#6973) 
Co-authored-by: pavanmantha <pavan.mantha@thevaslabs.io>
 2024-08-14 15:38:37 -07:00
Cyrus Leung 3f674a49b5
[VLM][Core] Support profiling with multiple multi-modal inputs per prompt (#7126) 2024-08-14 17:55:42 +00:00
youkaichao 199adbb7cf
[doc] update test script to include cudagraph (#7501) 2024-08-13 21:52:58 -07:00
Cyrus Leung dd164d72f3
[Bugfix][Docs] Update list of mock imports (#7493) 2024-08-13 20:37:30 -07:00
Woosuk Kwon a08df8322e
[TPU] Support multi-host inference (#7457) 2024-08-13 16:31:20 -07:00
Peter Salas 00c3d68e45
[Frontend][Core] Add plumbing to support audio language models (#7446) 2024-08-13 17:39:33 +00:00
Woosuk Kwon e20233d361
Revert "[Doc] Update supported_hardware.rst (#7276)" (#7467) 2024-08-13 01:37:08 -07:00
jon-chuang a046f86397
[Core/Bugfix] Add FP8 K/V Scale and dtype conversion for prefix/prefill Triton Kernel (#7208) 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-12 22:47:41 +00:00
Roger Wang e6e42e4b17
[Core][VLM] Support image embeddings as input (#6613) 2024-08-12 16:16:06 +08:00
Simon Mo f020a6297e
[Docs] Update readme (#7316) 2024-08-11 17:13:37 -07:00
tomeras91 02b1988b9f
[Doc] building vLLM with VLLM_TARGET_DEVICE=empty (#7403) 2024-08-11 14:38:17 -07:00
Woosuk Kwon 90bab18f24
[TPU] Use mark_dynamic to reduce compilation time (#7340) 2024-08-10 18:12:22 -07:00
Simon Mo 5923532e15
Add Skywork AI as Sponsor (#7314) 2024-08-08 13:59:57 -07:00
Jee Jee Li 757ac70a64
[Model] Rename MiniCPMVQwen2 to MiniCPMV2.6 (#7273) 2024-08-08 14:02:41 +00:00
Michael Goin 6d94420246
[Doc] Update supported_hardware.rst (#7276) 2024-08-07 14:21:50 -07:00
Stas Bekman 0e12cd67a8
[Doc] add online speculative decoding example (#7243) 2024-08-07 09:58:02 -07:00
Ilya Lavrenov 80cbe10c59
[OpenVINO] migrate to latest dependencies versions (#7251) 2024-08-07 09:49:10 -07:00
Roger Wang 2385c8f374
[Doc] Mock new dependencies for documentation (#7245) 2024-08-07 06:43:03 +00:00
Thomas Parnell 789937af2e
[Doc] [SpecDecode] Update MLPSpeculator documentation (#7100) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-08-05 23:29:43 +00:00
Simon Mo 4db5176d97
bump version to v0.5.4 (#7139) 2024-08-05 14:39:48 -07:00
Jee Jee Li 179a6a36f2
[Model]Refactor MiniCPMV (#7020) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 08:12:41 +00:00
Yihuan Bu 654bc5ca49
Support for guided decoding for offline LLM (#6878) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 03:12:09 +00:00
Michael Goin b482b9a5b1
[CI/Build] Add support for Python 3.12 (#7035) 2024-08-02 13:51:22 -07:00
Murali Andoorveedu fc912e0886
[Models] Support Qwen model with PP (#6974) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
 2024-08-01 12:40:43 -07:00
Jee Jee Li 7ecee34321
[Kernel][RFC] Refactor the punica kernel based on Triton (#5036) 2024-07-31 17:12:24 -07:00
Alphi 2f4e108f75
[Bugfix] Clean up MiniCPM-V (#6939) 
Co-authored-by: hezhihui <hzh7269@modelbest.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-07-31 14:39:19 +00:00
Cyrus Leung f230cc2ca6
[Bugfix] Fix broadcasting logic for `multi_modal_kwargs` (#6836) 2024-07-31 10:38:45 +08:00
Ilya Lavrenov 5895b24677
[OpenVINO] Updated OpenVINO requirements and build docs (#6948) 2024-07-30 11:33:01 -07:00
Isotr0py 7cbd9ec7a9
[Model] Initialize support for InternVL2 series models (#6514) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-29 10:16:30 +00:00
Woosuk Kwon fad5576c58
[TPU] Reduce compilation time & Upgrade PyTorch XLA version (#6856) 2024-07-27 10:28:33 -07:00
Chenggang Wu f954d0715c
[Docs] Add RunLLM chat widget (#6857) 2024-07-27 09:24:46 -07:00
Cyrus Leung 1ad86acf17
[Model] Initial support for BLIP-2 (#5920) 
Co-authored-by: ywang96 <ywang@roblox.com>
 2024-07-27 11:53:07 +00:00
Roger Wang ecb33a28cb
[CI/Build][Doc] Update CI and Doc for VLM example changes (#6860) 2024-07-27 09:54:14 +00:00
Harry Mellor c53041ae3b
[Doc] Add missing mock import to docs `conf.py` (#6834) 2024-07-27 04:47:33 +00:00
omrishiv 3c3012398e
[Doc] add VLLM_TARGET_DEVICE=neuron to documentation for neuron (#6844) 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
 2024-07-26 20:20:16 -07:00
Woosuk Kwon ced36cd89b
[ROCm] Upgrade PyTorch nightly version (#6845) 2024-07-26 20:16:13 -07:00
Zhanghao Wu 150a1ffbfd
[Doc] Update SkyPilot doc for wrong indents and instructions for update service (#4283) 2024-07-26 14:39:10 -07:00
Michael Goin 281977bd6e
[Doc] Add Nemotron to supported model docs (#6843) 2024-07-26 17:32:44 -04:00
Li, Jiang 3bbb4936dc
[Hardware] [Intel] Enable Multiprocessing and tensor parallel in CPU backend and update documentation (#6125) 2024-07-26 13:50:10 -07:00
youkaichao 85ad7e2d01
[doc][debugging] add known issues for hangs (#6816) 2024-07-25 21:48:05 -07:00
Woosuk Kwon b7215de2c5
[Docs] Publish 5th meetup slides (#6799) 2024-07-25 16:47:55 -07:00
youkaichao f3ff63c3f4
[doc][distributed] improve multinode serving doc (#6804) 2024-07-25 15:38:32 -07:00
Kuntai Du 6a1e25b151
[Doc] Add documentations for nightly benchmarks (#6412) 2024-07-25 11:57:16 -07:00