vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,557 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

159 Commits

Author SHA1 Message Date
Suraj Deshmukh 2a622d704a
k8s-config: Update the secret to use stringData (#11679) 
Signed-off-by: Suraj Deshmukh <surajd.service@gmail.com>
 2025-01-06 08:01:22 +00:00
Cyrus Leung 402d378360
[Doc] [1/N] Reorganize Getting Started section (#11645) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-06 02:18:33 +00:00
Chunyang Wen 84c35c374a
According to vllm.EngineArgs, the name should be distributed_executor_backend (#11689) 2025-01-02 18:14:16 +00:00
Cyrus Leung 32b4c63f02
[Doc] Convert list tables to MyST (#11594) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-29 15:56:22 +08:00
Cyrus Leung d427e5cfda
[Doc] Minor documentation fixes (#11580) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-28 21:53:59 +08:00
AlexHe99 d003f3ea39
Update deploying_with_k8s.md with AMD ROCm GPU example (#11465) 
Signed-off-by: Alex He <alehe@amd.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-12-27 10:00:04 +00:00
Robert Shaw 0c0c2015c5
Update openai_compatible_server.md (#11536) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-12-26 16:26:18 -08:00
Cyrus Leung 6ad909fdda
[Doc] Improve GitHub links (#11491) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-25 14:49:26 -08:00
Cyrus Leung 9edca6bf8f
[Frontend] Online Pooling API (#11457) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-24 17:54:30 +08:00
Rafael Vasquez 32aa2059ad
[Docs] Convert rST to MyST (Markdown) (#11145) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-23 22:35:38 +00:00
omer-dayan 995f56236b
[Core] Loading model from S3 using RunAI Model Streamer as optional loader (#10192) 
Signed-off-by: OmerD <omer@run.ai>
 2024-12-20 16:46:24 +00:00
youkaichao 7801f56ed7
[ci][gh200] dockerfile clean up (#11351) 
Signed-off-by: drikster80 <ed.sealing@gmail.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: drikster80 <ed.sealing@gmail.com>
Co-authored-by: cenzhiyao <2523403608@qq.com>
 2024-12-19 18:13:06 -08:00
kYLe 66d4b16724
[Frontend] Add OpenAI API support for input_audio (#11027) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-16 22:09:58 -08:00
youkaichao 35bae114a8
fix gh200 tests on main (#11246) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-16 17:22:38 -08:00
bk-TurbaAI 35ffa682b1
[Docs] hint to enable use of GPU performance counters in profiling tools for multi-node distributed serving (#11235) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-12-16 22:20:39 +00:00
cennn b3b1526f03
WIP: [CI/Build] simplify Dockerfile build for ARM64 / GH200 (#11212) 
Signed-off-by: drikster80 <ed.sealing@gmail.com>
Co-authored-by: drikster80 <ed.sealing@gmail.com>
 2024-12-16 09:20:49 +00:00
AlexHe99 da6f409246
Update deploying_with_k8s.rst (#10922) 2024-12-15 16:33:58 -08:00
Cyrus Leung 0920ab9131
[Doc] Reorganize online pooling APIs (#11172) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-14 00:22:22 +08:00
Ramon Ziai d4d5291cc2
fix(docs): typo in helm install instructions (#11141) 
Signed-off-by: Ramon Ziai <ramon.ziai@bettermarks.com>
 2024-12-12 17:36:32 +00:00
Yuan Tang 24a36d6d5f
Update link to LlamaStack remote vLLM guide in serving_with_llamastack.rst (#11112) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
 2024-12-12 02:39:21 +00:00
Maxime Fournioux fe2e10c71b
Add example of helm chart for vllm deployment on k8s (#9199) 
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
 2024-12-10 09:19:27 +00:00
Michael Goin 6d525288c1
[Docs] Add dedicated tool calling page to docs (#10554) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-12-09 20:15:34 -05:00
Sam Stoelinga 7406274041
[Doc] add KubeAI to serving integrations (#10837) 
Signed-off-by: Sam Stoelinga <sammiestoel@gmail.com>
 2024-12-06 17:03:56 +00:00
Cyrus Leung aa39a8e175
[Doc] Create a new "Usage" section (#10827) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-05 11:19:35 +08:00
Murali Andoorveedu db66e018ea
[Bugfix] Fix for Spec model TP + Chunked Prefill (#10232) 
Signed-off-by: andoorve <37849411+andoorve@users.noreply.github.com>
Signed-off-by: Sourashis Roy <sroy@roblox.com>
Co-authored-by: Sourashis Roy <sroy@roblox.com>
 2024-11-26 09:11:16 -08:00
Cyrus Leung 1b583cfefa
[Doc] Fix typos in docs (#10636) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-25 10:15:45 -08:00
Maximilien de Bayser 214efc2c3c
Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Michael Goin 9afa014552
Add small example to metrics.rst (#10550) 2024-11-21 23:43:43 +00:00
Li, Jiang 63f1fde277
[Hardware][CPU] Support chunked-prefill and prefix-caching on CPU (#10355) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2024-11-20 10:57:39 +00:00
Cyrus Leung b4be5a8adb
[Bugfix] Enforce no chunked prefill for embedding models (#10470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-20 05:12:51 +00:00
Cyrus Leung 32e46e000f
[Frontend] Automatic detection of chat content format from AST (#9919) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-16 13:35:40 +08:00
Mike Depinet f67ce05d0b
[Frontend] Pythonic tool parser (#9859) 
Signed-off-by: Mike Depinet <mike@fixie.ai>
 2024-11-14 04:14:34 +00:00
Guillaume Calmettes 36c513a076
[BugFix] Do not raise a `ValueError` when `tool_choice` is set to the supported `none` option and `tools` are not defined. (#10000) 
Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
 2024-11-12 11:13:46 +00:00
Yuan Tang 4800339c62
Add docs on serving with Llama Stack (#10183) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2024-11-11 11:28:55 -08:00
cjackal d88bff1b96
[Frontend] add `add_request_id` middleware (#9594) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2024-11-09 10:18:29 +00:00
Maximilien de Bayser ae62fd17c0
[Frontend] Tool calling parser for Granite 3.0 models (#9027) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-11-07 07:09:02 -08:00
Cyrus Leung db7db4aab9
[Misc] Consolidate ModelConfig code related to HF config (#10104) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-07 06:00:21 +00:00
Cyrus Leung 06386a64dd
[Frontend] Chat-based Embeddings API (#9759) 2024-11-01 08:13:35 +00:00
Joe Runde 031a7995f3
[Bugfix][Frontend] Reject guided decoding in multistep mode (#9892) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-11-01 01:09:46 +00:00
Guillaume Calmettes abbfb6134d
[Misc][OpenAI] deprecate max_tokens in favor of new max_completion_tokens field for chat completion endpoint (#9837) 2024-10-30 18:15:56 -07:00
youkaichao c2cd1a2142
[doc] update pp support (#9853) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-10-30 13:36:51 -07:00
Joe Runde 33d257735f
[Doc] link bug for multistep guided decoding (#9843) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-10-30 17:28:29 +00:00
Will Eaton 882a1ad0de
[Model] tool calling support for ibm-granite/granite-20b-functioncalling (#8339) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com>
 2024-10-29 15:07:37 -07:00
Vinay R Damodaran 33bab41060
[Bugfix]: Make chat content text allow type content (#9358) 
Signed-off-by: Vinay Damodaran <vrdn@hey.com>
 2024-10-24 05:05:49 +00:00
Seth Kimmel 208cb34c81
[Doc]: Update tensorizer docs to include vllm[tensorizer] (#7889) 
Co-authored-by: Kaunil Dhruv <dhruv.kaunil@gmail.com>
 2024-10-22 15:43:25 -07:00
Yuan 32a1ee74a0
[Hardware][Intel CPU][DOC] Update docs for CPU backend (#6212) 
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Gubrud, Aaron D <aaron.d.gubrud@intel.com>
Co-authored-by: adgubrud <96072084+adgubrud@users.noreply.github.com>
 2024-10-22 10:38:04 -07:00
tomeras91 d2b1bf55ec
[Frontend][Feature] Add jamba tool parser (#9154) 2024-10-18 10:27:48 +00:00
Wallas Henrique 8baf85e4e9
[Doc] Compatibility matrix for mutual exclusive features (#8512) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2024-10-11 11:18:50 -07:00
Yuan Tang acce7630c1
Update link to KServe deployment guide (#9173) 2024-10-09 03:58:49 +00:00
TimWang 93cf74a8a7
[Doc]: Add deploying_with_k8s guide (#8451) 2024-10-07 13:31:45 -07:00
Andy Dai 5df1834895
[Bugfix] Fix order of arguments matters in config.yaml (#8960) 2024-10-05 17:35:11 +00:00
代君 3dbb215b38
[Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405) 2024-10-04 10:36:39 +08:00
Maximilien de Bayser 344cd2b6f4
[Feature] Add support for Llama 3.1 and 3.2 tool use (#8343) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-09-26 17:01:42 -07:00
sroy745 2febcf2777
[Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (#7962) 2024-09-05 16:25:29 -04:00
Kyle Mistele e02ce498be
[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
Kaunil Dhruv 058344f89a
[Frontend]-config-cli-args (#7737) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Kaunil Dhruv <kaunil_dhruv@intuit.com>
 2024-08-30 08:21:02 -07:00
Kameshwara Pavan Kumar Mantha 22b39e11f2
llama_index serving integration documentation (#6973) 
Co-authored-by: pavanmantha <pavan.mantha@thevaslabs.io>
 2024-08-14 15:38:37 -07:00
Murali Andoorveedu fc912e0886
[Models] Support Qwen model with PP (#6974) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
 2024-08-01 12:40:43 -07:00
Zhanghao Wu 150a1ffbfd
[Doc] Update SkyPilot doc for wrong indents and instructions for update service (#4283) 2024-07-26 14:39:10 -07:00
youkaichao f3ff63c3f4
[doc][distributed] improve multinode serving doc (#6804) 2024-07-25 15:38:32 -07:00
youkaichao 71950af726
[doc][distributed] fix doc argument order (#6691) 2024-07-23 08:55:33 -07:00
youkaichao c051bfe4eb
[doc][distributed] doc for setting up multi-node environment (#6529) 
[doc][distributed] add more doc for setting up multi-node environment (#6529)
 2024-07-22 21:22:09 -07:00
Murali Andoorveedu 45ceb85a0c
[Docs] Update PP docs (#6598) 2024-07-19 16:38:21 -07:00
milo157 a38524f338
[DOC] - Add docker image to Cerebrium Integration (#6510) 2024-07-17 10:22:53 -07:00
Cyrus Leung 5bf35a91e4
[Doc][CI/Build] Update docs and tests to use `vllm serve` (#6431) 2024-07-17 07:43:21 +00:00
youkaichao 94b82e8c18
[doc][distributed] add suggestion for distributed inference (#6418) 2024-07-15 09:45:51 -07:00
Ethan Xu dbfe254eda
[Feature] vLLM CLI (#5090) 
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-07-14 15:36:43 -07:00
Murali Andoorveedu 673dd4cae9
[Docs] Docs update for Pipeline Parallel (#6222) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-07-09 16:24:58 -07:00
Roger Wang 8e0817c262
[Bugfix][Doc] Fix Doc Formatting (#6048) 2024-07-01 15:09:11 -07:00
ning.zhang 83bdcb6ac3
add FAQ doc under 'serving' (#5946) 2024-07-01 14:11:36 -07:00
youkaichao 4050d646e5
[doc][misc] remove deprecated api server in doc (#6037) 2024-07-01 12:52:43 -04:00
youkaichao 3fd02bda51
[doc][misc] add note for Kubernetes users (#5916) 2024-06-27 10:07:07 -07:00
youkaichao 294104c3f9
[doc] update usage of env var to avoid conflict (#5873) 2024-06-26 17:57:12 -04:00
youkaichao c246212952
[doc][faq] add warning to download models for every nodes (#5783) 2024-06-24 15:37:42 +08:00
Rafael Vasquez e83db9e7e3
[Doc] Update docker references (#5614) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-06-19 15:01:45 -07:00
milo157 2bd231a7b7
[Doc] Added cerebrium as Integration option (#5553) 2024-06-18 15:56:59 -07:00
Sanger Steel 6e2527a7cb
[Doc] Update documentation on Tensorizer (#5471) 2024-06-14 11:27:57 -07:00
Nick Hill 99dac099ab
[Core][Doc] Default to multiprocessing for single-node distributed case (#5230) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2024-06-11 11:10:41 -07:00
Roger Wang 7a9cb294ae
[Frontend] Add OpenAI Vision API Support (#5237) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-06-07 11:23:32 -07:00
Breno Faria f775a07e30
[FRONTEND] OpenAI `tools` support named functions (#5032) 2024-06-03 18:25:29 -05:00
Chansung Park 429d89720e
add doc about serving option on dstack (#3074) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-30 10:11:07 -07:00
youkaichao 4fbcb0f27e
[Doc][Build] update after removing vllm-nccl (#5103) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-05-29 23:51:18 +00:00
Cyrus Leung 5ae5ed1e60
[Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Kante Yin 8e7fb5d43a
Support to serve vLLM on Kubernetes with LWS (#4829) 
Signed-off-by: kerthcet <kerthcet@gmail.com>
 2024-05-16 16:37:29 -07:00
Cyrus Leung 4bfa7e7f75
[Doc] Add API reference for offline inference (#4710) 2024-05-13 17:47:42 -07:00
Cyrus Leung a3c124570a
[Bugfix] Fix CLI arguments in OpenAI server docs (#4709) 2024-05-09 09:53:14 -07:00
youkaichao 2d7bce9cd5
[Doc] add env vars to the doc (#4572) 2024-05-03 05:13:49 +00:00
Frαnçois e491c7e053
[Doc] update(example model): for OpenAI compatible serving (#4503) 2024-05-01 10:14:16 -07:00
youkaichao 2768884ac4
[Doc] Add note for docker user (#4340) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-24 21:09:44 +00:00
Zhanghao Wu ceaf4ed003
[Doc] Update the SkyPilot doc with serving and Llama-3 (#4276) 2024-04-22 15:34:31 -07:00
Frαnçois 92cd2e2f21
[Doc] Fix getting stared to use publicly available model (#3963) 2024-04-10 18:05:52 +00:00
yhu422 d8658c8cc1
Usage Stats Collection (#2852) 2024-03-28 22:16:12 -07:00
Simon Mo ef65dcfa6f
[Doc] Add docs about OpenAI compatible server (#3288) 2024-03-18 22:05:34 -07:00
Sherlock Xu b0925b3878
docs: Add BentoML deployment doc (#3336) 
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
 2024-03-12 10:34:30 -07:00
Yuan Tang 49d849b3ab
docs: Add tutorial on deploying vLLM model with KServe (#2586) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
 2024-03-01 11:04:14 -08:00
Massimiliano Pronesti 5ed704ec8c
docs: fix langchain (#2736) 2024-02-03 18:17:55 -08:00
Ronen Schaffer c17daa9f89
[Docs] Fix broken links (#2222) 2023-12-20 12:43:42 -08:00
Ikko Eltociear Ashimine c0ce15dfb2
Update run_on_sky.rst (#2025) 
sharable -> shareable
 2023-12-11 10:32:58 -08:00
Simon Mo c85b80c2b6
[Docker] Add cuda arch list as build option (#1950) 2023-12-08 09:53:47 -08:00
AguirreNicolas 24f60a54f4
[Docker] Adding number of nvcc_threads during build as envar (#1893) 2023-12-07 11:00:32 -08:00