vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,204 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

466 Commits

Author SHA1 Message Date
Cyrus Leung 7a64d24aad
[Core] Support image processor (#4197) 2024-06-02 22:56:41 -07:00
Daniil Arapov c2d6d2f960
[Bugfix]: Fix issues related to prefix caching example (#5177) (#5180) 2024-06-01 15:53:52 -07:00
chenqianfzh b9c0605a8e
[Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776) 2024-06-01 14:51:10 -06:00
Ronen Schaffer ae495c74ea
[Doc]Replace deprecated flag in readme (#4526) 2024-05-29 22:26:33 +00:00
Cyrus Leung 5ae5ed1e60
[Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Antoni Baum c5711ef985
[Doc] Update Ray Data distributed offline inference example (#4871) 2024-05-17 10:52:11 -07:00
Alex Wu dbc0754ddf
[docs] Fix typo in examples filename openi -> openai (#4864) 2024-05-17 00:42:17 +09:00
Aurick Qiao 30e754390c
[Core] Implement sharded state loader (#4690) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-05-15 22:11:54 -07:00
Alex Wu 52f8107cf2
[Frontend] Support OpenAI batch file format (#4794) 
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-05-15 19:13:36 -04:00
Sanger Steel 8bc68e198c
[Frontend] [Core] perf: Automatically detect vLLM-tensorized model, update `tensorizer` to version 2.9.0 (#4208) 2024-05-13 14:57:07 -07:00
Chang Su e254497b66
[Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) 2024-05-11 11:30:37 -07:00
Hao Zhang ebce310b74
[Model] Snowflake arctic model implementation (#4652) 
Co-authored-by: Dash Desai <1723932+iamontheinet@users.noreply.github.com>
Co-authored-by: Aurick Qiao <qiao@aurick.net>
Co-authored-by: Aurick Qiao <aurick.qiao@snowflake.com>
Co-authored-by: Aurick Qiao <aurickq@users.noreply.github.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-05-09 22:37:14 +00:00
Robert Shaw cea64430f6
[Bugfix] Update grafana.json (#4711) 2024-05-09 10:10:13 -07:00
Danny Guinther b8afa8b95a
[MISC] Rework logger to enable pythonic custom logging configuration to be provided (#4273) 2024-05-01 17:34:40 -07:00
Ronen Schaffer bf480c5302
Add more Prometheus metrics (#2764) 
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
 2024-04-28 15:59:33 -07:00
James Fleming 2b7949c1c2
AQLM CUDA support (#3287) 
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-04-23 13:59:33 -04:00
Harry Mellor 3d925165f2
Add example scripts to documentation (#4225) 
Co-authored-by: Harry Mellor <hmellor@oxts.com>
 2024-04-22 16:36:54 +00:00
Antoni Baum 69e1d2fb69
[Core] Refactor model loading code (#4097) 2024-04-16 11:34:39 -07:00
Sanger Steel d619ae2d19
[Doc] Add better clarity for tensorizer usage (#4090) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-04-15 13:28:25 -07:00
Sanger Steel 711a000255
[Frontend] [Core] feat: Add model loading using `tensorizer` (#3476) 2024-04-13 17:13:01 -07:00
Cade Daniel e0dd4d3589
[Misc] Fix linter issues in examples/fp8/quantizer/quantize.py (#3864) 2024-04-04 21:57:33 -07:00
Adrian Abeyta 2ff767b513
Enable scaled FP8 (e4m3fn) KV cache on ROCm (AMD GPU) (#3290) 
Co-authored-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: AdrianAbeyta <Adrian.Abeyta@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: root <root@gt-pla-u18-08.pla.dcgpu>
Co-authored-by: mawong-amd <156021403+mawong-amd@users.noreply.github.com>
Co-authored-by: ttbachyinsda <ttbachyinsda@outlook.com>
Co-authored-by: guofangze <guofangze@kuaishou.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: jacobthebanana <50071502+jacobthebanana@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-04-03 14:15:55 -07:00
Woosuk Kwon c0935c96d3
[Bugfix] Set enable_prefix_caching=True in prefix caching example (#3703) 2024-03-28 16:26:30 -07:00
Simon Mo a4075cba4d
[CI] Add test case to run examples scripts (#3638) 2024-03-28 14:36:10 -07:00
xwjiang2010 64172a976c
[Feature] Add vision language model support. (#3042) 2024-03-25 14:16:30 -07:00
SangBin Cho 01bfb22b41
[CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
Zhuohan Li e90fc21f2e
[Hardware][Neuron] Refactor neuron support (#3471) 2024-03-22 01:22:17 +00:00
Simon Mo 8e67598aa6
[Misc] fix line length for entire codebase (#3444) 2024-03-16 00:36:29 -07:00
Dinghow Yang cf6ff18246
Fix Baichuan chat template (#3340) 2024-03-15 21:02:12 -07:00
Dinghow Yang 253a98078a
Add chat templates for ChatGLM (#3418) 2024-03-14 23:19:22 -07:00
Dinghow Yang 21539e6856
Add chat templates for Falcon (#3420) 2024-03-14 23:19:02 -07:00
Allen.Dou a37415c31b
allow user to chose which vllm's merics to display in grafana (#3393) 2024-03-14 06:35:13 +00:00
DAIZHENWEI 654865e21d
Support Mistral Model Inference with transformers-neuronx (#3153) 2024-03-11 13:19:51 -07:00
Sage Moore ce4f5a29fb
Add Automatic Prefix Caching (#2762) 
Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-03-02 00:50:01 -08:00
Liangfu Chen 3b7178cfa4
[Neuron] Support inference with transformers-neuronx (#2569) 2024-02-28 09:34:34 -08:00
jvmncs 8f36444c4f
multi-LoRA as extra models in OpenAI server (#2775) 
how to serve the loras (mimicking the [multilora inference example](https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py)):
```terminal
$ export LORA_PATH=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
$ python -m vllm.entrypoints.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --enable-lora \
 --lora-modules sql-lora=$LORA_PATH sql-lora2=$LORA_PATH
```
the above server will list 3 separate values if the user queries `/models`: one for the base served model, and one each for the specified lora modules. in this case sql-lora and sql-lora2 point to the same underlying lora, but this need not be the case. lora config values take the same values they do in EngineArgs

no work has been done here to scope client permissions to specific models
 2024-02-17 12:00:48 -08:00
Cheng Su 4abf6336ec
Add one example to run batch inference distributed on Ray (#2696) 2024-02-02 15:41:42 -08:00
Robert Shaw 93b38bea5d
Refactor Prometheus and Add Request Level Metrics (#2316) 2024-01-31 14:58:07 -08:00
Simon Mo 1e4277d2d1
lint: format all python file instead of just source code (#2567) 2024-01-23 15:53:06 -08:00
Antoni Baum 9b945daaf1
[Experimental] Add multi-LoRA support (#1804) 
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
 2024-01-23 15:26:37 -08:00
Jason Zhu 5d80a9178b
Minor fix in prefill cache example (#2494) 2024-01-18 09:40:34 -08:00
shiyi.c_98 d10f8e1d43
[Experimental] Prefix Caching Support (#1669) 
Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-17 16:32:10 -08:00
arkohut 97460585d9
Add gradio chatbot for openai webserver (#2307) 2024-01-11 19:45:56 -08:00
KKY 74cd5abdd1
Add baichuan chat template jinjia file (#2390) 2024-01-09 09:13:02 -08:00
Ronen Schaffer 1066cbd152
Remove deprecated parameter: concurrency_count (#2315) 2024-01-03 09:56:21 -08:00
Massimiliano Pronesti c07a442854
chore(examples-docs): upgrade to OpenAI V1 (#1785) 2023-12-03 01:11:22 -08:00
Adam Brusselback 66785cc05c
Support chat template and `echo` for chat API (#1756) 2023-11-30 16:43:13 -08:00
iongpt ac8d36f3e5
Refactor LLMEngine demo script for clarity and modularity (#1413) 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2023-10-30 09:14:37 -07:00
Zhuohan Li 9d9072a069
Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Yunfeng Bai 09ff7f106a
API server support ipv4 / ipv6 dualstack (#1288) 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2023-10-07 15:15:54 -07:00
Woosuk Kwon 55fe8a81ec
Refactor scheduler (#658) 2023-08-02 16:42:01 -07:00
Zhuohan Li 1b0bd0fe8a
Add Falcon support (new) (#592) 2023-08-02 14:04:39 -07:00
Zhuohan Li 82ad323dee
[Fix] Add chat completion Example and simplify dependencies (#576) 2023-07-25 23:45:48 -07:00
Zhuohan Li d6fa1be3a8
[Quality] Add code formatter and linter (#326) 2023-07-03 11:31:55 -07:00
Woosuk Kwon 14f0b39cda
[Bugfix] Fix a bug in RequestOutput.finished (#202) 2023-06-22 00:17:24 -07:00
Woosuk Kwon 0b98ba15c7
Change the name to vLLM (#150) 2023-06-17 03:07:40 -07:00
Zhuohan Li e5464ee484
Rename servers to engines (#152) 2023-06-17 17:25:21 +08:00
Zhuohan Li eedb46bf03
Rename servers and change port numbers to reduce confusion (#149) 2023-06-17 00:13:02 +08:00
Woosuk Kwon 311490a720
Add script for benchmarking serving throughput (#145) 2023-06-14 19:55:38 -07:00
Zhuohan Li 5020e1e80c
Non-streaming simple fastapi server (#144) 2023-06-10 10:43:07 -07:00
Zhuohan Li 4298374265
Add docstrings for LLMServer and related classes and examples (#142) 2023-06-07 18:25:20 +08:00
Woosuk Kwon 211318d44a
Add throughput benchmarking script (#133) 2023-05-28 03:20:05 -07:00
Zhuohan Li 057daef778
OpenAI Compatible Frontend (#116) 2023-05-23 21:39:50 -07:00
Woosuk Kwon 655a5e48df
Introduce LLM class for offline inference (#115) 2023-05-21 17:04:18 -07:00
Woosuk Kwon f746ced08d
Implement stop strings and best_of (#114) 2023-05-21 11:18:00 -07:00
Woosuk Kwon c3442c1f6f
Refactor system architecture (#109) 2023-05-20 13:06:59 -07:00