vLLM
/
vllm
mirror of https://github.com/vllm-project/vllm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,421 Commits 73 Branches 67 Tags 518 MiB
Commit Graph

80 Commits

Author SHA1 Message Date
Brayden Zhong 5aa4a015ce
[Benchmark] Fix `Value of type "SampleRequest" is not indexable` (#18032) 
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-06-19 21:28:55 -07:00
Reid 6fa718a460
[Misc] Modularize CLI Argument Parsing in Benchmark Scripts (#19593) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
 2025-06-14 16:54:52 +08:00
Simon Mo 02f0c7b220
[Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Harry Mellor 009d9e7590
Convert `benchmarks` to `ruff format` (#18068) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-13 13:43:29 +00:00
Brayden Zhong 891b9d33de
[Fix] Benchmark `"EngineClient" has no attribute "model_config"` (#17976) 
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-11 22:55:53 -07:00
Kartik Ramesh 3b34fd5273
Raise error for data-parallel with benchmark_throughput (#16737) 
Signed-off-by: Kartik Ramesh <kartikx2000@gmail.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2025-04-21 23:51:43 +08:00
Jennifer Zhao 63d2705edb
[Benchmark][Bugfix] Fix SonnetDataset default values in benchmark_throughput.py (#16556) 2025-04-13 17:20:26 -07:00
WWW 268c325078
Fix range_ratio Bug in RandomDataset (#16126) 
Signed-off-by: jadewang21 <jadewangcn@outlook.com>
 2025-04-10 15:31:17 -07:00
Michael Goin b2ce859bd2
Fix `benchmark_throughput.py --backend=hf` (#16352) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-09 19:09:28 +00:00
Ziji Shi (Steven) 95862f7b4d
[Benchmark][Doc] Update throughput benchmark and README (#15998) 
Signed-off-by: StevenShi-23 <shi.ziji.sm@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-04-04 09:39:02 -07:00
Jennifer Zhao effc5d24fa
[Benchmark] Update Vision Arena Dataset and HuggingFaceDataset Setup (#15748) 
Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
 2025-03-31 15:38:58 +08:00
Chen Xia e7f720ea56
[Misc]add coding benchmark for speculative decoding (#15303) 
Signed-off-by: CXIAAAAA <cxia0209@gmail.com>
 2025-03-28 10:47:05 +08:00
Jennifer Zhao a6e0d096dd
[Feature] Add visionarena offline support for benchmark_throughput (#14654) 
Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>
Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>
Co-authored-by: Jennifer Zhao <JenZhao@users.noreply.github.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2025-03-14 04:07:54 +00:00
Jennifer Zhao 1253b15774
[Feature] Consolidate performance benchmark datasets (#14036) 
Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-03-10 07:23:11 +00:00
Jeremy Arnold 58abe35455
[Benchmarks] Make detokenization optional in benchmark scripts (#11697) 
Signed-off-by: Jeremy Arnold <Jeremy.Arnold@amd.com>
 2025-03-07 08:09:00 -08:00
Aleksandr Malyshev 0ca3b8e01c
[BUGFIX] Skip tokenization support for throughput benchmark (#12712) 
Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu>
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
 2025-03-07 02:51:47 -08:00
Harry Mellor cf069aa8aa
Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Huy Do e7ef74e26e
Fix some issues with benchmark data output (#13641) 
Signed-off-by: Huy Do <huydhn@gmail.com>
 2025-02-24 10:23:18 +08:00
Robin 8aca27fa11
[Bugfix] Fix benchmark script bug: inaccurate stats for vllm backend when max_model_len < input_len + output_len (#13691) 
Signed-off-by: WangErXiao <863579016@qq.com>
 2025-02-22 14:10:38 +08:00
Huy Do 45186834a0
Run v1 benchmark and integrate with PyTorch OSS benchmark database (#13068) 
Signed-off-by: Huy Do <huydhn@gmail.com>
 2025-02-17 08:16:32 +00:00
Russell Bryant e489ad7a21
[Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Varun Sundar Rabindranath 98356735ac
[misc] benchmark_throughput : Add LoRA (#11267) 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2024-12-19 15:43:16 +08:00
Michael Goin 4433195ab7
[Bugfix] Prevent benchmark_throughput.py from using duplicated random prompts (#10753) 2024-12-03 02:26:15 +00:00
lkchen d2e80332a7
[Feature] Update benchmark_throughput.py to support image input (#9851) 
Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
Co-authored-by: Linkun Chen <github+anyscale@lkchen.net>
 2024-11-05 19:30:02 +00:00
lkchen 9a5664d4a4
[Misc] Refactor benchmark_throughput.py (#9779) 
Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
Co-authored-by: Linkun Chen <lkchen@github.com>
Co-authored-by: Linkun Chen <github+anyscale@lkchen.net>
 2024-11-04 14:32:16 -08:00
Michael Goin fd0e2cfdb2
[Misc] Separate total and output tokens in benchmark_throughput.py (#8914) 2024-10-23 16:47:20 +00:00
Chen Zhang 65050a40e6
[Bugfix] Generate exactly input_len tokens in benchmark_throughput (#9592) 2024-10-22 17:45:35 -07:00
Jeremy Arnold cb6fdaa0a0
[Misc] Make benchmarks use EngineArgs (#9529) 2024-10-22 15:40:38 -07:00
Kuntai Du 81ede99ca4
[Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
sroy745 f3a507f1d3
[Core] Add an environment variable which needs to be set explicitly to allow BlockSpaceManagerV1 (#9149) 2024-10-10 14:17:17 +08:00
youkaichao 18b296fdb2
[core] remove beam search from the core (#9105) 2024-10-07 05:47:04 +00:00
Brendan Wong 168cab6bbf
[Frontend] API support for beam search (#9087) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-10-05 23:39:03 -07:00
youkaichao 0250dd68c5
re-implement beam search on top of vllm core (#8726) 
Co-authored-by: Brendan Wong <bjwpokemon@gmail.com>
 2024-09-23 22:08:12 -07:00
Kunshang Ji 855c8ae2c9
[MISC] remove engine_use_ray in benchmark_throughput.py (#8615) 2024-09-18 22:33:20 -07:00
Aarni Koskela 8baa454937
[Misc] Move device options to a single place (#8322) 2024-09-11 13:25:58 -07:00
Nick Hill d4db9f53c8
[Benchmark] Add `--async-engine` option to benchmark_throughput.py (#7964) 2024-09-03 20:57:41 -04:00
Megha Agarwal 2eedede875
[Core] Asynchronous Output Processor (#7049) 
Co-authored-by: Alexander Matveev <alexm@neuralmagic.com>
 2024-08-26 20:53:20 -07:00
Alexander Matveev 9db93de20c
[Core] Add multi-step support to LLMEngine (#7789) 2024-08-23 12:45:53 -07:00
Ilya Lavrenov 57f09a419c
[Hardware][Intel] OpenVINO vLLM backend (#5379) 2024-06-28 13:50:16 +00:00
Michael Goin 8065a7e220
[Frontend] Add FlexibleArgumentParser to support both underscore and dash in names (#5718) 2024-06-20 17:00:13 -06:00
Kunshang Ji 728c4c8a06
[Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (#3814) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>
 2024-06-17 11:01:25 -07:00
Cyrus Leung 0e9164b40a
[mypy] Enable type checking for test directory (#5017) 2024-06-15 04:45:31 +00:00
Kuntai Du 319ad7f1d3
[CI/Build][Misc] Add CI that benchmarks vllm performance on those PRs with `perf-benchmarks` label (#5073) 
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-06-13 22:36:20 -07:00
Woosuk Kwon 1a8bfd92d5
[Hardware] Initial TPU integration (#5292) 2024-06-12 11:53:03 -07:00
Benjamin Kitor b3376e5c76
[Misc] Add args for selecting distributed executor to benchmarks (#5335) 2024-06-08 09:20:16 +08:00
Cody Yu a3a73ab069
[Misc] Load FP8 kv-cache scaling factors from checkpoints (#4893) 
The 2nd PR for #4532.

This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).
 2024-05-22 13:28:20 -07:00
Simon Mo f09edd8a25
Add JSON output support for benchmark_latency and benchmark_throughput (#4848) 2024-05-16 10:02:56 -07:00
zifeitong a395a638c2
[Misc] Use public API in benchmark_throughput (#4300) 2024-04-24 21:10:24 +00:00
Michael Goin 53b018edcb
[Bugfix] Get available quantization methods from quantization registry (#4098) 2024-04-18 00:21:55 -07:00
SangBin Cho 67b4221a61
[Core][5/N] Fully working chunked prefill e2e (#3884) 2024-04-10 17:56:48 -07:00