Commit Graph

74 Commits

Author SHA1 Message Date
JmPotato 7793c676c4
feat(resourcecontrol): refine request bypass logic for internal analyze and system requests (#1772)
ref tidbcloud/cloud-storage-engine#3703

Signed-off-by: JmPotato <github@ipotato.me>

Co-authored-by: glorv <glorvs@163.com>
2025-10-27 06:44:10 +00:00
tangenta 837c06ce52
apicodec: do not decode empty key for codec v2 (#1734)
Signed-off-by: tangenta <tangenta@126.com>
2025-08-15 09:26:06 +00:00
zyguan 3761f7e254
util/async: make TestExecConcurrent stable (#1683)
close tikv/client-go#1682

Signed-off-by: zyguan <zhongyangguan@gmail.com>
2025-07-09 12:18:54 +00:00
lhy1024 e84f1a780f
update pd client for resource group and keyspace (#1645)
Signed-off-by: lhy1024 <admin@liudos.us>
2025-05-28 09:09:49 +00:00
tangenta e53096d518
support redact key in logs (#1612)
ref pingcap/tidb#59279

Signed-off-by: tangenta <tangenta@126.com>

Co-authored-by: you06 <you1474600@gmail.com>
2025-04-01 07:17:12 +00:00
Neil Shen 0c0c1d1c92
execdetails: export grpc process and wait time to time details (#1614)
Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: Bisheng Huang <hbisheng@gmail.com>
2025-03-25 11:05:43 +00:00
Neil Shen 0135225dd9
execdetails: export scheduler write details (#1606)
Signed-off-by: Neil Shen <overvenus@gmail.com>
2025-03-14 08:22:22 +00:00
okJiang aa7301d4bc
pdclient: Add caller info to pd client (#1516)
ref tikv/pd#8593

Signed-off-by: okJiang <819421878@qq.com>
2025-03-04 01:42:09 +00:00
zyguan 5ac118bbc6
util: define and implement core interfaces for async api (#1591)
ref tikv/client-go#1586

Signed-off-by: zyguan <zhongyangguan@gmail.com>
2025-02-27 04:05:35 +00:00
crazycs f2266d6bf2
*: Optimize executor runtime stats performance (#1532)
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2024-12-25 04:06:45 +00:00
JmPotato ff126df68a
Upgrade the PD client to adopt the latest refactor (#1525)
Signed-off-by: JmPotato <ghzpotato@gmail.com>
2024-12-20 05:23:27 +00:00
ShuNing 0e4728cfaa
*: add the traffic statistic (#1505)
ref pingcap/tidb#57543

Signed-off-by: nolouch <nolouch@gmail.com>
2024-12-16 06:58:54 +00:00
you06 14f3d2b5b7
region cache: add `BatchLocateKeyRanges` interface (#1367)
* add batch scan interface

Signed-off-by: you06 <you1474600@gmail.com>

fix bug

Signed-off-by: you06 <you1474600@gmail.com>

copy encoded ranges

Signed-off-by: you06 <you1474600@gmail.com>

fix panic

Signed-off-by: you06 <you1474600@gmail.com>

add batch scan region test

Signed-off-by: you06 <you1474600@gmail.com>

fix log

Signed-off-by: you06 <you1474600@gmail.com>

remove need-leader option

Signed-off-by: you06 <you1474600@gmail.com>

remove need-leader optino

Signed-off-by: you06 <you1474600@gmail.com>

add comment

Signed-off-by: you06 <you1474600@gmail.com>

deprecate ScanRegions

Signed-off-by: you06 <you1474600@gmail.com>

update kvproto & pd client

Signed-off-by: you06 <you1474600@gmail.com>

update go.mod

Signed-off-by: you06 <you1474600@gmail.com>

fix need leader

Signed-off-by: you06 <you1474600@gmail.com>

fix integration test

Signed-off-by: you06 <you1474600@gmail.com>

fix lint

Signed-off-by: you06 <you1474600@gmail.com>

export BatchLocateKeyRangesOpt

Signed-off-by: you06 <you1474600@gmail.com>

update integration test's tidb

Signed-off-by: you06 <you1474600@gmail.com>

refactor BatchLocateKeyRanges and add comments

Signed-off-by: you06 <you1474600@gmail.com>

* update tidb of integration test

Signed-off-by: you06 <you1474600@gmail.com>

* remove key in error since redact is not supported

Signed-off-by: you06 <you1474600@gmail.com>

* remove key in error since redact is not supported

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-06-24 16:06:06 +08:00
crazycs 5a4905d2f5
refine timedetail string (#1261)
* refine timedetail string

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

---------

Signed-off-by: crazycs520 <crazycs520@gmail.com>
2024-04-03 13:22:40 +08:00
cfzjywxk c2927c0ec6
cop: fix time detail merge (#1258)
* time detail merge

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* fix test

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

---------

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-04-03 12:05:38 +08:00
cfzjywxk 603dc7b2e7
cop: add kv read wall time to print result (#1248)
* add kv read wall time to print result

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* format

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

---------

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-03-27 10:30:46 +08:00
ekexium 87a984a72d
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB (#1212)
* feat: introduce GetLocal() for MemBuffer

Signed-off-by: ekexium <eke@fastmail.com>

* downgrade tools to 0.18.0

Signed-off-by: ekexium <eke@fastmail.com>

* fix: set flushingMemDB = nil when an error is returned from a flush

Signed-off-by: ekexium <eke@fastmail.com>

* impl BatchGet for MemBuffer

Signed-off-by: you06 <you1474600@gmail.com>

test membuffer batch get

Signed-off-by: you06 <you1474600@gmail.com>

* add Prefetch & GetPrefetchCache

Signed-off-by: you06 <you1474600@gmail.com>

* cache multi Prefetch call

Signed-off-by: you06 <you1474600@gmail.com>

* add tests

Signed-off-by: you06 <you1474600@gmail.com>

* workaround for golang ci lint failure

Signed-off-by: you06 <you1474600@gmail.com>

* replace assert with require

Signed-off-by: you06 <you1474600@gmail.com>

* workaround golangci lint

Signed-off-by: you06 <you1474600@gmail.com>

* lint

Signed-off-by: you06 <you1474600@gmail.com>

* remove prefetch interface, pipelined memdb will cache batch get result

Signed-off-by: you06 <you1474600@gmail.com>

* update tidb

Signed-off-by: you06 <you1474600@gmail.com>

* fix batch get cache when membuffer is empty

Signed-off-by: you06 <you1474600@gmail.com>

* fix returned delete value

Signed-off-by: you06 <you1474600@gmail.com>

* fix: handle resourceGroupTag of Flush

Signed-off-by: ekexium <eke@fastmail.com>

* fix: set resource group tag for committer if it's pipelined

Signed-off-by: ekexium <eke@fastmail.com>

* Update internal/unionstore/pipelined_memdb.go

Co-authored-by: ekexium <eke@fastmail.com>

* remove prefetch interface

Signed-off-by: you06 <you1474600@gmail.com>

* release mutex

Signed-off-by: you06 <you1474600@gmail.com>

* flush wait to avoid race

Signed-off-by: you06 <you1474600@gmail.com>

* fix unstopped test

Signed-off-by: you06 <you1474600@gmail.com>

* set resource group tags for committer

Signed-off-by: ekexium <eke@fastmail.com>

* skip test due to tikv image not updated yet

Signed-off-by: you06 <you1474600@gmail.com>

* skip more test

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 11:36:05 +08:00
crazycs 8d6a95f73d
replica selector refactor (#1142)
* init

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add ReplicaSelector interface

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add replica_selector_v2, todo: fix test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix all test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test in another way to compatible with old version about stale-read request

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* tiny refactor

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refactor to remove duplicate code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine onServerIsBusy region error handing logic

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* support forwarding by proxy

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* support busyThreshold and tiny reractor code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add config

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix proxy bug

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test and tiny refactor

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add some test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* tidy refine

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* address comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test in v1

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add backoff-cnt check and timeout test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add benchmark test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* make test stale

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix golangci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test and refine code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix race test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more comprehensive enumeration testing

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix accessFollower with label retry bug

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test and fix some bug

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test and add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine onNotLeader logic when new leader is not available

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine calculateScore logic, if the replica already tried, decrease the score

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine logic: replica-read request with mixed strategy and with label, should be able to retry all remain replicas

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* prefer-leader for mixed read with label for non stale-read req

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test case

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* resolve conflict

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* remove some duplicate test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refactor test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine code

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add busy_threshold test and fix bug

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix proxy bug and add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine code and add test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add learner test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add more test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* remove old test and refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* move test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* address comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* use new score calculation by flag bit

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* make test stable

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine comment,license and fix lint

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix race test timeout

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add flagNotAttemp in score

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* address comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

---------

Signed-off-by: crazycs520 <crazycs520@gmail.com>
2024-03-11 15:40:44 +08:00
you06 8d28d3cd3a
txn: introduce pipelined memdb, Flush and BufferBatchGet for pipelined DML (#1114)
* Support pipelined memdb.

Signed-off-by: you06 <you1474600@gmail.com>

* fix mutex for pipelined memdb

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-02-23 10:24:44 +08:00
Weizhen Wang fd2fc84032
use slices.Sort to eliminate bounds check (#1128)
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-23 11:24:15 +08:00
glorv 4ce1e45671
add a helper function for new ru details (#1078)
Signed-off-by: glorv <glorvs@163.com>
2023-12-20 10:35:27 +08:00
glorv e80e9ca1fe
resource_control: add ru details in ExecDetails (#1070)
* collecting the RU information by pasing point through context.Value (#1032)

Signed-off-by: zzm <zhouzemin@pingcap.com>

* add ruWaitDuration to RUDetails and update pd-client

Signed-off-by: glorv <glorvs@163.com>

---------

Signed-off-by: zzm <zhouzemin@pingcap.com>
Signed-off-by: glorv <glorvs@163.com>
Co-authored-by: zzm <zhouzemin@pingcap.com>
2023-12-04 15:40:48 +08:00
Yongbo Jiang 0ff16620f6
pd client: uprade and fit get region option (#1069)
* fit get region option

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

* replace tidb

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

* replace tidb

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

---------

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
2023-12-01 10:44:04 +08:00
ekexium c129e1b309
Fix comment of `SuspendTime` (#1057)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-11-14 15:10:49 +08:00
JmPotato 845e3b01a2
Support to integrate with the PD HTTP client (#1049)
Signed-off-by: JmPotato <ghzpotato@gmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-11-09 10:31:01 +08:00
ShuNing c8832b8466
resource_control: add stats task (#985)
Signed-off-by: nolouch <nolouch@gmail.com>
2023-09-21 19:32:00 +08:00
Smilencer 330fc8d843
add util to set dns server (#927)
Signed-off-by: Smityz <smityz@qq.com>
Co-authored-by: disksing <i@disksing.com>
2023-09-18 15:19:39 +08:00
crazycs520 ba3de39920 *: fix panic log
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2023-09-11 14:38:07 +08:00
husharp cedc94b3ee use config instead
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-16 15:59:30 +08:00
husharp c65aaeb8f4 address comment
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-16 11:56:09 +08:00
husharp b771e6317f make ci happy
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-14 14:24:53 +08:00
Hu# d8451eb284
Merge branch 'master' into get_min_resolved_ts_by_stores 2023-08-14 13:24:52 +08:00
husharp 1a932e727e suuport api bench for params
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-14 13:11:16 +08:00
ShuNing 45894d9d36
[pick] resource_control: bypass some internal urgent request (#938)
* resource_control: bypass some internal urgent request (#884)

Signed-off-by: nolouch <nolouch@gmail.com>

* resourcecontrol: fix nil pointer (#900)

Signed-off-by: nolouch <nolouch@gmail.com>

---------

Signed-off-by: nolouch <nolouch@gmail.com>
2023-08-11 03:40:06 -06:00
glorv 300545a8a3
add a function to set request source task type (#925)
* add a function to set request source task type

Signed-off-by: glorv <glorvs@163.com>
2023-08-08 22:03:15 -07:00
husharp 3f36b93a7d fix error
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-09 10:29:01 +08:00
ShuNing a0d916d116
request_source: remove default label (#890)
* request_source: remove default label

Signed-off-by: nolouch <nolouch@gmail.com>
2023-08-08 09:38:22 -07:00
husharp 0930be96a5 remove ci
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-02 18:40:29 +08:00
husharp c5bf330532 expand min-resolved-ts to support stores
Signed-off-by: husharp <jinhao.hu@pingcap.com>
2023-08-02 15:14:15 +08:00
Hu# 76d6d93e98
resource_manager: ignore ru metrics for background request (#872)
Signed-off-by: husharp <jinhao.hu@pingcap.com>
Co-authored-by: disksing <i@disksing.com>
2023-07-26 15:13:54 +08:00
ShuNing 178f6fa01a
add explicit request source type to label the external request like lightning/br (#868)
Signed-off-by: nolouch <nolouch@gmail.com>
2023-07-07 15:02:42 +08:00
Hu# 28247160f4
PDHTTPClient: when meet GetStoreMinResolvedTS error just write in debug log (#825)
Signed-off-by: husharp <jinhao.hu@pingcap.com>
Co-authored-by: disksing <i@disksing.com>
2023-06-05 16:51:12 +08:00
Hu# 1b28766936
PDHTTPClient: when meet GetStoreMinResolvedTS error just write in debug log (#814)
Signed-off-by: husharp <ihusharp@gmail.com>
2023-05-24 11:26:53 +08:00
Hu# adb48afeef
pd http: support api to get store min resolved ts (#793)
Signed-off-by: husharp <ihusharp@gmail.com>
2023-05-12 11:43:16 +08:00
ShuNing dd6caa4c03
execdetails: add RRU/WRU interface of RURuntimeStats (#769)
Signed-off-by: nolouch <nolouch@gmail.com>
2023-04-19 16:57:01 +08:00
JmPotato 9d950905d7
Introduce the RURuntimeStats (#732)
Signed-off-by: JmPotato <ghzpotato@gmail.com>
2023-03-16 10:19:36 +08:00
glorv c9119d02ce
support time detail v2 (#729)
* support time detail v2

Signed-off-by: glorv <glorvs@163.com>

---------

Signed-off-by: glorv <glorvs@163.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
2023-03-13 21:32:19 +08:00
Yongbo Jiang 3f7860f109
region_request: handle resource group error before Backoff (#719)
Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-03-02 13:40:57 +08:00
cfzjywxk 49323956ae
metric: seperate source for txn cmd (#721)
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-02-28 15:40:40 +08:00
Lucas bce56a5831
ReadMode: introduce AutoFollowerRead mechanism by adding a new `ReadMode == PreferLeader` (#671)
Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
2023-02-07 16:42:52 +08:00