Commit Graph

49 Commits

Author SHA1 Message Date
zyguan e85edd4206
client: refactor batch client send loop (#1564)
Signed-off-by: zyguan <zhongyangguan@gmail.com>
2025-01-24 06:43:32 +00:00
zyguan 61e09c6539
metrics: optimize rpc latency related metrics (#1556)
Signed-off-by: zyguan <zhongyangguan@gmail.com>
2025-01-17 03:49:19 +00:00
zyguan 20764920b4
*: reduce overhead of codec client (#1555)
Signed-off-by: zyguan <zhongyangguan@gmail.com>
2025-01-17 03:47:01 +00:00
ShuNing 0e4728cfaa
*: add the traffic statistic (#1505)
ref pingcap/tidb#57543

Signed-off-by: nolouch <nolouch@gmail.com>
2024-12-16 06:58:54 +00:00
crazycs e7894a7b27
config: change grpc-keepalive-timeout from uint to float64, for support millisecond timeout (#1461)
Signed-off-by: crazycs520 <crazycs520@gmail.com>

Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
2024-09-11 04:15:06 +00:00
zyguan 4c6b2171b2
improve batch efficiency for high throughput workloads (#1411)
* improve batch efficiency by multiple attempts

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add support for aggressive batching

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add some metrics for batch client

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* fetch more requests according to recent wait head durs

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add experimental batch options

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* optimize and refactor

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* some minor updates

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* fix the metric of head arrival interval

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* a minor update

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* some minor fixes

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* update according to the spec

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* fix the ut

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* fix batch condition

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add some doc comments

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* address https://github.com/tikv/client-go/pull/1411#discussion_r1714086744

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* rename some vars according to the comments

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add more comments

Signed-off-by: zyguan <zhongyangguan@gmail.com>

---------

Signed-off-by: zyguan <zhongyangguan@gmail.com>
2024-08-13 12:55:44 +08:00
yibin d4f0f4cf12
Revert "upgrade grpc 1.64 (#1346)" (#1369)
This reverts commit 1c06c5cbf5.

Signed-off-by: yibin <huyibin@pingcap.com>
2024-06-21 17:03:19 +08:00
crazycs cb580bc4ea
fix rpc client panic cause by concurrent close (#1359)
* fix rpc client panic cause by concurrent close

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* address comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine

Signed-off-by: crazycs520 <crazycs520@gmail.com>

---------

Signed-off-by: crazycs520 <crazycs520@gmail.com>
2024-05-31 18:21:21 +08:00
MyonKeminta 31a8ddab19
Support actively requesting update health feedback information by calling RPC to TiKV (#1287)
* Support actively requesting update health feedback information by calling RPC to TiKV

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto; follow the protocol change; and add more comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add integration tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* tune the frequency of active updating

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Revert changes to examples

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary empty line

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2024-05-27 14:36:27 +08:00
Weizhen Wang 1c06c5cbf5
upgrade grpc 1.64 (#1346)
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-05-17 12:20:15 +08:00
Liqi Geng 44b2944b64
Add grpc window size config and change the default value
Signed-off-by: gengliqi <gengliqiii@gmail.com>
2024-03-19 23:01:50 +08:00
crazycs 98a7df8f41
fix issue read req timeout bug cause by pr#1223 (#1232)
* fix issue read req timeout bug cause by pr#1223

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* add test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* implement Cause

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix race test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

---------

Signed-off-by: crazycs520 <crazycs520@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 19:58:42 +09:00
Ping Yu 7bce6ea4df
Pick connection handling improvements from tidb-cse-7.5 (#1223)
* close on canceled (#1121)

Signed-off-by: Ping Yu <yuping@pingcap.com>

* Fix wait recvloop timeout (#1134)

* wip

Signed-off-by: Ping Yu <yuping@pingcap.com>

* CloseAddr with ver

Signed-off-by: Ping Yu <yuping@pingcap.com>

* fix ErrConn

Signed-off-by: Ping Yu <yuping@pingcap.com>

* fix ut

Signed-off-by: Ping Yu <yuping@pingcap.com>

* cleanup

Signed-off-by: Ping Yu <yuping@pingcap.com>

---------

Signed-off-by: Ping Yu <yuping@pingcap.com>

* Use correct address for CloseAddr (#1140)

* wip

Signed-off-by: Ping Yu <yuping@pingcap.com>

* CloseAddr with ver

Signed-off-by: Ping Yu <yuping@pingcap.com>

* fix ErrConn

Signed-off-by: Ping Yu <yuping@pingcap.com>

* fix ut

Signed-off-by: Ping Yu <yuping@pingcap.com>

* polish

Signed-off-by: Ping Yu <yuping@pingcap.com>

* polish

Signed-off-by: Ping Yu <yuping@pingcap.com>

* polish

Signed-off-by: Ping Yu <yuping@pingcap.com>

---------

Signed-off-by: Ping Yu <yuping@pingcap.com>

* rebase

Signed-off-by: Ping Yu <yuping@pingcap.com>

---------

Signed-off-by: Ping Yu <yuping@pingcap.com>
Co-authored-by: iosmanthus <dengliming@pingcap.com>
2024-03-14 16:30:36 +08:00
MyonKeminta 03bbadb277
Support receving health feedback (#1153)
* Register health feedback handler in batch client

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* update protocol

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Collect the health status and record to the stores

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove the filter which looks not necessary

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add metrics

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix checkAndUpdateStoreHealthStatus panicking

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add logs

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Merge SetHealthFeedbackHandler to Client

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Try another pattern: separated callback registry

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Try region cache eventListener pattern

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Test receiving froim callback

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests to region cache

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary debug log

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* remove replace of tidb repo

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update comments; fix lint

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add the isSlow method of Store back

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* remove unused method for now to make lint happy

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Address comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2024-02-28 19:10:32 +08:00
tongjian 824302acd0
client: Cache tikv request in tidb client side (#1098)
* impl priority queue

Signed-off-by: bufferflies <1045931706@qq.com>

* replace priority queue

Signed-off-by: bufferflies <1045931706@qq.com>

* cache request in tidb side

Signed-off-by: bufferflies <1045931706@qq.com>

* fix gosimple

Signed-off-by: bufferflies <1045931706@qq.com>

* impl priority

Signed-off-by: bufferflies <1045931706@qq.com>

* pass ut

Signed-off-by: bufferflies <1045931706@qq.com>

* add

Signed-off-by: bufferflies <1045931706@qq.com>

* remove request if the request has been canceled

Signed-off-by: bufferflies <1045931706@qq.com>

* remove request if it has been canceled

Signed-off-by: bufferflies <1045931706@qq.com>

* add comment for cancel

Signed-off-by: bufferflies <1045931706@qq.com>

* not make the loop is busy

Signed-off-by: bufferflies <1045931706@qq.com>

* lint

Signed-off-by: bufferflies <1045931706@qq.com>

* revert busy loop

Signed-off-by: bufferflies <1045931706@qq.com>

* add unit test

Signed-off-by: bufferflies <1045931706@qq.com>

* not limit ehigh prioirty test

Signed-off-by: bufferflies <1045931706@qq.com>

* pass lint

Signed-off-by: bufferflies <1045931706@qq.com>

* support all

Signed-off-by: bufferflies <1045931706@qq.com>

* add comment

Signed-off-by: bufferflies <1045931706@qq.com>

* squash

Signed-off-by: bufferflies <1045931706@qq.com>

* revert all to All

Signed-off-by: bufferflies <1045931706@qq.com>

* remove index from entry

Signed-off-by: bufferflies <1045931706@qq.com>

* make fail reasons more clear

Signed-off-by: bufferflies <1045931706@qq.com>

---------

Signed-off-by: bufferflies <1045931706@qq.com>
2024-02-21 14:54:33 +08:00
Weizhen Wang f90605363e
support grpc's shard buffer pool (#1132)
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-24 17:57:47 +08:00
crazycs be2b4c78a9
Revert "*: fix batch-client wait too long and add some metrics (#973)" (#984)
This reverts commit adb7db13c3.

Signed-off-by: crazycs520 <crazycs520@gmail.com>
2023-09-19 11:15:11 +08:00
crazycs adb7db13c3
*: fix batch-client wait too long and add some metrics (#973)
Signed-off-by: crazycs520 <crazycs520@gmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-09-18 11:39:42 +08:00
glorv c9119d02ce
support time detail v2 (#729)
* support time detail v2

Signed-off-by: glorv <glorvs@163.com>

---------

Signed-off-by: glorv <glorvs@163.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
2023-03-13 21:32:19 +08:00
cfzjywxk bdd41058aa
metrics: seperate metrics with source scope for txn command (#723)
* seperate metrics with source scope

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

* fix async pessimistic rollback

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

* fix missing definition

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

---------

Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-03-06 22:12:45 +08:00
iosmanthus e0e6019f8a
attach keyspace_id to tikv/tiflash requests (#682) 2023-02-20 13:10:03 +08:00
Smilencer 35416c4c43
Add gRPC connection monitor and fix related bug (#665)
Signed-off-by: Smityz <smityz@qq.com>
Co-authored-by: disksing <i@disksing.com>
2023-01-31 16:28:48 +08:00
David f313ddf58d
reformat codec and add keyspace support (#649)
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
2023-01-10 15:15:33 +08:00
Jack Yu ce9203ef66
*: disable PermitWithoutStream to avoid too many pings (#574)
Signed-off-by: Jack Yu <jackysp@gmail.com>
2022-09-21 18:16:51 +08:00
Yilin Chen 2ed2113d10
Fix that histogram is not inserted into rpcNetLatencyHistCache (#563)
* Fix that histogram is not inserted into rpcNetLatencyHistCache

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* downgrade golangci-lint to avoid gofmt complaints temporarily

Signed-off-by: Yilin Chen <sticnarf@gmail.com>
2022-08-09 11:48:08 +08:00
zyguan e10841f2d1
support adding `ExecDetailsV2` to tracing (#559)
* support adding `ExecDetailsV2` to tracing

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* rename `TraceExecEnabled` to `TraceExecDetailsEnabled`

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* revert changes for #558

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* address comments

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* add unit test

Signed-off-by: zyguan <zhongyangguan@gmail.com>
2022-07-29 11:44:04 +08:00
Yilin Chen e38d2b07de
do not format error when response does not include ExecDetailsV2 (#553)
Signed-off-by: Yilin Chen <sticnarf@gmail.com>
2022-07-21 11:16:57 +08:00
Yilin Chen 57c12f7c64
Handle more detailed statistics from TiKV (#536)
* support more detailed statistics returned from TiKV

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix log formatting

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* add to test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>
2022-07-13 16:56:47 +08:00
iosmanthus 68cd1bc6c1
Introduce API v2 (#494)
* introduce raw.Client with api v2

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* migrate rawkv test to api v2

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* git fire: refactoring the interface SplitRegion of CodecClientV2

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* go mod tidy

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix split region issue in CodecClientV2

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* rearrange imports

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* add version info for RegionCache

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* delay request encoding to RPCClient

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* prevent multiple encoding

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* wip: debugging retry logic

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix EpochNotMatch region decoding

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* remove txn request branch

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* copy request before encoding it

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* add gRPCDialOptions for rawkv.Client

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix api breaking change of raw.NewClient

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* address menglong's comments

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* go mod tidy

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* address comments and fix linter

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix api_test

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix TestScan

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix TestScan

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* fix compatibility

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>
2022-07-09 14:49:46 +08:00
you06 bb026bdad8
client: support requests with source label (#506)
* support request scope and source.

Signed-off-by: you06 <you1474600@gmail.com>

* update

Signed-off-by: you06 <you1474600@gmail.com>

* use counter for requests with too-many source

Signed-off-by: you06 <you1474600@gmail.com>

* refine resolve lock

Signed-off-by: you06 <you1474600@gmail.com>

* remove request source from resolve details

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>
2022-06-24 18:17:09 +08:00
Lei Zhao 2807409d49
client: fix nil dereference when closing conns (#514)
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
2022-05-31 16:17:49 +08:00
guo-shaoge 700cbe60d2
support find stores with tiflash_mpp label (#481)
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Co-authored-by: Yilin Chen <sticnarf@gmail.com>
2022-05-23 11:22:53 +08:00
disksing ec236b6435
fix linter warning (#470)
Signed-off-by: disksing <i@disksing.com>
2022-04-11 21:29:30 +08:00
Lei Zhao f73ec0e675
client: fix the race between connArray.Close() and connArray.Get() (#465)
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
2022-04-06 17:12:03 +08:00
Lei Zhao 5b4040a95c
reconnect when meets store-not-match errors (#456)
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
2022-03-24 17:30:51 +08:00
disksing 589cf79b2a
*: update etcd client to v3.5.2 (#423)
Signed-off-by: disksing <i@disksing.com>
2022-02-17 22:01:16 +08:00
Song Gao b8cc5a319d
fix metrics error (#365)
Signed-off-by: yisaer <disxiaofei@163.com>
2021-11-02 12:05:33 +00:00
disksing 9ec50224be
*: replace pingcap/errors with pkg/errors (#357)
Signed-off-by: disksing <i@disksing.com>
2021-11-01 11:59:52 +08:00
disksing b074cb9ad3
internal/client: remove use of errors.Trace (#329)
Signed-off-by: disksing <i@disksing.com>
2021-10-11 16:59:49 +00:00
disksing 97fac134af
error: replace usages of terror.Log (#322)
Signed-off-by: disksing <i@disksing.com>
2021-10-11 06:08:21 +00:00
Song Gao 3cc2459779
add label (#314)
Signed-off-by: yisaer <disxiaofei@163.com>
2021-09-26 18:06:28 +08:00
Yilin Chen 16d902a3c7
reduce write backoffs but don't count busy errors (#271)
Signed-off-by: Yilin Chen <sticnarf@gmail.com>
2021-08-24 09:05:36 +00:00
tiancaiamao 9d2ad5209e
internal/client: remove the lock for idle connection recycle (#278)
Signed-off-by: tiancaiamao <tiancaiamao@gmail.com>
2021-08-23 15:23:11 +08:00
tiancaiamao 31906d4105
internal/client: fix recycle idle connection block sending request (#270)
Signed-off-by: tiancaiamao <tiancaiamao@gmail.com>
2021-08-19 23:40:46 +08:00
disksing df2119f51b
client: cleanup RPCClient options (#268)
Signed-off-by: disksing <i@disksing.com>
2021-08-16 07:11:08 +00:00
disksing 288c649636
fix license (#267)
Signed-off-by: disksing <i@disksing.com>
2021-08-15 01:48:01 +08:00
Jack Yu 55155ad2e5
*: make grpc request duration more accurate (#253)
Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>
2021-07-27 12:09:05 +00:00
disksing d0a98864a6
*: move logutil to internal directory (#207)
Signed-off-by: disksing <i@disksing.com>
2021-07-05 15:15:41 +08:00
disksing 4e8c0a34c8
move client to internal package (#194)
Signed-off-by: disksing <i@disksing.com>

Co-authored-by: iosmanthus <MyOsmanthusTree@gmail.com>
2021-06-30 16:18:10 +08:00