Commit Graph

153 Commits

Author SHA1 Message Date
ekexium 8635acc840
feat: return error if commit ts expired is returned for a non-primary key (#1316)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-24 19:16:26 +08:00
you06 0229f4077f
do not cache snapshot with version maxuint64 (#1320)
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-24 13:23:42 +08:00
you06 edba2544d5
txn: abort pipelined dml when pk rollback (#1317)
* fail p-dml when ttl manager is closed

Signed-off-by: you06 <you1474600@gmail.com>

* close ttl manager when pk is rollbcked

Signed-off-by: you06 <you1474600@gmail.com>

* stablize test

Signed-off-by: you06 <you1474600@gmail.com>

* add test

Signed-off-by: you06 <you1474600@gmail.com>

* fix race

Signed-off-by: you06 <you1474600@gmail.com>

* fix leak in test

Signed-off-by: you06 <you1474600@gmail.com>

* remote testlog

Signed-off-by: you06 <you1474600@gmail.com>

* check error message

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-04-23 17:36:44 +08:00
ekexium e02b025475
log: only log the first lock in every flush or prewrite response (#1295)
* log: only log the first lock in every flush or prewrite response

Signed-off-by: ekexium <eke@fastmail.com>

* log each conflicting txn at most once

Signed-off-by: ekexium <eke@fastmail.com>

* log: ttlManager

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 16:47:21 +08:00
ekexium 5b3e266723
log: log the resolve progress of pipelined-dml every 30s (#1291)
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:51:40 +08:00
ekexium 47891a931f
feat: stop flush when first error is met, like for prewrite (#1296)
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:40:37 +08:00
ekexium a23f6cac0c
log: silence schema check log (#1290)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-15 15:08:02 +08:00
you06 dcb9b95307
txn: cache result of snapshot.Get (#1278)
* also cache result of snapshot.Get

Signed-off-by: you06 <you1474600@gmail.com>

* fix test

Signed-off-by: you06 <you1474600@gmail.com>

fix test

Signed-off-by: you06 <you1474600@gmail.com>

* add Get cache test

Signed-off-by: you06 <you1474600@gmail.com>

* move RLock position

Signed-off-by: you06 <you1474600@gmail.com>

* remove comment

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-04-13 16:45:31 +08:00
you06 714958ccd4
implement rollback for pipelined dml (#1235)
* impl rollback for p-dml

Signed-off-by: you06 <you1474600@gmail.com>

* add test to ensure the rollback locks cannot be read

Signed-off-by: you06 <you1474600@gmail.com>

* remove TODO comment

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

address comment

Signed-off-by: you06 <you1474600@gmail.com>

* resolve conflict

Signed-off-by: you06 <you1474600@gmail.com>

* lint

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-04-09 10:27:18 +08:00
ekexium f800b05581
Distinguish identifier and name in task runner (#1271)
* refactor: distinguish identifier and name in task runner, to avoid messy metrics

Signed-off-by: ekexium <eke@fastmail.com>

* keep the original interface to avoid breaking compatibility

Signed-off-by: ekexium <eke@fastmail.com>

* log: use ctx logger

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
2024-04-08 19:35:52 +08:00
ekexium 7c70c54016
Allow setting session id of KVTxn (#1270)
* allow setting session id

Signed-off-by: ekexium <eke@fastmail.com>

* comment: explain usage

Signed-off-by: ekexium <eke@fastmail.com>

* try using ctx logger as much as possible

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-08 16:03:59 +08:00
ekexium 356eb45c5d
log: more items for diagnosing pipelined dml (#1252)
* log: more items for diagnosing pipelined dml

Signed-off-by: ekexium <eke@fastmail.com>

* log: flush duration

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-01 10:16:58 +08:00
ekexium 87a984a72d
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB (#1212)
* feat: introduce GetLocal() for MemBuffer

Signed-off-by: ekexium <eke@fastmail.com>

* downgrade tools to 0.18.0

Signed-off-by: ekexium <eke@fastmail.com>

* fix: set flushingMemDB = nil when an error is returned from a flush

Signed-off-by: ekexium <eke@fastmail.com>

* impl BatchGet for MemBuffer

Signed-off-by: you06 <you1474600@gmail.com>

test membuffer batch get

Signed-off-by: you06 <you1474600@gmail.com>

* add Prefetch & GetPrefetchCache

Signed-off-by: you06 <you1474600@gmail.com>

* cache multi Prefetch call

Signed-off-by: you06 <you1474600@gmail.com>

* add tests

Signed-off-by: you06 <you1474600@gmail.com>

* workaround for golang ci lint failure

Signed-off-by: you06 <you1474600@gmail.com>

* replace assert with require

Signed-off-by: you06 <you1474600@gmail.com>

* workaround golangci lint

Signed-off-by: you06 <you1474600@gmail.com>

* lint

Signed-off-by: you06 <you1474600@gmail.com>

* remove prefetch interface, pipelined memdb will cache batch get result

Signed-off-by: you06 <you1474600@gmail.com>

* update tidb

Signed-off-by: you06 <you1474600@gmail.com>

* fix batch get cache when membuffer is empty

Signed-off-by: you06 <you1474600@gmail.com>

* fix returned delete value

Signed-off-by: you06 <you1474600@gmail.com>

* fix: handle resourceGroupTag of Flush

Signed-off-by: ekexium <eke@fastmail.com>

* fix: set resource group tag for committer if it's pipelined

Signed-off-by: ekexium <eke@fastmail.com>

* Update internal/unionstore/pipelined_memdb.go

Co-authored-by: ekexium <eke@fastmail.com>

* remove prefetch interface

Signed-off-by: you06 <you1474600@gmail.com>

* release mutex

Signed-off-by: you06 <you1474600@gmail.com>

* flush wait to avoid race

Signed-off-by: you06 <you1474600@gmail.com>

* fix unstopped test

Signed-off-by: you06 <you1474600@gmail.com>

* set resource group tags for committer

Signed-off-by: ekexium <eke@fastmail.com>

* skip test due to tikv image not updated yet

Signed-off-by: you06 <you1474600@gmail.com>

* skip more test

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 11:36:05 +08:00
ekexium 280cb0852e
feat: add pipeliend field in TxnInfo (#1228)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-15 12:45:02 +08:00
ekexium d7000ea557
feat: pipelined dml has its own max-txn-ttl of 24 hours (#1224)
* feat: pipelined dml has its own max-txn-ttl of 24 hours

Signed-off-by: ekexium <eke@fastmail.com>

* fix gh actions

Signed-off-by: ekexium <eke@fastmail.com>

* style: fix lint

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
2024-03-14 10:54:19 +08:00
you06 88fbc938cd
p-dml: fix unstopped resolve (#1210)
* fix unstopped resolve when the txn cross regions

Signed-off-by: you06 <you1474600@gmail.com>

* cancel context when flush done or abort

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-03-11 21:20:38 +08:00
you06 af4f9a9b6e
fix data race when collapsing ResolveLockRequest (#1209)
* fix data race when collapsing ResolveLockRequest

Signed-off-by: you06 <you1474600@gmail.com>

* comment

Signed-off-by: you06 <you1474600@gmail.com>

* sleep to wait goroutine finish

Signed-off-by: you06 <you1474600@gmail.com>

* update comment

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-03-08 13:24:15 +08:00
you06 c72236a601
adjust parameter for pipelined memdb by failpoint (#1205)
* adjust parameter for pipelined memdb by failpoint

Signed-off-by: you06 <you1474600@gmail.com>

* update comment

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-06 16:44:48 +08:00
cfzjywxk 83ec5f2f33
add ts information in the pre split path (#1206)
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-03-06 14:46:42 +08:00
ystaticy da22d65e94
support gc v2 (#1180)
Signed-off-by: y_static_y@sina.com <y_static_y@sina.com>
2024-02-27 11:44:28 +08:00
you06 8d28d3cd3a
txn: introduce pipelined memdb, Flush and BufferBatchGet for pipelined DML (#1114)
* Support pipelined memdb.

Signed-off-by: you06 <you1474600@gmail.com>

* fix mutex for pipelined memdb

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2024-02-23 10:24:44 +08:00
cfzjywxk 11cb7985f0
txn: enable pessimistic region rollback (#1149)
* enable pessimistic region rollback

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* add key range parameters

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* change interface to use start and end keys

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

---------

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-02-05 15:11:26 +08:00
ekexium 70c148e84e
ErrQueryInterrupted with parameters (#1124)
* feat: ErrQueryInterrupted with parameters

Signed-off-by: ekexium <eke@fastmail.com>

* Revert "Revert "fix: check kill signal against 0 (#1102)" (#1129)"

This reverts commit 3480b5ed7c.

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-02-02 10:26:02 +08:00
cfzjywxk d2887d56ab
txn: use region pessimsitic lock rollback to speed up cleanup (#1125)
* use region pessimsitic lock rollback and clean for write-write conflict processing

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* format

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

* debug ci failure patch, disable region pessimistic rollback

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>

---------

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-01-31 09:53:00 +08:00
Weizhen Wang fd2fc84032
use slices.Sort to eliminate bounds check (#1128)
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-23 11:24:15 +08:00
YangKeao abda04c22a
export stats information about RPC count (#1110)
Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>
2024-01-12 15:16:19 +08:00
ekexium 2ea84a6c5a
feat: check kill signal in 2pc committer (#1084)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-12-26 14:54:34 +08:00
Smilencer 6ba909c4ad
config: enable user configuration for retry strategy. (#1055)
Signed-off-by: Smityz <smityz@qq.com>
Co-authored-by: disksing <i@disksing.com>
2023-11-14 14:54:10 +08:00
cfzjywxk 07432ef6c0
refactor the func name to make it clear (#1000)
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-10-10 14:18:02 +08:00
Weizhen Wang 18a8414120
*: fix data race at the aggressiveLockingDirty (#913)
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: disksing <i@disksing.com>
2023-10-08 14:40:27 +08:00
you06 295094e5b5
add retry info to request source (#953)
* add retry info to request source

Signed-off-by: you06 <you1474600@gmail.com>

* handle upper layer retry

Signed-off-by: you06 <you1474600@gmail.com>

* stabilize test

Signed-off-by: you06 <you1474600@gmail.com>

* retry in 3 dimension

Signed-off-by: you06 <you1474600@gmail.com>

* record and restore req.ReadType

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2023-08-29 08:28:46 +08:00
crazycs 8a214402da
use tidb_kv_read_timeout as first kv request timeout (#919)
* support tidb_kv_read_timeout as first round kv request timeout

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix ci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix ci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix ci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix ci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* fix ci

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* update comment

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* refine test

Signed-off-by: crazycs520 <crazycs520@gmail.com>

---------

Signed-off-by: crazycs520 <crazycs520@gmail.com>
2023-08-11 11:37:10 +08:00
Hangjie Mo 2f119351bd
*: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` (#883)
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
2023-07-14 10:36:07 +08:00
ShuNing 178f6fa01a
add explicit request source type to label the external request like lightning/br (#868)
Signed-off-by: nolouch <nolouch@gmail.com>
2023-07-07 15:02:42 +08:00
MyonKeminta c0cf773917
Fix the issue that primary pessimistic lock may be left not cleared after GC (#866)
* Fix the issue that primary pessimistic lock may be left not cleared after GC

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix mysteriously shown up thing that makes compilation failed

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix test effectiveness (forgot to set txn2 to pessimistic txn); add more strict checks

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Address comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-07-04 15:17:05 +08:00
zyguan 04dc78e522
lock_resolver: let getTxnStatusFromLock return error when backoff timeout (#847)
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-06-21 13:07:44 +08:00
glorv 0b4b0ca00e
kvrpc: deduplicate chained intercept by name (#832)
* dedup chained interceptor

Signed-off-by: glorv <glorvs@163.com>
2023-06-13 20:21:06 -07:00
you06 11121c6913
Remove stale-read flag after resolving lock (#792)
* remove stale-read when key-is-locked

Signed-off-by: you06 <you1474600@gmail.com>

* disable follower read for batch get

Signed-off-by: you06 <you1474600@gmail.com>

* the stale-read flag may already set to false

Signed-off-by: you06 <you1474600@gmail.com>

* fix batchget

Signed-off-by: you06 <you1474600@gmail.com>

* Reset busy-threshold when stale read fallback to leader read

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

* trigger CI

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
2023-05-26 13:55:53 +08:00
zyguan 2b0667c65c
lock_resolver: handle pessimistic locks in BatchResolveLocks (#794)
* lock_resolver: handle pessimistic locks in BatchResolveLocks

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test but failed on unistore because unistore's ScanLock doesnt return lock type

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Address comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix test

Signed-off-by: zyguan <zhongyangguan@gmail.com>

* Fix golangci

Signed-off-by: zyguan <zhongyangguan@gmail.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-05-11 12:26:01 +08:00
MyonKeminta 5c324b7c1e
lock_resolver: support verifying primary for check_txn_status (#777)
* support verifying primary for check_txn_status

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* update kvproto

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* add more failpoint usages

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* update depencency and fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Do not skip for unistore; refine logs

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Address comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-05-08 16:18:26 +08:00
Connor 62cc95123c
Fix wrong resource group name for some requests (#788)
* fix wrong resource group name

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
2023-04-28 19:03:56 +08:00
disksing aee2f311a5
txn: fix goroutine leak (#784)
Signed-off-by: disksing <i@disksing.com>
2023-04-27 17:39:07 +08:00
Connor 35c1ee47c4
Set resource group penalty (#772)
* Set resource group penalty 

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
2023-04-19 20:39:20 +08:00
ekexium a72e61dc64
log: print log if the assertion error is generated from client-go (#771)
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-04-19 17:42:17 +08:00
MyonKeminta 92db9f7b15
Support specifying checking forUpdateTS in prewrite (#756)
* Support specifying checking forUpdateTS in prewrite

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix go mod in tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add successful cases to the test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update dependencies

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check LockedWithConflictTS greater than requested ForUpdateTS

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-04-17 14:53:28 +08:00
Jack Yu c853ddc68c
metrics: add txn ttl lifetime (#766)
Signed-off-by: Jack Yu <jackysp@gmail.com>
Co-authored-by: Shirly <AndreMouche@126.com>
2023-04-14 13:59:29 +08:00
ekexium 67e56a956f
For KeyIsLocked error reported for timeout, if a lock is recently updated, don't try to resolve it. (#758)
* update client-go; format

Signed-off-by: ekexium <eke@fastmail.com>

* feat: do not resolve lock if duration_to_last_updated is short

Signed-off-by: ekexium <eke@fastmail.com>

* adjust the threshold to 1200ms to allow small deviation

Signed-off-by: ekexium <eke@fastmail.com>

* fix: don't treat it as WriteConflict, simply retry

Signed-off-by: ekexium <eke@fastmail.com>

* update kvproto

Signed-off-by: ekexium <eke@fastmail.com>

* set the threshold to 300ms

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
2023-04-13 16:14:28 +08:00
ekexium f3e87039d8
Finer `is_retry_request` settings (#763)
* set is_retry_request only for requests that is possible to have undetermined errors

Signed-off-by: ekexium <eke@fastmail.com>

* rename tryTimes to retryTimes

Signed-off-by: ekexium <eke@fastmail.com>

---------

Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2023-04-10 10:39:43 +08:00
MyonKeminta ea13e97002
Add exitAggressiveLockingIfInapplicable function and call within mutex of lockKeys (#749)
* Add exitAggressiveLockingIfInapplicable function and call within mutex of lockKeys

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* fix fmt

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

---------

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-03-28 16:41:04 +08:00
JmPotato 9d950905d7
Introduce the RURuntimeStats (#732)
Signed-off-by: JmPotato <ghzpotato@gmail.com>
2023-03-16 10:19:36 +08:00