41d133b6b6
membuffer: refactor the memdb to support multi implementations ( #1426 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-08-23 03:46:43 +00:00
aa8b33873f
Let memdb cache last traversed node ( #1389 )
...
* opt: let memdb cache last traversed node
Signed-off-by: ekexium <eke@fastmail.com>
* add a switch
Signed-off-by: ekexium <eke@fastmail.com>
* use atomic pointer
Signed-off-by: ekexium <eke@fastmail.com>
* fix: invalidate cache if the node is deleted
Signed-off-by: ekexium <eke@fastmail.com>
* feat: log flush_wait_duration (#1391 )
Signed-off-by: ekexium <eke@fastmail.com>
* feat: add cache metrics
Signed-off-by: ekexium <eke@fastmail.com>
* opt: parent node tracking in traverse
Signed-off-by: ekexium <eke@fastmail.com>
* opt: isNull
Signed-off-by: ekexium <eke@fastmail.com>
* log: silence verbose failpoint log
Signed-off-by: ekexium <eke@fastmail.com>
* metrics: track traverse duration
Signed-off-by: ekexium <eke@fastmail.com>
* Revert "metrics: track traverse duration"
This reverts commit 05fc82bad51711c5f0fadf7936e118ed2c4990ab.
Signed-off-by: ekexium <eke@fastmail.com>
* fix: flush wait
Signed-off-by: ekexium <eke@fastmail.com>
* fix: duration logs
Signed-off-by: ekexium <eke@fastmail.com>
* refine log message
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-08-08 17:20:52 +08:00
3ac46e8171
fix: duration logs ( #1405 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-07-31 13:12:27 +08:00
88ce38492f
feat: log flush_wait_duration ( #1391 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-07-23 09:16:43 +08:00
ac8fa1d73a
feat: introduce a new method MemHookSet ( #1362 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-06-14 14:44:55 +08:00
a6335a5aaf
Fix the problem that statement being oom-killed within DoneAggressiveLocking causing the transaction still in aggressive locking state ( #1355 )
...
* Quick fix the problem that statement being oom-killed within DoneAggressiveLocking causing the transaction still in aggressive locking state
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Add comments to explain the change
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2024-05-30 16:20:09 +08:00
c40432e3ab
feat: record flush_wait_ms in TxnInfo ( #1342 )
...
* feat: record flush_wait_ms in TxnInfo
Signed-off-by: ekexium <eke@fastmail.com>
* refactor: change FlushWaitDuration() to GetFlushMetrics()
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-05-15 11:13:15 +08:00
6cb0704fce
fail pipelined dml when max ttl exceeded ( #1329 )
...
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-04-30 22:52:41 +08:00
8635acc840
feat: return error if commit ts expired is returned for a non-primary key ( #1316 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-24 19:16:26 +08:00
edba2544d5
txn: abort pipelined dml when pk rollback ( #1317 )
...
* fail p-dml when ttl manager is closed
Signed-off-by: you06 <you1474600@gmail.com>
* close ttl manager when pk is rollbcked
Signed-off-by: you06 <you1474600@gmail.com>
* stablize test
Signed-off-by: you06 <you1474600@gmail.com>
* add test
Signed-off-by: you06 <you1474600@gmail.com>
* fix race
Signed-off-by: you06 <you1474600@gmail.com>
* fix leak in test
Signed-off-by: you06 <you1474600@gmail.com>
* remote testlog
Signed-off-by: you06 <you1474600@gmail.com>
* check error message
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-23 17:36:44 +08:00
e02b025475
log: only log the first lock in every flush or prewrite response ( #1295 )
...
* log: only log the first lock in every flush or prewrite response
Signed-off-by: ekexium <eke@fastmail.com>
* log each conflicting txn at most once
Signed-off-by: ekexium <eke@fastmail.com>
* log: ttlManager
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 16:47:21 +08:00
5b3e266723
log: log the resolve progress of pipelined-dml every 30s ( #1291 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:51:40 +08:00
47891a931f
feat: stop flush when first error is met, like for prewrite ( #1296 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:40:37 +08:00
a23f6cac0c
log: silence schema check log ( #1290 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-15 15:08:02 +08:00
714958ccd4
implement rollback for pipelined dml ( #1235 )
...
* impl rollback for p-dml
Signed-off-by: you06 <you1474600@gmail.com>
* add test to ensure the rollback locks cannot be read
Signed-off-by: you06 <you1474600@gmail.com>
* remove TODO comment
Signed-off-by: you06 <you1474600@gmail.com>
* address comment
Signed-off-by: you06 <you1474600@gmail.com>
address comment
Signed-off-by: you06 <you1474600@gmail.com>
* resolve conflict
Signed-off-by: you06 <you1474600@gmail.com>
* lint
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-04-09 10:27:18 +08:00
f800b05581
Distinguish identifier and name in task runner ( #1271 )
...
* refactor: distinguish identifier and name in task runner, to avoid messy metrics
Signed-off-by: ekexium <eke@fastmail.com>
* keep the original interface to avoid breaking compatibility
Signed-off-by: ekexium <eke@fastmail.com>
* log: use ctx logger
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-08 19:35:52 +08:00
7c70c54016
Allow setting session id of KVTxn ( #1270 )
...
* allow setting session id
Signed-off-by: ekexium <eke@fastmail.com>
* comment: explain usage
Signed-off-by: ekexium <eke@fastmail.com>
* try using ctx logger as much as possible
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-08 16:03:59 +08:00
356eb45c5d
log: more items for diagnosing pipelined dml ( #1252 )
...
* log: more items for diagnosing pipelined dml
Signed-off-by: ekexium <eke@fastmail.com>
* log: flush duration
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-01 10:16:58 +08:00
87a984a72d
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB ( #1212 )
...
* feat: introduce GetLocal() for MemBuffer
Signed-off-by: ekexium <eke@fastmail.com>
* downgrade tools to 0.18.0
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set flushingMemDB = nil when an error is returned from a flush
Signed-off-by: ekexium <eke@fastmail.com>
* impl BatchGet for MemBuffer
Signed-off-by: you06 <you1474600@gmail.com>
test membuffer batch get
Signed-off-by: you06 <you1474600@gmail.com>
* add Prefetch & GetPrefetchCache
Signed-off-by: you06 <you1474600@gmail.com>
* cache multi Prefetch call
Signed-off-by: you06 <you1474600@gmail.com>
* add tests
Signed-off-by: you06 <you1474600@gmail.com>
* workaround for golang ci lint failure
Signed-off-by: you06 <you1474600@gmail.com>
* replace assert with require
Signed-off-by: you06 <you1474600@gmail.com>
* workaround golangci lint
Signed-off-by: you06 <you1474600@gmail.com>
* lint
Signed-off-by: you06 <you1474600@gmail.com>
* remove prefetch interface, pipelined memdb will cache batch get result
Signed-off-by: you06 <you1474600@gmail.com>
* update tidb
Signed-off-by: you06 <you1474600@gmail.com>
* fix batch get cache when membuffer is empty
Signed-off-by: you06 <you1474600@gmail.com>
* fix returned delete value
Signed-off-by: you06 <you1474600@gmail.com>
* fix: handle resourceGroupTag of Flush
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set resource group tag for committer if it's pipelined
Signed-off-by: ekexium <eke@fastmail.com>
* Update internal/unionstore/pipelined_memdb.go
Co-authored-by: ekexium <eke@fastmail.com>
* remove prefetch interface
Signed-off-by: you06 <you1474600@gmail.com>
* release mutex
Signed-off-by: you06 <you1474600@gmail.com>
* flush wait to avoid race
Signed-off-by: you06 <you1474600@gmail.com>
* fix unstopped test
Signed-off-by: you06 <you1474600@gmail.com>
* set resource group tags for committer
Signed-off-by: ekexium <eke@fastmail.com>
* skip test due to tikv image not updated yet
Signed-off-by: you06 <you1474600@gmail.com>
* skip more test
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 11:36:05 +08:00
280cb0852e
feat: add pipeliend field in TxnInfo ( #1228 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-15 12:45:02 +08:00
d7000ea557
feat: pipelined dml has its own max-txn-ttl of 24 hours ( #1224 )
...
* feat: pipelined dml has its own max-txn-ttl of 24 hours
Signed-off-by: ekexium <eke@fastmail.com>
* fix gh actions
Signed-off-by: ekexium <eke@fastmail.com>
* style: fix lint
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-14 10:54:19 +08:00
88fbc938cd
p-dml: fix unstopped resolve ( #1210 )
...
* fix unstopped resolve when the txn cross regions
Signed-off-by: you06 <you1474600@gmail.com>
* cancel context when flush done or abort
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-03-11 21:20:38 +08:00
af4f9a9b6e
fix data race when collapsing ResolveLockRequest ( #1209 )
...
* fix data race when collapsing ResolveLockRequest
Signed-off-by: you06 <you1474600@gmail.com>
* comment
Signed-off-by: you06 <you1474600@gmail.com>
* sleep to wait goroutine finish
Signed-off-by: you06 <you1474600@gmail.com>
* update comment
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-03-08 13:24:15 +08:00
83ec5f2f33
add ts information in the pre split path ( #1206 )
...
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-03-06 14:46:42 +08:00
8d28d3cd3a
txn: introduce pipelined memdb, Flush and BufferBatchGet for pipelined DML ( #1114 )
...
* Support pipelined memdb.
Signed-off-by: you06 <you1474600@gmail.com>
* fix mutex for pipelined memdb
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-02-23 10:24:44 +08:00
11cb7985f0
txn: enable pessimistic region rollback ( #1149 )
...
* enable pessimistic region rollback
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* add key range parameters
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* change interface to use start and end keys
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
---------
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-02-05 15:11:26 +08:00
70c148e84e
ErrQueryInterrupted with parameters ( #1124 )
...
* feat: ErrQueryInterrupted with parameters
Signed-off-by: ekexium <eke@fastmail.com>
* Revert "Revert "fix: check kill signal against 0 (#1102 )" (#1129 )"
This reverts commit 3480b5ed7c
2024-02-02 10:26:02 +08:00
d2887d56ab
txn: use region pessimsitic lock rollback to speed up cleanup ( #1125 )
...
* use region pessimsitic lock rollback and clean for write-write conflict processing
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* format
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* debug ci failure patch, disable region pessimistic rollback
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
---------
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-01-31 09:53:00 +08:00
fd2fc84032
use slices.Sort to eliminate bounds check ( #1128 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-23 11:24:15 +08:00
2ea84a6c5a
feat: check kill signal in 2pc committer ( #1084 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-12-26 14:54:34 +08:00
6ba909c4ad
config: enable user configuration for retry strategy. ( #1055 )
...
Signed-off-by: Smityz <smityz@qq.com>
Co-authored-by: disksing <i@disksing.com>
2023-11-14 14:54:10 +08:00
18a8414120
*: fix data race at the aggressiveLockingDirty ( #913 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: disksing <i@disksing.com>
2023-10-08 14:40:27 +08:00
2f119351bd
*: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` ( #883 )
...
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
2023-07-14 10:36:07 +08:00
178f6fa01a
add explicit request source type to label the external request like lightning/br ( #868 )
...
Signed-off-by: nolouch <nolouch@gmail.com>
2023-07-07 15:02:42 +08:00
c0cf773917
Fix the issue that primary pessimistic lock may be left not cleared after GC ( #866 )
...
* Fix the issue that primary pessimistic lock may be left not cleared after GC
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix mysteriously shown up thing that makes compilation failed
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix test effectiveness (forgot to set txn2 to pessimistic txn); add more strict checks
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Address comments
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-07-04 15:17:05 +08:00
5c324b7c1e
lock_resolver: support verifying primary for check_txn_status ( #777 )
...
* support verifying primary for check_txn_status
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* update kvproto
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* add more failpoint usages
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* update depencency and fix test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Do not skip for unistore; refine logs
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Address comments
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-05-08 16:18:26 +08:00
aee2f311a5
txn: fix goroutine leak ( #784 )
...
Signed-off-by: disksing <i@disksing.com>
2023-04-27 17:39:07 +08:00
35c1ee47c4
Set resource group penalty ( #772 )
...
* Set resource group penalty
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
2023-04-19 20:39:20 +08:00
a72e61dc64
log: print log if the assertion error is generated from client-go ( #771 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-04-19 17:42:17 +08:00
92db9f7b15
Support specifying checking forUpdateTS in prewrite ( #756 )
...
* Support specifying checking forUpdateTS in prewrite
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix go mod in tests
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Add test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Add successful cases to the test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Update dependencies
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Check LockedWithConflictTS greater than requested ForUpdateTS
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-04-17 14:53:28 +08:00
c853ddc68c
metrics: add txn ttl lifetime ( #766 )
...
Signed-off-by: Jack Yu <jackysp@gmail.com>
Co-authored-by: Shirly <AndreMouche@126.com>
2023-04-14 13:59:29 +08:00
67e56a956f
For KeyIsLocked error reported for timeout, if a lock is recently updated, don't try to resolve it. ( #758 )
...
* update client-go; format
Signed-off-by: ekexium <eke@fastmail.com>
* feat: do not resolve lock if duration_to_last_updated is short
Signed-off-by: ekexium <eke@fastmail.com>
* adjust the threshold to 1200ms to allow small deviation
Signed-off-by: ekexium <eke@fastmail.com>
* fix: don't treat it as WriteConflict, simply retry
Signed-off-by: ekexium <eke@fastmail.com>
* update kvproto
Signed-off-by: ekexium <eke@fastmail.com>
* set the threshold to 300ms
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2023-04-13 16:14:28 +08:00
f3e87039d8
Finer `is_retry_request` settings ( #763 )
...
* set is_retry_request only for requests that is possible to have undetermined errors
Signed-off-by: ekexium <eke@fastmail.com>
* rename tryTimes to retryTimes
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2023-04-10 10:39:43 +08:00
ea13e97002
Add exitAggressiveLockingIfInapplicable function and call within mutex of lockKeys ( #749 )
...
* Add exitAggressiveLockingIfInapplicable function and call within mutex of lockKeys
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* fix fmt
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Add test
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-03-28 16:41:04 +08:00
9d950905d7
Introduce the RURuntimeStats ( #732 )
...
Signed-off-by: JmPotato <ghzpotato@gmail.com>
2023-03-16 10:19:36 +08:00
f555fdd2c9
txn: support force locking and lock only if exists be used together ( #727 )
...
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-03-09 18:08:32 +08:00
bdd41058aa
metrics: seperate metrics with source scope for txn command ( #723 )
...
* seperate metrics with source scope
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
* fix async pessimistic rollback
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
* fix missing definition
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
---------
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-03-06 22:12:45 +08:00
49323956ae
metric: seperate source for txn cmd ( #721 )
...
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-02-28 15:40:40 +08:00
596b79404d
set group name in twoPhaseCommitter ( #718 )
...
Signed-off-by: glorv <glorvs@163.com>
2023-02-27 16:39:23 +08:00
40a82457eb
tikv: configurable pool ( #714 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2023-02-27 11:23:58 +08:00
you06