bce97bbb3e
Do not broadcast txn status when heartbeat fail ( #1471 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-10-10 05:27:19 +00:00
527f80a186
feat: Broadcast min_commit_ts for pipelined transactions ( #1458 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-09-25 01:34:14 +00:00
f0ea917499
transaction: add hook for async commit to track the life cycle of the async-commit goroutine and secondary lock cleanup goroutine ( #1432 )
...
Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>
2024-09-04 09:21:32 +00:00
41d133b6b6
membuffer: refactor the memdb to support multi implementations ( #1426 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-08-23 03:46:43 +00:00
0820519da7
Add GetTimeDetail for SnapshotRuntimeStats ( #1422 )
...
Signed-off-by: yibin <huyibin@pingcap.cn>
2024-08-14 14:30:31 +08:00
aa8b33873f
Let memdb cache last traversed node ( #1389 )
...
* opt: let memdb cache last traversed node
Signed-off-by: ekexium <eke@fastmail.com>
* add a switch
Signed-off-by: ekexium <eke@fastmail.com>
* use atomic pointer
Signed-off-by: ekexium <eke@fastmail.com>
* fix: invalidate cache if the node is deleted
Signed-off-by: ekexium <eke@fastmail.com>
* feat: log flush_wait_duration (#1391 )
Signed-off-by: ekexium <eke@fastmail.com>
* feat: add cache metrics
Signed-off-by: ekexium <eke@fastmail.com>
* opt: parent node tracking in traverse
Signed-off-by: ekexium <eke@fastmail.com>
* opt: isNull
Signed-off-by: ekexium <eke@fastmail.com>
* log: silence verbose failpoint log
Signed-off-by: ekexium <eke@fastmail.com>
* metrics: track traverse duration
Signed-off-by: ekexium <eke@fastmail.com>
* Revert "metrics: track traverse duration"
This reverts commit 05fc82bad51711c5f0fadf7936e118ed2c4990ab.
Signed-off-by: ekexium <eke@fastmail.com>
* fix: flush wait
Signed-off-by: ekexium <eke@fastmail.com>
* fix: duration logs
Signed-off-by: ekexium <eke@fastmail.com>
* refine log message
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-08-08 17:20:52 +08:00
3ac46e8171
fix: duration logs ( #1405 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-07-31 13:12:27 +08:00
88ce38492f
feat: log flush_wait_duration ( #1391 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-07-23 09:16:43 +08:00
ac8fa1d73a
feat: introduce a new method MemHookSet ( #1362 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-06-14 14:44:55 +08:00
a6335a5aaf
Fix the problem that statement being oom-killed within DoneAggressiveLocking causing the transaction still in aggressive locking state ( #1355 )
...
* Quick fix the problem that statement being oom-killed within DoneAggressiveLocking causing the transaction still in aggressive locking state
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Add comments to explain the change
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2024-05-30 16:20:09 +08:00
c40432e3ab
feat: record flush_wait_ms in TxnInfo ( #1342 )
...
* feat: record flush_wait_ms in TxnInfo
Signed-off-by: ekexium <eke@fastmail.com>
* refactor: change FlushWaitDuration() to GetFlushMetrics()
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-05-15 11:13:15 +08:00
6cb0704fce
fail pipelined dml when max ttl exceeded ( #1329 )
...
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-04-30 22:52:41 +08:00
31a502b9ba
improve region request log for diagnose ( #1300 )
...
* improve region request log for diagnose
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* rename struct
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine region error metric with store id label and add rpc error metric
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine comment
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine code
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* restrict log
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine code
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine log
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine code
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix test
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* address comment
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine log
Signed-off-by: crazycs520 <crazycs520@gmail.com>
---------
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2024-04-29 15:56:32 +08:00
8635acc840
feat: return error if commit ts expired is returned for a non-primary key ( #1316 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-24 19:16:26 +08:00
0229f4077f
do not cache snapshot with version maxuint64 ( #1320 )
...
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-24 13:23:42 +08:00
edba2544d5
txn: abort pipelined dml when pk rollback ( #1317 )
...
* fail p-dml when ttl manager is closed
Signed-off-by: you06 <you1474600@gmail.com>
* close ttl manager when pk is rollbcked
Signed-off-by: you06 <you1474600@gmail.com>
* stablize test
Signed-off-by: you06 <you1474600@gmail.com>
* add test
Signed-off-by: you06 <you1474600@gmail.com>
* fix race
Signed-off-by: you06 <you1474600@gmail.com>
* fix leak in test
Signed-off-by: you06 <you1474600@gmail.com>
* remote testlog
Signed-off-by: you06 <you1474600@gmail.com>
* check error message
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-23 17:36:44 +08:00
e02b025475
log: only log the first lock in every flush or prewrite response ( #1295 )
...
* log: only log the first lock in every flush or prewrite response
Signed-off-by: ekexium <eke@fastmail.com>
* log each conflicting txn at most once
Signed-off-by: ekexium <eke@fastmail.com>
* log: ttlManager
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 16:47:21 +08:00
5b3e266723
log: log the resolve progress of pipelined-dml every 30s ( #1291 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:51:40 +08:00
47891a931f
feat: stop flush when first error is met, like for prewrite ( #1296 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-18 15:40:37 +08:00
a23f6cac0c
log: silence schema check log ( #1290 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-15 15:08:02 +08:00
dcb9b95307
txn: cache result of snapshot.Get ( #1278 )
...
* also cache result of snapshot.Get
Signed-off-by: you06 <you1474600@gmail.com>
* fix test
Signed-off-by: you06 <you1474600@gmail.com>
fix test
Signed-off-by: you06 <you1474600@gmail.com>
* add Get cache test
Signed-off-by: you06 <you1474600@gmail.com>
* move RLock position
Signed-off-by: you06 <you1474600@gmail.com>
* remove comment
Signed-off-by: you06 <you1474600@gmail.com>
* address comment
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-13 16:45:31 +08:00
714958ccd4
implement rollback for pipelined dml ( #1235 )
...
* impl rollback for p-dml
Signed-off-by: you06 <you1474600@gmail.com>
* add test to ensure the rollback locks cannot be read
Signed-off-by: you06 <you1474600@gmail.com>
* remove TODO comment
Signed-off-by: you06 <you1474600@gmail.com>
* address comment
Signed-off-by: you06 <you1474600@gmail.com>
address comment
Signed-off-by: you06 <you1474600@gmail.com>
* resolve conflict
Signed-off-by: you06 <you1474600@gmail.com>
* lint
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-04-09 10:27:18 +08:00
f800b05581
Distinguish identifier and name in task runner ( #1271 )
...
* refactor: distinguish identifier and name in task runner, to avoid messy metrics
Signed-off-by: ekexium <eke@fastmail.com>
* keep the original interface to avoid breaking compatibility
Signed-off-by: ekexium <eke@fastmail.com>
* log: use ctx logger
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-08 19:35:52 +08:00
7c70c54016
Allow setting session id of KVTxn ( #1270 )
...
* allow setting session id
Signed-off-by: ekexium <eke@fastmail.com>
* comment: explain usage
Signed-off-by: ekexium <eke@fastmail.com>
* try using ctx logger as much as possible
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-08 16:03:59 +08:00
356eb45c5d
log: more items for diagnosing pipelined dml ( #1252 )
...
* log: more items for diagnosing pipelined dml
Signed-off-by: ekexium <eke@fastmail.com>
* log: flush duration
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-04-01 10:16:58 +08:00
87a984a72d
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB ( #1212 )
...
* feat: introduce GetLocal() for MemBuffer
Signed-off-by: ekexium <eke@fastmail.com>
* downgrade tools to 0.18.0
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set flushingMemDB = nil when an error is returned from a flush
Signed-off-by: ekexium <eke@fastmail.com>
* impl BatchGet for MemBuffer
Signed-off-by: you06 <you1474600@gmail.com>
test membuffer batch get
Signed-off-by: you06 <you1474600@gmail.com>
* add Prefetch & GetPrefetchCache
Signed-off-by: you06 <you1474600@gmail.com>
* cache multi Prefetch call
Signed-off-by: you06 <you1474600@gmail.com>
* add tests
Signed-off-by: you06 <you1474600@gmail.com>
* workaround for golang ci lint failure
Signed-off-by: you06 <you1474600@gmail.com>
* replace assert with require
Signed-off-by: you06 <you1474600@gmail.com>
* workaround golangci lint
Signed-off-by: you06 <you1474600@gmail.com>
* lint
Signed-off-by: you06 <you1474600@gmail.com>
* remove prefetch interface, pipelined memdb will cache batch get result
Signed-off-by: you06 <you1474600@gmail.com>
* update tidb
Signed-off-by: you06 <you1474600@gmail.com>
* fix batch get cache when membuffer is empty
Signed-off-by: you06 <you1474600@gmail.com>
* fix returned delete value
Signed-off-by: you06 <you1474600@gmail.com>
* fix: handle resourceGroupTag of Flush
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set resource group tag for committer if it's pipelined
Signed-off-by: ekexium <eke@fastmail.com>
* Update internal/unionstore/pipelined_memdb.go
Co-authored-by: ekexium <eke@fastmail.com>
* remove prefetch interface
Signed-off-by: you06 <you1474600@gmail.com>
* release mutex
Signed-off-by: you06 <you1474600@gmail.com>
* flush wait to avoid race
Signed-off-by: you06 <you1474600@gmail.com>
* fix unstopped test
Signed-off-by: you06 <you1474600@gmail.com>
* set resource group tags for committer
Signed-off-by: ekexium <eke@fastmail.com>
* skip test due to tikv image not updated yet
Signed-off-by: you06 <you1474600@gmail.com>
* skip more test
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 11:36:05 +08:00
280cb0852e
feat: add pipeliend field in TxnInfo ( #1228 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-15 12:45:02 +08:00
d7000ea557
feat: pipelined dml has its own max-txn-ttl of 24 hours ( #1224 )
...
* feat: pipelined dml has its own max-txn-ttl of 24 hours
Signed-off-by: ekexium <eke@fastmail.com>
* fix gh actions
Signed-off-by: ekexium <eke@fastmail.com>
* style: fix lint
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-14 10:54:19 +08:00
88fbc938cd
p-dml: fix unstopped resolve ( #1210 )
...
* fix unstopped resolve when the txn cross regions
Signed-off-by: you06 <you1474600@gmail.com>
* cancel context when flush done or abort
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2024-03-11 21:20:38 +08:00
af4f9a9b6e
fix data race when collapsing ResolveLockRequest ( #1209 )
...
* fix data race when collapsing ResolveLockRequest
Signed-off-by: you06 <you1474600@gmail.com>
* comment
Signed-off-by: you06 <you1474600@gmail.com>
* sleep to wait goroutine finish
Signed-off-by: you06 <you1474600@gmail.com>
* update comment
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-03-08 13:24:15 +08:00
c72236a601
adjust parameter for pipelined memdb by failpoint ( #1205 )
...
* adjust parameter for pipelined memdb by failpoint
Signed-off-by: you06 <you1474600@gmail.com>
* update comment
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-06 16:44:48 +08:00
83ec5f2f33
add ts information in the pre split path ( #1206 )
...
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-03-06 14:46:42 +08:00
da22d65e94
support gc v2 ( #1180 )
...
Signed-off-by: y_static_y@sina.com <y_static_y@sina.com>
2024-02-27 11:44:28 +08:00
8d28d3cd3a
txn: introduce pipelined memdb, Flush and BufferBatchGet for pipelined DML ( #1114 )
...
* Support pipelined memdb.
Signed-off-by: you06 <you1474600@gmail.com>
* fix mutex for pipelined memdb
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-02-23 10:24:44 +08:00
11cb7985f0
txn: enable pessimistic region rollback ( #1149 )
...
* enable pessimistic region rollback
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* add key range parameters
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* change interface to use start and end keys
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
---------
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-02-05 15:11:26 +08:00
70c148e84e
ErrQueryInterrupted with parameters ( #1124 )
...
* feat: ErrQueryInterrupted with parameters
Signed-off-by: ekexium <eke@fastmail.com>
* Revert "Revert "fix: check kill signal against 0 (#1102 )" (#1129 )"
This reverts commit 3480b5ed7c
2024-02-02 10:26:02 +08:00
d2887d56ab
txn: use region pessimsitic lock rollback to speed up cleanup ( #1125 )
...
* use region pessimsitic lock rollback and clean for write-write conflict processing
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* format
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
* debug ci failure patch, disable region pessimistic rollback
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
---------
Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
2024-01-31 09:53:00 +08:00
fd2fc84032
use slices.Sort to eliminate bounds check ( #1128 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-23 11:24:15 +08:00
abda04c22a
export stats information about RPC count ( #1110 )
...
Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>
2024-01-12 15:16:19 +08:00
2ea84a6c5a
feat: check kill signal in 2pc committer ( #1084 )
...
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-12-26 14:54:34 +08:00
6ba909c4ad
config: enable user configuration for retry strategy. ( #1055 )
...
Signed-off-by: Smityz <smityz@qq.com>
Co-authored-by: disksing <i@disksing.com>
2023-11-14 14:54:10 +08:00
07432ef6c0
refactor the func name to make it clear ( #1000 )
...
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2023-10-10 14:18:02 +08:00
18a8414120
*: fix data race at the aggressiveLockingDirty ( #913 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: disksing <i@disksing.com>
2023-10-08 14:40:27 +08:00
295094e5b5
add retry info to request source ( #953 )
...
* add retry info to request source
Signed-off-by: you06 <you1474600@gmail.com>
* handle upper layer retry
Signed-off-by: you06 <you1474600@gmail.com>
* stabilize test
Signed-off-by: you06 <you1474600@gmail.com>
* retry in 3 dimension
Signed-off-by: you06 <you1474600@gmail.com>
* record and restore req.ReadType
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2023-08-29 08:28:46 +08:00
8a214402da
use tidb_kv_read_timeout as first kv request timeout ( #919 )
...
* support tidb_kv_read_timeout as first round kv request timeout
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* update comment
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* refine test
Signed-off-by: crazycs520 <crazycs520@gmail.com>
---------
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2023-08-11 11:37:10 +08:00
2f119351bd
*: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` ( #883 )
...
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
2023-07-14 10:36:07 +08:00
178f6fa01a
add explicit request source type to label the external request like lightning/br ( #868 )
...
Signed-off-by: nolouch <nolouch@gmail.com>
2023-07-07 15:02:42 +08:00
c0cf773917
Fix the issue that primary pessimistic lock may be left not cleared after GC ( #866 )
...
* Fix the issue that primary pessimistic lock may be left not cleared after GC
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix mysteriously shown up thing that makes compilation failed
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Fix test effectiveness (forgot to set txn2 to pessimistic txn); add more strict checks
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
* Address comments
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
---------
Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2023-07-04 15:17:05 +08:00
04dc78e522
lock_resolver: let getTxnStatusFromLock return error when backoff timeout ( #847 )
...
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Co-authored-by: disksing <i@disksing.com>
2023-06-21 13:07:44 +08:00
0b4b0ca00e
kvrpc: deduplicate chained intercept by name ( #832 )
...
* dedup chained interceptor
Signed-off-by: glorv <glorvs@163.com>
2023-06-13 20:21:06 -07:00
ekexium