183817ac81
memdb: introduce snapshot interface ( #1623 )
...
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: ekexium <eke@fastmail.com>
2025-04-15 14:17:37 +00:00
e53096d518
support redact key in logs ( #1612 )
...
ref pingcap/tidb#59279
Signed-off-by: tangenta <tangenta@126.com>
Co-authored-by: you06 <you1474600@gmail.com>
2025-04-01 07:17:12 +00:00
ddec8237fd
memdb: prevent iterator invalidation ( #1563 )
...
ref pingcap/tidb#59153
Signed-off-by: ekexium <eke@fastmail.com>
2025-02-19 08:02:06 +00:00
43d2db704a
memdb: refactor code of art to make it more understandable ( #1508 )
...
Signed-off-by: cfzjywxk <lsswxrxr@163.com>
2024-12-16 10:46:15 +00:00
05d115b3e8
memdb: retain old version nodes of ART to satisfy snapshot read ( #1503 )
...
ref pingcap/tidb#57425
Signed-off-by: you06 <you1474600@gmail.com>
2024-11-20 02:44:59 +00:00
86678db6fd
membuffer: fix memory leak in red-black tree ( #1483 )
...
close tikv/client-go#1375 , ref pingcap/tidb#56837
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <cfzjywxk@gmail.com>
2024-11-14 14:41:17 +00:00
c154447fa5
membuffer: compare the keys of ART by chunk ( #1482 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-11-07 08:45:30 +00:00
0232600d68
membuffer: improve ART's code and test coverage ( #1477 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-10-31 01:41:22 +00:00
e30a3e6e3a
*: upgrade go1.23 ( #1473 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-10-11 06:35:57 +00:00
c3e10ae7c8
membuffer: implement cache for ART ( #1470 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-10-08 08:58:09 +00:00
58f3322fc3
membuffer: implement snapshot get and iterator for ART ( #1467 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-09-25 07:03:02 +00:00
7d0f0bc93b
membuffer: support iterator for ART ( #1466 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-09-24 11:38:48 +00:00
271945f345
membuffer: support staging & checkpoint for ART ( #1465 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-09-23 03:55:13 +00:00
6beede6998
membuffer: implement ART with basic get/set ( #1451 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-09-18 09:25:22 +00:00
6b1453c358
memdb: use atomic u64 addr to reduce allocation ( #1453 )
...
Signed-off-by: you06 <you1474600@gmail.com>
2024-08-28 04:00:17 +00:00
41d133b6b6
membuffer: refactor the memdb to support multi implementations ( #1426 )
...
ref pingcap/tidb#55287
Signed-off-by: you06 <you1474600@gmail.com>
2024-08-23 03:46:43 +00:00
75e3705e58
memdb: fix memdb snapshot get/iter is not actually snapshot ( #1393 )
...
* fix memdb snapshot get/iter is not actually snapshot
Signed-off-by: you06 <you1474600@gmail.com>
* rename test
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-08-21 15:35:30 +08:00
aa8b33873f
Let memdb cache last traversed node ( #1389 )
...
* opt: let memdb cache last traversed node
Signed-off-by: ekexium <eke@fastmail.com>
* add a switch
Signed-off-by: ekexium <eke@fastmail.com>
* use atomic pointer
Signed-off-by: ekexium <eke@fastmail.com>
* fix: invalidate cache if the node is deleted
Signed-off-by: ekexium <eke@fastmail.com>
* feat: log flush_wait_duration (#1391 )
Signed-off-by: ekexium <eke@fastmail.com>
* feat: add cache metrics
Signed-off-by: ekexium <eke@fastmail.com>
* opt: parent node tracking in traverse
Signed-off-by: ekexium <eke@fastmail.com>
* opt: isNull
Signed-off-by: ekexium <eke@fastmail.com>
* log: silence verbose failpoint log
Signed-off-by: ekexium <eke@fastmail.com>
* metrics: track traverse duration
Signed-off-by: ekexium <eke@fastmail.com>
* Revert "metrics: track traverse duration"
This reverts commit 05fc82bad51711c5f0fadf7936e118ed2c4990ab.
Signed-off-by: ekexium <eke@fastmail.com>
* fix: flush wait
Signed-off-by: ekexium <eke@fastmail.com>
* fix: duration logs
Signed-off-by: ekexium <eke@fastmail.com>
* refine log message
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-08-08 17:20:52 +08:00
3ac46e8171
fix: duration logs ( #1405 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-07-31 13:12:27 +08:00
fce0abfd82
Let pipelined memdb ignore bufferLimit ( #1392 )
...
* fix: pipelined memdb ignores bufferLimit
Signed-off-by: ekexium <eke@fastmail.com>
* refactor: introduce unlimitedSize
Signed-off-by: ekexium <eke@fastmail.com>
* refactor: unlimitedSize
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-07-29 12:13:17 +08:00
ac8fa1d73a
feat: introduce a new method MemHookSet ( #1362 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-06-14 14:44:55 +08:00
c40432e3ab
feat: record flush_wait_ms in TxnInfo ( #1342 )
...
* feat: record flush_wait_ms in TxnInfo
Signed-off-by: ekexium <eke@fastmail.com>
* refactor: change FlushWaitDuration() to GetFlushMetrics()
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-05-15 11:13:15 +08:00
edba2544d5
txn: abort pipelined dml when pk rollback ( #1317 )
...
* fail p-dml when ttl manager is closed
Signed-off-by: you06 <you1474600@gmail.com>
* close ttl manager when pk is rollbcked
Signed-off-by: you06 <you1474600@gmail.com>
* stablize test
Signed-off-by: you06 <you1474600@gmail.com>
* add test
Signed-off-by: you06 <you1474600@gmail.com>
* fix race
Signed-off-by: you06 <you1474600@gmail.com>
* fix leak in test
Signed-off-by: you06 <you1474600@gmail.com>
* remote testlog
Signed-off-by: you06 <you1474600@gmail.com>
* check error message
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-04-23 17:36:44 +08:00
4183ab10fa
Use Mem() instead of Size() to estimate pipelined-memdb size ( #1286 )
...
* fix: use Mem() instead of Size() to evaluate pipelined-memdb size, for better memory control
Signed-off-by: ekexium <eke@fastmail.com>
* test: fix TestPipelinedFlushTrigger
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-04-15 16:53:35 +08:00
58ef395164
test: restore skipped test for pipelined dml ( #1238 )
...
* test: restore skipped test
Signed-off-by: ekexium <eke@fastmail.com>
* test: FlushWait() to avoid race
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-19 20:37:44 +08:00
a9128e8200
fix: check mem != nil before calling Mem() ( #1236 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-18 14:55:17 +08:00
73c0712c01
txn: implement staging for PipelinedMemDB ( #1230 )
...
* feat: implement staging for PipelinedMemDB
Signed-off-by: ekexium <eke@fastmail.com>
* comment: explain staging methods
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-18 14:25:05 +08:00
56418bf28f
feat: implement Mem() and SetMemoryFootprintChangeHook ( #1233 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2024-03-18 11:02:04 +08:00
87a984a72d
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB ( #1212 )
...
* feat: introduce GetLocal() for MemBuffer
Signed-off-by: ekexium <eke@fastmail.com>
* downgrade tools to 0.18.0
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set flushingMemDB = nil when an error is returned from a flush
Signed-off-by: ekexium <eke@fastmail.com>
* impl BatchGet for MemBuffer
Signed-off-by: you06 <you1474600@gmail.com>
test membuffer batch get
Signed-off-by: you06 <you1474600@gmail.com>
* add Prefetch & GetPrefetchCache
Signed-off-by: you06 <you1474600@gmail.com>
* cache multi Prefetch call
Signed-off-by: you06 <you1474600@gmail.com>
* add tests
Signed-off-by: you06 <you1474600@gmail.com>
* workaround for golang ci lint failure
Signed-off-by: you06 <you1474600@gmail.com>
* replace assert with require
Signed-off-by: you06 <you1474600@gmail.com>
* workaround golangci lint
Signed-off-by: you06 <you1474600@gmail.com>
* lint
Signed-off-by: you06 <you1474600@gmail.com>
* remove prefetch interface, pipelined memdb will cache batch get result
Signed-off-by: you06 <you1474600@gmail.com>
* update tidb
Signed-off-by: you06 <you1474600@gmail.com>
* fix batch get cache when membuffer is empty
Signed-off-by: you06 <you1474600@gmail.com>
* fix returned delete value
Signed-off-by: you06 <you1474600@gmail.com>
* fix: handle resourceGroupTag of Flush
Signed-off-by: ekexium <eke@fastmail.com>
* fix: set resource group tag for committer if it's pipelined
Signed-off-by: ekexium <eke@fastmail.com>
* Update internal/unionstore/pipelined_memdb.go
Co-authored-by: ekexium <eke@fastmail.com>
* remove prefetch interface
Signed-off-by: you06 <you1474600@gmail.com>
* release mutex
Signed-off-by: you06 <you1474600@gmail.com>
* flush wait to avoid race
Signed-off-by: you06 <you1474600@gmail.com>
* fix unstopped test
Signed-off-by: you06 <you1474600@gmail.com>
* set resource group tags for committer
Signed-off-by: ekexium <eke@fastmail.com>
* skip test due to tikv image not updated yet
Signed-off-by: you06 <you1474600@gmail.com>
* skip more test
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: you06 <you1474600@gmail.com>
2024-03-16 11:36:05 +08:00
d59fea5757
P-DML: for AlreadyExist error, attach the value from flushingMemDB ( #1218 )
...
* feat: for AlreadyExist error during Flush, attach the value from flushingMemDB to it
Signed-off-by: ekexium <eke@fastmail.com>
* style: apply comment suggestion
Signed-off-by: ekexium <eke@fastmail.com>
---------
Signed-off-by: ekexium <eke@fastmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-13 10:23:20 +08:00
c72236a601
adjust parameter for pipelined memdb by failpoint ( #1205 )
...
* adjust parameter for pipelined memdb by failpoint
Signed-off-by: you06 <you1474600@gmail.com>
* update comment
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
Co-authored-by: cfzjywxk <lsswxrxr@163.com>
2024-03-06 16:44:48 +08:00
8d28d3cd3a
txn: introduce pipelined memdb, Flush and BufferBatchGet for pipelined DML ( #1114 )
...
* Support pipelined memdb.
Signed-off-by: you06 <you1474600@gmail.com>
* fix mutex for pipelined memdb
Signed-off-by: you06 <you1474600@gmail.com>
---------
Signed-off-by: you06 <you1474600@gmail.com>
2024-02-23 10:24:44 +08:00
fd2fc84032
use slices.Sort to eliminate bounds check ( #1128 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2024-01-23 11:24:15 +08:00
2f119351bd
*: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` ( #883 )
...
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
2023-07-14 10:36:07 +08:00
df58452fe1
client_batch: add test for auto re-connect idle connection when wait connection ready and fix ci ( #835 )
...
* client_batch: add test
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* update go version
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci lint
Signed-off-by: crazycs520 <crazycs520@gmail.com>
* fix ci lint
Signed-off-by: crazycs520 <crazycs520@gmail.com>
---------
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2023-06-13 14:17:17 +08:00
6d5cb2da49
don't call onMemChange before all work is done in memdb ( #636 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2022-12-08 12:30:56 +08:00
ad59ca833a
fix: do not call onMemChange in alloc, since it will make the latest block empty, which breaks a precondition for some operations (e.g. revertToCheckpoint) ( #632 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2022-12-05 16:43:17 +08:00
81939ec8b2
fix data race at the memChangeHook ( #625 )
...
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
2022-11-28 10:56:02 +08:00
92f0a82e1a
fix: avoid data race when setting memdb footprint hook ( #621 )
...
Signed-off-by: ekexium <eke@fastmail.com>
2022-11-24 11:10:13 +08:00
0130f76738
Support postponed conflict check ( #556 )
...
* replace kvproto
Signed-off-by: ekexium <eke@fastmail.com>
* support NeedConflictCheck
Signed-off-by: ekexium <eke@fastmail.com>
* fix mutation encoding
Signed-off-by: ekexium <eke@fastmail.com>
* support temporary flag
Signed-off-by: ekexium <eke@fastmail.com>
* update kvproto
Signed-off-by: ekexium <eke@fastmail.com>
* fix style
Signed-off-by: ekexium <eke@fastmail.com>
* add an option to enable the behavior
Signed-off-by: ekexium <eke@fastmail.com>
* replace AfterCheckPoint with existing canModity
Signed-off-by: ekexium <eke@fastmail.com>
* UpdateFlag do not unset temporary flag
Signed-off-by: ekexium <eke@fastmail.com>
* remove unused function
Signed-off-by: ekexium <eke@fastmail.com>
* update tidb dependency
Signed-off-by: ekexium <eke@fastmail.com>
update tidb dependency
Signed-off-by: ekexium <eke@fastmail.com>
* fix test
Signed-off-by: ekexium <eke@fastmail.com>
* do no unset flag on read
Signed-off-by: ekexium <eke@fastmail.com>
* update tidb dependency
Signed-off-by: ekexium <eke@fastmail.com>
* update comment
Signed-off-by: ekexium <eke@fastmail.com>
Signed-off-by: ekexium <eke@fastmail.com>
2022-08-30 15:38:39 +08:00
79a0fc59e4
Add a callback when MemDB grows ( #520 )
...
* implement the footprint chagne hook
Signed-off-by: ekexium <ekexium@fastmail.com>
* simplify the signature
Signed-off-by: ekexium <ekexium@fastmail.com>
* also call the callback when arena shrinks
Signed-off-by: ekexium <ekexium@fastmail.com>
* also callback on reset
Signed-off-by: ekexium <ekexium@fastmail.com>
Co-authored-by: disksing <i@disksing.com>
2022-06-30 15:19:23 +08:00
75d3f86f3b
unionstore: add checkpoint API to support TiDB savepoint feature ( #490 )
...
Signed-off-by: crazycs520 <crazycs520@gmail.com>
2022-05-30 13:58:57 +08:00
3984ffee09
unionstore: make lock earlier ( #484 )
...
Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>
2022-04-29 12:57:25 +08:00
577843f1b9
internal/unionstore: avoid panic and add log for tidb issue/34307 ( #482 )
...
Signed-off-by: tiancaiamao <tiancaiamao@gmail.com>
2022-04-28 14:21:51 +08:00
300275dee6
fix data race and panic when exiting ( #396 )
...
* fix the data race when exiting
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
* remove error and change log level
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
* add warn log
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Yilin Chen <sticnarf@gmail.com>
2021-12-23 14:21:59 +08:00
350b6ce9fb
Add an RemoveFromBuffer method for test ( #387 )
...
* export DeleteKey
Signed-off-by: ekexium <ekexium@gmail.com>
* rename DeleteKey to UnsafeRemoveRecord
Signed-off-by: ekexium <ekexium@gmail.com>
* rename the old DeleteKey
Signed-off-by: ekexium <ekexium@gmail.com>
2021-11-30 19:43:54 +08:00
843a5378aa
add readable flag ( #320 )
...
Signed-off-by: you06 <you1474600@gmail.com>
2021-10-09 04:30:03 +00:00
f4a640e8e0
rechange to retry when tikv disk full ( #302 )
...
Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>
2021-09-15 13:01:35 +08:00
288c649636
fix license ( #267 )
...
Signed-off-by: disksing <i@disksing.com>
2021-08-15 01:48:01 +08:00
14892a598e
*: allow to custom disk_full_opt for transactions ( #256 )
...
Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>
2021-08-06 16:56:16 +08:00
you06