Commit Graph

1293 Commits

Author SHA1 Message Date
JmPotato 50a938a710
chore(deps): bump Go to 1.23.12 and update tidb-dashboard dependency (#9787)
ref tikv/pd#4399

- Upgraded Go version from 1.23.0 → 1.23.12 to adopt security fixes.
- Updated github.com/pingcap/tidb-dashboard to latest commit (e111aaee93).

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-29 03:18:58 +00:00
hujiatao 9450168983
scheduler: graceful shutdown implement (#9720)
close tikv/pd#9719

Add an is_stopping status to the StoreHeartbeat message. When TiKV receives a SIGTERM, it sets this flag. This change adds a new evict-stopping-store-scheduler to PD, which is analogous to the evict-slow-store-scheduler. It proactively transfers leaders away from nodes by inspecting the is_stopping status from store heartbeats.

Signed-off-by: hujiatao0 <hhjjtt110@gmail.com>
2025-09-28 06:46:16 +00:00
okJiang e86b044583
*: Revert resource manager micro service code (#9738)
ref tikv/pd#9737

Signed-off-by: okjiang <819421878@qq.com>
2025-09-28 02:56:40 +00:00
Ryan Leung ec8e27cea0
mcs: fix panic of store not found (#9781)
ref tikv/pd#5839

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-26 03:45:43 +00:00
JmPotato 0b32e6e185
feat(metering): add storage size metering and keyspace name caching (#9774)
ref tikv/pd#9707

- Introduced storage size metering:  
  - Added `storageSizeCollector` to periodically record row-based and column-based storage usage per keyspace.  
  - Integrated with cluster metering writer for reporting.  
- Extended `RegionInfo` and statistics with `ApproximateColumnarKvSize` tracking.  
- Added keyspace name caching and lookup in `KeyspaceManager` for efficient resolution.  
- Extracted common metering utilities (`NewRUValue`, `NewBytesValue`, constants for source/fields) into `pkg/metering/utils.go`.

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-25 10:38:55 +00:00
Ryan Leung f9be8dd186
mcs: support bucket for scheduling service (#9728)
ref tikv/pd#5839

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-25 08:31:07 +00:00
Neil Shen e2f7162d73
chore(dashboard): update TiDB Dashboard to 20250908031600-468ef95996bb (#9770)
ref tikv/pd#4257, close tikv/pd#9405

Upgrade TiDB Dashboard to fix login issue when keyspace is enabled.

Signed-off-by: Neil Shen <overvenus@gmail.com>
2025-09-23 08:13:58 +00:00
Hao W a3ed38ba2e
server/api: Add PD HTTP Endpoints for PiTR Restore Mode Marker (#9655)
close tikv/pd#9638

Signed-off-by: h-wang4 <h.wang4@airbnb.com>

Co-authored-by: h-wang4 <h.wang4@airbnb.com>
2025-09-23 07:27:31 +00:00
Ryan Leung a89e236124
mcs: implement basic framework of router service (#9696)
ref tikv/pd#9212

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-22 05:32:31 +00:00
JmPotato 71f13dd743
feat(metering): update SDK dependency and integrate config-based writer initialization (#9760)
ref tikv/pd#9707

- Bumped `github.com/pingcap/metering_sdk` to latest version.  
- Switched writer initialization from `NewMeteringWriter` to `NewMeteringWriterFromConfig`.

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-18 09:37:41 +00:00
Ryan Leung a3be87a64e
mcs: add health api for scheduling (#9750)
close tikv/pd#5839

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-18 05:44:47 +00:00
JmPotato 056e04c3e3
fix(metering): ensure the order of collector registration and writer startup (#9747)
ref tikv/pd#9707

- Delay registration of RU collector until after the metering writer has been started to ensure correct initialization order.
- Add integration test confirming that the metering writer starts correctly with config, and that RU collector is properly registered.

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-17 05:02:59 +00:00
JmPotato e2ce53841d
feat(metering): integrate RU metering and flush to object storage (#9717)
ref tikv/pd#9707

- Introduced a new **metering writer** (`pkg/metering`) that collects RU usage data and periodically flushes it to object storage via `pingcap/metering_sdk`.
- Implemented a **RU collector** (`ruCollector`) in the resource manager to track resource usage (OLTP/OLAP RU, cross-AZ traffic) per keyspace.

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-12 07:50:03 +00:00
Ryan Leung ebdfd2f291
mcs: refactor prepare checker (#9582)
ref tikv/pd#7671

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-12 06:46:00 +00:00
Ryan Leung ae017feaa0
test: fix `TestRedirect` (#9722)
close tikv/pd#8712

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-08 03:51:42 +00:00
JmPotato 9822378487
refactor(mcs): remove global service registration (#9718)
ref tikv/pd#9707

This change removes the global `ServerServiceRegistry` and the auto-registration `install` packages for microservices, as they are no longer used.

Signed-off-by: JmPotato <github@ipotato.me>
2025-09-04 03:02:12 +00:00
Ryan Leung 7fa31c6635
metrics: clean up metrics (#9535)
ref tikv/pd#9357

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-09-02 06:13:22 +00:00
JmPotato 4b91c02edd
feat(client/http): add support for fetching keyspace metadata by ID (#9695)
ref tikv/pd#7300

Introduce a new PD API endpoint for retrieving keyspace metadata by ID.

Signed-off-by: JmPotato <github@ipotato.me>
2025-08-27 09:00:46 +00:00
okJiang 14eb016237
scheduler: implement network slow store scheduler (#9451)
ref tikv/pd#9359

Signed-off-by: okjiang <819421878@qq.com>
2025-08-25 09:26:07 +00:00
lhy1024 1e3728e623
tests: accelerate testing speed by reducing clusters (Part 2) (#9666)
ref tikv/pd#9596

Signed-off-by: lhy1024 <admin@liudos.us>
2025-08-25 06:36:07 +00:00
Ryan Leung d0a0f38b72
metrics: reduce audit metrics cardinality (#9552)
ref tikv/pd#9357

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-08-18 07:12:38 +00:00
lhy1024 33d470186d
tests: accelerate testing speed by reducing clusters (Part 1) (#9598)
ref tikv/pd#9596

Signed-off-by: lhy1024 <admin@liudos.us>
2025-08-14 12:42:18 +00:00
okJiang 4e36646aac
testutil: fix LastHeartbeat assignment and streamline PutMetaStore call (#9634)
close tikv/pd#9481

test: fix flaky test that store may panic during test

Signed-off-by: okjiang <819421878@qq.com>
2025-08-12 08:20:39 +00:00
lhy1024 a1d89da952
api: distinguish bootstrap state in ready api (#9632)
close tikv/pd#9576

Signed-off-by: lhy1024 <admin@liudos.us>
2025-08-08 06:10:45 +00:00
Ryan Leung 4a225bd8bc
scheduler: remove bucket scheduler (#9621)
close tikv/pd#9622

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-08-07 08:07:46 +00:00
Ryan Leung 1d6cae220c
mcs: improve the log and rename (#9608)
ref tikv/pd#5766

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-08-07 08:07:39 +00:00
MyonKeminta 42471db841
server/api: Return pseudo service safe point for gc_worker to provide compabitility for some existing tests of CDC (#9626)
ref tikv/pd#8978

Make the service safe point HTTP API return a pseudo service safe point for "gc_worker".

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2025-08-06 08:28:51 +00:00
tiancaiamao 7d93890a99
gc: global GC barriers support (#9340)
ref tikv/pd#8978

Signed-off-by: tiancaiamao <tiancaiamao@gmail.com>
2025-08-05 04:07:39 +00:00
okJiang 4d498809b2
placement-rule: move placement rule initialization after LoadStores (#9616)
close tikv/pd#9602

Signed-off-by: okjiang <819421878@qq.com>
2025-08-04 08:13:12 +00:00
lhy1024 6c9e60f271
tests: use temp dir to ensure test isolation (#9603)
ref tikv/pd#9596

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-31 09:30:44 +00:00
Ryan Leung f0c67d8125
mcs: refactor election member interface (part 2) (#9601)
ref tikv/pd#5766

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-07-31 09:16:19 +00:00
tongjian d5d5a03917
ctl: support to query the members and primary by using pdctl (#9564)
close tikv/pd#9567

Signed-off-by: 童剑 <1045931706@qq.com>
2025-07-31 08:18:22 +00:00
Ryan Leung a37367fbc9
mcs: refactor election member interface (part 1) (#9599)
ref tikv/pd#5766

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-07-31 08:01:34 +00:00
Hao W 01d96415c6
maintenance: implement PD maintenance endpoints to serialize TiKV maintenance operations (#9478)
close tikv/pd#9477

Signed-off-by: h-wang4 <h.wang4@airbnb.com>

Co-authored-by: h-wang4 <h.wang4@airbnb.com>
2025-07-31 07:11:28 +00:00
JmPotato ca23c9299d
test(resourcemanager): add service limit integration tests (#9594)
ref tikv/pd#9296

Add integration tests for the service limit.

Signed-off-by: JmPotato <github@ipotato.me>
2025-07-30 03:27:46 +00:00
Ryan Leung 23882d656f
client: fix tso service discovery (#9595)
close tikv/pd#9593

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-07-29 05:36:46 +00:00
lhy1024 5b8c5f09ef
tests: improve tests with a unified `env.Reset` (#9581)
ref tikv/pd#4399

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-29 02:45:28 +00:00
lhy1024 ac5bc19f7b
*: remove `--verbose` in ci-test-job (#9575)
ref tikv/pd#4399

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-25 02:29:28 +00:00
MyonKeminta 0d311729ad
server, client: Deprecate old GC related APIs (#9292)
close tikv/pd#8978

This PR migrates the old GC APIs to the new implementation, and marks them as deprecated.
Also suppresses warnings on necessary invocations to them, such as tests.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
2025-07-24 04:11:01 +00:00
Ryan Leung 59b75f576c
tests: fix stale failpoint (#9573)
ref tikv/pd#4791

Signed-off-by: Ryan Leung <rleungx@gmail.com>
2025-07-23 06:46:17 +00:00
JmPotato ec48862f86
test: stablize the TSO client tests (#9571)
close tikv/pd#9412, close tikv/pd#9551

Stabilize TSO client tests by implementing resilient leader resignation with retries and enhancing the tests to accommodate leader change scenarios.

Signed-off-by: JmPotato <github@ipotato.me>
2025-07-23 02:53:05 +00:00
lhy1024 ca497a48ef
*: make `regionTestSuite` and `storeTestSuite` acceleration (#9550)
ref tikv/pd#4399

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-22 06:28:38 +00:00
JmPotato 9ab2fb2dcb
test: stablize the mcs TSO integration tests (#9568)
close tikv/pd#9501

This PR requires specifying the keyspace group ID when checking the keyspace's primary server,
preventing duplicate keyspace IDs across multiple keyspace groups from disrupting the tests.

Signed-off-by: JmPotato <github@ipotato.me>
2025-07-22 04:24:41 +00:00
tongjian 0bd02197ba
api: fix some wrong metrics and add bucket version (#9561)
close tikv/pd#9557

Signed-off-by: 童剑 <1045931706@qq.com>
2025-07-22 02:34:47 +00:00
lhy1024 24e6dcfabc
test: make TestSplitKeyspaceGroup and TestMergeKeyspaceGroup stable (#9534)
close tikv/pd#9363

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-21 08:53:55 +00:00
Ryan Leung e2abc033f3
*: add leakcheck (#9563)
ref tikv/pd#7782

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
2025-07-18 06:54:33 +00:00
JmPotato 735fcade52
deps(master): update to fix the vulnerabilities (#9559)
close tikv/pd#9556

Update the third-party dependencies to address vulnerabilities.

Signed-off-by: JmPotato <github@ipotato.me>
2025-07-18 03:08:18 +00:00
lhy1024 b34bf56628
test: make TestTransferLeaderForScheduler stable (#9517)
close tikv/pd#9515

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
2025-07-14 06:06:51 +00:00
lhy1024 5507301370
keyspace, tso: disable default keyspace in next-gen (#9499)
ref tikv/pd#9414

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
2025-07-11 09:43:18 +00:00
lhy1024 4bddd64401
*: enable errcheck in test (#9505)
ref tikv/pd#4322

Signed-off-by: lhy1024 <admin@liudos.us>
2025-07-10 09:53:00 +00:00