tikv
/
website
mirror of https://github.com/tikv/website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

polish style 

Signed-off-by: pingyu <yuping@pingcap.com>
This commit is contained in:
pingyu 2022-11-17 10:16:40 +08:00
parent 10e86ab540
commit c3e505b6c1
1 changed files with 38 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
content/docs/dev/concepts/explore-tikv-features/cdc/cdc.md
View File

@ -18,7 +18,9 @@ This page introduces what's RawKV Change Data Capture and how to use it.
By RawKV CDC, you can build a storage system with **Cross Cluster Replication**, to implement financial-level disaster recovery. By RawKV CDC, you can build a storage system with **Cross Cluster Replication**, to implement financial-level disaster recovery.
To use RawKV CDC, you need to enable [TiKV API V2] and deploy a **TiKV-CDC** cluster. The minimal required version of TiKV is [v6.2.0]. {{< info >}}
To use RawKV CDC, you need to enable [TiKV API V2](../../api-v2) and deploy a **TiKV-CDC** cluster. The minimal required version of TiKV is [v6.2.0](https://docs.pingcap.com/tidb/v6.2/release-6.2.0).
{{< /info >}}
{{< figure {{< figure
src="/img/docs/rawkv-cdc.png" src="/img/docs/rawkv-cdc.png"
@ -42,7 +44,9 @@ It forks from [TiCDC](https://github.com/pingcap/tiflow/blob/master/README_TiCDC
#### Deploy by TiUP #### Deploy by TiUP
_Note: [TiUP] >= `v1.11.0` is required._ {{< info >}}
[TiUP](https://tiup.io) >= v1.11.0 is required.
{{< /info >}}
##### Deploy a new TiDB/TiKV cluster including TiKV-CDC ##### Deploy a new TiDB/TiKV cluster including TiKV-CDC
@ -52,10 +56,10 @@ When you deploy a new TiDB/TiKV cluster using TiUP, you can also deploy TiKV-CDC
You can also use TiUP to add the TiKV-CDC component to an existing TiDB/TiKV cluster. Take the following procedures: You can also use TiUP to add the TiKV-CDC component to an existing TiDB/TiKV cluster. Take the following procedures:
1. Make sure that the current TiDB/TiKV version >= `6.2.0` and [TiKV API V2] is enabled. 1. Make sure that the current TiDB/TiKV version >= 6.2.0 and [TiKV API V2] is enabled.
2. Prepare a scale-out configuration file, refer to [template](https://github.com/tikv/migration/blob/main/cdc/deployments/tikv-cdc/config-templates/scale-out.example.yaml). 2. Prepare a scale-out configuration file, refer to [template](https://github.com/tikv/migration/blob/main/cdc/deployments/tikv-cdc/config-templates/scale-out.example.yaml).
3. Scale out by `tiup cluster scale-out`. Also refer to [Scale a TiDB Cluster Using TiUP](https://docs.pingcap.com/tidb/stable/scale-tidb-using-tiup). 3. Scale out by `tiup cluster scale-out`. Also refer to [Scale a TiDB Cluster Using TiUP](https://docs.pingcap.com/tidb/stable/scale-tidb-using-tiup).
``` ```bash
tiup cluster scale-out <cluster-name> scale-out.yaml tiup cluster scale-out <cluster-name> scale-out.yaml
``` ```
@ -93,10 +97,10 @@ tiup tikv-cdc cli --help
#### Manage TiKV-CDC service (`capture`) #### Manage TiKV-CDC service (`capture`)
##### Query the `capture` list ##### Query the `capture` list
``` ```bash
tikv-cdc cli capture list --pd=http://192.168.100.122:2379 tikv-cdc cli capture list --pd=http://192.168.100.122:2379
``` ```
``` ```bash
[ [
{ {
"id": "07684765-52df-42a0-8dd1-a4e9084bb7c1", "id": "07684765-52df-42a0-8dd1-a4e9084bb7c1",
@ -123,7 +127,7 @@ In the result above:
* `address`: The address to access to. * `address`: The address to access to.
If TLS is required: If TLS is required:
``` ```bash
tikv-cdc cli capture list --pd=http://192.168.100.122:2379 --ca=$TLS_DIR/ca.pem --cert=$TLS_DIR/client.pem --key=$TLS_DIR/client-key.pem tikv-cdc cli capture list --pd=http://192.168.100.122:2379 --ca=$TLS_DIR/ca.pem --cert=$TLS_DIR/client.pem --key=$TLS_DIR/client-key.pem
``` ```
@ -135,10 +139,10 @@ In the command above:
#### Manage Replication Tasks (`changefeed`) #### Manage Replication Tasks (`changefeed`)
##### Create a replication task ##### Create a replication task
``` ```bash
tikv-cdc cli changefeed create --pd=http://192.168.100.122:2379 --sink-uri="tikv://192.168.100.61:2379/" --changefeed-id="rawkv-replication-task" --start-ts=434716089136185435 tikv-cdc cli changefeed create --pd=http://192.168.100.122:2379 --sink-uri="tikv://192.168.100.61:2379/" --changefeed-id="rawkv-replication-task" --start-ts=434716089136185435
``` ```
``` ```bash
Create changefeed successfully! Create changefeed successfully!
ID: rawkv-replication-task ID: rawkv-replication-task
Info: {"sink-uri":"tikv://192.168.100.61:2379","opts":{},"create-time":"2022-07-20T15:35:47.860947953+08:00","start-ts":434714063103852547,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","scheduler":{"type":"keyspan-number","polling-time":-1},"state":"normal","history":null,"error":null} Info: {"sink-uri":"tikv://192.168.100.61:2379","opts":{},"create-time":"2022-07-20T15:35:47.860947953+08:00","start-ts":434714063103852547,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","scheduler":{"type":"keyspan-number","polling-time":-1},"state":"normal","history":null,"error":null}
@ -156,10 +160,12 @@ In the command and result above:
* `--start-ts`: Specifies the starting TSO of the changefeed. TiKV-CDC will replicate RawKV entries starting from this TSO. The default value is the current time. * `--start-ts`: Specifies the starting TSO of the changefeed. TiKV-CDC will replicate RawKV entries starting from this TSO. The default value is the current time.
> Refer to [How to replicate TiKV cluster with existing data](#how-to-replicate-tikv-cluster-with-existing-data) if the replication is deployed on a existing cluster. {{< info >}}
Refer to [How to replicate TiKV cluster with existing data](#how-to-replicate-tikv-cluster-with-existing-data) if the replication is deployed on a existing cluster.
{{< /info >}}
##### Configure sink URI with `tikv` ##### Configure sink URI with `tikv`
``` ```bash
--sink-uri="tikv://192.168.100.61:2379/" --sink-uri="tikv://192.168.100.61:2379/"
``` ```
| Parameter/Parameter Value | Description | | Parameter/Parameter Value | Description |
@ -167,7 +173,7 @@ In the command and result above:
| 192.168.100.61:2379 | The endpoints of the downstream PD. Multiple addresses are separated by comma. | | 192.168.100.61:2379 | The endpoints of the downstream PD. Multiple addresses are separated by comma. |
If TLS is required: If TLS is required:
``` ```bash
--sink-uri="tikv://192.168.100.61:2379/?ca-path=$TLS_DIR/ca.pem&cert-path=$TLS_DIR/client.pem&key-path=$TLS_DIR/client-key.pem" --sink-uri="tikv://192.168.100.61:2379/?ca-path=$TLS_DIR/ca.pem&cert-path=$TLS_DIR/client.pem&key-path=$TLS_DIR/client-key.pem"
``` ```
| Parameter/Parameter Value | Description | | Parameter/Parameter Value | Description |
@ -178,10 +184,10 @@ If TLS is required:
| key-path | The path of the private key file in PEM format. | | key-path | The path of the private key file in PEM format. |
##### Query the replication task list ##### Query the replication task list
``` ```bash
tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379
``` ```
``` ```bash
[ [
{ {
"id": "rawkv-replication-task", "id": "rawkv-replication-task",
@ -206,10 +212,10 @@ In the result above:
##### Query a specific replication task ##### Query a specific replication task
``` ```bash
tikv-cdc cli changefeed query -s --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed query -s --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379
``` ```
``` ```bash
{ {
"state": "normal", "state": "normal",
"tso": 434716089136185435, "tso": 434716089136185435,
@ -227,10 +233,10 @@ In the command and result above:
* `error` records whether an error has occurred in the current changefeed. * `error` records whether an error has occurred in the current changefeed.
``` ```bash
tikv-cdc cli changefeed query --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed query --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379
``` ```
``` ```bash
{ {
"info": { "info": {
"sink-uri": "tikv://192.168.100.61:2379/", "sink-uri": "tikv://192.168.100.61:2379/",
@ -294,11 +300,11 @@ In the result above:
##### Pause a replication task ##### Pause a replication task
``` ```bash
tikv-cdc cli changefeed pause --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed pause --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379
tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379
``` ```
``` ```bash
[ [
{ {
"id": "rawkv-replication-task", "id": "rawkv-replication-task",
@ -318,11 +324,11 @@ In the command above:
##### Resume a replication task ##### Resume a replication task
``` ```bash
tikv-cdc cli changefeed resume --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed resume --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379
tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379
``` ```
``` ```bash
[ [
{ {
"id": "rawkv-replication-task", "id": "rawkv-replication-task",
@ -337,19 +343,19 @@ tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379
``` ```
##### Remove a replication task ##### Remove a replication task
``` ```bash
tikv-cdc cli changefeed remove --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed remove --changefeed-id=rawkv-replication-task --pd=http://192.168.100.122:2379
tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379 tikv-cdc cli changefeed list --pd=http://192.168.100.122:2379
``` ```
``` ```bash
[] []
``` ```
#### Query processing units of replication sub-tasks (processor) #### Query processing units of replication sub-tasks (processor)
``` ```bash
tikv-cdc cli processor list --pd=http://192.168.100.122:2379` tikv-cdc cli processor list --pd=http://192.168.100.122:2379`
``` ```
``` ```bash
[ [
{ {
"changefeed_id": "rawkv-replication-task", "changefeed_id": "rawkv-replication-task",
@ -366,15 +372,18 @@ Use [TiKV-BR] to migrate the existing data to downstream cluster (network shared
We don't recommend replicating existing data by TiKV-CDC because: We don't recommend replicating existing data by TiKV-CDC because:
- First, as life time of garbage collection is short (defaults to `10` minutes), in most circumstance it's not applicable to perform the replication. You can not create a changefeed with `start-ts` earlier than **GC Safe Point**. - First, as life time of garbage collection is short (defaults to 10 minutes), in most circumstance it's not applicable to perform the replication. You can not create a changefeed with `start-ts` earlier than **GC Safe Point**.
- Second, if there are mass existing data, replication by TiKV-CDC is inefficiency, as all existing data must be gathered, hold, and sorted in TiKV-CDC, before finally write to downstream. By contrast, TiKV-BR can utilize the power of the whole cluster, as all regions are directly exported to and imported from the shared storage. - Second, if there are mass existing data, replication by TiKV-CDC is inefficiency, as all existing data must be gathered, hold, and sorted in TiKV-CDC, before finally write to downstream. By contrast, TiKV-BR can utilize the power of the whole cluster, as all regions are directly exported to and imported from the shared storage.
To replicate TiKV cluster with existing data: To replicate TiKV cluster with existing data:
1) Backup upstream cluster by TiKV-BR, with a long enough `--gcttl`. See [Backup Raw Data] for more details. 1) Backup upstream cluster by TiKV-BR, with a long enough `--gcttl`. See [Backup Raw Data] for more details.
> NOTE: value of `gcttl` should include duration of backup, restoration, and other preparation work. If you are not sure about the value of `gcttl`, you can [disable GC] temporarily, and enable it after changefeed has started. > NOTE: value of `--gcttl` should include duration of backup, restoration, and other preparation work. If you are not sure about the value of `--gcttl`, you can disable GC temporarily (`SET GLOBAL tidb_gc_enable = "OFF";`, see [tidb_gc_enable]), and enable it after changefeed has started (`SET GLOBAL tidb_gc_enable = "ON";`).
2) Record `backup-ts` from backup result in *Step 1*. 2) Record `backup-ts` from backup result in *Step 1*.
3) Restore to downstream cluster. Refer to [Restore Raw Data]. 3) Restore to downstream cluster. Refer to [Restore Raw Data].
4) Create changefeed with `--start-ts=<backup-ts>`. 4) Create changefeed with `--start-ts=<backup-ts>`.
@ -387,5 +396,5 @@ To replicate TiKV cluster with existing data:
[S3]: https://aws.amazon.com/s3/ [S3]: https://aws.amazon.com/s3/
[Backup Raw Data]: ../../backup-restore/#backup-raw-data [Backup Raw Data]: ../../backup-restore/#backup-raw-data
[Restore Raw Data]: ../../backup-restore/#restore-raw-data [Restore Raw Data]: ../../backup-restore/#restore-raw-data
[Disable GC]: https://docs.pingcap.com/tidb/stable/system-variables#tidb_gc_enable-new-in-v50 [tidb_gc_enable]: https://docs.pingcap.com/tidb/stable/system-variables#tidb_gc_enable-new-in-v50
[中文使用手册]: ../cdc-cn [中文使用手册]: ../cdc-cn