mirror of https://github.com/tikv/website.git
Add documentation for export (#308)
Signed-off-by: Peng Guanwen <pg999w@outlook.com>
This commit is contained in:
parent
5caf80b50b
commit
b049c359cc
|
|
@ -11,4 +11,31 @@ This document, like our Java API, is still a work in progress. In the meantime,
|
|||
|
||||
{{< warning >}}
|
||||
You should not use the Java client for production use until it is released.
|
||||
{{< /warning >}}
|
||||
{{< /warning >}}
|
||||
|
||||
## Parse SST file
|
||||
|
||||
```java
|
||||
import java.io.File;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
|
||||
import org.tikv.br.BackupDecoder;
|
||||
import org.tikv.br.BackupMetaDecoder;
|
||||
import org.tikv.br.SSTDecoder;
|
||||
import org.tikv.common.util.Pair;
|
||||
|
||||
BackupMetaDecoder metaDecoder = BackupMetaDecoder.parse("/path/to/sst/backupmeta");
|
||||
BackupDecoder sstBackup = new BackupDecoder(metaDecoder.getBackupMeta());
|
||||
File folder = new File("/path/to/sst");
|
||||
File[] listOfFiles = folder.listFiles();
|
||||
|
||||
for (File file : listOfFiles) {
|
||||
if (file.getName().endsWith("sst")) {
|
||||
SSTDecoder decoder = sstBackup.decodeSST(file.getAbsolutePath());
|
||||
Iterator<Pair<ByteString, ByteString>> iter = decoder.getIterator();
|
||||
// Do something with iter
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,63 @@
|
|||
---
|
||||
title: Export
|
||||
description: Export data to SST
|
||||
menu:
|
||||
"dev":
|
||||
parent: Tasks
|
||||
---
|
||||
|
||||
This document describes how to use [BR Raw KV backup](br-rawkv) capability to export KV data to remote storages as SST files.
|
||||
|
||||
[br-rawkv]: https://docs.pingcap.com/tidb/stable/use-br-command-line-tool#restore-raw-kv-experimental-feature
|
||||
|
||||
# Basic Usage
|
||||
|
||||
```bash
|
||||
br backup raw --pd ⟨pd address⟩ \
|
||||
-s ⟨storage url⟩ \
|
||||
--start ⟨start key⟩ \
|
||||
--end ⟨end key⟩ \
|
||||
--format ⟨key format⟩ \
|
||||
--ratelimit ⟨in MiB/s⟩
|
||||
```
|
||||
|
||||
This will export all KV data in the range `[start key, end key)` to the specified storage in the format of SST.
|
||||
|
||||
## Supported storage
|
||||
|
||||
The storage URL support following schemes:
|
||||
|
||||
| Service | Scheme | Example |
|
||||
|---------------------------------------------|---------|-------------------------------------|
|
||||
| Local filesystem, distributed on every node | local | `local:///path/to/dest/` |
|
||||
| Hadoop HDFS and other compatible services | hdfs | `hdfs:///prefix/of/dest/` |
|
||||
| Amazon S3 and other compatible services | s3 | `s3://bucket-name/prefix/of/dest/` |
|
||||
| GCS | gcs, gs | `gcs://bucket-name/prefix/of/dest/` |
|
||||
| Write to nowhere(for benchmark only) | noop | `noop://` |
|
||||
|
||||
S3 and GCS can be configured using URL and command line parameters, see the BR documentation [External Storage](br-external-storage) for more information.
|
||||
|
||||
[br-external-storage]: https://docs.pingcap.com/tidb/stable/backup-and-restore-storages
|
||||
|
||||
### HDFS configuration
|
||||
|
||||
To use HDFS storage, [Apache Hadoop](hdfs-client) or compatible client should be installed and currectly configured on all BR and TiKV machines. The `bin/hdfs` binary in hadoop installation will be used by BR and TiKV.
|
||||
|
||||
Various configuration should be provided for HDFS storage to work, see the following table.
|
||||
|
||||
| Component | Configuration | Environment variable | Configuration file item |
|
||||
|-----------|---------------------------------------|----------------------|--------------------------|
|
||||
| BR | hadoop installation directory | HADOOP_HOME | (None) |
|
||||
| TiKV | hadoop installation directory | HADOOP_HOME | backup.hadoop.home |
|
||||
| TiKV | linux user to use when calling hadoop | HADOOP_LINUX_USER | backup.hadoop.linux-user |
|
||||
|
||||
For TiKV, the configuration file have higher priority than environment variables.
|
||||
|
||||
[hdfs-client]: https://hadoop.apache.org/docs/stable/
|
||||
|
||||
### Parse SST file
|
||||
|
||||
#### Java Client
|
||||
|
||||
The exported SST file can be parsed using [Java Client](../../../reference/clients/java).
|
||||
|
||||
Loading…
Reference in New Issue