diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index b0840d6377..9e52b0ea05 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -1,15 +1,17 @@ # Summary + + - [Introduction](./introduction/introduction.md) - [Start With Examples](./examples/introduction.md) - [Quick Start](./examples/quick-start.md) - [Interact with TiKV RawKV API](./examples/rawkv.md) - [Interact with TiKV TxnKV API](./examples/txnkv.md) - - [TiKV RawKV Bulk Load](./examples/bulk-load.md) + - [TiKV RawKV Bulk Load]() (./examples/bulk-load.md) -- [Performance](./performance/introduction.md) - - [YCSB Benchmarks](./performance/ycsb.md) +- [Performance]() (./performance/introduction.md) + - [YCSB Benchmarks]() (./performance/ycsb.md) - [Administration](./administration/introduction.md) - [Configuration](./administration/configuration.md) @@ -17,12 +19,12 @@ - [Troubleshooting](./troubleshooting/introduction.md) - [Slow Request Diagnosis](./troubleshooting/slow-request.md) - - [Error Request Diagnosis](./troubleshooting/error-request.md) + - [Error Request Diagnosis]() (./troubleshooting/error-request.md) - [Architecture and Internals](./architecture/introduction.md) - [The Lifecycle of A Request](./architecture/request-lifecycle.md) - [Backoff and Retry Policy](./architecture/availability.md) - - [Slow Log and Metrics](./architecture/observability.md) - - [Region Cache](./architecture/region-cache.md) + - [Slow Log and Metrics]() (./architecture/observability.md) + - [Region Cache]() (./architecture/region-cache.md) - [Contribution Guide](./contribution/introduction.md) diff --git a/docs/src/architecture/availability.md b/docs/src/architecture/availability.md index 2a2146c821..34c1f019ea 100644 --- a/docs/src/architecture/availability.md +++ b/docs/src/architecture/availability.md @@ -1 +1,47 @@ # Availability: Backoff and Retry Policy + +## BackOffer + +The retry and timeout mechanism for a request is controlled by a `BackOffer` object, which is created one per `RawKVClient` method. The `BackOffer` will decide how much time the next sleep and retry should spend, and whether to timeout the request if not enough time is left for retrying the request. +If we need a back off sleep, we call backOffer.doBackOff(funcType, exception), and the current thread will sleep for a decided time. If the current operation will timeout after sleep, the doBackOff simply throw an exception to abort the operation. + +## callWithRetry + +RegionStoreClient.callWithRetry inherits from AbstractGRPCClient.callWithRetry. The concrete logic is in RetryPolicy.callWithRetry, which implements a retry mechanism, but the specific retry strategy is determined by the ErrorHandler. +ErrorHandler’s handler{Request, Response}Error function returns a boolean value indicating whether to retry inside callWithRetry. +The control flow for callWithRetry is as follows: + +![callWithRetry](./callWithRetry.jpg) + +The error handler is chosen obeying the following table: + +| gPRC request | the result | handler | +| -- | -- | -- | +| throws exception | - | handleRequestError +| no exception | is null | handleRequestError +| no exception | is error | handleResponseError +| no exception | normal | normal return + +The handleRequestError function copes with the following situations: + +| situation | retry within callWithRetry | note | +|----------|---------------|------------------------------| +| invalid store in region manager | true | refresh ClientStub | +| region has not got multiple copies | false | | +| successfully switched to new leader | true | | +| seekProxyStore | true if success | only when `tikv.enable_grpc_forward` is set | +| other | false | | + +The handleResponseError function copes with the following gRPC errors: + +| error | retry within callWithRetry | +|----------------------|----------------------------| +| NotLeader | true if leader unchanged | +| StoreNotMatch | false | +| EphochNotMatch | true if region epoch in `ctx` is ahead of TiKV's | +| ServerIsBusy | true | +| StaleCommand | true | +| RaftEntryTooLarge | throw | +| KeyNotInRegion | throw | +| Raft ProposalDropped | true | +| other | false | \ No newline at end of file diff --git a/docs/src/architecture/callWithRetry.jpg b/docs/src/architecture/callWithRetry.jpg new file mode 100644 index 0000000000..a476eae595 Binary files /dev/null and b/docs/src/architecture/callWithRetry.jpg differ diff --git a/docs/src/architecture/introduction.md b/docs/src/architecture/introduction.md index c79bec1ac6..3d1f5d86dd 100644 --- a/docs/src/architecture/introduction.md +++ b/docs/src/architecture/introduction.md @@ -1 +1,3 @@ # Architecture + +This section includes in-depthA description of the client architecture. \ No newline at end of file diff --git a/docs/src/architecture/request-lifecycle.md b/docs/src/architecture/request-lifecycle.md index a95445a276..108db567de 100644 --- a/docs/src/architecture/request-lifecycle.md +++ b/docs/src/architecture/request-lifecycle.md @@ -1 +1,7 @@ # The Lifecycle of A Request + +![time graph](./timegraph.png) + +The client talks to TiKV store directly using gRPC requests, which are created in RegionStoreClient. If a request failed, the client could retry after a back off sleep. The retry logic is delegated to AbstractGRPCClient::callWithRetry method. callWithRetry may decide to retry request within the function, or, if the RegionStoreClient must be recreated (due to, for example, region split), return a failure and let outermost RawKVClient to do the retry. + +![request-overview](./request-overview.jpg) diff --git a/docs/src/architecture/request-overview.jpg b/docs/src/architecture/request-overview.jpg new file mode 100644 index 0000000000..2bbfa00b2d Binary files /dev/null and b/docs/src/architecture/request-overview.jpg differ diff --git a/docs/src/architecture/timegraph.png b/docs/src/architecture/timegraph.png new file mode 100644 index 0000000000..3aae5de893 Binary files /dev/null and b/docs/src/architecture/timegraph.png differ diff --git a/docs/src/contribution/introduction.md b/docs/src/contribution/introduction.md index 1d66832d32..b0e6a7c745 100644 --- a/docs/src/contribution/introduction.md +++ b/docs/src/contribution/introduction.md @@ -1,11 +1,18 @@ +# Contribution Guide -## How to build +## Build the package ``` mvn clean package -Dmaven.test.skip=true ``` -## How to run test +## Install the package to local maven repository + +``` +mvn clean install -Dmaven.test.skip=true +``` + +## Run tests ``` export RAWKV_PD_ADDRESSES=127.0.0.1:2379 diff --git a/docs/src/examples/introduction.md b/docs/src/examples/introduction.md index a48e8db34a..e1752e5187 100644 --- a/docs/src/examples/introduction.md +++ b/docs/src/examples/introduction.md @@ -1 +1,3 @@ # Start With Examples + +This section contains examples to demonstrate basic usages of the java client. \ No newline at end of file diff --git a/docs/src/examples/quick-start.md b/docs/src/examples/quick-start.md index e69de29bb2..36078f2673 100644 --- a/docs/src/examples/quick-start.md +++ b/docs/src/examples/quick-start.md @@ -0,0 +1,109 @@ +# Quick Start + +The package is hosted on maven central repository. To build from source, refer to the [Contribution Guide](../contribution/introduction.html). + +## Create a maven project + +First download [maven] and follow the [installation instructoins][install]. Then `mvn` command should be available in the `$PATH`. + +[maven]: https://maven.apache.org/download.html +[install]: https://maven.apache.org/install.html + +create a maven project by following command: + +``` +mvn archetype:generate -DgroupId=com.example -DartifactId=java-client-example -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false +cd java-client-example +``` + +## Add dependency + +Add maven dependency to `pom.xml`. + +```xml + + org.tikv + tikv-client-java + 3.1.0 + + + org.slf4j + slf4j-log4j12 + 1.7.32 + +``` + +Now `pom.xml` should look like this: + +```xml + + 4.0.0 + com.example + java-project + jar + 1.0-SNAPSHOT + java-project + http://maven.apache.org + + 1.8 + 1.8 + + + + junit + junit + 4.12 + test + + + org.tikv + tikv-client-java + 3.1.0 + + + org.slf4j + slf4j-log4j12 + 1.7.32 + + + +``` + +## Writing code + +To interact with TiKV, we should first create a `TiConfiguration` with PD address, create a `TiSession` using `TiSession.create`, and then create a client. +For example, if we want to put a `World` in `Hello` key in RawKV, write the following code in `src/main/java/com/example/App.java`. + +```java +import org.tikv.common.TiConfiguration; +import org.tikv.common.TiSession; +import org.tikv.raw.RawKVClient; +import org.tikv.shade.com.google.protobuf.ByteString; + +public class App { + public static void main(String[] args) throws Exception { + String pdAddr = "127.0.0.1:2379"; + // You MUST create a raw configuration if you are using RawKVClient. + TiConfiguration conf = TiConfiguration.createRawDefault(pdAddr); + try (TiSession session = TiSession.create(conf)) { + try (RawKVClient client = session.createRawClient()) { + client.put(ByteString.copyFromUtf8("Hello"), ByteString.copyFromUtf8("World")); + ByteString value = client.get(ByteString.copyFromUtf8("Hello")); + System.out.println(value); + } + } + } +} +``` + +More examples for RawKV and TxnKV are in following chapters. + +## Running program + +Run following command: + +``` +mvn assembly:assembly -DdescriptorId=jar-with-dependencies +java -cp target/java-client-example-1.0-SNAPSHOT-jar-with-dependencies.jar com.example.App +``` diff --git a/docs/src/examples/rawkv.md b/docs/src/examples/rawkv.md index e69de29bb2..41ea662942 100644 --- a/docs/src/examples/rawkv.md +++ b/docs/src/examples/rawkv.md @@ -0,0 +1,51 @@ +# RawKV + +Below is the basic usages of RawKV. See [API document] to see a full list of methods available. + +[API document]: https://tikv.github.io/client-java/apidocs/org/tikv/raw/RawKVClient + +```java +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import org.tikv.common.TiConfiguration; +import org.tikv.common.TiSession; +import org.tikv.kvproto.Kvrpcpb; +import org.tikv.raw.RawKVClient; +import org.tikv.shade.com.google.protobuf.ByteString; + +public class Main { + public static void main() { + // You MUST create a raw configuration if you are using RawKVClient. + TiConfiguration conf = TiConfiguration.createRawDefault("127.0.0.1:2379"); + TiSession session = TiSession.create(conf); + RawKVClient client = session.createRawClient(); + + // put + client.put(ByteString.copyFromUtf8("k1"), ByteString.copyFromUtf8("Hello")); + client.put(ByteString.copyFromUtf8("k2"), ByteString.copyFromUtf8(",")); + client.put(ByteString.copyFromUtf8("k3"), ByteString.copyFromUtf8("World")); + client.put(ByteString.copyFromUtf8("k4"), ByteString.copyFromUtf8("!")); + client.put(ByteString.copyFromUtf8("k5"), ByteString.copyFromUtf8("Raw KV")); + + // get + Optional result = client.get(ByteString.copyFromUtf8("k1")); + System.out.println(result.get().toStringUtf8()); + + // batch get + List list = client.batchGet(new ArrayList() {{ + add(ByteString.copyFromUtf8("k1")); + add(ByteString.copyFromUtf8("k3")); + }}); + System.out.println(list); + + // scan + list = client.scan(ByteString.copyFromUtf8("k1"), ByteString.copyFromUtf8("k6"), 10); + System.out.println(list); + + // close + client.close(); + session.close(); + } +} +``` \ No newline at end of file diff --git a/docs/src/examples/txnkv.md b/docs/src/examples/txnkv.md index e69de29bb2..9bcfb430c4 100644 --- a/docs/src/examples/txnkv.md +++ b/docs/src/examples/txnkv.md @@ -0,0 +1,69 @@ +# TxnKV + +Below is the basic usages of TxnKV. +Data should be written into TxnKV using [`TwoPhaseCommitter`], and be read using [`org.tikv.txn.KVClient`][KVClient]. + +[`TwoPhaseCommitter`]: https://tikv.github.io/client-java/apidocs/org/tikv/txn/TwoPhaseCommitter.html +[KVClient]: https://tikv.github.io/client-java/apidocs/org/tikv/txn/KVClient.html + +```java +import java.util.Arrays; +import java.util.List; + +import org.tikv.common.BytePairWrapper; +import org.tikv.common.ByteWrapper; +import org.tikv.common.TiConfiguration; +import org.tikv.common.TiSession; +import org.tikv.common.util.BackOffer; +import org.tikv.common.util.ConcreteBackOffer; +import org.tikv.kvproto.Kvrpcpb.KvPair; +import org.tikv.shade.com.google.protobuf.ByteString; +import org.tikv.txn.KVClient; +import org.tikv.txn.TwoPhaseCommitter; + +public class App { + public static void main(String[] args) throws Exception { + TiConfiguration conf = TiConfiguration.createDefault("127.0.0.1:2389"); + try (TiSession session = TiSession.create(conf)) { + // two-phrase write + long startTS = session.getTimestamp().getVersion(); + try (TwoPhaseCommitter twoPhaseCommitter = new TwoPhaseCommitter(session, startTS)) { + BackOffer backOffer = ConcreteBackOffer.newCustomBackOff(1000); + byte[] primaryKey = "key1".getBytes("UTF-8"); + byte[] key2 = "key2".getBytes("UTF-8"); + + // first phrase: prewrite + twoPhaseCommitter.prewritePrimaryKey(backOffer, primaryKey, "val1".getBytes("UTF-8")); + List pairs = Arrays + .asList(new BytePairWrapper(key2, "val2".getBytes("UTF-8"))); + twoPhaseCommitter.prewriteSecondaryKeys(primaryKey, pairs.iterator(), 1000); + + // second phrase: commit + long commitTS = session.getTimestamp().getVersion(); + twoPhaseCommitter.commitPrimaryKey(backOffer, primaryKey, commitTS); + List keys = Arrays.asList(new ByteWrapper(key2)); + twoPhaseCommitter.commitSecondaryKeys(keys.iterator(), commitTS, 1000); + } + + try (KVClient kvClient = session.createKVClient()) { + long version = session.getTimestamp().getVersion(); + ByteString key1 = ByteString.copyFromUtf8("key1"); + ByteString key2 = ByteString.copyFromUtf8("key2"); + + // get value of a single key + ByteString val = kvClient.get(key1, version); + System.out.println(val); + + // get value of multiple keys + BackOffer backOffer = ConcreteBackOffer.newCustomBackOff(1000); + List kvPairs = kvClient.batchGet(backOffer, Arrays.asList(key1, key2), version); + System.out.println(kvPairs); + + // get value of a range of keys + kvPairs = kvClient.scan(key1, ByteString.copyFromUtf8("key3"), version); + System.out.println(kvPairs); + } + } + } +} +``` \ No newline at end of file diff --git a/docs/src/troubleshooting/slow-request.md b/docs/src/troubleshooting/slow-request.md index 1de796a0ab..798acf2838 100644 --- a/docs/src/troubleshooting/slow-request.md +++ b/docs/src/troubleshooting/slow-request.md @@ -1 +1,34 @@ # Slow Request Diagnosis + +If a request take too much time, we can collect the detailed time spend in each component in a “slow log”. + + +
+
+2022-02-11 11:07:56 WARN  SlowLogImpl:88 - A request spent 55 ms. start=11:07:56.938, end=11:07:56.993, SlowLog:{"trace_id":4361090673996453790,"spans":[{"event":"put","begin":"11:07:56.938","duration_ms":55,"properties":{"region":"{Region[2] ConfVer[5] Version[60] Store[1] KeyRange[]:[]}","key":"Hello"}},{"event":"getRegionByKey","begin":"11:07:56.938","duration_ms":0},{"event":"callWithRetry","begin":"11:07:56.943","duration_ms":49,"properties":{"method":"tikvpb.Tikv/RawPut"}},{"event":"gRPC","begin":"11:07:56.943","duration_ms":49,"properties":{"method":"tikvpb.Tikv/RawPut"}}]}
+
+
+ +## Slow log configurations + +| SlowLog settings | default value | +| -- | -- | +| tikv.rawkv.read_slowlog_in_ms | tikv.grpc.timeout_in_ms * 2 | +| tikv.rawkv.write_slowlog_in_ms | tikv.grpc.timeout_in_ms * 2 | +| tikv.rawkv.batch_read_slowlog_in_ms | tikv.grpc.timeout_in_ms * 2 | +| tikv.rawkv.batch_write_slowlog_in_ms | tikv.grpc.timeout_in_ms * 2 | +| tikv.rawkv.scan_slowlog_in_ms | 5s | + +Each settings can be set by system properties, configuration files or `set...` method of `TiConfiguration`. + +System properties can be set by `-D` parameter of `java` command. + +``` +java -cp target/java-client-example-1.0-SNAPSHOT-jar-with-dependencies.jar -Dtikv.rawkv.read_slowlog_in_ms=100 com.example.App +``` + +Configuration file is `src/main/resources/tikv.properties` in maven projects. + +## Visualize slow log + +TBD \ No newline at end of file diff --git a/src/main/java/org/tikv/common/TiSession.java b/src/main/java/org/tikv/common/TiSession.java index a8d6fe25c7..968b214da8 100644 --- a/src/main/java/org/tikv/common/TiSession.java +++ b/src/main/java/org/tikv/common/TiSession.java @@ -49,8 +49,10 @@ import org.tikv.txn.TxnKVClient; /** * TiSession is the holder for PD Client, Store pdClient and PD Cache All sessions share common - * region store connection pool but separated PD conn and cache for better concurrency TiSession is - * thread-safe but it's also recommended to have multiple session avoiding lock contention + * region store connection pool but separated PD conn and cache for better concurrency + * + *

TiSession is thread-safe but it's also recommended to have multiple session avoiding lock + * contention */ public class TiSession implements AutoCloseable {