etcd-io
/
dbtester
mirror of https://github.com/etcd-io/dbtester.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #286 from gyuho/update-with-new-test-results 

*: new test results
This commit is contained in:
Gyu-Ho Lee 2017-02-10 13:52:25 -08:00 committed by GitHub
parent d43ccbf27d 7093e7851a
commit a6cb280e74
11 changed files with 299 additions and 310 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

347
README.md
View File

@ -28,49 +28,39 @@ Snapshot, when writing 1-million entries (256-byte key, 1KB value value), with 3
```
# snapshot warnings
cd 2017Q1-00-etcd-zookeeper-consul/02-write-1M-keys-best-throughput
grep -r -i fsync-ing\ the zookeeper-r3.4.9-java8-* | less
...
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2017-02-06 22:02:19,388 [myid:2] - WARN [SyncThread:2:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:2 took 1083ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-02-06 22:02:20,000 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570038, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570024, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570051, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570044, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570037, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570059, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570039, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570026, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a570013, timeout of 4000ms exceeded
2017-02-06 22:02:20,526 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a57004f, timeout of 4000ms exceeded
2017-02-06 22:02:20,527 [myid:2] - INFO [SessionTracker:ZooKeeperServer@358] - Expiring session 0x15a15722a57005b, timeout of 4000ms exceeded
2017-02-06 22:02:20,527 [myid:2] - WARN [SyncThread:2:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:2 took 1023ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
...
2017-02-10 18:55:38,997 [myid:3] - WARN [SyncThread:3:SyncRequestProcessor@148] - Too busy to snap, skipping
2017-02-10 18:55:38,998 [myid:3] - INFO [SyncThread:3:FileTxnLog@203] - Creating new log file: log.1000c0c51
2017-02-10 18:55:40,855 [myid:3] - INFO [SyncThread:3:FileTxnLog@203] - Creating new log file: log.1000cd2e6
2017-02-10 18:55:40,855 [myid:3] - INFO [Snapshot Thread:FileTxnSnapLog@240] - Snapshotting: 0x1000cd1ca to /home/gyuho/zookeeper/zookeeper.data/version-2/snapshot.1000cd1ca
2017-02-10 18:55:46,382 [myid:3] - WARN [SyncThread:3:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:3 took 1062ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-02-10 18:55:47,471 [myid:3] - WARN [SyncThread:3:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:3 took 1084ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-02-10 18:55:49,425 [myid:3] - WARN [SyncThread:3:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:3 took 1142ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-02-10 18:55:51,188 [myid:3] - WARN [SyncThread:3:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:3 took 1201ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2017-02-10 18:55:52,292 [myid:3] - WARN [SyncThread:3:FileTxnLog@338] - fsync-ing the write ahead log in SyncThread:3 took 1102ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
```
When writing more than 2-million entries (256-byte key, 1KB value value) with 300 concurrent clients
```
# leader election
cd 2017Q1-00-etcd-zookeeper-consul/04-write-too-many-keys
grep -r -i election\ took zookeeper-r3.4.9-java8-* | less
...
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:15:50,966 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 15202
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:16:14,346 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 13354
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:16:24,781 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 6426
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:16:48,327 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 12808
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:16:55,542 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 3207
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:17:19,225 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 13652
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:17:37,581 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 15147
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:18:01,784 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 13368
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:18:20,758 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 15025
zookeeper-r3.4.9-java8-1-database.log:2017-02-06 15:18:43,963 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 13176
...
# leader election is taking more than 10 seconds...
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:22:16,549 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 22978
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:23:02,279 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 10210
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:23:14,498 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 203
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:23:36,303 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 9791
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:23:52,151 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 3836
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:24:13,849 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 9686
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:24:29,694 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 3573
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:24:51,392 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 8686
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:25:07,231 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 3827
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:25:28,940 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 9697
zookeeper-r3.4.9-java8-2-database.log:2017-02-10 19:25:44,772 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 3820
```
@ -81,24 +71,23 @@ Snapshot, when writing 1-million entries (256-byte key, 1KB value value), with 5
```
# snapshot warnings
grep -r -i installed\ remote consul-v0.7.3-go1.7.4-* | less
cd 2017Q1-00-etcd-zookeeper-consul/02-write-1M-keys-best-throughput
grep -r -i installed\ remote consul-v0.7.4-go1.7.5-* | less
2017/02/06 22:04:23 [ERR] raft-net: Failed to flush response: write tcp 10.240.0.31:8300->10.240.0.30:46032: write: connection reset by peer
2017/02/06 22:04:24 [INFO] snapshot: Creating new snapshot at /home/gyuho/consul.data/raft/snapshots/2-740201-1486418664486.tmp
2017/02/06 22:04:26 [INFO] snapshot: reaping snapshot /home/gyuho/consul.data/raft/snapshots/2-688951-1486418642600
2017/02/06 22:04:26 [INFO] raft: Copied 1005637598 bytes to local snapshot
2017/02/06 22:04:34 [INFO] raft: Compacting logs from 724865 to 725120
2017/02/06 22:04:34 [INFO] raft: Installed remote snapshot
2017/02/06 22:04:35 [INFO] snapshot: Creating new snapshot at /home/gyuho/consul.data/raft/snapshots/2-756515-1486418675363.tmp
2017/02/06 22:04:37 [INFO] snapshot: reaping snapshot /home/gyuho/consul.data/raft/snapshots/2-734528-1486418655488
2017/02/06 22:04:37 [INFO] raft: Copied 1027808324 bytes to local snapshot
2017/02/06 22:04:45 [INFO] consul.fsm: snapshot created in 18.935µs
2017/02/06 22:04:45 [INFO] raft: Compacting logs from 725121 to 725120
2017/02/06 22:04:46 [INFO] raft: Installed remote snapshot
2017/02/06 22:04:46 [INFO] raft: Starting snapshot up to 756515
2017/02/10 18:58:43 [INFO] snapshot: Creating new snapshot at /home/gyuho/consul.data/raft/snapshots/2-900345-1486753123478.tmp
2017/02/10 18:58:45 [INFO] snapshot: reaping snapshot /home/gyuho/consul.data/raft/snapshots/2-849399-1486753096972
2017/02/10 18:58:46 [INFO] raft: Copied 1223270573 bytes to local snapshot
2017/02/10 18:58:55 [INFO] raft: Compacting logs from 868354 to 868801
2017/02/10 18:58:56 [INFO] raft: Installed remote snapshot
2017/02/10 18:58:57 [INFO] snapshot: Creating new snapshot at /home/gyuho/consul.data/raft/snapshots/2-911546-1486753137827.tmp
2017/02/10 18:58:59 [INFO] consul.fsm: snapshot created in 32.255µs
2017/02/10 18:59:01 [INFO] snapshot: reaping snapshot /home/gyuho/consul.data/raft/snapshots/2-873921-1486753116619
2017/02/10 18:59:02 [INFO] raft: Copied 1238491373 bytes to local snapshot
2017/02/10 18:59:11 [INFO] raft: Compacting logs from 868802 to 868801
2017/02/10 18:59:11 [INFO] raft: Installed remote snapshot
```
Logs do not tell much but average latency spikes (e.g. from 258.28656 ms to 6265.185836 ms)
Logs do not tell much but average latency spikes (e.g. from 70.27517 ms to 10407.900082 ms)
@ -108,54 +97,54 @@ Logs do not tell much but average latency spikes (e.g. from 258.28656 ms to 6265
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 338.7661 sec | 344.3563 sec | 703.7060 sec |
| TOTAL-SECONDS | 342.2984 sec | 320.9968 sec | 888.9235 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 35,147 req/sec | 31,726 req/sec | 15,328 req/sec |
| AVG-THROUGHPUT | 2,951 req/sec | 2,903 req/sec | 1,421 req/sec |
| MIN-THROUGHPUT | 90 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 1.1001 ms | 1.1093 ms | 2.9964 ms |
| AVG-LATENCY | 13.8862 ms | 34.9948 ms | 72.5791 ms |
| SLOWEST-LATENCY | 109.4800 ms | 2618.2703 ms | 20860.6692 ms |
| Latency p10 | 2.295037 ms | 2.620473 ms | 3.982040 ms |
| Latency p25 | 5.788546 ms | 3.932461 ms | 7.888984 ms |
| Latency p50 | 9.935599 ms | 7.747493 ms | 21.950488 ms |
| Latency p75 | 17.040088 ms | 16.891088 ms | 58.936521 ms |
| Latency p90 | 28.513968 ms | 28.114578 ms | 126.568085 ms |
| Latency p95 | 44.023164 ms | 44.599685 ms | 165.331967 ms |
| Latency p99 | 60.351324 ms | 1063.554863 ms | 430.071868 ms |
| Latency p99.9 | 94.058105 ms | 2336.144865 ms | 12648.734251 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.1 GB | 5.4 GB | 7.9 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.4 GB | 6.6 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 270 MB | 357 MB | 202 MB |
| MAX-THROUGHPUT | 34,747 req/sec | 43,558 req/sec | 16,486 req/sec |
| AVG-THROUGHPUT | 2,921 req/sec | 3,115 req/sec | 1,124 req/sec |
| MIN-THROUGHPUT | 29 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 1.1239 ms | 1.1194 ms | 3.1083 ms |
| AVG-LATENCY | 13.9400 ms | 36.2980 ms | 152.1034 ms |
| SLOWEST-LATENCY | 115.2305 ms | 2626.1766 ms | 20615.4531 ms |
| Latency p10 | 2.309089 ms | 2.512687 ms | 3.992750 ms |
| Latency p25 | 5.936953 ms | 3.853528 ms | 8.522133 ms |
| Latency p50 | 10.201295 ms | 6.619094 ms | 36.083945 ms |
| Latency p75 | 16.895621 ms | 13.507464 ms | 107.288320 ms |
| Latency p90 | 28.576840 ms | 20.348043 ms | 313.922534 ms |
| Latency p95 | 44.602367 ms | 28.054291 ms | 478.639678 ms |
| Latency p99 | 60.177421 ms | 1108.928408 ms | 623.624914 ms |
| Latency p99.9 | 92.142990 ms | 2619.595630 ms | 14534.009968 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.4 GB | 6.7 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.3 GB | 5.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 270 MB | 356 MB | 210 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.5 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 434.00 % | 600.67 % | 416.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.3 GB | 4.0 GB | 5.4 GB |
| CLIENT-MAX-CPU-USAGE | 540.00 % | 322.00 % | 204.00 % |
| CLIENT-MAX-MEMORY-USAGE | 330 MB | 3.6 GB | 199 MB |
| CLIENT-ERROR-COUNT | 0 | 24 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 76 | 334 | 66 |
| SERVER-MAX-CPU-USAGE | 431.33 % | 601.67 % | 466.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.3 GB | 3.9 GB | 5.0 GB |
| CLIENT-MAX-CPU-USAGE | 559.00 % | 685.00 % | 231.00 % |
| CLIENT-MAX-MEMORY-USAGE | 327 MB | 4.4 GB | 201 MB |
| CLIENT-ERROR-COUNT | 0 | 13 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 72 | 389 | 255 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 1,217,294 | 953,784 | 2,381,092 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 714,152 | 9,304,072 | 10,711,132 |
| SERVER-AVG-DISK-SPACE-USAGE | 3.0 GB | 7.9 GB | 3.0 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 1,525,739 | 1,199,135 | 4,434,402 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 32,134,464 | 40,860,544 | 183,627,904 |
| SERVER-AVG-DISK-SPACE-USAGE | 3.1 GB | 7.6 GB | 2.8 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 24)
"zk: could not connect to a server" (count 13)
```
@ -226,60 +215,60 @@ zookeeper errors:
<br><br><hr>
##### Write 1M keys, 256-byte key, 1KB value, Best Throughput (etcd 700, Zookeeper 300, Consul 500 clients)
##### Write 1M keys, 256-byte key, 1KB value, Best Throughput (etcd 1,000, Zookeeper 500, Consul 500 clients)
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 37.3284 sec | 75.0700 sec | 304.4858 sec |
| TOTAL-SECONDS | 36.5583 sec | 63.6203 sec | 261.0246 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 33,567 req/sec | 35,499 req/sec | 15,141 req/sec |
| AVG-THROUGHPUT | 26,789 req/sec | 13,274 req/sec | 3,284 req/sec |
| MIN-THROUGHPUT | 10,018 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.2842 ms | 2.7405 ms | 11.4297 ms |
| AVG-LATENCY | 26.0603 ms | 18.2231 ms | 152.1359 ms |
| SLOWEST-LATENCY | 520.8716 ms | 4264.2996 ms | 28029.3953 ms |
| Latency p10 | 10.171289 ms | 6.401553 ms | 30.579107 ms |
| Latency p25 | 12.254908 ms | 7.300705 ms | 35.763003 ms |
| Latency p50 | 17.138243 ms | 8.302805 ms | 48.467608 ms |
| Latency p75 | 23.925669 ms | 9.453586 ms | 80.519456 ms |
| Latency p90 | 48.690057 ms | 10.764813 ms | 248.959013 ms |
| Latency p95 | 76.533161 ms | 11.992104 ms | 349.281928 ms |
| Latency p99 | 146.318242 ms | 153.580393 ms | 1324.508306 ms |
| Latency p99.9 | 183.924901 ms | 1935.929712 ms | 10622.316021 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 6.2 GB | 11 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 5.1 GB | 10 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 274 MB | 350 MB | 216 MB |
| MAX-THROUGHPUT | 35,187 req/sec | 44,883 req/sec | 15,009 req/sec |
| AVG-THROUGHPUT | 27,353 req/sec | 15,684 req/sec | 3,831 req/sec |
| MIN-THROUGHPUT | 13,891 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.7388 ms | 2.4777 ms | 14.0583 ms |
| AVG-LATENCY | 36.3305 ms | 22.5664 ms | 130.4658 ms |
| SLOWEST-LATENCY | 346.3847 ms | 3517.8313 ms | 19316.7564 ms |
| Latency p10 | 15.483941 ms | 7.766447 ms | 32.229589 ms |
| Latency p25 | 18.596901 ms | 9.156112 ms | 37.179339 ms |
| Latency p50 | 23.920164 ms | 10.532408 ms | 47.314383 ms |
| Latency p75 | 44.685986 ms | 12.228320 ms | 70.740623 ms |
| Latency p90 | 64.373404 ms | 14.214414 ms | 104.953131 ms |
| Latency p95 | 113.441501 ms | 16.216272 ms | 282.116427 ms |
| Latency p99 | 167.148590 ms | 340.373624 ms | 1092.781013 ms |
| Latency p99.9 | 194.350987 ms | 2151.870409 ms | 11587.696623 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.4 GB | 10 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 4.3 GB | 9.1 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 352 MB | 218 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 407.67 % | 704.97 % | 380.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 5.1 GB | 6.2 GB |
| CLIENT-MAX-CPU-USAGE | 454.00 % | 292.00 % | 202.00 % |
| CLIENT-MAX-MEMORY-USAGE | 210 MB | 1.7 GB | 88 MB |
| CLIENT-ERROR-COUNT | 0 | 3,452 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 5 | 212 | 270 |
| SERVER-MAX-CPU-USAGE | 446.67 % | 731.27 % | 379.33 % |
| SERVER-MAX-MEMORY-USAGE | 1.2 GB | 4.6 GB | 5.8 GB |
| CLIENT-MAX-CPU-USAGE | 568.00 % | 404.00 % | 223.00 % |
| CLIENT-MAX-MEMORY-USAGE | 248 MB | 3.1 GB | 81 MB |
| CLIENT-ERROR-COUNT | 0 | 2,150 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 2 | 213 | 147 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 112,190 | 109,945 | 681,774 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 492,444 | 10,249,020 | 32,988,480 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.8 GB | 7.3 GB | 2.9 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 102,486 | 86,871 | 641,255 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 20,504,912 | 26,252,736 | 110,850,712 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 6.9 GB | 3.0 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 3,152)
"zk: connection closed" (count 300)
"zk: could not connect to a server" (count 765)
"zk: connection closed" (count 1,385)
```
@ -355,55 +344,55 @@ zookeeper errors:
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 36.3917 sec | 72.1698 sec | 140.9480 sec |
| TOTAL-SECONDS | 36.5091 sec | 59.0429 sec | 116.4349 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 36,861 req/sec | 39,923 req/sec | 14,455 req/sec |
| AVG-THROUGHPUT | 27,478 req/sec | 13,704 req/sec | 7,094 req/sec |
| MIN-THROUGHPUT | 7,088 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.7509 ms | 4.3350 ms | 12.9159 ms |
| AVG-LATENCY | 36.2639 ms | 49.9165 ms | 140.4538 ms |
| SLOWEST-LATENCY | 244.3595 ms | 6056.0204 ms | 21808.2916 ms |
| Latency p10 | 13.700258 ms | 14.719617 ms | 65.494475 ms |
| Latency p25 | 16.855903 ms | 20.289440 ms | 71.570399 ms |
| Latency p50 | 21.895662 ms | 24.861967 ms | 80.420560 ms |
| Latency p75 | 43.387011 ms | 28.165478 ms | 96.334874 ms |
| Latency p90 | 69.199925 ms | 32.086272 ms | 161.048409 ms |
| Latency p95 | 133.035887 ms | 58.675701 ms | 210.603403 ms |
| Latency p99 | 173.021441 ms | 880.253639 ms | 484.701759 ms |
| Latency p99.9 | 188.144407 ms | 3260.921550 ms | 15472.281038 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.5 GB | 5.6 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.4 GB | 4.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 357 MB | 243 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 443.00 % | 695.00 % | 414.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 4.8 GB | 4.9 GB |
| CLIENT-MAX-CPU-USAGE | 566.00 % | 324.00 % | 205.00 % |
| CLIENT-MAX-MEMORY-USAGE | 278 MB | 5.3 GB | 114 MB |
| CLIENT-ERROR-COUNT | 0 | 10,929 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 2 | 216 | 64 |
| MAX-THROUGHPUT | 35,162 req/sec | 50,196 req/sec | 14,615 req/sec |
| AVG-THROUGHPUT | 27,390 req/sec | 16,854 req/sec | 8,588 req/sec |
| MIN-THROUGHPUT | 14,654 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.9384 ms | 1.4337 ms | 15.3309 ms |
| AVG-LATENCY | 36.3098 ms | 44.2298 ms | 115.8624 ms |
| SLOWEST-LATENCY | 353.6275 ms | 7585.4064 ms | 12902.7031 ms |
| Latency p10 | 14.627933 ms | 12.299326 ms | 67.088002 ms |
| Latency p25 | 17.957404 ms | 15.830300 ms | 73.441500 ms |
| Latency p50 | 23.290321 ms | 18.660221 ms | 82.581044 ms |
| Latency p75 | 43.118899 ms | 21.623243 ms | 96.395872 ms |
| Latency p90 | 64.488242 ms | 26.143493 ms | 166.706106 ms |
| Latency p95 | 126.257124 ms | 77.780210 ms | 218.288165 ms |
| Latency p99 | 173.468949 ms | 843.928907 ms | 323.392356 ms |
| Latency p99.9 | 194.822900 ms | 2909.696060 ms | 7179.211229 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.3 GB | 5.6 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 4.3 GB | 4.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 371 MB | 243 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.5 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 433.67 % | 634.40 % | 478.67 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 4.9 GB | 5.1 GB |
| CLIENT-MAX-CPU-USAGE | 582.00 % | 422.00 % | 219.00 % |
| CLIENT-MAX-MEMORY-USAGE | 266 MB | 4.9 GB | 115 MB |
| CLIENT-ERROR-COUNT | 0 | 4,886 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 70 | 217 | 46 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 100,985 | 91,154 | 289,238 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 552,592 | 9,923,988 | 9,537,168 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 6.7 GB | 3.1 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 102,259 | 80,718 | 377,108 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 20,520,584 | 27,706,344 | 48,882,704 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 5.9 GB | 3.1 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 8,818)
"zk: connection closed" (count 2,111)
"zk: connection closed" (count 2,381)
"zk: could not connect to a server" (count 2,505)
```
@ -474,60 +463,60 @@ zookeeper errors:
<br><br><hr>
##### Write 3-million keys, 256-byte key, 1KB value, Best Throughput (etcd 700, Zookeeper 300, Consul 500 clients)
##### Write 3-million keys, 256-byte key, 1KB value, Best Throughput (etcd 1,000, Zookeeper 500, Consul 500 clients)
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 141.1728 sec | 4134.1565 sec | 2979.6229 sec |
| TOTAL-SECONDS | 138.0049 sec | 2242.5513 sec | 2361.4227 sec |
| TOTAL-REQUEST-NUMBER | 3,000,000 | 3,000,000 | 3,000,000 |
| MAX-THROUGHPUT | 32,215 req/sec | 38,259 req/sec | 15,784 req/sec |
| AVG-THROUGHPUT | 21,250 req/sec | 465 req/sec | 1,006 req/sec |
| MIN-THROUGHPUT | 9,081 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.6479 ms | 1.8912 ms | 10.3322 ms |
| AVG-LATENCY | 32.9036 ms | 28.5018 ms | 496.5874 ms |
| SLOWEST-LATENCY | 1030.3392 ms | 4729.8104 ms | 34484.1252 ms |
| Latency p10 | 10.882650 ms | 5.694308 ms | 35.045139 ms |
| Latency p25 | 13.005974 ms | 6.357600 ms | 44.711879 ms |
| Latency p50 | 18.743999 ms | 7.379354 ms | 66.739638 ms |
| Latency p75 | 31.054943 ms | 8.886502 ms | 116.150396 ms |
| Latency p90 | 96.027241 ms | 11.474394 ms | 540.305996 ms |
| Latency p95 | 123.844111 ms | 14.391699 ms | 1209.603980 ms |
| Latency p99 | 148.837786 ms | 722.847010 ms | 11957.052613 ms |
| Latency p99.9 | 162.994644 ms | 2457.522611 ms | 26068.610794 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 15 GB | 26 GB | 102 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 12 GB | 24 GB | 98 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 832 MB | 978 MB | 688 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 4.3 GB | 3.3 GB | 4.4 GB |
| SERVER-MAX-CPU-USAGE | 421.00 % | 752.33 % | 410.67 % |
| SERVER-MAX-MEMORY-USAGE | 2.8 GB | 7.1 GB | 17 GB |
| CLIENT-MAX-CPU-USAGE | 442.00 % | 376.00 % | 217.00 % |
| CLIENT-MAX-MEMORY-USAGE | 265 MB | 1.9 GB | 174 MB |
| CLIENT-ERROR-COUNT | 0 | 1,076,632 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 14 | 309 | 94,864 |
| MAX-THROUGHPUT | 33,963 req/sec | 42,804 req/sec | 16,033 req/sec |
| AVG-THROUGHPUT | 21,738 req/sec | 906 req/sec | 1,270 req/sec |
| MIN-THROUGHPUT | 7,006 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.2812 ms | 1.2984 ms | 16.9263 ms |
| AVG-LATENCY | 45.9259 ms | 42.7162 ms | 393.5551 ms |
| SLOWEST-LATENCY | 259.3153 ms | 6921.5532 ms | 30425.8070 ms |
| Latency p10 | 15.658060 ms | 8.200137 ms | 34.534783 ms |
| Latency p25 | 18.852826 ms | 9.417761 ms | 42.917753 ms |
| Latency p50 | 23.711419 ms | 10.902322 ms | 64.087200 ms |
| Latency p75 | 53.911600 ms | 12.742504 ms | 91.427820 ms |
| Latency p90 | 130.485283 ms | 15.178863 ms | 167.938406 ms |
| Latency p95 | 151.376530 ms | 38.140465 ms | 951.339409 ms |
| Latency p99 | 171.722975 ms | 1540.586327 ms | 10968.875166 ms |
| Latency p99.9 | 188.102916 ms | 2276.156658 ms | 18546.023298 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 15 GB | 12 GB | 89 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 12 GB | 10 GB | 85 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 852 MB | 997 MB | 685 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 4.3 GB | 3.4 GB | 4.4 GB |
| SERVER-MAX-CPU-USAGE | 442.00 % | 785.67 % | 439.33 % |
| SERVER-MAX-MEMORY-USAGE | 2.8 GB | 7.4 GB | 15 GB |
| CLIENT-MAX-CPU-USAGE | 554.00 % | 368.00 % | 222.00 % |
| CLIENT-MAX-MEMORY-USAGE | 348 MB | 3.1 GB | 157 MB |
| CLIENT-ERROR-COUNT | 0 | 967,222 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 168 | 443 | 23,290 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 333,196 | 360,314 | 3,689,221 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 1,604,172 | 67,295,610 | 794,919,246 |
| SERVER-AVG-DISK-SPACE-USAGE | 6.5 GB | 27 GB | 8.2 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 303,974 | 231,817 | 2,976,074 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 62,193,472 | 84,808,712 | 990,569,940 |
| SERVER-AVG-DISK-SPACE-USAGE | 6.5 GB | 21 GB | 8.3 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: connection closed" (count 6,678)
"zk: could not connect to a server" (count 1,069,954)
"zk: could not connect to a server" (count 963,503)
"zk: connection closed" (count 3,719)
```

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-1M-large.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-1M-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-1M.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 46 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-3M-by-key-large.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-3M-by-key-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-latency-3M-by-key.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 62 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-memory-1M-by-key-large.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-memory-1M-by-key-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
test-results/2017Q1-00-etcd-zookeeper-consul/2017Q1-00-etcd-zookeeper-consul-memory-1M-by-key.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 48 KiB

262
test-results/2017Q1-00-etcd-zookeeper-consul/README.md
View File

@ -6,54 +6,54 @@
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 338.7661 sec | 344.3563 sec | 703.7060 sec |
| TOTAL-SECONDS | 342.2984 sec | 320.9968 sec | 888.9235 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 35,147 req/sec | 31,726 req/sec | 15,328 req/sec |
| AVG-THROUGHPUT | 2,951 req/sec | 2,903 req/sec | 1,421 req/sec |
| MIN-THROUGHPUT | 90 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 1.1001 ms | 1.1093 ms | 2.9964 ms |
| AVG-LATENCY | 13.8862 ms | 34.9948 ms | 72.5791 ms |
| SLOWEST-LATENCY | 109.4800 ms | 2618.2703 ms | 20860.6692 ms |
| Latency p10 | 2.295037 ms | 2.620473 ms | 3.982040 ms |
| Latency p25 | 5.788546 ms | 3.932461 ms | 7.888984 ms |
| Latency p50 | 9.935599 ms | 7.747493 ms | 21.950488 ms |
| Latency p75 | 17.040088 ms | 16.891088 ms | 58.936521 ms |
| Latency p90 | 28.513968 ms | 28.114578 ms | 126.568085 ms |
| Latency p95 | 44.023164 ms | 44.599685 ms | 165.331967 ms |
| Latency p99 | 60.351324 ms | 1063.554863 ms | 430.071868 ms |
| Latency p99.9 | 94.058105 ms | 2336.144865 ms | 12648.734251 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.1 GB | 5.4 GB | 7.9 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.4 GB | 6.6 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 270 MB | 357 MB | 202 MB |
| MAX-THROUGHPUT | 34,747 req/sec | 43,558 req/sec | 16,486 req/sec |
| AVG-THROUGHPUT | 2,921 req/sec | 3,115 req/sec | 1,124 req/sec |
| MIN-THROUGHPUT | 29 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 1.1239 ms | 1.1194 ms | 3.1083 ms |
| AVG-LATENCY | 13.9400 ms | 36.2980 ms | 152.1034 ms |
| SLOWEST-LATENCY | 115.2305 ms | 2626.1766 ms | 20615.4531 ms |
| Latency p10 | 2.309089 ms | 2.512687 ms | 3.992750 ms |
| Latency p25 | 5.936953 ms | 3.853528 ms | 8.522133 ms |
| Latency p50 | 10.201295 ms | 6.619094 ms | 36.083945 ms |
| Latency p75 | 16.895621 ms | 13.507464 ms | 107.288320 ms |
| Latency p90 | 28.576840 ms | 20.348043 ms | 313.922534 ms |
| Latency p95 | 44.602367 ms | 28.054291 ms | 478.639678 ms |
| Latency p99 | 60.177421 ms | 1108.928408 ms | 623.624914 ms |
| Latency p99.9 | 92.142990 ms | 2619.595630 ms | 14534.009968 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.4 GB | 6.7 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.3 GB | 5.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 270 MB | 356 MB | 210 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.5 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 434.00 % | 600.67 % | 416.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.3 GB | 4.0 GB | 5.4 GB |
| CLIENT-MAX-CPU-USAGE | 540.00 % | 322.00 % | 204.00 % |
| CLIENT-MAX-MEMORY-USAGE | 330 MB | 3.6 GB | 199 MB |
| CLIENT-ERROR-COUNT | 0 | 24 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 76 | 334 | 66 |
| SERVER-MAX-CPU-USAGE | 431.33 % | 601.67 % | 466.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.3 GB | 3.9 GB | 5.0 GB |
| CLIENT-MAX-CPU-USAGE | 559.00 % | 685.00 % | 231.00 % |
| CLIENT-MAX-MEMORY-USAGE | 327 MB | 4.4 GB | 201 MB |
| CLIENT-ERROR-COUNT | 0 | 13 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 72 | 389 | 255 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 1,217,294 | 953,784 | 2,381,092 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 714,152 | 9,304,072 | 10,711,132 |
| SERVER-AVG-DISK-SPACE-USAGE | 3.0 GB | 7.9 GB | 3.0 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 1,525,739 | 1,199,135 | 4,434,402 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 32,134,464 | 40,860,544 | 183,627,904 |
| SERVER-AVG-DISK-SPACE-USAGE | 3.1 GB | 7.6 GB | 2.8 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 24)
"zk: could not connect to a server" (count 13)
```
@ -124,60 +124,60 @@ zookeeper errors:
<br><br><hr>
##### Write 1M keys, 256-byte key, 1KB value, Best Throughput (etcd 700, Zookeeper 300, Consul 500 clients)
##### Write 1M keys, 256-byte key, 1KB value, Best Throughput (etcd 1,000, Zookeeper 500, Consul 500 clients)
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 37.3284 sec | 75.0700 sec | 304.4858 sec |
| TOTAL-SECONDS | 36.5583 sec | 63.6203 sec | 261.0246 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 33,567 req/sec | 35,499 req/sec | 15,141 req/sec |
| AVG-THROUGHPUT | 26,789 req/sec | 13,274 req/sec | 3,284 req/sec |
| MIN-THROUGHPUT | 10,018 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.2842 ms | 2.7405 ms | 11.4297 ms |
| AVG-LATENCY | 26.0603 ms | 18.2231 ms | 152.1359 ms |
| SLOWEST-LATENCY | 520.8716 ms | 4264.2996 ms | 28029.3953 ms |
| Latency p10 | 10.171289 ms | 6.401553 ms | 30.579107 ms |
| Latency p25 | 12.254908 ms | 7.300705 ms | 35.763003 ms |
| Latency p50 | 17.138243 ms | 8.302805 ms | 48.467608 ms |
| Latency p75 | 23.925669 ms | 9.453586 ms | 80.519456 ms |
| Latency p90 | 48.690057 ms | 10.764813 ms | 248.959013 ms |
| Latency p95 | 76.533161 ms | 11.992104 ms | 349.281928 ms |
| Latency p99 | 146.318242 ms | 153.580393 ms | 1324.508306 ms |
| Latency p99.9 | 183.924901 ms | 1935.929712 ms | 10622.316021 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 6.2 GB | 11 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 5.1 GB | 10 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 274 MB | 350 MB | 216 MB |
| MAX-THROUGHPUT | 35,187 req/sec | 44,883 req/sec | 15,009 req/sec |
| AVG-THROUGHPUT | 27,353 req/sec | 15,684 req/sec | 3,831 req/sec |
| MIN-THROUGHPUT | 13,891 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.7388 ms | 2.4777 ms | 14.0583 ms |
| AVG-LATENCY | 36.3305 ms | 22.5664 ms | 130.4658 ms |
| SLOWEST-LATENCY | 346.3847 ms | 3517.8313 ms | 19316.7564 ms |
| Latency p10 | 15.483941 ms | 7.766447 ms | 32.229589 ms |
| Latency p25 | 18.596901 ms | 9.156112 ms | 37.179339 ms |
| Latency p50 | 23.920164 ms | 10.532408 ms | 47.314383 ms |
| Latency p75 | 44.685986 ms | 12.228320 ms | 70.740623 ms |
| Latency p90 | 64.373404 ms | 14.214414 ms | 104.953131 ms |
| Latency p95 | 113.441501 ms | 16.216272 ms | 282.116427 ms |
| Latency p99 | 167.148590 ms | 340.373624 ms | 1092.781013 ms |
| Latency p99.9 | 194.350987 ms | 2151.870409 ms | 11587.696623 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.4 GB | 10 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 4.3 GB | 9.1 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 352 MB | 218 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 407.67 % | 704.97 % | 380.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 5.1 GB | 6.2 GB |
| CLIENT-MAX-CPU-USAGE | 454.00 % | 292.00 % | 202.00 % |
| CLIENT-MAX-MEMORY-USAGE | 210 MB | 1.7 GB | 88 MB |
| CLIENT-ERROR-COUNT | 0 | 3,452 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 5 | 212 | 270 |
| SERVER-MAX-CPU-USAGE | 446.67 % | 731.27 % | 379.33 % |
| SERVER-MAX-MEMORY-USAGE | 1.2 GB | 4.6 GB | 5.8 GB |
| CLIENT-MAX-CPU-USAGE | 568.00 % | 404.00 % | 223.00 % |
| CLIENT-MAX-MEMORY-USAGE | 248 MB | 3.1 GB | 81 MB |
| CLIENT-ERROR-COUNT | 0 | 2,150 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 2 | 213 | 147 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 112,190 | 109,945 | 681,774 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 492,444 | 10,249,020 | 32,988,480 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.8 GB | 7.3 GB | 2.9 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 102,486 | 86,871 | 641,255 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 20,504,912 | 26,252,736 | 110,850,712 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 6.9 GB | 3.0 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 3,152)
"zk: connection closed" (count 300)
"zk: could not connect to a server" (count 765)
"zk: connection closed" (count 1,385)
```
@ -253,55 +253,55 @@ zookeeper errors:
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 36.3917 sec | 72.1698 sec | 140.9480 sec |
| TOTAL-SECONDS | 36.5091 sec | 59.0429 sec | 116.4349 sec |
| TOTAL-REQUEST-NUMBER | 1,000,000 | 1,000,000 | 1,000,000 |
| MAX-THROUGHPUT | 36,861 req/sec | 39,923 req/sec | 14,455 req/sec |
| AVG-THROUGHPUT | 27,478 req/sec | 13,704 req/sec | 7,094 req/sec |
| MIN-THROUGHPUT | 7,088 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.7509 ms | 4.3350 ms | 12.9159 ms |
| AVG-LATENCY | 36.2639 ms | 49.9165 ms | 140.4538 ms |
| SLOWEST-LATENCY | 244.3595 ms | 6056.0204 ms | 21808.2916 ms |
| Latency p10 | 13.700258 ms | 14.719617 ms | 65.494475 ms |
| Latency p25 | 16.855903 ms | 20.289440 ms | 71.570399 ms |
| Latency p50 | 21.895662 ms | 24.861967 ms | 80.420560 ms |
| Latency p75 | 43.387011 ms | 28.165478 ms | 96.334874 ms |
| Latency p90 | 69.199925 ms | 32.086272 ms | 161.048409 ms |
| Latency p95 | 133.035887 ms | 58.675701 ms | 210.603403 ms |
| Latency p99 | 173.021441 ms | 880.253639 ms | 484.701759 ms |
| Latency p99.9 | 188.144407 ms | 3260.921550 ms | 15472.281038 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.5 GB | 5.6 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.9 GB | 4.4 GB | 4.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 357 MB | 243 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.4 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 443.00 % | 695.00 % | 414.00 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 4.8 GB | 4.9 GB |
| CLIENT-MAX-CPU-USAGE | 566.00 % | 324.00 % | 205.00 % |
| CLIENT-MAX-MEMORY-USAGE | 278 MB | 5.3 GB | 114 MB |
| CLIENT-ERROR-COUNT | 0 | 10,929 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 2 | 216 | 64 |
| MAX-THROUGHPUT | 35,162 req/sec | 50,196 req/sec | 14,615 req/sec |
| AVG-THROUGHPUT | 27,390 req/sec | 16,854 req/sec | 8,588 req/sec |
| MIN-THROUGHPUT | 14,654 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 4.9384 ms | 1.4337 ms | 15.3309 ms |
| AVG-LATENCY | 36.3098 ms | 44.2298 ms | 115.8624 ms |
| SLOWEST-LATENCY | 353.6275 ms | 7585.4064 ms | 12902.7031 ms |
| Latency p10 | 14.627933 ms | 12.299326 ms | 67.088002 ms |
| Latency p25 | 17.957404 ms | 15.830300 ms | 73.441500 ms |
| Latency p50 | 23.290321 ms | 18.660221 ms | 82.581044 ms |
| Latency p75 | 43.118899 ms | 21.623243 ms | 96.395872 ms |
| Latency p90 | 64.488242 ms | 26.143493 ms | 166.706106 ms |
| Latency p95 | 126.257124 ms | 77.780210 ms | 218.288165 ms |
| Latency p99 | 173.468949 ms | 843.928907 ms | 323.392356 ms |
| Latency p99.9 | 194.822900 ms | 2909.696060 ms | 7179.211229 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 5.0 GB | 5.3 GB | 5.6 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 3.8 GB | 4.3 GB | 4.4 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 282 MB | 371 MB | 243 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 1.4 GB | 1.5 GB | 1.5 GB |
| SERVER-MAX-CPU-USAGE | 433.67 % | 634.40 % | 478.67 % |
| SERVER-MAX-MEMORY-USAGE | 1.1 GB | 4.9 GB | 5.1 GB |
| CLIENT-MAX-CPU-USAGE | 582.00 % | 422.00 % | 219.00 % |
| CLIENT-MAX-MEMORY-USAGE | 266 MB | 4.9 GB | 115 MB |
| CLIENT-ERROR-COUNT | 0 | 4,886 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 70 | 217 | 46 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 100,985 | 91,154 | 289,238 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 552,592 | 9,923,988 | 9,537,168 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 6.7 GB | 3.1 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 102,259 | 80,718 | 377,108 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 20,520,584 | 27,706,344 | 48,882,704 |
| SERVER-AVG-DISK-SPACE-USAGE | 2.7 GB | 5.9 GB | 3.1 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: could not connect to a server" (count 8,818)
"zk: connection closed" (count 2,111)
"zk: connection closed" (count 2,381)
"zk: could not connect to a server" (count 2,505)
```
@ -372,60 +372,60 @@ zookeeper errors:
<br><br><hr>
##### Write 3-million keys, 256-byte key, 1KB value, Best Throughput (etcd 700, Zookeeper 300, Consul 500 clients)
##### Write 3-million keys, 256-byte key, 1KB value, Best Throughput (etcd 1,000, Zookeeper 500, Consul 500 clients)
- Google Cloud Compute Engine
- 4 machines of 16 vCPUs + 30 GB Memory + 150 GB SSD (1 for client)
- Ubuntu 16.10
- etcd v3.1 (Go 1.7.4)
- etcd v3.1 (Go 1.7.5)
- Zookeeper r3.4.9
- Java 8
- javac 1.8.0_121
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Consul v0.7.3 (Go 1.7.4)
- Consul v0.7.4 (Go 1.7.5)
```
+---------------------------------------+-------------------+------------------------+-----------------------+
| | etcd-v3.1-go1.7.4 | zookeeper-r3.4.9-java8 | consul-v0.7.3-go1.7.4 |
| | etcd-v3.1-go1.7.5 | zookeeper-r3.4.9-java8 | consul-v0.7.4-go1.7.5 |
+---------------------------------------+-------------------+------------------------+-----------------------+
| TOTAL-SECONDS | 141.1728 sec | 4134.1565 sec | 2979.6229 sec |
| TOTAL-SECONDS | 138.0049 sec | 2242.5513 sec | 2361.4227 sec |
| TOTAL-REQUEST-NUMBER | 3,000,000 | 3,000,000 | 3,000,000 |
| MAX-THROUGHPUT | 32,215 req/sec | 38,259 req/sec | 15,784 req/sec |
| AVG-THROUGHPUT | 21,250 req/sec | 465 req/sec | 1,006 req/sec |
| MIN-THROUGHPUT | 9,081 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.6479 ms | 1.8912 ms | 10.3322 ms |
| AVG-LATENCY | 32.9036 ms | 28.5018 ms | 496.5874 ms |
| SLOWEST-LATENCY | 1030.3392 ms | 4729.8104 ms | 34484.1252 ms |
| Latency p10 | 10.882650 ms | 5.694308 ms | 35.045139 ms |
| Latency p25 | 13.005974 ms | 6.357600 ms | 44.711879 ms |
| Latency p50 | 18.743999 ms | 7.379354 ms | 66.739638 ms |
| Latency p75 | 31.054943 ms | 8.886502 ms | 116.150396 ms |
| Latency p90 | 96.027241 ms | 11.474394 ms | 540.305996 ms |
| Latency p95 | 123.844111 ms | 14.391699 ms | 1209.603980 ms |
| Latency p99 | 148.837786 ms | 722.847010 ms | 11957.052613 ms |
| Latency p99.9 | 162.994644 ms | 2457.522611 ms | 26068.610794 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 15 GB | 26 GB | 102 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 12 GB | 24 GB | 98 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 832 MB | 978 MB | 688 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 4.3 GB | 3.3 GB | 4.4 GB |
| SERVER-MAX-CPU-USAGE | 421.00 % | 752.33 % | 410.67 % |
| SERVER-MAX-MEMORY-USAGE | 2.8 GB | 7.1 GB | 17 GB |
| CLIENT-MAX-CPU-USAGE | 442.00 % | 376.00 % | 217.00 % |
| CLIENT-MAX-MEMORY-USAGE | 265 MB | 1.9 GB | 174 MB |
| CLIENT-ERROR-COUNT | 0 | 1,076,632 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 14 | 309 | 94,864 |
| MAX-THROUGHPUT | 33,963 req/sec | 42,804 req/sec | 16,033 req/sec |
| AVG-THROUGHPUT | 21,738 req/sec | 906 req/sec | 1,270 req/sec |
| MIN-THROUGHPUT | 7,006 req/sec | 0 req/sec | 0 req/sec |
| FASTEST-LATENCY | 3.2812 ms | 1.2984 ms | 16.9263 ms |
| AVG-LATENCY | 45.9259 ms | 42.7162 ms | 393.5551 ms |
| SLOWEST-LATENCY | 259.3153 ms | 6921.5532 ms | 30425.8070 ms |
| Latency p10 | 15.658060 ms | 8.200137 ms | 34.534783 ms |
| Latency p25 | 18.852826 ms | 9.417761 ms | 42.917753 ms |
| Latency p50 | 23.711419 ms | 10.902322 ms | 64.087200 ms |
| Latency p75 | 53.911600 ms | 12.742504 ms | 91.427820 ms |
| Latency p90 | 130.485283 ms | 15.178863 ms | 167.938406 ms |
| Latency p95 | 151.376530 ms | 38.140465 ms | 951.339409 ms |
| Latency p99 | 171.722975 ms | 1540.586327 ms | 10968.875166 ms |
| Latency p99.9 | 188.102916 ms | 2276.156658 ms | 18546.023298 ms |
| SERVER-TOTAL-NETWORK-RX-DATA-SUM | 15 GB | 12 GB | 89 GB |
| SERVER-TOTAL-NETWORK-TX-DATA-SUM | 12 GB | 10 GB | 85 GB |
| CLIENT-TOTAL-NETWORK-RX-SUM | 852 MB | 997 MB | 685 MB |
| CLIENT-TOTAL-NETWORK-TX-SUM | 4.3 GB | 3.4 GB | 4.4 GB |
| SERVER-MAX-CPU-USAGE | 442.00 % | 785.67 % | 439.33 % |
| SERVER-MAX-MEMORY-USAGE | 2.8 GB | 7.4 GB | 15 GB |
| CLIENT-MAX-CPU-USAGE | 554.00 % | 368.00 % | 222.00 % |
| CLIENT-MAX-MEMORY-USAGE | 348 MB | 3.1 GB | 157 MB |
| CLIENT-ERROR-COUNT | 0 | 967,222 | 0 |
| SERVER-AVG-READS-COMPLETED-DELTA-SUM | 168 | 443 | 23,290 |
| SERVER-AVG-SECTORS-READS-DELTA-SUM | 0 | 0 | 0 |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 333,196 | 360,314 | 3,689,221 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 1,604,172 | 67,295,610 | 794,919,246 |
| SERVER-AVG-DISK-SPACE-USAGE | 6.5 GB | 27 GB | 8.2 GB |
| SERVER-AVG-WRITES-COMPLETED-DELTA-SUM | 303,974 | 231,817 | 2,976,074 |
| SERVER-AVG-SECTORS-WRITTEN-DELTA-SUM | 62,193,472 | 84,808,712 | 990,569,940 |
| SERVER-AVG-DISK-SPACE-USAGE | 6.5 GB | 21 GB | 8.3 GB |
+---------------------------------------+-------------------+------------------------+-----------------------+
zookeeper errors:
"zk: connection closed" (count 6,678)
"zk: could not connect to a server" (count 1,069,954)
"zk: could not connect to a server" (count 963,503)
"zk: connection closed" (count 3,719)
```