mirror of https://github.com/grpc/grpc.io.git
323 lines
12 KiB
HTML
323 lines
12 KiB
HTML
|
|
|
|
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css" integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">
|
|
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,400" rel="stylesheet">
|
|
<link rel="stylesheet" type="text/css" href="/css/style.css">
|
|
<title>
|
|
Benchmarking – gRPC
|
|
</title>
|
|
|
|
|
|
<link rel="apple-touch-icon" href="/favicons/apple-touch-icon.png" sizes="180x180">
|
|
<link rel="icon" type="image/png" href="/favicons/android-chrome-192x192.png" sizes="192x192" >
|
|
<link rel="icon" type="image/png" href="/favicons/favicon-32x32.png" sizes="32x32">
|
|
<link rel="icon" type="image/png" href="/favicons/favicon-16x16.png" sizes="16x16">
|
|
<link rel="manifest" href="/favicons/manifest.json">
|
|
<link rel="mask-icon" href="/favicons/safari-pinned-tab.svg" color="#2DA6B0">
|
|
<meta name="msapplication-TileColor" content="#ffffff">
|
|
<meta name="msapplication-TileImage" content="/favicons/mstile-150x150.png">
|
|
|
|
|
|
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-60127042-1"></script>
|
|
<script>
|
|
window.dataLayer = window.dataLayer || [];
|
|
function gtag(){dataLayer.push(arguments);}
|
|
gtag('js', new Date());
|
|
|
|
gtag('config', 'UA-60127042-1');
|
|
</script>
|
|
|
|
</head>
|
|
<body>
|
|
<div id="landing-content">
|
|
<div class="row">
|
|
<div class="topbannersub">
|
|
<nav class="navbar navbar-expand-md navbar-dark topnav">
|
|
<a class="navbar-brand" href="https://cjyabraham.github.io/">
|
|
<img src="https://cjyabraham.github.io/img/grpc-logo.png" width="114" height="50">
|
|
</a>
|
|
|
|
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
|
|
<span class="navbar-toggler-icon"></span>
|
|
</button>
|
|
|
|
<div class="topnav, collapse navbar-collapse" id="navbarSupportedContent" style="float:right !important">
|
|
<ul class="navbar-nav ml-auto">
|
|
<li class="nav-item ">
|
|
<a class="nav-link" href="https://cjyabraham.github.io/about/">About</a>
|
|
</li>
|
|
<li class="nav-item dropdown active">
|
|
<a class="nav-link dropdown-toggle" href="https://cjyabraham.github.io/docs/" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
|
|
Docs
|
|
</a>
|
|
|
|
<div class="dropdown-menu" aria-labelledby="navbarDropdown">
|
|
|
|
|
|
<a class="dropdown-item" href="/docs">
|
|
Overview
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/quickstart/">
|
|
Quick Start
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/guides/">
|
|
Guides
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/tutorials/">
|
|
Tutorials
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/reference/">
|
|
Reference
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/samples/">
|
|
Samples
|
|
</a>
|
|
|
|
|
|
<a class="dropdown-item" href="/docs/talks">
|
|
Presentations
|
|
</a>
|
|
|
|
</div>
|
|
</li>
|
|
<li class="nav-item ">
|
|
<a class="nav-link" href="/blog">
|
|
Blog
|
|
</a>
|
|
</li>
|
|
<li class="nav-item ">
|
|
<a class="nav-link" href="/community">Community</a>
|
|
</li>
|
|
<li class="nav-item">
|
|
<a class="nav-link" href="https://packages.grpc.io/">
|
|
Packages
|
|
</a>
|
|
</li>
|
|
<li class="nav-item ">
|
|
<a class="nav-link" href="https://cjyabraham.github.io/faq/">FAQ</a>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</nav>
|
|
|
|
|
|
|
|
|
|
|
|
<div class="headertext">Documentation</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="subnav d-none d-md-block">
|
|
<a href="https://cjyabraham.github.io/docs/" >Overview</a>
|
|
| <a href="https://cjyabraham.github.io/docs/quickstart/" >Quick Start</a>
|
|
| <a href="https://cjyabraham.github.io/docs/guides/" class="active">Guides</a>
|
|
| <a href="https://cjyabraham.github.io/docs/tutorials/" >Tutorials</a>
|
|
| <a href="https://cjyabraham.github.io/docs/reference/" >Reference</a>
|
|
| <a href="https://cjyabraham.github.io/docs/samples/" >Samples</a>
|
|
| <a href="https://cjyabraham.github.io/docs/talks/" >Presentations</a>
|
|
</div>
|
|
|
|
|
|
<div class="quickstartcols">
|
|
|
|
<div class="quickstartcol1">
|
|
|
|
<h8>Guides</h8>
|
|
|
|
|
|
<a href="/docs/guides/">
|
|
What is gRPC?
|
|
</a>
|
|
|
|
|
|
<a href="/docs/guides/concepts/">
|
|
gRPC Concepts
|
|
</a>
|
|
|
|
|
|
<a href="/docs/guides/auth/">
|
|
Authentication
|
|
</a>
|
|
|
|
|
|
<a href="/docs/guides/error/">
|
|
Error handling and debugging
|
|
</a>
|
|
|
|
|
|
<a href="/docs/guides/benchmarking/" class="active">
|
|
Benchmarking
|
|
</a>
|
|
|
|
|
|
<a href="https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md">
|
|
gRPC Wire Format
|
|
</a>
|
|
|
|
|
|
<h8 style="margin-top:25%">Related Guides</h8>
|
|
<a href="https://developers.google.com/protocol-buffers/docs/overview">Protocol Buffers</a>
|
|
</div>
|
|
|
|
<div class="quickstartcol2" style="margin-top:4%">
|
|
<h3 style="margin-top:0px;">
|
|
Benchmarking
|
|
</h3>
|
|
|
|
|
|
|
|
<p class="lead">gRPC is designed to support high-performance
|
|
open-source RPCs in many languages. This document describes the
|
|
performance benchmarking tools, the scenarios considered by the tests,
|
|
and the testing infrastructure.</p>
|
|
|
|
<div id="toc"></div>
|
|
|
|
<p><a name="Overview"></a></p>
|
|
|
|
<h3 id="overview">Overview</h3>
|
|
|
|
<p>gRPC is designed for both high-performance and high-productivity
|
|
design of distributed applications. Continuous performance
|
|
benchmarking is a critical part of the gRPC development
|
|
workflow. Multi-language performance tests run hourly against
|
|
the master branch, and these numbers are reported to a dashboard for
|
|
visualization.</p>
|
|
|
|
<ul>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528">Multi-language performance dashboard @latest_release (lastest available stable release)</a></li>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5652536396611584">Multi-language performance dashboard @master (latest dev version)</a></li>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5685265389584384">C++ detailed performance dashboard @master (latest dev version)</a></li>
|
|
</ul>
|
|
|
|
<p>Additional benchmarking provides fine grained insights into where
|
|
CPU is spent.</p>
|
|
|
|
<ul>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5684961520648192">C++ full-stack microbenchmarks</a></li>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5740240702537728">C Core filter benchmarks</a></li>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5641826627223552&container=789696829&widget=512792852">C Core shared component benchmarks</a></li>
|
|
<li><a href="https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5732910535540736">C Core HTTP/2 microbenchmarks</a></li>
|
|
</ul>
|
|
|
|
<h3 id="performance-testing-design">Performance testing design</h3>
|
|
|
|
<p>Each language implements a performance testing worker that implements
|
|
a gRPC
|
|
<a href="https://github.com/grpc/grpc/blob/master/src/proto/grpc/testing/worker_service.proto">WorkerService</a>. This
|
|
service directs the worker to act as either a client or a server for
|
|
the actual benchmark test, represented as
|
|
<a href="https://github.com/grpc/grpc/blob/master/src/proto/grpc/testing/benchmark_service.proto">BenchmarkService</a>. That
|
|
service has two methods:</p>
|
|
|
|
<ul>
|
|
<li>UnaryCall - a unary RPC of a simple request that specifies the number of bytes to return in the response</li>
|
|
<li>StreamingCall - a streaming RPC that allows repeated ping-pongs of request and response messages akin to the UnaryCall</li>
|
|
</ul>
|
|
|
|
<p><img src="/img/testing_framework.png" alt="gRPC performance testing worker diagram" /></p>
|
|
|
|
<p>These workers are controlled by a
|
|
<a href="https://github.com/grpc/grpc/blob/master/test/cpp/qps/qps_json_driver.cc">driver</a>
|
|
that takes as input a scenario description (in JSON format) and an
|
|
environment variable specifying the host:port of each worker process.</p>
|
|
|
|
<p><a name="Languages under test"></a></p>
|
|
|
|
<h3 id="languages-under-test">Languages under test</h3>
|
|
|
|
<p>The following languages have continuous performance testing as both
|
|
clients and servers at master:</p>
|
|
|
|
<ul>
|
|
<li>C++</li>
|
|
<li>Java</li>
|
|
<li>Go</li>
|
|
<li>C#</li>
|
|
<li>node.js</li>
|
|
<li>Python</li>
|
|
<li>Ruby</li>
|
|
</ul>
|
|
|
|
<p>Additionally, all languages derived from C core have limited
|
|
performance testing (smoke testing) conducted at every pull request.</p>
|
|
|
|
<p>In addition to running as both the client-side and server-side of
|
|
performance tests, all languages are tested as clients against a C++
|
|
server, and as servers against a C++ client. This test aims to provide
|
|
the current upper bound of performance for a given language’s client or
|
|
server implementation without testing the other side.</p>
|
|
|
|
<p>Although PHP or mobile environments do not support a gRPC server
|
|
(which is needed for our performance tests), their client-side
|
|
performance can be benchmarked using a proxy WorkerService written in
|
|
another language. This code is implemented for PHP but is not yet in
|
|
continuous testing mode.</p>
|
|
|
|
<p><a name="Scenarios under test"></a></p>
|
|
|
|
<h3 id="scenarios-under-test">Scenarios under test</h3>
|
|
|
|
<p>There are several important scenarios under test and displayed in the dashboards
|
|
above, including the following:</p>
|
|
|
|
<ul>
|
|
<li>Contentionless latency - the median and tail response latencies seen with only 1 client sending a single message at a time using StreamingCall</li>
|
|
<li>QPS - the messages/second rate when there are 2 clients and a total of 64 channels, each of which has 100 outstanding messages at a time sent using StreamingCall</li>
|
|
<li>Scalability (for selected languages) - the number of messages/second per server core</li>
|
|
</ul>
|
|
|
|
<p>Most performance testing is using secure communication and
|
|
protobufs. Some C++ tests additionally use insecure communication and
|
|
the generic (non-protobuf) API to display peak performance. Additional
|
|
scenarios may be added in the future.</p>
|
|
|
|
<p><a name="Testing infrastructure"></a></p>
|
|
|
|
<h3 id="testing-infrastructure">Testing infrastructure</h3>
|
|
|
|
<p>All performance benchmarks are run as instances in GCE through our
|
|
Jenkins testing infrastructure. In addition to the gRPC performance
|
|
scenarios described above, we also run baseline <a href="http://www.netperf.org">netperf
|
|
TCP_RR</a> latency numbers in order to understand
|
|
the underlying network characteristics. These numbers are present on
|
|
our dashboard and sometimes vary depending on where our instances
|
|
happen to be allocated within GCE.</p>
|
|
|
|
<p>Most test instances are 8-core systems, and these are used for both
|
|
latency and QPS measurement. For C++ and Java, we additionally support
|
|
QPS testing on 32-core systems. All QPS tests use 2 identical client machines
|
|
for each server, to make sure that QPS measurement is not client-limited.</p>
|
|
|
|
</div>
|
|
</div>
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
|
|
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
|
|
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/js/bootstrap.min.js" integrity="sha384-ChfqqxuZUCnJSK3+MXmPNIyE6ZbWh2IMqE241rYiqJxyMiZ6OW/JmZQ5stwEULTy" crossorigin="anonymous"></script>
|
|
|
|
</body>
|
|
</html>
|