Use a load-aware balancer (#251)

Currently, the conduit proxy uses a simplistic Round-Robin load
balancing algorithm. This strategy degrades severely when individual
endpoints exhibit abnormally high latency.

This change improves this situation somewhat by making the load balancer
aware of the number of outstanding requests to each endpoint. When nodes
exhibit high latency, they should tend to have more pending requests
than faster nodes; and the Power-of-Two-Choices node selector can be
used to distribute requests to lesser-loaded instances.

From the finagle guide:

    The algorithm randomly picks two nodes from the set of ready endpoints
    and selects the least loaded of the two. By repeatedly using this
    strategy, we can expect a manageable upper bound on the maximum load of
    any server.

    The maximum load variance between any two servers is bound by
    ln(ln(n))` where `n` is the number of servers in the cluster.

Signed-off-by: Oliver Gould <ver@buoyant.io>

Cargo.lock

@@ -147,6 +147,7 @@ dependencies = [
  "prost 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "prost-types 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "quickcheck 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "rand 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
  "tokio-connect 0.1.0 (git+https://github.com/carllerche/tokio-connect)",
  "tokio-core 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)",
  "tokio-io 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",

proxy/Cargo.toml

@@ -22,6 +22,7 @@ hyper = { version = "0.11.15", features = ["compat"] }
 ipnet = "1.0"
 log = "0.3"
 ordermap = "0.2"
+rand = "0.4"
 url = "1.5"
 
 tokio-core = "0.1"

proxy/src/lib.rs

@@ -28,6 +28,7 @@ extern crate prost_types;
 #[cfg(test)]
 #[macro_use]
 extern crate quickcheck;
+extern crate rand;
 extern crate tokio_connect;
 extern crate tokio_core;
 extern crate tokio_io;

proxy/src/outbound.rs

@@ -1,9 +1,8 @@
-use std::sync::Arc;
-
 use http;
-
+use rand;
+use std::sync::Arc;
 use tower;
-use tower_balance::{self, choose, Balance};
+use tower_balance::{self, choose, load, Balance};
 use tower_buffer::Buffer;
 use tower_h2;
 use conduit_proxy_router::Recognize;
@@ -49,8 +48,8 @@ where
     type Key = (FullyQualifiedAuthority, Protocol);
     type RouteError = ();
     type Service = Buffer<Balance<
-        Discovery<B>,
-        choose::RoundRobin, // TODO: better load balancer.
+        load::WithPendingRequests<Discovery<B>>,
+        choose::PowerOfTwoChoices<rand::ThreadRng>,
     >>;
 
     fn recognize(&self, req: &Self::Request) -> Option<Self::Key> {
@@ -89,8 +88,9 @@ where
             self.bind.clone().with_protocol(protocol),
         );
 
-        // TODO: move to p2c lb.
-        let balance = tower_balance::round_robin(resolve);
+        let loaded = tower_balance::load::WithPendingRequests::new(resolve);
+
+        let balance = tower_balance::power_of_two_choices(loaded, rand::thread_rng());
 
         // Wrap with buffering. This currently is an unbounded buffer,
         // which is not ideal.