There's no reason that we should ever need to cache proxy binaries
between CI runs.
For instance, for each pull request build we appear to cache a ~125M
binary.
Cache interactions seem to account for much of our CI times, so any
reduction here is helpful.
The `DstLabels` type has a large an undefined scope. Over time, it's
become critical in a variety of contexts, even outside of the scope of
prometheus endpoint labels.
This change makes `DstLabels` private to the
`telemetry::metrics::labels` module. This more clearly separates the
concerns of prometheus labeling from discovery, etc.
Now, destination metadata includes an index map of arbitrary metadata
labels.
Previously, dst label strings would be formatted eagerly at
service-discovery-time. This change moves this string formatting into
the data path(!). I think this is an acceptable tradeoff temporarily,
while we work to stop constructing RequestLabels in the data path
altogether.
There are many different types of things that implement `fmt::Display`
in the metrics library: lists of key-value labels, individial label keys
or values, groups of metrics output, etc. While modifying metrics code,
it can be very easy to use something that's not a label in the context
of a label.
In order to start dismantling the monolithic ctx structures, this change
removes the root `ctx::Process` type. This simplifies `ctx::Proxy` such
that is copyable and need not be `Arc`ed.
`telemetry::metrics::labels::Direction` has been updated to decorate
`ctx::Proxy` instead of modeling the same variants directly as an enum.
Previously, transport metrics have been stored in two "scopes": an
"open" scope, storing metrics by transport class, and a "close" scope
storing metrics by transport and end-of-stream classes.
Instead of storing these as parallel maps, this changes transport
metrics to be stored hierarchically so that the end-of-stream metrics
are stored _within_ the transport metrics type.
This will make it possible to share the transport metrics structure
directly with the transport wrapper so that a global lock need only be
taken at connect-time. Subsequent updates will then be scoped to the
shared transport structure.
This helps to setup linkerd/linkerd2#1430, among other improvements.
Furthermore, the `Scopes` type is no longer used for transport metrics,
since it's much easier to use the full affordances of `indexmap`,
especially now that it isn't part of an API.
The `telemetry::metrics::transport` module exposes much of its
implementation details to callers, which makes it difficult to make
changes to how the module is structured. In preparation for further
refactors, this change narrows the module's public API:
All labels and scopes types have been made private. A single, public
`Transports` type has been introduce to describe the entire public
interface of the module. This has been crafted to be free of `event`
types and to have minimal external dependencies.
A new `transport::Eos` type has been introduced to replace the
overloaded `labels::Classification` type -- this type was initially
introduced to model _HTTP response_ classification, but it was
reused for transports. This is an undesirable coupling that will have to
be broken when we start to adddress HTTP response classification
properly.
005d4f1 made `Metric::fmt_scopes` generic over an Iterator, but
maintained some of the unnecessary type bounds inherited from its
initial implementation.
This change relaxes the constraints on the label value so that it need
not be passed as a reference or be hashable--our only requirement is
that it can be rendered.
The `tcp_connect_err` tests do not pass on Linux.
I initially observed that the errno produced is ECONNREFUSED on Linux,
not EXFULL, but even changing this does not help the test to pass
reliably, as the socket teardown semantics appear to be slightly
different.
In order to prevent spurious test failures during development, this
change marks these two tests as macos-specific.
Various transport-specific labels are defined in the common
`metrics::labels` module.
In preparation to refactor transport metrics into Sensor and Report
halves, this change moves all transport-specific labels into the
transport metrics module.
Only the `DstLabels` and `Serve` types in `metrics` are used outside of
`telemetry`.
This change narrows visibility so that `metrics` is not referenced
outside of `telemetry`.
The `Scopes` type shouldn't be critical to metrics formatting.
This change makes the `Metric` type unaware of `Scopes` so that other
labeling mechanisms can be used if appropriate.
As we begin to remove event-based metrics telemetry, the `record` bench
test becomes an obstacle that needs to be updated at every step.
Given that we have already removed the event channel and plan to remove
the `Record` type entirely, it's suitable to remove the bench test
entirely.
While this is done, the `test-benches` Make target and rust-nightly
travis stage are obsoleted.
The metrics::Scopes type exposes its internal implementation to many of
its uses.
By extracting the type into its own module, we are forced to provide an
explicit public interface, hiding its IndexMap implementation details.
Following on #52, this change moves the
`telemetry::metrics::tls_config_reload` module to
`telemetry::tls_config_reload`.
The `Fmt` type has been renamed to `Report`.
Furthermore, two helper methods have been added to `metrics::Scopes` so
that its internal representation need not be made public.
As a follow-on to #50, which begins to tease apart metrics
infrastructure from application logic, this change moves the
`telemetry::metrics::process` module into `telemetry::process`.
The `Process` type has been renamed to `Report` and some other minor
cosmetic changes have been made.
The `telemetry::metrics` module has a mix of core metrics types and
application-specific code. I'd like to change this module to only
contain core types. All app-specific module will be moved into
`telemetry`.
In order to make these changes, the `metrics!` macro (and related types)
must be visible to to `telemetry`. This change moves the macro
definition into the parent mod.rs, and makes the `Metric`, `FmtMetric`,
and `Scopes` types public.
In subsequent PRs, app-specific logic will be moved from
`telemetry::metrics` to `telemetry`, and the `telemetry::metrics`
module's public API will be minimized.
Metric eviction was introduced to protect against the situation where,
over time, the proxy addresses an effectively unbounded number of
endpoints. Metrics with endpoint-specific dimensions must be removed
over time to prevent a form of memory leak.
Eviction is implemented for transport stats. However, transport stats do
not yet contain per-endpoint dimensions. The eviction logic is
superfluous in this case, since transport metrics are bounded and small.
In preparation of changes to transport telemetry, this disables tracking
of transport metrics for eviction. This logic will be restored when we
support per-endpoint transport metrics.
The `mk_err_enum` macro is only used in one place: the `labels::errno` module.
In order to make the main `labels` module simpler, the macro definition
can by moved into the errno module. If this macro is really generally
useful, it can be factored out later.
The current implementation of TLS config reload telemetry has several
layers: a sensor emits events into a dispatcher that updates metrics.
This can be simplified by sharing a common metrics object between the
metrics registry and config module.
The metrics registry is updated to hold a read-only `tls_config_reload::Fmt`;
and the config module holds a `tls_config_reload::Sensor`.
The `Sensor` type holds a strong reference to an inner metrics
structure and acquires a lock on updates.
The `Fmt` type holds a weak reference to the metrics structure so
that the metrics server can print the state as long as it's actually held
in memory. If the `Sensor` is dropped (for instance, because TLS has
been administratively disabled), then no metrics will be formatted by
`Fmt`.
Details of TLS configuration reload metrics span several modules.
This change consolidates all of this logic into a single
module with a single public type, `TlsConfigReload`, that supports
recording successful reloads, recording errors, and formatting its
current state.
This sets up further simplifications, eventually leading to the removal
of the `Event` API.
The `process` module exposes a `Sensor` type that is different from
other types called `Sensor`. Most `Sensor` types instrument other
types with telemetry. The `process::Sensor` type, on the other hand,
is used to read system metrics from the `/proc` filesystem, returning
a metrics summary.
Furthermore, `telemetry::metrics::Root` owns the process start time
metric.
In the interest of making the telemetry system more modular, this moves
all process-related telemetry concerns into the `process` module.
Instead of exposing a `Sensor` that produces metrics, a single public
`Process` type implements `fmt::Display` directly.
This removes process-related concerns from `telemetry/metrics/mod.rs` to
setup further refactoring along these lines.
This branch should not make any functional changes.
This branch makes two minor refactorings to the `client` module in
`control::destination::background`:
1. Remove the `AddOrigin` middleware and replace it with the
`tower-add-origin` crate from `tower-http`. These middlewares are
functionally identical, but the Tower version has tests.
2. Change `ClientService` from a type alias to a tuple struct. This
means that some of the middleware that are used only in this module
(`LogErrors` and `Backoff`) are no longer part of a publicly visible
type and can be made private to the module.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
When the destination service returns a hint that an endpoint is another
proxy, eligible HTTP1 requests are translated into HTTP2 and sent over
an HTTP2 connection. The original protocol details are encoded in a
header, `l5d-orig-proto`. When a proxy receives an inbound HTTP2
request with this header, the request is translated back into its HTTP/1
representation before being passed to the internal service.
Signed-off-by: Sean McArthur <sean@buoyant.io>
Required for linkerd/linkerd2#1322.
Currently, the proxy places a limit on the number of active routes
in the route cache. This limit defaults to 100 routes, and is intended
to prevent the proxy from requesting more than 100 lookups from the
Destination service.
However, in some cases, such as Prometheus scraping a large number of
pods, the proxy hits this limit even though none of those requests
actually result in requests to service discovery (since Prometheus
scrapes pods by their IP addresses).
This branch implements @briansmith's suggestion in
https://github.com/linkerd/linkerd2/issues/1322#issuecomment-407161829.
It splits the router capacity limit to two separate, configurable
limits, one that sets an upper bound on the number of concurrently
active destination lookups, and one that limits the capacity of the
router cache.
I've done some preliminary testing using the `lifecycle` tests, where a
single Prometheus instance is configured to scrape a very large number
of proxies. In these tests, neither limit is reached. Furthermore, I've added
integration tests in `tests/discovery` to exercise the destination service
query limit. These tests ensure that query capacity is released when inactive
routes which create queries are evicted from the router cache, and that the
limit does _not_ effect DNS queries.
This branch obsoletes and closes#27, which contained an earlier version of
these changes.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Both tcp_connect_err tests frequently fail, even during local
development. This seems to happen because the proxy doesn't necessarily
observe the socket closure.
Instead of shutting down the socket gracefully, we can just drop it!
This helps the test pass much more reliably.
This is strictly a refactor which should make no functional changes.
Currently, the function used to construct new Destination service
queries (`Background::query_destination_service_if_relevant`) is a
function rather than a method on `Background`, although it takes as an
argument a field from `Background`. This is because in some cases, it is
called where `self.destinations` is borrowed mutably, preventing `self`
from being borrowed immutably.
Right now, this means that one additional field has to be passed
explicitly. However, in order to add the limit on active Destination
service queries, it was necessary to add two additional fields to
`Background` that have to be passed to this function. Since these
arguments should always come from fields on `Background`, it would be
preferable for this to be a method.
This branch breaks out some of the fields on
`control::destination::Background` into their own structs:
`DestinationCache`, which holds the map of DNS names to
`DestinationSet`s and the queue of DNS names that need reconnects; and
`Config`, which holds the configuration necessary to create a new
Destination service query (currently just the `Namespaces` config). This
allows us to have separate borrows on the `DestinationCache` and
`Config`.
When I make the additional changes necessary to add the limit on active
destination queries, the two additional fields necessary can be added to
`Confiq`, rather than having to explicitly pass them into
`query_destination_service_if_relevant` every time it's called.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This branch is purely a refactor and should result in no functional
changes.
The `control::destination::background` module has become quite large,
making the code difficult to read and review changes to. This branch
separates out the `DestinationSet` type and the destination service
client code into their own modules inside of `background`. Furthermore,
it rolls the `control::utils` module into the `client` submodule of
`background`, as that code is only used in the `client` module.
I think there's some additional work that can be done to make this code
clearer beyond simply splitting apart some of these large files, and I
intend to do some refactoring in additional follow-up branches.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This branch updates the proxy's `h2` dependency to v0.1.11. This version
removes a busy loop when shutting down an idle server
(carllerche/h2#296), and fixes a potential panic when dropping clients
(carllerche/h2#295).
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
The proxy's telemetry system is implemented with a channel: the proxy thread
generates events and the control thread consumes these events to record
metrics and satisfy Tap observations. This design was intended to minimize
latency overhead in the data path.
However, this design leads to substantial CPU overhead: the control thread's
work scales with the proxy thread's work, leading to resource contention in
busy, resource-limited deployments. This design also has other drawbacks in
terms of allocation & makes it difficult to implement planned features like
payload-aware Tapping.
This change removes the event channel so that all telemetry is recorded
instantaneously in the data path, setting up for further simplifications so
that, eventually, the metrics registry properly uses service lifetimes to
support eviction.
This change has a potentially negative side effect: metrics scrapes obtain
the same lock that the data path uses to write metrics so, if the metrics
server gets heavy traffic, it can directly impact proxy latency. These
effects will be ameliorated by future changes that reduce the need for the
Mutex in the proxy thread.
* Add initial tests for client and server connection handling w.r.t. TLS.
Add a simple framework for TLS connection handling and some initial tests
that use it.
An explicit effort has been made to keep the test configuration as close to the
production configuration as possible; e.g. we use regular TCP sockets instead of some
mock TCP sockets. This matters less now, but will matter more later, if/when we
implement more low-level TLS-over-TCP optimizations.
Rename `ConnectionConfig::identity` to `ConnectionConfig::server_identity` to make
it clearer that it is always the identity of the server, regardless of which role
the `ConnectionConfig` is being used in.
Signed-off-by: Brian Smith <brian@briansmith.org>
Specifically proxied bodies would make use of an optimization in hyper
that resulted in the connection not knowing (but it did know! just
didn't tell itself...) that the body was finished, and so the connection
was closed. 0.12.7 includes the fix in hyper.
As part of this upgrade, the keep-alive tests have been adjusted to send
a small body, since the empty body was not triggering this case.
This branch adds a label displaying the Unix error code name (or the raw
error code, on other operating systems or if the error code was not
recognized) to the metrics generated for TCP connection failures.
It also adds a couple of tests for label formatting.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
The way TLS is done for a bound port is fixed based on its role and whatever
the TLS settings are, so it makes sense to configure the TLS aspects of the
bound port during construction. This will also make writing tests easier.
Signed-off-by: Brian Smith <brian@briansmith.org>
Previously it wasn't possible to create objects that need to watch the TLS
configuration until the telemetry sensors were created. Split the watching
initialization into two parts so that in the (near) future TLS-using objects can
be created before the telemetry sensors are created.
This will allow us to initialize `BoundPort` with the TLS settings during
creation instead of later in `BoundPort::listen_and_fold()`. This will also
facilitate TLS testing.
Signed-off-by: Brian Smith <brian@briansmith.org>
Currently, the messages that the proxy logs on errors are often not
very useful. For example, when an error occurred that caused the proxy
to return HTTP 500, we log a message like this:
```
ERR! proxy={server=in listen=0.0.0.0:4143 remote=127.0.0.1:57416} linkerd2_proxy turning Error caused by underlying HTTP/2 error: protocol error: unexpected internal error encountered into 500
```
Note that:
+ Regardless of what the error actually was, the current log message
*always* says protocol error: unexpected internal error encountered,
which is both fairly unclear *and* often not actually the case.
+ Regardless of whether the error was encountered by an HTTP/1 or
HTTP/2 client, the error message always includes the string
"underlying HTTP/2 error". This is probably fairly confusing for
users who are, say, only proxying HTTP/1 traffic.
This branch fixes several related issues around the clarity of the
proxy's error messages:
+ A couple cases in the `transparency` module that returned
`io::Error::from(io::ErrorKind::Other)` have been replaced with
more descriptive errors that propagate the underlying error. This
necessitated adding bounds on some error types.
+ Introduced a new `transparency::client::Error` enum that can be
either a `h2::Error` or a `hyper::Error`, depending on whether
the client is HTTP/1 or HTTP/2, and proxies its' `std::error::Error`
impl to the wrapped error types. This way, we don't return a
`tower_h2::client::Error` (with format output that includes the
string "HTTP/2") from everything, and discard significantly less
error information.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Closeslinkerd/linkerd2#1272.
Currently, if TLS is enabled but the TLS configuration isn't available
(yet), the proxy will pass through all traffic to the application.
However, the destination service will tell other proxies to send TLS
traffic to the pod unconditionally, so the proxy will pass through TLS
handshakes to the application that are destined for the proxy itself.
In linkerd/linkerd2#1272, @briansmith suggested that we change the
proxy so that when it hasn't yet loaded a TLS configuration, it will
accept TLS handshakes, but fail them. This branch implements that
behaviour by making the `rustls::sign::CertifiedKey` in `CertResolver`
optional, and changing the `CertResolver` to return `None` when
`rustls` asks it to resolve a certificate in that case. The server
config watch is now initially created with `Conditional::Some` with an
empty `CertResolver`, rather than `Conditional::None(NoConfig)`, so
that the proxy will accept incoming handshakes, but fail them.
Signed-off-by: Eliza Weisman <eliza@buoyant.io
Fixeslinkerd/linkerd2#1239.
The proxy's `process_cpu_seconds_total` metric is currently calculated
incorrectly and will differ from the CPU stats reported by other
sources. This is because it currently calculates the CPU time by summing
the `utime` and `stime` fields of the stat struct returned by `procinfo`.
However, those numbers are expressed in _clock ticks_, not seconds, so
the metric is expressed in the wrong unit.
This branch fixes this issue by using `sysconf` to get the number of
clock ticks per second when the process sensor is created, and then
dividing `utime + stime` by that number, so that the value is expressed
in seconds.
## Demonstration:
(Note that the last column in `ps aux` output is the CPU time total)
```
eliza@ares:~$ ps aux | grep linkerd2-proxy | grep -v grep
eliza 40703 0.2 0.0 45580 14864 pts/0 Sl+ 13:49 0:03 target/debug/linkerd2-proxy
eliza@ares:~$ curl localhost:4191/metrics
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 19
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 46673920
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 15220736
# HELP process_start_time_seconds Time that the process started (in seconds since the UNIX epoch)
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1531428584
eliza@ares:~$
```
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This is purely aesthetic; the TLS logic doesn't care about the product name.
The test data was regenerated by re-running gen-certs.sh after modifying it.
Signed-off-by: Brian Smith <brian@briansmith.org>
Sean McArthur
Oliver Gould
Eliza Weisman
Sean McArthur
Markus Jais
Brian Smith