Initial Telemetry API docs (#10496)

* Initial Telemetry API docs

Currently, the docs only have Telemetry for tracing. This adds a common
Telemetry API doc for high level info, shared for all 3 types. It also
adds some info about access logging via Telemetry.

We should likely add a similar page for Metrics, but I did not do it
here.

* fix gen

* fix dead links

* fix typo

* dead link

* one more

* 1 more?

* I hope this is it...

* again???
This commit is contained in:
John Howard 2021-11-17 17:20:09 -08:00 committed by GitHub
parent a851161e68
commit ad49bf308d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
14 changed files with 209 additions and 208 deletions

View File

@ -4,4 +4,4 @@ weight: 40
---
Istio, via Envoy, currently supports a percentage-based sampling strategy for trace generation.
Please see [this section](/docs/tasks/observability/distributed-tracing/configurability/#trace-sampling) for more information on how to set this sampling rate.
Please see [this section](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling) for more information on how to set this sampling rate.

View File

@ -7,7 +7,7 @@ Since Istio 1.0.3, the sampling rate for tracing has been reduced to 1% in the `
[configuration profile](/docs/setup/additional-setup/config-profiles/).
This means that only 1 out of 100 trace instances captured by Istio will be reported to the tracing backend.
The sampling rate in the `demo` profile is still set to 100%. See
[this section](/docs/tasks/observability/distributed-tracing/configurability/#trace-sampling)
[this section](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling)
for more information on how to set the sampling rate.
If you still do not see any trace data, please confirm that your ports conform to the Istio [port naming conventions](/about/faq/#naming-port-convention) and that the appropriate container port is exposed (via pod spec, for example) to enable

View File

@ -1,7 +0,0 @@
---
title: Configuration
description: This task guides you on how to configure Istio tracing behavior.
weight: 2
keywords: [telemetry,tracing]
test: n/a
---

View File

@ -1,190 +0,0 @@
---
title: Configuring tracing using the Telemetry API
description: How to configure tracing options using the Telemetry API.
weight: 10
keywords: [telemetry,tracing]
owner: istio/wg-policies-and-telemetry-maintainers
test: no
status: Experimental
---
Istio provides a Telemetry API that enables flexible configuration of tracing behavior. The
Telemetry API offers control over tracing options such as sampling rates and custom tags for individual
spans, as well as backend provider selection.
## Before you begin
1. Ensure that your applications [propagate tracing headers](/docs/tasks/observability/distributed-tracing/overview/).
1. Follow the tracing installation guide located under [Integrations](/docs/ops/integrations/)
to install your preferred tracing provider.
## Telemetry API: Tracing Overview
The Telemetry API offers tracing behavior configuration control over the following at the mesh, namespace, and workload levels:
- **provider selection** - allows selection of [backend providers](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider) for reporting.
- **sampling percentage** - allows control of the rate of trace sampling applied to received requests *for which no prior sampling decision has been made*.
- **custom tags** - allows control over any custom tags to add to each generated tracing span.
- **tracing participation** - allows opting services out of reporting spans to the selected tracing providers.
### Workload Selection
Individual workloads within a namespace are selected via a [`selector`](/docs/reference/config/type/workload-selector/#WorkloadSelector)
which allows label-based selection of workloads.
It is not valid to have two different `Telemetry` resources select the same workload using `selector`. Likewise, it is not valid to have two
distinct `Telemetry` resources in a namespace with no `selector` specified.
### Scope, Inheritance, and Overrides
Telemetry API resources inherit configuration from parent resources in the Istio configuration hierarchy:
1. root configuration namespace (example: `istio-system`)
1. local namespace (namespace-scoped resource with **no** workload `selector`)
1. workload (namespace-scoped resource with a workload `selector`)
A Telemetry API resource in the root configuration namespace, typically `istio-system`, provides mesh-wide defaults for behavior.
Any workload-specific selector in the root configuration namespace will be ignored/rejected. It is not valid to define multiple
mesh-wide Telemetry API resources in the root configuration namespace.
Namespace-specific overrides for the mesh-wide configuration can be achieved by applying a new `Telemetry` resource in the desired
namespace (without a workload selector). Any `Tracing` fields specified in the namespace configuration will completely override
the field from the parent configuration (in the root configuration namespace).
Workload-specific overrides can be achieved by applying a new Telemetry resource in the desired namespace *with a workload selector*.
Any `Tracing` fields specified in the namespace configuration will completely override the field from any parent configuration
(root configuration or local namespace).
## Using the Telemetry API for Tracing Configuration
### Configuring tracing providers
Tracing providers are the backend collectors and processors that receive tracing spans
and process them for storage and retrieval. Example providers include Zipkin, Jaeger, Lightstep,
Datadog, and Apache SkyWalking.
For Istio, tracing providers are configured for use within the mesh via `MeshConfig`. To configure new
providers to use in tracing, edit the `MeshConfig` for your mesh via:
{{< text bash >}}
$ kubectl -n istio-system edit configmap istio
{{< /text >}}
The full set of configuration options is described in the [reference docs for MeshConfig](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig).
Typical configuration includes service address and port for the provider, as well as establishing
a limit on max tag length supported by the provider.
Each configured provider *must* be uniquely named. That name will be used to refer to the provider in the
Telemetry API.
An example set of provider configuration in `MeshConfig` is:
{{< text yaml >}}
data:
mesh: |-
extensionProviders: # The following content defines two example tracing providers.
- name: "localtrace"
zipkin:
service: "zipkin.istio-system.svc.cluster.local"
port: 9411
maxTagLength: 56
- name: "cloudtrace"
stackdriver:
maxTagLength: 256
defaultProviders: # If a default provider is not specified, Telemetry resources must fully-specify a provider
tracing:
- "cloudtrace"
{{< /text >}}
### Configuring mesh-wide tracing behavior
Telemetry API resources inherit from the root configuration namespace for a mesh, typically `istio-system`. To configure
mesh-wide behavior, add a new (or edit the existing) `Telemetry` resource in the root configuration namespace.
Here is an example configuration that uses the provider configuration from the prior section:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers: # only a single tracing provider is supported at this time
- name: localtrace
customTags:
foo:
literal:
value: bar
randomSamplingPercentage: 100
{{< /text >}}
This configuration overrides the default provider from `MeshConfig`, setting the mesh default to be the `"localtrace"`
provider. It also sets the mesh-wide sampling percentage to be `100`, and configures a tag to be added to all trace
spans with a name of `foo` and a value of `bar`.
### Configuring namespace-scoped tracing behavior
To tailor the tracing behavior for individual namespaces, add a `Telemetry` resource to the desired namespace. Any tracing
fields specified in the namespace resource will completely override the inherited field configuration from the configuration hierarchy. For example:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: namespace-override
namespace: myapp
spec:
tracing:
- customTags:
userId:
header:
name: userId
defaultValue: unknown
{{< /text >}}
When deployed with into a mesh with the prior mesh-wide example configuration, this will result in
tracing behavior in the `myapp` namespace that sends trace spans to the `localtrace` provider and
randomly selects requests for tracing at a `100%` rate, but that sets custom tags for each span with
a name of `userId` and a value taken from the `userId` request header. Importantly, the `foo: bar` tag
from the parent configuration will not be used in the `myapp` namespace. The custom tags behavior completely
overrides the behavior configured in the `mesh-default.istio-system` resource.
{{< tip >}}
Any tracing configuration in a `Telemetry` resource completely overrides configuration of its parent resource in the
configuration hierarchy. This includes provider selection.
{{< /tip >}}
### Configuring workload-specific tracing behavior
To tailor the tracing behavior for individual workloads, add a `Telemetry` resource to the desired namespace and use a
`selector`. Any tracing fields specified in the workload-specific resource will completely override the inherited
field configuration from the configuration hierarchy.
For example:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: workload-override
namespace: myapp
spec:
selector:
matchLabels:
service.istio.io/canonical-name: frontend
tracing:
- disableSpanReporting: true
{{< /text >}}
In this case, tracing will be disabled for the `frontend` workload in the `myapp` namespace. Istio will still forward the tracing headers,
but no spans will be reported to the configured tracing provider.
{{< tip >}}
It is not valid to have two `Telemetry` resources with workload selectors select the same workload. In those cases,
Istio tracing behavior is undefined.
{{< /tip >}}

View File

@ -22,7 +22,7 @@ To learn how Istio handles tracing, visit this task's [overview](../overview/).
1. When you enable tracing, you can set the sampling rate that Istio uses for tracing.
Use the `meshConfig.defaultConfig.tracing.sampling` option during installation to
[set the sampling rate](/docs/tasks/observability/distributed-tracing/configurability/#customizing-trace-sampling).
[set the sampling rate](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling).
The default sampling rate is 1%.
1. Deploy the [Bookinfo](/docs/examples/bookinfo/#deploying-the-application) sample application.

View File

@ -178,7 +178,7 @@ server-side span of the call. The screenshot shows that the call took 14.60 ms.
## Trace sampling
Istio captures traces at a configurable trace sampling percentage. To learn how to modify the trace sampling percentage,
visit the [Distributed Tracing trace sampling section](../configurability/#trace-sampling).
visit the [Distributed Tracing trace sampling section](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling).
When using Lightstep, we do not recommend reducing the trace sampling percentage below 100%. To handle a high traffic mesh, consider scaling up the size of your satellite pool.

View File

@ -5,13 +5,14 @@ weight: 11
keywords: [telemetry,tracing]
aliases:
- /docs/tasks/observability/distributed-tracing/configurability/
- /docs/tasks/observability/distributed-tracing/configurability/mesh-and-proxy-config/
owner: istio/wg-policies-and-telemetry-maintainers
test: no
status: Beta/Experimental
---
{{< tip >}}
Users are encouraged to transition to the [Telemetry API](/docs/tasks/observability/distributed-tracing/configurability/telemetry-api/) for tracing configuration.
Users are encouraged to transition to the [Telemetry API](/docs/tasks/observability/telemetry/) for tracing configuration.
{{</ tip >}}
Istio provides the ability to configure advanced tracing options,

View File

@ -22,7 +22,7 @@ To learn how Istio handles tracing, visit this task's [overview](../overview/).
1. When you enable tracing, you can set the sampling rate that Istio uses for tracing.
Use the `meshConfig.defaultConfig.tracing.sampling` option during installation to
[set the sampling rate](/docs/tasks/observability/distributed-tracing/configurability/#customizing-trace-sampling).
[set the sampling rate](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling).
The default sampling rate is 1%.
1. Deploy the [Bookinfo](/docs/examples/bookinfo/#deploying-the-application) sample application.

View File

@ -1,5 +1,5 @@
---
title: Getting Envoy's Access Logs
title: Envoy Access Logs
description: This task shows you how to configure Envoy proxies to print access logs to their standard output.
weight: 10
keywords: [telemetry,logs]
@ -21,6 +21,31 @@ The standard output of Envoy's containers can then be printed by the `kubectl lo
## Enable Envoy's access logging
Istio offers a few ways to enable access logs. Use of the Telemetry API is recommended
### Using Telemetry API
The Telemetry API can be used to enable or disable access logs:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
accessLogging:
- {}
{{< /text >}}
The above example provides and empty object (`{}`), as we do not want to configure anything other than default settings.
Similar configuration can also be applied on an individual namespace, or to an individual workload, to control logging at a fine grained level.
For more information about using the Telemetry API, see the [Telemetry API overview](/docs/tasks/observability/telemetry/).
### Using Mesh Config
If you used an `IstioOperator` CR to install Istio, add the following field to your configuration:
{{< text yaml >}}

View File

@ -22,7 +22,18 @@
source "content/en/boilerplates/snips/before-you-begin-egress.sh"
source "content/en/boilerplates/snips/start-httpbin-service.sh"
! read -r -d '' snip_enable_envoys_access_logging_1 <<\ENDSNIP
! read -r -d '' snip_using_telemetry_api_1 <<\ENDSNIP
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
accessLogging:
- {}
ENDSNIP
! read -r -d '' snip_using_mesh_config_1 <<\ENDSNIP
spec:
meshConfig:
accessLogFile: /dev/stdout

View File

@ -0,0 +1,161 @@
---
title: Telemetry API
description: This task shows you how to configure the Telemetry API.
weight: 0
keywords: [telemetry]
owner: istio/wg-policies-and-telemetry-maintainers
test: no
---
Istio provides a [Telemetry API](/docs/reference/config/telemetry/) that enables flexible configuration of
[metrics](/docs/tasks/observability/metrics/), [access logs](/docs/tasks/observability/logs/), and [tracing](/docs/tasks/observability/distributed-tracing/).
## Using the API
### Scope, Inheritance, and Overrides
Telemetry API resources inherit configuration from parent resources in the Istio configuration hierarchy:
1. root configuration namespace (example: `istio-system`)
1. local namespace (namespace-scoped resource with **no** workload `selector`)
1. workload (namespace-scoped resource with a workload `selector`)
A Telemetry API resource in the root configuration namespace, typically `istio-system`, provides mesh-wide defaults for behavior.
Any workload-specific selector in the root configuration namespace will be ignored/rejected. It is not valid to define multiple
mesh-wide Telemetry API resources in the root configuration namespace.
Namespace-specific overrides for the mesh-wide configuration can be achieved by applying a new `Telemetry` resource in the desired
namespace (without a workload selector). Any fields specified in the namespace configuration will completely override
the field from the parent configuration (in the root configuration namespace).
Workload-specific overrides can be achieved by applying a new Telemetry resource in the desired namespace *with a workload selector*.
### Workload Selection
Individual workloads within a namespace are selected via a [`selector`](/docs/reference/config/type/workload-selector/#WorkloadSelector)
which allows label-based selection of workloads.
It is not valid to have two different `Telemetry` resources select the same workload using `selector`. Likewise, it is not valid to have two
distinct `Telemetry` resources in a namespace with no `selector` specified.
### Provider Selection
The Telemetry API uses the concept of providers to indicate the protocol or type of integration to use. Providers can be configured in [`MeshConfig`](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider).
An example set of provider configuration in `MeshConfig` is:
{{< text yaml >}}
data:
mesh: |-
extensionProviders: # The following content defines two example tracing providers.
- name: "localtrace"
zipkin:
service: "zipkin.istio-system.svc.cluster.local"
port: 9411
maxTagLength: 56
- name: "cloudtrace"
stackdriver:
maxTagLength: 256
{{< /text >}}
For convenience, Istio comes with a few providers configured out of the box with default settings:
|Provider Name|Functionality|
|-------------|-------------|
|`prometheus` |Metrics|
|`stackdriver`|Metrics, Tracing, Access Logging|
|`envoy` |Access Logging|
In additional, a [default provider](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-DefaultProviders) can be set which
will be used when the `Telemetry` resources do not specify a provider.
## Examples
### Configuring mesh-wide behavior
Telemetry API resources inherit from the root configuration namespace for a mesh, typically `istio-system`. To configure
mesh-wide behavior, add a new (or edit the existing) `Telemetry` resource in the root configuration namespace.
Here is an example configuration that uses the provider configuration from the prior section:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: localtrace
customTags:
foo:
literal:
value: bar
randomSamplingPercentage: 100
{{< /text >}}
This configuration overrides the default provider from `MeshConfig`, setting the mesh default to be the `localtrace`
provider. It also sets the mesh-wide sampling percentage to be `100`, and configures a tag to be added to all trace
spans with a name of `foo` and a value of `bar`.
### Configuring namespace-scoped tracing behavior
To tailor the behavior for individual namespaces, add a `Telemetry` resource to the desired namespace.
Any fields specified in the namespace resource will completely override the inherited field configuration from the configuration hierarchy.
For example:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: namespace-override
namespace: myapp
spec:
tracing:
- customTags:
userId:
header:
name: userId
defaultValue: unknown
{{< /text >}}
When deployed into a mesh with the prior mesh-wide example configuration, this will result in
tracing behavior in the `myapp` namespace that sends trace spans to the `localtrace` provider and
randomly selects requests for tracing at a `100%` rate, but that sets custom tags for each span with
a name of `userId` and a value taken from the `userId` request header.
Importantly, the `foo: bar` tag from the parent configuration will not be used in the `myapp` namespace.
The custom tags behavior completely overrides the behavior configured in the `mesh-default.istio-system` resource.
{{< tip >}}
Any configuration in a `Telemetry` resource completely overrides configuration of its parent resource in the configuration hierarchy. This includes provider selection.
{{< /tip >}}
### Configuring workload-specific behavior
To tailor the behavior for individual workloads, add a `Telemetry` resource to the desired namespace and use a
`selector`. Any fields specified in the workload-specific resource will completely override the inherited
field configuration from the configuration hierarchy.
For example:
{{< text yaml >}}
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: workload-override
namespace: myapp
spec:
selector:
matchLabels:
service.istio.io/canonical-name: frontend
tracing:
- disableSpanReporting: true
{{< /text >}}
In this case, tracing will be disabled for the `frontend` workload in the `myapp` namespace.
Istio will still forward the tracing headers, but no spans will be reported to the configured tracing provider.
{{< tip >}}
It is not valid to have two `Telemetry` resources with workload selectors select the same workload. In those cases, behavior is undefined.
{{< /tip >}}

View File

@ -30,7 +30,7 @@ We're pleased to announce the availability of Istio 1.0.3. Please see below for
- Pilot performance and scalability has been greatly enhanced. Pilot now delivers endpoint updates to 500 sidecars in under 1 second.
- Default [trace sampling](/docs/tasks/observability/distributed-tracing/configurability/#trace-sampling) is set to 1%.
- Default [trace sampling](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#trace-sampling) is set to 1%.
## Policy and telemetry

View File

@ -31,7 +31,7 @@ publishdate: 2020-05-21
## Telemetry
- ***Added*** experimental support for [request classification](/docs/tasks/observability/metrics/classify-metrics/) filters. This enables operators to configure new attributes for use in telemetry, based on request information. A primary use case for this feature is labeling of traffic by API method.
- ***Added*** an experimental [mesh-wide tracing configuration API](/docs/tasks/observability/distributed-tracing/configurability/). This API provides control of trace sampling rates, the [maximum tag lengths](https://github.com/istio/istio/issues/14563) for URL tags, and [custom tags extraction](https://github.com/istio/istio/issues/13018) for all traces within the mesh.
- ***Added*** an experimental [mesh-wide tracing configuration API](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/). This API provides control of trace sampling rates, the [maximum tag lengths](https://github.com/istio/istio/issues/14563) for URL tags, and [custom tags extraction](https://github.com/istio/istio/issues/13018) for all traces within the mesh.
- ***Added*** standard Prometheus scrape annotations to proxies and the control plane workloads, which improves the Prometheus integration experience. This removes the need for specialized configuration to discover and consume Istio metrics. More details are available in the [operations guide for Prometheus](/docs/ops/integrations/prometheus#option-2-metrics-merging/).
- ***Added*** the ability for mesh operators to add and remove labels used in Istio metrics, based on expressions over the set of available request and response attributes. This improves Istio's support for [customizing v2 metrics generation](/docs/tasks/observability/metrics/customize-metrics/).
- ***Updated*** default telemetry v2 configuration to avoid using host header to extract destination service name at the gateway. This prevents unbound cardinality due to an untrusted host header, and implies that destination service labels are going to be omitted for requests that hit `blackhole` and `passthrough` at the gateway.

View File

@ -150,7 +150,7 @@ features:
area: Observability
- name: "Trace Sampling"
id: "observability.trace_sampling"
link: "/docs/tasks/observability/distributed-tracing/configurability/#trace-sampling"
link: "/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#trace-sampling"
level:
checklist: ""
maturity: Beta