From 221d4e817c07a9de2af531bf79f360755367de52 Mon Sep 17 00:00:00 2001
From: Sally O'Malley <somalley@redhat.com>
Date: Tue, 9 Aug 2022 14:12:08 -0400
Subject: [PATCH] add to cluster-admin/system-traces docs

---
 .../cluster-administration/system-traces.md   | 31 ++++++++++++++++++-
 .../feature-gates.md                          |  6 ++--
 2 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/content/en/docs/concepts/cluster-administration/system-traces.md b/content/en/docs/concepts/cluster-administration/system-traces.md
index f324604b16..9f1e4e7b06 100644
--- a/content/en/docs/concepts/cluster-administration/system-traces.md
+++ b/content/en/docs/concepts/cluster-administration/system-traces.md
@@ -61,7 +61,7 @@ as the kube-apiserver is often a public endpoint.
 
 To enable tracing, enable the `APIServerTracing`
 [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-on the kube-apiserver. Also, provide the kube-apiserver with a tracing configration file
+on the kube-apiserver. Also, provide the kube-apiserver with a tracing configuration file
 with `--tracing-config-file=<path-to-config>`. This is an example config that records
 spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
 
@@ -76,6 +76,35 @@ samplingRatePerMillion: 100
 For more information about the `TracingConfiguration` struct, see
 [API server config API (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/#apiserver-k8s-io-v1alpha1-TracingConfiguration).
 
+### kubelet traces
+
+{{< feature-state for_k8s_version="v1.25" state="alpha" >}}
+
+The kubelet CRI interface and authenticated http servers are instrumented to generate
+trace spans. As with the apiserver, the endpoint and sampling rate are configurable.
+Trace context propagation is also configured. A parent span's sampling decision is always respected.
+A provided tracing configuration sampling rate will apply to spans without a parent.
+Enabled without a configured endpoint, the default OpenTelemetry Collector reciever address of "localhost:4317" is set.
+
+#### Enabling tracing in the kubelet
+
+To enable tracing, enable the `KubeletTracing`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+on the kubelet. Also, provide the kubelet with a
+[tracing configuration](https://github.com/kubernetes/component-base/blob/release-1.25/tracing/api/v1/types.go).
+This is an example snippet of a kubelet config that records spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
+
+```yaml
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+featureGates:
+  KubeletTracing: true
+tracing:
+  # default value
+  #endpoint: localhost:4317
+  samplingRatePerMillion: 100
+```
+
 ## Stability
 
 Tracing instrumentation is still under active development, and may change
diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md
index 254ebdf51a..a3b4991dc6 100644
--- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md
+++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md
@@ -625,6 +625,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `APIResponseCompression`: Compress the API responses for `LIST` or `GET` requests.
 - `APIServerIdentity`: Assign each API server an ID in a cluster.
 - `APIServerTracing`: Add support for distributed tracing in the API server.
+  See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/traces-for-kubernetes-system-components) for more details.
 - `Accelerators`: Provided an early form of plugin to enable Nvidia GPU support when using
   Docker Engine; no longer available. See
   [Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for
@@ -998,9 +999,8 @@ Each feature gate is designed for enabling/disabling a specific feature:
   feature-specific labels provided by `NodeDisruptionExclusion` and `ServiceNodeExclusion`.
 - `KubeletTracing`: Add support for distributed tracing in the kubelet.
   When enabled, kubelet CRI interface and authenticated http servers are instrumented to generate
-  OpenTelemetry trace spans. Sampling rate and collector address are configurable. When enabled without
-  a tracing configuration, a parent span's sampling decision is respected and the default collector
-  address of 0.0.0.0:4317 is set.
+  OpenTelemetry trace spans.
+  See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/traces-for-kubernetes-system-components) for more details.
 - `LegacyServiceAccountTokenNoAutoGeneration`: Stop auto-generation of Secret-based
   [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens).
 - `LocalStorageCapacityIsolation`: Enable the consumption of