Update stats inclusion to use proxy config instead (#8325)

* update stats inclusion to use proxy config instead

* lint

* lint

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Douglas Reid <douglas-reid@users.noreply.github.com>

* fix

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Rachael Graham <rachael.graham@ibm.com>

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Rachael Graham <rachael.graham@ibm.com>

* Update content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

Co-authored-by: Rachael Graham <rachael.graham@ibm.com>

Co-authored-by: Douglas Reid <douglas-reid@users.noreply.github.com>
Co-authored-by: Rachael Graham <rachael.graham@ibm.com>

content/en/docs/ops/configuration/telemetry/envoy-stats/index.md

@@ -21,16 +21,22 @@ To see the statistics for a pod:
 $ kubectl exec $POD -c istio-proxy -- pilot-agent request GET stats
 {{< /text >}}
 
-See [the Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats)
-for an explanation of the data recorded.
+Envoy generates statistics about its behavior, scoping the statistics by proxy function. Examples include:
 
-By default, Istio configures Envoy to record minimal statistics.  The default collection
+- [Upstream connection](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats)
+- [Listener](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats)
+- [HTTP Connection Manager](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats)
+- [TCP proxy](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/tcp_proxy_filter#statistics)
+- [Router](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter.html?highlight=vhost#statistics)
+
+By default, Istio configures Envoy to record a minimal set of statistics to reduce the overall CPU and memory footprint of the installed proxies. The default collection
 keys are:
 
 - `cluster_manager`
 - `listener_manager`
 - `server`
 - `cluster.xds-grpc`
+- `wasm`
 
 To see the Envoy settings for statistics data collection use
 [`istioctl proxy-config bootstrap`](/docs/reference/commands/istioctl/#istioctl-proxy-config-bootstrap) and follow the
@@ -38,13 +44,39 @@ To see the Envoy settings for statistics data collection use
 Envoy only collects statistical data on items matching the `inclusion_list` within
 the `stats_matcher` JSON element.
 
-To Configure Envoy to record statistics for inbound or outbound traffic, add the
-`sidecar.istio.io/statsInclusionPrefixes` annotation to the pod template in the Kubernetes `Deployment`.
-Add the `cluster.outbound` prefix to gather data about outbound traffic activity and circuit breaking.
-To gather data on inbound traffic, add the `listener` prefix.  The sample
-[fortio-deploy.yaml]({{< github_file>}}/samples/httpbin/sample-client/fortio-deploy.yaml)
-shows use of `sidecar.istio.io/statsInclusionPrefixes` with the `cluster.outbound` prefix.
+{{< tip >}}
+Note: The names of Envoy statistics can vary based on the composition of Envoy configuration. As a result, the exposed names of statistics for Envoys managed by Istio are subject to the configuration behavior of Istio.
+If you build or maintain dashboards or alerts based on Envoy statistics, it is **strongly recommended** that you examine the
+statistics in a canary environment **before upgrading Istio**.
+{{< /tip >}}
 
-You can override the Envoy defaults to gather less data than usual.  Use
-`sidecar.istio.io/statsInclusionPrefixes: cluster_manager,listener_manager`
-to collect the least statistics possible.
+To configure Istio proxy to record additional statistics, you can add [`ProxyConfig.ProxyStatsMatcher`](/docs/reference/config/istio.mesh.v1alpha1/#ProxyStatsMatcher) to your mesh config. For example, to enable stats for circuit breaker, retry, and upstream connections globally, you can specify stats matcher as follow:
+
+{{< tip >}}
+Proxy needs to restart to pick up the stats matcher configuration.
+{{< /tip >}}
+
+{{< text yaml >}}
+proxyStatsMatcher:
+  inclusionRegexps:
+    - ".*circuit_breakers.*"
+  inclusionPrefixes:
+    - "upstream_rq_retry"
+    - "upstream_cx"
+{{< /text >}}
+
+You can also override the global stats matching configuration per proxy by using the `proxy.istio.io/config` annotation. For example, to configure the same stats generation inclusion as above, you can add the annotation to a gateway proxy or a workload as follow:
+
+{{< text yaml >}}
+proxy.istio.io/config: |-
+  proxyStatsMatcher:
+    inclusionRegexps:
+    - ".*circuit_breakers.*"
+    inclusionPrefixes:
+    - "upstream_rq_retry"
+    - "upstream_cx"
+{{< /text >}}
+
+{{< tip >}}
+Note: If you are using `sidecar.istio.io/statsInclusionPrefixes`, `sidecar.istio.io/statsInclusionRegexps`, and `sidecar.istio.io/statsInclusionSuffixes`, consider switching to the `ProxyConfig`-based configuration as it provides a global default and a uniform way to override at the gateway and sidecar proxy.
+{{< /tip >}}