Attach kubernetes attributes to traces blogpost (#1454)
* Attach kubernetes attributes to traces blogpost Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Run markdown prettifier and made some copyedits * address some comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * address comments about resources vs attributes Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * change to internal url Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * update date, title and linktitle Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * minor changes Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * rephrase from attach spans to attach to resources Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * improvments to paragraphs Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * improve conclusions Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Copyedits * Run prettifier * Add link definition * Fix app spelling * address comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * address more comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com> * More comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Copyedits - fix use of link def and vert.x app name * change diagram Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * change name of files Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Use today's date Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
This commit is contained in:
Ruben Vargas
GitHub
parent
ae690a7e5e
commit
3a417e1cfb
|
|
@ -0,0 +1,260 @@
|
|||
---
|
||||
title: Improved troubleshooting using k8s metadata
|
||||
linkTitle: Kubernetes metadata
|
||||
date: 2022-06-29
|
||||
author: Ruben Vargas
|
||||
spelling:
|
||||
cSpell:ignore k8sattributes k8sattributesprocessor K8sattributes k8sprocessor
|
||||
cSpell:ignore K8sprocessor KUBE
|
||||
---
|
||||
|
||||
Attaching Kubernetes resource metadata to OpenTelemetry traces is useful to
|
||||
identify which resource (such as a pod) is failing or having performance
|
||||
problems. It is also useful for correlating across other signals, for example:
|
||||
you can correlated logs and spans that were generated by the same pod.
|
||||
|
||||
In this article, you'll learn how to configure the OpenTelemetry collector to
|
||||
use the [k8sattributesprocessor][] in different scenarios.
|
||||
|
||||
Details of the OpenTelemetry collector pipeline won't be covered in this post.
|
||||
For those details, refer to the [collector documentation](/docs/collector/).
|
||||
|
||||
## How k8s attributes are attached
|
||||
|
||||
At a high level, k8s attributes are attached to traces as
|
||||
[resources](/docs/concepts/glossary/#resource). This is for two reasons:
|
||||
|
||||
1. K8s attributes fit the definition of what a resource is: an entity for which
|
||||
telemetry is recorded
|
||||
2. It centralizes this metadata, which is relevant for any generated span.
|
||||
|
||||
Let's dive in and see how to do it!
|
||||
|
||||
## Using k8sattributes processor
|
||||
|
||||
This is an OpenTelemetry processor that automatically discovers pod metadata and
|
||||
attaches it to a resource associated with the spans generated by that pod. If
|
||||
the pod belongs to a `Deployment` or a `ReplicaSet`, it will also discover it's
|
||||
attributes.
|
||||
|
||||
Some attributes we can attach to the resource are:
|
||||
|
||||
- Node name `k8s.node.name`
|
||||
- Pod name `k8s.pod.name`
|
||||
- Pod UID `k8s.pod.uid`
|
||||
- Namespace `k8s.namespace.name`
|
||||
- Deployment name, `k8s.deployment.name` if the pod was created by a deployment
|
||||
|
||||
Such attributes adhere to OpenTelemetry semantic conventions. For details, see
|
||||
the [Kubernetes resource semantic conventions][].
|
||||
|
||||
The processor internally maintains a list of pods and an associated attribute,
|
||||
usually the IP address of the pod, and uses this attribute to know which pod
|
||||
generates a certain span.
|
||||
|
||||
|
||||
|
||||
In the figure above you can see how the data flows: The table of pods is fetched
|
||||
using Kubernetes API, while the pod IP is extracted from the connection context
|
||||
between the pod and the collector.
|
||||
|
||||
The `k8sattributesprocessor` can work in different modes depending on how the
|
||||
collector is configured. Let's explore one common scenario, when the collector
|
||||
is deployed as daemonset.
|
||||
|
||||
### Daemonset mode
|
||||
|
||||
Let’s take a look at how we can configure the collector in daemonset mode, also
|
||||
known as an agent mode in the k8sattributes documentation.
|
||||
|
||||
When we deploy the collector in daemonset mode, we have one collector pod per
|
||||
node. We need to configure the collector service account to have permissions to
|
||||
fetch all pod information. In order to do that, we will create a `ClusterRole`
|
||||
with the necessary permissions.
|
||||
|
||||
Here are the minimum permissions required to make the `k8sattributesprocessor`
|
||||
work:
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: otel-collector
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "namespaces"]
|
||||
verbs: ["get", "watch", "list"]
|
||||
```
|
||||
|
||||
Next, deploy the collector in daemonset mode. We recommend that you set a filter
|
||||
to only fetch the pods that belong to the node in which the collector is
|
||||
deployed. This is because if you have a large cluster, you don’t want to
|
||||
maintain a huge list of pods.
|
||||
|
||||
This is the manifest used in this blog to show how the processor works:
|
||||
|
||||
```yaml
|
||||
apiVersion: opentelemetry.io/v1alpha1
|
||||
kind: OpenTelemetryCollector
|
||||
metadata:
|
||||
name: otel-collector-daemonset
|
||||
spec:
|
||||
mode: daemonset
|
||||
image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.47.0
|
||||
serviceAccount: attributes-account
|
||||
env:
|
||||
- name: KUBE_NODE_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: spec.nodeName
|
||||
config: |
|
||||
receivers:
|
||||
jaeger:
|
||||
protocols:
|
||||
grpc:
|
||||
thrift_binary:
|
||||
thrift_compact:
|
||||
thrift_http:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
http:
|
||||
|
||||
processors:
|
||||
k8sattributes:
|
||||
filter:
|
||||
node_from_env_var: KUBE_NODE_NAME
|
||||
exporters:
|
||||
jaeger:
|
||||
endpoint: jaeger-all-in-one-collector:14250
|
||||
tls:
|
||||
insecure: true
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp, jaeger]
|
||||
processors: [k8sattributes]
|
||||
exporters: [jaeger]
|
||||
```
|
||||
|
||||
The main parts to note are that it uses the contrib collector image. The
|
||||
`k8sattributesprocessor` is not part of the OpenTelemetry collector core, but
|
||||
the contrib distribution has it. Other things to notice are the filter mentioned
|
||||
above, and the use of a previously-created specific service account, which
|
||||
contains the permissions to fetch the pod list.
|
||||
|
||||
Next, deploy the manifest and the [vert.x example app][] to generate some
|
||||
traces.
|
||||
|
||||
|
||||
|
||||
As you can see, each span of the trace now has the corresponding pod attributes
|
||||
attached to it.
|
||||
|
||||
You can restrict the configuration above to a certain namespace if you add the
|
||||
namespace on the `k8sattributesprocessor` filter like this:
|
||||
|
||||
```yaml
|
||||
processors:
|
||||
k8sattributes:
|
||||
filter:
|
||||
namespace: my_namespace
|
||||
```
|
||||
|
||||
In this way, you can create a `Role` and don’t need to create a `ClusterRole`,
|
||||
reducing the scope of the collector service account to a single namespace.
|
||||
|
||||
## Using Resource detector processor
|
||||
|
||||
As of [recently][pr#832], the [OpenTelemetry operator][] sets the
|
||||
`OTEL_RESOURCE_ATTRIBUTES` environment variable on the collector container with
|
||||
the k8s pod attributes. This lets you to use the resource detector processor,
|
||||
which attaches the environment variable values to the spans. This only works
|
||||
when the collector is deployed in sidecar mode.
|
||||
|
||||
For example, if you deploy the following manifest:
|
||||
|
||||
```yaml
|
||||
apiVersion: opentelemetry.io/v1alpha1
|
||||
kind: OpenTelemetryCollector
|
||||
metadata:
|
||||
name: sidecar-for-my-app
|
||||
spec:
|
||||
mode: sidecar
|
||||
image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.47.0
|
||||
config: |
|
||||
receivers:
|
||||
jaeger:
|
||||
protocols:
|
||||
grpc:
|
||||
thrift_binary:
|
||||
thrift_compact:
|
||||
thrift_http:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
http:
|
||||
|
||||
processors:
|
||||
resourcedetection:
|
||||
detectors: [env]
|
||||
timeout: 2s
|
||||
override: false
|
||||
|
||||
exporters:
|
||||
jaeger:
|
||||
endpoint: jaeger-all-in-one-collector:14250
|
||||
tls:
|
||||
insecure: true
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp, jaeger]
|
||||
processors: [resourcedetection]
|
||||
exporters: [jaeger]
|
||||
```
|
||||
|
||||
And then deploy the [vert.x app example][], you can see the
|
||||
`OTEL_RESOURCE_ATTRIBUTES` environment variable gets injected with some values
|
||||
in the sidecar container. Some of them use the Kubernetes downward API to get
|
||||
the attribute values.
|
||||
|
||||
Here's an example of the value of the environment variable:
|
||||
|
||||
```yaml
|
||||
- name: OTEL_RESOURCE_ATTRIBUTES
|
||||
value: k8s.deployment.name=dep-vert-x,k8s.deployment.uid=ef3fe26b-a690-4746-9119-d2dbd94b469f,
|
||||
k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=
|
||||
(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.pod.uid=$(OTEL_RESOURCE_ATTRIBUTES_POD_UID),k8s.replicaset
|
||||
name=dep-vert-x-59b6f76585,k8s.replicaset.uid=5127bc38-e298-40e1-95df-f4a777e3176c
|
||||
```
|
||||
|
||||
## Learn more
|
||||
|
||||
This post covers how to configure the OpenTelemetry collector to attach
|
||||
Kubernetes resource metadata as resource attributes to OpenTelemetry traces. The
|
||||
scenarios covered, although basic, illustrate how to add this kind of metadata
|
||||
to traces so that you can incorporate the technique into other more
|
||||
sophisticated scenarios. If you want to learn more about different scenarios or
|
||||
options for configure the processors you can see the [K8sattributes processor
|
||||
documentation][k8sattributesprocessor] where you can find more scenarios like
|
||||
sidecar, or when one collector as an agent report to another collector.
|
||||
|
||||
## References
|
||||
|
||||
- [K8sattributes processor documentation][k8sattributesprocessor]
|
||||
- [K8sattributes processor RBAC](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor#hdr-RBAC)
|
||||
- [OpenTelemetry Kubernetes attributes](/docs/reference/specification/resource/semantic_conventions/k8s)
|
||||
- [Resource detector processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md)
|
||||
|
||||
[pr#832]: https://github.com/open-telemetry/opentelemetry-operator/pull/832
|
||||
[opentelemetry operator]:
|
||||
https://github.com/open-telemetry/opentelemetry-operator
|
||||
[k8sattributesprocessor]:
|
||||
https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
|
||||
[kubernetes resource semantic conventions]:
|
||||
/docs/reference/specification/resource/semantic_conventions/k8s
|
||||
[vert.x example app]: https://github.com/jaegertracing/vertx-create-span
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 154 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
Loading…
Reference in New Issue