Attach kubernetes attributes to traces blogpost (#1454)

* Attach kubernetes attributes to traces blogpost Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Run markdown prettifier and made some copyedits * address some comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * address comments about resources vs attributes Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * change to internal url Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * update date, title and linktitle Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * minor changes Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * rephrase from attach spans to attach to resources Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * improvments to paragraphs Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Phillip Carter <pcarter@fastmail.com> * improve conclusions Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Copyedits * Run prettifier * Add link definition * Fix app spelling * address comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * address more comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update content/en/blog/2022/k8sattributes.md Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com> * More comments Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Copyedits - fix use of link def and vert.x app name * change diagram Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * change name of files Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Use today's date Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
2022-06-29 08:17:55 -05:00 · 2022-06-29 08:17:55 -05:00 · 3a417e1cfb
parent ae690a7e5e
commit 3a417e1cfb
3 changed files with 260 additions and 0 deletions
--- a/content/en/blog/2022/k8s-metadata.md
+++ b/content/en/blog/2022/k8s-metadata.md
@ -0,0 +1,260 @@
 ---
 title: Improved troubleshooting using k8s metadata
 linkTitle: Kubernetes metadata
 date: 2022-06-29
 author: Ruben Vargas
 spelling:
  cSpell:ignore k8sattributes k8sattributesprocessor K8sattributes k8sprocessor
  cSpell:ignore K8sprocessor KUBE
 ---
 Attaching Kubernetes resource metadata to OpenTelemetry traces is useful to
 identify which resource (such as a pod) is failing or having performance
 problems. It is also useful for correlating across other signals, for example:
 you can correlated logs and spans that were generated by the same pod.
 In this article, you'll learn how to configure the OpenTelemetry collector to
 use the [k8sattributesprocessor][] in different scenarios.
 Details of the OpenTelemetry collector pipeline won't be covered in this post.
 For those details, refer to the [collector documentation](/docs/collector/).
 ## How k8s attributes are attached
 At a high level, k8s attributes are attached to traces as
 [resources](/docs/concepts/glossary/#resource). This is for two reasons:
 1.  K8s attributes fit the definition of what a resource is: an entity for which
    telemetry is recorded
 2.  It centralizes this metadata, which is relevant for any generated span.
 Let's dive in and see how to do it!
 ## Using k8sattributes processor
 This is an OpenTelemetry processor that automatically discovers pod metadata and
 attaches it to a resource associated with the spans generated by that pod. If
 the pod belongs to a `Deployment` or a `ReplicaSet`, it will also discover it's
 attributes.
 Some attributes we can attach to the resource are:
 - Node name `k8s.node.name`
 - Pod name `k8s.pod.name`
 - Pod UID `k8s.pod.uid`
 - Namespace `k8s.namespace.name`
 - Deployment name, `k8s.deployment.name` if the pod was created by a deployment
 Such attributes adhere to OpenTelemetry semantic conventions. For details, see
 the [Kubernetes resource semantic conventions][].
 The processor internally maintains a list of pods and an associated attribute,
 usually the IP address of the pod, and uses this attribute to know which pod
 generates a certain span.
 ![k8sattributes processor data flow](/img/blog-k8s-metadata/k8sprocessor.png)
 In the figure above you can see how the data flows: The table of pods is fetched
 using Kubernetes API, while the pod IP is extracted from the connection context
 between the pod and the collector.
 The `k8sattributesprocessor` can work in different modes depending on how the
 collector is configured. Let's explore one common scenario, when the collector
 is deployed as daemonset.
 ### Daemonset mode
 Let’s take a look at how we can configure the collector in daemonset mode, also
 known as an agent mode in the k8sattributes documentation.
 When we deploy the collector in daemonset mode, we have one collector pod per
 node. We need to configure the collector service account to have permissions to
 fetch all pod information. In order to do that, we will create a `ClusterRole`
 with the necessary permissions.
 Here are the minimum permissions required to make the `k8sattributesprocessor`
 work:
 ```yaml
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
  name: otel-collector
 rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "watch", "list"]
 ```
 Next, deploy the collector in daemonset mode. We recommend that you set a filter
 to only fetch the pods that belong to the node in which the collector is
 deployed. This is because if you have a large cluster, you don’t want to
 maintain a huge list of pods.
 This is the manifest used in this blog to show how the processor works:
 ```yaml
 apiVersion: opentelemetry.io/v1alpha1
 kind: OpenTelemetryCollector
 metadata:
  name: otel-collector-daemonset
 spec:
  mode: daemonset
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.47.0
  serviceAccount: attributes-account
  env:
    - name: KUBE_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
  config: |
    receivers:
      jaeger:
          protocols:
              grpc:
              thrift_binary:
              thrift_compact:
              thrift_http:
      otlp:
          protocols:
              grpc:
              http:
    processors:
         k8sattributes:
             filter:
                 node_from_env_var: KUBE_NODE_NAME
    exporters:
      jaeger:
        endpoint: jaeger-all-in-one-collector:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp, jaeger]
          processors: [k8sattributes]
          exporters: [jaeger]
 ```
 The main parts to note are that it uses the contrib collector image. The
 `k8sattributesprocessor` is not part of the OpenTelemetry collector core, but
 the contrib distribution has it. Other things to notice are the filter mentioned
 above, and the use of a previously-created specific service account, which
 contains the permissions to fetch the pod list.
 Next, deploy the manifest and the [vert.x example app][] to generate some
 traces.
 ![Jaeger UI showing the span attributes](/img/blog-k8s-metadata/jaeger-k8sattributes.png)
 As you can see, each span of the trace now has the corresponding pod attributes
 attached to it.
 You can restrict the configuration above to a certain namespace if you add the
 namespace on the `k8sattributesprocessor` filter like this:
 ```yaml
 processors:
  k8sattributes:
    filter:
      namespace: my_namespace
 ```
 In this way, you can create a `Role` and don’t need to create a `ClusterRole`,
 reducing the scope of the collector service account to a single namespace.
 ## Using Resource detector processor
 As of [recently][pr#832], the [OpenTelemetry operator][] sets the
 `OTEL_RESOURCE_ATTRIBUTES` environment variable on the collector container with
 the k8s pod attributes. This lets you to use the resource detector processor,
 which attaches the environment variable values to the spans. This only works
 when the collector is deployed in sidecar mode.
 For example, if you deploy the following manifest:
 ```yaml
 apiVersion: opentelemetry.io/v1alpha1
 kind: OpenTelemetryCollector
 metadata:
  name: sidecar-for-my-app
 spec:
  mode: sidecar
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.47.0
  config: |
    receivers:
      jaeger:
        protocols:
          grpc:
          thrift_binary:
          thrift_compact:
          thrift_http:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      resourcedetection:
        detectors: [env]
        timeout: 2s
        override: false
    exporters:
      jaeger:
        endpoint: jaeger-all-in-one-collector:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp, jaeger]
          processors: [resourcedetection]
          exporters: [jaeger]
 ```
 And then deploy the [vert.x app example][], you can see the
 `OTEL_RESOURCE_ATTRIBUTES` environment variable gets injected with some values
 in the sidecar container. Some of them use the Kubernetes downward API to get
 the attribute values.
 Here's an example of the value of the environment variable:
 ```yaml
 - name: OTEL_RESOURCE_ATTRIBUTES
  value: k8s.deployment.name=dep-vert-x,k8s.deployment.uid=ef3fe26b-a690-4746-9119-d2dbd94b469f,
  k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=
  (OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.pod.uid=$(OTEL_RESOURCE_ATTRIBUTES_POD_UID),k8s.replicaset
  name=dep-vert-x-59b6f76585,k8s.replicaset.uid=5127bc38-e298-40e1-95df-f4a777e3176c
 ```
 ## Learn more
 This post covers how to configure the OpenTelemetry collector to attach
 Kubernetes resource metadata as resource attributes to OpenTelemetry traces. The
 scenarios covered, although basic, illustrate how to add this kind of metadata
 to traces so that you can incorporate the technique into other more
 sophisticated scenarios. If you want to learn more about different scenarios or
 options for configure the processors you can see the [K8sattributes processor
 documentation][k8sattributesprocessor] where you can find more scenarios like
 sidecar, or when one collector as an agent report to another collector.
 ## References
 - [K8sattributes processor documentation][k8sattributesprocessor]
 - [K8sattributes processor RBAC](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor#hdr-RBAC)
 - [OpenTelemetry Kubernetes attributes](/docs/reference/specification/resource/semantic_conventions/k8s)
 - [Resource detector processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md)
 [pr#832]: https://github.com/open-telemetry/opentelemetry-operator/pull/832
 [opentelemetry operator]:
  https://github.com/open-telemetry/opentelemetry-operator
 [k8sattributesprocessor]:
  https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
 [kubernetes resource semantic conventions]:
  /docs/reference/specification/resource/semantic_conventions/k8s
 [vert.x example app]: https://github.com/jaegertracing/vertx-create-span
--- a/static/img/blog-k8s-metadata/jaeger-k8sattributes.png
+++ b/static/img/blog-k8s-metadata/jaeger-k8sattributes.png
--- a/static/img/blog-k8s-metadata/k8sprocessor.png
+++ b/static/img/blog-k8s-metadata/k8sprocessor.png