Move Observability docs to Serving and Eventing guides (#4387)

* move observability docs

* Update nav links

* Update collecting-metrics.md

Pretty sure this fixes some weird formatting

Co-authored-by: Omer Bensaadon <obensaadon@vmware.com>

config/nav.yml

@@ -117,12 +117,11 @@ nav:
           - Feature and extension flags: admin/serving/feature-flags.md
           - Configure the Defaults ConfigMap: admin/serving/config-defaults.md
       # Serving - observability
-      # TODO: Create separate files for Serving and Eventing
       - Observability:
-          - Collecting logs: admin/observability/collecting-logs/collecting-logs.md
+        - Collecting logs: serving/observability/logging/collecting-logs.md
-          - Configuring logging: admin/observability/config-logging/logging.md
+        - Configuring logging: serving/observability/logging/config-logging.md
-          - Collecting metrics: admin/observability/collecting-metrics/collecting-metrics.md
+        - Collecting metrics: serving/observability/metrics/collecting-metrics.md
-          - Knative Serving metrics: admin/observability/collecting-metrics/serving-metrics/metrics.md
+        - Knative Serving metrics: serving/observability/metrics/serving-metrics.md
       # Serving - troubleshooting
       - Troubleshooting:
         - Debugging application issues: developer/serving/troubleshooting/debugging-application-issues.md
@@ -200,12 +199,11 @@ nav:
           - Configure Kafka Channel defaults: admin/eventing/kafka-channel-configuration.md
           - Configure event source defaults: admin/eventing/sources-configuration.md
       # Eventing - observability
-      # TODO: Create separate files for Serving and Eventing
       - Observability:
-        - Collecting logs: admin/observability/collecting-logs/collecting-logs.md
+        - Collecting logs: eventing/observability/logging/collecting-logs.md
-        - Configuring logging: admin/observability/config-logging/logging.md
+        - Configuring logging: eventing/observability/logging/config-logging.md
-        - Collecting metrics: admin/observability/collecting-metrics/collecting-metrics.md
+        - Collecting metrics: eventing/observability/metrics/collecting-metrics.md
-        - Knative Eventing metrics: admin/observability/collecting-metrics/eventing-metrics/metrics.md
+        - Knative Eventing metrics: eventing/observability/metrics/eventing-metrics.md
       # Eventing - troubleshooting
       - Troubleshooting:
         - Debugging: eventing/debugging/README.md

config/redirects.yml

@@ -24,10 +24,10 @@ plugins:
       serving/deploying-from-private-registry.md: developer/serving/deploying-from-private-registry.md
       serving/samples/blue-green-deployment.md: developer/serving/traffic-management.md
       serving/samples/traffic-splitting/README.md: developer/serving/traffic-management.md
-      admin/collecting-logs/README.md: admin/observability/collecting-logs/collecting-logs.md
+      admin/collecting-logs/README.md: serving/observability/logging/collecting-logs.md
-      admin/collecting-metrics/README.md: admin/observability/collecting-metrics/collecting-metrics.md
+      admin/collecting-metrics/README.md: serving/observability/metrics/collecting-metrics.md
-      admin/collecting-metrics/eventing-metrics/metrics.md: admin/observability/collecting-metrics/eventing-metrics/metrics.md
+      admin/collecting-metrics/eventing-metrics/metrics.md: eventing/observability/metrics/eventing-metrics.md
-      admin/collecting-metrics/serving-metrics/metrics.md: admin/observability/collecting-metrics/serving-metrics/metrics.md
+      admin/collecting-metrics/serving-metrics/metrics.md: serving/observability/metrics/serving-metrics.md
       admin/install/install-eventing-with-yaml.md: admin/install/eventing/install-eventing-with-yaml.md
       admin/install/install-serving-with-yaml.md: admin/install/serving/install-serving-with-yaml.md
       client/connecting-kn-to-your-cluster/index.md: client/README.md
@@ -38,7 +38,7 @@ plugins:
       eventing/broker/kafka-broker/kafka-configmap.md: admin/eventing/kafka-channel-configuration.md
       eventing/channels/default-channels/index.md: eventing/channels/README.md
       eventing/event-delivery.md: developer/eventing/event-delivery.md
-      eventing/metrics.md: admin/observability/collecting-metrics/eventing-metrics/metrics.md
+      eventing/metrics.md: eventing/observability/metrics/eventing-metrics.md
       eventing/parallel.md: eventing/flows/parallel.md
       eventing/samples/apache-camel-source/index.md: developer/eventing/sources/apache-camel-source/README.md
       eventing/samples/gcp-pubsub-source/README.md: eventing/samples/cloud-pubsub-source/README.md
@@ -60,8 +60,8 @@ plugins:
       help/README.md: help/contributor/README.md
       install/README.md: admin/install/README.md
       install/check-install-version.md: check-install-version.md
-      install/collecting-logs/index.md: admin/observability/collecting-logs/collecting-logs.md
+      install/collecting-logs/index.md: serving/observability/logging/collecting-logs.md
-      install/collecting-metrics/index.md: admin/observability/collecting-metrics/collecting-metrics.md
+      install/collecting-metrics/index.md: serving/observability/metrics/collecting-metrics.md
       install/getting-started-knative-app/index.md: getting-started/README.md
       serving/getting-started-knative-app.md: getting-started/README.md
       install/install-eventing-with-yaml.md: admin/install/eventing/install-eventing-with-yaml.md
@@ -84,7 +84,7 @@ plugins:
       serving/configuring-autoscaling/index.md: serving/autoscaling/README.md
       serving/creating-domain-mappings.md: developer/serving/services/custom-domains.md
       serving/debugging-application-issues.md: developer/serving/troubleshooting/debugging-application-issues.md
-      serving/metrics.md: admin/observability/collecting-metrics/serving-metrics/metrics.md
+      serving/metrics.md: serving/observability/metrics/serving-metrics.md
       serving/samples/autoscale-go/index.md: serving/autoscaling/autoscale-go/README.md
       serving/services/README.md: developer/serving/services/README.md
       serving/services/creating-services.md: developer/serving/services/creating-services.md

docs/eventing/observability/logging/collecting-logs.md

@@ -0,0 +1 @@
+--8<-- "collecting-logs.md"

docs/eventing/observability/logging/config-logging.md

@@ -0,0 +1 @@
+--8<-- "configuring-logs.md"

docs/eventing/observability/metrics/collecting-metrics.md

@@ -0,0 +1 @@
+--8<-- "collecting-metrics.md"

docs/eventing/observability/metrics/eventing-metrics.md

@@ -38,7 +38,7 @@ By aggregating the metrics over the http code, events can be separated into two
 !!! note
     A number of metrics eg. controller, Go runtime and others are omitted here as they are common
     across most components. For more about these metrics check the
-    [Serving metrics API section](../serving-metrics/metrics.md).
+    [Serving metrics API section](../../../serving/observability/metrics/serving-metrics.md).
 
 ## Eventing sources
 

docs/serving/README.md

@@ -71,7 +71,7 @@ For more information on the resources and their interactions, see the [Resource
 
 ## Observability
 
-- [Serving Metrics API](../admin/observability/collecting-metrics/serving-metrics/metrics.md)
+- [Serving Metrics API](observability/metrics/serving-metrics.md)
 
 ## Known Issues
 

docs/serving/observability/logging/collecting-logs.md

@@ -0,0 +1 @@
+--8<-- "collecting-logs.md"

docs/serving/observability/logging/config-logging.md

@@ -0,0 +1 @@
+--8<-- "configuring-logs.md"

docs/serving/observability/logging/fluent-bit-collector.yaml

@@ -0,0 +1,167 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: logging
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: log-collector-config
+  namespace: logging
+  labels:
+    k8s-app: log-collector
+data:
+  # Configuration files: server, input, filters and output
+  # ======================================================
+  fluent-bit.conf: |
+    [SERVICE]
+        Flush         1
+        Log_Level     info
+        Daemon        off
+        HTTP_Server   On
+        HTTP_Listen   0.0.0.0
+        HTTP_Port     2020
+
+    @INCLUDE input-forward.conf
+    @INCLUDE filter-simplify.conf
+    @INCLUDE output-files.conf
+
+  input-forward.conf: |
+    [INPUT]
+        Name              forward
+        Port              ${FLUENT_PORT}
+
+  # This filter simplifies the fluentbit "tag" which is used to choose the
+  # output filename. First match Knative services, then match pods with an "app"
+  # label, then use the pod name.
+  filter-simplify.conf: |
+    [FILTER]
+      Name      rewrite_tag
+      Match     kube.*
+      Rule      $kubernetes['labels']['serving.knative.dev/configuration'] ^(.+)$ knative.$kubernetes['namespace_name'].$1.log false
+
+    [FILTER]
+      Name      rewrite_tag
+      Match     kube.*
+      Rule      $kubernetes['labels']['app'] ^(.+)$ app.$kubernetes['namespace_name'].$1.log false
+
+    [FILTER]
+      Name      rewrite_tag
+      Match     kube.*
+      Rule      $kubernetes['pod_name'] ^(.+)$ pod.$kubernetes['namespace_name'].$1.log false
+
+  output-files.conf: |
+    [OUTPUT]
+        Name            file
+        Path            /logs
+        Format          plain
+        Match           *
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: log-nginx-config
+  namespace: logging
+  labels:
+    k8s-app: log-collector
+data:
+  nginx.conf: |
+    events {
+    }
+
+    http {
+      include mime.types;
+      default_type application/octet-stream;
+      sendfile on;
+
+      server {
+        listen 8080;
+        server_name localhost;
+
+        location / {
+          root html;
+          autoindex on;
+        }
+      }
+    }
+  mime.types: |
+    types {
+      text/html        html htm shtml;
+      text/plain       txt log;
+    }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: log-collector
+  namespace: logging
+  labels:
+    k8s-apps: log-collector
+spec:
+  ports:
+  - port: 24224
+    name: forward
+  - port: 80
+    name: viewer
+    targetPort: viewer
+  selector:
+    k8s-apps: log-collector
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: log-collector
+  namespace: logging
+  labels:
+    k8s-apps: log-collector
+spec:
+  selector:
+    matchLabels:
+      k8s-apps: log-collector
+  serviceName: log-collector
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        k8s-apps: log-collector
+    spec:
+      terminationGracePeriodSeconds: 10
+      containers:
+      - name: fluent-bit
+        image: fluent/fluent-bit:1.5
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 24224
+        volumeMounts:
+        - name: logs
+          mountPath: /logs
+        - name: fluentd-config
+          mountPath: /fluent-bit/etc/
+        env:
+        - name: FLUENT_PORT
+          value: "24224"
+      - name: nginx
+        image: k8s.gcr.io/nginx-slim:0.8
+        ports:
+        - containerPort: 8080
+          name: viewer
+        volumeMounts:
+        - name: logs
+          mountPath: /usr/share/nginx/html
+        - name: nginx-config
+          mountPath: /etc/nginx
+      volumes:
+        - name: fluentd-config
+          configMap:
+            name: log-collector-config
+        - name: nginx-config
+          configMap:
+            name: log-nginx-config
+  volumeClaimTemplates:
+  - metadata:
+      name: logs
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      resources:
+        requests:
+          storage: 40Gi

docs/serving/observability/logging/fluent-bit-configmap.yaml

@@ -0,0 +1,106 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: fluent-bit-config
+  namespace: metrics
+  labels:
+    k8s-app: fluent-bit
+data:
+  # Configuration files: server, input, filters and output
+  # ======================================================
+  fluent-bit.conf: |
+    [SERVICE]
+        Flush         1
+        Log_Level     info
+        Daemon        off
+        Parsers_File  parsers.conf
+        HTTP_Server   On
+        HTTP_Listen   0.0.0.0
+        HTTP_Port     2020
+
+    @INCLUDE input-kubernetes.conf
+    @INCLUDE filter-kubernetes.conf
+    @INCLUDE output-forward.conf
+
+  input-kubernetes.conf: |
+    [INPUT]
+        Name              tail
+        Tag               kube.*
+        Path              /var/log/containers/*.log
+        Parser            docker
+        DB                /var/log/flb_kube.db
+        Mem_Buf_Limit     5MB
+        Skip_Long_Lines   On
+        Refresh_Interval  10
+
+  filter-kubernetes.conf: |
+    [FILTER]
+        Name                kubernetes
+        Match               kube.*
+        Kube_URL            https://kubernetes.default.svc:443
+        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
+        Kube_Tag_Prefix     kube.var.log.containers.
+        Merge_Log           On
+        Merge_Log_Key       log_processed
+        K8S-Logging.Parser  On
+        K8S-Logging.Exclude Off
+
+  output-null.conf: |
+    [OUTPUT]
+        Name            null
+        Match           *
+
+  output-forward.conf: |
+    [OUTPUT]
+        Name            forward
+        Host            log-collector.logging
+        Port            24224
+        Require_ack_response  True
+
+  parsers.conf: |
+    [PARSER]
+        Name   apache
+        Format regex
+        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
+        Time_Key time
+        Time_Format %d/%b/%Y:%H:%M:%S %z
+
+    [PARSER]
+        Name   apache2
+        Format regex
+        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
+        Time_Key time
+        Time_Format %d/%b/%Y:%H:%M:%S %z
+
+    [PARSER]
+        Name   apache_error
+        Format regex
+        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
+
+    [PARSER]
+        Name   nginx
+        Format regex
+        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
+        Time_Key time
+        Time_Format %d/%b/%Y:%H:%M:%S %z
+
+    [PARSER]
+        Name   json
+        Format json
+        Time_Key time
+        Time_Format %d/%b/%Y:%H:%M:%S %z
+
+    [PARSER]
+        Name        docker
+        Format      json
+        Time_Key    time
+        Time_Format %Y-%m-%dT%H:%M:%S.%L
+        Time_Keep   On
+
+    [PARSER]
+        Name        syslog
+        Format      regex
+        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
+        Time_Key    time
+        Time_Format %b %d %H:%M:%S

docs/serving/observability/metrics/collecting-metrics.md

@@ -0,0 +1 @@
+--8<-- "collecting-metrics.md"

docs/serving/observability/metrics/collector.yaml

@@ -0,0 +1,96 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: otel-collector-config
+  namespace: metrics
+data:
+  collector.yaml: |
+    receivers:
+      opencensus:
+        endpoint: "0.0.0.0:55678"
+
+    exporters:
+      logging:
+      prometheus:
+        endpoint: "0.0.0.0:8889"
+    extensions:
+        health_check:
+        pprof:
+        zpages:
+    service:
+      extensions: [health_check, pprof, zpages]
+      pipelines:
+        metrics:
+          receivers: [opencensus]
+          processors: []
+          exporters: [prometheus]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: otel-collector
+  namespace: metrics
+  labels:
+    app: otel-collector
+spec:
+  selector:
+    matchLabels:
+      app: otel-collector
+  replicas: 1  # This can be increased for a larger system.
+  template:
+    metadata:
+      labels:
+        app: otel-collector
+    spec:
+      containers:
+      - name: collector
+        args:
+        - --config=/conf/collector.yaml
+        image: otel/opentelemetry-collector:latest
+        resources:
+          requests:  # Note: these are suitable for a small instance, but may need to be increased for a large instance.
+            memory: 100Mi
+            cpu: 50m
+        ports:
+        - name: otel
+          containerPort: 55678
+        - name: prom-export
+          containerPort: 8889
+        - name: zpages  # A /debug page
+          containerPort: 55679
+        volumeMounts:
+          - mountPath: /conf
+            name: config
+      volumes:
+      - name: config
+        configMap:
+          name: otel-collector-config
+          items:
+            - key: collector.yaml
+              path: collector.yaml
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: otel-collector
+  namespace: metrics
+spec:
+  selector:
+    app: "otel-collector"
+  ports:
+  - port: 55678
+    name: otel
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: otel-export
+  namespace: metrics
+  labels:
+    app: otel-export
+spec:
+  selector:
+    app: otel-collector
+  ports:
+  - port: 8889
+    name: prom-export

docs/snippets/collecting-logs.md

@@ -3,7 +3,7 @@
 You can use [Fluent Bit](https://docs.fluentbit.io/), a log processor and forwarder, to collect
 Kubernetes logs in a central directory.
 This is not required to run Knative, but can be helpful with
-[Knative Serving](../../../serving/README.md), which automatically deletes pods and associated logs when they are no longer needed.
+[Knative Serving](/docs/serving/), which automatically deletes pods and associated logs when they are no longer needed.
 
 Fluent Bit supports exporting to a number of other log providers. If you already have an existing log provider, for example, Splunk, Datadog, ElasticSearch, or Stackdriver, you can follow the [FluentBit documentation](https://docs.fluentbit.io/manual/pipeline/outputs) to configure log forwarders.
 

docs/snippets/collecting-metrics.md

@@ -1,6 +1,7 @@
 # Collecting Metrics in Knative
 
 Knative supports different popular tools for collecting metrics:
+
 - [Prometheus](https://prometheus.io/)
 - [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)