4.8 KiB

Raw Blame History

Monitoring, Logging and Tracing Installation

Knative Serving offers two different monitoring setups: Elasticsearch, Kibana, Prometheus and Grafana or Stackdriver, Prometheus and Grafana. You can install only one of these two setups and side-by-side installation of these two are not supported.

Elasticsearch, Kibana, Prometheus & Grafana Setup

If you installed the full Knative release, skip this step and continue to Create Elasticsearch Indices

Install Knative monitoring components from the root of the Serving repository:

kubectl apply --recursive --filename config/monitoring/100-common \
    --filename config/monitoring/150-elasticsearch \
    --filename third_party/config/monitoring/common \
    --filename third_party/config/monitoring/elasticsearch \
    --filename config/monitoring/200-common \
    --filename config/monitoring/200-common/100-istio.yaml

The installation is complete when logging & monitoring components are all reported Running or Completed:

kubectl get pods --namespace monitoring --watch

NAME                                  READY     STATUS    RESTARTS   AGE
elasticsearch-logging-0               1/1       Running   0          2d
elasticsearch-logging-1               1/1       Running   0          2d
fluentd-ds-5kc85                      1/1       Running   0          2d
fluentd-ds-vhrcq                      1/1       Running   0          2d
fluentd-ds-xghk9                      1/1       Running   0          2d
grafana-798cf569ff-v4q74              1/1       Running   0          2d
kibana-logging-7d474fbb45-6qb8x       1/1       Running   0          2d
kube-state-metrics-75bd4f5b8b-8t2h2   4/4       Running   0          2d
node-exporter-cr6bh                   2/2       Running   0          2d
node-exporter-mf6k7                   2/2       Running   0          2d
node-exporter-rhzr7                   2/2       Running   0          2d
prometheus-system-0                   1/1       Running   0          2d
prometheus-system-1                   1/1       Running   0          2d

CTRL+C to exit watch.

Create Elasticsearch Indices

To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will create two indices in Elasticsearch using Logstash for application logs and Zipkin for request traces.

To open the Kibana UI (the visualization tool for Elasticsearch), start a local proxy with the following command:
```
kubectl proxy
```
This command starts a local proxy of Kibana on port 8001. For security reasons, the Kibana UI is exposed only within the cluster.
Navigate to the Kibana UI. It might take a couple of minutes for the proxy to work.
Within the "Configure an index pattern" page, enter logstash-* to Index pattern and select @timestamp from Time Filter field name and click on Create button.

To create the second index, select Create Index Pattern button on top left of the page. Enter zipkin* to Index pattern and select timestamp_millis from Time Filter field name and click on Create button.

Stackdriver, Prometheus & Grafana Setup

If your Knative Serving is not built on a Google Cloud Platform (GCP) based cluster or you want to send logs to another GCP project, you need to build your own Fluentd image and modify the configuration first. See

Install Fluentd image on Knative Serving.
Set up a logging plugin.
Install Knative monitoring components:

kubectl apply --recursive --filename config/monitoring/100-common \
    --filename config/monitoring/150-stackdriver-prod \
    --filename third_party/config/monitoring/common \
    --filename config/monitoring/200-common \
    --filename config/monitoring/200-common/100-istio.yaml

Learn More

Learn more about accessing logs, metrics, and traces:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License.

4.8 KiB Raw Blame History