mirror of https://github.com/knative/docs.git
Removing references to old monitoring stack (#2946)
* Removing references to old monitoring stack * removed mentions, deleted files related * remove telemetry go * removed mention from landing pages * re-add traces file * add back traces file * readd lines
This commit is contained in:
Ashleigh Brennan
GitHub
parent
ec3b20653d
commit
15153b3e46
|
|
@ -39,15 +39,6 @@ Follow the links below to learn more about Knative.
|
||||||
- [All samples for serving](./serving/samples/)
|
- [All samples for serving](./serving/samples/)
|
||||||
- [All samples for eventing](./eventing/samples/)
|
- [All samples for eventing](./eventing/samples/)
|
||||||
|
|
||||||
### Logging and metrics
|
|
||||||
|
|
||||||
- [Installing logging, metrics and traces](./serving/installing-logging-metrics-traces.md)
|
|
||||||
- [Accessing logs](./serving/accessing-logs.md)
|
|
||||||
- [Accessing metrics](./serving/accessing-metrics.md)
|
|
||||||
- [Accessing traces](./serving/accessing-traces.md)
|
|
||||||
- [Setting up a logging plugin](./serving/setting-up-a-logging-plugin.md)
|
|
||||||
|
|
||||||
### Debugging
|
### Debugging
|
||||||
|
|
||||||
- [Debugging application issues](./serving/debugging-application-issues.md)
|
- [Debugging application issues](./serving/debugging-application-issues.md)
|
||||||
- [Debugging performance issues](./serving/debugging-performance-issues.md)
|
|
||||||
|
|
|
||||||
|
|
@ -13,16 +13,6 @@ visualize and trace your requests.
|
||||||
|
|
||||||
You must have a Knative cluster running with the Eventing component installed. [Learn more](../install/README.md)
|
You must have a Knative cluster running with the Eventing component installed. [Learn more](../install/README.md)
|
||||||
|
|
||||||
## Installing observability plugins
|
|
||||||
|
|
||||||
Knative Eventing uses the same tracing plugin as Knative Serving. See the
|
|
||||||
[Tracing installation instructions](./../serving/installing-logging-metrics-traces.md#end-to-end-request-tracing)
|
|
||||||
in the Knative Serving section for details. Note that you do not need to install the
|
|
||||||
Knative Serving component itself.
|
|
||||||
|
|
||||||
To enable request tracing in Knative Eventing,
|
|
||||||
you must install Elasticsearch and either the Zipkin or Jaeger plugins.
|
|
||||||
|
|
||||||
## Configuring tracing
|
## Configuring tracing
|
||||||
|
|
||||||
With the exception of importers, the Knative Eventing tracing is configured through the
|
With the exception of importers, the Knative Eventing tracing is configured through the
|
||||||
|
|
@ -30,7 +20,7 @@ With the exception of importers, the Knative Eventing tracing is configured thro
|
||||||
|
|
||||||
Most importers do _not_ use the ConfigMap and instead, use a static 1% sampling rate.
|
Most importers do _not_ use the ConfigMap and instead, use a static 1% sampling rate.
|
||||||
|
|
||||||
You can use the `config-tracing` ConfigMap to configure the following Eventing subcomponents:
|
You can use the `config-tracing` ConfigMap to configure the following Eventing components:
|
||||||
- Brokers
|
- Brokers
|
||||||
- Triggers
|
- Triggers
|
||||||
- InMemoryChannel
|
- InMemoryChannel
|
||||||
|
|
|
||||||
|
|
@ -37,8 +37,7 @@ respective repositories in GitHub.
|
||||||
|
|
||||||
Before upgrading, view the status of the pods for the namespaces you plan on
|
Before upgrading, view the status of the pods for the namespaces you plan on
|
||||||
upgrading. This allows you to compare the before and after state of your
|
upgrading. This allows you to compare the before and after state of your
|
||||||
namespace. For example, if you are upgrading Knative Serving, Eventing, and the
|
namespace. For example, if you are upgrading Knative Serving and Eventing, enter the following commands to see the current state of
|
||||||
monitoring plug-in, enter the following commands to see the current state of
|
|
||||||
each namespace:
|
each namespace:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -49,16 +48,10 @@ kubectl get pods --namespace knative-serving
|
||||||
kubectl get pods --namespace knative-eventing
|
kubectl get pods --namespace knative-eventing
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl get pods --namespace knative-monitoring
|
|
||||||
```
|
|
||||||
|
|
||||||
### Upgrading plug-ins
|
### Upgrading plug-ins
|
||||||
|
|
||||||
If you have a plug-in installed, make sure to upgrade it at the same time as
|
If you have a plug-in installed, make sure to upgrade it at the same time as
|
||||||
you upgrade your Knative components. For example, if you have the
|
you upgrade your Knative components.
|
||||||
monitoring plug-in installed, upgrade it alongside Knative Serving and Eventing.
|
|
||||||
|
|
||||||
|
|
||||||
### Run pre-install tools before upgrade
|
### Run pre-install tools before upgrade
|
||||||
|
|
||||||
|
|
@ -90,13 +83,12 @@ kubectl create --filename {{< artifact repo="serving" file="serving-post-install
|
||||||
To upgrade, apply the `.yaml` files for the subsequent minor versions of all
|
To upgrade, apply the `.yaml` files for the subsequent minor versions of all
|
||||||
your installed Knative components and features, remembering to only
|
your installed Knative components and features, remembering to only
|
||||||
upgrade by one minor version at a time. For a cluster running v0.15.2 of the
|
upgrade by one minor version at a time. For a cluster running v0.15.2 of the
|
||||||
Knative Serving and Eventing components and the monitoring plug-in, the
|
Knative Serving and Eventing components, the
|
||||||
following command upgrades the installation to v0.16.0:
|
following command upgrades the installation to v0.16.0:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.16.0/serving-core.yaml \
|
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.16.0/serving-core.yaml \
|
||||||
--filename https://github.com/knative/eventing/releases/download/v0.16.0/eventing.yaml \
|
--filename https://github.com/knative/eventing/releases/download/v0.16.0/eventing.yaml \
|
||||||
--filename https://github.com/knative/serving/releases/download/v0.16.0/monitoring.yaml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Run post-install tools after the upgrade
|
### Run post-install tools after the upgrade
|
||||||
|
|
@ -111,11 +103,9 @@ kubectl apply --filename {{< artifact repo="eventing" file="eventing-post-instal
|
||||||
|
|
||||||
## Verifying the upgrade
|
## Verifying the upgrade
|
||||||
|
|
||||||
To confirm that your components and plugins have successfully upgraded, view the
|
To confirm that your components and plugins have successfully upgraded, view the status of their pods in the relevant namespaces.
|
||||||
status of their pods in the relevant namespaces. All pods will restart during
|
All pods will restart during the upgrade and their age will reset.
|
||||||
the upgrade and their age will reset. If you upgraded Knative Serving, Eventing,
|
If you upgraded Knative Serving and Eventing, enter the following commands to get information about the pods for each namespace:
|
||||||
and the monitoring plug-in, enter the following commands to get information
|
|
||||||
about the pods for each namespace:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl get pods --namespace knative-serving
|
kubectl get pods --namespace knative-serving
|
||||||
|
|
@ -125,10 +115,6 @@ kubectl get pods --namespace knative-serving
|
||||||
kubectl get pods --namespace knative-eventing
|
kubectl get pods --namespace knative-eventing
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl get pods --namespace knative-monitoring
|
|
||||||
```
|
|
||||||
|
|
||||||
These commands return something similar to:
|
These commands return something similar to:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -150,27 +136,7 @@ imc-dispatcher-f59b7c57-q9xcl 1/1 Running 0 80s
|
||||||
sources-controller-8596684d7b-jxkmd 1/1 Running 0 83s
|
sources-controller-8596684d7b-jxkmd 1/1 Running 0 83s
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
If the age of all your pods has been reset and all pods are up and running, the upgrade was completed successfully.
|
||||||
NAME READY STATUS RESTARTS AGE
|
You might notice a status of `Terminating` for the old pods as they are cleaned up.
|
||||||
NAME READY STATUS RESTARTS AGE
|
|
||||||
elasticsearch-logging-0 1/1 Running 0 117s
|
|
||||||
elasticsearch-logging-1 1/1 Running 0 83s
|
|
||||||
fluentd-ds-dqzr7 1/1 Running 0 116s
|
|
||||||
fluentd-ds-dspjc 1/1 Running 0 116s
|
|
||||||
fluentd-ds-x9xg7 1/1 Running 0 116s
|
|
||||||
grafana-59568f8f48-bz2ll 1/1 Running 0 111s
|
|
||||||
kibana-logging-b5d75f556-pwzkg 1/1 Running 0 116s
|
|
||||||
kube-state-metrics-5cb5c6986b-qp6pw 1/1 Running 0 116s
|
|
||||||
node-exporter-bgtsb 2/2 Running 0 112s
|
|
||||||
node-exporter-cqrqv 2/2 Running 0 112s
|
|
||||||
node-exporter-xwv7f 2/2 Running 0 112s
|
|
||||||
prometheus-system-0 1/1 Running 0 110s
|
|
||||||
prometheus-system-1 1/1 Running 0 110s
|
|
||||||
```
|
|
||||||
|
|
||||||
If the age of all your pods has been reset and all pods are up and running, the
|
If necessary, repeat the upgrade process until you reach your desired minor version number.
|
||||||
upgrade was completed successfully. You might notice a status of "Terminating"
|
|
||||||
for the old pods as they are cleaned up.
|
|
||||||
|
|
||||||
If necessary, repeat the upgrade process until you reach your desired minor
|
|
||||||
version number.
|
|
||||||
|
|
|
||||||
|
|
@ -59,18 +59,9 @@ in the Knative Serving repository.
|
||||||
|
|
||||||
- [Knative Serving code samples](./samples/README.md)
|
- [Knative Serving code samples](./samples/README.md)
|
||||||
|
|
||||||
## Setting up Logging and Metrics
|
|
||||||
|
|
||||||
- [Installing Logging, Metrics and Traces](./installing-logging-metrics-traces.md)
|
|
||||||
- [Accessing Logs](./accessing-logs.md)
|
|
||||||
- [Accessing Metrics](./accessing-metrics.md)
|
|
||||||
- [Accessing Traces](./accessing-traces.md)
|
|
||||||
- [Setting up a logging plugin](./setting-up-a-logging-plugin.md)
|
|
||||||
|
|
||||||
## Debugging Knative Serving issues
|
## Debugging Knative Serving issues
|
||||||
|
|
||||||
- [Debugging Application Issues](./debugging-application-issues.md)
|
- [Debugging Application Issues](./debugging-application-issues.md)
|
||||||
- [Debugging Performance Issues](./debugging-performance-issues.md)
|
|
||||||
|
|
||||||
## Configuration and Networking
|
## Configuration and Networking
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,120 +0,0 @@
|
||||||
---
|
|
||||||
title: "Accessing logs"
|
|
||||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
|
||||||
weight: 5
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
If you have not yet installed the logging and monitoring components, go through
|
|
||||||
the [installation instructions](./installing-logging-metrics-traces.md) to set
|
|
||||||
up the necessary components first.
|
|
||||||
|
|
||||||
## Kibana and Elasticsearch
|
|
||||||
|
|
||||||
- To open the Kibana UI (the visualization tool for
|
|
||||||
[Elasticsearch](https://info.elastic.co)), start a local proxy with the
|
|
||||||
following command:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl proxy
|
|
||||||
```
|
|
||||||
|
|
||||||
This command starts a local proxy of Kibana on port 8001. For security
|
|
||||||
reasons, the Kibana UI is exposed only within the cluster.
|
|
||||||
|
|
||||||
- Navigate to the
|
|
||||||
[Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana).
|
|
||||||
_It might take a couple of minutes for the proxy to work_.
|
|
||||||
|
|
||||||
The Discover tab of the Kibana UI looks like this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
You can change the time frame of logs Kibana displays in the upper right
|
|
||||||
corner of the screen. The main search bar is across the top of the Discover
|
|
||||||
page.
|
|
||||||
|
|
||||||
- As more logs are ingested, new fields will be discovered. To have them
|
|
||||||
indexed, go to "Management" > "Index Patterns" > Refresh button (on top
|
|
||||||
right) > "Refresh fields".
|
|
||||||
|
|
||||||
<!-- TODO: create a video walkthrough of the Kibana UI -->
|
|
||||||
|
|
||||||
### Accessing stdout/stderr logs
|
|
||||||
|
|
||||||
To find the logs sent to `stdout/stderr` from your application in the Kibana UI:
|
|
||||||
|
|
||||||
1. Click `Discover` on the left side bar.
|
|
||||||
1. Choose `logstash-*` index pattern on the left top.
|
|
||||||
1. Input `tag: kubernetes*` in the top search bar then search.
|
|
||||||
|
|
||||||
### Accessing request logs
|
|
||||||
|
|
||||||
To access the request logs (if enabled), enter the following search in Kibana:
|
|
||||||
|
|
||||||
```text
|
|
||||||
_exists_:"httpRequest.requestUrl"
|
|
||||||
```
|
|
||||||
|
|
||||||
Request logs contain customized details about requests served by the revision.
|
|
||||||
Below is a sample request log:
|
|
||||||
|
|
||||||
```text
|
|
||||||
@timestamp July 10th 2018, 10:09:28.000
|
|
||||||
kubernetes.labels.serving_knative_dev/configuration helloworld-go
|
|
||||||
kubernetes.labels.serving_knative_dev/revision helloworld-go-6vf4x
|
|
||||||
kubernetes.labels.serving_knative_dev/service helloworld-go
|
|
||||||
httpRequest.protocol HTTP/1.1
|
|
||||||
httpRequest.referer
|
|
||||||
httpRequest.remoteIp 10.32.0.2:46866
|
|
||||||
httpRequest.requestMethod GET
|
|
||||||
httpRequest.requestSize 0
|
|
||||||
httpRequest.requestUrl /
|
|
||||||
httpRequest.responseSize 20
|
|
||||||
httpRequest.serverIp 10.32.1.36
|
|
||||||
httpRequest.status 200
|
|
||||||
httpRequest.userAgent curl/7.60.0
|
|
||||||
traceId 0def9abf835ad90e9d824f7a492e2dcb
|
|
||||||
```
|
|
||||||
|
|
||||||
### Accessing configuration and revision logs
|
|
||||||
|
|
||||||
To access the logs for a configuration:
|
|
||||||
|
|
||||||
- Find the configuration's name with the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get configurations
|
|
||||||
```
|
|
||||||
|
|
||||||
- Replace `<CONFIGURATION_NAME>` and enter the following search query in Kibana:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubernetes.labels.serving_knative_dev\/configuration: <CONFIGURATION_NAME>
|
|
||||||
```
|
|
||||||
|
|
||||||
To access logs for a revision:
|
|
||||||
|
|
||||||
- Find the revision's name with the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get revisions
|
|
||||||
```
|
|
||||||
|
|
||||||
- Replace `<REVISION_NAME>` and enter the following search query in Kibana:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubernetes.labels.serving_knative_dev\/revision: <REVISION_NAME>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Accessing end to end request traces
|
|
||||||
|
|
||||||
See [Accessing Traces](./accessing-traces.md) page for details.
|
|
||||||
|
|
||||||
## Stackdriver
|
|
||||||
|
|
||||||
Go to the
|
|
||||||
[GCP Console logging page](https://console.cloud.google.com/logs/viewer) for
|
|
||||||
your GCP project, which stores your logs via Stackdriver.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1,82 +0,0 @@
|
||||||
---
|
|
||||||
title: "Accessing metrics"
|
|
||||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
|
||||||
weight: 10
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
You access metrics through the [Grafana](https://grafana.com/) UI. Grafana is
|
|
||||||
the visualization tool for [Prometheus](https://prometheus.io/).
|
|
||||||
|
|
||||||
## Grafana
|
|
||||||
|
|
||||||
1. To open Grafana, enter the following command:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl port-forward --namespace knative-monitoring \
|
|
||||||
$(kubectl get pods --namespace knative-monitoring \
|
|
||||||
--selector=app=grafana --output=jsonpath="{.items..metadata.name}") \
|
|
||||||
3000
|
|
||||||
```
|
|
||||||
|
|
||||||
- This starts a local proxy of Grafana on port 3000. For security reasons, the
|
|
||||||
Grafana UI is exposed only within the cluster.
|
|
||||||
|
|
||||||
2. Navigate to the Grafana UI at [http://localhost:3000](http://localhost:3000).
|
|
||||||
|
|
||||||
3. Select the **Home** button on the top of the page to see the list of
|
|
||||||
pre-installed dashboards (screenshot below):
|
|
||||||
|
|
||||||
|
|
||||||
The following dashboards are pre-installed with Knative Serving:
|
|
||||||
|
|
||||||
- **Revision HTTP Requests:** HTTP request count, latency, and size metrics per
|
|
||||||
revision and per configuration
|
|
||||||
- **Nodes:** CPU, memory, network, and disk metrics at node level
|
|
||||||
- **Pods:** CPU, memory, and network metrics at pod level
|
|
||||||
- **Deployment:** CPU, memory, and network metrics aggregated at deployment
|
|
||||||
level
|
|
||||||
- **Istio, Mixer and Pilot:** Detailed Istio mesh, Mixer, and Pilot metrics
|
|
||||||
- **Kubernetes:** Dashboards giving insights into cluster health, deployments,
|
|
||||||
and capacity usage
|
|
||||||
|
|
||||||
4. Set up an administrator account to modify or add dashboards by signing in
|
|
||||||
with username: `admin` and password: `admin`.
|
|
||||||
|
|
||||||
- Before you expose the Grafana UI outside the cluster, make sure to change the
|
|
||||||
password. You will be prompted to set a password on first login, and it can
|
|
||||||
later be changed at
|
|
||||||
[http://localhost:3000/org/users](http://localhost:3000/org/users).
|
|
||||||
|
|
||||||
## Prometheus
|
|
||||||
|
|
||||||
Although Grafana provides a richer set of graphing functionality, Prometheus can
|
|
||||||
still be useful for debugging.
|
|
||||||
|
|
||||||
1. To open Prometheus, enter the following command:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl port-forward -n knative-monitoring \
|
|
||||||
$(kubectl get pods -n knative-monitoring \
|
|
||||||
--selector=app=prometheus --output=jsonpath="{.items[0].metadata.name}") \
|
|
||||||
9090
|
|
||||||
```
|
|
||||||
|
|
||||||
- This starts a local proxy of Prometheus on port 9090. For security reasons,
|
|
||||||
the Prometheus UI is exposed only within the cluster.
|
|
||||||
|
|
||||||
2. Navigate to the Prometheus UI at
|
|
||||||
[http://localhost:9090](http://localhost:9090)
|
|
||||||
|
|
||||||
### Metrics Troubleshooting
|
|
||||||
|
|
||||||
You can use the Prometheus web UI to troubleshoot publishing and service
|
|
||||||
discovery issues for metrics.
|
|
||||||
|
|
||||||
- To see the targets that are being scraped, go to Status -> Targets
|
|
||||||
- To see what Prometheus service discovery is picking up vs. dropping, go to
|
|
||||||
Status -> Service Discovery
|
|
||||||
- To see a specific metric you can search for in the in search box on the top of
|
|
||||||
the page
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -9,10 +9,6 @@ Depending on the request tracing tool that you have installed on your Knative
|
||||||
Serving cluster, see the corresponding section for details about how to
|
Serving cluster, see the corresponding section for details about how to
|
||||||
visualize and trace your requests.
|
visualize and trace your requests.
|
||||||
|
|
||||||
If you have not yet installed the logging and monitoring components, go through
|
|
||||||
the [installation instructions](./installing-logging-metrics-traces.md) to set
|
|
||||||
up the necessary components.
|
|
||||||
|
|
||||||
## Configuring Traces
|
## Configuring Traces
|
||||||
|
|
||||||
You can update the configuration file for tracing in [config-tracing.yaml](https://github.com/knative/serving/blob/master/config/config-tracing.yaml).
|
You can update the configuration file for tracing in [config-tracing.yaml](https://github.com/knative/serving/blob/master/config/config-tracing.yaml).
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,6 @@ A demonstration of the autoscaling capabilities of a Knative Serving Revision.
|
||||||
|
|
||||||
1. A Kubernetes cluster with [Knative Serving](../../../install/README.md)
|
1. A Kubernetes cluster with [Knative Serving](../../../install/README.md)
|
||||||
installed.
|
installed.
|
||||||
1. A [metrics installation](../../installing-logging-metrics-traces.md) for
|
|
||||||
viewing scaling graphs (optional).
|
|
||||||
1. The `hey` load generator installed (`go get -u github.com/rakyll/hey`).
|
1. The `hey` load generator installed (`go get -u github.com/rakyll/hey`).
|
||||||
1. Clone this repository, and move into the sample directory:
|
1. Clone this repository, and move into the sample directory:
|
||||||
|
|
||||||
|
|
@ -207,18 +205,6 @@ autoscaler classes built into Knative:
|
||||||
View the [Kubecon Demo](https://youtu.be/OPSIPr-Cybs) of Knative autoscaler
|
View the [Kubecon Demo](https://youtu.be/OPSIPr-Cybs) of Knative autoscaler
|
||||||
customization (32 minutes).
|
customization (32 minutes).
|
||||||
|
|
||||||
### Dashboards
|
|
||||||
|
|
||||||
View the Knative Serving Scaling and Request dashboards (if configured).
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl port-forward --namespace knative-monitoring $(kubectl get pods --namespace knative-monitoring --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Other Experiments
|
### Other Experiments
|
||||||
|
|
||||||
1. Send 60 seconds of traffic maintaining 100 concurrent requests.
|
1. Send 60 seconds of traffic maintaining 100 concurrent requests.
|
||||||
|
|
|
||||||
|
|
@ -27,11 +27,6 @@ for: "STDIN": Internal error occurred: admission webhook "webhook.knative.dev" d
|
||||||
ERROR: Non-zero return code '1' from command: Process exited with status 1
|
ERROR: Non-zero return code '1' from command: Process exited with status 1
|
||||||
```
|
```
|
||||||
|
|
||||||
## Check application logs
|
|
||||||
|
|
||||||
Knative Serving provides default out-of-the-box logs for your application.
|
|
||||||
Access your application logs using [Accessing Logs](./accessing-logs.md) page.
|
|
||||||
|
|
||||||
## Check Route status
|
## Check Route status
|
||||||
|
|
||||||
Run the following command to get the `status` of the `Route` object with which
|
Run the following command to get the `status` of the `Route` object with which
|
||||||
|
|
@ -169,6 +164,3 @@ Choose one and use the following command to see detailed information for its
|
||||||
kubectl get pod <pod-name> --output yaml
|
kubectl get pod <pod-name> --output yaml
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
If you see issues with "user-container" container in the containerStatuses,
|
|
||||||
check your application logs as described below.
|
|
||||||
|
|
|
||||||
|
|
@ -1,123 +0,0 @@
|
||||||
---
|
|
||||||
title: "Investigating performance issues"
|
|
||||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
|
||||||
weight: 30
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
You deployed your application or function to Knative Serving but its performance
|
|
||||||
doesn't meet your expectations. Knative Serving provides various dashboards and
|
|
||||||
tools to help investigate such issues. This document reviews these dashboards
|
|
||||||
and tools.
|
|
||||||
|
|
||||||
## Request metrics
|
|
||||||
|
|
||||||
Start your investigation with the "Revision - HTTP Requests" dashboard.
|
|
||||||
|
|
||||||
1. To open this dashboard, open the Grafana UI as described in
|
|
||||||
[Accessing Metrics](./accessing-metrics.md) and navigate to "Knative
|
|
||||||
Serving - Revision HTTP Requests".
|
|
||||||
|
|
||||||
1. Select your configuration and revision from the menu on top left of the
|
|
||||||
page. You will see a page like this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This dashboard gives visibility into the following for each revision:
|
|
||||||
|
|
||||||
- Request volume
|
|
||||||
- Request volume per HTTP response code
|
|
||||||
- Response time
|
|
||||||
- Response time per HTTP response code
|
|
||||||
- Request and response sizes
|
|
||||||
|
|
||||||
This dashboard can show traffic volume or latency discrepancies between
|
|
||||||
different revisions. If, for example, a revision's latency is higher than others
|
|
||||||
revisions, then focus your investigation on the offending revision through the
|
|
||||||
rest of this guide.
|
|
||||||
|
|
||||||
## Request traces
|
|
||||||
|
|
||||||
Next, look into request traces to find out where the time is spent for a single
|
|
||||||
request.
|
|
||||||
|
|
||||||
1. To access request traces, open the Zipkin UI as described in
|
|
||||||
[Accessing Traces](./accessing-traces.md).
|
|
||||||
|
|
||||||
1. Select your revision from the "Service Name" dropdown, and then click the
|
|
||||||
"Find Traces" button. You'll get a view that looks like this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
In this example, you can see that the request spent most of its time in the
|
|
||||||
[span](https://github.com/opentracing/specification/blob/master/specification.md#the-opentracing-data-model)
|
|
||||||
right before the last, so focus your investigation on that specific span.
|
|
||||||
|
|
||||||
1. Click that span to see a view like the following:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This view shows detailed information about the specific span, such as the
|
|
||||||
micro service or external URL that was called. In this example, the call to
|
|
||||||
a Grafana URL is taking the most time. Focus your investigation on why that
|
|
||||||
URL is taking that long.
|
|
||||||
|
|
||||||
## Autoscaler metrics
|
|
||||||
|
|
||||||
If request metrics or traces do not show any obvious hot spots, or if they show
|
|
||||||
that most of the time is spent in your own code, look at autoscaler metrics
|
|
||||||
next.
|
|
||||||
|
|
||||||
1. To open the autoscaler dashboard, open Grafana UI and select "Knative
|
|
||||||
Serving - Autoscaler" dashboard, which looks like this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This view shows 4 key metrics from the Knative Serving autoscaler:
|
|
||||||
|
|
||||||
- Actual pod count: # of pods that are running a given revision
|
|
||||||
- Desired pod count: # of pods that autoscaler thinks should serve the revision
|
|
||||||
- Requested pod count: # of pods that the autoscaler requested from Kubernetes
|
|
||||||
- Panic mode: If 0, the autoscaler is operating in
|
|
||||||
[stable mode](https://github.com/knative/serving/blob/master/docs/scaling/DEVELOPMENT.md#stable-mode).
|
|
||||||
If 1, the autoscaler is operating in
|
|
||||||
[panic mode](https://github.com/knative/serving/blob/master/docs/scaling/DEVELOPMENT.md#panic-mode).
|
|
||||||
|
|
||||||
A large gap between the actual pod count and the requested pod count indicates
|
|
||||||
that the Kubernetes cluster is unable to keep up allocating new resources fast
|
|
||||||
enough, or that the Kubernetes cluster is out of requested resources.
|
|
||||||
|
|
||||||
A large gap between the requested pod count and the desired pod count indicates
|
|
||||||
that the Knative Serving autoscaler is unable to communicate with the Kubernetes
|
|
||||||
API to make the request.
|
|
||||||
|
|
||||||
In the preceding example, the autoscaler requested 18 pods to optimally serve
|
|
||||||
the traffic but was only granted 8 pods because the cluster is out of resources.
|
|
||||||
|
|
||||||
## CPU and memory usage
|
|
||||||
|
|
||||||
You can access total CPU and memory usage of your revision from the "Knative
|
|
||||||
Serving - Revision CPU and Memory Usage" dashboard, which looks like this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The first chart shows rate of the CPU usage across all pods serving the
|
|
||||||
revision. The second chart shows total memory consumed across all pods serving
|
|
||||||
the revision. Both of these metrics are further divided into per container
|
|
||||||
usage.
|
|
||||||
|
|
||||||
- user-container: This container runs the user code (application, function, or
|
|
||||||
container).
|
|
||||||
- [istio-proxy](https://github.com/istio/proxy): Sidecar container to form an
|
|
||||||
[Istio](https://istio.io/docs/concepts/what-is-istio/) mesh.
|
|
||||||
- queue-proxy: Knative Serving owned sidecar container to enforce request
|
|
||||||
concurrency limits.
|
|
||||||
- autoscaler: Knative Serving owned sidecar container to provide autoscaling for
|
|
||||||
the revision.
|
|
||||||
- fluentd-proxy: Sidecar container to collect logs from /var/log.
|
|
||||||
|
|
||||||
## Profiling
|
|
||||||
|
|
||||||
...To be filled...
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
---
|
|
||||||
title: "Fluentd container image requirements"
|
|
||||||
linkTitle: "Fluentd requirements"
|
|
||||||
weight: 10
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
Knative Serving uses a [Fluentd](https://www.fluentd.org/) docker image to
|
|
||||||
collect logs. Operators can customize their own Fluentd docker image and
|
|
||||||
configuration to define logging output.
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
Knative requires the customized Fluentd docker image with the following plugins
|
|
||||||
installed:
|
|
||||||
|
|
||||||
- [fluentd](https://github.com/fluent/fluentd) >= v0.14.0
|
|
||||||
- [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) >=
|
|
||||||
1.0.0 AND < 2.1.0: To enrich log entries with Kubernetes metadata.
|
|
||||||
- [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions) >=
|
|
||||||
0.0.9: To combine multi-line exception stack traces logs into one log entry.
|
|
||||||
- [fluent-plugin-multi-format-parser](https://github.com/repeatedly/fluent-plugin-multi-format-parser) >=
|
|
||||||
1.0.0: To detect log format as Json or plain text.
|
|
||||||
|
|
||||||
## Sample images
|
|
||||||
|
|
||||||
Operators can use any Docker image which meets the requirements above and
|
|
||||||
includes the desired output plugin. Two examples below:
|
|
||||||
|
|
||||||
### Send logs to Elasticsearch
|
|
||||||
|
|
||||||
Operators can use
|
|
||||||
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image)
|
|
||||||
which includes
|
|
||||||
[fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch)
|
|
||||||
that allows sending logs to a Elasticsearch service.
|
|
||||||
|
|
||||||
### Send logs to Stackdriver
|
|
||||||
|
|
||||||
This sample [Dockerfile](./stackdriver/Dockerfile) is based on
|
|
||||||
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch).
|
|
||||||
It additionally adds one more plugin -
|
|
||||||
[fluent-plugin-google-cloud](https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud)
|
|
||||||
which allows sending logs to Stackdriver.
|
|
||||||
|
|
||||||
Operators can build this image and push it to a container registry which their
|
|
||||||
Kubernetes cluster has access to. See
|
|
||||||
[Setting Up A Logging Plugin](./setting-up-a-logging-plugin.md) for details.
|
|
||||||
|
|
||||||
> **NOTE**: Operators must add the credentials file the Stackdriver agent needs
|
|
||||||
> to the Docker image if their Knative Serving is not built on a Google Cloud
|
|
||||||
> Platform-based cluster or if they want to send logs to another Google Cloud
|
|
||||||
> Platform project. See
|
|
||||||
> [here](https://cloud.google.com/logging/docs/agent/authorization) for more
|
|
||||||
> information.
|
|
||||||
|
|
@ -1,335 +0,0 @@
|
||||||
---
|
|
||||||
title: "Installing logging, metrics, and traces"
|
|
||||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
|
||||||
weight: 40
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
If you followed one of the
|
|
||||||
[comprehensive install guides](../install/README.md#install-guides) or you
|
|
||||||
performed a custom installation and included the `monitoring.yaml` file in your
|
|
||||||
installation, all of the observability features are already installed and you
|
|
||||||
can skip down to the
|
|
||||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
|
||||||
|
|
||||||
If you have not yet installed any observability plugins, continue to the next
|
|
||||||
sections to do so now.
|
|
||||||
|
|
||||||
## Metrics
|
|
||||||
|
|
||||||
1. Run the following command and follow the instructions below to enable request
|
|
||||||
metrics if they are wanted:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl edit cm -n knative-serving config-observability
|
|
||||||
```
|
|
||||||
|
|
||||||
Add `metrics.request-metrics-backend-destination: prometheus` to `data`
|
|
||||||
field. You can find detailed information in `data._example` field in the
|
|
||||||
`ConfigMap` you are editing.
|
|
||||||
|
|
||||||
1. Run the following command to install Prometheus and Grafana:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-metrics-prometheus.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Ensure that the `grafana-*`, `kibana-logging-*`, `kube-state-metrics-*`,
|
|
||||||
`node-exporter-*` and `prometheus-system-*` pods all report a `Running`
|
|
||||||
status:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get pods --namespace knative-monitoring --watch
|
|
||||||
```
|
|
||||||
|
|
||||||
For example:
|
|
||||||
|
|
||||||
```text
|
|
||||||
NAME READY STATUS RESTARTS AGE
|
|
||||||
grafana-798cf569ff-v4q74 1/1 Running 0 2d
|
|
||||||
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
|
|
||||||
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
|
|
||||||
node-exporter-cr6bh 2/2 Running 0 2d
|
|
||||||
node-exporter-mf6k7 2/2 Running 0 2d
|
|
||||||
node-exporter-rhzr7 2/2 Running 0 2d
|
|
||||||
prometheus-system-0 1/1 Running 0 2d
|
|
||||||
prometheus-system-1 1/1 Running 0 2d
|
|
||||||
```
|
|
||||||
|
|
||||||
Tip: Hit CTRL+C to exit watch mode.
|
|
||||||
|
|
||||||
[Accessing Metrics](./accessing-metrics.md) for more information about metrics
|
|
||||||
in Knative.
|
|
||||||
|
|
||||||
## Logs
|
|
||||||
|
|
||||||
### Enable Request Logs
|
|
||||||
|
|
||||||
Run the following command and follow the instructions below to enable request
|
|
||||||
logs if they are wanted:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl edit cm -n knative-serving config-observability
|
|
||||||
```
|
|
||||||
|
|
||||||
Copy `logging.request-log-template` from `data._example` field to`data` field in
|
|
||||||
the `ConfigMap` you are editing. You can find detailed information in
|
|
||||||
`data._example` field to customize the request log format.
|
|
||||||
|
|
||||||
### Choose One Logging Backend
|
|
||||||
|
|
||||||
Knative offers three different setups for collecting logs. Choose one to
|
|
||||||
install:
|
|
||||||
|
|
||||||
1. [Elasticsearch and Kibana](#elasticsearch-and-kibana)
|
|
||||||
1. [Stackdriver](#stackdriver)
|
|
||||||
1. [Custom logging plugin](./setting-up-a-logging-plugin.md)
|
|
||||||
|
|
||||||
#### Elasticsearch and Kibana
|
|
||||||
|
|
||||||
1. Run the following command to install an ELK stack:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-logs-elasticsearch.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Ensure that the `elasticsearch-logging-*`, `fluentd-ds-*`, and
|
|
||||||
`kibana-logging-*` pods all report a `Running` status:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get pods --namespace knative-monitoring --watch
|
|
||||||
```
|
|
||||||
|
|
||||||
For example:
|
|
||||||
|
|
||||||
```text
|
|
||||||
NAME READY STATUS RESTARTS AGE
|
|
||||||
elasticsearch-logging-0 1/1 Running 0 2d
|
|
||||||
elasticsearch-logging-1 1/1 Running 0 2d
|
|
||||||
fluentd-ds-5kc85 1/1 Running 0 2d
|
|
||||||
fluentd-ds-vhrcq 1/1 Running 0 2d
|
|
||||||
fluentd-ds-xghk9 1/1 Running 0 2d
|
|
||||||
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
|
|
||||||
```
|
|
||||||
|
|
||||||
Tip: Hit CTRL+C to exit watch mode.
|
|
||||||
|
|
||||||
1. Verify that each of your nodes have the
|
|
||||||
`beta.kubernetes.io/fluentd-ds-ready=true` label:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
|
|
||||||
```
|
|
||||||
|
|
||||||
1. If you receive the `No Resources Found` response:
|
|
||||||
|
|
||||||
1. Run the following command to ensure that the Fluentd DaemonSet runs on all
|
|
||||||
your nodes:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Run the following command to ensure that the `fluentd-ds` daemonset is
|
|
||||||
ready on at least one node:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get daemonset fluentd-ds --namespace knative-monitoring --watch
|
|
||||||
```
|
|
||||||
|
|
||||||
Tip: Hit CTRL+C to exit watch mode.
|
|
||||||
|
|
||||||
1. When the installation is complete and all the resources are running, you can
|
|
||||||
continue to the next section and begin creating your Elasticsearch indices.
|
|
||||||
|
|
||||||
##### Create Elasticsearch Indices
|
|
||||||
|
|
||||||
To visualize logs with Kibana, you need to set which Elasticsearch indices to
|
|
||||||
explore.
|
|
||||||
|
|
||||||
- To open the Kibana UI (the visualization tool for
|
|
||||||
[Elasticsearch](https://info.elastic.co)), you must start a local proxy by
|
|
||||||
running the following command:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl proxy
|
|
||||||
```
|
|
||||||
|
|
||||||
This command starts a local proxy of Kibana on port 8001. For security
|
|
||||||
reasons, the Kibana UI is exposed only within the cluster.
|
|
||||||
|
|
||||||
- Navigate to the
|
|
||||||
[Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana).
|
|
||||||
_It might take a couple of minutes for the proxy to work_.
|
|
||||||
|
|
||||||
- Within the "Configure an index pattern" page, enter `logstash-*` to
|
|
||||||
`Index pattern` and select `@timestamp` from `Time Filter field name` and
|
|
||||||
click on `Create` button.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
See [Accessing Logs](./accessing-logs.md) for more information about logs in
|
|
||||||
Knative.
|
|
||||||
|
|
||||||
#### Stackdriver
|
|
||||||
|
|
||||||
To configure and setup monitoring:
|
|
||||||
|
|
||||||
1. Clone the Knative Serving repository:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
git clone -b {{< version >}} https://github.com/knative/serving knative-serving
|
|
||||||
cd knative-serving
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Choose a container image that meets the
|
|
||||||
[Fluentd image requirements](./fluentd-requirements.md#requirements). For
|
|
||||||
example, you can use a public image. Or you can create a custom one and
|
|
||||||
upload the image to a container registry which your cluster has read access
|
|
||||||
to.
|
|
||||||
|
|
||||||
You must configure and build your own Fluentd image if either of the
|
|
||||||
following are true:
|
|
||||||
|
|
||||||
- Your Knative Serving component is not hosted on a Google Cloud Platform
|
|
||||||
(GCP) based cluster.
|
|
||||||
- You want to send logs to another GCP project.
|
|
||||||
|
|
||||||
1. Follow the instructions in
|
|
||||||
[Setting up a logging plugin](./setting-up-a-logging-plugin.md#Configuring)
|
|
||||||
to configure the stackdriver components settings.
|
|
||||||
|
|
||||||
1. Install Knative Stackdriver components by running the following command from
|
|
||||||
the root directory of [knative/serving](https://github.com/knative/serving)
|
|
||||||
repository:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --recursive --filename config/monitoring/100-namespace.yaml \
|
|
||||||
--filename config/monitoring/logging/stackdriver
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Ensure that the `fluentd-ds-*` pods all report a `Running` status:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get pods --namespace knative-monitoring --watch
|
|
||||||
```
|
|
||||||
|
|
||||||
For example:
|
|
||||||
|
|
||||||
```text
|
|
||||||
NAME READY STATUS RESTARTS AGE
|
|
||||||
fluentd-ds-5kc85 1/1 Running 0 2d
|
|
||||||
fluentd-ds-vhrcq 1/1 Running 0 2d
|
|
||||||
fluentd-ds-xghk9 1/1 Running 0 2d
|
|
||||||
```
|
|
||||||
|
|
||||||
Tip: Hit CTRL+C to exit watch mode.
|
|
||||||
|
|
||||||
1. Verify that each of your nodes have the
|
|
||||||
`beta.kubernetes.io/fluentd-ds-ready=true` label:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
|
|
||||||
```
|
|
||||||
|
|
||||||
1. If you receive the `No Resources Found` response:
|
|
||||||
|
|
||||||
1. Run the following command to ensure that the Fluentd DaemonSet runs on
|
|
||||||
all your nodes:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Run the following command to ensure that the `fluentd-ds` daemonset is
|
|
||||||
ready on at least one node:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl get daemonset fluentd-ds --namespace knative-monitoring
|
|
||||||
```
|
|
||||||
|
|
||||||
See [Accessing Logs](./accessing-logs.md) for more information about logs in
|
|
||||||
Knative.
|
|
||||||
|
|
||||||
## End to end request tracing
|
|
||||||
|
|
||||||
You can choose from one of the following options to enable request tracing in
|
|
||||||
your Knative Serving cluster.
|
|
||||||
|
|
||||||
**Important**: Your cluster supports only a single request trace tool. If you
|
|
||||||
want to replace a currently installed request trace tool, you must first
|
|
||||||
uninstall that tool before installing the new tool.
|
|
||||||
|
|
||||||
### Zipkin
|
|
||||||
|
|
||||||
1. Install support for Zipkin:
|
|
||||||
|
|
||||||
- If Elasticsearch is not installed or if you don't want to persist end to
|
|
||||||
end traces, run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-zipkin-in-mem.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
- If Elasticsearch is installed and you want to persist end to end traces,
|
|
||||||
first run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-zipkin.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Create an Elasticsearch index for end to end traces:
|
|
||||||
|
|
||||||
1. Open Kibana UI as described in
|
|
||||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
|
||||||
|
|
||||||
1. Select `Create Index Pattern` button on top left of the page. Enter
|
|
||||||
`zipkin*` to `Index pattern` and select `timestamp_millis` from
|
|
||||||
`Time Filter field name` and click on `Create` button.
|
|
||||||
|
|
||||||
Visit [Accessing Traces](./accessing-traces.md) for more information on end to
|
|
||||||
end traces.
|
|
||||||
|
|
||||||
### Jaeger
|
|
||||||
|
|
||||||
1. Install the Jaeger operator. Use the instructions in
|
|
||||||
jaegertracing/jaeger-operator repository and follow only the steps in the
|
|
||||||
[Installing the operator](https://github.com/jaegertracing/jaeger-operator#installing-the-operator)
|
|
||||||
section.
|
|
||||||
|
|
||||||
1. Install support for Jaeger:
|
|
||||||
|
|
||||||
- If Elasticsearch is not installed or if you don't want to persist end to
|
|
||||||
end traces, run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-jaeger-in-mem.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
- If Elasticsearch is installed and you want to persist end to end traces,
|
|
||||||
first run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-jaeger.yaml" >}}
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Create an Elasticsearch index for end to end traces:
|
|
||||||
|
|
||||||
1. Open Kibana UI as described in
|
|
||||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
|
||||||
|
|
||||||
1. Select `Create Index Pattern` button on top left of the page. Enter
|
|
||||||
`jaeger*` to `Index pattern` and select `timestamp_millis` from
|
|
||||||
`Time Filter field name` and click on `Create` button.
|
|
||||||
|
|
||||||
Visit [Accessing Traces](./accessing-traces.md) for more information on end to
|
|
||||||
end traces.
|
|
||||||
|
|
||||||
## Learn More
|
|
||||||
|
|
||||||
- Learn more about accessing logs, metrics, and traces:
|
|
||||||
- [Accessing Logs](./accessing-logs.md)
|
|
||||||
- [Accessing Metrics](./accessing-metrics.md)
|
|
||||||
- [Accessing Traces](./accessing-traces.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
# Copyright 2018 The Knative Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
FROM golang AS builder
|
|
||||||
|
|
||||||
WORKDIR /go/src/github.com/knative/docs/docs/
|
|
||||||
ADD . /go/src/github.com/knative/docs/
|
|
||||||
|
|
||||||
RUN CGO_ENABLED=0 go build ./serving/samples/telemetry-go/
|
|
||||||
|
|
||||||
FROM gcr.io/distroless/base
|
|
||||||
|
|
||||||
EXPOSE 8080
|
|
||||||
COPY --from=builder /go/src/github.com/knative/docs/docs/telemetry-go /sample
|
|
||||||
|
|
||||||
ENTRYPOINT ["/sample"]
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
# The OWNERS file is used by prow to automatically merge approved PRs.
|
|
||||||
|
|
||||||
approvers:
|
|
||||||
- mdemirhan
|
|
||||||
- yanweiguo
|
|
||||||
|
|
@ -1,166 +0,0 @@
|
||||||
This sample runs a simple web server that makes calls to other in-cluster
|
|
||||||
services and responds to requests with "Hello World!". The purpose of this
|
|
||||||
sample is to show generating [metrics](../../accessing-metrics.md),
|
|
||||||
[logs](../../accessing-logs.md) and distributed
|
|
||||||
[traces](../../accessing-traces.md). This sample also shows how to create a
|
|
||||||
dedicated Prometheus instance rather than using the default installation.
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
1. A Kubernetes cluster with [Knative Serving](../../../install/README.md)
|
|
||||||
installed and DNS configured.
|
|
||||||
2. Check if Knative monitoring components are installed:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get pods --namespace knative-monitoring
|
|
||||||
```
|
|
||||||
|
|
||||||
- If pods aren't found, install
|
|
||||||
[Knative monitoring component](../../installing-logging-metrics-traces.md).
|
|
||||||
|
|
||||||
3. Install
|
|
||||||
[Docker](https://docs.docker.com/get-started/#prepare-your-docker-environment).
|
|
||||||
4. Check out the code:
|
|
||||||
|
|
||||||
```
|
|
||||||
go get -d github.com/knative/docs/docs/serving/samples/telemetry-go
|
|
||||||
```
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
Build the application container and publish it to a container registry:
|
|
||||||
|
|
||||||
1. Move into the sample directory:
|
|
||||||
|
|
||||||
```
|
|
||||||
cd $GOPATH/src/github.com/knative/docs
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Set your preferred container registry:
|
|
||||||
|
|
||||||
```
|
|
||||||
export REPO="gcr.io/<YOUR_PROJECT_ID>"
|
|
||||||
```
|
|
||||||
|
|
||||||
This example shows how to use Google Container Registry (GCR). You will need a
|
|
||||||
Google Cloud Project and to enable the
|
|
||||||
[Google Container Registry API](https://console.cloud.google.com/apis/library/containerregistry.googleapis.com).
|
|
||||||
|
|
||||||
3. Use Docker to build your application container:
|
|
||||||
|
|
||||||
```
|
|
||||||
docker build \
|
|
||||||
--tag "${REPO}/docs/serving/samples/telemetry-go" \
|
|
||||||
--file=docs/serving/samples/telemetry-go/Dockerfile .
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Push your container to a container registry:
|
|
||||||
|
|
||||||
```
|
|
||||||
docker push "${REPO}/docs/serving/samples/telemetry-go"
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Replace the image reference path with our published image path in the
|
|
||||||
configuration file (`docs/serving/samples/telemetry-go/sample.yaml`):
|
|
||||||
|
|
||||||
- Manually replace:
|
|
||||||
`image: github.com/knative/docs/docs/serving/samples/telemetry-go` with
|
|
||||||
`image: <YOUR_CONTAINER_REGISTRY>/docs/serving/samples/telemetry-go`
|
|
||||||
|
|
||||||
|
|
||||||
Or
|
|
||||||
|
|
||||||
- Use run this command:
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
perl -pi -e "s@github.com/knative/docs@${REPO}@g" docs/serving/samples/telemetry-go/sample.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Deploy the Service
|
|
||||||
|
|
||||||
Deploy this application to Knative Serving:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl apply --filename docs/serving/samples/telemetry-go/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Explore the Service
|
|
||||||
|
|
||||||
Inspect the created resources with the `kubectl` commands:
|
|
||||||
|
|
||||||
- View the created Route resource:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get route --output yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
- View the created Configuration resource:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get configurations --output yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
- View the Revision that was created by the Configuration:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get revisions --output yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Access the Service
|
|
||||||
|
|
||||||
To access this service via `curl`, you need to determine its ingress address.
|
|
||||||
|
|
||||||
1. To determine if your route is ready:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl get route
|
|
||||||
```
|
|
||||||
|
|
||||||
When the route is ready, you'll see the following fields reported as:
|
|
||||||
|
|
||||||
```YAML
|
|
||||||
telemetrysample-route http://telemetrysample-route.default.example.com True
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Make a request to the service to see the `Hello World!` message:
|
|
||||||
|
|
||||||
```
|
|
||||||
curl http://telemetrysample-route.default.example.com
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Make a request to the `/log` endpoint to generate logs to the `stdout` file
|
|
||||||
and generate files under `/var/log` in both `JSON` and plain text formats:
|
|
||||||
|
|
||||||
```
|
|
||||||
curl http://telemetrysample-route.default.example.com/log
|
|
||||||
```
|
|
||||||
|
|
||||||
## Access Logs
|
|
||||||
|
|
||||||
You can access to the logs from Kibana UI - see [Logs](../../accessing-logs.md)
|
|
||||||
for more information.
|
|
||||||
|
|
||||||
## Access per Request Traces
|
|
||||||
|
|
||||||
You can access to per request traces from Zipkin UI - see
|
|
||||||
[Traces](../../accessing-traces.md) for more information.
|
|
||||||
|
|
||||||
## Accessing Custom Metrics
|
|
||||||
|
|
||||||
You can see published metrics using Prometheus UI. To access to the UI, forward
|
|
||||||
the Prometheus server to your machine:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl port-forward $(kubectl get pods --selector=app=prometheus-test --output=jsonpath="{.items[0].metadata.name}") 9090
|
|
||||||
```
|
|
||||||
|
|
||||||
Then browse to http://localhost:9090.
|
|
||||||
|
|
||||||
## Clean up
|
|
||||||
|
|
||||||
To clean up the sample service:
|
|
||||||
|
|
||||||
```
|
|
||||||
kubectl delete --filename docs/serving/samples/telemetry-go/
|
|
||||||
```
|
|
||||||
|
|
@ -1,8 +0,0 @@
|
||||||
---
|
|
||||||
title: "Observability: Monitoring, logging, and tracing sample - Go"
|
|
||||||
linkTitle: "Observability - Go"
|
|
||||||
weight: 1
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
{{% readfile file="README.md" %}}
|
|
||||||
|
|
@ -1,122 +0,0 @@
|
||||||
# Copyright 2018 The Knative Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: prometheus-test
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRole
|
|
||||||
metadata:
|
|
||||||
name: prometheus-test
|
|
||||||
rules:
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources:
|
|
||||||
- nodes
|
|
||||||
- services
|
|
||||||
- endpoints
|
|
||||||
- pods
|
|
||||||
verbs: ["get", "list", "watch"]
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources:
|
|
||||||
- configmaps
|
|
||||||
verbs: ["get"]
|
|
||||||
- nonResourceURLs: ["/metrics"]
|
|
||||||
verbs: ["get"]
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: prometheus-test
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: prometheus-test
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: prometheus-test
|
|
||||||
namespace: default
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: prometheus-test-scrape-config
|
|
||||||
labels:
|
|
||||||
name: prometheus-test-scrape-config
|
|
||||||
namespace: default
|
|
||||||
data:
|
|
||||||
prometheus.yml: |-
|
|
||||||
scrape_configs:
|
|
||||||
- job_name: telemetrysample-test
|
|
||||||
scrape_interval: 3s
|
|
||||||
scrape_timeout: 3s
|
|
||||||
kubernetes_sd_configs:
|
|
||||||
- role: endpoints
|
|
||||||
relabel_configs:
|
|
||||||
# Scrape only the the targets matching the following metadata
|
|
||||||
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_serving_knative_dev_configuration, __meta_kubernetes_endpoint_port_name]
|
|
||||||
action: keep
|
|
||||||
regex: default;telemetrysample-configuration;http
|
|
||||||
# Rename metadata labels to be reader friendly
|
|
||||||
- source_labels: [__meta_kubernetes_namespace]
|
|
||||||
action: replace
|
|
||||||
regex: (.*)
|
|
||||||
target_label: namespace
|
|
||||||
replacement: $1
|
|
||||||
- source_labels: [__meta_kubernetes_pod_name]
|
|
||||||
action: replace
|
|
||||||
regex: (.*)
|
|
||||||
target_label: pod
|
|
||||||
replacement: $1
|
|
||||||
- source_labels: [__meta_kubernetes_service_name]
|
|
||||||
action: replace
|
|
||||||
regex: (.*)
|
|
||||||
target_label: service
|
|
||||||
replacement: $1
|
|
||||||
---
|
|
||||||
apiVersion: apps/v1beta1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: prometheus-test-deployment
|
|
||||||
namespace: default
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: prometheus-test
|
|
||||||
spec:
|
|
||||||
serviceAccountName: prometheus-test
|
|
||||||
containers:
|
|
||||||
- name: prometheus
|
|
||||||
image: prom/prometheus:v2.2.1
|
|
||||||
args:
|
|
||||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
||||||
- "--storage.tsdb.path=/prometheus/"
|
|
||||||
ports:
|
|
||||||
- containerPort: 9090
|
|
||||||
volumeMounts:
|
|
||||||
- name: prometheus-config-volume
|
|
||||||
mountPath: /etc/prometheus/
|
|
||||||
- name: prometheus-storage-volume
|
|
||||||
mountPath: /prometheus/
|
|
||||||
volumes:
|
|
||||||
- name: prometheus-config-volume
|
|
||||||
configMap:
|
|
||||||
defaultMode: 420
|
|
||||||
name: prometheus-test-scrape-config
|
|
||||||
|
|
||||||
- name: prometheus-storage-volume
|
|
||||||
emptyDir: {}
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
||||||
# Copyright 2018 The Knative Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
apiVersion: serving.knative.dev/v1
|
|
||||||
kind: Route
|
|
||||||
metadata:
|
|
||||||
name: telemetrysample-route
|
|
||||||
namespace: default
|
|
||||||
spec:
|
|
||||||
traffic:
|
|
||||||
- configurationName: telemetrysample-configuration
|
|
||||||
percent: 100
|
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
# Copyright 2018 The Knative Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
apiVersion: serving.knative.dev/v1
|
|
||||||
kind: Configuration
|
|
||||||
metadata:
|
|
||||||
name: telemetrysample-configuration
|
|
||||||
namespace: default
|
|
||||||
labels:
|
|
||||||
app: telemetrysample
|
|
||||||
spec:
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
knative.dev/type: app
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- # This is the Go import path for the binary to containerize
|
|
||||||
# and substitute here.
|
|
||||||
image: github.com/knative/docs/docs/serving/samples/telemetry-go
|
|
||||||
|
|
@ -1,269 +0,0 @@
|
||||||
/*
|
|
||||||
Copyright 2018 The Knative Authors
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
*/
|
|
||||||
|
|
||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"log"
|
|
||||||
"math/rand"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
openzipkin "github.com/openzipkin/zipkin-go"
|
|
||||||
httpreporter "github.com/openzipkin/zipkin-go/reporter/http"
|
|
||||||
|
|
||||||
"go.opencensus.io/exporter/prometheus"
|
|
||||||
"go.opencensus.io/exporter/zipkin"
|
|
||||||
"go.opencensus.io/plugin/ochttp"
|
|
||||||
"go.opencensus.io/plugin/ochttp/propagation/b3"
|
|
||||||
"go.opencensus.io/stats"
|
|
||||||
"go.opencensus.io/stats/view"
|
|
||||||
"go.opencensus.io/tag"
|
|
||||||
"go.opencensus.io/trace"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
// Create a measurement to keep track of incoming request counts.
|
|
||||||
// For more information on measurements, see https://pkg.go.dev/go.opencensus.io/stats
|
|
||||||
requestCount = stats.Int64("request_count", "Total number of requests.", stats.UnitNone)
|
|
||||||
|
|
||||||
// Create a measurement to keep track of request durations.
|
|
||||||
requestDuration = stats.Int64("request_duration", "Histogram of the request duration.", stats.UnitMilliseconds)
|
|
||||||
|
|
||||||
// Capture the HTTP response code in a tag so that we can aggregate and visualize
|
|
||||||
// this metric based on different response codes (see count of all 400 vs 200 for example).
|
|
||||||
// For more information on tags, see https://pkg.go.dev/go.opencensus.io/tag
|
|
||||||
requestStatusTagKey tag.Key
|
|
||||||
)
|
|
||||||
|
|
||||||
func main() {
|
|
||||||
flag.Parse()
|
|
||||||
log.SetPrefix("TelemetrySample: ")
|
|
||||||
log.Print("Telemetry sample started.")
|
|
||||||
|
|
||||||
//
|
|
||||||
// Metrics setup
|
|
||||||
//
|
|
||||||
|
|
||||||
// We want our metrics in Prometheus. Create the exporter to do that.
|
|
||||||
promExporter, err := prometheus.NewExporter(prometheus.Options{Namespace: "telemetrysample"})
|
|
||||||
if err != nil {
|
|
||||||
log.Fatal(err)
|
|
||||||
}
|
|
||||||
view.RegisterExporter(promExporter)
|
|
||||||
|
|
||||||
// Create the tag keys that will be used to add tags to our measurements.
|
|
||||||
requestStatusTagKey, err = tag.NewKey("status")
|
|
||||||
if err != nil {
|
|
||||||
log.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create view to see our measurements cumulatively.
|
|
||||||
err = view.Register(
|
|
||||||
&view.View{
|
|
||||||
Description: "Total number of requests.",
|
|
||||||
Measure: requestCount,
|
|
||||||
Aggregation: view.Count(),
|
|
||||||
TagKeys: []tag.Key{requestStatusTagKey},
|
|
||||||
},
|
|
||||||
&view.View{
|
|
||||||
Description: "Histogram of the request duration.",
|
|
||||||
Measure: requestDuration,
|
|
||||||
Aggregation: view.Distribution(10, 25, 50, 100, 250, 500, 1000, 2500),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Cannot subscribe to the view: %v", err)
|
|
||||||
}
|
|
||||||
view.SetReportingPeriod(1 * time.Second)
|
|
||||||
|
|
||||||
//
|
|
||||||
// Tracing setup
|
|
||||||
//
|
|
||||||
|
|
||||||
// If your service only calls other Knative Serving revisions, then Zipkin setup below
|
|
||||||
// is not needed. All that you need is to extract the following headers from incoming
|
|
||||||
// requests and copy them to your outgoing requests.
|
|
||||||
// x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context
|
|
||||||
//
|
|
||||||
// For richer instrumentation, or to instrument calls that are made to external services,
|
|
||||||
// you can instrument the code using Zipkin's Go client library as shown below.
|
|
||||||
//
|
|
||||||
// Zipkin is installed in istio-system namespace because istio assumes that zipkin is installed there.
|
|
||||||
// Ideally this value should be config driven, but for demo purposes, we will hardcode it here.
|
|
||||||
// For unit tests, reporter.noopReporter can be used instead of the httpreporter below.
|
|
||||||
reporter := httpreporter.NewReporter("http://zipkin.istio-system.svc.cluster.local:9411/api/v2/spans")
|
|
||||||
defer reporter.Close()
|
|
||||||
localEndpoint, err := openzipkin.NewEndpoint("TelemetrySample", "localhost:8080")
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Unable to create zipkin local endpoint: %+v\n", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The OpenCensus exporter wraps the Zipkin reporter
|
|
||||||
zipkinExporter := zipkin.NewExporter(reporter, localEndpoint)
|
|
||||||
trace.RegisterExporter(zipkinExporter)
|
|
||||||
|
|
||||||
// For example purposes, sample every trace.
|
|
||||||
trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})
|
|
||||||
|
|
||||||
// Use an http.RoundTripper that instruments all outgoing requests with stats and tracing.
|
|
||||||
client := &http.Client{Transport: &ochttp.Transport{Propagation: &b3.HTTPFormat{}}}
|
|
||||||
|
|
||||||
// Implements an http.Handler that automatically extracts traces
|
|
||||||
// from the incoming request and writes them back to the response.
|
|
||||||
handler := &ochttp.Handler{Propagation: &b3.HTTPFormat{}, Handler: rootHandler(client)}
|
|
||||||
|
|
||||||
mux := http.NewServeMux()
|
|
||||||
mux.Handle("/", handler)
|
|
||||||
mux.Handle("/log", &ochttp.Handler{Propagation: &b3.HTTPFormat{}, Handler: logHandler(client)})
|
|
||||||
// Setup OpenCensus Prometheus exporter to handle scraping for metrics.
|
|
||||||
mux.Handle("/metrics", promExporter)
|
|
||||||
http.ListenAndServe(":8080", mux)
|
|
||||||
}
|
|
||||||
|
|
||||||
var statusCodes = [...]int{
|
|
||||||
http.StatusOK,
|
|
||||||
http.StatusCreated,
|
|
||||||
http.StatusAccepted,
|
|
||||||
http.StatusBadRequest,
|
|
||||||
http.StatusUnauthorized,
|
|
||||||
}
|
|
||||||
|
|
||||||
func rootHandler(client *http.Client) http.HandlerFunc {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
// Write the http request headers to the log for demonstration purposes.
|
|
||||||
log.Printf("Request received. Request headers: %v", r.Header)
|
|
||||||
|
|
||||||
// Pick a random return code - this is used for demonstrating metrics & logs
|
|
||||||
// with different responses.
|
|
||||||
status := statusCodes[rand.Intn(len(statusCodes))]
|
|
||||||
|
|
||||||
// Before returning from this function, update requestCount and requestDuration metrics.
|
|
||||||
defer func(start time.Time) {
|
|
||||||
// Create a new context with the metric tags and their values.
|
|
||||||
ctx, err := tag.New(r.Context(), tag.Insert(requestStatusTagKey, fmt.Sprint(status)))
|
|
||||||
if err != nil {
|
|
||||||
log.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Increment the request count by one.
|
|
||||||
stats.Record(ctx, requestCount.M(1))
|
|
||||||
|
|
||||||
// Record the request duration.
|
|
||||||
stats.Record(ctx, requestDuration.M(time.Since(start).Nanoseconds()/int64(time.Millisecond)))
|
|
||||||
}(time.Now())
|
|
||||||
|
|
||||||
getWithContext := func(url string) (*http.Response, error) {
|
|
||||||
req, err := http.NewRequest("GET", url, nil)
|
|
||||||
if err != nil {
|
|
||||||
log.Printf("Failed to create a new request: %v", err)
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
// If we don't attach the incoming request's context, we will end up creating
|
|
||||||
// a new trace id (i.e. a new root span) rather than attaching to the incoming
|
|
||||||
// request's trace id. To ensure the continuity of the trace, use incoming
|
|
||||||
// request's context for the outgoing request.
|
|
||||||
req = req.WithContext(r.Context())
|
|
||||||
return client.Do(req)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Simulate a few extra calls to other services to demostrate the distributed tracing capabilities.
|
|
||||||
// In sequence, call three different other services. For each call, we will create a new span
|
|
||||||
// to track that call in the call graph. See http://opentracing.io/documentation/ for more information
|
|
||||||
// on these concepts.
|
|
||||||
res, err := getWithContext("http://prometheus-system-np.knative-monitoring.svc.cluster.local:8080")
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
// Close directly rather than defer to ensure that spans are finished now rather than
|
|
||||||
// end of this function. This applies to the remaining res.Body().Close calls below.
|
|
||||||
res.Body.Close()
|
|
||||||
|
|
||||||
res, err = getWithContext("http://grafana.knative-monitoring.svc.cluster.local:30802")
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
res.Body.Close()
|
|
||||||
|
|
||||||
// Let's call a non-existent URL to demonstrate the failure scenarios in distributed tracing.
|
|
||||||
res, err = getWithContext("http://invalidurl.svc.cluster.local")
|
|
||||||
if err != nil {
|
|
||||||
log.Printf("Request failed: %v", err)
|
|
||||||
} else {
|
|
||||||
res.Body.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
|
||||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
|
||||||
w.WriteHeader(status)
|
|
||||||
w.Write([]byte("Hello world!\n"))
|
|
||||||
log.Printf("Request complete. Status: %v", status)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func logHandler(client *http.Client) http.HandlerFunc {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
timestamp := time.Now()
|
|
||||||
|
|
||||||
// Send logs to STDOUT
|
|
||||||
msg := "A log in plain text format to STDOUT"
|
|
||||||
fmt.Fprintln(os.Stdout, msg)
|
|
||||||
|
|
||||||
data := map[string]string{
|
|
||||||
"log": "A log in json format to STDOUT",
|
|
||||||
"foo": "bar",
|
|
||||||
"time": timestamp.String(),
|
|
||||||
// Cluster operator can configure which field is used as time key and what
|
|
||||||
// the format is. For example, in config/monitoring/150-elasticsearch/100-fluentd-configmap.yaml,
|
|
||||||
// fluentd-time is the reserved key to tell fluentd the logging time. It
|
|
||||||
// must be in the format of RFC3339Nano, i.e. %Y-%m-%dT%H:%M:%S.%NZ.
|
|
||||||
// Without this, fluentd uses the time when it collect the log as an
|
|
||||||
// event time.
|
|
||||||
"fluentd-time": timestamp.Format(time.RFC3339Nano),
|
|
||||||
}
|
|
||||||
jsonOutput, _ := json.Marshal(data)
|
|
||||||
fmt.Fprintln(os.Stdout, string(jsonOutput))
|
|
||||||
|
|
||||||
// Send logs to /var/log
|
|
||||||
fileName := "/var/log/sample.log"
|
|
||||||
f, err := os.OpenFile(fileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
|
|
||||||
defer f.Close()
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
msg = "A log in plain text format to /var/log\n"
|
|
||||||
if _, err := f.WriteString(msg); err != nil {
|
|
||||||
fmt.Fprintf(os.Stderr, "Failed to write to %s: %v", fileName, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
data["log"] = "A log in json format to /var/log"
|
|
||||||
jsonOutput, _ := json.Marshal(data)
|
|
||||||
if _, err := f.WriteString(string(jsonOutput) + "\n"); err != nil {
|
|
||||||
fmt.Fprintf(os.Stderr, "Failed to write to %s: %v", fileName, err)
|
|
||||||
}
|
|
||||||
f.Sync()
|
|
||||||
} else {
|
|
||||||
fmt.Fprintf(os.Stderr, "Failed to create %s: %v", fileName, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprint(w, "Sending logs done.\n")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,90 +0,0 @@
|
||||||
---
|
|
||||||
title: "Setting up a logging plugin"
|
|
||||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
|
||||||
weight: 50
|
|
||||||
type: "docs"
|
|
||||||
---
|
|
||||||
|
|
||||||
Knative allows cluster operators to use different backends for their logging
|
|
||||||
needs. This document describes how to change these settings. Knative currently
|
|
||||||
requires changes in Fluentd configuration files, however we plan on abstracting
|
|
||||||
logging configuration in the future
|
|
||||||
([#906](https://github.com/knative/serving/issues/906)). Once
|
|
||||||
[#906](https://github.com/knative/serving/issues/906) is complete, the
|
|
||||||
methodology described in this document will no longer be valid and migration to
|
|
||||||
a new process will be required. In order to minimize the effort for a future
|
|
||||||
migration, we recommend only changing the output configuration of Fluentd and
|
|
||||||
leaving the rest intact.
|
|
||||||
|
|
||||||
**NOTE**: All the files mentioned below are in
|
|
||||||
[knative/serving](https://github.com/knative/serving) repository. You run the
|
|
||||||
commands mentioned below from the root directory of `knative/serving`.
|
|
||||||
|
|
||||||
## Configuring
|
|
||||||
|
|
||||||
### Configure the DaemonSet for stdout/stderr logs
|
|
||||||
|
|
||||||
Operators can do the following steps to configure the Fluentd DaemonSet for
|
|
||||||
collecting `stdout/stderr` logs from the containers:
|
|
||||||
|
|
||||||
1. Replace `900.output.conf` part in
|
|
||||||
[100-fluentd-configmap.yaml](https://github.com/knative/serving/blob/master/config/monitoring/logging/elasticsearch/100-fluentd-configmap.yaml)
|
|
||||||
with the desired output configuration. Knative provides a sample for sending
|
|
||||||
logs to Elasticsearch or Stackdriver. Developers can simply use
|
|
||||||
`100-fluentd-configmap.yaml` or override any with other configuration.
|
|
||||||
2. Replace the `image` field of `fluentd-ds` container of `fluentd-ds` DaemonSet
|
|
||||||
in
|
|
||||||
[200-fluentd.yaml](https://github.com/knative/serving/blob/master/config/monitoring/logging/elasticsearch/200-fluentd.yaml)
|
|
||||||
with the Fluentd image including the desired Fluentd output plugin. See
|
|
||||||
[here](./fluentd-requirements.md) for the requirements of Flunetd image on
|
|
||||||
Knative.
|
|
||||||
|
|
||||||
### Configure the DaemonSet for log files under /var/log
|
|
||||||
|
|
||||||
The Fluentd DaemonSet can also capture `/var/log` logs from the containers. To
|
|
||||||
enable:
|
|
||||||
|
|
||||||
- Set `logging.enable-var-log-collection` to `true` in
|
|
||||||
[config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml)
|
|
||||||
|
|
||||||
## Deploying
|
|
||||||
|
|
||||||
Operators need to deploy Knative components after the configuring:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# Deploy the configuration for enabling /var/log collection
|
|
||||||
kubectl apply --filename config/config-observability.yaml
|
|
||||||
|
|
||||||
# Deploy the DaemonSet to make configuration for DaemonSet take effect
|
|
||||||
kubectl apply --recursive --filename config/monitoring/100-namespace.yaml \
|
|
||||||
--filename <path-of-fluentd-daemonset-config>
|
|
||||||
```
|
|
||||||
|
|
||||||
In the commands above, replace `<path-of-fluentd-daemonset-config>` with the
|
|
||||||
Fluentd DaemonSet configuration file, e.g.
|
|
||||||
`config/monitoring/logging/stackdriver`.
|
|
||||||
|
|
||||||
**NOTE**: The deployment above will not affect the existing pods. Developers
|
|
||||||
need to redeploy their app to get the newest configuration for `/var/log`
|
|
||||||
collection.
|
|
||||||
|
|
||||||
**NOTE**: Operators sometimes need to deploy extra services as the logging
|
|
||||||
backends. For example, if they desire Elasticsearch&Kibana, they have to deploy
|
|
||||||
the Elasticsearch and Kibana services. Knative provides this sample:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl apply --recursive --filename third_party/config/monitoring/logging/elasticsearch
|
|
||||||
```
|
|
||||||
|
|
||||||
See [here](./installing-logging-metrics-traces.md) for deploying the whole
|
|
||||||
Knative monitoring components.
|
|
||||||
|
|
||||||
## Uninstalling
|
|
||||||
|
|
||||||
To uninstall a logging plugin, run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
kubectl delete --recursive --filename <path-of-fluentd-daemonset-config>
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue