mirror of https://github.com/knative/docs.git
Removing references to old monitoring stack (#2946)
* Removing references to old monitoring stack * removed mentions, deleted files related * remove telemetry go * removed mention from landing pages * re-add traces file * add back traces file * readd lines
This commit is contained in:
Ashleigh Brennan
GitHub
parent
ec3b20653d
commit
15153b3e46
|
|
@ -39,15 +39,6 @@ Follow the links below to learn more about Knative.
|
|||
- [All samples for serving](./serving/samples/)
|
||||
- [All samples for eventing](./eventing/samples/)
|
||||
|
||||
### Logging and metrics
|
||||
|
||||
- [Installing logging, metrics and traces](./serving/installing-logging-metrics-traces.md)
|
||||
- [Accessing logs](./serving/accessing-logs.md)
|
||||
- [Accessing metrics](./serving/accessing-metrics.md)
|
||||
- [Accessing traces](./serving/accessing-traces.md)
|
||||
- [Setting up a logging plugin](./serving/setting-up-a-logging-plugin.md)
|
||||
|
||||
### Debugging
|
||||
|
||||
- [Debugging application issues](./serving/debugging-application-issues.md)
|
||||
- [Debugging performance issues](./serving/debugging-performance-issues.md)
|
||||
|
|
|
|||
|
|
@ -13,16 +13,6 @@ visualize and trace your requests.
|
|||
|
||||
You must have a Knative cluster running with the Eventing component installed. [Learn more](../install/README.md)
|
||||
|
||||
## Installing observability plugins
|
||||
|
||||
Knative Eventing uses the same tracing plugin as Knative Serving. See the
|
||||
[Tracing installation instructions](./../serving/installing-logging-metrics-traces.md#end-to-end-request-tracing)
|
||||
in the Knative Serving section for details. Note that you do not need to install the
|
||||
Knative Serving component itself.
|
||||
|
||||
To enable request tracing in Knative Eventing,
|
||||
you must install Elasticsearch and either the Zipkin or Jaeger plugins.
|
||||
|
||||
## Configuring tracing
|
||||
|
||||
With the exception of importers, the Knative Eventing tracing is configured through the
|
||||
|
|
@ -30,7 +20,7 @@ With the exception of importers, the Knative Eventing tracing is configured thro
|
|||
|
||||
Most importers do _not_ use the ConfigMap and instead, use a static 1% sampling rate.
|
||||
|
||||
You can use the `config-tracing` ConfigMap to configure the following Eventing subcomponents:
|
||||
You can use the `config-tracing` ConfigMap to configure the following Eventing components:
|
||||
- Brokers
|
||||
- Triggers
|
||||
- InMemoryChannel
|
||||
|
|
|
|||
|
|
@ -37,8 +37,7 @@ respective repositories in GitHub.
|
|||
|
||||
Before upgrading, view the status of the pods for the namespaces you plan on
|
||||
upgrading. This allows you to compare the before and after state of your
|
||||
namespace. For example, if you are upgrading Knative Serving, Eventing, and the
|
||||
monitoring plug-in, enter the following commands to see the current state of
|
||||
namespace. For example, if you are upgrading Knative Serving and Eventing, enter the following commands to see the current state of
|
||||
each namespace:
|
||||
|
||||
```bash
|
||||
|
|
@ -49,16 +48,10 @@ kubectl get pods --namespace knative-serving
|
|||
kubectl get pods --namespace knative-eventing
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl get pods --namespace knative-monitoring
|
||||
```
|
||||
|
||||
### Upgrading plug-ins
|
||||
|
||||
If you have a plug-in installed, make sure to upgrade it at the same time as
|
||||
you upgrade your Knative components. For example, if you have the
|
||||
monitoring plug-in installed, upgrade it alongside Knative Serving and Eventing.
|
||||
|
||||
you upgrade your Knative components.
|
||||
|
||||
### Run pre-install tools before upgrade
|
||||
|
||||
|
|
@ -90,13 +83,12 @@ kubectl create --filename {{< artifact repo="serving" file="serving-post-install
|
|||
To upgrade, apply the `.yaml` files for the subsequent minor versions of all
|
||||
your installed Knative components and features, remembering to only
|
||||
upgrade by one minor version at a time. For a cluster running v0.15.2 of the
|
||||
Knative Serving and Eventing components and the monitoring plug-in, the
|
||||
Knative Serving and Eventing components, the
|
||||
following command upgrades the installation to v0.16.0:
|
||||
|
||||
```bash
|
||||
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.16.0/serving-core.yaml \
|
||||
--filename https://github.com/knative/eventing/releases/download/v0.16.0/eventing.yaml \
|
||||
--filename https://github.com/knative/serving/releases/download/v0.16.0/monitoring.yaml
|
||||
```
|
||||
|
||||
### Run post-install tools after the upgrade
|
||||
|
|
@ -111,11 +103,9 @@ kubectl apply --filename {{< artifact repo="eventing" file="eventing-post-instal
|
|||
|
||||
## Verifying the upgrade
|
||||
|
||||
To confirm that your components and plugins have successfully upgraded, view the
|
||||
status of their pods in the relevant namespaces. All pods will restart during
|
||||
the upgrade and their age will reset. If you upgraded Knative Serving, Eventing,
|
||||
and the monitoring plug-in, enter the following commands to get information
|
||||
about the pods for each namespace:
|
||||
To confirm that your components and plugins have successfully upgraded, view the status of their pods in the relevant namespaces.
|
||||
All pods will restart during the upgrade and their age will reset.
|
||||
If you upgraded Knative Serving and Eventing, enter the following commands to get information about the pods for each namespace:
|
||||
|
||||
```bash
|
||||
kubectl get pods --namespace knative-serving
|
||||
|
|
@ -125,10 +115,6 @@ kubectl get pods --namespace knative-serving
|
|||
kubectl get pods --namespace knative-eventing
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl get pods --namespace knative-monitoring
|
||||
```
|
||||
|
||||
These commands return something similar to:
|
||||
|
||||
```bash
|
||||
|
|
@ -150,27 +136,7 @@ imc-dispatcher-f59b7c57-q9xcl 1/1 Running 0 80s
|
|||
sources-controller-8596684d7b-jxkmd 1/1 Running 0 83s
|
||||
```
|
||||
|
||||
```bash
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
elasticsearch-logging-0 1/1 Running 0 117s
|
||||
elasticsearch-logging-1 1/1 Running 0 83s
|
||||
fluentd-ds-dqzr7 1/1 Running 0 116s
|
||||
fluentd-ds-dspjc 1/1 Running 0 116s
|
||||
fluentd-ds-x9xg7 1/1 Running 0 116s
|
||||
grafana-59568f8f48-bz2ll 1/1 Running 0 111s
|
||||
kibana-logging-b5d75f556-pwzkg 1/1 Running 0 116s
|
||||
kube-state-metrics-5cb5c6986b-qp6pw 1/1 Running 0 116s
|
||||
node-exporter-bgtsb 2/2 Running 0 112s
|
||||
node-exporter-cqrqv 2/2 Running 0 112s
|
||||
node-exporter-xwv7f 2/2 Running 0 112s
|
||||
prometheus-system-0 1/1 Running 0 110s
|
||||
prometheus-system-1 1/1 Running 0 110s
|
||||
```
|
||||
If the age of all your pods has been reset and all pods are up and running, the upgrade was completed successfully.
|
||||
You might notice a status of `Terminating` for the old pods as they are cleaned up.
|
||||
|
||||
If the age of all your pods has been reset and all pods are up and running, the
|
||||
upgrade was completed successfully. You might notice a status of "Terminating"
|
||||
for the old pods as they are cleaned up.
|
||||
|
||||
If necessary, repeat the upgrade process until you reach your desired minor
|
||||
version number.
|
||||
If necessary, repeat the upgrade process until you reach your desired minor version number.
|
||||
|
|
|
|||
|
|
@ -59,18 +59,9 @@ in the Knative Serving repository.
|
|||
|
||||
- [Knative Serving code samples](./samples/README.md)
|
||||
|
||||
## Setting up Logging and Metrics
|
||||
|
||||
- [Installing Logging, Metrics and Traces](./installing-logging-metrics-traces.md)
|
||||
- [Accessing Logs](./accessing-logs.md)
|
||||
- [Accessing Metrics](./accessing-metrics.md)
|
||||
- [Accessing Traces](./accessing-traces.md)
|
||||
- [Setting up a logging plugin](./setting-up-a-logging-plugin.md)
|
||||
|
||||
## Debugging Knative Serving issues
|
||||
|
||||
- [Debugging Application Issues](./debugging-application-issues.md)
|
||||
- [Debugging Performance Issues](./debugging-performance-issues.md)
|
||||
|
||||
## Configuration and Networking
|
||||
|
||||
|
|
|
|||
|
|
@ -1,120 +0,0 @@
|
|||
---
|
||||
title: "Accessing logs"
|
||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
||||
weight: 5
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
If you have not yet installed the logging and monitoring components, go through
|
||||
the [installation instructions](./installing-logging-metrics-traces.md) to set
|
||||
up the necessary components first.
|
||||
|
||||
## Kibana and Elasticsearch
|
||||
|
||||
- To open the Kibana UI (the visualization tool for
|
||||
[Elasticsearch](https://info.elastic.co)), start a local proxy with the
|
||||
following command:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
This command starts a local proxy of Kibana on port 8001. For security
|
||||
reasons, the Kibana UI is exposed only within the cluster.
|
||||
|
||||
- Navigate to the
|
||||
[Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana).
|
||||
_It might take a couple of minutes for the proxy to work_.
|
||||
|
||||
The Discover tab of the Kibana UI looks like this:
|
||||
|
||||
|
||||
|
||||
You can change the time frame of logs Kibana displays in the upper right
|
||||
corner of the screen. The main search bar is across the top of the Discover
|
||||
page.
|
||||
|
||||
- As more logs are ingested, new fields will be discovered. To have them
|
||||
indexed, go to "Management" > "Index Patterns" > Refresh button (on top
|
||||
right) > "Refresh fields".
|
||||
|
||||
<!-- TODO: create a video walkthrough of the Kibana UI -->
|
||||
|
||||
### Accessing stdout/stderr logs
|
||||
|
||||
To find the logs sent to `stdout/stderr` from your application in the Kibana UI:
|
||||
|
||||
1. Click `Discover` on the left side bar.
|
||||
1. Choose `logstash-*` index pattern on the left top.
|
||||
1. Input `tag: kubernetes*` in the top search bar then search.
|
||||
|
||||
### Accessing request logs
|
||||
|
||||
To access the request logs (if enabled), enter the following search in Kibana:
|
||||
|
||||
```text
|
||||
_exists_:"httpRequest.requestUrl"
|
||||
```
|
||||
|
||||
Request logs contain customized details about requests served by the revision.
|
||||
Below is a sample request log:
|
||||
|
||||
```text
|
||||
@timestamp July 10th 2018, 10:09:28.000
|
||||
kubernetes.labels.serving_knative_dev/configuration helloworld-go
|
||||
kubernetes.labels.serving_knative_dev/revision helloworld-go-6vf4x
|
||||
kubernetes.labels.serving_knative_dev/service helloworld-go
|
||||
httpRequest.protocol HTTP/1.1
|
||||
httpRequest.referer
|
||||
httpRequest.remoteIp 10.32.0.2:46866
|
||||
httpRequest.requestMethod GET
|
||||
httpRequest.requestSize 0
|
||||
httpRequest.requestUrl /
|
||||
httpRequest.responseSize 20
|
||||
httpRequest.serverIp 10.32.1.36
|
||||
httpRequest.status 200
|
||||
httpRequest.userAgent curl/7.60.0
|
||||
traceId 0def9abf835ad90e9d824f7a492e2dcb
|
||||
```
|
||||
|
||||
### Accessing configuration and revision logs
|
||||
|
||||
To access the logs for a configuration:
|
||||
|
||||
- Find the configuration's name with the following command:
|
||||
|
||||
```
|
||||
kubectl get configurations
|
||||
```
|
||||
|
||||
- Replace `<CONFIGURATION_NAME>` and enter the following search query in Kibana:
|
||||
|
||||
```
|
||||
kubernetes.labels.serving_knative_dev\/configuration: <CONFIGURATION_NAME>
|
||||
```
|
||||
|
||||
To access logs for a revision:
|
||||
|
||||
- Find the revision's name with the following command:
|
||||
|
||||
```
|
||||
kubectl get revisions
|
||||
```
|
||||
|
||||
- Replace `<REVISION_NAME>` and enter the following search query in Kibana:
|
||||
|
||||
```
|
||||
kubernetes.labels.serving_knative_dev\/revision: <REVISION_NAME>
|
||||
```
|
||||
|
||||
### Accessing end to end request traces
|
||||
|
||||
See [Accessing Traces](./accessing-traces.md) page for details.
|
||||
|
||||
## Stackdriver
|
||||
|
||||
Go to the
|
||||
[GCP Console logging page](https://console.cloud.google.com/logs/viewer) for
|
||||
your GCP project, which stores your logs via Stackdriver.
|
||||
|
||||
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
---
|
||||
title: "Accessing metrics"
|
||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
||||
weight: 10
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
You access metrics through the [Grafana](https://grafana.com/) UI. Grafana is
|
||||
the visualization tool for [Prometheus](https://prometheus.io/).
|
||||
|
||||
## Grafana
|
||||
|
||||
1. To open Grafana, enter the following command:
|
||||
|
||||
```shell
|
||||
kubectl port-forward --namespace knative-monitoring \
|
||||
$(kubectl get pods --namespace knative-monitoring \
|
||||
--selector=app=grafana --output=jsonpath="{.items..metadata.name}") \
|
||||
3000
|
||||
```
|
||||
|
||||
- This starts a local proxy of Grafana on port 3000. For security reasons, the
|
||||
Grafana UI is exposed only within the cluster.
|
||||
|
||||
2. Navigate to the Grafana UI at [http://localhost:3000](http://localhost:3000).
|
||||
|
||||
3. Select the **Home** button on the top of the page to see the list of
|
||||
pre-installed dashboards (screenshot below):
|
||||
|
||||
|
||||
The following dashboards are pre-installed with Knative Serving:
|
||||
|
||||
- **Revision HTTP Requests:** HTTP request count, latency, and size metrics per
|
||||
revision and per configuration
|
||||
- **Nodes:** CPU, memory, network, and disk metrics at node level
|
||||
- **Pods:** CPU, memory, and network metrics at pod level
|
||||
- **Deployment:** CPU, memory, and network metrics aggregated at deployment
|
||||
level
|
||||
- **Istio, Mixer and Pilot:** Detailed Istio mesh, Mixer, and Pilot metrics
|
||||
- **Kubernetes:** Dashboards giving insights into cluster health, deployments,
|
||||
and capacity usage
|
||||
|
||||
4. Set up an administrator account to modify or add dashboards by signing in
|
||||
with username: `admin` and password: `admin`.
|
||||
|
||||
- Before you expose the Grafana UI outside the cluster, make sure to change the
|
||||
password. You will be prompted to set a password on first login, and it can
|
||||
later be changed at
|
||||
[http://localhost:3000/org/users](http://localhost:3000/org/users).
|
||||
|
||||
## Prometheus
|
||||
|
||||
Although Grafana provides a richer set of graphing functionality, Prometheus can
|
||||
still be useful for debugging.
|
||||
|
||||
1. To open Prometheus, enter the following command:
|
||||
|
||||
```shell
|
||||
kubectl port-forward -n knative-monitoring \
|
||||
$(kubectl get pods -n knative-monitoring \
|
||||
--selector=app=prometheus --output=jsonpath="{.items[0].metadata.name}") \
|
||||
9090
|
||||
```
|
||||
|
||||
- This starts a local proxy of Prometheus on port 9090. For security reasons,
|
||||
the Prometheus UI is exposed only within the cluster.
|
||||
|
||||
2. Navigate to the Prometheus UI at
|
||||
[http://localhost:9090](http://localhost:9090)
|
||||
|
||||
### Metrics Troubleshooting
|
||||
|
||||
You can use the Prometheus web UI to troubleshoot publishing and service
|
||||
discovery issues for metrics.
|
||||
|
||||
- To see the targets that are being scraped, go to Status -> Targets
|
||||
- To see what Prometheus service discovery is picking up vs. dropping, go to
|
||||
Status -> Service Discovery
|
||||
- To see a specific metric you can search for in the in search box on the top of
|
||||
the page
|
||||
|
||||
|
||||
|
|
@ -9,10 +9,6 @@ Depending on the request tracing tool that you have installed on your Knative
|
|||
Serving cluster, see the corresponding section for details about how to
|
||||
visualize and trace your requests.
|
||||
|
||||
If you have not yet installed the logging and monitoring components, go through
|
||||
the [installation instructions](./installing-logging-metrics-traces.md) to set
|
||||
up the necessary components.
|
||||
|
||||
## Configuring Traces
|
||||
|
||||
You can update the configuration file for tracing in [config-tracing.yaml](https://github.com/knative/serving/blob/master/config/config-tracing.yaml).
|
||||
|
|
|
|||
|
|
@ -4,8 +4,6 @@ A demonstration of the autoscaling capabilities of a Knative Serving Revision.
|
|||
|
||||
1. A Kubernetes cluster with [Knative Serving](../../../install/README.md)
|
||||
installed.
|
||||
1. A [metrics installation](../../installing-logging-metrics-traces.md) for
|
||||
viewing scaling graphs (optional).
|
||||
1. The `hey` load generator installed (`go get -u github.com/rakyll/hey`).
|
||||
1. Clone this repository, and move into the sample directory:
|
||||
|
||||
|
|
@ -207,18 +205,6 @@ autoscaler classes built into Knative:
|
|||
View the [Kubecon Demo](https://youtu.be/OPSIPr-Cybs) of Knative autoscaler
|
||||
customization (32 minutes).
|
||||
|
||||
### Dashboards
|
||||
|
||||
View the Knative Serving Scaling and Request dashboards (if configured).
|
||||
|
||||
```
|
||||
kubectl port-forward --namespace knative-monitoring $(kubectl get pods --namespace knative-monitoring --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Other Experiments
|
||||
|
||||
1. Send 60 seconds of traffic maintaining 100 concurrent requests.
|
||||
|
|
|
|||
|
|
@ -27,11 +27,6 @@ for: "STDIN": Internal error occurred: admission webhook "webhook.knative.dev" d
|
|||
ERROR: Non-zero return code '1' from command: Process exited with status 1
|
||||
```
|
||||
|
||||
## Check application logs
|
||||
|
||||
Knative Serving provides default out-of-the-box logs for your application.
|
||||
Access your application logs using [Accessing Logs](./accessing-logs.md) page.
|
||||
|
||||
## Check Route status
|
||||
|
||||
Run the following command to get the `status` of the `Route` object with which
|
||||
|
|
@ -169,6 +164,3 @@ Choose one and use the following command to see detailed information for its
|
|||
kubectl get pod <pod-name> --output yaml
|
||||
|
||||
```
|
||||
|
||||
If you see issues with "user-container" container in the containerStatuses,
|
||||
check your application logs as described below.
|
||||
|
|
|
|||
|
|
@ -1,123 +0,0 @@
|
|||
---
|
||||
title: "Investigating performance issues"
|
||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
||||
weight: 30
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
You deployed your application or function to Knative Serving but its performance
|
||||
doesn't meet your expectations. Knative Serving provides various dashboards and
|
||||
tools to help investigate such issues. This document reviews these dashboards
|
||||
and tools.
|
||||
|
||||
## Request metrics
|
||||
|
||||
Start your investigation with the "Revision - HTTP Requests" dashboard.
|
||||
|
||||
1. To open this dashboard, open the Grafana UI as described in
|
||||
[Accessing Metrics](./accessing-metrics.md) and navigate to "Knative
|
||||
Serving - Revision HTTP Requests".
|
||||
|
||||
1. Select your configuration and revision from the menu on top left of the
|
||||
page. You will see a page like this:
|
||||
|
||||
|
||||
|
||||
This dashboard gives visibility into the following for each revision:
|
||||
|
||||
- Request volume
|
||||
- Request volume per HTTP response code
|
||||
- Response time
|
||||
- Response time per HTTP response code
|
||||
- Request and response sizes
|
||||
|
||||
This dashboard can show traffic volume or latency discrepancies between
|
||||
different revisions. If, for example, a revision's latency is higher than others
|
||||
revisions, then focus your investigation on the offending revision through the
|
||||
rest of this guide.
|
||||
|
||||
## Request traces
|
||||
|
||||
Next, look into request traces to find out where the time is spent for a single
|
||||
request.
|
||||
|
||||
1. To access request traces, open the Zipkin UI as described in
|
||||
[Accessing Traces](./accessing-traces.md).
|
||||
|
||||
1. Select your revision from the "Service Name" dropdown, and then click the
|
||||
"Find Traces" button. You'll get a view that looks like this:
|
||||
|
||||
|
||||
|
||||
In this example, you can see that the request spent most of its time in the
|
||||
[span](https://github.com/opentracing/specification/blob/master/specification.md#the-opentracing-data-model)
|
||||
right before the last, so focus your investigation on that specific span.
|
||||
|
||||
1. Click that span to see a view like the following:
|
||||
|
||||
|
||||
|
||||
This view shows detailed information about the specific span, such as the
|
||||
micro service or external URL that was called. In this example, the call to
|
||||
a Grafana URL is taking the most time. Focus your investigation on why that
|
||||
URL is taking that long.
|
||||
|
||||
## Autoscaler metrics
|
||||
|
||||
If request metrics or traces do not show any obvious hot spots, or if they show
|
||||
that most of the time is spent in your own code, look at autoscaler metrics
|
||||
next.
|
||||
|
||||
1. To open the autoscaler dashboard, open Grafana UI and select "Knative
|
||||
Serving - Autoscaler" dashboard, which looks like this:
|
||||
|
||||
|
||||
|
||||
This view shows 4 key metrics from the Knative Serving autoscaler:
|
||||
|
||||
- Actual pod count: # of pods that are running a given revision
|
||||
- Desired pod count: # of pods that autoscaler thinks should serve the revision
|
||||
- Requested pod count: # of pods that the autoscaler requested from Kubernetes
|
||||
- Panic mode: If 0, the autoscaler is operating in
|
||||
[stable mode](https://github.com/knative/serving/blob/master/docs/scaling/DEVELOPMENT.md#stable-mode).
|
||||
If 1, the autoscaler is operating in
|
||||
[panic mode](https://github.com/knative/serving/blob/master/docs/scaling/DEVELOPMENT.md#panic-mode).
|
||||
|
||||
A large gap between the actual pod count and the requested pod count indicates
|
||||
that the Kubernetes cluster is unable to keep up allocating new resources fast
|
||||
enough, or that the Kubernetes cluster is out of requested resources.
|
||||
|
||||
A large gap between the requested pod count and the desired pod count indicates
|
||||
that the Knative Serving autoscaler is unable to communicate with the Kubernetes
|
||||
API to make the request.
|
||||
|
||||
In the preceding example, the autoscaler requested 18 pods to optimally serve
|
||||
the traffic but was only granted 8 pods because the cluster is out of resources.
|
||||
|
||||
## CPU and memory usage
|
||||
|
||||
You can access total CPU and memory usage of your revision from the "Knative
|
||||
Serving - Revision CPU and Memory Usage" dashboard, which looks like this:
|
||||
|
||||
|
||||
|
||||
The first chart shows rate of the CPU usage across all pods serving the
|
||||
revision. The second chart shows total memory consumed across all pods serving
|
||||
the revision. Both of these metrics are further divided into per container
|
||||
usage.
|
||||
|
||||
- user-container: This container runs the user code (application, function, or
|
||||
container).
|
||||
- [istio-proxy](https://github.com/istio/proxy): Sidecar container to form an
|
||||
[Istio](https://istio.io/docs/concepts/what-is-istio/) mesh.
|
||||
- queue-proxy: Knative Serving owned sidecar container to enforce request
|
||||
concurrency limits.
|
||||
- autoscaler: Knative Serving owned sidecar container to provide autoscaling for
|
||||
the revision.
|
||||
- fluentd-proxy: Sidecar container to collect logs from /var/log.
|
||||
|
||||
## Profiling
|
||||
|
||||
...To be filled...
|
||||
|
||||
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
---
|
||||
title: "Fluentd container image requirements"
|
||||
linkTitle: "Fluentd requirements"
|
||||
weight: 10
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
Knative Serving uses a [Fluentd](https://www.fluentd.org/) docker image to
|
||||
collect logs. Operators can customize their own Fluentd docker image and
|
||||
configuration to define logging output.
|
||||
|
||||
## Requirements
|
||||
|
||||
Knative requires the customized Fluentd docker image with the following plugins
|
||||
installed:
|
||||
|
||||
- [fluentd](https://github.com/fluent/fluentd) >= v0.14.0
|
||||
- [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) >=
|
||||
1.0.0 AND < 2.1.0: To enrich log entries with Kubernetes metadata.
|
||||
- [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions) >=
|
||||
0.0.9: To combine multi-line exception stack traces logs into one log entry.
|
||||
- [fluent-plugin-multi-format-parser](https://github.com/repeatedly/fluent-plugin-multi-format-parser) >=
|
||||
1.0.0: To detect log format as Json or plain text.
|
||||
|
||||
## Sample images
|
||||
|
||||
Operators can use any Docker image which meets the requirements above and
|
||||
includes the desired output plugin. Two examples below:
|
||||
|
||||
### Send logs to Elasticsearch
|
||||
|
||||
Operators can use
|
||||
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image)
|
||||
which includes
|
||||
[fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch)
|
||||
that allows sending logs to a Elasticsearch service.
|
||||
|
||||
### Send logs to Stackdriver
|
||||
|
||||
This sample [Dockerfile](./stackdriver/Dockerfile) is based on
|
||||
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch).
|
||||
It additionally adds one more plugin -
|
||||
[fluent-plugin-google-cloud](https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud)
|
||||
which allows sending logs to Stackdriver.
|
||||
|
||||
Operators can build this image and push it to a container registry which their
|
||||
Kubernetes cluster has access to. See
|
||||
[Setting Up A Logging Plugin](./setting-up-a-logging-plugin.md) for details.
|
||||
|
||||
> **NOTE**: Operators must add the credentials file the Stackdriver agent needs
|
||||
> to the Docker image if their Knative Serving is not built on a Google Cloud
|
||||
> Platform-based cluster or if they want to send logs to another Google Cloud
|
||||
> Platform project. See
|
||||
> [here](https://cloud.google.com/logging/docs/agent/authorization) for more
|
||||
> information.
|
||||
|
|
@ -1,335 +0,0 @@
|
|||
---
|
||||
title: "Installing logging, metrics, and traces"
|
||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
||||
weight: 40
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
If you followed one of the
|
||||
[comprehensive install guides](../install/README.md#install-guides) or you
|
||||
performed a custom installation and included the `monitoring.yaml` file in your
|
||||
installation, all of the observability features are already installed and you
|
||||
can skip down to the
|
||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
||||
|
||||
If you have not yet installed any observability plugins, continue to the next
|
||||
sections to do so now.
|
||||
|
||||
## Metrics
|
||||
|
||||
1. Run the following command and follow the instructions below to enable request
|
||||
metrics if they are wanted:
|
||||
|
||||
```
|
||||
kubectl edit cm -n knative-serving config-observability
|
||||
```
|
||||
|
||||
Add `metrics.request-metrics-backend-destination: prometheus` to `data`
|
||||
field. You can find detailed information in `data._example` field in the
|
||||
`ConfigMap` you are editing.
|
||||
|
||||
1. Run the following command to install Prometheus and Grafana:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-metrics-prometheus.yaml" >}}
|
||||
```
|
||||
|
||||
1. Ensure that the `grafana-*`, `kibana-logging-*`, `kube-state-metrics-*`,
|
||||
`node-exporter-*` and `prometheus-system-*` pods all report a `Running`
|
||||
status:
|
||||
|
||||
```shell
|
||||
kubectl get pods --namespace knative-monitoring --watch
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
grafana-798cf569ff-v4q74 1/1 Running 0 2d
|
||||
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
|
||||
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
|
||||
node-exporter-cr6bh 2/2 Running 0 2d
|
||||
node-exporter-mf6k7 2/2 Running 0 2d
|
||||
node-exporter-rhzr7 2/2 Running 0 2d
|
||||
prometheus-system-0 1/1 Running 0 2d
|
||||
prometheus-system-1 1/1 Running 0 2d
|
||||
```
|
||||
|
||||
Tip: Hit CTRL+C to exit watch mode.
|
||||
|
||||
[Accessing Metrics](./accessing-metrics.md) for more information about metrics
|
||||
in Knative.
|
||||
|
||||
## Logs
|
||||
|
||||
### Enable Request Logs
|
||||
|
||||
Run the following command and follow the instructions below to enable request
|
||||
logs if they are wanted:
|
||||
|
||||
```
|
||||
kubectl edit cm -n knative-serving config-observability
|
||||
```
|
||||
|
||||
Copy `logging.request-log-template` from `data._example` field to`data` field in
|
||||
the `ConfigMap` you are editing. You can find detailed information in
|
||||
`data._example` field to customize the request log format.
|
||||
|
||||
### Choose One Logging Backend
|
||||
|
||||
Knative offers three different setups for collecting logs. Choose one to
|
||||
install:
|
||||
|
||||
1. [Elasticsearch and Kibana](#elasticsearch-and-kibana)
|
||||
1. [Stackdriver](#stackdriver)
|
||||
1. [Custom logging plugin](./setting-up-a-logging-plugin.md)
|
||||
|
||||
#### Elasticsearch and Kibana
|
||||
|
||||
1. Run the following command to install an ELK stack:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-logs-elasticsearch.yaml" >}}
|
||||
```
|
||||
|
||||
1. Ensure that the `elasticsearch-logging-*`, `fluentd-ds-*`, and
|
||||
`kibana-logging-*` pods all report a `Running` status:
|
||||
|
||||
```shell
|
||||
kubectl get pods --namespace knative-monitoring --watch
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
elasticsearch-logging-0 1/1 Running 0 2d
|
||||
elasticsearch-logging-1 1/1 Running 0 2d
|
||||
fluentd-ds-5kc85 1/1 Running 0 2d
|
||||
fluentd-ds-vhrcq 1/1 Running 0 2d
|
||||
fluentd-ds-xghk9 1/1 Running 0 2d
|
||||
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
|
||||
```
|
||||
|
||||
Tip: Hit CTRL+C to exit watch mode.
|
||||
|
||||
1. Verify that each of your nodes have the
|
||||
`beta.kubernetes.io/fluentd-ds-ready=true` label:
|
||||
|
||||
```shell
|
||||
kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
|
||||
```
|
||||
|
||||
1. If you receive the `No Resources Found` response:
|
||||
|
||||
1. Run the following command to ensure that the Fluentd DaemonSet runs on all
|
||||
your nodes:
|
||||
|
||||
```shell
|
||||
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
|
||||
```
|
||||
|
||||
1. Run the following command to ensure that the `fluentd-ds` daemonset is
|
||||
ready on at least one node:
|
||||
|
||||
```shell
|
||||
kubectl get daemonset fluentd-ds --namespace knative-monitoring --watch
|
||||
```
|
||||
|
||||
Tip: Hit CTRL+C to exit watch mode.
|
||||
|
||||
1. When the installation is complete and all the resources are running, you can
|
||||
continue to the next section and begin creating your Elasticsearch indices.
|
||||
|
||||
##### Create Elasticsearch Indices
|
||||
|
||||
To visualize logs with Kibana, you need to set which Elasticsearch indices to
|
||||
explore.
|
||||
|
||||
- To open the Kibana UI (the visualization tool for
|
||||
[Elasticsearch](https://info.elastic.co)), you must start a local proxy by
|
||||
running the following command:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
This command starts a local proxy of Kibana on port 8001. For security
|
||||
reasons, the Kibana UI is exposed only within the cluster.
|
||||
|
||||
- Navigate to the
|
||||
[Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana).
|
||||
_It might take a couple of minutes for the proxy to work_.
|
||||
|
||||
- Within the "Configure an index pattern" page, enter `logstash-*` to
|
||||
`Index pattern` and select `@timestamp` from `Time Filter field name` and
|
||||
click on `Create` button.
|
||||
|
||||
|
||||
|
||||
See [Accessing Logs](./accessing-logs.md) for more information about logs in
|
||||
Knative.
|
||||
|
||||
#### Stackdriver
|
||||
|
||||
To configure and setup monitoring:
|
||||
|
||||
1. Clone the Knative Serving repository:
|
||||
|
||||
```shell
|
||||
git clone -b {{< version >}} https://github.com/knative/serving knative-serving
|
||||
cd knative-serving
|
||||
```
|
||||
|
||||
1. Choose a container image that meets the
|
||||
[Fluentd image requirements](./fluentd-requirements.md#requirements). For
|
||||
example, you can use a public image. Or you can create a custom one and
|
||||
upload the image to a container registry which your cluster has read access
|
||||
to.
|
||||
|
||||
You must configure and build your own Fluentd image if either of the
|
||||
following are true:
|
||||
|
||||
- Your Knative Serving component is not hosted on a Google Cloud Platform
|
||||
(GCP) based cluster.
|
||||
- You want to send logs to another GCP project.
|
||||
|
||||
1. Follow the instructions in
|
||||
[Setting up a logging plugin](./setting-up-a-logging-plugin.md#Configuring)
|
||||
to configure the stackdriver components settings.
|
||||
|
||||
1. Install Knative Stackdriver components by running the following command from
|
||||
the root directory of [knative/serving](https://github.com/knative/serving)
|
||||
repository:
|
||||
|
||||
```shell
|
||||
kubectl apply --recursive --filename config/monitoring/100-namespace.yaml \
|
||||
--filename config/monitoring/logging/stackdriver
|
||||
```
|
||||
|
||||
1. Ensure that the `fluentd-ds-*` pods all report a `Running` status:
|
||||
|
||||
```shell
|
||||
kubectl get pods --namespace knative-monitoring --watch
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
fluentd-ds-5kc85 1/1 Running 0 2d
|
||||
fluentd-ds-vhrcq 1/1 Running 0 2d
|
||||
fluentd-ds-xghk9 1/1 Running 0 2d
|
||||
```
|
||||
|
||||
Tip: Hit CTRL+C to exit watch mode.
|
||||
|
||||
1. Verify that each of your nodes have the
|
||||
`beta.kubernetes.io/fluentd-ds-ready=true` label:
|
||||
|
||||
```shell
|
||||
kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
|
||||
```
|
||||
|
||||
1. If you receive the `No Resources Found` response:
|
||||
|
||||
1. Run the following command to ensure that the Fluentd DaemonSet runs on
|
||||
all your nodes:
|
||||
|
||||
```shell
|
||||
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
|
||||
```
|
||||
|
||||
1. Run the following command to ensure that the `fluentd-ds` daemonset is
|
||||
ready on at least one node:
|
||||
|
||||
```shell
|
||||
kubectl get daemonset fluentd-ds --namespace knative-monitoring
|
||||
```
|
||||
|
||||
See [Accessing Logs](./accessing-logs.md) for more information about logs in
|
||||
Knative.
|
||||
|
||||
## End to end request tracing
|
||||
|
||||
You can choose from one of the following options to enable request tracing in
|
||||
your Knative Serving cluster.
|
||||
|
||||
**Important**: Your cluster supports only a single request trace tool. If you
|
||||
want to replace a currently installed request trace tool, you must first
|
||||
uninstall that tool before installing the new tool.
|
||||
|
||||
### Zipkin
|
||||
|
||||
1. Install support for Zipkin:
|
||||
|
||||
- If Elasticsearch is not installed or if you don't want to persist end to
|
||||
end traces, run:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-zipkin-in-mem.yaml" >}}
|
||||
```
|
||||
|
||||
- If Elasticsearch is installed and you want to persist end to end traces,
|
||||
first run:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-zipkin.yaml" >}}
|
||||
```
|
||||
|
||||
1. Create an Elasticsearch index for end to end traces:
|
||||
|
||||
1. Open Kibana UI as described in
|
||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
||||
|
||||
1. Select `Create Index Pattern` button on top left of the page. Enter
|
||||
`zipkin*` to `Index pattern` and select `timestamp_millis` from
|
||||
`Time Filter field name` and click on `Create` button.
|
||||
|
||||
Visit [Accessing Traces](./accessing-traces.md) for more information on end to
|
||||
end traces.
|
||||
|
||||
### Jaeger
|
||||
|
||||
1. Install the Jaeger operator. Use the instructions in
|
||||
jaegertracing/jaeger-operator repository and follow only the steps in the
|
||||
[Installing the operator](https://github.com/jaegertracing/jaeger-operator#installing-the-operator)
|
||||
section.
|
||||
|
||||
1. Install support for Jaeger:
|
||||
|
||||
- If Elasticsearch is not installed or if you don't want to persist end to
|
||||
end traces, run:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-jaeger-in-mem.yaml" >}}
|
||||
```
|
||||
|
||||
- If Elasticsearch is installed and you want to persist end to end traces,
|
||||
first run:
|
||||
|
||||
```shell
|
||||
kubectl apply --filename {{< artifact repo="serving" file="monitoring-tracing-jaeger.yaml" >}}
|
||||
```
|
||||
|
||||
1. Create an Elasticsearch index for end to end traces:
|
||||
|
||||
1. Open Kibana UI as described in
|
||||
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
|
||||
|
||||
1. Select `Create Index Pattern` button on top left of the page. Enter
|
||||
`jaeger*` to `Index pattern` and select `timestamp_millis` from
|
||||
`Time Filter field name` and click on `Create` button.
|
||||
|
||||
Visit [Accessing Traces](./accessing-traces.md) for more information on end to
|
||||
end traces.
|
||||
|
||||
## Learn More
|
||||
|
||||
- Learn more about accessing logs, metrics, and traces:
|
||||
- [Accessing Logs](./accessing-logs.md)
|
||||
- [Accessing Metrics](./accessing-metrics.md)
|
||||
- [Accessing Traces](./accessing-traces.md)
|
||||
|
||||
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
# Copyright 2018 The Knative Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
FROM golang AS builder
|
||||
|
||||
WORKDIR /go/src/github.com/knative/docs/docs/
|
||||
ADD . /go/src/github.com/knative/docs/
|
||||
|
||||
RUN CGO_ENABLED=0 go build ./serving/samples/telemetry-go/
|
||||
|
||||
FROM gcr.io/distroless/base
|
||||
|
||||
EXPOSE 8080
|
||||
COPY --from=builder /go/src/github.com/knative/docs/docs/telemetry-go /sample
|
||||
|
||||
ENTRYPOINT ["/sample"]
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
# The OWNERS file is used by prow to automatically merge approved PRs.
|
||||
|
||||
approvers:
|
||||
- mdemirhan
|
||||
- yanweiguo
|
||||
|
|
@ -1,166 +0,0 @@
|
|||
This sample runs a simple web server that makes calls to other in-cluster
|
||||
services and responds to requests with "Hello World!". The purpose of this
|
||||
sample is to show generating [metrics](../../accessing-metrics.md),
|
||||
[logs](../../accessing-logs.md) and distributed
|
||||
[traces](../../accessing-traces.md). This sample also shows how to create a
|
||||
dedicated Prometheus instance rather than using the default installation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. A Kubernetes cluster with [Knative Serving](../../../install/README.md)
|
||||
installed and DNS configured.
|
||||
2. Check if Knative monitoring components are installed:
|
||||
|
||||
```
|
||||
kubectl get pods --namespace knative-monitoring
|
||||
```
|
||||
|
||||
- If pods aren't found, install
|
||||
[Knative monitoring component](../../installing-logging-metrics-traces.md).
|
||||
|
||||
3. Install
|
||||
[Docker](https://docs.docker.com/get-started/#prepare-your-docker-environment).
|
||||
4. Check out the code:
|
||||
|
||||
```
|
||||
go get -d github.com/knative/docs/docs/serving/samples/telemetry-go
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
Build the application container and publish it to a container registry:
|
||||
|
||||
1. Move into the sample directory:
|
||||
|
||||
```
|
||||
cd $GOPATH/src/github.com/knative/docs
|
||||
```
|
||||
|
||||
2. Set your preferred container registry:
|
||||
|
||||
```
|
||||
export REPO="gcr.io/<YOUR_PROJECT_ID>"
|
||||
```
|
||||
|
||||
This example shows how to use Google Container Registry (GCR). You will need a
|
||||
Google Cloud Project and to enable the
|
||||
[Google Container Registry API](https://console.cloud.google.com/apis/library/containerregistry.googleapis.com).
|
||||
|
||||
3. Use Docker to build your application container:
|
||||
|
||||
```
|
||||
docker build \
|
||||
--tag "${REPO}/docs/serving/samples/telemetry-go" \
|
||||
--file=docs/serving/samples/telemetry-go/Dockerfile .
|
||||
```
|
||||
|
||||
4. Push your container to a container registry:
|
||||
|
||||
```
|
||||
docker push "${REPO}/docs/serving/samples/telemetry-go"
|
||||
```
|
||||
|
||||
5. Replace the image reference path with our published image path in the
|
||||
configuration file (`docs/serving/samples/telemetry-go/sample.yaml`):
|
||||
|
||||
- Manually replace:
|
||||
`image: github.com/knative/docs/docs/serving/samples/telemetry-go` with
|
||||
`image: <YOUR_CONTAINER_REGISTRY>/docs/serving/samples/telemetry-go`
|
||||
|
||||
|
||||
Or
|
||||
|
||||
- Use run this command:
|
||||
|
||||
|
||||
```
|
||||
perl -pi -e "s@github.com/knative/docs@${REPO}@g" docs/serving/samples/telemetry-go/sample.yaml
|
||||
```
|
||||
|
||||
## Deploy the Service
|
||||
|
||||
Deploy this application to Knative Serving:
|
||||
|
||||
```
|
||||
kubectl apply --filename docs/serving/samples/telemetry-go/
|
||||
```
|
||||
|
||||
## Explore the Service
|
||||
|
||||
Inspect the created resources with the `kubectl` commands:
|
||||
|
||||
- View the created Route resource:
|
||||
|
||||
```
|
||||
kubectl get route --output yaml
|
||||
```
|
||||
|
||||
- View the created Configuration resource:
|
||||
|
||||
```
|
||||
kubectl get configurations --output yaml
|
||||
```
|
||||
|
||||
- View the Revision that was created by the Configuration:
|
||||
|
||||
```
|
||||
kubectl get revisions --output yaml
|
||||
```
|
||||
|
||||
## Access the Service
|
||||
|
||||
To access this service via `curl`, you need to determine its ingress address.
|
||||
|
||||
1. To determine if your route is ready:
|
||||
|
||||
```
|
||||
kubectl get route
|
||||
```
|
||||
|
||||
When the route is ready, you'll see the following fields reported as:
|
||||
|
||||
```YAML
|
||||
telemetrysample-route http://telemetrysample-route.default.example.com True
|
||||
```
|
||||
|
||||
1. Make a request to the service to see the `Hello World!` message:
|
||||
|
||||
```
|
||||
curl http://telemetrysample-route.default.example.com
|
||||
```
|
||||
|
||||
1. Make a request to the `/log` endpoint to generate logs to the `stdout` file
|
||||
and generate files under `/var/log` in both `JSON` and plain text formats:
|
||||
|
||||
```
|
||||
curl http://telemetrysample-route.default.example.com/log
|
||||
```
|
||||
|
||||
## Access Logs
|
||||
|
||||
You can access to the logs from Kibana UI - see [Logs](../../accessing-logs.md)
|
||||
for more information.
|
||||
|
||||
## Access per Request Traces
|
||||
|
||||
You can access to per request traces from Zipkin UI - see
|
||||
[Traces](../../accessing-traces.md) for more information.
|
||||
|
||||
## Accessing Custom Metrics
|
||||
|
||||
You can see published metrics using Prometheus UI. To access to the UI, forward
|
||||
the Prometheus server to your machine:
|
||||
|
||||
```
|
||||
kubectl port-forward $(kubectl get pods --selector=app=prometheus-test --output=jsonpath="{.items[0].metadata.name}") 9090
|
||||
```
|
||||
|
||||
Then browse to http://localhost:9090.
|
||||
|
||||
## Clean up
|
||||
|
||||
To clean up the sample service:
|
||||
|
||||
```
|
||||
kubectl delete --filename docs/serving/samples/telemetry-go/
|
||||
```
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
---
|
||||
title: "Observability: Monitoring, logging, and tracing sample - Go"
|
||||
linkTitle: "Observability - Go"
|
||||
weight: 1
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
{{% readfile file="README.md" %}}
|
||||
|
|
@ -1,122 +0,0 @@
|
|||
# Copyright 2018 The Knative Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: prometheus-test
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: prometheus-test
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- nodes
|
||||
- services
|
||||
- endpoints
|
||||
- pods
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- configmaps
|
||||
verbs: ["get"]
|
||||
- nonResourceURLs: ["/metrics"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: prometheus-test
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: prometheus-test
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: prometheus-test
|
||||
namespace: default
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus-test-scrape-config
|
||||
labels:
|
||||
name: prometheus-test-scrape-config
|
||||
namespace: default
|
||||
data:
|
||||
prometheus.yml: |-
|
||||
scrape_configs:
|
||||
- job_name: telemetrysample-test
|
||||
scrape_interval: 3s
|
||||
scrape_timeout: 3s
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
relabel_configs:
|
||||
# Scrape only the the targets matching the following metadata
|
||||
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_serving_knative_dev_configuration, __meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: default;telemetrysample-configuration;http
|
||||
# Rename metadata labels to be reader friendly
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: namespace
|
||||
replacement: $1
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: pod
|
||||
replacement: $1
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: service
|
||||
replacement: $1
|
||||
---
|
||||
apiVersion: apps/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus-test-deployment
|
||||
namespace: default
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus-test
|
||||
spec:
|
||||
serviceAccountName: prometheus-test
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:v2.2.1
|
||||
args:
|
||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||
- "--storage.tsdb.path=/prometheus/"
|
||||
ports:
|
||||
- containerPort: 9090
|
||||
volumeMounts:
|
||||
- name: prometheus-config-volume
|
||||
mountPath: /etc/prometheus/
|
||||
- name: prometheus-storage-volume
|
||||
mountPath: /prometheus/
|
||||
volumes:
|
||||
- name: prometheus-config-volume
|
||||
configMap:
|
||||
defaultMode: 420
|
||||
name: prometheus-test-scrape-config
|
||||
|
||||
- name: prometheus-storage-volume
|
||||
emptyDir: {}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
# Copyright 2018 The Knative Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Route
|
||||
metadata:
|
||||
name: telemetrysample-route
|
||||
namespace: default
|
||||
spec:
|
||||
traffic:
|
||||
- configurationName: telemetrysample-configuration
|
||||
percent: 100
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
# Copyright 2018 The Knative Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: telemetrysample-configuration
|
||||
namespace: default
|
||||
labels:
|
||||
app: telemetrysample
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
knative.dev/type: app
|
||||
spec:
|
||||
containers:
|
||||
- # This is the Go import path for the binary to containerize
|
||||
# and substitute here.
|
||||
image: github.com/knative/docs/docs/serving/samples/telemetry-go
|
||||
|
|
@ -1,269 +0,0 @@
|
|||
/*
|
||||
Copyright 2018 The Knative Authors
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/rand"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
openzipkin "github.com/openzipkin/zipkin-go"
|
||||
httpreporter "github.com/openzipkin/zipkin-go/reporter/http"
|
||||
|
||||
"go.opencensus.io/exporter/prometheus"
|
||||
"go.opencensus.io/exporter/zipkin"
|
||||
"go.opencensus.io/plugin/ochttp"
|
||||
"go.opencensus.io/plugin/ochttp/propagation/b3"
|
||||
"go.opencensus.io/stats"
|
||||
"go.opencensus.io/stats/view"
|
||||
"go.opencensus.io/tag"
|
||||
"go.opencensus.io/trace"
|
||||
)
|
||||
|
||||
var (
|
||||
// Create a measurement to keep track of incoming request counts.
|
||||
// For more information on measurements, see https://pkg.go.dev/go.opencensus.io/stats
|
||||
requestCount = stats.Int64("request_count", "Total number of requests.", stats.UnitNone)
|
||||
|
||||
// Create a measurement to keep track of request durations.
|
||||
requestDuration = stats.Int64("request_duration", "Histogram of the request duration.", stats.UnitMilliseconds)
|
||||
|
||||
// Capture the HTTP response code in a tag so that we can aggregate and visualize
|
||||
// this metric based on different response codes (see count of all 400 vs 200 for example).
|
||||
// For more information on tags, see https://pkg.go.dev/go.opencensus.io/tag
|
||||
requestStatusTagKey tag.Key
|
||||
)
|
||||
|
||||
func main() {
|
||||
flag.Parse()
|
||||
log.SetPrefix("TelemetrySample: ")
|
||||
log.Print("Telemetry sample started.")
|
||||
|
||||
//
|
||||
// Metrics setup
|
||||
//
|
||||
|
||||
// We want our metrics in Prometheus. Create the exporter to do that.
|
||||
promExporter, err := prometheus.NewExporter(prometheus.Options{Namespace: "telemetrysample"})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
view.RegisterExporter(promExporter)
|
||||
|
||||
// Create the tag keys that will be used to add tags to our measurements.
|
||||
requestStatusTagKey, err = tag.NewKey("status")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Create view to see our measurements cumulatively.
|
||||
err = view.Register(
|
||||
&view.View{
|
||||
Description: "Total number of requests.",
|
||||
Measure: requestCount,
|
||||
Aggregation: view.Count(),
|
||||
TagKeys: []tag.Key{requestStatusTagKey},
|
||||
},
|
||||
&view.View{
|
||||
Description: "Histogram of the request duration.",
|
||||
Measure: requestDuration,
|
||||
Aggregation: view.Distribution(10, 25, 50, 100, 250, 500, 1000, 2500),
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatalf("Cannot subscribe to the view: %v", err)
|
||||
}
|
||||
view.SetReportingPeriod(1 * time.Second)
|
||||
|
||||
//
|
||||
// Tracing setup
|
||||
//
|
||||
|
||||
// If your service only calls other Knative Serving revisions, then Zipkin setup below
|
||||
// is not needed. All that you need is to extract the following headers from incoming
|
||||
// requests and copy them to your outgoing requests.
|
||||
// x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context
|
||||
//
|
||||
// For richer instrumentation, or to instrument calls that are made to external services,
|
||||
// you can instrument the code using Zipkin's Go client library as shown below.
|
||||
//
|
||||
// Zipkin is installed in istio-system namespace because istio assumes that zipkin is installed there.
|
||||
// Ideally this value should be config driven, but for demo purposes, we will hardcode it here.
|
||||
// For unit tests, reporter.noopReporter can be used instead of the httpreporter below.
|
||||
reporter := httpreporter.NewReporter("http://zipkin.istio-system.svc.cluster.local:9411/api/v2/spans")
|
||||
defer reporter.Close()
|
||||
localEndpoint, err := openzipkin.NewEndpoint("TelemetrySample", "localhost:8080")
|
||||
if err != nil {
|
||||
log.Fatalf("Unable to create zipkin local endpoint: %+v\n", err)
|
||||
}
|
||||
|
||||
// The OpenCensus exporter wraps the Zipkin reporter
|
||||
zipkinExporter := zipkin.NewExporter(reporter, localEndpoint)
|
||||
trace.RegisterExporter(zipkinExporter)
|
||||
|
||||
// For example purposes, sample every trace.
|
||||
trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})
|
||||
|
||||
// Use an http.RoundTripper that instruments all outgoing requests with stats and tracing.
|
||||
client := &http.Client{Transport: &ochttp.Transport{Propagation: &b3.HTTPFormat{}}}
|
||||
|
||||
// Implements an http.Handler that automatically extracts traces
|
||||
// from the incoming request and writes them back to the response.
|
||||
handler := &ochttp.Handler{Propagation: &b3.HTTPFormat{}, Handler: rootHandler(client)}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("/", handler)
|
||||
mux.Handle("/log", &ochttp.Handler{Propagation: &b3.HTTPFormat{}, Handler: logHandler(client)})
|
||||
// Setup OpenCensus Prometheus exporter to handle scraping for metrics.
|
||||
mux.Handle("/metrics", promExporter)
|
||||
http.ListenAndServe(":8080", mux)
|
||||
}
|
||||
|
||||
var statusCodes = [...]int{
|
||||
http.StatusOK,
|
||||
http.StatusCreated,
|
||||
http.StatusAccepted,
|
||||
http.StatusBadRequest,
|
||||
http.StatusUnauthorized,
|
||||
}
|
||||
|
||||
func rootHandler(client *http.Client) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
// Write the http request headers to the log for demonstration purposes.
|
||||
log.Printf("Request received. Request headers: %v", r.Header)
|
||||
|
||||
// Pick a random return code - this is used for demonstrating metrics & logs
|
||||
// with different responses.
|
||||
status := statusCodes[rand.Intn(len(statusCodes))]
|
||||
|
||||
// Before returning from this function, update requestCount and requestDuration metrics.
|
||||
defer func(start time.Time) {
|
||||
// Create a new context with the metric tags and their values.
|
||||
ctx, err := tag.New(r.Context(), tag.Insert(requestStatusTagKey, fmt.Sprint(status)))
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Increment the request count by one.
|
||||
stats.Record(ctx, requestCount.M(1))
|
||||
|
||||
// Record the request duration.
|
||||
stats.Record(ctx, requestDuration.M(time.Since(start).Nanoseconds()/int64(time.Millisecond)))
|
||||
}(time.Now())
|
||||
|
||||
getWithContext := func(url string) (*http.Response, error) {
|
||||
req, err := http.NewRequest("GET", url, nil)
|
||||
if err != nil {
|
||||
log.Printf("Failed to create a new request: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
// If we don't attach the incoming request's context, we will end up creating
|
||||
// a new trace id (i.e. a new root span) rather than attaching to the incoming
|
||||
// request's trace id. To ensure the continuity of the trace, use incoming
|
||||
// request's context for the outgoing request.
|
||||
req = req.WithContext(r.Context())
|
||||
return client.Do(req)
|
||||
}
|
||||
|
||||
// Simulate a few extra calls to other services to demostrate the distributed tracing capabilities.
|
||||
// In sequence, call three different other services. For each call, we will create a new span
|
||||
// to track that call in the call graph. See http://opentracing.io/documentation/ for more information
|
||||
// on these concepts.
|
||||
res, err := getWithContext("http://prometheus-system-np.knative-monitoring.svc.cluster.local:8080")
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
// Close directly rather than defer to ensure that spans are finished now rather than
|
||||
// end of this function. This applies to the remaining res.Body().Close calls below.
|
||||
res.Body.Close()
|
||||
|
||||
res, err = getWithContext("http://grafana.knative-monitoring.svc.cluster.local:30802")
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
res.Body.Close()
|
||||
|
||||
// Let's call a non-existent URL to demonstrate the failure scenarios in distributed tracing.
|
||||
res, err = getWithContext("http://invalidurl.svc.cluster.local")
|
||||
if err != nil {
|
||||
log.Printf("Request failed: %v", err)
|
||||
} else {
|
||||
res.Body.Close()
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.WriteHeader(status)
|
||||
w.Write([]byte("Hello world!\n"))
|
||||
log.Printf("Request complete. Status: %v", status)
|
||||
}
|
||||
}
|
||||
|
||||
func logHandler(client *http.Client) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
timestamp := time.Now()
|
||||
|
||||
// Send logs to STDOUT
|
||||
msg := "A log in plain text format to STDOUT"
|
||||
fmt.Fprintln(os.Stdout, msg)
|
||||
|
||||
data := map[string]string{
|
||||
"log": "A log in json format to STDOUT",
|
||||
"foo": "bar",
|
||||
"time": timestamp.String(),
|
||||
// Cluster operator can configure which field is used as time key and what
|
||||
// the format is. For example, in config/monitoring/150-elasticsearch/100-fluentd-configmap.yaml,
|
||||
// fluentd-time is the reserved key to tell fluentd the logging time. It
|
||||
// must be in the format of RFC3339Nano, i.e. %Y-%m-%dT%H:%M:%S.%NZ.
|
||||
// Without this, fluentd uses the time when it collect the log as an
|
||||
// event time.
|
||||
"fluentd-time": timestamp.Format(time.RFC3339Nano),
|
||||
}
|
||||
jsonOutput, _ := json.Marshal(data)
|
||||
fmt.Fprintln(os.Stdout, string(jsonOutput))
|
||||
|
||||
// Send logs to /var/log
|
||||
fileName := "/var/log/sample.log"
|
||||
f, err := os.OpenFile(fileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
|
||||
defer f.Close()
|
||||
|
||||
if err == nil {
|
||||
msg = "A log in plain text format to /var/log\n"
|
||||
if _, err := f.WriteString(msg); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to write to %s: %v", fileName, err)
|
||||
}
|
||||
|
||||
data["log"] = "A log in json format to /var/log"
|
||||
jsonOutput, _ := json.Marshal(data)
|
||||
if _, err := f.WriteString(string(jsonOutput) + "\n"); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to write to %s: %v", fileName, err)
|
||||
}
|
||||
f.Sync()
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Failed to create %s: %v", fileName, err)
|
||||
}
|
||||
|
||||
fmt.Fprint(w, "Sending logs done.\n")
|
||||
}
|
||||
}
|
||||
|
|
@ -1,90 +0,0 @@
|
|||
---
|
||||
title: "Setting up a logging plugin"
|
||||
#linkTitle: "OPTIONAL_ALTERNATE_NAV_TITLE"
|
||||
weight: 50
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
Knative allows cluster operators to use different backends for their logging
|
||||
needs. This document describes how to change these settings. Knative currently
|
||||
requires changes in Fluentd configuration files, however we plan on abstracting
|
||||
logging configuration in the future
|
||||
([#906](https://github.com/knative/serving/issues/906)). Once
|
||||
[#906](https://github.com/knative/serving/issues/906) is complete, the
|
||||
methodology described in this document will no longer be valid and migration to
|
||||
a new process will be required. In order to minimize the effort for a future
|
||||
migration, we recommend only changing the output configuration of Fluentd and
|
||||
leaving the rest intact.
|
||||
|
||||
**NOTE**: All the files mentioned below are in
|
||||
[knative/serving](https://github.com/knative/serving) repository. You run the
|
||||
commands mentioned below from the root directory of `knative/serving`.
|
||||
|
||||
## Configuring
|
||||
|
||||
### Configure the DaemonSet for stdout/stderr logs
|
||||
|
||||
Operators can do the following steps to configure the Fluentd DaemonSet for
|
||||
collecting `stdout/stderr` logs from the containers:
|
||||
|
||||
1. Replace `900.output.conf` part in
|
||||
[100-fluentd-configmap.yaml](https://github.com/knative/serving/blob/master/config/monitoring/logging/elasticsearch/100-fluentd-configmap.yaml)
|
||||
with the desired output configuration. Knative provides a sample for sending
|
||||
logs to Elasticsearch or Stackdriver. Developers can simply use
|
||||
`100-fluentd-configmap.yaml` or override any with other configuration.
|
||||
2. Replace the `image` field of `fluentd-ds` container of `fluentd-ds` DaemonSet
|
||||
in
|
||||
[200-fluentd.yaml](https://github.com/knative/serving/blob/master/config/monitoring/logging/elasticsearch/200-fluentd.yaml)
|
||||
with the Fluentd image including the desired Fluentd output plugin. See
|
||||
[here](./fluentd-requirements.md) for the requirements of Flunetd image on
|
||||
Knative.
|
||||
|
||||
### Configure the DaemonSet for log files under /var/log
|
||||
|
||||
The Fluentd DaemonSet can also capture `/var/log` logs from the containers. To
|
||||
enable:
|
||||
|
||||
- Set `logging.enable-var-log-collection` to `true` in
|
||||
[config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml)
|
||||
|
||||
## Deploying
|
||||
|
||||
Operators need to deploy Knative components after the configuring:
|
||||
|
||||
```shell
|
||||
# Deploy the configuration for enabling /var/log collection
|
||||
kubectl apply --filename config/config-observability.yaml
|
||||
|
||||
# Deploy the DaemonSet to make configuration for DaemonSet take effect
|
||||
kubectl apply --recursive --filename config/monitoring/100-namespace.yaml \
|
||||
--filename <path-of-fluentd-daemonset-config>
|
||||
```
|
||||
|
||||
In the commands above, replace `<path-of-fluentd-daemonset-config>` with the
|
||||
Fluentd DaemonSet configuration file, e.g.
|
||||
`config/monitoring/logging/stackdriver`.
|
||||
|
||||
**NOTE**: The deployment above will not affect the existing pods. Developers
|
||||
need to redeploy their app to get the newest configuration for `/var/log`
|
||||
collection.
|
||||
|
||||
**NOTE**: Operators sometimes need to deploy extra services as the logging
|
||||
backends. For example, if they desire Elasticsearch&Kibana, they have to deploy
|
||||
the Elasticsearch and Kibana services. Knative provides this sample:
|
||||
|
||||
```shell
|
||||
kubectl apply --recursive --filename third_party/config/monitoring/logging/elasticsearch
|
||||
```
|
||||
|
||||
See [here](./installing-logging-metrics-traces.md) for deploying the whole
|
||||
Knative monitoring components.
|
||||
|
||||
## Uninstalling
|
||||
|
||||
To uninstall a logging plugin, run:
|
||||
|
||||
```shell
|
||||
kubectl delete --recursive --filename <path-of-fluentd-daemonset-config>
|
||||
```
|
||||
|
||||
|
||||
Loading…
Reference in New Issue