mirror of https://github.com/knative/docs.git
Redundant content
This commit is contained in:
Ryan Gregg
GitHub
parent
fc35fa2a0b
commit
460e53e5ef
|
|
@ -1,377 +0,0 @@
|
|||
# Logs and metrics
|
||||
|
||||
## Monitoring components setup
|
||||
|
||||
First, deploy monitoring components.
|
||||
|
||||
### Elasticsearch, Kibana, Prometheus, and Grafana Setup
|
||||
|
||||
You can use two different setups:
|
||||
|
||||
1. **150-elasticsearch-prod**: This configuration collects logs & metrics from
|
||||
user containers, build controller and Istio requests.
|
||||
|
||||
```shell
|
||||
kubectl apply -R -f config/monitoring/100-common \
|
||||
-f config/monitoring/150-elasticsearch-prod \
|
||||
-f third_party/config/monitoring/common \
|
||||
-f third_party/config/monitoring/elasticsearch \
|
||||
-f config/monitoring/200-common \
|
||||
-f config/monitoring/200-common/100-istio.yaml
|
||||
```
|
||||
|
||||
1. **150-elasticsearch-dev**: This configuration collects everything **150
|
||||
-elasticsearch-prod** does, plus Knative Serving controller logs.
|
||||
|
||||
```shell
|
||||
kubectl apply -R -f config/monitoring/100-common \
|
||||
-f config/monitoring/150-elasticsearch-dev \
|
||||
-f third_party/config/monitoring/common \
|
||||
-f third_party/config/monitoring/elasticsearch \
|
||||
-f config/monitoring/200-common \
|
||||
-f config/monitoring/200-common/100-istio.yaml
|
||||
```
|
||||
|
||||
### Stackdriver, Prometheus, and Grafana Setup
|
||||
|
||||
If your Knative Serving is not built on a Google Cloud Platform based cluster,
|
||||
or you want to send logs to another GCP project, you need to build your own
|
||||
Fluentd image and modify the configuration first. See
|
||||
|
||||
1. [Fluentd image on Knative Serving](/image/fluentd/README.md)
|
||||
2. [Setting up a logging plugin](setting-up-a-logging-plugin.md)
|
||||
|
||||
Then you can use two different setups:
|
||||
|
||||
1. **150-stackdriver-prod**: This configuration collects logs and metrics from
|
||||
user containers, build controller, and Istio requests.
|
||||
|
||||
```shell
|
||||
kubectl apply -R -f config/monitoring/100-common \
|
||||
-f config/monitoring/150-stackdriver-prod \
|
||||
-f third_party/config/monitoring/common \
|
||||
-f config/monitoring/200-common \
|
||||
-f config/monitoring/200-common/100-istio.yaml
|
||||
```
|
||||
|
||||
2. **150-stackdriver-dev**: This configuration collects everything **150
|
||||
-stackdriver-prod** does, plus Knative Serving controller logs.
|
||||
|
||||
```shell
|
||||
kubectl apply -R -f config/monitoring/100-common \
|
||||
-f config/monitoring/150-stackdriver-dev \
|
||||
-f third_party/config/monitoring/common \
|
||||
-f config/monitoring/200-common \
|
||||
-f config/monitoring/200-common/100-istio.yaml
|
||||
```
|
||||
|
||||
## Accessing logs
|
||||
|
||||
### Kibana and Elasticsearch
|
||||
|
||||
To open the Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co),
|
||||
enter the following command:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
This starts a local proxy of Kibana on port 8001. The Kibana UI is only exposed within
|
||||
the cluster for security reasons.
|
||||
|
||||
Navigate to the [Kibana UI](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana)
|
||||
(*It might take a couple of minutes for the proxy to work*).
|
||||
|
||||
When Kibana is opened the first time, it will ask you to create an index.
|
||||
Accept the default options:
|
||||
|
||||
|
||||
|
||||
The Discover tab of the Kibana UI looks like this:
|
||||
|
||||
|
||||
|
||||
You can change the time frame of logs Kibana displays in the upper right corner
|
||||
of the screen. The main search bar is across the top of the Dicover page.
|
||||
|
||||
As more logs are ingested, new fields will be discovered. To have them indexed,
|
||||
go to Management > Index Patterns > Refresh button (on top right) > Refresh
|
||||
fields.
|
||||
|
||||
<!-- TODO: create a video walkthrough of the Kibana UI -->
|
||||
|
||||
#### Accessing configuration and revision logs
|
||||
|
||||
To access the logs for a configuration, enter the following search query in Kibana:
|
||||
|
||||
```
|
||||
kubernetes.labels.knative_dev\/configuration: "configuration-example"
|
||||
```
|
||||
|
||||
Replace `configuration-example` with your configuration's name. Enter the following
|
||||
command to get your configuration's name:
|
||||
|
||||
```shell
|
||||
kubectl get configurations
|
||||
```
|
||||
|
||||
To access logs for a revision, enter the following search query in Kibana:
|
||||
|
||||
```
|
||||
kubernetes.labels.knative_dev\/revision: "configuration-example-00001"
|
||||
```
|
||||
|
||||
Replace `configuration-example-00001` with your revision's name.
|
||||
|
||||
#### Accessing build logs
|
||||
|
||||
To access the logs for a build, enter the following search query in Kibana:
|
||||
|
||||
```
|
||||
kubernetes.labels.build\-name: "test-build"
|
||||
```
|
||||
|
||||
Replace `test-build` with your build's name. The build name is specified in the `.yaml` file as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: build.knative.dev/v1alpha1
|
||||
kind: Build
|
||||
metadata:
|
||||
name: test-build
|
||||
```
|
||||
|
||||
### Stackdriver
|
||||
|
||||
Go to the [Google Cloud Console logging page](https://console.cloud.google.com/logs/viewer) for
|
||||
your GCP project which stores your logs via Stackdriver.
|
||||
|
||||
## Accessing metrics
|
||||
|
||||
Enter:
|
||||
|
||||
```shell
|
||||
kubectl port-forward -n monitoring $(kubectl get pods -n monitoring --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000
|
||||
```
|
||||
|
||||
Then open the Grafana UI at [http://localhost:3000](http://localhost:3000). The following dashboards are
|
||||
pre-installed with Knative Serving:
|
||||
|
||||
* **Revision HTTP Requests:** HTTP request count, latency and size metrics per revision and per configuration
|
||||
* **Nodes:** CPU, memory, network and disk metrics at node level
|
||||
* **Pods:** CPU, memory and network metrics at pod level
|
||||
* **Deployment:** CPU, memory and network metrics aggregated at deployment level
|
||||
* **Istio, Mixer and Pilot:** Detailed Istio mesh, Mixer and Pilot metrics
|
||||
* **Kubernetes:** Dashboards giving insights into cluster health, deployments and capacity usage
|
||||
|
||||
### Accessing per request traces
|
||||
|
||||
Before you can view per request metrics, you'll need to create a new index pattern that will store
|
||||
per request traces captured by Zipkin:
|
||||
|
||||
1. Start the Kibana UI serving on local port 8001 by entering the following command:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
1. Open the [Kibana UI](http://localhost:8001/api/v1/namespaces/monitoring/services/kibana-logging/proxy/app/kibana).
|
||||
|
||||
1. Navigate to Management -> Index Patterns -> Create Index Pattern.
|
||||
|
||||
1. Enter `zipkin*` in the "Index pattern" text field.
|
||||
|
||||
1. Click **Create**.
|
||||
|
||||
After you've created the Zipkin index pattern, open the
|
||||
[Zipkin UI](http://localhost:8001/api/v1/namespaces/istio-system/services/zipkin:9411/proxy/zipkin/).
|
||||
Click on "Find Traces" to see the latest traces. You can search for a trace ID
|
||||
or look at traces of a specific application. Click on a trace to see a detailed
|
||||
view of a specific call.
|
||||
|
||||
To see a demo of distributed tracing, deploy the
|
||||
[Telemetry sample](../sample/telemetrysample/README.md), send some traffic to it,
|
||||
then explore the traces it generates from Zipkin UI.
|
||||
|
||||
<!--TODO: Consider adding a video here. -->
|
||||
|
||||
## Default metrics
|
||||
|
||||
The following metrics are collected by default:
|
||||
* Knative Serving controller metrics
|
||||
* Istio metrics (mixer, envoy and pilot)
|
||||
* Node and pod metrics
|
||||
|
||||
There are several other collectors that are pre-configured but not enabled.
|
||||
To see the full list, browse to config/monitoring/prometheus-exporter
|
||||
and config/monitoring/prometheus-servicemonitor folders and deploy them
|
||||
using `kubectl apply -f`.
|
||||
|
||||
## Default logs
|
||||
|
||||
Deployment above enables collection of the following logs:
|
||||
|
||||
* stdout & stderr from all user-container
|
||||
* stdout & stderr from build-controller
|
||||
|
||||
To enable log collection from other containers and destinations, see
|
||||
[setting up a logging plugin](setting-up-a-logging-plugin.md).
|
||||
|
||||
## Metrics troubleshooting
|
||||
|
||||
You can use the Prometheus web UI to troubleshoot publishing and service
|
||||
discovery issues for metrics. To access to the web UI, forward the Prometheus
|
||||
server to your machine:
|
||||
|
||||
```shell
|
||||
kubectl port-forward -n monitoring $(kubectl get pods -n monitoring --selector=app=prometheus --output=jsonpath="{.items[0].metadata.name}") 9090
|
||||
```
|
||||
|
||||
Then browse to http://localhost:9090 to access the UI.
|
||||
|
||||
* To see the targets that are being scraped, go to Status -> Targets
|
||||
* To see what Prometheus service discovery is picking up vs. dropping, go to Status -> Service Discovery
|
||||
|
||||
## Generating metrics
|
||||
|
||||
If you want to send metrics from your controller, follow the steps below. These
|
||||
steps are already applied to autoscaler and controller. For those controllers,
|
||||
simply add your new metric definitions to the `view`, create new `tag.Key`s if
|
||||
necessary and instrument your code as described in step 3.
|
||||
|
||||
In the example below, we will setup the service to host the metrics and
|
||||
instrument a sample 'Gauge' type metric using the setup.
|
||||
|
||||
1. First, go through [OpenCensus Go Documentation](https://godoc.org/go.opencensus.io).
|
||||
2. Add the following to your application startup:
|
||||
|
||||
```go
|
||||
import (
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"go.opencensus.io/stats"
|
||||
"go.opencensus.io/stats/view"
|
||||
"go.opencensus.io/tag"
|
||||
)
|
||||
|
||||
var (
|
||||
desiredPodCountM *stats.Int64Measure
|
||||
namespaceTagKey tag.Key
|
||||
revisionTagKey tag.Key
|
||||
)
|
||||
|
||||
func main() {
|
||||
exporter, err := prometheus.NewExporter(prometheus.Options{Namespace: "{your metrics namespace (eg: autoscaler)}"})
|
||||
if err != nil {
|
||||
glog.Fatal(err)
|
||||
}
|
||||
view.RegisterExporter(exporter)
|
||||
view.SetReportingPeriod(10 * time.Second)
|
||||
|
||||
// Create a sample gauge
|
||||
var r = &Reporter{}
|
||||
desiredPodCountM = stats.Int64(
|
||||
"desired_pod_count",
|
||||
"Number of pods autoscaler wants to allocate",
|
||||
stats.UnitNone)
|
||||
|
||||
// Tag the statistics with namespace and revision labels
|
||||
var err error
|
||||
namespaceTagKey, err = tag.NewKey("namespace")
|
||||
if err != nil {
|
||||
// Error handling
|
||||
}
|
||||
revisionTagKey, err = tag.NewKey("revision")
|
||||
if err != nil {
|
||||
// Error handling
|
||||
}
|
||||
|
||||
// Create view to see our measurement.
|
||||
err = view.Register(
|
||||
&view.View{
|
||||
Description: "Number of pods autoscaler wants to allocate",
|
||||
Measure: r.measurements[DesiredPodCountM],
|
||||
Aggregation: view.LastValue(),
|
||||
TagKeys: []tag.Key{namespaceTagKey, configTagKey, revisionTagKey},
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
// Error handling
|
||||
}
|
||||
|
||||
// Start the endpoint for Prometheus scraping
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("/metrics", exporter)
|
||||
http.ListenAndServe(":8080", mux)
|
||||
}
|
||||
```
|
||||
|
||||
3. In your code where you want to instrument, set the counter with the
|
||||
appropriate label values - example:
|
||||
|
||||
```go
|
||||
ctx := context.TODO()
|
||||
tag.New(
|
||||
ctx,
|
||||
tag.Insert(namespaceTagKey, namespace),
|
||||
tag.Insert(revisionTagKey, revision))
|
||||
stats.Record(ctx, desiredPodCountM.M({Measurement Value}))
|
||||
```
|
||||
|
||||
4. Add the following to scape config file located at
|
||||
config/monitoring/200-common/300-prometheus/100-scrape-config.yaml:
|
||||
|
||||
```yaml
|
||||
- job_name: <YOUR SERVICE NAME>
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
relabel_configs:
|
||||
# Scrape only the the targets matching the following metadata
|
||||
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_label_app, __meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: {SERVICE NAMESPACE};{APP LABEL};{PORT NAME}
|
||||
# Rename metadata labels to be reader friendly
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: namespace
|
||||
replacement: $1
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: pod
|
||||
replacement: $1
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: replace
|
||||
regex: (.*)
|
||||
target_label: service
|
||||
replacement: $1
|
||||
```
|
||||
|
||||
5. Redeploy prometheus and its configuration:
|
||||
```sh
|
||||
kubectl delete -f config/monitoring/200-common/300-prometheus
|
||||
kubectl apply -f config/monitoring/200-common/300-prometheus
|
||||
```
|
||||
|
||||
6. Add a dashboard for your metrics - you can see examples of it under
|
||||
config/grafana/dashboard-definition folder. An easy way to generate JSON
|
||||
definitions is to use Grafana UI (make sure to login with as admin user) and
|
||||
[export JSON](http://docs.grafana.org/reference/export_import) from it.
|
||||
|
||||
7. Validate the metrics flow either by Grafana UI or Prometheus UI (see
|
||||
Troubleshooting section above to enable Prometheus UI)
|
||||
|
||||
## Distributed tracing with Zipkin
|
||||
Check [Telemetry sample](../sample/telemetrysample/README.md) as an example usage of
|
||||
[OpenZipkin](https://zipkin.io/pages/existing_instrumentations)'s Go client library.
|
||||
|
||||
## Delete monitoring components
|
||||
Enter:
|
||||
|
||||
```shell
|
||||
ko delete --ignore-not-found=true \
|
||||
-f config/monitoring/200-common/100-istio.yaml \
|
||||
-f config/monitoring/200-common/100-zipkin.yaml \
|
||||
-f config/monitoring/100-common
|
||||
```
|
||||
Loading…
Reference in New Issue