Final monitoring doc revision (#433)

* Correct helm chart option

* Revise monitoring docs

* update howto readme

* update concepts

* fix cr

* wip

* Fix dashboard for minikube

* Fix steps for prometheus howto

* Update ordering

* fix ct

* wip

* Update README.md
This commit is contained in:
Young Bu Park 2020-03-12 10:40:27 -07:00 committed by GitHub
parent 57bb1df233
commit 0faaeb37d7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
18 changed files with 2461 additions and 2477 deletions

View File

@ -1,6 +1,6 @@
# Observability
Observability is a term from control theory. Observability means you can answer questions about whats happening on the inside of a system by observing the outside of the system, without having to ship new code to answer new questions. Observability is critical in production environments and services to debug, operate and monitor Dapr system services, components and user applications.
Observability is a term from control theory. Observability means you can answer questions about what's happening on the inside of a system by observing the outside of the system, without having to ship new code to answer new questions. Observability is critical in production environments and services to debug, operate and monitor Dapr system services, components and user applications.
The observability capabilities enable users to monitor the Dapr system services, their interaction with user applications and understand how these monitored services behave. The observability capabilities are divided into three main areas;
@ -17,8 +17,9 @@ The table below shows the current status of each of the observabilty capabilites
|Tracing | Yes | N/A | N/A | *Planned* | N/A |
|Logs | Yes | Yes | Yes | Yes | Yes |
## Supported monitoring tools
The observability tools listed below are ones that have been tested to work with Dapr
## Monitoring tools
The observability tools listed below are ones that have been tested to work with Dapr.
### Metrics

View File

@ -44,10 +44,10 @@ The following steps describe how to configure JSON formatted logs for Kubernetes
### Install Dapr to your cluster using the Helm chart
You can enable JSON formatted logs for Dapr system services by adding `--set global.LogASJSON=true` option to Helm command.
You can enable JSON formatted logs for Dapr system services by adding `--set global.logAsJson=true` option to Helm command.
```bash
helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
```
### Enable JSON formatted log for Dapr sidecars

View File

@ -1,30 +1,21 @@
# Metrics
Dapr exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain
a greater understanding of how Dapr is behaving and to setup alerts for specific conditions.
Dapr exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving and to setup alerts for specific conditions.
## Configuration
The metrics endpoint is enabled by default, you can disable it by passing the command line argument
`--enable-metrics=false` to daprd.
The default metrics port is `9090`. This can be overridden by passing the command line argument
`--metrics-port` to darpd.
The metrics endpoint is enabled by default, you can disable it by passing the command line argument `--enable-metrics=false` to dapr system processes.
The default metrics port is `9090`. This can be overridden by passing the command line argument `--metrics-port` to darpd.
## Metrics
Dapr will expose metrics for Darp's HTTP pipeline, gPRC pipeline, Go runtime, metrics server and process.
> Note: The gRPC metrics are all initialized with zero values and are therefore immediately visible on the
metrics endpoint. HTTP metrics are not zero initialized so will only become visible once a HTTP request has
passed through the HTTP pipeline.
## Prometheus
To consume these metrics you'll need to install Prometheus or something compatible with scraping Prometheus
metrics.
Each Dapr system process emits Go runtime/process metrics by default and have their own metrics:
For more detailed instructions on installing and configuring Prometheus to work with Dapr, please refer to the [Observe Metrics with Prometheus](../../howto/observe-metrics-with-prometheus/README.md) article.
- [Dapr runtime metric list](https://github.com/dapr/dapr/blob/master/pkg/diagnostics/README.md)
## References
* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Howto: Run Prometheus locally](../../howto/setup-monitoring-tools/observe-metrics-with-prometheus-locally.md)
* [Howto: Set up Prometheus and Grafana for metrics](../../howto/setup-monitoring-tools/setup-prometheus-grafana.md)
* [Howto: Set up Azure monitor to search logs and collect metrics for Dapr](../../howto/setup-monitoring-tools/setup-azure-monitor.md)

View File

@ -51,10 +51,18 @@ For Actors How Tos see the SDK documentation
* [Java Actors](https://github.com/dapr/java-sdk)
## Observerability
### Metric and logs
* [Set up Azure monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md)
* [Set up Fleuntd, Elastic search, and Kibana in Kubernetes](./setup-monitoring-tools/setup-fluentd-es-kibana.md)
* [Set up Prometheus and Grafana for metrics](./setup-monitoring-tools/setup-prometheus-grafana.md)
### Distributed Tracing
* [Diagnose your services with distributed tracing](./diagnose-with-tracing)
## Security
### Mutual Transport Layer Security (TLS)
@ -87,4 +95,4 @@ For Actors How Tos see the SDK documentation
## Infrastructure integration
* [Autoscale on Kubernetes using KEDA and Dapr bindings](./autoscale-with-keda)
* [Autoscale on Kubernetes using KEDA and Dapr bindings](./autoscale-with-keda)

Binary file not shown.

After

Width:  |  Height:  |  Size: 361 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

View File

@ -1,4 +1,5 @@
# Observe Metrics with Prometheus
# Observe Metrics with Prometheus locally
Dapr exposes a Prometheus metrics endpoint you can use to collect time-series
data relating to the execution of the Dapr runtime itself.
@ -55,36 +56,3 @@ docker run \
`--net=host` ensures that the Prometheus instance will be able to connect to any Dapr instances running on the host machine. If you plan to run your Dapr apps in containers as well, you'll need to run them on a shared Docker network and update the configuration with the correct target address.
Once Prometheus is running, you'll be able to visit its dashboard by visiting `http://localhost:8080`.
## Setup Prometheus on Kubernetes
Prometheus can be installed onto a Kubernetes cluster in a number of different ways. These are documented comprehensively [here](https://github.com/coreos/kube-prometheus).
Once you have installed Prometheus on your Kubernetes cluster, you can add your Dapr sidecars' service address to your Prometheus configuration.
The Dapr sidecar addresses can be fetched by using
```
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
nodeapp-dapr ClusterIP 10.0.163.251 <none> 80/TCP,50001/TCP,9090/TCP
...
```
You can then add the services DNS to your Prometheus configuration.
```yaml
...
scrape_configs:
...
- job_name: 'nodeapp-dapr'
metrics_path: /
static_configs:
- targets: ['nodeapp-dapr.<namespace>.svc.cluster.local:9090'] # Replace with Dapr metrics port if not default
...
```
To see you Prometheus dashboard in Kubernetes, you'll need to port forward to your Prometheus pod and then hit the local endpoint `http://localhost:9090`.
```
kubectl port-forward <prometheus-pod-name> 9090:9090
```

View File

@ -59,7 +59,7 @@ kubectl apply -f ./azm-config.map.yaml
1. Install Dapr with enabling JSON-formatted logs
```bash
helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
```
2. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.

View File

@ -86,7 +86,7 @@ fluentd-sdrld 1/1 Running 0 14s
1. Install Dapr with enabling JSON-formatted logs
```bash
helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
```
2. Enable JSON formatted log in Dapr sidecar

View File

@ -30,10 +30,7 @@ helm install dapr-prom stable/prometheus -n dapr-monitoring
If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
```bash
helm install prometheus stable/prometheus -n dapr-monitoring
--set alertmanager.persistentVolume.enable=false
--set pushgateway.persistentVolume.enabled=false
--set server.persistentVolume.enabled=false
helm install dapr-prom stable/prometheus -n dapr-monitoring --set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
```
3. Install Grafana
@ -44,11 +41,20 @@ helm install grafana stable/grafana -n dapr-monitoring
If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
```bash
```bash
helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false
```
4. Validation
4. Retrieve admin password for Grafana Login
> Note: remove `%` character from the password that this command returns. The admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`.
```
kubernetes get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode
cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%
```
5. Validation
Ensure Prometheus and Grafana are running in your cluster.
@ -80,15 +86,17 @@ Handling connection for 8080
2. Browse `http://localhost:8080`
3. Click Configuration Settings -> Data Sources
3. Login with admin and password
4. Click Configuration Settings -> Data Sources
![data source](./img/grafana-datasources.png)
4. Add Prometheus as a data soruce.
5. Add Prometheus as a data soruce.
![add data source](./img/grafana-datasources.png)
5. Enter Promethesus server address in your cluster.
6. Enter Promethesus server address in your cluster.
You can get the prometheus server address by running following command.
@ -107,30 +115,35 @@ kibana-kibana ClusterIP 10.0.188.224 <none> 56
```
In this set up tutorial, the server is `dapr-prom-prometheus-server`.
In this howto, the server is `dapr-prom-prometheus-server`.
So you need to provide `http://dapr-prom-prometheus-server.dapr-monitoring` in the URL field.
So you need to set up Prometheus data source with the below settings:
![prometheus server](./img/grafana-prometheus-server-url.png)
- Name: `Dapr`
- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring`
- Default: On
6. Click Save & Test button to verify that connected succeeded.
![prometheus server](./img/grafana-prometheus-dapr-server-url.png)
7. Import Dapr dashboards.
7. Click `Save & Test` button to verify that connected succeeded.
You can now import built-in [Grafana dashboard templates](https://github.com/dapr/docs/tree/master/monitoring/grafana/dashboards).
8. Import Dapr dashboards.
Refer [here](https://github.com/dapr/docs/tree/master/monitoring/grafana) for details.
You can now import built-in [Grafana dashboard templates](../../reference/dashboard/README.md).
Refer [here](../../reference/dashboard/README.md) for details.
![upload json](./img/grafana-uploadjson.png)
You can find screenshots of Dapr dashboards [here](https://github.com/dapr/docs/tree/master/monitoring/grafana/img).
9. Find the dashboard that you imported and enjoy!
![upload json](../../reference/dashboard/img/system-service-dashboard.png)
You can find more screenshots of Dapr dashboards [here](../../reference/dashboard/img/).
# References
* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)
* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)

View File

@ -1,9 +0,0 @@
# Grafana Dashboard Templates
Dapr offers the below dashboard templates to monitor Dapr system component and sidecars. You can import the templates to your Grafana dashboard.
1. [Dapr System Service Dashboard](./dashboards/system-services-dashboard.json)
- [Shows Dapr system component status](./img/system-service-dashboard.png) - dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement
2. [Dapr Sidecar Dashboard](./dashboards/sidecar-dashboard.json)
- [Shows Dapr Sidecar status](./img/sidecar-dashboard.png) - sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.

File diff suppressed because it is too large Load Diff

View File

@ -9,3 +9,4 @@
- [Service Invocation](./api/service_invocation_api.md)
- [State Management](./api/state_api.md)
- **[Dapr Binding Specs](./specs/bindings)**: Bindings provide triggers and interactions with external resources and services
- **[Monitoring Dashboard templates](./monitoring/README.md)**: Monitoring dashboard templates help Dapr user to monitor Dapr services by importing templates to their monitoring tools

View File

@ -0,0 +1,13 @@
# Monitoring Dashboard
This includes dashboard templates to monitor Dapr system services and sidecars. For more detail information, please read [Dapr Observability](../../concepts/observability/README.md).
## Grafana
You can set up [Prometheus and Grafana](../../howto/setup-monitoring-tools/setup-prometheus-grafana.md) and import the templates to your Grafana dashboard to monitor Dapr.
1. [Dapr System Service Dashboard](./grafana/system-services-dashboard.json)
- [Shows Dapr system component status](./img/system-service-dashboard.png) - dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement
2. [Dapr Sidecar Dashboard](./grafana/sidecar-dashboard.json)
- [Shows Dapr Sidecar status](./img/sidecar-dashboard.png) - sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.

File diff suppressed because it is too large Load Diff

View File

@ -15,7 +15,7 @@
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 18,
"id": 1,
"links": [],
"panels": [
{
@ -1514,7 +1514,7 @@
"list": []
},
"time": {
"from": "now-24h",
"from": "now-1h",
"to": "now"
},
"timepicker": {
@ -1534,5 +1534,5 @@
"timezone": "",
"title": "Dapr System Services Dashboard",
"uid": "RHSwiHXWk",
"version": 29
"version": 2
}

View File

Before

Width:  |  Height:  |  Size: 605 KiB

After

Width:  |  Height:  |  Size: 605 KiB

View File

Before

Width:  |  Height:  |  Size: 523 KiB

After

Width:  |  Height:  |  Size: 523 KiB