dapr
/
docs
mirror of https://github.com/dapr/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Restructuring the steps for setting up Prometheus and Grafana dashboard (#781) 

* Restructure the steps for setting up Prometheus and Grafana dashboard

* Update observe-metrics-with-grafana.md

* Update setup-prometheus-grafana.md

* Update setup-prometheus-grafana.md

* Resolving review comments

* Update setup-prometheus-grafana.md

* Update README.md

Co-authored-by: Mark Fussell <mfussell@microsoft.com>
This commit is contained in:
Shalabh Mohan Shrivastava 2020-09-11 16:19:29 -07:00 committed by GitHub
parent dbc5667eba
commit 1ba5968481
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 139 additions and 91 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
howto/README.md
View File

@ -50,11 +50,12 @@ For Actors How Tos see the SDK documentation
## Observability ## Observability
### Metric and logs ### Metrics and Logs
* [Set up Azure monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md) * [Set up Azure Monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md)
* [Set up Fleuntd, Elastic search, and Kibana in Kubernetes](./setup-monitoring-tools/setup-fluentd-es-kibana.md) * [Set up Fleuntd, Elastic search, and Kibana in Kubernetes](./setup-monitoring-tools/setup-fluentd-es-kibana.md)
* [Set up Prometheus and Grafana for metrics](./setup-monitoring-tools/setup-prometheus-grafana.md) * [Set up Prometheus and Grafana in Kubernetes](./setup-monitoring-tools/setup-prometheus-grafana.md)
* [Observe metrics with Grafana](./setup-monitoring-tools/observe-metrics-with-grafana.md)
### Distributed Tracing ### Distributed Tracing

95
howto/setup-monitoring-tools/observe-metrics-with-grafana.md Normal file
View File

@ -0,0 +1,95 @@
# Observe metrics with Grafana
This document describes how to view Dapr metrics in a Grafana dashboard. You can see example screenshots of the Dapr dashboards [here](../../reference/dashboard/img/).
Or watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana Dapr metrics dashboard.
## Prerequisites
- [Set up Prometheus and Grafana](./setup-prometheus-grafana.md)
## Steps to view metrics
- [Configure Prometheus as Data Source](#configure-prometheus-as-data-source)
- [Import Dashboards in Grafana](#import-dashboards-in-grafana)
### Configure Prometheus as data source
First you need to connect Prometheus as a data source to Grafana.
1. Port-forward to svc/grafana
```
$ kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000
Handling connection for 8080
Handling connection for 8080
```
2. Browse `http://localhost:8080`
3. Login with admin and password
4. Click Configuration Settings -> Data Sources
![data source](./img/grafana-datasources.png)
5. Add Prometheus as a data soruce.
![add data source](./img/grafana-datasources.png)
6. Enter Promethesus server address in your cluster.
You can get the Prometheus server address by running the following command.
```bash
kubectl get svc -n dapr-monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 <none> 8080/TCP 7d9h
dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 <none> 80/TCP 7d9h
dapr-prom-prometheus-node-exporter ClusterIP None <none> 9100/TCP 7d9h
dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 <none> 9091/TCP 7d9h
dapr-prom-prometheus-server ClusterIP 10.0.172.191 <none> 80/TCP 7d9h
elasticsearch-master ClusterIP 10.0.36.146 <none> 9200/TCP,9300/TCP 7d10h
elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 7d10h
grafana ClusterIP 10.0.15.229 <none> 80/TCP 5d5h
kibana-kibana ClusterIP 10.0.188.224 <none> 5601/TCP 7d10h
```
In this Howto, the server is `dapr-prom-prometheus-server`.
You now need to set up Prometheus data source with the following settings:
- Name: `Dapr`
- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring`
- Default: On
![prometheus server](./img/grafana-prometheus-dapr-server-url.png)
7. Click `Save & Test` button to verify that the connection succeeded.
### Import dashboards in Grafana
Next you import the Dapr dashboards into Grafana.
In the upper left, click the "+" then "Import".
You can now import built-in [Grafana dashboard templates](https://github.com/dapr/dapr/tree/master/grafana).
The Grafana dashboards are part of [release assets](https://github.com/dapr/dapr/releases) with this URL https://github.com/dapr/dapr/releases/
You can find `grafana-actor-dashboard.json`, `grafana-sidecar-dashboard.json` and `grafana-system-services-dashboard.json` in release assets location.
![upload json](./img/grafana-uploadjson.png)
8. Find the dashboard that you imported and enjoy!
![upload json](../../reference/dashboard/img/system-service-dashboard.png)
# References
* [Set up Prometheus and Grafana](./setup-prometheus-grafana.md)
* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)

128
howto/setup-monitoring-tools/setup-prometheus-grafana.md
View File

@ -1,8 +1,8 @@
# Set up Prometheus and Grafana # Set up Prometheus and Grafana in Kubernetes
This document shows how to install Prometheus and Grafana to view metrics. This document describes how to install Prometheus and Grafana on a Kubernetes cluster which can then be used to view the Dapr metrics dashboards.
Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana metrics dashboard. Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana Dapr metrics dashboards.
## Prerequisites ## Prerequisites
@ -12,12 +12,12 @@ Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&
## Contents ## Contents
- [Install Prometheus and Grafana](#install-prometheus-and-grafana) - [Install Prometheus](#install-prometheus)
- [View metrics](#view-metrics) - [Install Grafana](#install-grafana)
## Install Prometheus and Grafana ## Install Prometheus
1. Create namespace for monitoring tool 1. First create namespace that can be used to deploy the Grafana and Prometheus monitoring tools
```bash ```bash
kubectl create namespace dapr-monitoring kubectl create namespace dapr-monitoring
@ -31,39 +31,19 @@ helm repo update
helm install dapr-prom stable/prometheus -n dapr-monitoring helm install dapr-prom stable/prometheus -n dapr-monitoring
``` ```
If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command. If you are Minikube user or want to disable persistent volume for development purposes, you can disable it by using the following command.
```bash ```bash
helm install dapr-prom stable/prometheus -n dapr-monitoring --set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false helm install dapr-prom stable/prometheus -n dapr-monitoring --set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
``` ```
3. Install Grafana 3. Validation
```bash Ensure Prometheus is running in your cluster.
helm install grafana stable/grafana -n dapr-monitoring
```
If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
```bash
helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false
```
4. Retrieve admin password for Grafana Login
> Note: remove `%` character from the password that this command returns. The admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`.
```
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode
cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%
```
5. Validation
Ensure Prometheus and Grafana are running in your cluster.
```bash ```bash
kubectl get pods -n dapr-monitoring kubectl get pods -n dapr-monitoring
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
@ -72,81 +52,53 @@ dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0
dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s
dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s
dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
grafana-c49889cff-x56vj 1/1 Running 0 5m10s
``` ```
## View metrics ## Install Grafana
1. Port-forward to svc/grafana 1. Install Grafana
```
$ kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000
Handling connection for 8080
Handling connection for 8080
```
2. Browse `http://localhost:8080`
3. Login with admin and password
4. Click Configuration Settings -> Data Sources
![data source](./img/grafana-datasources.png)
5. Add Prometheus as a data soruce.
![add data source](./img/grafana-datasources.png)
6. Enter Promethesus server address in your cluster.
You can get the prometheus server address by running following command.
```bash ```bash
kubectl get svc -n dapr-monitoring helm install grafana stable/grafana -n dapr-monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 <none> 8080/TCP 7d9h
dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 <none> 80/TCP 7d9h
dapr-prom-prometheus-node-exporter ClusterIP None <none> 9100/TCP 7d9h
dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 <none> 9091/TCP 7d9h
dapr-prom-prometheus-server ClusterIP 10.0.172.191 <none> 80/TCP 7d9h
elasticsearch-master ClusterIP 10.0.36.146 <none> 9200/TCP,9300/TCP 7d10h
elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 7d10h
grafana ClusterIP 10.0.15.229 <none> 80/TCP 5d5h
kibana-kibana ClusterIP 10.0.188.224 <none> 5601/TCP 7d10h
``` ```
In this howto, the server is `dapr-prom-prometheus-server`. If you are Minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
So you need to set up Prometheus data source with the below settings: ```bash
helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false
```
- Name: `Dapr` 2. Retrieve the admin password for Grafana login
- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring`
- Default: On
![prometheus server](./img/grafana-prometheus-dapr-server-url.png) > Note: Remove the `%` character from the password that this command returns. For example, the admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`.
7. Click `Save & Test` button to verify that connected succeeded. ```
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode
cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%
```
8. Import Dapr dashboards. 3. Validation
In the upper left, click the "+" then "Import". Ensure Grafana is running in your cluster (see last line below)
You can now import built-in [Grafana dashboard templates](../../reference/dashboard/README.md). Please see the link for the templates. ```bash
kubectl get pods -n dapr-monitoring
![upload json](./img/grafana-uploadjson.png) NAME READY STATUS RESTARTS AGE
dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
9. Find the dashboard that you imported and enjoy! dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
dapr-prom-prometheus-node-exporter-5jh8p 1/1 Running 0 4m58s
![upload json](../../reference/dashboard/img/system-service-dashboard.png) dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 4m58s
dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s
You can find more screenshots of Dapr dashboards [here](../../reference/dashboard/img/). dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s
dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
grafana-c49889cff-x56vj 1/1 Running 0 5m10s
```
Next steps are to [observe metrics with Grafana](../observe-metrics-with-grafana.md)
# References # References
* [Observe metrics with Grafana](../observe-metrics-with-grafana.md)
* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator) * [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus) * [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator) * [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)