diff --git a/howto/README.md b/howto/README.md index ec7630e26..294cc5aad 100644 --- a/howto/README.md +++ b/howto/README.md @@ -50,11 +50,12 @@ For Actors How Tos see the SDK documentation ## Observability -### Metric and logs +### Metrics and Logs -* [Set up Azure monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md) +* [Set up Azure Monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md) * [Set up Fleuntd, Elastic search, and Kibana in Kubernetes](./setup-monitoring-tools/setup-fluentd-es-kibana.md) -* [Set up Prometheus and Grafana for metrics](./setup-monitoring-tools/setup-prometheus-grafana.md) +* [Set up Prometheus and Grafana in Kubernetes](./setup-monitoring-tools/setup-prometheus-grafana.md) +* [Observe metrics with Grafana](./setup-monitoring-tools/observe-metrics-with-grafana.md) ### Distributed Tracing diff --git a/howto/setup-monitoring-tools/observe-metrics-with-grafana.md b/howto/setup-monitoring-tools/observe-metrics-with-grafana.md new file mode 100644 index 000000000..38aee9b7c --- /dev/null +++ b/howto/setup-monitoring-tools/observe-metrics-with-grafana.md @@ -0,0 +1,95 @@ +# Observe metrics with Grafana + +This document describes how to view Dapr metrics in a Grafana dashboard. You can see example screenshots of the Dapr dashboards [here](../../reference/dashboard/img/). + +Or watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana Dapr metrics dashboard. + +## Prerequisites + +- [Set up Prometheus and Grafana](./setup-prometheus-grafana.md) + +## Steps to view metrics + +- [Configure Prometheus as Data Source](#configure-prometheus-as-data-source) +- [Import Dashboards in Grafana](#import-dashboards-in-grafana) + +### Configure Prometheus as data source +First you need to connect Prometheus as a data source to Grafana. + +1. Port-forward to svc/grafana + +``` +$ kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring +Forwarding from 127.0.0.1:8080 -> 3000 +Forwarding from [::1]:8080 -> 3000 +Handling connection for 8080 +Handling connection for 8080 +``` + +2. Browse `http://localhost:8080` + +3. Login with admin and password + +4. Click Configuration Settings -> Data Sources + + ![data source](./img/grafana-datasources.png) + +5. Add Prometheus as a data soruce. + + ![add data source](./img/grafana-datasources.png) + +6. Enter Promethesus server address in your cluster. + + You can get the Prometheus server address by running the following command. + +```bash +kubectl get svc -n dapr-monitoring + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 8080/TCP 7d9h +dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 80/TCP 7d9h +dapr-prom-prometheus-node-exporter ClusterIP None 9100/TCP 7d9h +dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 9091/TCP 7d9h +dapr-prom-prometheus-server ClusterIP 10.0.172.191 80/TCP 7d9h +elasticsearch-master ClusterIP 10.0.36.146 9200/TCP,9300/TCP 7d10h +elasticsearch-master-headless ClusterIP None 9200/TCP,9300/TCP 7d10h +grafana ClusterIP 10.0.15.229 80/TCP 5d5h +kibana-kibana ClusterIP 10.0.188.224 5601/TCP 7d10h + +``` + +In this Howto, the server is `dapr-prom-prometheus-server`. + +You now need to set up Prometheus data source with the following settings: + +- Name: `Dapr` +- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring` +- Default: On + +![prometheus server](./img/grafana-prometheus-dapr-server-url.png) + +7. Click `Save & Test` button to verify that the connection succeeded. + +### Import dashboards in Grafana +Next you import the Dapr dashboards into Grafana. + +In the upper left, click the "+" then "Import". + +You can now import built-in [Grafana dashboard templates](https://github.com/dapr/dapr/tree/master/grafana). + +The Grafana dashboards are part of [release assets](https://github.com/dapr/dapr/releases) with this URL https://github.com/dapr/dapr/releases/ +You can find `grafana-actor-dashboard.json`, `grafana-sidecar-dashboard.json` and `grafana-system-services-dashboard.json` in release assets location. + +![upload json](./img/grafana-uploadjson.png) + +8. Find the dashboard that you imported and enjoy! + +![upload json](../../reference/dashboard/img/system-service-dashboard.png) + +# References + +* [Set up Prometheus and Grafana](./setup-prometheus-grafana.md) +* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator) +* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus) +* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator) +* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/) diff --git a/howto/setup-monitoring-tools/setup-prometheus-grafana.md b/howto/setup-monitoring-tools/setup-prometheus-grafana.md index e6b033bee..b3f96523e 100644 --- a/howto/setup-monitoring-tools/setup-prometheus-grafana.md +++ b/howto/setup-monitoring-tools/setup-prometheus-grafana.md @@ -1,8 +1,8 @@ -# Set up Prometheus and Grafana +# Set up Prometheus and Grafana in Kubernetes -This document shows how to install Prometheus and Grafana to view metrics. +This document describes how to install Prometheus and Grafana on a Kubernetes cluster which can then be used to view the Dapr metrics dashboards. -Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana metrics dashboard. +Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be&t=2580) for a demonstration of the Grafana Dapr metrics dashboards. ## Prerequisites @@ -12,12 +12,12 @@ Watch this [video](https://www.youtube.com/watch?v=8W-iBDNvCUM&feature=youtu.be& ## Contents - - [Install Prometheus and Grafana](#install-prometheus-and-grafana) - - [View metrics](#view-metrics) + - [Install Prometheus](#install-prometheus) + - [Install Grafana](#install-grafana) -## Install Prometheus and Grafana +## Install Prometheus -1. Create namespace for monitoring tool +1. First create namespace that can be used to deploy the Grafana and Prometheus monitoring tools ```bash kubectl create namespace dapr-monitoring @@ -31,39 +31,19 @@ helm repo update helm install dapr-prom stable/prometheus -n dapr-monitoring ``` - If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command. +If you are Minikube user or want to disable persistent volume for development purposes, you can disable it by using the following command. ```bash helm install dapr-prom stable/prometheus -n dapr-monitoring --set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false ``` -3. Install Grafana +3. Validation -```bash -helm install grafana stable/grafana -n dapr-monitoring -``` - - If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command. - -```bash -helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false -``` - -4. Retrieve admin password for Grafana Login - -> Note: remove `%` character from the password that this command returns. The admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`. - -``` -kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode -cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1% -``` - -5. Validation - -Ensure Prometheus and Grafana are running in your cluster. +Ensure Prometheus is running in your cluster. ```bash kubectl get pods -n dapr-monitoring + NAME READY STATUS RESTARTS AGE dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s @@ -72,81 +52,53 @@ dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s -grafana-c49889cff-x56vj 1/1 Running 0 5m10s - ``` -## View metrics +## Install Grafana -1. Port-forward to svc/grafana - -``` -$ kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring -Forwarding from 127.0.0.1:8080 -> 3000 -Forwarding from [::1]:8080 -> 3000 -Handling connection for 8080 -Handling connection for 8080 -``` - -2. Browse `http://localhost:8080` - -3. Login with admin and password - -4. Click Configuration Settings -> Data Sources - - ![data source](./img/grafana-datasources.png) - -5. Add Prometheus as a data soruce. - - ![add data source](./img/grafana-datasources.png) - -6. Enter Promethesus server address in your cluster. - -You can get the prometheus server address by running following command. +1. Install Grafana ```bash -kubectl get svc -n dapr-monitoring -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 8080/TCP 7d9h -dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 80/TCP 7d9h -dapr-prom-prometheus-node-exporter ClusterIP None 9100/TCP 7d9h -dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 9091/TCP 7d9h -dapr-prom-prometheus-server ClusterIP 10.0.172.191 80/TCP 7d9h -elasticsearch-master ClusterIP 10.0.36.146 9200/TCP,9300/TCP 7d10h -elasticsearch-master-headless ClusterIP None 9200/TCP,9300/TCP 7d10h -grafana ClusterIP 10.0.15.229 80/TCP 5d5h -kibana-kibana ClusterIP 10.0.188.224 5601/TCP 7d10h - +helm install grafana stable/grafana -n dapr-monitoring ``` -In this howto, the server is `dapr-prom-prometheus-server`. +If you are Minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command. -So you need to set up Prometheus data source with the below settings: +```bash +helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false +``` -- Name: `Dapr` -- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring` -- Default: On +2. Retrieve the admin password for Grafana login -![prometheus server](./img/grafana-prometheus-dapr-server-url.png) +> Note: Remove the `%` character from the password that this command returns. For example, the admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`. -7. Click `Save & Test` button to verify that connected succeeded. +``` +kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode +cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1% +``` -8. Import Dapr dashboards. +3. Validation -In the upper left, click the "+" then "Import". +Ensure Grafana is running in your cluster (see last line below) -You can now import built-in [Grafana dashboard templates](../../reference/dashboard/README.md). Please see the link for the templates. +```bash +kubectl get pods -n dapr-monitoring -![upload json](./img/grafana-uploadjson.png) - -9. Find the dashboard that you imported and enjoy! - -![upload json](../../reference/dashboard/img/system-service-dashboard.png) - -You can find more screenshots of Dapr dashboards [here](../../reference/dashboard/img/). +NAME READY STATUS RESTARTS AGE +dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s +dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s +dapr-prom-prometheus-node-exporter-5jh8p 1/1 Running 0 4m58s +dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 4m58s +dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s +dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s +dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s +grafana-c49889cff-x56vj 1/1 Running 0 5m10s +``` +Next steps are to [observe metrics with Grafana](../observe-metrics-with-grafana.md) # References +* [Observe metrics with Grafana](../observe-metrics-with-grafana.md) * [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator) * [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus) * [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)