docker
/
docs
mirror of https://github.com/docker/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

procedure formatting fixed

This commit is contained in:
ddeyo 2018-09-14 10:16:33 -07:00
parent 66b01469c6
commit 3d8a6fb9ff
1 changed files with 132 additions and 124 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

116
ee/ucp/admin/configure/collect-cluster-metrics.md
View File

@ -25,27 +25,33 @@ Events, logs, and metrics are sources of data that provide observability of your
The Docker EE platform provides a base set of metrics that gets you running and into production without having to rely on external or 3rd party tools. Docker strongly encourages the use of additional monitoring to provide more comprehensive visibility into your specific Docker environment, but recognizes the need for a basic set of metrics built into the product. The following are examples of these metrics: The Docker EE platform provides a base set of metrics that gets you running and into production without having to rely on external or 3rd party tools. Docker strongly encourages the use of additional monitoring to provide more comprehensive visibility into your specific Docker environment, but recognizes the need for a basic set of metrics built into the product. The following are examples of these metrics:
- **Business metrics**: These are high-level aggregate metrics that typically combine technical, financial, and organizational data to create metrics for business leaders of the IT infrastructure. Some examples of business metrics might be: ## Business metrics ##
These are high-level aggregate metrics that typically combine technical, financial, and organizational data to create metrics for business leaders of the IT infrastructure. Some examples of business metrics might be:
- Company or division-level application downtime - Company or division-level application downtime
- Aggregate resource utilization - Aggregate resource utilization
- Application resource demand growth - Application resource demand growth
## Application metrics ##
- **Application metrics**: These are metrics about domain of APM tools like AppDynamics or DynaTrace and provide metrics about the state or performance of the application itself. These are metrics about domain of APM tools like AppDynamics or DynaTrace and provide metrics about the state or performance of the application itself.
- Service state metrics - Service state metrics
- Container platform metrics - Container platform metrics
- Host infrastructure metrics - Host infrastructure metrics
Docker EE 2.1 does not collect or expose application level metrics. The following are metrics Docker EE 2.1 collects, aggregates, and exposes: Docker EE 2.1 does not collect or expose application level metrics. The following are metrics Docker EE 2.1 collects, aggregates, and exposes:
- **Service state metrics**: These are metrics about the state of services running on the container platform. These types of metrics have very low cardinality, meaning the values are typically from a small fixed set of possibilities, commonly binary. ## Service state metrics ##
These are metrics about the state of services running on the container platform. These types of metrics have very low cardinality, meaning the values are typically from a small fixed set of possibilities, commonly binary.
- Application health - Application health
- Convergence of K8s deployments and Swarm services - Convergence of K8s deployments and Swarm services
- Cluster load by number of services or containers or pods - Cluster load by number of services or containers or pods
- **Host infrastructure metrics**: These are metrics taken from te software & hardware infrastructure. ## Host infrastructure metrics ##
These are metrics taken from te software & hardware infrastructure.
- CPU - Container-level CPU utilization, Node-level load average - CPU - Container-level CPU utilization, Node-level load average
- Memory - RSS, swap - Memory - RSS, swap
- Network I/O - bandwidth, packets, drops - Network I/O - bandwidth, packets, drops
@ -53,7 +59,9 @@ Docker EE 2.1 does not collect or expose application level metrics. The followin
- Operating System – file descriptors, open network connections, number of processes/threads - Operating System – file descriptors, open network connections, number of processes/threads
- **Container infrastructure system metrics**: These are application-level metrics derived from the container platform itself. ## Container infrastructure system metrics ##
These are application-level metrics derived from the container platform itself.
- Infrastructure Quorum Leader - Swarm RAFT, etcd, rethink - Infrastructure Quorum Leader - Swarm RAFT, etcd, rethink
- UCP Component health - Healthy / Unhealthy - UCP Component health - Healthy / Unhealthy
@ -66,34 +74,34 @@ To deploy Prometheus on worker nodes in a cluster:
2. Verify that ucp-metrics pods are running on all managers. 2. Verify that ucp-metrics pods are running on all managers.
``` ```
$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide
NAME READY STATUS RESTARTS AGE IP NODE NAME READY STATUS RESTARTS AGE IP NODE
ucp-metrics-hvkr7 3/3 Running 0 4h 192.168.80.66 3a724a-0 ucp-metrics-hvkr7 3/3 Running 0 4h 192.168.80.66 3a724a-0
``` ```
3. Add a Kubernetes node label to one or more workers. Here we add a label with key "ucp-metrics" and value "". 3. Add a Kubernetes node label to one or more workers. Here we add a label with key "ucp-metrics" and value "".
``` ```
$ kubectl label node 3a724a-1 ucp-metrics= $ kubectl label node 3a724a-1 ucp-metrics=
node "test-3a724a-1" labeled node "test-3a724a-1" labeled
``` ```
4. Patch the ucp-metrics DaemonSet's nodeSelector using the same key and value used for the node label. This example shows the key “ucp-metrics” and the value “”. 4. Patch the ucp-metrics DaemonSet's nodeSelector using the same key and value used for the node label. This example shows the key “ucp-metrics” and the value “”.
``` ```
$ kubectl -n kube-system patch daemonset ucp-metrics --type json -p '[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value": {"ucp-metrics": ""}}]' $ kubectl -n kube-system patch daemonset ucp-metrics --type json -p '[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value": {"ucp-metrics": ""}}]'
daemonset "ucp-metrics" patched daemonset "ucp-metrics" patched
``` ```
5. Observe that ucp-metrics pods are running only on the labeled workers. 5. Observe that ucp-metrics pods are running only on the labeled workers.
``` ```
$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide
NAME READY STATUS RESTARTS AGE IP NODE NAME READY STATUS RESTARTS AGE IP NODE
ucp-metrics-88lzx 3/3 Running 0 12s 192.168.83.1 3a724a-1 ucp-metrics-88lzx 3/3 Running 0 12s 192.168.83.1 3a724a-1
ucp-metrics-hvkr7 3/3 Terminating 0 4h 192.168.80.66 3a724a-0 ucp-metrics-hvkr7 3/3 Terminating 0 4h 192.168.80.66 3a724a-0
``` ```
## Configure external Prometheus to scrape metrics from UCP ## Configure external Prometheus to scrape metrics from UCP
@ -103,21 +111,21 @@ To configure your external Prometheus server to scrape metrics from Prometheus i
2. Create a Kubernetes secret containing your bundle’s TLS material. 2. Create a Kubernetes secret containing your bundle’s TLS material.
``` ```
(cd $DOCKER_CERT_PATH && kubectl create secret generic prometheus --from-file=ca.pem --from-file=cert.pem --from-file=key.pem) (cd $DOCKER_CERT_PATH && kubectl create secret generic prometheus --from-file=ca.pem --from-file=cert.pem --from-file=key.pem)
``` ```
3. Create a Prometheus deployment and ClusterIP service using YAML as follows. 3. Create a Prometheus deployment and ClusterIP service using YAML as follows.
On AWS with Kube’s cloud provider configured, you can replace `ClusterIP` with `LoadBalancer` in the service YAML then access the service through the load balancer. If running Prometheus external to UCP, change the following domain for the inventory container in the Prometheus deployment from `ucp-controller.kube-system.svc.cluster.local` to an external domain to access UCP from the Prometheus node. On AWS with Kube’s cloud provider configured, you can replace `ClusterIP` with `LoadBalancer` in the service YAML then access the service through the load balancer. If running Prometheus external to UCP, change the following domain for the inventory container in the Prometheus deployment from `ucp-controller.kube-system.svc.cluster.local` to an external domain to access UCP from the Prometheus node.
``` ```
kubectl apply -f - <<EOF kubectl apply -f - &lt;&lt;EOF
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
name: prometheus name: prometheus
data: data:
prometheus.yaml: | prometheus.yaml: |
global: global:
scrape_interval: 10s scrape_interval: 10s
@ -132,12 +140,12 @@ data:
file_sd_configs: file_sd_configs:
- files: - files:
- /inventory/inventory.json - /inventory/inventory.json
--- ---
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: prometheus name: prometheus
spec: spec:
replicas: 2 replicas: 2
selector: selector:
matchLabels: matchLabels:
@ -187,33 +195,33 @@ spec:
- name: inventory - name: inventory
emptyDir: emptyDir:
medium: Memory medium: Memory
--- ---
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: prometheus name: prometheus
spec: spec:
ports: ports:
- port: 9090 - port: 9090
targetPort: 9090 targetPort: 9090
selector: selector:
app: prometheus app: prometheus
sessionAffinity: ClientIP sessionAffinity: ClientIP
EOF EOF
``` ```
4. Determine the service ClusterIP. 4. Determine the service ClusterIP.
``` ```
$ kubectl get service prometheus $ kubectl get service prometheus
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus ClusterIP 10.96.254.107 <none> 9090/TCP 1h prometheus ClusterIP 10.96.254.107 <none> 9090/TCP 1h
``` ```
5. Forward port 9090 on the local host to the ClusterIP. The tunnel created does not need to be kept alive and is only intended to expose the Prometheus API. 5. Forward port 9090 on the local host to the ClusterIP. The tunnel created does not need to be kept alive and is only intended to expose the Prometheus API.
``` ```
ssh -L 9090:10.96.254.107:9090 ANY_NODE ssh -L 9090:10.96.254.107:9090 ANY_NODE
``` ```
6. Visit `http://127.0.0.1:9090` to explore the UCP metrics being collected by Prometheus. 6. Visit `http://127.0.0.1:9090` to explore the UCP metrics being collected by Prometheus.