Merge pull request #741 from piosz/metrics-server

Metrics server design proposal

contributors/design-proposals/metrics-server.md

@@ -0,0 +1,89 @@
+Metrics Server
+==============
+
+Resource Metrics API is an effort to provide a first-class Kubernetes API
+(stable, versioned, discoverable, available through apiserver and with client support)
+that serves resource usage metrics for pods and nodes. The use cases were discussed
+and the API was proposed a while ago in
+[another proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md).
+This document describes the architecture and the design of the second part of this effort:
+making the mentioned API available in the same way as the other Kubernetes APIs.
+
+### Scalability limitations ###
+We want to collect up to 10 metrics from each pod and node running in a cluster.
+Starting with Kubernetes 1.6 we support 5000 nodes clusters with 30 pods per node.
+Assuming we want to collect metrics with 1 minute granularity this means:
+```
+10 x 5000 x 30 / 60 = 25000 metrics per second by average
+```
+ 
+Kubernetes apiserver persists all Kubernetes resources in its key-value store [etcd](https://coreos.com/etcd/).
+It’s not able to handle such load. On the other hand metrics tend to change frequently,
+are temporary and in case of loss of them we can collect them during the next housekeeping operation.
+We will store them in memory then. This means that we can’t reuse the main apiserver
+and instead we will introduce a new one - metrics server.
+
+### Current status ###
+The API has been already implemented in Heapster, but users and Kubernetes components
+can only access it through master proxy mechanism and have to decode it on their own.
+Heapster serves the API using go http library which doesn’t offer a number of functionality
+that is offered by Kubernetes API server like authorization/authentication or client generation.
+There is also a prototype of Heapster using [generic apiserver](https://github.com/kubernetes/apiserver) library.
+ 
+The API is in alpha and there is a plan to graduate it to beta (and later to GA),
+but it’s out of the scope of this document.
+
+### Dependencies ###
+In order to make metrics server available for users in exactly the same way
+as the regular Kubernetes API we need a mechanism that redirects requests to `/apis/metrics`
+endpoint from the apiserver to metrics server. The solution for this problem is
+[kube-aggregator](https://github.com/kubernetes/kube-aggregator).
+The effort is on track to be completed for Kubernetes 1.7 release.
+Previously metrics server was blocked on this dependency.
+
+### Design ###
+Metrics server will be implemented in line with
+[Kubernetes monitoring architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md)
+and inspired by [Heapster](https://github.com/kubernetes/heapster).
+It will be a cluster level component which periodically scrapes metrics from all Kubernetes nodes
+served by Kubelet through Summary API. Then metrics will be aggregated, 
+stored in memory (see Scalability limitations) and served in
+[Metrics API](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1alpha1/types.go) format.
+ 
+Metrics server will use apiserver library to implement http server functionality.
+The library offers common Kubernetes functionality like authorization/authentication,
+versioning, support for auto-generated client. To store data in memory we will replace
+the default storage layer (etcd) by introducing in-memory store which will implement
+[Storage interface](https://github.com/kubernetes/apiserver/blob/master/pkg/registry/rest/rest.go).
+ 
+Only the most recent value of each metric will be remembered. If a user needs an access
+to historical data they should either use 3rd party monitoring solution or
+archive the metrics on their own (more details in the mentioned vision).
+ 
+Since the metrics are stored in memory, once the component is restarted, all data are lost.
+This is an acceptable behavior because shortly after the restart the newest metrics will be collected,
+though we will try to minimize the priority of this (see also Deployment).
+
+### Deployment ###
+Since metrics server is prerequisite for a number of Kubernetes components (HPA, scheduler, kubectl top)
+it will run by default in all Kubernetes clusters. Metrics server initiates connections to nodes,
+due to security reasons (our policy allows only connection in the opposite direction) so it has to run on user’s node.
+ 
+There will be only one instance of metrics server running in each cluster. In order to handle
+high metrics volume, metrics server will be vertically autoscaled by
+[addon-resizer](https://github.com/kubernetes/contrib/tree/master/addon-resizer).
+We will measure its resource usage characteristic. Our experience from profiling Heapster shows
+that it scales vertically effectively. If we hit performance limits we will consider scaling it 
+horizontally, though it’s rather complicated and is out of the scope of this doc.
+ 
+Metrics server will be Kubernetes addon, create by kube-up script and managed by
+[addon-manager](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/addon-manager).
+Since there is a number of dependant components, it will be marked as a critical addon.
+In the future when the priority/preemption feature is introduced we will migrate to use this
+proper mechanism for marking it as a high-priority, system component.
+
+### Users migration ###
+In order to make the API usable we will provide auto-generated set of clients.
+Currently the API is being used by a number of components and after we will introduce
+the metrics server we will migrate all of them to use the new path.
+