main_test.go: Add model based test for sharding
In order to ensure a sharded system behaves equal to an unsharded
system, a model based test has been introduced. It scrapes an unsharded
setup and compares its output with the union of a sharded setup
therefore ensuring semantic equality.
When kube-state-metrics does not have the correct roles to list or watch
on a resource, it just logs this error but not actually error out. This
is a problem as the pod never restars and it is hard to catch any
problems as other resource metrics continue to be created correctly. We
only see this error in the noisy kube-state-metrics logs.
This registers two metrics `kube_state_metrics_watch_total` and
`kube_state_metrics_list_total`. With the following labels:
"result" label is the type of action count relates to, error or success.
"resource" label contains the resource <apiVersion.Kind>.
This way we can do a rate alert when kube-state-metrics error rate is
too high.
Example of the metrics:
kube_state_metrics_list_total{resource="*v1.Namespace",result="success"} 1
kube_state_metrics_list_total{resource="*v1.Node",result="error"} 52
kube_state_metrics_watch_total{resource="*v1beta1.Ingress",result="success"} 1
This fixes an issue where it was impossible to specify a collector that
was available but not selected by default.
Instead of checking whether chosen collectors are valid at flag parse
time, this moves the check into the builder, where we can reference it
against the availableStores in the builder. As a bonus, the error
message also prints out a list of available collectors:
```
kube-state-metrics --collectors non-existent-collector
I0618 15:23:34.517532 50719 main.go:88] Using collectors non-existent-collector
F0618 15:23:34.519132 50719 main.go:90] Error: collector non-existent-collector does not exist. Available collectors: persistentvolumeclaims,configmaps,limitranges,nodes,namespaces,persistentvolumes,pods,replicasets,services,cronjobs,deployments,ingresses,horizontalpodautoscalers,jobs,poddisruptionbudgets,secrets,certificatesigningrequests,daemonsets,endpoints,storageclasses,replicationcontrollers,resourcequotas,statefulsets
```
Since the removal of collector, this introduces both the concept of the
store and the resources instead of collectors that the user passes in.
The user facing logs and flags were not changed as that would be a
regression.
This makes the rest of the packages useful to be used in a standalone
library without importing the kube-state-metrics specific collectors.
* Rename collectors -> collector package
* Rename metrics -> metric package
* Add metricFamily mocking in tests to prevent cyclic dependency.
Clear comment --- "TODO : What if not ok?". Because the collector's name
has been checked in CollectorSet.Set function. It will exit if collector' name
is not in DefaultCollectors. As a result, all collectors in Build.enabledCollectors
will find its constructor function. So I think it doesn't exist "not ok" situation.
Signed-off-by: MIBc <lvjiawei@cmss.chinamobile.com>
Instead of iterating a map of enabled collectors, iterate a sorted
slice to achieve determinism across scrapes.
Having a consistent order in the metrics output enables Proemetheus to
apply optimizations during metric parsing and ingestion.
Reenable feature to filter the exposition of metric families by their
name. This is now done at startup time, thereby not slowing down the
critical path.
Dependent on the environment users might or might not want to encode the
responses of kube-state-metrics via gzip independent of the
`Accept-Header` that the client (Prometheus) sends.
This patch introduces a custom client go store, MetricsStore. It is
updated by a corresponding reflector. Instead of generating the
Prometheus custom metrics on demand, metrics are generated on new
Kubernetes object arrival.
In addition this patch replaces most of the logic of the Prometheus
client_golang package by its optimized custom logic to render metrics in
text format.
Next to these main changes this patch also includes:
- Documentation/design: Add performance optimization proposal
- tests/e2e.sh: Ignore "no help text" warnings by promtool
- .travis.yml: Run tests in parallel
As non of the Travis tests depend on each other, there is no reason to
run them in sequence. Instead this patch makes them run in parallel.
This patch adds a simple go benchmark test, injecting Kubernetes objects
and simulating scrape requests. It uses the Kubernetes client-go fake
client. Alongside comes some refactoring of each collectors structure
using informer factories to be compatible with the fake client.
The patch lays the groundwork to make future performance optimizations
comparable with past versions.
How to run test:
`go test -race -bench . -memprofile=mem.out -cpuprofile=cpu.out`
Frederic Branczyk
Tariq Ibrahim
Lili Cosic
Kubernetes Prow Robot
Miles Bryant
Andreas Kohn
Lv Jiawei
Max Leonard Inden
Andy Xie
Tomas Dohnalek
Matthias Loibl
Raphael Deem
jordigilh