When kube-state-metrics does not have the correct roles to list or watch
on a resource, it just logs this error but not actually error out. This
is a problem as the pod never restars and it is hard to catch any
problems as other resource metrics continue to be created correctly. We
only see this error in the noisy kube-state-metrics logs.
This registers two metrics `kube_state_metrics_watch_total` and
`kube_state_metrics_list_total`. With the following labels:
"result" label is the type of action count relates to, error or success.
"resource" label contains the resource <apiVersion.Kind>.
This way we can do a rate alert when kube-state-metrics error rate is
too high.
Example of the metrics:
kube_state_metrics_list_total{resource="*v1.Namespace",result="success"} 1
kube_state_metrics_list_total{resource="*v1.Node",result="error"} 52
kube_state_metrics_watch_total{resource="*v1beta1.Ingress",result="success"} 1
This fixes an issue where it was impossible to specify a collector that
was available but not selected by default.
Instead of checking whether chosen collectors are valid at flag parse
time, this moves the check into the builder, where we can reference it
against the availableStores in the builder. As a bonus, the error
message also prints out a list of available collectors:
```
kube-state-metrics --collectors non-existent-collector
I0618 15:23:34.517532 50719 main.go:88] Using collectors non-existent-collector
F0618 15:23:34.519132 50719 main.go:90] Error: collector non-existent-collector does not exist. Available collectors: persistentvolumeclaims,configmaps,limitranges,nodes,namespaces,persistentvolumes,pods,replicasets,services,cronjobs,deployments,ingresses,horizontalpodautoscalers,jobs,poddisruptionbudgets,secrets,certificatesigningrequests,daemonsets,endpoints,storageclasses,replicationcontrollers,resourcequotas,statefulsets
```
Since the removal of collector, this introduces both the concept of the
store and the resources instead of collectors that the user passes in.
The user facing logs and flags were not changed as that would be a
regression.
This makes the rest of the packages useful to be used in a standalone
library without importing the kube-state-metrics specific collectors.
* Rename collectors -> collector package
* Rename metrics -> metric package
* Add metricFamily mocking in tests to prevent cyclic dependency.
Clear comment --- "TODO : What if not ok?". Because the collector's name
has been checked in CollectorSet.Set function. It will exit if collector' name
is not in DefaultCollectors. As a result, all collectors in Build.enabledCollectors
will find its constructor function. So I think it doesn't exist "not ok" situation.
Signed-off-by: MIBc <lvjiawei@cmss.chinamobile.com>
Instead of iterating a map of enabled collectors, iterate a sorted
slice to achieve determinism across scrapes.
Having a consistent order in the metrics output enables Proemetheus to
apply optimizations during metric parsing and ingestion.
Reenable feature to filter the exposition of metric families by their
name. This is now done at startup time, thereby not slowing down the
critical path.
Dependent on the environment users might or might not want to encode the
responses of kube-state-metrics via gzip independent of the
`Accept-Header` that the client (Prometheus) sends.
This patch introduces a custom client go store, MetricsStore. It is
updated by a corresponding reflector. Instead of generating the
Prometheus custom metrics on demand, metrics are generated on new
Kubernetes object arrival.
In addition this patch replaces most of the logic of the Prometheus
client_golang package by its optimized custom logic to render metrics in
text format.
Next to these main changes this patch also includes:
- Documentation/design: Add performance optimization proposal
- tests/e2e.sh: Ignore "no help text" warnings by promtool
- .travis.yml: Run tests in parallel
As non of the Travis tests depend on each other, there is no reason to
run them in sequence. Instead this patch makes them run in parallel.
This patch adds a simple go benchmark test, injecting Kubernetes objects
and simulating scrape requests. It uses the Kubernetes client-go fake
client. Alongside comes some refactoring of each collectors structure
using informer factories to be compatible with the fake client.
The patch lays the groundwork to make future performance optimizations
comparable with past versions.
How to run test:
`go test -race -bench . -memprofile=mem.out -cpuprofile=cpu.out`
It makes a sentence clear and shows a default value.
```
--collectors string Comma-separated list of collectors to be enabled. Defaults to "services,persistentvolumeclaims,nodes,replicationcontrollers,statefulsets,replicasets,resourcequotas,jobs,namespaces,daemonsets,deployments,limitranges,pods,cronjobs"
```
* Don't set a default --kubeconfig flag value so NewDefaultClientConfigLoadingRules
will pick up the default locations
* Set an ExplicitPath using --kubeconfig flag value
* Remove canonical import path
This either causes issues for the CI or people having the
repo checked out into the correct k8s.io/kube-state-metrics directory
.
* Add Makefile
This commit reworks node metrics in the same manner as deployment
metrics.
The node states are broken out into one time series for each possible
conditional state. One constant metric holds textual meta information
about cluster nodes.
* Rework container restart metric
This commit reworks the container restart metric in the same manner as
deployment metrics. The format remains unchanged to the previous format
as in one metric per container via labels.
* Normalize expected string representation
To be able to compare strings easier when a failure occurs, remove empty
lines and trim whitespace for each line of the expected metrics string.
* Update prometheus client vendoring
* Rework deployment metrics
This change extracts the deployment metrics into their own
deployment collector. The metrics are collected synchronously with
each scrape rather than every 10 seconds.
The `deployment` label replaces the `name` label to avoid name
collisions further up in the monitoring chain.
Frederic Branczyk
Tariq Ibrahim
Lili Cosic
Kubernetes Prow Robot
Miles Bryant
Andreas Kohn
Lv Jiawei
Max Leonard Inden
Andy Xie
Tomas Dohnalek
Matthias Loibl
Raphael Deem
jordigilh
zouyee
joe miller
Kazuki Suda
chenguoyan01
Yecheng Fu
Stijn De Haes
Andrew Davidoff
Dominik Schulz
koudaiii
Ferran Rodenas
slaws
Dominik Schulz
Fabian Reinartz
Daniel Sachse
Sam Ghods