Prometheus 2.16.0 introduced a breaking change that prevents
_any_ metrics from being ingested from a target in case
_even a single_ metric has duplicate labels.
See https://github.com/prometheus/prometheus/pull/6664
This becomes a problem for ksm as Kubernetes label naming is less restrictive
than Prometheus label naming, so we have a "sanitization" process that replaces
unsupported characters with an underscore.
A side effect of this process is that it can produce duplicate labels, e.g.
Kubernetes labels "foo.bar", "foo-bar" and "foo_bar" are mapped to the same
Prometheus label "foo_bar".
Based on the discussion in #1141, we've decided to sort and then automatically
rename conflicting labels, so "foo-bar" and "foo_bar" will become
"label_foo_bar_conflict1" and "label_foo_bar_conflict2" correspondingly.
If you'd like to have more control over how this conflict is resolved,
you should consider addressing this on a different level of the stack,
e.g. by standardizing Kubernetes labels using Admission Webhook that
ensures that there are no possible conflicts.
Kubernetes has a new resource type: `VolumeAttachments`. They provide
helpful information on where a volume is attached and to alert on
unexpected attachment status (for example, differences between
information scraped from node-exporter and kube-state-metrics).
The collector adds a bunch of new metrics. Each VolumeAttachment (ie.,
each CSI-attached volume) will have one of each, so we do not overly
pollute the metrics space. Most metrics are rather unsurprising.
- `kube_volumeattachment_status_attachment_metadata`: provides a
label-like export of the attachment metadata map. Generalizing the
label-conversion function slightly helps at providing this metric.
- `kube_volumeattachment_created`: as VolumeAttachments are
automatically created and we already suffered from duplicate
`VolumeAttachments`, this can be invaluable for debugging
misattachments.
- `kube_volumeattachment_spec_source_persistentvolume`: will only be
generated when the volume source is of `PersistentVolume` type. The
other type `inlineVolumeSpec` is still alpha-level and hard to map to
metrics.
No end-to-end test manifest was added, as `VolumeAttachment`s are
automatically generated when mounting volumes.
Signed-off-by: Jens Erat <email@jenserat.de>
Previously metrics containing the annotations or labels of an object
rendered in the metrics endpoint in a non-deterministic way, as it
relied on the order of the map these were initialized in, since a map
has non-deterministic ordering.
Prometheus caches the byte representation of time-series for
fast-path inserts, so having this be deterministic causes lower cache
misses for Prometheus, hence better performance. Additionally
deterministic ordering allows for better testability.
Since the removal of collector, this introduces both the concept of the
store and the resources instead of collectors that the user passes in.
The user facing logs and flags were not changed as that would be a
regression.
Manuel Rüger
leiwingqueen
Pranshu Srivastava
Sherif Abdel-Naby
Sylvain Rabot
Mikulas (Mike) Dite
Lili Cosic
Quentin Bisson
Joel Whittaker-Smith
Damien Grisonnet
George Miroshnykov
Jens Erat
Ben Wells
Tariq Ibrahim
Frederic Branczyk