Prometheus 2.16.0 introduced a breaking change that prevents
_any_ metrics from being ingested from a target in case
_even a single_ metric has duplicate labels.
See https://github.com/prometheus/prometheus/pull/6664
This becomes a problem for ksm as Kubernetes label naming is less restrictive
than Prometheus label naming, so we have a "sanitization" process that replaces
unsupported characters with an underscore.
A side effect of this process is that it can produce duplicate labels, e.g.
Kubernetes labels "foo.bar", "foo-bar" and "foo_bar" are mapped to the same
Prometheus label "foo_bar".
Based on the discussion in #1141, we've decided to sort and then automatically
rename conflicting labels, so "foo-bar" and "foo_bar" will become
"label_foo_bar_conflict1" and "label_foo_bar_conflict2" correspondingly.
If you'd like to have more control over how this conflict is resolved,
you should consider addressing this on a different level of the stack,
e.g. by standardizing Kubernetes labels using Admission Webhook that
ensures that there are no possible conflicts.
Previously metrics containing the annotations or labels of an object
rendered in the metrics endpoint in a non-deterministic way, as it
relied on the order of the map these were initialized in, since a map
has non-deterministic ordering.
Prometheus caches the byte representation of time-series for
fast-path inserts, so having this be deterministic causes lower cache
misses for Prometheus, hence better performance. Additionally
deterministic ordering allows for better testability.
Since the removal of collector, this introduces both the concept of the
store and the resources instead of collectors that the user passes in.
The user facing logs and flags were not changed as that would be a
regression.
Manuel Rüger
Sherif Abdel-Naby
Sylvain Rabot
Damien Grisonnet
George Miroshnykov
Tariq Ibrahim
Frederic Branczyk
Lili Cosic