In order to better identify, prioritize, and debug webhook latency
issues it is important to have a metric that would point to the resource
it is responsible for. However, it is not possible to have that
dimension in the metrics exposed by Kubernetes because of the unbound
cardinality that such a label would have.
The name of the webhook could be an alternative since it usually
contains some information about the resource that the webhook targets,
however this is not very practical to use in multi-tenants
environments.
A solution for these kind of platform is to tie a specific webhook to a
namespace in order to be able to know which tenant manages it and take
actions depending on that. This is achieveable by leveraging the client
config information of webhooks configured via WebhookConfiguration
resources since Services are namespaced objects.
With these new metrics, users will be able to split the alerting
severity of webhook latency / rejection rate per namespace on top of
being able to do it based on the webhook name. This is key in
environment where administrators don't have control over the webhooks
installed by the various tenants.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
The resource version metrics expose the resource version as a string in the label.
This value can change often and would therefore create huge cardinality.
This changes is to be the number value of the metric itself instead.
- To allow other external Stores, remove the `FamilyByteSlicer` interface
and give access directly to `metric.Family`.
- Move functions present in `pkg/metric/generator.go` to a dedicated package
`generator` in `pkg/metric_generator/generator.go`.
Signed-off-by: cedric lamoriniere <cedric.lamoriniere@datadoghq.com>
Damien Grisonnet
Quentin Bisson
qw1mb0
Kubernetes Prow Robot
Lili Cosic
cedric lamoriniere
Nataly Sheinin