Commit Graph

28 Commits

Author SHA1 Message Date
Ben Luddy 681941e62b Don't bypass ResponseWriter wrappers for apiserver healthz errors.
The effective layering of ResponseWriters is today, from outside to
inside, httplog(timeout(audit(metrics(original)))). From
6e3fd91e1aa3259d7bd67e0a65693e346ade347d, calls to http.Error in the
apiserver's root healthz handler use an unwrapped ResponseWriter --
effectively timeout(audit(metrics(original))) -- to avoid logging
stack traces for those requests.

From 0d50c969c587c8a6c16e0962118305ac652c5a6b, the same call to
http.Error receives a completely-unwrapped ResponseWriter. This has
the effect of bypassing not only the httplog wrapper, but also
timeout, audit, and metrics. The timeout wrapper defends against
the (disallowed) use of underyling ResponseWriter after the completion
of its request's ServeHTTP call. Since that defensive behavior is
being bypassed, it's possible for the root healthz handler to panic
when health probes time out.

Instead of continuing to use a wrapper-aware means of disabling stack
traces, this commit adds a new function to httplog that allows
customization of the stack trace logging predicate on a per-request
basis.

Kubernetes-commit: ff849fe8b688606d5173d5ee0213a96cffae23c0
2021-12-14 16:23:36 -05:00
Abu Kashem 6013da2b94 apiserver: wrap ResponseWriter using abstraction
Kubernetes-commit: 0d50c969c587c8a6c16e0962118305ac652c5a6b
2021-09-20 11:21:56 -04:00
Lukasz Szaszkiewicz 279d11fb1e readyz signals when the handler succeeds for the first time.
Co-authored-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

Kubernetes-commit: ca108d109d55e927c292e3e558fc4f761a3a4e7c
2021-06-30 11:04:44 +02:00
Dr. Stefan Schimanski 4cf3f7725b apiserver: fix healthz vs. livez vs. readyz log output
Kubernetes-commit: 480b4c74b858a9caac0b33cb4d9a1ab30ae65cf4
2020-09-15 15:46:02 +02:00
ialidzhikov f1d71be938 Fix panic on /readyz
Signed-off-by: ialidzhikov <i.alidjikov@gmail.com>

Kubernetes-commit: 1f06c88dd38f17b1687d83208572a0093c2b2905
2020-07-31 14:30:28 +03:00
Maciej Borsz ea82586fc5 Wait for all informers to sync in /readyz.
Kubernetes-commit: 3f680002038c81b02541e4cc4bd7c019b7e7d6e6
2020-06-18 15:21:12 +02:00
David Eads 75c70e8856 make log output of failed healthchecks more focused
Kubernetes-commit: e87f62dcd57fb455135e8cd6fb7e604afbc2e1c4
2020-06-16 15:35:26 -04:00
Jordan Liggitt c02923d8c7 apiserver: increment metric for deprecated API use
Kubernetes-commit: e4bb1daecf36aac3051d36a20dfdf7ea3050de58
2020-04-30 12:21:29 -04:00
Davanum Srinivas 5879417a28 switch over k/k to use klog v2
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: 442a69c3bdf6fe8e525b05887e57d89db1e2f3a5
2020-04-17 15:25:06 -04:00
Jonathan Tomer 114cbd8352 healthz: instrument root healthz requests for metrics
Kubernetes-commit: 9dadafa8979619495317f166bf8bd4803ec89452
2019-10-07 15:21:17 -07:00
Han Kang 050d1b223c add /livez as a liveness endpoint for kube-apiserver
go fmt

make func private

refactor config_test

Two primary refactorings:

1. config test checkPath method is now each a distinct test
run (which makes it easier to see what is actually failing)

2. TestNewWithDelegate's root path check now parses the json output and
does a comparison against a list of expected paths (no more whitespace
and ordering issues when updating this test, yay).

go fmt

modify and simplify existing integration test for readyz/livez

simplify integration test

set default rbac policy rules for livez

rename a few functions and the entrypoint command line argument (and etcetera)

simplify interface for installing readyz and livez and make auto-register completion a bootstrapped check

untangle some of the nested functions, restructure the code

Kubernetes-commit: aa1b2d6d35c92a31be17357fc66cfc1eca8a67e0
2019-08-26 16:57:25 -07:00
Han Kang 2b0c93afef rename healthz methodNames to be more consistent w/ present day usages
Kubernetes-commit: 2e23788fda86c68e7f17cf0b66ee1017594c1055
2019-08-13 12:42:13 -07:00
Dr. Stefan Schimanski c4286f7b60 apiserver: store httplog in context
Kubernetes-commit: e848eaa566b750ad5e5fd26a64eecfc883f936f3
2019-06-14 13:09:13 +02:00
Dr. Stefan Schimanski 41c11d856b apiserver: don't log stack trace on /healthz error
Kubernetes-commit: 6e3fd91e1aa3259d7bd67e0a65693e346ade347d
2019-06-14 11:37:48 +02:00
Han Kang 7fd71e31ef add readyz endpoint for kube-apiserver readiness checks
add startup sequence duration and readyz endpoint

add rbac bootstrapping policy for readyz

add integration test around grace period and readyz

rename startup sequence duration flag

copy health checks to fields

rename health-check installed boolean, refactor clock injection logic

cleanup clock injection code

remove todo about poststarthook url registration from healthz

Kubernetes-commit: 54dcf5c9c46fc4782d4861936309349b5a71a1ac
2019-05-30 11:19:49 -07:00
Tim Allclair b1e85edc1f Avoid the default server mux
Kubernetes-commit: dd5c8e14fd2a1715be7795c37fb5b92478867494
2019-05-16 17:31:16 -07:00
David Eads 121f70dcfd provide verbose output when health check fails
Kubernetes-commit: 3874e43550385c0f4adddc4ccbedf12478233692
2019-03-08 15:49:29 -05:00
Davanum Srinivas 0202761ce2 Ensure we set a content-type for healthz
Change-Id: I453b1433c69bf26c28da873dbdd1ac25006b8d60

Co-Authored-By: Tim Allclair (St. Clair) <tallclair@google.com>

Kubernetes-commit: eedb70355ad59f9d7054dba198a8352554274e0a
2019-01-03 10:05:58 -05:00
Davanum Srinivas b87560e3bd Log Health Check failure at Verbosity Level - 4
Our default recommendation as well as our CI default is 4, so we should
be able to see the health check errors at the same level 4 itself.

Change-Id: I388f945d3522d1f63f0a85f35e765c2d4f2c548a

Kubernetes-commit: 3d6b3857b29b9c738f258344781ce4a9b40b9a0b
2019-01-06 21:32:57 -05:00
Han Kang f61020971c add ability to exclude health checks from failing healthz by passing in a query param
Kubernetes-commit: f1f1bc83fd07450a191ecf94b945f1b772d1dc7c
2018-11-12 16:17:36 -08:00
Davanum Srinivas 2710b17b80 Move from glog to klog
- Move from the old github.com/golang/glog to k8s.io/klog
- klog as explicit InitFlags() so we add them as necessary
- we update the other repositories that we vendor that made a similar
change from glog to klog
  * github.com/kubernetes/repo-infra
  * k8s.io/gengo/
  * k8s.io/kube-openapi/
  * github.com/google/cadvisor
- Entirely remove all references to glog
- Fix some tests by explicit InitFlags in their init() methods

Change-Id: I92db545ff36fcec83afe98f550c9e630098b3135

Kubernetes-commit: 954996e231074dc7429f7be1256a579bedd8344c
2018-11-09 13:49:10 -05:00
Han Kang 85a1725c91 fix healthz checkerNames test so that it tests against the expected output
Kubernetes-commit: 0623f630ab37ad75961bf836195e190e6bcf560e
2018-11-07 09:26:11 -08:00
Jordan Liggitt 6c34ac4aa5 Add healthz check to ensure logging is not blocked
Kubernetes-commit: b7b4b84afe4405cde976ceeeccb62acecac1c4f0
2018-06-09 17:32:14 -04:00
Jacob Tanenbaum 6a0cc50341 Add InstallPathHandler which allows for more then one path to be associated with health checking.
Currently it is only possible to have one group of checks which must all pass for the handler to report success.
Allowing multiple paths for these checks allows use of the same machinery for other kinds of checks, i.e. readiness.

Kubernetes-commit: 2082a0f42851c47620ce31f257dcb5536abae014
2018-05-10 16:21:39 -04:00
Henrik Schmidt 9493c48653 Log error when a healthz check fails
Kubernetes-commit: 1bcfe909125acc567258d4937fc2c08206d14d08
2017-09-26 12:58:12 +02:00
Morgan Bauer c5c5445b49 basic logging for healthz installer
- InstallHandler is the public interface through which all interaction
   occurs.
 - It is good to know whether the default ping is occurring to know due
   to manual installation or automatic installation.
 - It is good to know how many handlers are installed to see whether
   code changes are taking effect.
 - It is good to know the names of the handlers that are installed to
   make sure that a handler a user thinks is installed is being
   installed at runtime.
 - Print all the checkers once

Kubernetes-commit: efa66227d4fbcfad9fec21755b898f5d10d3344c
2017-09-03 14:04:11 +00:00
deads2k 025f3f79e4 redact detailed errors from healthz and expose in default policy 2017-02-27 15:10:33 -05:00
Dr. Stefan Schimanski dc90e9a003 genericapiserver: move packages
Towards a sane k8s.io/apiserver package structure.
2017-01-19 10:27:27 -05:00