The fact that we're building the OpenAPI using the proto.Models is an
implementation detail that we shouldn't have to expose. Since we're
going to change the way this is transformed, let's first hide it behind
the common NewTypeConverter so that the next change is transparent.
This will also enable other clean-ups like hiding the gvkParser which
shouldn't be exposed and prevent some refactoring.
Kubernetes-commit: a7ab6b86db83e31ff599e4d21a065f6845fb93dd
Serving profiling information can leak information or expose the
apiserver to possible DoS attacks. Serving on a UDS is more secure
though slightly less convenient. One can't use `go tool pprof` directly
against the socket since it's not supported, but can either run a proxy
to copy from the socket over to http, or use `curl --unix-socket` to
download the profile and then use `go tool pprof`.
Kubernetes-commit: 667599b0ddfad8ba760d3bbfe006aae0d8f7dec6
so that aggregated-apiservers can also take advantage. discovered by e2e tests with feature enabled
Kubernetes-commit: c9b34884004079ed3f184b475f7408984f9226f4
Also make some design changes exposed in testing and review.
Do not remove the ambiguous old metric
`apiserver_flowcontrol_request_concurrency_limit` because reviewers
though it is too early. This creates a problem, that metric can not
keep both of its old meanings. I chose the configured concurrency
limit.
Testing has revealed a design flaw, which concerns the initialization
of the seat demand state tracking. The current design in the KEP is
as follows.
> Adjustment is also done on configuration change … For a newly
> introduced priority level, we set HighSeatDemand, AvgSeatDemand, and
> SmoothSeatDemand to NominalCL-LendableSD/2 and StDevSeatDemand to
> zero.
But this does not work out well at server startup. As part of its
construction, the APF controller does a configuration change with zero
objects read, to initialize its request-handling state. As always,
the two mandatory priority levels are implicitly added whenever they
are not read. So this initial reconfig has one non-exempt priority
level, the mandatory one called catch-all --- and it gets its
SmoothSeatDemand initialized to the whole server concurrency limit.
From there it decays slowly, as per the regular design. So for a
fairly long time, it appears to have a high demand and competes
strongly with the other priority levels. Its Target is higher than
all the others, once they start to show up. It properly gets a low
NominalCL once other levels show up, which actually makes it compete
harder for borrowing: it has an exceptionally high Target and a rather
low NominalCL.
I have considered the following fix. The idea is that the designed
initialization is not appropriate before all the default objects are
read. So the fix is to have a mode bit in the controller. In the
initial state, those seat demand tracking variables are set to zero.
Once the config-producing controller detects that all the default
objects are pre-existing, it flips the mode bit. In the later mode,
the seat demand tracking variables are initialized as originally
designed.
However, that still gives preferential treatment to the default
PriorityLevelConfiguration objects, over any that may be added later.
So I have made a universal and simpler fix: always initialize those
seat demand tracking variables to zero. Even if a lot of load shows
up quickly, remember that adjustments are frequent (every 10 sec) and
the very next one will fully respond to that load.
Also: revise logging logic, to log at numerically lower V level when
there is a change.
Also: bug fix in float64close.
Also, separate imports in some file
Co-authored-by: Han Kang <hankang@google.com>
Kubernetes-commit: feb42277884bc7cfbd6f0bb1d875cc63b1b6caac
This change enables hot reload of encryption config file when api server
flag --encryption-provider-config-automatic-reload is set to true. This
allows the user to change the encryption config file without restarting
kube-apiserver. The change is detected by polling the file and is done
by using fsnotify watcher. When file is updated it's process to generate
new set of transformers and close the old ones.
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>
Kubernetes-commit: 761b7822fca569d475f782b135ef433e5b014147
Co-authored-by: Alexander Zielenski <zielenski@google.com>
Co-authored-by: Joe Betz <jpbetz@google.com>
Kubernetes-commit: e7d83a1fb7b3e4f6a75ed73bc6e410946e12ad9f
This change adds a flag --encryption-provider-config-automatic-reload
which will be used to drive automatic reloading of the encryption
config at runtime. While this flag is set to true, or when KMS v2
plugins are used without KMS v1 plugins, the /healthz endpoints
associated with said plugins are collapsed into a single endpoint at
/healthz/kms-providers - in this state, it is not possible to
configure exclusions for specific KMS providers while including the
remaining ones - ex: using /readyz?exclude=kms-provider-1 to exclude
a particular KMS is not possible. This single healthz check handles
checking all configured KMS providers. When reloading is enabled
but no KMS providers are configured, it is a no-op.
k8s.io/apiserver does not support dynamic addition and removal of
healthz checks at runtime. Reloading will instead have a single
static healthz check and swap the underlying implementation at
runtime when a config change occurs.
Signed-off-by: Monis Khan <mok@microsoft.com>
Kubernetes-commit: 22e540bc48d9bf698c4f381ccb56ed57dea0dae2
* add superuser fallback to authorizer
* change the order of authorizers
* change the order of authorizers
* remove the duplicate superuser authorizer
* add integration test for superuser permissions
Kubernetes-commit: f86acbad68baf1a99d6fa153f6f0cdc7b93932e4
Default api server manifest whose liveness check looks like:
"/livez?exclude=etcd&exclude=kms-provider-0&exclude=kms-provider-1"
Which causes spurious messages in apiserver logs every 10 mins:
```
W1017 00:03:39.938956 9 healthz.go:256] cannot exclude some health checks, no health checks are installed matching "kms-provider-0","kms-provider-1"
```
Let's not log excessive messages especially at warning level. We should
do this at a higher level (6 instead of 4).
NOTE: we don't change the message returned to the http request, we keep
that as-is (does not change on log level)
Also see:
https://github.com/aws/eks-distro/blob/v1-19-eks-12/projects/kubernetes/kubernetes/1-19/patches/0016-EKS-PATCH-apiserver-healthz-upper-log-verbosity-for-.patch
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: 20de240d5bdb7fc50de3fe9b8cdd95f81bf47034
receiver name obj should be consistent with previous receiver name s for SimpleStream
error var hookNotFinished should have name of the form errFoo
Kubernetes-commit: ae385ee874a81cd01ee4fef98efc1bd5c219c9b7
This change updates the API server code to load the encryption
config once at start up instead of multiple times. Previously the
code would set up the storage transformers and the etcd healthz
checks in separate parse steps. This is problematic for KMS v2 key
ID based staleness checks which need to be able to assert that the
API server has a single view into the KMS plugin's current key ID.
Signed-off-by: Monis Khan <mok@microsoft.com>
Kubernetes-commit: f507bc255382b2e2095351053bc17e74f7100d35
- add feature gate
- add encrypted object and run generated_files
- generate protobuf for encrypted object and add unit tests
- move parse endpoint to util and refactor
- refactor interface and remove unused interceptor
- add protobuf generate to update-generated-kms.sh
- add integration tests
- add defaulting for apiVersion in kmsConfiguration
- handle v1/v2 and default in encryption config parsing
- move metrics to own pkg and reuse for v2
- use Marshal and Unmarshal instead of serializer
- add context for all service methods
- check version and keyid for healthz
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
Kubernetes-commit: f19f3f409938ff9ac8a61966e47fbe9c6075ec90
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
v1.43.0 marked grpc.WithInsecure() deprecated so this commit moves to use
what is the recommended replacement:
grpc.WithTransportCredentials(insecure.NewCredentials())
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Kubernetes-commit: 2c8bfad9106039aa15233b5bf7282b25a7b7e0a0
Make the test accept all the legitimate outcomes.
Expand the explanation of how TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter/priority_level_concurrency_is_set_to_1,_queue_length_is_1,_first_request_should_time_out_and_second_(enqueued)_request_should_time_out_as_well is supposed to work.
Expand debug information that is available when the test fails.
Kubernetes-commit: 1f450695ffd5b2d028c87328b8b32630a8052129
Members are not used in (waiting,executing) pairs, so stopped
using the wrapper that adds such pairing.
Kubernetes-commit: cd33c7cf2260b351dd345497223a944e80bc7b61
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:
- Logging a message only above a certain verbosity threshold without
recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
this matters when using a logging backend which records the verbosity
level.
- Passing a format string with parameters to a logging function that
doesn't do string formatting.
All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.
In some cases it reports false positives, but those can be suppressed with
source code comments.
Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
This commit includes all the changes needed for APIServer. Instead of modifying the existing signatures for the methods which either generate or return stopChannel, we generate a context from the channel and use the generated context to be passed to the controllers which are started in APIServer. This ensures we don't have to touch APIServer dependencies.
Kubernetes-commit: 8b84a793b39fed2a62af0876b2eda461a68008c9
Since flowDistinguisher may hold data identifying a user accessing the
cluster this can be a source of a PII leak.
Kubernetes-commit: 94c92f78e5b02c27502f3b9d59b4e194e476a6f4
When an envelope transformer calls out to KMS (for instance), it will be
very helpful to pass a `context.Context` to allow for cancellation. This
patch does that, while passing the previously-expected additional data
via a context value.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 27312feb9983c18d1daf00afba788727d024cdd0
In the following code pattern, the log message will get logged with v=0 in JSON
output although conceptually it has a higher verbosity:
if klog.V(5).Enabled() {
klog.Info("hello world")
}
Having the actual verbosity in the JSON output is relevant, for example for
filtering out only the important info messages. The solution is to use
klog.V(5).Info or something similar.
Whether the outer if is necessary at all depends on how complex the parameters
are. The return value of klog.V can be captured in a variable and be used
multiple times to avoid the overhead for that function call and to avoid
repeating the verbosity level.
Kubernetes-commit: 9eaa2dc554e0c3d4485d4c916dfdbc2f517db2e0
The effective layering of ResponseWriters is today, from outside to
inside, httplog(timeout(audit(metrics(original)))). From
6e3fd91e1aa3259d7bd67e0a65693e346ade347d, calls to http.Error in the
apiserver's root healthz handler use an unwrapped ResponseWriter --
effectively timeout(audit(metrics(original))) -- to avoid logging
stack traces for those requests.
From 0d50c969c587c8a6c16e0962118305ac652c5a6b, the same call to
http.Error receives a completely-unwrapped ResponseWriter. This has
the effect of bypassing not only the httplog wrapper, but also
timeout, audit, and metrics. The timeout wrapper defends against
the (disallowed) use of underyling ResponseWriter after the completion
of its request's ServeHTTP call. Since that defensive behavior is
being bypassed, it's possible for the root healthz handler to panic
when health probes time out.
Instead of continuing to use a wrapper-aware means of disabling stack
traces, this commit adds a new function to httplog that allows
customization of the stack trace logging predicate on a per-request
basis.
Kubernetes-commit: ff849fe8b688606d5173d5ee0213a96cffae23c0
In 18177e2bdeafbddeb3d66fec0b8cb88794cd69ff, PostGoRestful field has
been renamed to NonGoRestfulMux, but the documentation change did not
follow.
This commit fixes that to avoid potential confusion.
Signed-off-by: Mateusz Gozdek <mgozdek@microsoft.com>
Kubernetes-commit: 36ede8de945adcc06bde21c49dc157e9c741e0d2
This PR removes Serve function and uses all required places
ServeWithListenerStopped which takes place new Serve function.
This function returns ListenerStopped channel can be used to drain
requests before shutting down the server.
Kubernetes-commit: a8d2b3a7926394b1c53621804cdeb93e4a61b7c8
Print information from both the original path registration and the new
path registration stack traces when encountering a duplicate. This helps
the developer determine where the duplication is coming from and makes
it much easier to resolve.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
Kubernetes-commit: 04aa8f9dcdbc575fde37e25e45315359b0aa1ca6
the new field exists primarily to avoid returning a 404 response when a resource actually exists but we haven't installed the path to a handler.
it is exposed for easier composition of the individual servers.
the primary users of this field are the WithMuxCompleteProtection filter and the NotFoundHandler.
Kubernetes-commit: ddfbb5d2bb57ee44b3e10f0b58f9cc7001f55802
This is a Config specialized for a GroupResource.
It will support generating new resource-specific metrics.
Kubernetes-commit: 85bcd243aa3c8769a5904a1aea44ce704f5e7174
This reverts commit fc5863b8b276e0789f717859e8cce58d7d060181, reversing
changes made to 027fe2554fd18343b8be39eddc8ff6570a6c390f.
Kubernetes-commit: f9f08725907b7db2104ee5fe9f82ab0752726533
This reverts commit edb0a72cff0e43bab72a02cada8486d562ee1cd5, reversing
changes made to 80feff6f407be9f0898c449ba3f9d4d013f05ec9.
Kubernetes-commit: 8844d3092a46a40915b4df6e3b9944d5081f8268
add a new mode for graceful termination with the new server run option
'shutdown-send-retry-after'
- shutdown-send-retry-after=true: we initiate shutdown of the
HTTP Server when all in-flight request(s) have been drained. during
this window all incoming requests are rejected with status code
429 and the following response headers:
- 'Retry-After: N' - client should retry after N seconds
- 'Connection: close' - tear down the TCP connection
- shutdown-send-retry-after=false: we initiate shutdown of the
HTTP Server as soon as shutdown-delay-duration has elapsed. This
is in keeping with the current behavior.
Kubernetes-commit: 3182b69e970bd1fd036ff839fdf811f14e790244
DynamicFileCAContent and DynamicCertKeyPairContent used periodical job
to check whether the file content has changed, leading to 1 minute of
delay in worst case. This patch improves it by leveraging fsnotify
watcher. The content change will be reflected immediately.
Kubernetes-commit: 3cfe3d048ff37c1c6994d131ed8557f3c8bddc8a
Abu Kashem