so that aggregated-apiservers can also take advantage. discovered by e2e tests with feature enabled
Kubernetes-commit: c9b34884004079ed3f184b475f7408984f9226f4
Also make some design changes exposed in testing and review.
Do not remove the ambiguous old metric
`apiserver_flowcontrol_request_concurrency_limit` because reviewers
though it is too early. This creates a problem, that metric can not
keep both of its old meanings. I chose the configured concurrency
limit.
Testing has revealed a design flaw, which concerns the initialization
of the seat demand state tracking. The current design in the KEP is
as follows.
> Adjustment is also done on configuration change … For a newly
> introduced priority level, we set HighSeatDemand, AvgSeatDemand, and
> SmoothSeatDemand to NominalCL-LendableSD/2 and StDevSeatDemand to
> zero.
But this does not work out well at server startup. As part of its
construction, the APF controller does a configuration change with zero
objects read, to initialize its request-handling state. As always,
the two mandatory priority levels are implicitly added whenever they
are not read. So this initial reconfig has one non-exempt priority
level, the mandatory one called catch-all --- and it gets its
SmoothSeatDemand initialized to the whole server concurrency limit.
From there it decays slowly, as per the regular design. So for a
fairly long time, it appears to have a high demand and competes
strongly with the other priority levels. Its Target is higher than
all the others, once they start to show up. It properly gets a low
NominalCL once other levels show up, which actually makes it compete
harder for borrowing: it has an exceptionally high Target and a rather
low NominalCL.
I have considered the following fix. The idea is that the designed
initialization is not appropriate before all the default objects are
read. So the fix is to have a mode bit in the controller. In the
initial state, those seat demand tracking variables are set to zero.
Once the config-producing controller detects that all the default
objects are pre-existing, it flips the mode bit. In the later mode,
the seat demand tracking variables are initialized as originally
designed.
However, that still gives preferential treatment to the default
PriorityLevelConfiguration objects, over any that may be added later.
So I have made a universal and simpler fix: always initialize those
seat demand tracking variables to zero. Even if a lot of load shows
up quickly, remember that adjustments are frequent (every 10 sec) and
the very next one will fully respond to that load.
Also: revise logging logic, to log at numerically lower V level when
there is a change.
Also: bug fix in float64close.
Also, separate imports in some file
Co-authored-by: Han Kang <hankang@google.com>
Kubernetes-commit: feb42277884bc7cfbd6f0bb1d875cc63b1b6caac
This change enables hot reload of encryption config file when api server
flag --encryption-provider-config-automatic-reload is set to true. This
allows the user to change the encryption config file without restarting
kube-apiserver. The change is detected by polling the file and is done
by using fsnotify watcher. When file is updated it's process to generate
new set of transformers and close the old ones.
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>
Kubernetes-commit: 761b7822fca569d475f782b135ef433e5b014147
Co-authored-by: Alexander Zielenski <zielenski@google.com>
Co-authored-by: Joe Betz <jpbetz@google.com>
Kubernetes-commit: e7d83a1fb7b3e4f6a75ed73bc6e410946e12ad9f
This change adds a flag --encryption-provider-config-automatic-reload
which will be used to drive automatic reloading of the encryption
config at runtime. While this flag is set to true, or when KMS v2
plugins are used without KMS v1 plugins, the /healthz endpoints
associated with said plugins are collapsed into a single endpoint at
/healthz/kms-providers - in this state, it is not possible to
configure exclusions for specific KMS providers while including the
remaining ones - ex: using /readyz?exclude=kms-provider-1 to exclude
a particular KMS is not possible. This single healthz check handles
checking all configured KMS providers. When reloading is enabled
but no KMS providers are configured, it is a no-op.
k8s.io/apiserver does not support dynamic addition and removal of
healthz checks at runtime. Reloading will instead have a single
static healthz check and swap the underlying implementation at
runtime when a config change occurs.
Signed-off-by: Monis Khan <mok@microsoft.com>
Kubernetes-commit: 22e540bc48d9bf698c4f381ccb56ed57dea0dae2
* add superuser fallback to authorizer
* change the order of authorizers
* change the order of authorizers
* remove the duplicate superuser authorizer
* add integration test for superuser permissions
Kubernetes-commit: f86acbad68baf1a99d6fa153f6f0cdc7b93932e4
Default api server manifest whose liveness check looks like:
"/livez?exclude=etcd&exclude=kms-provider-0&exclude=kms-provider-1"
Which causes spurious messages in apiserver logs every 10 mins:
```
W1017 00:03:39.938956 9 healthz.go:256] cannot exclude some health checks, no health checks are installed matching "kms-provider-0","kms-provider-1"
```
Let's not log excessive messages especially at warning level. We should
do this at a higher level (6 instead of 4).
NOTE: we don't change the message returned to the http request, we keep
that as-is (does not change on log level)
Also see:
https://github.com/aws/eks-distro/blob/v1-19-eks-12/projects/kubernetes/kubernetes/1-19/patches/0016-EKS-PATCH-apiserver-healthz-upper-log-verbosity-for-.patch
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: 20de240d5bdb7fc50de3fe9b8cdd95f81bf47034
receiver name obj should be consistent with previous receiver name s for SimpleStream
error var hookNotFinished should have name of the form errFoo
Kubernetes-commit: ae385ee874a81cd01ee4fef98efc1bd5c219c9b7
This change updates the API server code to load the encryption
config once at start up instead of multiple times. Previously the
code would set up the storage transformers and the etcd healthz
checks in separate parse steps. This is problematic for KMS v2 key
ID based staleness checks which need to be able to assert that the
API server has a single view into the KMS plugin's current key ID.
Signed-off-by: Monis Khan <mok@microsoft.com>
Kubernetes-commit: f507bc255382b2e2095351053bc17e74f7100d35
Tim Allclair
Monis Khan
Jordan Liggitt
Jefftree
David Ashpole
Cici Huang
Mike Spreitzer
Alexander Zielenski
Nilekh Chaudhari
Joseph Anttila Hall
Anish Ramasekar
Paweł Banaszewski
Andrew Sy Kim
SataQiu
Nick Turner
Mohammad Zuber Khan
Davanum Srinivas
Abu Kashem
Maciej Wyrzuc
twilight0620
Paco Xu
Jiahui Feng
Han Kang
Manish Kumar
Wojciech Tyczyński
xueqzhan