Happy middle ground with what the KEP says?
Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>
Kubernetes-commit: 56fc11f3bef9f6af16aa30731050168e732754a2
This change makes us use the generic workqueue throughout the project in
order to improve type safety and readability of the code.
Kubernetes-commit: 6d0ac8c561a7ac66c21e4ee7bd1976c2ecedbf32
Max seats from prioriy & fairness work estimator is now min(0.15 x
nominalCL, nominalCL/handSize)
'Max seats' calculated by work estimator is currently hard coded to 10.
When using lower values for --max-requests-inflight, a single
LIST request taking up 10 seats could end up using all if not most seats in
the priority level. This change updates the default work estimator
config such that 'max seats' is at most 10% of the
maximum concurrency limit for a priority level, with an upper limit of 10.
This ensures seats taken from LIST request is proportional to the total
available seats.
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Kubernetes-commit: d3ef2d4fe95c3ef7b1c606ad01be1183659da391
* apiserver: add latency tracker for priority & fairness queue wait time
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: exclude priority & fairness wait times to SLO/SLI latency metrics
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: update TestLatencyTrackersFrom to check latency from PriorityAndFairnessTracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add helper function observeQueueWaitTime to consolidate metric and latency tracker calls
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: replace time.Now() / time.Since() with clock.Now() / clock.Since() for better testability
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add unit test TestQueueWaitTimeLatencyTracker to validate queue wait times recorded by latency tracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
---------
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Kubernetes-commit: ee18f602523e11a80823a659bed8f70f98a12914
Also make some design changes exposed in testing and review.
Do not remove the ambiguous old metric
`apiserver_flowcontrol_request_concurrency_limit` because reviewers
though it is too early. This creates a problem, that metric can not
keep both of its old meanings. I chose the configured concurrency
limit.
Testing has revealed a design flaw, which concerns the initialization
of the seat demand state tracking. The current design in the KEP is
as follows.
> Adjustment is also done on configuration change … For a newly
> introduced priority level, we set HighSeatDemand, AvgSeatDemand, and
> SmoothSeatDemand to NominalCL-LendableSD/2 and StDevSeatDemand to
> zero.
But this does not work out well at server startup. As part of its
construction, the APF controller does a configuration change with zero
objects read, to initialize its request-handling state. As always,
the two mandatory priority levels are implicitly added whenever they
are not read. So this initial reconfig has one non-exempt priority
level, the mandatory one called catch-all --- and it gets its
SmoothSeatDemand initialized to the whole server concurrency limit.
From there it decays slowly, as per the regular design. So for a
fairly long time, it appears to have a high demand and competes
strongly with the other priority levels. Its Target is higher than
all the others, once they start to show up. It properly gets a low
NominalCL once other levels show up, which actually makes it compete
harder for borrowing: it has an exceptionally high Target and a rather
low NominalCL.
I have considered the following fix. The idea is that the designed
initialization is not appropriate before all the default objects are
read. So the fix is to have a mode bit in the controller. In the
initial state, those seat demand tracking variables are set to zero.
Once the config-producing controller detects that all the default
objects are pre-existing, it flips the mode bit. In the later mode,
the seat demand tracking variables are initialized as originally
designed.
However, that still gives preferential treatment to the default
PriorityLevelConfiguration objects, over any that may be added later.
So I have made a universal and simpler fix: always initialize those
seat demand tracking variables to zero. Even if a lot of load shows
up quickly, remember that adjustments are frequent (every 10 sec) and
the very next one will fully respond to that load.
Also: revise logging logic, to log at numerically lower V level when
there is a change.
Also: bug fix in float64close.
Also, separate imports in some file
Co-authored-by: Han Kang <hankang@google.com>
Kubernetes-commit: feb42277884bc7cfbd6f0bb1d875cc63b1b6caac
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:
- Logging a message only above a certain verbosity threshold without
recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
this matters when using a logging backend which records the verbosity
level.
- Passing a format string with parameters to a logging function that
doesn't do string formatting.
All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.
In some cases it reports false positives, but those can be suppressed with
source code comments.
Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
This reverts commit 83ca74541216405323ddfb67f5f80ad5717da826, reversing
changes made to 1c216c6ec86e700170620fe4c75fa3a2a2817530.
Kubernetes-commit: b0b460921b81b260473d5c393d85beeb5a03e834
This reverts commit 6faa4f001008a5a29476f5722f66430c35f48229, reversing
changes made to 33a2c50bce334467640e016f68cf19e9382ba1a7.
Kubernetes-commit: 8fb33338635565f2f755a4557b94c26039c175d9
In the following code pattern, the log message will get logged with v=0 in JSON
output although conceptually it has a higher verbosity:
if klog.V(5).Enabled() {
klog.Info("hello world")
}
Having the actual verbosity in the JSON output is relevant, for example for
filtering out only the important info messages. The solution is to use
klog.V(5).Info or something similar.
Whether the outer if is necessary at all depends on how complex the parameters
are. The return value of klog.V can be captured in a variable and be used
multiple times to avoid the overhead for that function call and to avoid
repeating the verbosity level.
Kubernetes-commit: 9eaa2dc554e0c3d4485d4c916dfdbc2f517db2e0
Instead of a plain `Mutex`, use an `RWMutex` so that the common
operations can proceed in parallel.
Kubernetes-commit: 58927c1abede11ce7a8a74104328cf823df1b39e
The cmp comparison is relatively expensive (#104821). If we're not
going to log it, we shouldn't make the comparison.
Kubernetes-commit: f9f556dc7061df1dfc8c1628db983eeb97149317
- add plumbing that allows us to estimated "width" of a request
- the default implementation returns 1 as the "width" of all
incoming requests, this is in keeping with the current behavior.
Kubernetes-commit: 9b72eb1929a64b9d5a5234090a631ba312fb4d41
- don't expose the internal states of the apf controller to the caller
- return a boolean, instead of the priority level states
Kubernetes-commit: f20c6cb2d9060920cae9ff5cade1739c7e0b7f7a
When the error is due to the object having been deleted, the
controller does not need to do anything before the coming
notification.
Kubernetes-commit: ef1e2039b5fc7f955ec4f9c636a64aa403cba2ab
Mike Spreitzer
Alvaro Aleman
Abu Kashem
Andrew Sy Kim
leilei.wan.cn
Wojciech Tyczyński
Jordan Liggitt
Patrick Ohly
Justin SB
Maciej Borsz
Manu Gupta
youhonglian
David Eads