Commit Graph

56 Commits

Author SHA1 Message Date
Abu Kashem 7c07064678 apf: fix test flake
Kubernetes-commit: fa0952ee778aaf7df3e593305575f5d121b55cf1
2021-03-29 16:09:49 -04:00
David Eads 4c81a6672b Enable, rate limit, and test APF controller fights
Using real time.

Kubernetes-commit: 80ff06fe846c5e565e0cbd4b70f5f1e8a4ea7295
2020-12-02 16:50:05 -05:00
yoyinzyc 1a8abfc56f add context to metrics in util/flowcontrol.
Kubernetes-commit: 57d0bc301a017c41d890baee0a3a287f448c664d
2020-12-16 17:08:43 -08:00
Adhityaa Chandrasekar ec9a97b26e flowcontrol: make TestConfigConsumer less time-consuming
Signed-off-by: Adhityaa Chandrasekar <adtac@google.com>

Kubernetes-commit: c1f3e90969d38f5b8410b67f0640d5df0d098e6b
2021-02-02 11:20:18 +00:00
Mike Spreitzer b2c3c4f86c Tweak up TestSampler in response to review
Kubernetes-commit: 736a87a4ee516a3feca0a79a4c48806ad4916aa3
2021-01-24 23:38:48 -05:00
Mike Spreitzer 39f5df86ff Added comment explaining the new unit test
Kubernetes-commit: 42f8d4e28fc05f3532f5cf7d2294049e3314896a
2021-01-18 00:57:09 -05:00
Mike Spreitzer 60021e2fed Explain warnings in new test
Add info message that explains what warnings are deliberate and expected.

Kubernetes-commit: 194c22fed54ba89d985085412795a9850bedeaf3
2021-01-06 12:56:16 -05:00
Mike Spreitzer 71c9d22768 Add unit test for sample-and-watermark histograms
Kubernetes-commit: 5fe48be506fe2b946d4db9cd7976ad0c4d4e7e1a
2021-01-05 23:31:06 -05:00
pacoxu 2343af38bb fix all keps links 404 for kep folder migration
Signed-off-by: pacoxu <paco.xu@daocloud.io>

Kubernetes-commit: a10bdfed09e96f3c2703454caafa2c884914e064
2021-01-14 14:03:21 +08:00
Abu Kashem 84ce907827 fix apf controller unit test
- don't expose the internal states of the apf controller to the caller
- return a boolean, instead of the priority level states

Kubernetes-commit: f20c6cb2d9060920cae9ff5cade1739c7e0b7f7a
2021-01-13 11:06:17 -05:00
Mike Spreitzer 16288a4d61 move all variables in sampleAndWaterMarkHistograms::innerSet
to tiptoe around https://github.com/golang/go/issues/43570 for #97685

Kubernetes-commit: 611184aa59d0cd40466bc3bc4b40a3712a038171
2021-01-08 13:32:38 -05:00
Mike Spreitzer 41788f53e2 Relax apf_controller's reaction to certain errors
When the error is due to the object having been deleted, the
controller does not need to do anything before the coming
notification.

Kubernetes-commit: ef1e2039b5fc7f955ec4f9c636a64aa403cba2ab
2020-12-17 10:46:14 -05:00
Abu Kashem eea0d66fcd clean up executing request on panic
Kubernetes-commit: 13cedca0eb5337b13e5176983ea5e784ec38df22
2020-12-10 12:57:21 -05:00
Abu Kashem 59a718ed18 fix the deadlock in priority and fairness config controller
Kubernetes-commit: 7114319b3e66e331250c116ac926f1a57352072f
2020-12-15 12:19:56 -05:00
Mike Spreitzer a011399e98 Define TestableConfig in k/apiserver/pkg/util/flowcontrol
Collect the parameters of newTestableController into a named type.

Also tolerate the surprising situation in which a request's user
groups include neither `system:authenticated` nor
`system:unauthenticated` --- because this is observed to happen in
some tests.

Also a few other minor fixups.

Kubernetes-commit: 10df6d459b5bca7b42471f9409182417fbc3f253
2020-12-16 18:58:45 -05:00
Mike Spreitzer 2ceb0fcbe8 Create OWNERS for most of the API Priority and Fairness impl
Kubernetes-commit: 23de8061fbb63928c62aa6020b443a62491a0a1d
2020-12-11 17:54:59 -05:00
corehello b6f30b469b Fix typo for ServiceAccount
Kubernetes-commit: c20b2c8cd79f9e7dafa86fb4245dee9dd2d795c7
2020-12-02 01:38:46 +08:00
Ken Sipe 0da065589a fix S1023 redundant return statement
Signed-off-by: Ken Sipe <kensipe@gmail.com>

Kubernetes-commit: 1d795002919675b8cf3bf462b6543fef843b5b54
2020-06-25 17:29:42 -05:00
Adhityaa Chandrasekar 43cf7154be APF matching: fallback to catch-all if nothing matches
Signed-off-by: Adhityaa Chandrasekar <adtac@google.com>

Kubernetes-commit: 794a3ec0997caffa116c2e6f78feb53ebfca6052
2020-11-13 03:58:07 +00:00
yue9944882 a4a3fc9b87 APF: graduate API and types to beta
Signed-off-by: Adhityaa Chandrasekar <adtac@google.com>

Kubernetes-commit: 849be447f563fc93a27a0827fb1185b885b57114
2020-11-04 16:33:14 +08:00
Mike Spreitzer 70bbff23ac Fixed quantization and made monotic time reversal not panic
Changed quantization to use monotonic clock readings.

Stopping panicing when monotonic clock readings go backwards because
https://github.com/kubernetes/kubernetes/issues/96459 reported it
happening.

Re-organized sampleAndWaterMarkHistograms::innerSet to `defer` the
Unlock call, because that is the preferred pattern.

Kubernetes-commit: 0809220fd04d26892f0fcfd3bf54cc4baeda22da
2020-11-12 15:17:07 -05:00
Adhityaa Chandrasekar 8b21b5725d APF metrics: set StabilityLevel to ALPHA
Signed-off-by: Adhityaa Chandrasekar <adtac@google.com>

Kubernetes-commit: b16f36b251ddbfef5f12fed58640de53512631f0
2020-11-05 15:35:39 +00:00
Adhityaa Chandrasekar ebe254b2e6 APF: use snake_case in metric labels
Signed-off-by: Adhityaa Chandrasekar <adtac@google.com>

Kubernetes-commit: f9d57a8d5db3e58f79a1b1958d80c049c63d6cde
2020-11-04 22:19:52 +00:00
yue9944882 5474822749 fixes max-min fairness
Kubernetes-commit: fd889ec8ae37437a9e75386542291bd0e2cc605e
2020-10-29 18:57:38 +08:00
Mike Spreitzer 86c572cf3f Remove phantoms from dump_requests output
No longer writes a single info-free line for each exempt priority level.

Kubernetes-commit: 6aeca5fb345256e104b9b2224a0f8797c1921214
2020-07-24 00:35:26 -04:00
Ken Sipe 4ad3e91efa changes in imports was unintentional
Signed-off-by: Ken Sipe <kensipe@gmail.com>

Kubernetes-commit: 09cd56b561d0c31421dfc6374b68e0b18c6cc746
2020-06-28 15:26:40 -05:00
Ken Sipe 32533315c9 fix S1000 simplify ch switch cases
Signed-off-by: Ken Sipe <kensipe@gmail.com>

Kubernetes-commit: 268c2f81c7ab94cbab68a8d6c00725144b81fa09
2020-06-26 10:45:30 -05:00
Mike Spreitzer b1ede52e21 Make sampleAndWaterMarkHistograms not fall very far behind
Kubernetes-commit: 9e89b92a92c02cdd2c70c0f52a30936e9c3309c7
2020-08-20 16:43:11 -04:00
yue9944882 543449d042 removes empty columns and fixes request details
Kubernetes-commit: 40e3feb7f1b7f54a512785af7e28810c0874ac63
2020-07-23 14:25:11 +08:00
Mike Spreitzer e28ab56bd4 Introduce more metrics on concurrency
Introduce min, average, and standard deviation for the number of
executing mutating and readOnly requests.

Introduce min, max, average, and standard deviation for the number
waiting and number waiting per priority level.

Later:

Revised to use a series of windows

Use three individuals instead of array of powers

Later:

Add coarse queue count metrics, removed windowed avg and stddev

Add metrics for number of queued mutating and readOnly requests,
to complement metrics for number executing.

Later:

Removed windowed average and standard deviation because consumers can
derive such from integrals of consumer's chosen window.

Also replaced "requestKind" Prometheus label with "request_kind".

Later:

Revised to focus on sampling

Make the clock intrinsic to a TimedObserver

... so that the clock can be read while holding the observer's lock;
otherwise, forward progress is not guaranteed (and violations were
observed in testing).

Bug fixes and histogram buckets revision

SetX1 to 1 when queue length limit is zero, beause dividing by zero is nasty.

Remove obsolete argument in gen_test.go.

Add a bucket boundary at 0 for sample-and-water-mark histograms, to
distinguish zeroes from non-zeros.

This includes adding Integrator test.

Simplified test code.

More pervasively used "ctlr" instead of "ctl" as abbreviation for
"controller".

Kubernetes-commit: 57ecea22296797a93b0157169db0ff2e477f58d0
2020-05-17 01:02:25 -04:00
Mike Spreitzer f0320d394e Add test for windup problem in APF
The current design for Fair Queueing for Server Requests has a
problem: if the min-max fair result stays different from an even
division for a long time and no queue involved in the imbalance goes
empty then the imbalance keeps accruing in queue virtual state times.

This commit adds a test that demonstrates the problem.

It also has some other tweaks to make other tests less flaky.

Factor the big scenario-testing func into pieces, with supporting
structs.

Kubernetes-commit: f3fdd5cf9f51f2a03ebfddef82bc2014e04b62a0
2020-06-04 00:15:08 -04:00
yue9944882 7f1f8c2578 introduce apf debug endpoint
Kubernetes-commit: 567becd5eedd9a1f3802f2b3b0b6b8efc445d2a1
2020-05-22 13:02:10 +08:00
Mike Spreitzer 705ee42713 Clean up fairqueuing/interface.go
Fixed outdated comment on QueueSet::StartRequest.

Removed unused interface EmptyHandler.

Kubernetes-commit: 28671fc4cf34e9e34eee545312c848ce40ab148b
2020-05-17 22:36:30 -04:00
Davanum Srinivas 5879417a28 switch over k/k to use klog v2
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: 442a69c3bdf6fe8e525b05887e57d89db1e2f3a5
2020-04-17 15:25:06 -04:00
Mike Spreitzer 9df60c9fe6 Renaming: "Change" -> "Add" for consistency with underlying method
Kubernetes-commit: c7b098ac6c276d65a79db6cfeb04f5f0f86eb315
2020-03-05 15:17:33 -05:00
Mike Spreitzer 8235385f97 Fix queued request accounting, extended queueset test
Kubernetes-commit: 8a1b60320986eca05cb281bcce45332e0969268e
2020-03-05 15:13:46 -05:00
Mike Spreitzer 6ae3e470a2 Make some metrics finer-grained, add dispatch counts, note immediate reject
Also add testing of metrics for queuesets.

Kubernetes-commit: f535a9c9ed4b6a0def47c354acad0ac2a8f961b0
2020-03-01 20:22:58 -05:00
Mike Spreitzer 219ab512ff Generalized NonResourcePolicyRule.NonResourceURLs impl
... to match the comment on that field.

Also generalized the test case generator to exercise the new
generality.

Kubernetes-commit: 2e97d3c8732147c3ba2f11d668f50b44e6374348
2020-03-04 00:00:39 -05:00
Mike Spreitzer f9c1780311 Hopefully plainer test strings
Kubernetes-commit: 4a4852ca9a1ffc439e2c476d7057a8be9f081055
2020-03-04 21:47:19 -05:00
Mike Spreitzer f44645645b Added non-randomized tests of matching FlowSchema rules
Kubernetes-commit: 21f78f3348736f242541f7b66e191aa1dce78c47
2020-02-27 23:36:19 -05:00
yue9944882 f452a698b0 register metrics from comp-base
Kubernetes-commit: 11656478be93d4a9e54129ec35cd2b9558e901ac
2020-02-27 17:04:17 +08:00
yue9944882 f93a7a8312 homogenize metrics naming
Kubernetes-commit: a1523a049ff9fc47d7dc2c4354b16b69d2eb4be2
2020-02-19 16:34:49 +08:00
Ted Yu f2eace2821 Do not dereference qcAPI which maybe nil
Kubernetes-commit: 583f804652ffff898437c80ce357e4f0c379bfb9
2020-02-14 13:30:40 -08:00
Mike Spreitzer 1ef67fe134 Version the API Priority and Fairness FieldManager values
Kubernetes-commit: b142868eaf46b24fce0e6b01e06087c924351857
2020-02-14 13:50:11 -05:00
Mike Spreitzer 1c79014daa Added API Priority and Fairness filter and config consumer
Kubernetes-commit: 73614ddd4e42728a36c7ac6b7b20f27c8032cafb
2020-01-25 19:47:12 -05:00
Mike Spreitzer fffb0fade5 brushed up according to review
Added LockingWriteMultipleOnly and LockingWriteOnceOnly interfaces,
so that further extensions are possible (in this package or others).

Moved common SetLocked behavior into promisoid.

Made comments say things that were implied.

Kubernetes-commit: cbdd3a279e6161d73f2c4e8a2b916ae74b258621
2020-01-24 14:33:15 -05:00
Mike Spreitzer 27a36b04f5 Tweak new names
Kubernetes-commit: a65f525aed8d9f28f677028f77e2ab41313ae037
2020-01-21 20:35:45 -05:00
Mike Spreitzer 3a091e906d Simplified logic around context cancel, removing bugs
Previously, a `decisionCancel` could overwrite a `decisionReject` or
`decisionExecute`, causing confusion.  Now a request gets exactly one
decision and there is no confusion.

Also added write-once to the promise package and refactored.

Kubernetes-commit: 1c092bf635954bde9c9c363672fa156b9430206b
2020-01-09 14:03:13 -05:00
Mike Spreitzer 4fead639b7 Refactored QueueSet configuration into two phases
So that errors can be detected before resolving concurrency shares
into concurrency counts.

Kubernetes-commit: 1e170637c3ce6c4ccd378275d9e52192f4be12b7
2020-01-18 01:46:11 -05:00
Mike Spreitzer d0f4b93c91 Brushed up fairqueuing package
This commit responds to the comments on PR #85192 that were not yet
addressed at the time it merged, apart from the one fixed in PR

Generalized fairqueuing to allow for zero queues, to support a
priority level that limits concurrency but does no queuing.

Kubernetes-commit: b123a43e7117e977606bacd31d77f4a30d2ed212
2019-11-13 23:34:10 -05:00