the new field exists primarily to avoid returning a 404 response when a resource actually exists but we haven't installed the path to a handler.
it is exposed for easier composition of the individual servers.
the primary users of this field are the WithMuxCompleteProtection filter and the NotFoundHandler.
Kubernetes-commit: ddfbb5d2bb57ee44b3e10f0b58f9cc7001f55802
This reverts commit fc5863b8b276e0789f717859e8cce58d7d060181, reversing
changes made to 027fe2554fd18343b8be39eddc8ff6570a6c390f.
Kubernetes-commit: f9f08725907b7db2104ee5fe9f82ab0752726533
add a new mode for graceful termination with the new server run option
'shutdown-send-retry-after'
- shutdown-send-retry-after=true: we initiate shutdown of the
HTTP Server when all in-flight request(s) have been drained. during
this window all incoming requests are rejected with status code
429 and the following response headers:
- 'Retry-After: N' - client should retry after N seconds
- 'Connection: close' - tear down the TCP connection
- shutdown-send-retry-after=false: we initiate shutdown of the
HTTP Server as soon as shutdown-delay-duration has elapsed. This
is in keeping with the current behavior.
Kubernetes-commit: 3182b69e970bd1fd036ff839fdf811f14e790244
- refactor graceful termination logic so we can write unit tests
to assert on the expected behavior.
Kubernetes-commit: d85619030e3a5fec5960ad00136e8d9bd030b5f8
- add plumbing that allows us to estimated "width" of a request
- the default implementation returns 1 as the "width" of all
incoming requests, this is in keeping with the current behavior.
Kubernetes-commit: 9b72eb1929a64b9d5a5234090a631ba312fb4d41
Manage the audit ID early in the request handling logic so that it can
be used by different layers to improve correlation.
- If the caller does not specify a value for Audit-ID in the request
header, we generate a new audit ID
- If a user specified Audit-ID is too large, we truncate it
- We echo the Audit-ID value to the caller via the response
Header 'Audit-ID'
Kubernetes-commit: 31653bacb9b979ee2f878ebece7e25f79d3f9aa6
- as soon as a request is received by the apiserver, determine the
timeout of the request and set a new request context with the deadline.
- the timeout filter that times out non-long-running requests should
use the request context as opposed to a fixed 60s wait today.
- admission and storage layer uses the same request context with the
deadline specified.
we use the default timeout enforced by the apiserver:
- if the user has specified a timeout of 0s, this implies no timeout on the user's part.
- if the user has specified a timeout that exceeds the maximum deadline allowed by the apiserver.
Kubernetes-commit: e416c9e574c49fd0190c8cdac58322aa33a935cf
- as soon as a request is received by the apiserver, determine the
timeout of the request and set a new request context with the deadline.
- the timeout filter that times out non-long-running requests should
use the request context as opposed to a fixed 60s wait today.
- admission and storage layer uses the same request context with the
deadline specified.
Kubernetes-commit: 83f869ee1350da1b65d508725749fb70d0f535f2
Aborted requests are the ones that were disrupted with http.ErrAbortHandler.
For example, the timeout handler will panic with http.ErrAbortHandler when a response to the client has been already sent
and the timeout elapsed.
Additionally, a new metric requestAbortsTotal was defined to count aborted requests. The new metric allows for aggregation for each group, version, verb, resource, subresource and scope.
Kubernetes-commit: 057986e32c1bb7284b0edbc161f0380f1548492f
StorageVersions are updated during apiserver bootstrap.
Also add a poststarthook to the aggregator which updates the
StorageVersions via the storageversion.Manager
Kubernetes-commit: 721897871697db007c2439ac298c579c0f201388
The MaxInFlight and PriorityAndFairness apiserver filters maintain
watermarks with histogram metrics that are observed when requests
are handled. When a request is received, the watermark observer
needs to fill out observations for the entire time period since the
last request was received. If it has been a long time since a
request has been received, then it can take an inordinate amount of
time to fill out the observations, to the extent that the request
may time out. To combat this, these changes will have the filters
fill out the observations on a 10-second interval, so that the
observations never fall too far behind.
This follows a similar approach taken in
9e89b92a92c02cdd2c70c0f52a30936e9c3309c7.
https://github.com/kubernetes/kubernetes/issues/95300
The Priority-and-Fairness and Max-in-Flight filters start goroutines to
handle some maintenance tasks on the watermarks for those filters. Once
started, these goroutines run forever. Instead, the goroutines should
have a lifetime tied to the lifetime of the apiserver.
These changes move the functionality for starting the goroutines to
a PostStartHook. The goroutines have been changed to accept a stop channel
and only run until the stop channel is closed.
Kubernetes-commit: 6c9b86646871f13a4431361310ba6a0785372053
apiserver_request_duration_seconds does not take into account the
time a request spends in the server filters. If a filter takes longer
then the latency incurred will not be reflected in the apiserver
latency metrics.
For example, the amount of time a request spends in priority and
fairness machineries or in shuffle queues will not be accounted for.
- Add a server filter that attaches request received timestamp to the
request context very early in in the handler chain (as soon as
net/http hands over control to us).
- Use the above received timestamp in the apiserver latency metrics
apiserver_request_duration_seconds.
- Use the above received timestamp in the audit layer to set
RequestReceivedTimestamp.
Kubernetes-commit: d74ab9e1a4929be208d4529fd12b76d3fcd5d546
Wojciech Tyczyński
Jefftree
Abu Kashem
Kermit Alexander
Paweł Banaszewski
Lukasz Szaszkiewicz
Ryan Phillips
David Ashpole
Daniel Smith
David Eads
Gautier Delorme
Shihang Zhang
Kevin Delgado
Marek Siarkowicz
KeZhang
drfish
Jordan Liggitt
yue9944882
Haowei Cai
Chao Xu
staebler
Mike Spreitzer
Joe Betz