- as soon as a request is received by the apiserver, determine the
timeout of the request and set a new request context with the deadline.
- the timeout filter that times out non-long-running requests should
use the request context as opposed to a fixed 60s wait today.
- admission and storage layer uses the same request context with the
deadline specified.
Kubernetes-commit: 83f869ee1350da1b65d508725749fb70d0f535f2
Aborted requests are the ones that were disrupted with http.ErrAbortHandler.
For example, the timeout handler will panic with http.ErrAbortHandler when a response to the client has been already sent
and the timeout elapsed.
Additionally, a new metric requestAbortsTotal was defined to count aborted requests. The new metric allows for aggregation for each group, version, verb, resource, subresource and scope.
Kubernetes-commit: 057986e32c1bb7284b0edbc161f0380f1548492f
without APIServerIdentity enabled, stale apiserver leases won't be GC'ed
and the same for stale storage version entries. In that case the storage
migrator won't operate correctly without manual intervention.
Kubernetes-commit: 1c2d446648662529282a3bb1528a6dbb50700fdb
StorageVersions are updated during apiserver bootstrap.
Also add a poststarthook to the aggregator which updates the
StorageVersions via the storageversion.Manager
Kubernetes-commit: 721897871697db007c2439ac298c579c0f201388
Previously no timeout was set. Requests without explicit timeout might potentially hang forever and lead to starvation of the application.
When no timeout was specified a default one will be applied.
Kubernetes-commit: 7340c3498ac23f46fc8b6bff4d5ac664a9c64a3b
The MaxInFlight and PriorityAndFairness apiserver filters maintain
watermarks with histogram metrics that are observed when requests
are handled. When a request is received, the watermark observer
needs to fill out observations for the entire time period since the
last request was received. If it has been a long time since a
request has been received, then it can take an inordinate amount of
time to fill out the observations, to the extent that the request
may time out. To combat this, these changes will have the filters
fill out the observations on a 10-second interval, so that the
observations never fall too far behind.
This follows a similar approach taken in
9e89b92a92c02cdd2c70c0f52a30936e9c3309c7.
https://github.com/kubernetes/kubernetes/issues/95300
The Priority-and-Fairness and Max-in-Flight filters start goroutines to
handle some maintenance tasks on the watermarks for those filters. Once
started, these goroutines run forever. Instead, the goroutines should
have a lifetime tied to the lifetime of the apiserver.
These changes move the functionality for starting the goroutines to
a PostStartHook. The goroutines have been changed to accept a stop channel
and only run until the stop channel is closed.
Kubernetes-commit: 6c9b86646871f13a4431361310ba6a0785372053
Currently webhook retry backoff parameters are hard coded, we want
to have the ability to configure the backoff parameters for webhook
retry logic.
Kubernetes-commit: 53a1307f68ccf6c9ffd252eeea2b333e818c1103
previously no timeout was set. Requests without explicit timeout might potentially hang forever and lead to starvation of the application.
Kubernetes-commit: 2160cbc53fdd27a3cbc1b361e523abda4c39ac42
apiserver_request_duration_seconds does not take into account the
time a request spends in the server filters. If a filter takes longer
then the latency incurred will not be reflected in the apiserver
latency metrics.
For example, the amount of time a request spends in priority and
fairness machineries or in shuffle queues will not be accounted for.
- Add a server filter that attaches request received timestamp to the
request context very early in in the handler chain (as soon as
net/http hands over control to us).
- Use the above received timestamp in the apiserver latency metrics
apiserver_request_duration_seconds.
- Use the above received timestamp in the audit layer to set
RequestReceivedTimestamp.
Kubernetes-commit: d74ab9e1a4929be208d4529fd12b76d3fcd5d546
Introduce min, average, and standard deviation for the number of
executing mutating and readOnly requests.
Introduce min, max, average, and standard deviation for the number
waiting and number waiting per priority level.
Later:
Revised to use a series of windows
Use three individuals instead of array of powers
Later:
Add coarse queue count metrics, removed windowed avg and stddev
Add metrics for number of queued mutating and readOnly requests,
to complement metrics for number executing.
Later:
Removed windowed average and standard deviation because consumers can
derive such from integrals of consumer's chosen window.
Also replaced "requestKind" Prometheus label with "request_kind".
Later:
Revised to focus on sampling
Make the clock intrinsic to a TimedObserver
... so that the clock can be read while holding the observer's lock;
otherwise, forward progress is not guaranteed (and violations were
observed in testing).
Bug fixes and histogram buckets revision
SetX1 to 1 when queue length limit is zero, beause dividing by zero is nasty.
Remove obsolete argument in gen_test.go.
Add a bucket boundary at 0 for sample-and-water-mark histograms, to
distinguish zeroes from non-zeros.
This includes adding Integrator test.
Simplified test code.
More pervasively used "ctlr" instead of "ctl" as abbreviation for
"controller".
Kubernetes-commit: 57ecea22296797a93b0157169db0ff2e477f58d0
Fixes:
* Don't call LogArgs if log will not be written due low verbosity
* Create separate slice for hijacked to avoid append on main path
* Shorten log message as this log is to common to be verbose
name old time/op new time/op delta
WithLogging-4 4.95µs ± 3% 3.52µs ± 1% -28.80% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
WithLogging-4 2.93kB ± 0% 1.22kB ± 0% -58.45% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
WithLogging-4 32.0 ± 0% 20.0 ± 0% -37.50% (p=0.000 n=10+10)
Kubernetes-commit: 303e1c19225149868d735b5c876d8ca9d3e1b5c9
* fix a number of unbounded dimensions in request metrics
* add test suite for cleanVerb and cleanContentType
* Properly validate that the content-type and charset (if applicable) are RFC compliant
* add additional test case
* truncate list of content-types
Change-Id: Ia5fe0d2e2c602e4def4b8e0849cc19f3f9251818
Kubernetes-commit: 6c588c3f441252f42fd37526297ed92d1e1f3acf
This change adds the generic ability for request handlers that run
before WithAudit to set annotations in the audit.Event.Annotations
map.
Note that this change does not use this capability yet. Determining
which handlers should set audit annotations and what keys and values
should be used requires further discussion (this data will become
part of our public API).
Signed-off-by: Monis Khan <mok@vmware.com>
Kubernetes-commit: 0bc62112adf270ef4efada37286319c229324c7b
This change removes support for basic authn in v1.19 via the
--basic-auth-file flag. This functionality was deprecated in v1.16
in response to ATR-K8S-002: Non-constant time password comparison.
Similar functionality is available via the --token-auth-file flag
for development purposes.
Signed-off-by: Monis Khan <mok@vmware.com>
Kubernetes-commit: df292749c9d063b06861d0f4f1741c37b815a2fa
So multiple instances of kube-apiserver can bind on the same address and
port, to provide seamless upgrades.
Signed-off-by: Mateusz Gozdek <mateusz@kinvolk.io>
Kubernetes-commit: dfe1f968ac31ba9b81a353d4de86d28e73d22d4e
Currently, /readyz starts reporting failure after ShutdownDelayDuration
elapses. We expect /readyz to start returning failure as soon as
shutdown is initiated. This gives the load balancer a window defined by
ShutdownDelayDuration to detect that /readyz is red and stop sending
traffic to this server.
Kubernetes-commit: 4134494fa51402ec5e5ea3fa1c51c0be55c955fd
The old flag name doesn't make sense with the renamed API Priority and
Fairness feature, and it's still safe to change the flag since it hasn't done
anything useful in a released k8s version yet.
Kubernetes-commit: 711c1e17209cc410440eecd3723e7b4906ca0e42
- Add handlers for service account issuer metadata.
- Add option to manually override JWKS URI.
- Add unit and integration tests.
- Add a separate ServiceAccountIssuerDiscovery feature gate.
Additional notes:
- If not explicitly overridden, the JWKS URI will be based on
the API server's external address and port.
- The metadata server is configured with the validating key set rather
than the signing key set. This allows for key rotation because tokens
can still be validated by the keys exposed in the JWKs URL, even if the
signing key has been rotated (note this may still be a short window if
tokens have short lifetimes).
- The trust model of OIDC discovery requires that the relying party
fetch the issuer metadata via HTTPS; the trust of the issuer metadata
comes from the server presenting a TLS certificate with a trust chain
back to the from the relying party's root(s) of trust. For tests, we use
a local issuer (https://kubernetes.default.svc) for the certificate
so that workloads within the cluster can authenticate it when fetching
OIDC metadata. An API server cannot validly claim https://kubernetes.io,
but within the cluster, it is the authority for kubernetes.default.svc,
according to the in-cluster config.
Co-authored-by: Michael Taufen <mtaufen@google.com>
Kubernetes-commit: 5a176ac77241ff059f22609fc569ac219334238c
From the listen godoc:
For TCP networks, if the host in the address parameter is empty or a
literal unspecified IP address, Listen listens on all available unicast
and anycast IP addresses of the local system.
Since the BindNetwork options is "tcp" by default, using an unspecified
address doesn't bind the listener to the IP family.
Kubernetes-commit: 05010d23ac7751a17aa26fb5cc011eb4f2127b1e
apiserver adds localhost to the alternateDNS field
if the bind address is 0.0.0.0.
This PR considers the IPv6 unspecified address too.
Kubernetes-commit: 29ec87f769c6494fad5c0a0c624efe2dc6eeab13
The old name is too broad, we wanted a name that is more specific to
the actual feature.
This is an alpha gate, and no release has yet associated any
functionality with this gate.
Kubernetes-commit: 76d090e30f917888c5882228f7261ed31a34a2ab
Also rename utilnet.ChooseBindAddress() to ResolveBindAddress(), to
better describe its functionality.
Kubernetes-commit: afa0b808f873b515c9d58a9ead788972ea7d2533
Reload SNI certificate cert and key file from disk every minute and notify
the dynamic certificate controller when they change, allowing serving
tls config to be updated.
Kubernetes-commit: d9adf535f35051be1d79d1309c72762939593d7c
Reload certificate cert and key file from disk every minute and notify
the dynamic certificate controller when they change, allowing serving
tls config to be updated.
Kubernetes-commit: 3f5fbfbfac281f40c11de2f57d58cc332affc37b
We create and use a dynamic certificate provider for the SNI serving
certificates. Currently we only use static serving certificate
providers, so the files are not reloaded, but we should be able to move
to a provider that is able to reload later on.
Kubernetes-commit: 84bc6b151d0726db0c737443874577f044e0e9c8
Currently we record request metrics during the normal request flow and
we also manually invoke `Record` in the timeout handler to record
timeouts. This means that we effectively double count whenever we
timeout. This PR renames the `Record` function to `RecordRequestError`
to more accurately reflect the intended side-effect of the function
call.
Change-Id: Ie37fd0c1e501bd525640a434433d364a5fd6dde2
Kubernetes-commit: 4c6e7247878477a1f2efc26df7f141258010374f
We create and use a dynamic certificate provider for the main apiserver
serving certificates: SNI will be handled separately later. Currently we
only use static serving certificate providers, so the files are not
reloaded, but we should be able to move to a provider that is able to
reload later on.
Kubernetes-commit: 6f89f1027488aeba4af0786988c2fd2ace9a6f1d
This change updates NewPrefixTransformers to not short-circuit on
the first transformer that has a matching prefix. If the same type
of encryption ProviderConfiguration is used more than once, they
will share the same prefix. A failure in the first one should not
prevent a later match from being attempted.
Added TestCBCKeyRotationWithOverlappingProviders unit test to
prevent regressions. Note that this test explicitly exercises this
flow using an EncryptionConfiguration object as the structure of the
resulting transformer is an important part of the check.
Signed-off-by: Monis Khan <mkhan@redhat.com>
Kubernetes-commit: 4dc16f29a7285a4bcaff1915728953d8a55e1b6e
If konnectivity service is enabled, the etcd client will now use it.
This did require moving a few methods to break circular dependencies.
Factored in feedback from lavalamp and wenjiaswe.
Kubernetes-commit: edbb0fa2fe2084d5d9ce0cf9dc0d1b2d820bb392
go fmt
make func private
refactor config_test
Two primary refactorings:
1. config test checkPath method is now each a distinct test
run (which makes it easier to see what is actually failing)
2. TestNewWithDelegate's root path check now parses the json output and
does a comparison against a list of expected paths (no more whitespace
and ordering issues when updating this test, yay).
go fmt
modify and simplify existing integration test for readyz/livez
simplify integration test
set default rbac policy rules for livez
rename a few functions and the entrypoint command line argument (and etcetera)
simplify interface for installing readyz and livez and make auto-register completion a bootstrapped check
untangle some of the nested functions, restructure the code
Kubernetes-commit: aa1b2d6d35c92a31be17357fc66cfc1eca8a67e0
Got the proxy-server coming up in the master.
Added certs and have it comiung up with those certs.
Added a daemonset to run the network-agent.
Adding support for agent running as a sameon set on every node.
Added quick hack to test that proxy server/agent were correctly
tunneling traffic to the kubelet.
Added more WIP for reading network proxy configuration.
Get flags set correctly and fix connection services.
Adding missing ApplyTo
Added ConnectivityService.
Fixed build directives. Added connectivity service configuration.
Fixed log levels.
Fixed minor issues for feature turned off.
Fixed boilerplate and format.
Moved log dialer initialization earlier as per Liggits suggestion.
Fixed a few minor issues in the configuration for GCE.
Fixed scheme allocation
Adding unit test.
Added test for direct connectivity service.
Switching to injecting the Lookup method rather than using a Singleton.
First round of mikedaneses feedback.
Fixed deployment to use yaml and other changes suggested by MikeDanese.
Switched network proxy server/agent which are kebab-case not camelCase.
Picked up DIAL_RSP fix.
Factored in deads2k feedback.
Feedback from mikedanese
Factored in second round of feedback from David.
Fix path in verify.
Factored in anfernee's feedback.
First part of lavalamps feedback.
Factored in more changes from lavalamp and mikedanese.
Renamed network-proxy to konnectivity-server and konnectivity-agent.
Fixed tolerations and config file checking.
Added missing strptr
Finished lavalamps requested rename.
Disambiguating konnectivity service by renaming it egress selector.
Switched feature flag to KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE
Kubernetes-commit: ebb65c5f4c51340f42c260cf00bde8629ed68d74
The previous HTTP compression implementation functioned as a filter, which
required it to deal with a number of special cases that complicated the
implementation.
Instead, when we write an API object to a response, handle only that one
case. This will allow a more limited implementation that does not impact
other code flows.
Also, to prevent excessive CPU use on small objects, compression is
disabled on responses smaller than 128Kb in size.
Kubernetes-commit: 4ed2b9875d0498b5c577095075bda341e96fcec2
There was a typo, the test was there but the return was not.
Added test that exposes the difference.
Kubernetes-commit: 7056e216addc7203f24c37a95c5c14ad194dddca
add startup sequence duration and readyz endpoint
add rbac bootstrapping policy for readyz
add integration test around grace period and readyz
rename startup sequence duration flag
copy health checks to fields
rename health-check installed boolean, refactor clock injection logic
cleanup clock injection code
remove todo about poststarthook url registration from healthz
Kubernetes-commit: 54dcf5c9c46fc4782d4861936309349b5a71a1ac
* fix duplicated imports of api/core/v1
* fix duplicated imports of client-go/kubernetes
* fix duplicated imports of rest code
* change import name to more reasonable
Kubernetes-commit: 5268f69405251a4a74130fa903e055a59071179a
firstly, split into two tests: TestLoopbackHostPortIPv4 and TestLoopbackHostPortIPv6.
then improve error handling, going to fail with explicit error message when run host
that does not support ipv6 or ipv4
Kubernetes-commit: 6b73b5031c463437064b1593d00a60107d09dd99
Antonio Ojea