Commit Graph

3290 Commits

Author SHA1 Message Date
Cici Huang 00ae118a76 Add stability tests for CEL cost estimation.
Kubernetes-commit: 93951bc00d3656c42bec050b82f2fc364b1cad31
2023-10-22 23:12:44 -07:00
Wojciech Tyczyński 324d2bc3cf Cleanup paging parameters in etcd3 store
Kubernetes-commit: 3f4d3b67682335db510f85deb65b322127a3a0a1
2023-10-20 15:38:59 +02:00
Wojciech Tyczyński 5bf4f58ab8 Remove storageConfig.Paging parameter
Kubernetes-commit: b386120da239bf9652fc02b2d2cbbd0fcc3cd121
2023-10-20 15:35:58 +02:00
Wojciech Tyczyński 71519801e7 Remove disablePaging param
Kubernetes-commit: 106f58686d58a790a86dc05b4c93a5da041de389
2023-10-20 15:30:03 +02:00
Sean Sullivan 10cbfd9a3c StreamTranslator and FallbackExecutor for WebSockets
Kubernetes-commit: 168998e87bfd49a1b0bc6402761fafd5ace3bb3b
2023-07-06 21:22:07 -07:00
tao.yang 47998d1ee6 cleanup: omit comparison with bool constants
Signed-off-by: tao.yang <tao.yang@daocloud.io>

Kubernetes-commit: b35357b6c08f21ba0fd312536051394c2567ec79
2023-09-04 16:59:23 +08:00
Ben Luddy a270d45ae5 Add validation for --storage-media-type option.
Kubernetes-commit: cf836309dc278d8d4f046e1580649179b1531143
2023-10-19 10:54:16 -04:00
Ben Luddy 91d0f39545 Restrict supported media types for new apiservers.
This is to prevent the enablement of new data formats (CBOR) in the early stages of phased
implementation.

Kubernetes-commit: ced56a6adabdd86f99455b100b1c0c7a2b4f3c55
2023-10-17 14:06:46 -04:00
machine424 340fd67b3a storage/etcd3/metrics: add unit test for apiserver_storage_objects
Kubernetes-commit: a96b97e40c0837f727c8e6d42f021b66922bb26c
2023-09-22 16:12:40 +02:00
carlory 5ac339fec6 update pretty param description
Kubernetes-commit: 75f20ee64da5317f4473de643eac43686fe9215e
2023-10-16 16:36:31 +08:00
Nabarun Pal 4693682515 Add --authorization-config flag to apiserver
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

Kubernetes-commit: 22e5a806a73e48486a90491fc3eb03d208b520a0
2023-09-25 09:18:11 +05:30
Nabarun Pal 5873bbb7bf add feature gates for authorization config
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

Kubernetes-commit: 007ef653ad089180d02a58782bbd3912e875354d
2023-08-24 15:56:56 +05:30
Jiahui Feng 140ffa083d set maxLength to longest enum.
Kubernetes-commit: 302d350e88eac519e1df020b82256371c171b861
2023-10-09 11:00:45 -07:00
Nabarun Pal b259861486 staging/apiserver: correct KubeConfigFile type in authorization types
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

Kubernetes-commit: 2bf2c4f3a413d3a2e070fe61aeba6fb309bf2e5e
2023-09-27 17:48:38 +05:30
Siyuan Zhang 77032c52b8 k8s.io/apiserver/storage: add 3 new unit tests for delete.
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 26a4e06c92c248748dd3c50c74d75f8adc3c6823
2023-10-11 10:38:01 -07:00
Jefftree 2e1024671d Fix v3 spec
Kubernetes-commit: b30c6bdff817cec28b3d88b3bb3e12f1e86488d0
2023-10-04 12:55:49 -04:00
Siyuan Zhang 70af178d56 k8s.io/apiserver/storage: add a new TestCreate case.
Add a test case of create with rv set.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 2f923e356e042d9dce88e8f78abf30f414051e71
2023-10-11 10:24:31 -07:00
Alexander Zielenski 8a3fe0e45c ratcheting: disable correlation by index
discussion: https://github.com/kubernetes/kubernetes/pull/121118#discussion_r1358865893

Kubernetes-commit: fb1fc8b4a72758688d1251278579b2b0ac666fc7
2023-10-13 14:36:46 -07:00
Alexander Zielenski b5ac4f9a61 comments: clear up correlateOldValueForChildAtNewIndex godoc
Kubernetes-commit: d991ed56c29e646c0c5c51ce1ebd2376f34fce28
2023-10-13 14:11:02 -07:00
Alexander Zielenski fbd7474961 cleanup: use swtich in CachedDeepEqual and add more comments
Kubernetes-commit: 0ed67c9e41dcfc3eef6953ca63082454c189443b
2023-10-13 14:05:47 -07:00
Alexander Zielenski a504910cff cleanup: consistently support nil receiver and document
Kubernetes-commit: 60c90fc0854eb04b95e74d445d88f45c212900fe
2023-10-13 13:57:55 -07:00
Alexander Zielenski 541189e16c cleanup: clarify correlatedOldValueForChildAtNewIndex comment
Kubernetes-commit: abb68591afd30cf263b0d6bb2942f9693eb420d7
2023-10-13 13:54:53 -07:00
Alexander Zielenski 2970233dd7 cleanup: consistent interface{} and any
Kubernetes-commit: e1fa1df3ae8414104f3710c064014e323e45aade
2023-10-13 13:50:52 -07:00
Alexander Zielenski fecc880526 cleanup: add godoc
Kubernetes-commit: 0495616230a13dcc19c9da8ec7b8b2a38e2b6a33
2023-10-13 13:50:19 -07:00
Alexander Zielenski 662079f048 test: fix boilerplate
Kubernetes-commit: 4dedabf2a659ee702cbcd93a482c63296910d5c6
2023-10-12 15:51:25 -07:00
Alexander Zielenski e501fcbbf6 test: few more correlatedobject test cases
Kubernetes-commit: 0149c1f8b315d704d6d80c00861526e2899001e5
2023-10-11 15:45:48 -07:00
Alexander Zielenski 4ec87cdde2 test: add correlatedobject test cases
Kubernetes-commit: ba9347230e6577140eaa0ac3d9ef99d0163a7934
2023-10-11 14:03:28 -07:00
Alexander Zielenski 5edc046b33 cleanup: add header and fix spelling
Kubernetes-commit: c08a9321eed6a917a2fbc13b8e023d2f4122ee36
2023-10-11 13:51:49 -07:00
Alexander Zielenski 1234a74f8e refactor: move correlatedObject to its own file
no changes except package naming

Kubernetes-commit: 27cb869e5596525cec9884ecb9b02bfcfe5273e4
2023-10-10 10:53:12 -07:00
Damien Grisonnet b5b4cd7758 apiserver: rename request body size metric
Rename the apiserver_request_body_sizes metric to
apiserver_request_body_size_bytes to conform with Prometheus best
practices.

This can be done safely without deprecation because that metric wasn't
registered before.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

Kubernetes-commit: 08006c842fd6a584bb8e2511587c999ffe7ced9d
2023-09-07 19:16:31 +02:00
Jiahui Feng 6b0a70e192 typed variables support.
Kubernetes-commit: c03579bfa40dcb39e1ffe24c12f933720e4eb204
2023-10-04 16:39:24 -07:00
Alexander Zielenski 3029a9f674 add rest of accessors to common.Schema
needed for declarative validation, CRD ratcheting

Kubernetes-commit: 438c0daab7587bdb094e714e68b5ba2f9f6ae963
2023-10-09 17:49:37 -07:00
Siyuan Zhang a0dede6875 k8s.io/apiserver/storage: add some ResourceVersion validation in GetList unit tests
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: baac8bb573c8efb314b037f4fbac116556c03c83
2023-10-11 10:46:30 -07:00
Abu Kashem 28ed1d7ad4 fix data race in apf unit test
Kubernetes-commit: 52c58d970e54bf10b78512c68602f70b0a970f31
2023-09-22 14:42:43 -04:00
Monis Khan 87ef6687ab Skip TestUnauthenticatedHTTP2ClientConnectionClose http1 tests
These occasionally flake on CI:

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/121200/pull-kubernetes-unit-go-compatibility/1712589824344461312

=== Failed
=== FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true/http/1.1 (0.19s)
    authentication_test.go:653: expect TCP connection: 1, actual: 2
        --- FAIL: TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true/http/1.1 (0.19s)

=== FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true (0.23s)
    --- FAIL: TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true (0.23s)

=== FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose (2.30s)

Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: cd5db9b7f23b0156bf5535fc0124361fbef0ce6a
2023-10-12 19:13:07 -04:00
Monis Khan 445b713906 Prevent rapid reset http2 DOS on API server
This change fully addresses CVE-2023-44487 and CVE-2023-39325 for
the API server when the client is unauthenticated.

The changes to util/runtime are required because otherwise a large
number of requests can get blocked on the time.Sleep calls.

For unauthenticated clients (either via 401 or the anonymous user),
we simply no longer allow such clients to hold open http2
connections.  They can use http2, but with the performance of http1
(with keep-alive disabled).

Since this change has the potential to cause issues, the
UnauthenticatedHTTP2DOSMitigation feature gate can be disabled to
remove this protection (it is enabled by default).  For example,
when the API server is fronted by an L7 load balancer that is set up
to mitigate http2 attacks, unauthenticated clients could force
disable connection reuse between the load balancer and the API
server (many incoming connections could share the same backend
connection).  An API server that is on a private network may opt to
disable this protection to prevent performance regressions for
unauthenticated clients.

For all other clients, we rely on the golang.org/x/net fix in
b225e7ca6d
That change is not sufficient to adequately protect against a
motivated client - future changes to Kube and/or golang.org/x/net
will be explored to address this gap.

The Kube API server now uses a max stream of 100 instead of 250
(this matches the Go http2 client default).  This lowers the abuse
limit from 1000 to 400.

Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 800a8eaba7f25bd223fefe6e7613e39a5d7f1eeb
2023-10-07 21:50:37 -04:00
Siyuan Zhang e15d4d2e0b k8s.io/apiserver/storage/etcd: refactor getCurrentState.
Extract getCurrentState as a separate method that can be reused.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: ebca5d438d9cb2c82d0b99dbcb0aeca8879db441
2023-10-11 09:56:07 -07:00
Qiming Teng e014cf25b9 Generated files
Kubernetes-commit: c65fe450d8a3229cfe531a3806939775dd52e7e0
2023-10-03 20:16:10 +08:00
Qiming Teng d763e7d132 Fix API docs for audit APIs
The `*`s in the source comment is confusing the API reference generator.
They are treated as symbols for bold texts when generating reference docs.
This PR replaces the quote marks with backtiqs so that the reference
generator can properly handle them.

Kubernetes-commit: e7b2aeee930188eec125bbb91096d9d3fd6f3b5c
2023-10-03 17:18:23 +08:00
David Ashpole d35f091281 fix missing http.target trace attribute
Kubernetes-commit: 80269d5d3497acc8ad155cb9bfbfaa7fd9e20d1f
2023-10-06 18:09:29 +00:00
Nabarun Pal 70eb989b94 k8s.io/apiserver: fix levelling of the name field in AuthorizationConfiguration
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

Kubernetes-commit: 11ce6d29157daf7437d6da7fdeb11cabf2e774aa
2023-10-04 10:33:58 +05:30
Abu Kashem 04acc93a12 apf: request ejected from queue should use reason 'time-out'
Kubernetes-commit: 6297067ff1afaa2d63d83183a23e8744865d7c2b
2023-08-29 16:30:02 -04:00
Abu Kashem d64c9b18da apf: remove RequestWaitLimit from queueset config
Kubernetes-commit: 11ef9514dad6f46a4315198978fee14132c4bbca
2023-08-29 12:11:08 -04:00
Abu Kashem 290096a4d0 apf: remove timeoutOldRequestsAndRejectOrEnqueueLocked function
Kubernetes-commit: da8a472206623d0727ba486489d34780c4b6c1d9
2023-08-28 17:26:11 -04:00
Abu Kashem a2e63604f2 apf: use context for queue wait
Kubernetes-commit: f39213a7e44f21a8cedcdf38d3c2531456a526d6
2023-08-28 17:01:16 -04:00
Dr. Stefan Schimanski 2f3285287e controlplane: make option structs uniformly optional
Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

Kubernetes-commit: 63950491764535a8635cb2c4810db59a9a1fad25
2023-09-24 11:50:38 +02:00
Lukasz Szaszkiewicz 1c49f6c8ba endpoints/metrics: define watchListLatencies metric and associated functions
Kubernetes-commit: a97f4b7a3123c9768ec7136b6ca32be926e16cd6
2023-09-19 03:05:37 +02:00
Lukasz Szaszkiewicz cd87b8f62d handlers/watch: calculate and record WatchList latency metric.
Kubernetes-commit: 772b1f4cd84a738f632716e28d4067c00f0b7f13
2023-09-19 03:05:00 +02:00
Lukasz Szaszkiewicz db34e9b7c7 storage/util: introduce HasInitialEventsEndBookmarkAnnotation
Kubernetes-commit: 70ae9c441cee9d78a88c0526592796ac4f491096
2023-09-18 11:39:08 +02:00
Lukasz Szaszkiewicz 90ba08909c storage/etcd: the watcher supports the API streaming
Kubernetes-commit: ca562fd280a9dd5db952eb8c14b93fcf6668cc49
2023-08-22 14:02:39 +02:00
Lukasz Szaszkiewicz e7424d8207 storage/testing/watcher_tests: add new flavour to RunWatchSemantics test and fill the gap
the new flavour proves that a storage layer follows RV
which is equal to the global rv

Kubernetes-commit: 0e238c4c28f11d1ec7cf5f506c81195c1cc463fe
2023-09-22 09:32:23 +02:00
HirazawaUi 91aa0e1c62 Add namespace to apiserver tracing
Kubernetes-commit: 45b9b0df41fb67cf2c71263b3577f3b3daff4899
2023-09-03 21:50:47 +08:00
HirazawaUi f5f4a6148a Add subresource to apiserver tracing
Kubernetes-commit: c83eb6dcaaacfe8b5706f7fdc969000fbef2d119
2023-09-03 21:37:13 +08:00
HirazawaUi 2b665e42e6 Add name to apiserver tracing
Kubernetes-commit: bcb59a03ebb50a966cc6921dea016cc68e0bf843
2023-09-03 21:33:48 +08:00
HirazawaUi 79f4968b20 Add api-version to apiserver tracing
Kubernetes-commit: 646053d73dbda9cae5df5fd105a720661ff051dc
2023-09-03 21:24:13 +08:00
HirazawaUi c51632c143 Add group to apiserver tracing
Kubernetes-commit: 8c34208a62f09904cd270906889d57c7a613b5d6
2023-09-03 21:11:08 +08:00
Lukasz Szaszkiewicz f0dfb17949 storage/testing/watcher_tests: refactor RunWatchSemantics tests
Kubernetes-commit: dc5cfe3cff289ecb97153716bfd5817579d00fb3
2023-09-21 12:16:16 +02:00
Lukasz Szaszkiewicz c8dcfeed34 storage/testing/watcher_tests: make TestCacherWatchSemantics storage agnostic
Kubernetes-commit: 91bb75883c613d45563f3b7c01a69dde8194bfdc
2023-09-15 12:28:36 +02:00
Lukasz Szaszkiewicz c13e210d56 storage/testing/watcher_tests: move TestCacherWatchSemantics (no-op)
Kubernetes-commit: afbb1a6ef98b548b9e57b168614ca6e15fd0034c
2023-09-14 11:19:36 +02:00
Lukasz Szaszkiewicz 3c3bb86696 storage/testing/utils: add helper functions
Kubernetes-commit: 3a96baf3fa8e837159b27b98b05c5024143b06a3
2023-09-15 12:26:11 +02:00
Lukasz Szaszkiewicz fe7ae035ef storage/etcd3/watcher_test: refactor TestWatchErrorWhenNoNewFunc to a table test
Kubernetes-commit: 4b60c18183f11207d0cb2d89bf28090156222c61
2023-09-19 10:35:09 +02:00
Nabarun Pal 1eae2482e2 Bootstrap API Types for Structured Authorization Configuration
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

Kubernetes-commit: 52c582ca77c775ee13300a999a29f8c4180750a2
2023-09-14 19:19:29 +05:30
Lukasz Szaszkiewicz f5adfca95e storage/testing: a simple refactor
Kubernetes-commit: 39af594f312ad46715dfb705d71ad4e607e27e5d
2023-09-14 11:16:10 +02:00
Han Kang bd7db816d7 promote component SLIs to GA; remove feature gates for component slis
Kubernetes-commit: e6435e98eda93fbf96e94110af6ca5a10ad1e926
2023-09-11 08:52:50 -07:00
machine424 98d028d0d0 clarify the description of the metric apiserver_storage_objects
Kubernetes-commit: 084deb7a7cf147bbafd131ab1a36852a79f08a64
2023-09-12 11:16:36 +02:00
Rita Zhang cdb2cea24a kms: remove livez check
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 7710128636a16c73045291d4729675339a7d57f7
2023-09-11 16:47:29 -07:00
Rita Zhang 2bed5d11d9 kmsv2: add apiserver identity to metrics
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 43ccf6c4e8f173d981edebb6146c58b523fc21b7
2023-09-05 13:03:18 -07:00
Monis Khan 9c40486020 kmsv2: enable KMSv2KDF feature gate by default
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 657cc2045ee46922b00d4fd7c126f57d1e8ecc43
2023-09-05 12:27:55 -04:00
Anish Ramasekar 9b1c514777 register API types only once for encryption config
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 418af0f4dcbe15147b21462b9e5db5a5ba769c12
2023-09-01 17:24:20 +00:00
Anish Ramasekar 25d893ad5f add loading config and wire feature flag
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 9e1ff1e51201ac41ddb1eed0d5cc015b4b6aa3df
2023-08-10 22:45:07 +00:00
Anish Ramasekar 1fbafe88b9 add StructuredAuthenticationConfiguration feature flag
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 1bf90f9484c5dbcd941251f0036af65fa25ee193
2023-08-10 22:06:41 +00:00
Monis Khan 87aad93082 kmsv2: add legacy data integration test
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 95121fe846f875c4f3feb2b1bf42a9553566f097
2023-09-01 11:21:38 -04:00
Monis Khan 49c6151dee kmsv2: fix race in simpleCache.set when setting cache size metric
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: b10697c7880848d7ec110fd6b3e67015bbe74fa8
2023-08-27 15:14:04 -04:00
Lukasz Szaszkiewicz 957c305d94 storage/etcd: add TestWatchDispatchBookmarkEvents unit test
Kubernetes-commit: 875b00137fdfbc74756a0fc5b1c9b2adbeb78e55
2023-09-01 13:45:44 +02:00
Lukasz Szaszkiewicz 51e79bdd97 storage: document ProgressNotify from storage.ListOptions
At first glance, it seems that the fields storage.ListOptions.ProgressNotify and storage.ListOptions.Predicate.AllowWatchBookmarks
are the same. Unfortunately, this is not the case.

This PR documents the differences and motivations for why these fields are actually distinct.

Kubernetes-commit: 6058540f3d0edc405a1f1b8a96bd82ceca99c240
2023-09-01 10:48:36 +02:00
Marek Siarkowicz 8b7824a7e8 Avoid creating local variables that don't change
Having local variables gives false impression that this is overwritten
in the function block.

Kubernetes-commit: e01bd641447a315e28fab8148e99ac6afba9bcd7
2023-08-30 16:51:40 +02:00
Marek Siarkowicz 90adbb75eb Flatten switch case
Kubernetes-commit: 10553a1966892e305252c65ebeb9043416304f48
2023-08-30 16:42:34 +02:00
Marek Siarkowicz 5f9b5dda94 Refactor common WithRange case
From API call WithRange and WithPrefix work the same, they just set the range end.
The difference is when the range end is provided:
* WithRange(end) requires providing the end while calling
* WithPrefix() calculates the end based on key provided to the Get.

For example, those are equal:
* client.Get(ctx, "/pods/", WithPrefix())
* client.Get(ctx, "/pods/", WithRange(GetPrfixRangeEnd("/pods/")))

As keyPrefix is equal preparedKey there should not be a difference.

Kubernetes-commit: 1f4f2a5d6014dc8f98b25a9484d4a6064a6ae18e
2023-08-30 16:26:20 +02:00
Wojciech Tyczyński 3bcdefc0fd Refactor transformers for watch to implement Encoder interface
Kubernetes-commit: 160589a0edb2038f2e22e376b6a0b71a24d21f22
2023-08-11 21:43:49 +02:00
Wojciech Tyczyński 70b023f369 Minor cleanup tranformers interface
Kubernetes-commit: 3fcc045bce9d77704f872d6089d7c09342e26a44
2023-08-09 19:49:59 +02:00
Wojciech Tyczyński 6311828461 Cleanup setting non-nil Items field
Kubernetes-commit: 172a41192c65324b1dc9dc4d90903552c538d664
2023-08-07 21:17:03 +02:00
Wojciech Tyczyński 3fd71eeb11 Refactor WatchServer to prepare for using encoders
Kubernetes-commit: ff56d3b6914dbc9e16683ab731eb8c0a485b4ab4
2023-08-03 21:50:08 +02:00
Marek Siarkowicz a8fba27b08 Remove duplicated returnRV variable
returnRV was was equal to withRev, but updated at different time.

When preparing the request they are set equal to each other.

The only difference was during the for loop.
returnRV was always set no matter if pagination was enabled, while withRev only when paginating.

Kubernetes-commit: be4692864bb983e94e8d7b6b6aa1a9c22fe23bce
2023-08-30 17:36:05 +02:00
wackxu f40bea647b paginate initial list inside the storage watcher
Signed-off-by: wackxu <xushiwei5@huawei.com>

Kubernetes-commit: f5d6c65186d63647a46400762f849d500e6ac591
2023-07-28 16:43:34 +08:00
Lukasz Szaszkiewicz 206231e0d0 storage/etcd3: error when progressNotify option set and newFunc was provided for a registry
Kubernetes-commit: 1988c31fc8115bf9eec1adf2bffed3fd677d1a9f
2023-08-28 17:50:42 +02:00
Patrick Ohly 037fa7c2c2 apiserver: fix data race in etcd metrics
7a63997c8a1a9ba1 added a global variable which gets set multiple times by
different goroutines in integration tests, leading to a data race:

WARNING: DATA RACE
Write at 0x00000a626928 by goroutine 87080:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Previous write at 0x00000a626928 by goroutine 87079:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Mutex locking avoids the data race. Whether this variable really can be used
safely by those concurrent (?) tests is a different question...

Kubernetes-commit: 13a8ad12b8296c0360afe3f66218027dae6c1805
2023-08-25 10:42:17 +02:00
Anish Ramasekar fdfc990c33 wiring existing oidc flags with internal API struct
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 1bad3cbbf59a61805a48f609b8cc0a2a40c168ef
2023-06-28 06:04:45 +00:00
Anish Ramasekar 496ba1943b add AuthenticationConfiguration v1alpha1 api
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: f909eb630cac310adf7267b85922f1340508ca79
2023-06-27 21:07:47 +00:00
Abu Kashem 27772523df apf: refactor promise to use a context
Kubernetes-commit: 0039f24d74d0f57c8ba868ae361821d37fd908d6
2023-08-21 15:19:31 -04:00
Lukasz Szaszkiewicz d8d65bebc5 storage/util: introduce AnnotateInitialEventsEndBookmark function
// AnnotateInitialEventsEndBookmark adds a special annotation to the given object
// which indicates that the initial events have been sent.
//
// Note that this function assumes that the obj's annotation
// field is a reference type (i.e. a map).

Kubernetes-commit: 47d9a47a08856613e2e6ae6aa8a1bdeb1e281f97
2023-08-24 12:19:10 +02:00
Lukasz Szaszkiewicz f75c503352 storage/factory: extend the Create method by newList and resourcePrefix params
Kubernetes-commit: ccabc01093a1344ebb27c32c946e9da3b8e91fd2
2023-07-28 09:53:01 +02:00
Lukasz Szaszkiewicz 52849e2b21 storage/cacher: duplicate logging when a watch is closed due to unresponsiveness.
Kubernetes-commit: 1327a57aaf8f5ffaea3d33204dbe64dccef4e0ef
2023-08-23 12:51:50 +02:00
Joe Betz eedba684cf Add CEL library lifecycle linter
Kubernetes-commit: 980fa6a2b9edae577d1b71a28834afe1ca7dfe82
2023-08-23 17:17:32 -04:00
Joe Betz f0fbaa3fab Bump cel string lib to v2, add tests
Kubernetes-commit: 3fb14cf4e7a0230d57f579b86262d9df6997e5e3
2023-08-22 13:05:09 -04:00
Joe Betz c6c8291384 Update k8s code to handle cel-go changes
Kubernetes-commit: 640a2ec33274f4b6311afc7dbb0c7e15bd5d792a
2023-08-18 17:32:41 -04:00
Divya Sri Sanaganapalli 956f1b4799 Incorporating feedback on 119341
Kubernetes-commit: 24877f96fbb60f34c1c808e7ac76870019eee86b
2023-08-21 15:20:30 +00:00
Wojciech Tyczyński f56b9ee7f5 Graduate RemainingItemCount to GA
Kubernetes-commit: 4e2e059c7b205d2e4b246a262128223258a49498
2023-07-21 15:22:51 +02:00
Wojciech Tyczyński 3030f660a8 Graduate APIListChunking to GA
Kubernetes-commit: 6acfa3cb4ac876e46ead5ba4772ba18e480435ce
2023-07-21 11:35:21 +02:00
Joe Betz 6f6cacf8bf check for overflow
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>

Kubernetes-commit: b1223e0d1ac626d5472436ed2cd0bc2962394004
2023-08-22 10:02:10 -07:00
Joe Betz 3e172f1316 Fix CEL cost handling of zero length replacement strings
Kubernetes-commit: 87cf890632c538e83e961035e73f0f2e8d3ceeba
2023-08-21 22:48:11 -04:00
Mike Spreitzer 7c7ff34a5a Handle edge cases in seat demand stats
Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Kubernetes-commit: b0e249f657a18c1435ace391fa752711dafce732
2023-08-21 15:26:38 -04:00
seantywork bf05e35835 kubernetes mutual (2-way) x509 comment
Kubernetes-commit: 48260b4a77b423b178ec5e262ac67be52d49f455
2023-08-18 01:31:22 +00:00
Lukasz Szaszkiewicz 7ecfbf7914 storage/etcd: simplify passing the transformer
Kubernetes-commit: a5600b6925169a7d99376d52472e5f8e8635082b
2023-08-21 12:27:41 +02:00
Lukasz Szaszkiewicz dc7ff6e034 storage/etcd: remove newWatcher function
Kubernetes-commit: 6161752ecbe434b8fb157dd3a30bb5c0cef7a23e
2023-08-21 12:13:12 +02:00
Lukasz Szaszkiewicz 6f2daefacf storage/util: move GetCurrentResourceVersionFromStorage
Kubernetes-commit: 9a253d896a096b4e1ffccf4b1f84e5cac1e1aad0
2023-07-26 15:53:13 +02:00
Lukasz Szaszkiewicz 2d9be35745 storage/etcd: no-op, refactor watcher.Watch method signature
Kubernetes-commit: f7e659db236286ca15707bf08acb08dc0ac4ab0e
2023-07-24 12:33:03 +02:00
Antonio Ojea 49f7deccb5 fix race on etcd client constructor for healthchecks
Change-Id: Id29b5b377989dcb5377316cfcdea367071a47365

Kubernetes-commit: 77b3bb0f69bfa4e5eb56ba484f724476304616cc
2023-08-08 13:55:14 +00:00
Rita Zhang 2eac3ca68c kmsv2 test feature enablement unit test
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: d86e72202c4b039e1dceccbfbae559fb1c54471d
2023-08-01 10:17:01 -07:00
Rita Zhang 7ebae7f76f kmsv2: add metric for DEK cache filled
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 3cbecf218dfea2e99ae95310ac03406d1d87a072
2023-08-09 12:28:01 -07:00
Joe Betz 539b445fa2 Add test coverage of result size of string operations
Kubernetes-commit: e4d16f34c15affdf4411d5bd1b19991b5fa27f2a
2023-08-07 12:41:52 -04:00
Monis Khan 64eaf11221 wsstream: use a single approach to detect connection upgrade
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 62b063b74b5eb1b7e72ebac7b5348593249f732b
2023-08-01 18:37:34 -04:00
Jefftree 35302d6383 Move adding GroupVersion log until after an update is confirmed
Kubernetes-commit: 49e00e8bdeec6b1cdaf691ec85ba8510892ebbc2
2023-08-08 14:28:54 +00:00
Dr. Stefan Schimanski 13a3aab581 apiserver/httplog: pretty up impersonation output
```
I0807 09:09:16.419239       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/batch/v1?timeout=32s" latency="214.666µs" userAgent="kubernetes-provider/v0.0.0 (linux/arm64) kubernetes/$Format" audit-ID="948ef6b2-474d-45a7-ad5f-894ce93d05f7" srcIP="192.168.139.202:35542" apf_pl="exempt" apf_fs="exempt" apf_execution_time="129.5µs" resp=200 addedInfo=<

      &{kubernetes-admin  [system:masters system:authenticated] map[]} is acting as &{foo  [system:authenticated] map[]}
       >
```

to

```
I0807 09:09:16.419239       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/batch/v1?timeout=32s" latency="214.666µs" userAgent="kubernetes-provider/v0.0.0 (linux/arm64) kubernetes/$Format" audit-ID="948ef6b2-474d-45a7-ad5f-894ce93d05f7" srcIP="192.168.139.202:35542" apf_pl="exempt" apf_fs="exempt" apf_execution_time="129.5µs" resp=200 addedInfo="kubernetes-admin[system:masters system:authenticated] is impersonating foo[system:authenticated]"
```

Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

Kubernetes-commit: 37730c07dd658ba585ffee3861780e18947ca534
2023-08-07 11:23:30 +02:00
Jiahui Feng bf2563c6cd CEL lazy map: add test for boolean short-circuiting
Kubernetes-commit: 66aa2af0979cc6007cd63720876fd21dda3b17dc
2023-07-25 14:37:20 -07:00
Andrew Sy Kim 066c7cb8cc apiserver: add flow control metric current_inqueue_seats
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

Kubernetes-commit: fb9646fd60d4b8e79223b729c1cb54fc6818fdd1
2023-07-24 19:40:05 +00:00
Stephen Kitt 7fb4ad7511 api-machinery: stop using deprecated io/ioutil
This replaces deprecated ioutil functions as follows:

* ioutil.ReadAll -> io.ReadAll
* ioutil.ReadFile -> os.ReadFile
* ioutil.TempDir -> os.MkdirTemp
* ioutil.TempFile -> os.CreateTemp
* ioutil.WriteFile -> os.WriteFile

Signed-off-by: Stephen Kitt <skitt@redhat.com>

Kubernetes-commit: b60a3a58df2791ae67764f6325be31aea5eca5a0
2023-05-02 15:08:18 +02:00
Damien Grisonnet 245d131967 apiserver/etcd3: fix segv during metric collection
Fix a segfault when collecting the storage size metrics when the getters
used to collect the data on etcd haven't been initialized properly. This
happens when the EtcdOptions are not applied which is the case for
aggregated apiservers that don't care about storage.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

Kubernetes-commit: c6efaf16c1ed07ce37485b7a272628f653cbf06f
2023-08-10 17:01:17 +02:00
Jordan Liggitt 1edd7d6157 Avoid returning nil responseKind in v1beta1 aggregated discovery
Kubernetes-commit: 1876ddf71497bad349f7c4df24c2e22356d3bad9
2023-08-08 14:25:56 -04:00
Alexander Zielenski 09a47412b5 bugfix: use matched resource for AdmissionRequest.resource, not the resource it was converted from
use existing admission request for audit annotation eval

populate matchResource in empty rules case

Kubernetes-commit: e1b0bc3d0a7fb89a1e60f4ec1ee34b10de22d00a
2023-07-21 18:13:24 -07:00
Alexander Zielenski eea6b57f73 bump validatingadmissionpolicy alpha->beta
Kubernetes-commit: 5e2e8c806475d21bc22f10ccc810451c1bcb21a7
2023-07-21 16:27:51 -07:00
Alexander Zielenski a690957dd1 update codegen
Kubernetes-commit: d6479587445a5a6fa736ee7fb3012a29f4e6e5e7
2023-07-19 16:21:22 -07:00
Alexander Zielenski df86e524c7 refactor: replace usage of v1alpha1 with v1beta1
v1alpha -> v1beta

fill in DenyAction where there is no ParameterNotFoundAction

Kubernetes-commit: ef8670c946d53fda523341658919f9d8bd242d40
2023-07-19 15:53:31 -07:00
Monis Khan 8e93c650b5 kmsv2: KDF based nonce extension
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: bf49c727ba10881d5378e9242f31dc00dede51be
2023-03-25 14:41:04 -04:00
Marek Siarkowicz e9acd0c76d Fix the semantic meaning of etcd server within component statuses and metrics.
Instead of numerating all the etcd endpoints known by apiserver, we will
group them by purpose. `etcd-0` will be the default etcd, `etcd-1` will
be the first resource override, `etcd-2` will be the second override and
so on.

Kubernetes-commit: 03aad1f823cb719fa6e6b6d33fefa2a2140cc760
2023-07-19 14:25:54 +02:00
Alexander Zielenski d501de662c feature: add multiple params capability to VAP controller
Kubernetes-commit: b5e9e0168cf9383dacbd730893c6bc426581e64b
2023-07-10 18:40:45 -07:00
Alexander Zielenski 1f9118f187 refactor: make scope of ParamKind available to vap controller
Kubernetes-commit: 3f63a2d17d4f70dc3ac191a52ad36897086efa7c
2023-07-11 12:04:07 -07:00
Alexander Zielenski 6a8d8652f7 refactor: use the provided sharedInformerFactory for params
Kubernetes-commit: 6323c106e9b5b0edd452a2a223d569a5dae8a832
2023-06-12 18:19:33 -07:00
Nilekh Chaudhari 3373784322 feat: improves metric and logging
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: 2c8288ac873b89ae6e351df5e805b1a825aae60d
2023-07-07 23:01:05 +00:00
Richa Banker 8a38d429d8 Add impl for uvip
Kubernetes-commit: cd5f3d9f9d5ae3153206178e6114d573dc24ad73
2023-03-29 17:20:25 -07:00
Mike Spreitzer b8bc556baa Add tracking and reporting of executing requests
Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Kubernetes-commit: a8a2fb317c8bc9c64ced023988802b2517d34f81
2023-06-30 22:55:35 -04:00
Andrew Sy Kim f00505bddc priority & fairness: support dynamically configuring work estimator max seats
Max seats from prioriy & fairness work estimator is now min(0.15 x
nominalCL, nominalCL/handSize)

'Max seats' calculated by work estimator is currently hard coded to 10.
When using lower values for --max-requests-inflight, a single
LIST request taking up 10 seats could end up using all if not most seats in
the priority level. This change updates the default work estimator
config such that 'max seats' is at most 10% of the
maximum concurrency limit for a priority level, with an upper limit of 10.
This ensures seats taken from LIST request is proportional to the total
available seats.

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

Kubernetes-commit: d3ef2d4fe95c3ef7b1c606ad01be1183659da391
2023-04-26 11:13:14 +00:00
VillageHeadHuang e6c3a57fb9 replace strings.Index with strings.Contains instead
replace strings.Index() !=-1  with strings.Contains instead

Kubernetes-commit: 037847066ce7ebb622c4b207355ac97814414841
2023-07-14 15:25:36 +08:00
Amine 623afac324 Enable admissionWebhookMatchCondition by default
Kubernetes-commit: 94c8ad289bb83c6457d1711a94c8a2cce9a7e8bc
2023-07-17 18:40:07 +01:00
Amine 5782becb79 Graduate AdmissionWebhookMatchCondition to beta
Kubernetes-commit: 0074b24ca42c8afca43f4ef3360f2d101594a567
2023-07-17 18:19:35 +01:00
Divya Sri Sanaganapalli e613190aba Skip apiserver_admission_webhook_request_total during context-canceled
Kubernetes-commit: d3c506133f1d5da6b8681423fc855d0513e8647e
2023-07-17 19:52:43 +00:00
Divya Sri Sanaganapalli 437ae54e84 Ignore context canceled from validate and mutate webhook failopen metric
Kubernetes-commit: 1732b23a343bc0cedbab3dd1df3b7eee4d280036
2023-07-14 20:20:33 +00:00
Mike Spreitzer 003feef4b3 Fix, deprecate apiserver_flowcontrol_request_concurrency_limit
Because it is redundant and has a bad name and its HELP string was
outdated.

Also note intended retention period for request_concurrency_in_use.

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Kubernetes-commit: 75186b1c32a7e9e92ced270eb303a686315a5c44
2023-06-29 01:29:15 -04:00
Andrew Sy Kim 73f18d34af promote the following APF metrics to beta:
apiserver_flowcontrol_request_wait_duration_seconds
apiserver_flowcontrol_request_concurrency_in_use
apiserver_flowcontrol_request_concurrency_limit
apiserver_flowcontrol_rejected_requests_total
apiserver_flowcontrol_dispatched_requests_total
apiserver_flowcontrol_current_inqueue_requests
apiserver_flowcontrol_current_executing_requests

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

Kubernetes-commit: 0bb419b1498a664d1dda3b487e9f15fd220ea363
2023-07-05 18:19:36 +00:00
Cici Huang 04b26c4697 ValidatingAdmissionPolicy: support namespace access (#118267)
* Support namespace access from cel expression in validatingadmissionpolicy.

* Whitelist the exposed fields in namespace object and add test

* better handling of cluster-scoped resources.

* [API REVIEW] namespaceObject in Expression doc.

* compatibility with composition.

* generated: ./hack/update-codegen.sh && ./hack/update-openapi-spec.sh

* workaround namespace of namespace is unexpectedly set.

* basic test coverage for namespaceObject.

---------

Co-authored-by: Jiahui Feng <jhf@google.com>

Kubernetes-commit: 13172cba5c0e1c6a076dbda4aeebbccaf658c7f1
2023-07-15 01:33:59 +00:00
Igor Velichkovich a541a7b473 remove todo/spelling
Kubernetes-commit: 8a4a29d59177699a78f6194861f83789763aac25
2023-07-14 11:08:00 -05:00
Igor Velichkovich c0cd27c353 update histogram
Kubernetes-commit: bef43788fc01775ea156be26d6731c87efbd7b37
2023-07-14 10:10:56 -05:00
Igor Velichkovich 496cd9c142 matchCondition metrics
Kubernetes-commit: 01b9f4b6eb819e4cd4a6d192d703961b34841f18
2023-07-13 19:59:27 -05:00
David Eads 5ed33dc31d add list of served versions to storage version
Kubernetes-commit: 90ab7580aaeca1c6e949df15554ad5bc408dca8e
2023-07-12 18:27:27 -04:00
Mike Spreitzer 90abb3933b QueueSet and controller part of borrowing by exempt PLs
Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Kubernetes-commit: 7305c967101813f794449db21a6edfbc05df2575
2023-06-20 23:35:21 -04:00
Abu Kashem dfc035926b apf: add validation to exempt for borrowing
Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Kubernetes-commit: f8e4e8abac8637f6510838d7d476a838ce612659
2023-05-15 12:08:18 -04:00
Amine 4f6b63aa11 Pre-allocate webhook accessors arrays for mutating and validating
webhooks

Kubernetes-commit: 49d03468021e24434171fde5458df34f6a753a32
2023-07-13 23:43:12 +01:00
Amine 408cf7b500 Improve naming and code comments
Kubernetes-commit: 0695853a3061ece0f602c1f267c82ced3f8c880d
2023-07-12 16:20:14 +01:00
Amine a4fba6de9b Move DeleteFunc logging to level 2
Kubernetes-commit: 28b6c90696680c87cab04b539c0bca6af7d66bac
2023-05-24 13:50:50 -05:00
Amine 278cd016a7 Properly setup mutatingWebhookConfigurationManager{}
Kubernetes-commit: 761016482da9e3febbbfe5ab3cf6c2b31692c1b0
2023-05-17 22:44:18 -05:00
Amine 907129f4f1 run `./hack/verify-gofmt.sh`
Kubernetes-commit: 747dbd9b6b72c21c94ddd17bd2c82bbb2ff583ad
2023-05-17 18:52:37 -05:00