diff --git a/.github/ISSUE_TEMPLATE/support.md b/.github/ISSUE_TEMPLATE/support.md index fd223c8991..cab9c54a05 100644 --- a/.github/ISSUE_TEMPLATE/support.md +++ b/.github/ISSUE_TEMPLATE/support.md @@ -11,7 +11,7 @@ STOP -- PLEASE READ! GitHub is not the right place for support requests. -If you're looking for help, check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) +If you're looking for help, check [Server Fault](https://serverfault.com/questions/tagged/kubernetes). You can also post your question on the [Kubernetes Slack](http://slack.k8s.io/) or the [Discuss Kubernetes](https://discuss.kubernetes.io/) forum. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 88d163d67c..13590450bf 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,4 +1,4 @@ - -本仓库包含了所有用于构建 [Kubernetes 网站和文档](https://kubernetes.io/) 的软件资产。 -我们非常高兴您想要参与贡献！ +本仓库包含了所有用于构建 [Kubernetes 网站和文档](https://kubernetes.io/)的软件资产。 +我们非常高兴你想要参与贡献！ +- [为文档做贡献](#为文档做贡献) +- [README.md 本地化](#readmemd-本地化) + + @@ -30,7 +37,6 @@ To use this repository, you need the following installed locally: - [Go](https://golang.org/) - [Hugo (Extended version)](https://gohugo.io/) - A container runtime, like [Docker](https://www.docker.com/). - --> ## 前提条件 @@ -38,15 +44,15 @@ To use this repository, you need the following installed locally: - [npm](https://www.npmjs.com/) - [Go](https://golang.org/) -- [Hugo (Extended version)](https://gohugo.io/) -- 容器运行时，比如 [Docker](https://www.docker.com/). +- [Hugo（Extended 版本）](https://gohugo.io/) +- 容器运行时，比如 [Docker](https://www.docker.com/)。 开始前，先安装这些依赖。克隆本仓库并进入对应目录： -``` +```bash git clone https://github.com/kubernetes/website.git cd website ``` @@ -55,32 +61,40 @@ cd website The Kubernetes website uses the [Docsy Hugo theme](https://github.com/google/docsy#readme). Even if you plan to run the website in a container, we strongly recommend pulling in the submodule and other development dependencies by running the following: --> -Kubernetes 网站使用的是 [Docsy Hugo 主题](https://github.com/google/docsy#readme)。 即使你打算在容器中运行网站，我们也强烈建议你通过运行以下命令来引入子模块和其他开发依赖项： +Kubernetes 网站使用的是 [Docsy Hugo 主题](https://github.com/google/docsy#readme)。 +即使你打算在容器中运行网站，我们也强烈建议你通过运行以下命令来引入子模块和其他开发依赖项： -``` -# pull in the Docsy submodule +```bash +# 引入 Docsy 子模块 git submodule update --init --recursive --depth 1 ``` ## 在容器中运行网站 -要在容器中构建网站，请通过以下命令来构建容器镜像并运行： +要在容器中构建网站，请运行以下命令： -``` -make container-image +```bash +# 你可以将 $CONTAINER_ENGINE 设置为任何 Docker 类容器工具的名称 make container-serve ``` -启动浏览器，打开 http://localhost:1313 来查看网站。 +如果你看到错误，这可能意味着 Hugo 容器没有足够的可用计算资源。 +要解决这个问题，请增加机器（[MacOSX](https://docs.docker.com/docker-for-mac/#resources) +和 [Windows](https://docs.docker.com/docker-for-windows/#resources)）上 +Docker 允许的 CPU 和内存使用量。 + + +启动浏览器，打开 来查看网站。 当你对源文件作出修改时，Hugo 会更新网站并强制浏览器执行刷新操作。 上述命令会在端口 1313 上启动本地 Hugo 服务器。 -启动浏览器，打开 http://localhost:1313 来查看网站。 +启动浏览器，打开 来查看网站。 当你对源文件作出修改时，Hugo 会更新网站并强制浏览器执行刷新操作。 + +## 构建 API 参考页面 + + +位于 `content/en/docs/reference/kubernetes-api` 的 API 参考页面是根据 Swagger 规范构建的，使用 。 + +要更新 Kubernetes 新版本的参考页面，请执行以下步骤： + + +1. 拉取 `api-ref-generator` 子模块： + + ```bash + git submodule update --init --recursive --depth 1 + ``` + + +2. 更新 Swagger 规范： + + ```bash + curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json + ``` + + +3. 在 `api-ref-assets/config/` 中，调整文件 `toc.yaml` 和 `fields.yaml` 以反映新版本的变化。 + + +4. 接下来，构建页面： + + ```bash + make api-reference + ``` + + + 你可以通过从容器映像创建和提供站点来在本地测试结果： + + ```bash + make container-image + make container-serve + ``` + + + 在 Web 浏览器中，打开 查看 API 参考。 + + +5. 当所有新的更改都反映到配置文件 `toc.yaml` 和 `fields.yaml` 中时，使用新生成的 API 参考页面创建一个 Pull Request。 + ## 故障排除 -### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version +### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version 由于技术原因，Hugo 会发布两套二进制文件。 当前网站仅基于 **Hugo Extended** 版本运行。 -在 [发布页面](https://github.com/gohugoio/hugo/releases) 中查找名称为 `extended` 的归档。可以运行 `huge version` 查看是否有单词 `extended` 来确认。 +在[发布页面](https://github.com/gohugoio/hugo/releases)中查找名称为 `extended` 的归档。 +可以运行 `hugo version` 查看是否有单词 `extended` 来确认。 -### 对 macOs 上打开太多文件的故障排除 +### 对 macOS 上打开太多文件的故障排除 如果在 macOS 上运行 `make serve` 收到以下错误： -``` +```bash ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files make: *** [serve] Error 1 ``` + 试着查看一下当前打开文件数的限制： `launchctl limit maxfiles` -然后运行以下命令（参考https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c）： + +然后运行以下命令（参考 ）： -``` +```shell #!/bin/sh -# These are the original gist links, linking to my gists now. +# 这些是原始的 gist 链接，立即链接到我的 gist。 # curl -O https://gist.githubusercontent.com/a2ikm/761c2ab02b7b3935679e55af5d81786a/raw/ab644cb92f216c019a2f032bbf25e258b01d87f9/limit.maxfiles.plist # curl -O https://gist.githubusercontent.com/a2ikm/761c2ab02b7b3935679e55af5d81786a/raw/ab644cb92f216c019a2f032bbf25e258b01d87f9/limit.maxproc.plist @@ -165,8 +251,56 @@ sudo chown root:wheel /Library/LaunchDaemons/limit.maxproc.plist sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist ``` + 这适用于 Catalina 和 Mojave macOS。 +### 对执行 make container-image 命令部分地区访问超时的故障排除 + +现象如下： + +```shell +langs/language.go:23:2: golang.org/x/text@v0.3.7: Get "https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip": dial tcp 142.251.43.17:443: i/o timeout +langs/language.go:24:2: golang.org/x/text@v0.3.7: Get "https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip": dial tcp 142.251.43.17:443: i/o timeout +common/text/transform.go:21:2: golang.org/x/text@v0.3.7: Get "https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip": dial tcp 142.251.43.17:443: i/o timeout +common/text/transform.go:22:2: golang.org/x/text@v0.3.7: Get "https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip": dial tcp 142.251.43.17:443: i/o timeout +common/text/transform.go:23:2: golang.org/x/text@v0.3.7: Get "https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip": dial tcp 142.251.43.17:443: i/o timeout +hugolib/integrationtest_builder.go:29:2: golang.org/x/tools@v0.1.11: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.11.zip": dial tcp 142.251.42.241:443: i/o timeout +deploy/google.go:24:2: google.golang.org/api@v0.76.0: Get "https://proxy.golang.org/google.golang.org/api/@v/v0.76.0.zip": dial tcp 142.251.43.17:443: i/o timeout +parser/metadecoders/decoder.go:32:2: gopkg.in/yaml.v2@v2.4.0: Get "https://proxy.golang.org/gopkg.in/yaml.v2/@v/v2.4.0.zip": dial tcp 142.251.42.241:443: i/o timeout +The command '/bin/sh -c mkdir $HOME/src && cd $HOME/src && curl -L https://github.com/gohugoio/hugo/archive/refs/tags/v${HUGO_VERSION}.tar.gz | tar -xz && cd "hugo-${HUGO_VERS ION}" && go install --tags extended' returned a non-zero code: 1 +make: *** [Makefile:69：container-image] error 1 +``` + +请修改 `Dockerfile` 文件，为其添加网络代理。修改内容如下： + +```dockerfile +... +FROM golang:1.18-alpine + +LABEL maintainer="Luc Perkins " + +ENV GO111MODULE=on # 需要添加内容1 + +ENV GOPROXY=https://proxy.golang.org,direct # 需要添加内容2 + +RUN apk add --no-cache \ + curl \ + gcc \ + g++ \ + musl-dev \ + build-base \ + libc6-compat + +ARG HUGO_VERSION +... +``` + +将 "https://proxy.golang.org" 替换为本地可以使用的代理地址。 + +**注意：** 此部分仅适用于中国大陆 + -# 参与 SIG Docs 工作 +## 参与 SIG Docs 工作 -通过 [社区页面](https://github.com/kubernetes/community/tree/master/sig-docs#meetings) -进一步了解 SIG Docs Kubernetes 社区和会议信息。 +通过[社区页面](https://github.com/kubernetes/community/tree/master/sig-docs#meetings)进一步了解 +SIG Docs Kubernetes 社区和会议信息。 你也可以通过以下渠道联系本项目的维护人员： -- [Slack](https://kubernetes.slack.com/messages/sig-docs) [加入Slack](https://slack.k8s.io/) +- [Slack](https://kubernetes.slack.com/messages/sig-docs) + - [获得此 Slack 的邀请](https://slack.k8s.io/) - [邮件列表](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) -# 为文档做贡献 +## 为文档做贡献 你也可以点击屏幕右上方区域的 **Fork** 按钮，在你自己的 GitHub -账号下创建本仓库的拷贝。此拷贝被称作 *fork*。 +账号下创建本仓库的拷贝。此拷贝被称作 **fork**。 你可以在自己的拷贝中任意地修改文档，并在你已准备好将所作修改提交给我们时， 在你自己的拷贝下创建一个拉取请求（Pull Request），以便让我们知道。 一旦你创建了拉取请求，某个 Kubernetes 评审人会负责提供明确的、可执行的反馈意见。 -作为拉取请求的拥有者，*修改拉取请求以解决 Kubernetes -评审人所提出的反馈是你的责任*。 +作为拉取请求的拥有者，**修改拉取请求以解决 Kubernetes 评审人所提出的反馈是你的责任**。 还要提醒的一点，有时可能会有不止一个 Kubernetes 评审人为你提供反馈意见。 有时候，某个评审人的意见和另一个最初被指派的评审人的意见不同。 -更进一步，在某些时候，评审人之一可能会在需要的时候请求 Kubernetes -技术评审人来执行技术评审。 -评审人会尽力及时地提供反馈意见，不过具体的响应时间可能会因时而异。 +另外在某些时候，某个评审人可能会在需要的时候请求一名 Kubernetes 技术评审人来执行技术评审。 +这些评审人会尽力及时地提供反馈意见，不过具体的响应时间可能会因时而异。 有关为 Kubernetes 文档做出贡献的更多信息，请参阅： -* [贡献 Kubernetes 文档](https://kubernetes.io/docs/contribute/) -* [页面内容类型](https://kubernetes.io/docs/contribute/style/page-content-types/) -* [文档风格指南](https://kubernetes.io/docs/contribute/style/style-guide/) -* [本地化 Kubernetes 文档](https://kubernetes.io/docs/contribute/localization/) +- [贡献 Kubernetes 文档](https://kubernetes.io/zh-cn/docs/contribute/) +- [页面内容类型](https://kubernetes.io/zh-cn/docs/contribute/style/page-content-types/) +- [文档风格指南](https://kubernetes.io/zh-cn/docs/contribute/style/style-guide/) +- [本地化 Kubernetes 文档](https://kubernetes.io/zh-cn/docs/contribute/localization/) + + +### 新贡献者大使 + + +如果你在贡献时需要帮助，[新贡献者大使](https://kubernetes.io/zh-cn/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador)是一个很好的联系人。 +这些是 SIG Docs 批准者，其职责包括指导新贡献者并帮助他们完成最初的几个拉取请求。 +联系新贡献者大使的最佳地点是 [Kubernetes Slack](https://slack.k8s.io/)。 +SIG Docs 的当前新贡献者大使： + + +| 姓名 | Slack | GitHub | +| -------------------------- | -------------------------- | -------------------------- | +| Arsh Sharma | @arsh | @RinkiyaKeDad | + + +## `README.md` 本地化 + + +| 语言 | 语言 | +| -------------------------- | -------------------------- | +| [中文](README-zh.md) | [韩语](README-ko.md) | +| [法语](README-fr.md) | [波兰语](README-pl.md) | +| [德语](README-de.md) | [葡萄牙语](README-pt.md) | +| [印地语](README-hi.md) | [俄语](README-ru.md) | +| [印尼语](README-id.md) | [西班牙语](README-es.md) | +| [意大利语](README-it.md) | [乌克兰语](README-uk.md) | +| [日语](README-ja.md) | [越南语](README-vi.md) | # 中文本地化 @@ -238,22 +420,22 @@ For more information about contributing to the Kubernetes documentation, see: * Rui Chen ([GitHub - @chenrui333](https://github.com/chenrui333)) * He Xiaolong ([GitHub - @markthink](https://github.com/markthink)) -* [Slack channel](https://kubernetes.slack.com/messages/kubernetes-docs-zh) +* [Slack 频道](https://kubernetes.slack.com/messages/kubernetes-docs-zh) -# 行为准则 +## 行为准则 -参与 Kubernetes 社区受 [CNCF 行为准则](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) 约束。 +参与 Kubernetes 社区受 [CNCF 行为准则](https://github.com/cncf/foundation/blob/main/code-of-conduct.md)约束。 -# 感谢！ +## 感谢你 -Kubernetes 因为社区的参与而蓬勃发展，感谢您对我们网站和文档的贡献！ +Kubernetes 因为社区的参与而蓬勃发展，感谢你对我们网站和文档的贡献！ diff --git a/README.md b/README.md index 74f99e2ffc..f037e0426b 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,10 @@ git submodule update --init --recursive --depth 1 ## Running the website using a container -To build the site in a container, run the following to build the container image and run it: +To build the site in a container, run the following: ```bash -make container-image +# You can set $CONTAINER_ENGINE to the name of any Docker-like container tool make container-serve ``` @@ -65,9 +65,9 @@ This will start the local Hugo server on port 1313. Open up your browser to . -To update the reference pages for a new Kubernetes release (replace v1.20 in the following examples with the release to update to): +To update the reference pages for a new Kubernetes release follow these steps: -1. Pull the `kubernetes-resources-reference` submodule: +1. Pull in the `api-ref-generator` submodule: ```bash git submodule update --init --recursive --depth 1 @@ -75,9 +75,9 @@ To update the reference pages for a new Kubernetes release (replace v1.20 in the 2. Update the Swagger specification: -``` -curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json -``` + ```bash + curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json + ``` 3. In `api-ref-assets/config/`, adapt the files `toc.yaml` and `fields.yaml` to reflect the changes of the new release. @@ -146,7 +146,8 @@ Learn more about SIG Docs Kubernetes community and meetings on the [community pa You can also reach the maintainers of this project at: -- [Slack](https://kubernetes.slack.com/messages/sig-docs) [Get an invite for this Slack](https://slack.k8s.io/) +- [Slack](https://kubernetes.slack.com/messages/sig-docs) + - [Get an invite for this Slack](https://slack.k8s.io/) - [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) ## Contributing to the docs @@ -166,6 +167,14 @@ For more information about contributing to the Kubernetes documentation, see: - [Documentation Style Guide](https://kubernetes.io/docs/contribute/style/style-guide/) - [Localizing Kubernetes Documentation](https://kubernetes.io/docs/contribute/localization/) +### New contributor ambassadors + +If you need help at any point when contributing, the [New Contributor Ambassadors](https://kubernetes.io/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador) are a good point of contact. These are SIG Docs approvers whose responsibilities include mentoring new contributors and helping them through their first few pull requests. The best place to contact the New Contributors Ambassadors would be on the [Kubernetes Slack](https://slack.k8s.io/). Current New Contributors Ambassadors for SIG Docs: + +| Name | Slack | GitHub | +| -------------------------- | -------------------------- | -------------------------- | +| Arsh Sharma | @arsh | @RinkiyaKeDad | + ## Localization `README.md`'s | Language | Language | @@ -180,7 +189,7 @@ For more information about contributing to the Kubernetes documentation, see: ## Code of conduct -Participation in the Kubernetes community is governed by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). +Participation in the Kubernetes community is governed by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). ## Thank you diff --git a/SECURITY_CONTACTS b/SECURITY_CONTACTS index 77910dc182..839e7bed14 100644 --- a/SECURITY_CONTACTS +++ b/SECURITY_CONTACTS @@ -10,5 +10,6 @@ # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ +divya-mohan0209 jimangel -sftim \ No newline at end of file +sftim diff --git a/api-ref-assets/api/swagger.json b/api-ref-assets/api/swagger.json index 5404b05fb9..399f47223f 100644 --- a/api-ref-assets/api/swagger.json +++ b/api-ref-assets/api/swagger.json @@ -367,7 +367,7 @@ "type": "object" }, "io.k8s.api.apiserverinternal.v1alpha1.StorageVersion": { - "description": "\n Storage version of a specific resource.", + "description": "Storage version of a specific resource.", "properties": { "apiVersion": { "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", @@ -750,7 +750,7 @@ "type": "integer" }, "numberReady": { - "description": "The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready.", + "description": "numberReady is the number of nodes that should be running the daemon pod and have one or more of the daemon pod running with a Ready Condition.", "format": "int32", "type": "integer" }, @@ -786,7 +786,7 @@ "description": "Rolling update config params. Present only if type = \"RollingUpdate\"." }, "type": { - "description": "Type of daemon set update. Can be \"RollingUpdate\" or \"OnDelete\". Default is RollingUpdate.", + "description": "Type of daemon set update. Can be \"RollingUpdate\" or \"OnDelete\". Default is RollingUpdate.\n\n", "type": "string" } }, @@ -969,7 +969,7 @@ "type": "integer" }, "readyReplicas": { - "description": "Total number of ready pods targeted by this deployment.", + "description": "readyReplicas is the number of pods targeted by this Deployment with a Ready Condition.", "format": "int32", "type": "integer" }, @@ -999,7 +999,7 @@ "description": "Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate." }, "type": { - "description": "Type of deployment. Can be \"Recreate\" or \"RollingUpdate\". Default is RollingUpdate.", + "description": "Type of deployment. Can be \"Recreate\" or \"RollingUpdate\". Default is RollingUpdate.\n\n", "type": "string" } }, @@ -1158,7 +1158,7 @@ "type": "integer" }, "readyReplicas": { - "description": "The number of ready replicas for this replica set.", + "description": "readyReplicas is the number of pods targeted by this ReplicaSet with a Ready Condition.", "format": "int32", "type": "integer" }, @@ -1204,8 +1204,12 @@ "io.k8s.api.apps.v1.RollingUpdateStatefulSetStrategy": { "description": "RollingUpdateStatefulSetStrategy is used to communicate parameter for RollingUpdateStatefulSetStrategyType.", "properties": { + "maxUnavailable": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.util.intstr.IntOrString", + "description": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0. Defaults to 1. This field is alpha-level and is only honored by servers that enable the MaxUnavailableStatefulSet feature. The field applies to all pods in the range 0 to Replicas-1. That means if there is any unavailable pod in the range 0 to Replicas-1, it will be counted towards MaxUnavailable." + }, "partition": { - "description": "Partition indicates the ordinal at which the StatefulSet should be partitioned. Default value is 0.", + "description": "Partition indicates the ordinal at which the StatefulSet should be partitioned for updates. During a rolling update, all pods from ordinal Replicas-1 to Partition are updated. All pods from ordinal Partition-1 to 0 remain untouched. This is helpful in being able to do a canary based deployment. The default value is 0.", "format": "int32", "type": "integer" } @@ -1310,6 +1314,20 @@ } ] }, + "io.k8s.api.apps.v1.StatefulSetPersistentVolumeClaimRetentionPolicy": { + "description": "StatefulSetPersistentVolumeClaimRetentionPolicy describes the policy used for PVCs created from the StatefulSet VolumeClaimTemplates.", + "properties": { + "whenDeleted": { + "description": "WhenDeleted specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is deleted. The default policy of `Retain` causes PVCs to not be affected by StatefulSet deletion. The `Delete` policy causes those PVCs to be deleted.", + "type": "string" + }, + "whenScaled": { + "description": "WhenScaled specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is scaled down. The default policy of `Retain` causes PVCs to not be affected by a scaledown. The `Delete` policy causes the associated PVCs for any excess pods above the replica count to be deleted.", + "type": "string" + } + }, + "type": "object" + }, "io.k8s.api.apps.v1.StatefulSetSpec": { "description": "A StatefulSetSpec is the specification of a StatefulSet.", "properties": { @@ -1318,8 +1336,12 @@ "format": "int32", "type": "integer" }, + "persistentVolumeClaimRetentionPolicy": { + "$ref": "#/definitions/io.k8s.api.apps.v1.StatefulSetPersistentVolumeClaimRetentionPolicy", + "description": "persistentVolumeClaimRetentionPolicy describes the lifecycle of persistent volume claims created from volumeClaimTemplates. By default, all persistent volume claims are created as needed and retained until manually deleted. This policy allows the lifecycle to be altered, for example by deleting persistent volume claims when their stateful set is deleted, or when their pod is scaled down. This requires the StatefulSetAutoDeletePVC feature gate to be enabled, which is alpha. +optional" + }, "podManagementPolicy": { - "description": "podManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. The default policy is `OrderedReady`, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling down, the pods are removed in the opposite order. The alternative policy is `Parallel` which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.", + "description": "podManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. The default policy is `OrderedReady`, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling down, the pods are removed in the opposite order. The alternative policy is `Parallel` which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.\n\n", "type": "string" }, "replicas": { @@ -1367,7 +1389,7 @@ "description": "StatefulSetStatus represents the current state of a StatefulSet.", "properties": { "availableReplicas": { - "description": "Total number of available pods (ready for at least minReadySeconds) targeted by this statefulset. This is an alpha field and requires enabling StatefulSetMinReadySeconds feature gate. Remove omitempty when graduating to beta", + "description": "Total number of available pods (ready for at least minReadySeconds) targeted by this statefulset. This is a beta field and enabled/disabled by StatefulSetMinReadySeconds feature gate.", "format": "int32", "type": "integer" }, @@ -1400,7 +1422,7 @@ "type": "integer" }, "readyReplicas": { - "description": "readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.", + "description": "readyReplicas is the number of pods created for this StatefulSet with a Ready Condition.", "format": "int32", "type": "integer" }, @@ -1432,7 +1454,7 @@ "description": "RollingUpdate is used to communicate parameters when Type is RollingUpdateStatefulSetStrategyType." }, "type": { - "description": "Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate.", + "description": "Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate.\n\n", "type": "string" } }, @@ -2237,6 +2259,601 @@ ], "type": "object" }, + "io.k8s.api.autoscaling.v2.ContainerResourceMetricSource": { + "description": "ContainerResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source. Only one \"target\" type should be set.", + "properties": { + "container": { + "description": "container is the name of the container in the pods of the scaling target", + "type": "string" + }, + "name": { + "description": "name is the name of the resource in question.", + "type": "string" + }, + "target": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricTarget", + "description": "target specifies the target value for the given metric" + } + }, + "required": [ + "name", + "target", + "container" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ContainerResourceMetricStatus": { + "description": "ContainerResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing a single container in each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source.", + "properties": { + "container": { + "description": "Container is the name of the container in the pods of the scaling target", + "type": "string" + }, + "current": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricValueStatus", + "description": "current contains the current value for the given metric" + }, + "name": { + "description": "Name is the name of the resource in question.", + "type": "string" + } + }, + "required": [ + "name", + "current", + "container" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.CrossVersionObjectReference": { + "description": "CrossVersionObjectReference contains enough information to let you identify the referred resource.", + "properties": { + "apiVersion": { + "description": "API version of the referent", + "type": "string" + }, + "kind": { + "description": "Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\"", + "type": "string" + }, + "name": { + "description": "Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names", + "type": "string" + } + }, + "required": [ + "kind", + "name" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ExternalMetricSource": { + "description": "ExternalMetricSource indicates how to scale on a metric not associated with any Kubernetes object (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cluster).", + "properties": { + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + }, + "target": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricTarget", + "description": "target specifies the target value for the given metric" + } + }, + "required": [ + "metric", + "target" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ExternalMetricStatus": { + "description": "ExternalMetricStatus indicates the current value of a global metric not associated with any Kubernetes object.", + "properties": { + "current": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricValueStatus", + "description": "current contains the current value for the given metric" + }, + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + } + }, + "required": [ + "metric", + "current" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HPAScalingPolicy": { + "description": "HPAScalingPolicy is a single policy which must hold true for a specified past interval.", + "properties": { + "periodSeconds": { + "description": "PeriodSeconds specifies the window of time for which the policy should hold true. PeriodSeconds must be greater than zero and less than or equal to 1800 (30 min).", + "format": "int32", + "type": "integer" + }, + "type": { + "description": "Type is used to specify the scaling policy.", + "type": "string" + }, + "value": { + "description": "Value contains the amount of change which is permitted by the policy. It must be greater than zero", + "format": "int32", + "type": "integer" + } + }, + "required": [ + "type", + "value", + "periodSeconds" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HPAScalingRules": { + "description": "HPAScalingRules configures the scaling behavior for one direction. These Rules are applied after calculating DesiredReplicas from metrics for the HPA. They can limit the scaling velocity by specifying scaling policies. They can prevent flapping by specifying the stabilization window, so that the number of replicas is not set instantly, instead, the safest value from the stabilization window is chosen.", + "properties": { + "policies": { + "description": "policies is a list of potential scaling polices which can be used during scaling. At least one policy must be specified, otherwise the HPAScalingRules will be discarded as invalid", + "items": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HPAScalingPolicy" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + }, + "selectPolicy": { + "description": "selectPolicy is used to specify which policy should be used. If not set, the default value Max is used.", + "type": "string" + }, + "stabilizationWindowSeconds": { + "description": "StabilizationWindowSeconds is the number of seconds for which past recommendations should be considered while scaling up or scaling down. StabilizationWindowSeconds must be greater than or equal to zero and less than or equal to 3600 (one hour). If not set, use the default values: - For scale up: 0 (i.e. no stabilization is done). - For scale down: 300 (i.e. the stabilization window is 300 seconds long).", + "format": "int32", + "type": "integer" + } + }, + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler": { + "description": "HorizontalPodAutoscaler is the configuration for a horizontal pod autoscaler, which automatically manages the replica count of any resource implementing the scale subresource based on the metrics specified.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", + "description": "metadata is the standard object metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + }, + "spec": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerSpec", + "description": "spec is the specification for the behaviour of the autoscaler. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status." + }, + "status": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerStatus", + "description": "status is the current information about the autoscaler." + } + }, + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + ] + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerBehavior": { + "description": "HorizontalPodAutoscalerBehavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively).", + "properties": { + "scaleDown": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HPAScalingRules", + "description": "scaleDown is scaling policy for scaling Down. If not set, the default value is to allow to scale down to minReplicas pods, with a 300 second stabilization window (i.e., the highest recommendation for the last 300sec is used)." + }, + "scaleUp": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HPAScalingRules", + "description": "scaleUp is scaling policy for scaling Up. If not set, the default value is the higher of:\n * increase no more than 4 pods per 60 seconds\n * double the number of pods per 60 seconds\nNo stabilization is used." + } + }, + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerCondition": { + "description": "HorizontalPodAutoscalerCondition describes the state of a HorizontalPodAutoscaler at a certain point.", + "properties": { + "lastTransitionTime": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", + "description": "lastTransitionTime is the last time the condition transitioned from one status to another" + }, + "message": { + "description": "message is a human-readable explanation containing details about the transition", + "type": "string" + }, + "reason": { + "description": "reason is the reason for the condition's last transition.", + "type": "string" + }, + "status": { + "description": "status is the status of the condition (True, False, Unknown)", + "type": "string" + }, + "type": { + "description": "type describes the current condition", + "type": "string" + } + }, + "required": [ + "type", + "status" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerList": { + "description": "HorizontalPodAutoscalerList is a list of horizontal pod autoscaler objects.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "items": { + "description": "items is the list of horizontal pod autoscaler objects.", + "items": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + }, + "type": "array" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", + "description": "metadata is the standard list metadata." + } + }, + "required": [ + "items" + ], + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "autoscaling", + "kind": "HorizontalPodAutoscalerList", + "version": "v2" + } + ] + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerSpec": { + "description": "HorizontalPodAutoscalerSpec describes the desired functionality of the HorizontalPodAutoscaler.", + "properties": { + "behavior": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerBehavior", + "description": "behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used." + }, + "maxReplicas": { + "description": "maxReplicas is the upper limit for the number of replicas to which the autoscaler can scale up. It cannot be less that minReplicas.", + "format": "int32", + "type": "integer" + }, + "metrics": { + "description": "metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used). The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods. Ergo, metrics used must decrease as the pod count is increased, and vice-versa. See the individual metric source types for more information about how each type of metric must respond. If not set, the default metric will be set to 80% average CPU utilization.", + "items": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricSpec" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + }, + "minReplicas": { + "description": "minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one Object or External metric is configured. Scaling is active as long as at least one metric value is available.", + "format": "int32", + "type": "integer" + }, + "scaleTargetRef": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.CrossVersionObjectReference", + "description": "scaleTargetRef points to the target resource to scale, and is used to the pods for which metrics should be collected, as well as to actually change the replica count." + } + }, + "required": [ + "scaleTargetRef", + "maxReplicas" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerStatus": { + "description": "HorizontalPodAutoscalerStatus describes the current status of a horizontal pod autoscaler.", + "properties": { + "conditions": { + "description": "conditions is the set of conditions required for this autoscaler to scale its target, and indicates whether or not those conditions are met.", + "items": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerCondition" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "type" + ], + "x-kubernetes-list-type": "map", + "x-kubernetes-patch-merge-key": "type", + "x-kubernetes-patch-strategy": "merge" + }, + "currentMetrics": { + "description": "currentMetrics is the last read state of the metrics used by this autoscaler.", + "items": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricStatus" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + }, + "currentReplicas": { + "description": "currentReplicas is current number of replicas of pods managed by this autoscaler, as last seen by the autoscaler.", + "format": "int32", + "type": "integer" + }, + "desiredReplicas": { + "description": "desiredReplicas is the desired number of replicas of pods managed by this autoscaler, as last calculated by the autoscaler.", + "format": "int32", + "type": "integer" + }, + "lastScaleTime": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", + "description": "lastScaleTime is the last time the HorizontalPodAutoscaler scaled the number of pods, used by the autoscaler to control how often the number of pods is changed." + }, + "observedGeneration": { + "description": "observedGeneration is the most recent generation observed by this autoscaler.", + "format": "int64", + "type": "integer" + } + }, + "required": [ + "desiredReplicas" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.MetricIdentifier": { + "description": "MetricIdentifier defines the name and optionally selector for a metric", + "properties": { + "name": { + "description": "name is the name of the given metric", + "type": "string" + }, + "selector": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", + "description": "selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics." + } + }, + "required": [ + "name" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.MetricSpec": { + "description": "MetricSpec specifies how to scale based on a single metric (only `type` and one other matching field should be set at once).", + "properties": { + "containerResource": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ContainerResourceMetricSource", + "description": "containerResource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod of the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source. This is an alpha feature and can be enabled by the HPAContainerMetrics feature flag." + }, + "external": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ExternalMetricSource", + "description": "external refers to a global metric that is not associated with any Kubernetes object. It allows autoscaling based on information coming from components running outside of cluster (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cluster)." + }, + "object": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ObjectMetricSource", + "description": "object refers to a metric describing a single kubernetes object (for example, hits-per-second on an Ingress object)." + }, + "pods": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.PodsMetricSource", + "description": "pods refers to a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value." + }, + "resource": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ResourceMetricSource", + "description": "resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source." + }, + "type": { + "description": "type is the type of metric source. It should be one of \"ContainerResource\", \"External\", \"Object\", \"Pods\" or \"Resource\", each mapping to a matching field in the object. Note: \"ContainerResource\" type is available on when the feature-gate HPAContainerMetrics is enabled", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.MetricStatus": { + "description": "MetricStatus describes the last-read state of a single metric.", + "properties": { + "containerResource": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ContainerResourceMetricStatus", + "description": "container resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source." + }, + "external": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ExternalMetricStatus", + "description": "external refers to a global metric that is not associated with any Kubernetes object. It allows autoscaling based on information coming from components running outside of cluster (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cluster)." + }, + "object": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ObjectMetricStatus", + "description": "object refers to a metric describing a single kubernetes object (for example, hits-per-second on an Ingress object)." + }, + "pods": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.PodsMetricStatus", + "description": "pods refers to a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value." + }, + "resource": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.ResourceMetricStatus", + "description": "resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source." + }, + "type": { + "description": "type is the type of metric source. It will be one of \"ContainerResource\", \"External\", \"Object\", \"Pods\" or \"Resource\", each corresponds to a matching field in the object. Note: \"ContainerResource\" type is available on when the feature-gate HPAContainerMetrics is enabled", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.MetricTarget": { + "description": "MetricTarget defines the target value, average value, or average utilization of a specific metric", + "properties": { + "averageUtilization": { + "description": "averageUtilization is the target value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods. Currently only valid for Resource metric source type", + "format": "int32", + "type": "integer" + }, + "averageValue": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "averageValue is the target value of the average of the metric across all relevant pods (as a quantity)" + }, + "type": { + "description": "type represents whether the metric type is Utilization, Value, or AverageValue", + "type": "string" + }, + "value": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "value is the target value of the metric (as a quantity)." + } + }, + "required": [ + "type" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.MetricValueStatus": { + "description": "MetricValueStatus holds the current value for a metric", + "properties": { + "averageUtilization": { + "description": "currentAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods.", + "format": "int32", + "type": "integer" + }, + "averageValue": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "averageValue is the current value of the average of the metric across all relevant pods (as a quantity)" + }, + "value": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "value is the current value of the metric (as a quantity)." + } + }, + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ObjectMetricSource": { + "description": "ObjectMetricSource indicates how to scale on a metric describing a kubernetes object (for example, hits-per-second on an Ingress object).", + "properties": { + "describedObject": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.CrossVersionObjectReference", + "description": "describedObject specifies the descriptions of a object,such as kind,name apiVersion" + }, + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + }, + "target": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricTarget", + "description": "target specifies the target value for the given metric" + } + }, + "required": [ + "describedObject", + "target", + "metric" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ObjectMetricStatus": { + "description": "ObjectMetricStatus indicates the current value of a metric describing a kubernetes object (for example, hits-per-second on an Ingress object).", + "properties": { + "current": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricValueStatus", + "description": "current contains the current value for the given metric" + }, + "describedObject": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.CrossVersionObjectReference", + "description": "DescribedObject specifies the descriptions of a object,such as kind,name apiVersion" + }, + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + } + }, + "required": [ + "metric", + "current", + "describedObject" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.PodsMetricSource": { + "description": "PodsMetricSource indicates how to scale on a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value.", + "properties": { + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + }, + "target": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricTarget", + "description": "target specifies the target value for the given metric" + } + }, + "required": [ + "metric", + "target" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.PodsMetricStatus": { + "description": "PodsMetricStatus indicates the current value of a metric describing each pod in the current scale target (for example, transactions-processed-per-second).", + "properties": { + "current": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricValueStatus", + "description": "current contains the current value for the given metric" + }, + "metric": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricIdentifier", + "description": "metric identifies the target metric by name and selector" + } + }, + "required": [ + "metric", + "current" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ResourceMetricSource": { + "description": "ResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source. Only one \"target\" type should be set.", + "properties": { + "name": { + "description": "name is the name of the resource in question.", + "type": "string" + }, + "target": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricTarget", + "description": "target specifies the target value for the given metric" + } + }, + "required": [ + "name", + "target" + ], + "type": "object" + }, + "io.k8s.api.autoscaling.v2.ResourceMetricStatus": { + "description": "ResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source.", + "properties": { + "current": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.MetricValueStatus", + "description": "current contains the current value for the given metric" + }, + "name": { + "description": "Name is the name of the resource in question.", + "type": "string" + } + }, + "required": [ + "name", + "current" + ], + "type": "object" + }, "io.k8s.api.autoscaling.v2beta1.ContainerResourceMetricSource": { "description": "ContainerResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the \"pods\" source. Only one \"target\" type should be set.", "properties": { @@ -2533,8 +3150,7 @@ }, "required": [ "currentReplicas", - "desiredReplicas", - "conditions" + "desiredReplicas" ], "type": "object" }, @@ -3092,8 +3708,7 @@ }, "required": [ "currentReplicas", - "desiredReplicas", - "conditions" + "desiredReplicas" ], "type": "object" }, @@ -3413,7 +4028,7 @@ "description": "CronJobSpec describes how the job execution will look like and when it will actually run.", "properties": { "concurrencyPolicy": { - "description": "Specifies how to treat concurrent executions of a Job. Valid values are: - \"Allow\" (default): allows CronJobs to run concurrently; - \"Forbid\": forbids concurrent runs, skipping next run if previous run hasn't finished yet; - \"Replace\": cancels currently running job and replaces it with a new one", + "description": "Specifies how to treat concurrent executions of a Job. Valid values are: - \"Allow\" (default): allows CronJobs to run concurrently; - \"Forbid\": forbids concurrent runs, skipping next run if previous run hasn't finished yet; - \"Replace\": cancels currently running job and replaces it with a new one\n\n", "type": "string" }, "failedJobsHistoryLimit": { @@ -3442,6 +4057,10 @@ "suspend": { "description": "This flag tells the controller to suspend subsequent executions, it does not apply to already started executions. Defaults to false.", "type": "boolean" + }, + "timeZone": { + "description": "The time zone for the given schedule, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. If not specified, this will rely on the time zone of the kube-controller-manager process. ALPHA: This field is in alpha and must be enabled via the `CronJobTimeZone` feature gate.", + "type": "string" } }, "required": [ @@ -3588,7 +4207,7 @@ "type": "integer" }, "completionMode": { - "description": "CompletionMode specifies how Pod completions are tracked. It can be `NonIndexed` (default) or `Indexed`.\n\n`NonIndexed` means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.\n\n`Indexed` means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is `Indexed`, .spec.completions must be specified and `.spec.parallelism` must be less than or equal to 10^5. In addition, The Pod name takes the form `$(job-name)-$(index)-$(random-string)`, the Pod hostname takes the form `$(job-name)-$(index)`.\n\nThis field is beta-level. More completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, the controller skips updates for the Job.", + "description": "CompletionMode specifies how Pod completions are tracked. It can be `NonIndexed` (default) or `Indexed`.\n\n`NonIndexed` means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.\n\n`Indexed` means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is `Indexed`, .spec.completions must be specified and `.spec.parallelism` must be less than or equal to 10^5. In addition, The Pod name takes the form `$(job-name)-$(index)-$(random-string)`, the Pod hostname takes the form `$(job-name)-$(index)`.\n\nMore completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, which is possible during upgrades due to version skew, the controller skips updates for the Job.", "type": "string" }, "completions": { @@ -3610,7 +4229,7 @@ "description": "A label query over pods that should match the pod count. Normally, the system sets this field for you. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors" }, "suspend": { - "description": "Suspend specifies whether the Job controller should create Pods or not. If a Job is created with suspend set to true, no Pods are created by the Job controller. If a Job is suspended after creation (i.e. the flag goes from false to true), the Job controller will delete all active Pods associated with this Job. Users must design their workload to gracefully handle this. Suspending a Job will reset the StartTime field of the Job, effectively resetting the ActiveDeadlineSeconds timer too. Defaults to false.\n\nThis field is beta-level, gated by SuspendJob feature flag (enabled by default).", + "description": "Suspend specifies whether the Job controller should create Pods or not. If a Job is created with suspend set to true, no Pods are created by the Job controller. If a Job is suspended after creation (i.e. the flag goes from false to true), the Job controller will delete all active Pods associated with this Job. Users must design their workload to gracefully handle this. Suspending a Job will reset the StartTime field of the Job, effectively resetting the ActiveDeadlineSeconds timer too. Defaults to false.", "type": "boolean" }, "template": { @@ -3618,7 +4237,7 @@ "description": "Describes the pod that will be created when executing a job. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" }, "ttlSecondsAfterFinished": { - "description": "ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won't be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. This field is alpha-level and is only honored by servers that enable the TTLAfterFinished feature.", + "description": "ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won't be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes.", "format": "int32", "type": "integer" } @@ -3632,7 +4251,7 @@ "description": "JobStatus represents the current state of a Job.", "properties": { "active": { - "description": "The number of actively running pods.", + "description": "The number of pending and running pods.", "format": "int32", "type": "integer" }, @@ -3659,6 +4278,11 @@ "format": "int32", "type": "integer" }, + "ready": { + "description": "The number of pods which have a Ready condition.\n\nThis field is beta-level. The job controller populates the field when the feature gate JobReadyPods is enabled (enabled by default).", + "format": "int32", + "type": "integer" + }, "startTime": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", "description": "Represents time when the job controller started processing a job. When a Job is created in the suspended state, this field is not set until the first time it is resumed. This field is reset every time a Job is resumed from suspension. It is represented in RFC3339 form and is in UTC." @@ -3670,7 +4294,7 @@ }, "uncountedTerminatedPods": { "$ref": "#/definitions/io.k8s.api.batch.v1.UncountedTerminatedPods", - "description": "UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters.\n\nThe job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding\n counter.\n\nThis field is alpha-level. The job controller only makes use of this field when the feature gate PodTrackingWithFinalizers is enabled. Old jobs might not be tracked using this field, in which case the field remains null." + "description": "UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters.\n\nThe job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding\n counter.\n\nThis field is beta-level. The job controller only makes use of this field when the feature gate JobTrackingWithFinalizers is enabled (enabled by default). Old jobs might not be tracked using this field, in which case the field remains null." } }, "type": "object" @@ -3812,6 +4436,10 @@ "suspend": { "description": "This flag tells the controller to suspend subsequent executions, it does not apply to already started executions. Defaults to false.", "type": "boolean" + }, + "timeZone": { + "description": "The time zone for the given schedule, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. If not specified, this will rely on the time zone of the kube-controller-manager process. ALPHA: This field is in alpha and must be enabled via the `CronJobTimeZone` feature gate.", + "type": "string" } }, "required": [ @@ -3963,7 +4591,7 @@ "description": "CertificateSigningRequestSpec contains the certificate request.", "properties": { "expirationSeconds": { - "description": "expirationSeconds is the requested duration of validity of the issued certificate. The certificate signer may issue a certificate with a different validity duration so a client must check the delta between the notBefore and and notAfter fields in the issued certificate to determine the actual duration.\n\nThe v1.22+ in-tree implementations of the well-known Kubernetes signers will honor this field as long as the requested duration is not greater than the maximum duration they will honor per the --cluster-signing-duration CLI flag to the Kubernetes controller manager.\n\nCertificate signers may not honor this field for various reasons:\n\n 1. Old signer that is unaware of the field (such as the in-tree\n implementations prior to v1.22)\n 2. Signer whose configured maximum is shorter than the requested duration\n 3. Signer whose configured minimum is longer than the requested duration\n\nThe minimum valid value for expirationSeconds is 600, i.e. 10 minutes.\n\nAs of v1.22, this field is beta and is controlled via the CSRDuration feature gate.", + "description": "expirationSeconds is the requested duration of validity of the issued certificate. The certificate signer may issue a certificate with a different validity duration so a client must check the delta between the notBefore and and notAfter fields in the issued certificate to determine the actual duration.\n\nThe v1.22+ in-tree implementations of the well-known Kubernetes signers will honor this field as long as the requested duration is not greater than the maximum duration they will honor per the --cluster-signing-duration CLI flag to the Kubernetes controller manager.\n\nCertificate signers may not honor this field for various reasons:\n\n 1. Old signer that is unaware of the field (such as the in-tree\n implementations prior to v1.22)\n 2. Signer whose configured maximum is shorter than the requested duration\n 3. Signer whose configured minimum is longer than the requested duration\n\nThe minimum valid value for expirationSeconds is 600, i.e. 10 minutes.", "format": "int32", "type": "integer" }, @@ -4137,20 +4765,20 @@ "description": "Represents a Persistent Disk resource in AWS.\n\nAn AWS EBS disk must exist before mounting to a container. The disk must also be in the same AWS zone as the kubelet. An AWS EBS disk can only be mounted as read/write once. AWS EBS volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", + "description": "fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", "type": "string" }, "partition": { - "description": "The partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as \"1\". Similarly, the volume partition for /dev/sda is \"0\" (or you can leave the property empty).", + "description": "partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as \"1\". Similarly, the volume partition for /dev/sda is \"0\" (or you can leave the property empty).", "format": "int32", "type": "integer" }, "readOnly": { - "description": "Specify \"true\" to force and set the ReadOnly property in VolumeMounts to \"true\". If omitted, the default is \"false\". More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", + "description": "readOnly value true will force the readOnly setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", "type": "boolean" }, "volumeID": { - "description": "Unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", + "description": "volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore", "type": "string" } }, @@ -4199,27 +4827,27 @@ "description": "AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod.", "properties": { "cachingMode": { - "description": "Host Caching mode: None, Read Only, Read Write.", + "description": "cachingMode is the Host Caching mode: None, Read Only, Read Write.", "type": "string" }, "diskName": { - "description": "The Name of the data disk in the blob storage", + "description": "diskName is the Name of the data disk in the blob storage", "type": "string" }, "diskURI": { - "description": "The URI the data disk in the blob storage", + "description": "diskURI is the URI of data disk in the blob storage", "type": "string" }, "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "kind": { - "description": "Expected values Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared", + "description": "kind expected values are Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" } }, @@ -4233,19 +4861,19 @@ "description": "AzureFile represents an Azure File Service mount on the host and bind mount to the pod.", "properties": { "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretName": { - "description": "the name of secret that contains Azure Storage Account Name and Key", + "description": "secretName is the name of secret that contains Azure Storage Account Name and Key", "type": "string" }, "secretNamespace": { - "description": "the namespace of the secret that contains Azure Storage Account Name and Key default is the same as the Pod", + "description": "secretNamespace is the namespace of the secret that contains Azure Storage Account Name and Key default is the same as the Pod", "type": "string" }, "shareName": { - "description": "Share Name", + "description": "shareName is the azure Share Name", "type": "string" } }, @@ -4259,15 +4887,15 @@ "description": "AzureFile represents an Azure File Service mount on the host and bind mount to the pod.", "properties": { "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretName": { - "description": "the name of secret that contains Azure Storage Account Name and Key", + "description": "secretName is the name of secret that contains Azure Storage Account Name and Key", "type": "string" }, "shareName": { - "description": "Share Name", + "description": "shareName is the azure share Name", "type": "string" } }, @@ -4314,41 +4942,41 @@ "properties": { "controllerExpandSecretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "ControllerExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerExpandVolume call. This is an alpha field and requires enabling ExpandCSIVolumes feature gate. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." + "description": "controllerExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerExpandVolume call. This is an alpha field and requires enabling ExpandCSIVolumes feature gate. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." }, "controllerPublishSecretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "ControllerPublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerPublishVolume and ControllerUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." + "description": "controllerPublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerPublishVolume and ControllerUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." }, "driver": { - "description": "Driver is the name of the driver to use for this volume. Required.", + "description": "driver is the name of the driver to use for this volume. Required.", "type": "string" }, "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\".", + "description": "fsType to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\".", "type": "string" }, "nodePublishSecretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "NodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." + "description": "nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." }, "nodeStageSecretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "NodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolume and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." + "description": "nodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolume and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed." }, "readOnly": { - "description": "Optional: The value to pass to ControllerPublishVolumeRequest. Defaults to false (read/write).", + "description": "readOnly value to pass to ControllerPublishVolumeRequest. Defaults to false (read/write).", "type": "boolean" }, "volumeAttributes": { "additionalProperties": { "type": "string" }, - "description": "Attributes of the volume to publish.", + "description": "volumeAttributes of the volume to publish.", "type": "object" }, "volumeHandle": { - "description": "VolumeHandle is the unique volume name returned by the CSI volume plugin’s CreateVolume to refer to the volume on all subsequent calls. Required.", + "description": "volumeHandle is the unique volume name returned by the CSI volume plugin’s CreateVolume to refer to the volume on all subsequent calls. Required.", "type": "string" } }, @@ -4362,26 +4990,26 @@ "description": "Represents a source location of a volume to mount, managed by an external CSI driver", "properties": { "driver": { - "description": "Driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster.", + "description": "driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster.", "type": "string" }, "fsType": { - "description": "Filesystem type to mount. Ex. \"ext4\", \"xfs\", \"ntfs\". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply.", + "description": "fsType to mount. Ex. \"ext4\", \"xfs\", \"ntfs\". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply.", "type": "string" }, "nodePublishSecretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "NodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed." + "description": "nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed." }, "readOnly": { - "description": "Specifies a read-only configuration for the volume. Defaults to false (read/write).", + "description": "readOnly specifies a read-only configuration for the volume. Defaults to false (read/write).", "type": "boolean" }, "volumeAttributes": { "additionalProperties": { "type": "string" }, - "description": "VolumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values.", + "description": "volumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values.", "type": "object" } }, @@ -4414,30 +5042,30 @@ "description": "Represents a Ceph Filesystem mount that lasts the lifetime of a pod Cephfs volumes do not support ownership management or SELinux relabeling.", "properties": { "monitors": { - "description": "Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "items": { "type": "string" }, "type": "array" }, "path": { - "description": "Optional: Used as the mounted root, rather than the full Ceph tree, default is /", + "description": "path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /", "type": "string" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "boolean" }, "secretFile": { - "description": "Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "string" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it" + "description": "secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it" }, "user": { - "description": "Optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "user is Optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "string" } }, @@ -4450,30 +5078,30 @@ "description": "Represents a Ceph Filesystem mount that lasts the lifetime of a pod Cephfs volumes do not support ownership management or SELinux relabeling.", "properties": { "monitors": { - "description": "Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "items": { "type": "string" }, "type": "array" }, "path": { - "description": "Optional: Used as the mounted root, rather than the full Ceph tree, default is /", + "description": "path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /", "type": "string" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "boolean" }, "secretFile": { - "description": "Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "string" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it" + "description": "secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it" }, "user": { - "description": "Optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", + "description": "user is optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it", "type": "string" } }, @@ -4486,19 +5114,19 @@ "description": "Represents a cinder volume resource in Openstack. A Cinder volume must exist before mounting to a container. The volume must also be in the same region as the kubelet. Cinder volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "fsType Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "string" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "Optional: points to a secret object containing parameters used to connect to OpenStack." + "description": "secretRef is Optional: points to a secret object containing parameters used to connect to OpenStack." }, "volumeID": { - "description": "volume id used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "string" } }, @@ -4511,19 +5139,19 @@ "description": "Represents a cinder volume resource in Openstack. A Cinder volume must exist before mounting to a container. The volume must also be in the same region as the kubelet. Cinder volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "string" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "Optional: points to a secret object containing parameters used to connect to OpenStack." + "description": "secretRef is optional: points to a secret object containing parameters used to connect to OpenStack." }, "volumeID": { - "description": "volume id used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", + "description": "volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md", "type": "string" } }, @@ -4788,7 +5416,7 @@ "description": "Adapts a ConfigMap into a projected volume.\n\nThe contents of the target ConfigMap's Data field will be presented in a projected volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. Note that this is identical to a configmap volume source without the default mode.", "properties": { "items": { - "description": "If unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", + "description": "items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.KeyToPath" }, @@ -4799,7 +5427,7 @@ "type": "string" }, "optional": { - "description": "Specify whether the ConfigMap or its keys must be defined", + "description": "optional specify whether the ConfigMap or its keys must be defined", "type": "boolean" } }, @@ -4809,12 +5437,12 @@ "description": "Adapts a ConfigMap into a volume.\n\nThe contents of the target ConfigMap's Data field will be presented in a volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. ConfigMap volumes support ownership management and SELinux relabeling.", "properties": { "defaultMode": { - "description": "Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", + "description": "defaultMode is optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", "format": "int32", "type": "integer" }, "items": { - "description": "If unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", + "description": "items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.KeyToPath" }, @@ -4825,7 +5453,7 @@ "type": "string" }, "optional": { - "description": "Specify whether the ConfigMap or its keys must be defined", + "description": "optional specify whether the ConfigMap or its keys must be defined", "type": "boolean" } }, @@ -4835,14 +5463,14 @@ "description": "A single application container that you want to run within a pod.", "properties": { "args": { - "description": "Arguments to the entrypoint. The docker image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", + "description": "Arguments to the entrypoint. The container image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", "items": { "type": "string" }, "type": "array" }, "command": { - "description": "Entrypoint array. Not executed within a shell. The docker image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", + "description": "Entrypoint array. Not executed within a shell. The container image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", "items": { "type": "string" }, @@ -4865,11 +5493,11 @@ "type": "array" }, "image": { - "description": "Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets.", + "description": "Container image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets.", "type": "string" }, "imagePullPolicy": { - "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images", + "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n\n", "type": "string" }, "lifecycle": { @@ -4927,7 +5555,7 @@ "type": "string" }, "terminationMessagePolicy": { - "description": "Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.", + "description": "Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.\n\n", "type": "string" }, "tty": { @@ -5002,7 +5630,7 @@ "type": "string" }, "protocol": { - "description": "Protocol for port. Must be UDP, TCP, or SCTP. Defaults to \"TCP\".", + "description": "Protocol for port. Must be UDP, TCP, or SCTP. Defaults to \"TCP\".\n\n", "type": "string" } }, @@ -5043,7 +5671,7 @@ "description": "ContainerStateTerminated is a terminated state of a container.", "properties": { "containerID": { - "description": "Container's ID in the format 'docker://'", + "description": "Container's ID in the format '://'", "type": "string" }, "exitCode": { @@ -5096,11 +5724,11 @@ "description": "ContainerStatus contains details for the current status of this container.", "properties": { "containerID": { - "description": "Container's ID in the format 'docker://'.", + "description": "Container's ID in the format '://'.", "type": "string" }, "image": { - "description": "The image the container is running. More info: https://kubernetes.io/docs/concepts/containers/images", + "description": "The image the container is running. More info: https://kubernetes.io/docs/concepts/containers/images.", "type": "string" }, "imageID": { @@ -5120,7 +5748,7 @@ "type": "boolean" }, "restartCount": { - "description": "The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. This value will get capped at 5 by GC.", + "description": "The number of times the container has been restarted.", "format": "int32", "type": "integer" }, @@ -5217,12 +5845,12 @@ "description": "Represents an empty directory for a pod. Empty directory volumes support ownership management and SELinux relabeling.", "properties": { "medium": { - "description": "What type of storage medium should back this directory. The default is \"\" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir", + "description": "medium represents what type of storage medium should back this directory. The default is \"\" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir", "type": "string" }, "sizeLimit": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", - "description": "Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir" + "description": "sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir" } }, "type": "object" @@ -5257,7 +5885,7 @@ "description": "EndpointPort is a tuple that describes a single port.", "properties": { "appProtocol": { - "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", + "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", "type": "string" }, "name": { @@ -5270,7 +5898,7 @@ "type": "integer" }, "protocol": { - "description": "The IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.", + "description": "The IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.\n\n", "type": "string" } }, @@ -5436,17 +6064,17 @@ "type": "object" }, "io.k8s.api.core.v1.EphemeralContainer": { - "description": "An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.", + "description": "An EphemeralContainer is a temporary container that you may add to an existing Pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a Pod is removed or restarted. The kubelet may evict a Pod if an ephemeral container causes the Pod to exceed its resource allocation.\n\nTo add an ephemeral container, use the ephemeralcontainers subresource of an existing Pod. Ephemeral containers may not be removed or restarted.\n\nThis is a beta feature available on clusters that haven't disabled the EphemeralContainers feature gate.", "properties": { "args": { - "description": "Arguments to the entrypoint. The docker image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", + "description": "Arguments to the entrypoint. The image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", "items": { "type": "string" }, "type": "array" }, "command": { - "description": "Entrypoint array. Not executed within a shell. The docker image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", + "description": "Entrypoint array. Not executed within a shell. The image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell", "items": { "type": "string" }, @@ -5469,11 +6097,11 @@ "type": "array" }, "image": { - "description": "Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images", + "description": "Container image name. More info: https://kubernetes.io/docs/concepts/containers/images", "type": "string" }, "imagePullPolicy": { - "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images", + "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n\n", "type": "string" }, "lifecycle": { @@ -5493,7 +6121,14 @@ "items": { "$ref": "#/definitions/io.k8s.api.core.v1.ContainerPort" }, - "type": "array" + "type": "array", + "x-kubernetes-list-map-keys": [ + "containerPort", + "protocol" + ], + "x-kubernetes-list-type": "map", + "x-kubernetes-patch-merge-key": "containerPort", + "x-kubernetes-patch-strategy": "merge" }, "readinessProbe": { "$ref": "#/definitions/io.k8s.api.core.v1.Probe", @@ -5520,7 +6155,7 @@ "type": "boolean" }, "targetContainerName": { - "description": "If set, the name of the container from PodSpec that this ephemeral container targets. The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container. If not set then the ephemeral container is run in whatever namespaces are shared for the pod. Note that the container runtime must support this feature.", + "description": "If set, the name of the container from PodSpec that this ephemeral container targets. The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container. If not set then the ephemeral container uses the namespaces configured in the Pod spec.\n\nThe container runtime must implement support for this feature. If the runtime does not support namespace targeting then the result of setting this field is undefined.", "type": "string" }, "terminationMessagePath": { @@ -5528,7 +6163,7 @@ "type": "string" }, "terminationMessagePolicy": { - "description": "Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.", + "description": "Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.\n\n", "type": "string" }, "tty": { @@ -5545,7 +6180,7 @@ "x-kubernetes-patch-strategy": "merge" }, "volumeMounts": { - "description": "Pod volumes to mount into the container's filesystem. Cannot be updated.", + "description": "Pod volumes to mount into the container's filesystem. Subpath mounts are not allowed for ephemeral containers. Cannot be updated.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.VolumeMount" }, @@ -5740,27 +6375,27 @@ "description": "Represents a Fibre Channel volume. Fibre Channel volumes can only be mounted as read/write once. Fibre Channel volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "lun": { - "description": "Optional: FC target lun number", + "description": "lun is Optional: FC target lun number", "format": "int32", "type": "integer" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "targetWWNs": { - "description": "Optional: FC target worldwide names (WWNs)", + "description": "targetWWNs is Optional: FC target worldwide names (WWNs)", "items": { "type": "string" }, "type": "array" }, "wwids": { - "description": "Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously.", + "description": "wwids Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously.", "items": { "type": "string" }, @@ -5773,27 +6408,27 @@ "description": "FlexPersistentVolumeSource represents a generic persistent volume resource that is provisioned/attached using an exec based plugin.", "properties": { "driver": { - "description": "Driver is the name of the driver to use for this volume.", + "description": "driver is the name of the driver to use for this volume.", "type": "string" }, "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script.", + "description": "fsType is the Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script.", "type": "string" }, "options": { "additionalProperties": { "type": "string" }, - "description": "Optional: Extra command options if any.", + "description": "options is Optional: this field holds extra command options if any.", "type": "object" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts." + "description": "secretRef is Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts." } }, "required": [ @@ -5805,27 +6440,27 @@ "description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.", "properties": { "driver": { - "description": "Driver is the name of the driver to use for this volume.", + "description": "driver is the name of the driver to use for this volume.", "type": "string" }, "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script.", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script.", "type": "string" }, "options": { "additionalProperties": { "type": "string" }, - "description": "Optional: Extra command options if any.", + "description": "options is Optional: this field holds extra command options if any.", "type": "object" }, "readOnly": { - "description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts." + "description": "secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts." } }, "required": [ @@ -5837,11 +6472,11 @@ "description": "Represents a Flocker volume mounted by the Flocker agent. One and only one of datasetName and datasetUUID should be set. Flocker volumes do not support ownership management or SELinux relabeling.", "properties": { "datasetName": { - "description": "Name of the dataset stored as metadata -> name on the dataset for Flocker should be considered as deprecated", + "description": "datasetName is Name of the dataset stored as metadata -> name on the dataset for Flocker should be considered as deprecated", "type": "string" }, "datasetUUID": { - "description": "UUID of the dataset. This is unique identifier of a Flocker dataset", + "description": "datasetUUID is the UUID of the dataset. This is unique identifier of a Flocker dataset", "type": "string" } }, @@ -5851,20 +6486,20 @@ "description": "Represents a Persistent Disk resource in Google Compute Engine.\n\nA GCE PD must exist before mounting to a container. The disk must also be in the same GCE project and zone as the kubelet. A GCE PD can only be mounted as read/write once or read-only many times. GCE PDs support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", + "description": "fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", "type": "string" }, "partition": { - "description": "The partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as \"1\". Similarly, the volume partition for /dev/sda is \"0\" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", + "description": "partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as \"1\". Similarly, the volume partition for /dev/sda is \"0\" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", "format": "int32", "type": "integer" }, "pdName": { - "description": "Unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", + "description": "pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", + "description": "readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk", "type": "boolean" } }, @@ -5873,19 +6508,36 @@ ], "type": "object" }, + "io.k8s.api.core.v1.GRPCAction": { + "properties": { + "port": { + "description": "Port number of the gRPC service. Number must be in the range 1 to 65535.", + "format": "int32", + "type": "integer" + }, + "service": { + "description": "Service is the name of the service to place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\nIf this is not specified, the default behavior is defined by gRPC.", + "type": "string" + } + }, + "required": [ + "port" + ], + "type": "object" + }, "io.k8s.api.core.v1.GitRepoVolumeSource": { "description": "Represents a volume that is populated with the contents of a git repository. Git repo volumes do not support ownership management. Git repo volumes support SELinux relabeling.\n\nDEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container.", "properties": { "directory": { - "description": "Target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name.", + "description": "directory is the target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name.", "type": "string" }, "repository": { - "description": "Repository URL", + "description": "repository is the URL", "type": "string" }, "revision": { - "description": "Commit hash for the specified revision.", + "description": "revision is the commit hash for the specified revision.", "type": "string" } }, @@ -5898,19 +6550,19 @@ "description": "Represents a Glusterfs mount that lasts the lifetime of a pod. Glusterfs volumes do not support ownership management or SELinux relabeling.", "properties": { "endpoints": { - "description": "EndpointsName is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "endpoints is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "string" }, "endpointsNamespace": { - "description": "EndpointsNamespace is the namespace that contains Glusterfs endpoint. If this field is empty, the EndpointNamespace defaults to the same namespace as the bound PVC. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "endpointsNamespace is the namespace that contains Glusterfs endpoint. If this field is empty, the EndpointNamespace defaults to the same namespace as the bound PVC. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "string" }, "path": { - "description": "Path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "boolean" } }, @@ -5924,15 +6576,15 @@ "description": "Represents a Glusterfs mount that lasts the lifetime of a pod. Glusterfs volumes do not support ownership management or SELinux relabeling.", "properties": { "endpoints": { - "description": "EndpointsName is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "endpoints is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "string" }, "path": { - "description": "Path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", + "description": "readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod", "type": "boolean" } }, @@ -5965,7 +6617,7 @@ "description": "Name or number of the port to access on the container. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME." }, "scheme": { - "description": "Scheme to use for connecting to the host. Defaults to HTTP.", + "description": "Scheme to use for connecting to the host. Defaults to HTTP.\n\n", "type": "string" } }, @@ -5992,24 +6644,6 @@ ], "type": "object" }, - "io.k8s.api.core.v1.Handler": { - "description": "Handler defines a specific action that should be taken", - "properties": { - "exec": { - "$ref": "#/definitions/io.k8s.api.core.v1.ExecAction", - "description": "One and only one of the following should be specified. Exec specifies the action to take." - }, - "httpGet": { - "$ref": "#/definitions/io.k8s.api.core.v1.HTTPGetAction", - "description": "HTTPGet specifies the http request to perform." - }, - "tcpSocket": { - "$ref": "#/definitions/io.k8s.api.core.v1.TCPSocketAction", - "description": "TCPSocket specifies an action involving a TCP port. TCP hooks not yet supported" - } - }, - "type": "object" - }, "io.k8s.api.core.v1.HostAlias": { "description": "HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.", "properties": { @@ -6031,11 +6665,11 @@ "description": "Represents a host path mapped into a pod. Host path volumes do not support ownership management or SELinux relabeling.", "properties": { "path": { - "description": "Path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath", + "description": "path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath", "type": "string" }, "type": { - "description": "Type for HostPath Volume Defaults to \"\" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath", + "description": "type for HostPath Volume Defaults to \"\" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath", "type": "string" } }, @@ -6048,51 +6682,51 @@ "description": "ISCSIPersistentVolumeSource represents an ISCSI disk. ISCSI volumes can only be mounted as read/write once. ISCSI volumes support ownership management and SELinux relabeling.", "properties": { "chapAuthDiscovery": { - "description": "whether support iSCSI Discovery CHAP authentication", + "description": "chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication", "type": "boolean" }, "chapAuthSession": { - "description": "whether support iSCSI Session CHAP authentication", + "description": "chapAuthSession defines whether support iSCSI Session CHAP authentication", "type": "boolean" }, "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi", + "description": "fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi", "type": "string" }, "initiatorName": { - "description": "Custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection.", + "description": "initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection.", "type": "string" }, "iqn": { - "description": "Target iSCSI Qualified Name.", + "description": "iqn is Target iSCSI Qualified Name.", "type": "string" }, "iscsiInterface": { - "description": "iSCSI Interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).", + "description": "iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).", "type": "string" }, "lun": { - "description": "iSCSI Target Lun number.", + "description": "lun is iSCSI Target Lun number.", "format": "int32", "type": "integer" }, "portals": { - "description": "iSCSI Target Portal List. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", + "description": "portals is the iSCSI Target Portal List. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", "items": { "type": "string" }, "type": "array" }, "readOnly": { - "description": "ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.", + "description": "readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "CHAP Secret for iSCSI target and initiator authentication" + "description": "secretRef is the CHAP Secret for iSCSI target and initiator authentication" }, "targetPortal": { - "description": "iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", + "description": "targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", "type": "string" } }, @@ -6107,51 +6741,51 @@ "description": "Represents an ISCSI disk. ISCSI volumes can only be mounted as read/write once. ISCSI volumes support ownership management and SELinux relabeling.", "properties": { "chapAuthDiscovery": { - "description": "whether support iSCSI Discovery CHAP authentication", + "description": "chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication", "type": "boolean" }, "chapAuthSession": { - "description": "whether support iSCSI Session CHAP authentication", + "description": "chapAuthSession defines whether support iSCSI Session CHAP authentication", "type": "boolean" }, "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi", + "description": "fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi", "type": "string" }, "initiatorName": { - "description": "Custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection.", + "description": "initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection.", "type": "string" }, "iqn": { - "description": "Target iSCSI Qualified Name.", + "description": "iqn is the target iSCSI Qualified Name.", "type": "string" }, "iscsiInterface": { - "description": "iSCSI Interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).", + "description": "iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).", "type": "string" }, "lun": { - "description": "iSCSI Target Lun number.", + "description": "lun represents iSCSI Target Lun number.", "format": "int32", "type": "integer" }, "portals": { - "description": "iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", + "description": "portals is the iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", "items": { "type": "string" }, "type": "array" }, "readOnly": { - "description": "ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.", + "description": "readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "CHAP Secret for iSCSI target and initiator authentication" + "description": "secretRef is the CHAP Secret for iSCSI target and initiator authentication" }, "targetPortal": { - "description": "iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", + "description": "targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).", "type": "string" } }, @@ -6166,16 +6800,16 @@ "description": "Maps a string key to a path within a volume.", "properties": { "key": { - "description": "The key to project.", + "description": "key is the key to project.", "type": "string" }, "mode": { - "description": "Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", + "description": "mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", "format": "int32", "type": "integer" }, "path": { - "description": "The relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'.", + "description": "path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'.", "type": "string" } }, @@ -6189,12 +6823,30 @@ "description": "Lifecycle describes actions that the management system should take in response to container lifecycle events. For the PostStart and PreStop lifecycle handlers, management of the container blocks until the action is complete, unless the container process fails, in which case the handler is aborted.", "properties": { "postStart": { - "$ref": "#/definitions/io.k8s.api.core.v1.Handler", + "$ref": "#/definitions/io.k8s.api.core.v1.LifecycleHandler", "description": "PostStart is called immediately after a container is created. If the handler fails, the container is terminated and restarted according to its restart policy. Other management of the container blocks until the hook completes. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" }, "preStop": { - "$ref": "#/definitions/io.k8s.api.core.v1.Handler", - "description": "PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The reason for termination is passed to the handler. The Pod's termination grace period countdown begins before the PreStop hooked is executed. Regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period. Other management of the container blocks until the hook completes or until the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" + "$ref": "#/definitions/io.k8s.api.core.v1.LifecycleHandler", + "description": "PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The Pod's termination grace period countdown begins before the PreStop hook is executed. Regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period (unless delayed by finalizers). Other management of the container blocks until the hook completes or until the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" + } + }, + "type": "object" + }, + "io.k8s.api.core.v1.LifecycleHandler": { + "description": "LifecycleHandler defines a specific action that should be taken in a lifecycle hook. One and only one of the fields, except TCPSocket must be specified.", + "properties": { + "exec": { + "$ref": "#/definitions/io.k8s.api.core.v1.ExecAction", + "description": "Exec specifies the action to take." + }, + "httpGet": { + "$ref": "#/definitions/io.k8s.api.core.v1.HTTPGetAction", + "description": "HTTPGet specifies the http request to perform." + }, + "tcpSocket": { + "$ref": "#/definitions/io.k8s.api.core.v1.TCPSocketAction", + "description": "Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept for the backward compatibility. There are no validation of this field and lifecycle hooks will fail in runtime when tcp handler is specified." } }, "type": "object" @@ -6377,11 +7029,11 @@ "description": "Local represents directly-attached storage with node affinity (Beta feature)", "properties": { "fsType": { - "description": "Filesystem type to mount. It applies only when the Path is a block device. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default value is to auto-select a fileystem if unspecified.", + "description": "fsType is the filesystem type to mount. It applies only when the Path is a block device. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default value is to auto-select a filesystem if unspecified.", "type": "string" }, "path": { - "description": "The full path to the volume on the node. It can be either a directory or block device (disk, partition, ...).", + "description": "path of the full path to the volume on the node. It can be either a directory or block device (disk, partition, ...).", "type": "string" } }, @@ -6394,15 +7046,15 @@ "description": "Represents an NFS mount that lasts the lifetime of a pod. NFS volumes do not support ownership management or SELinux relabeling.", "properties": { "path": { - "description": "Path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", + "description": "path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", + "description": "readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", "type": "boolean" }, "server": { - "description": "Server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", + "description": "server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs", "type": "string" } }, @@ -6533,7 +7185,7 @@ "x-kubernetes-patch-strategy": "merge" }, "phase": { - "description": "Phase is the current lifecycle phase of the namespace. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/", + "description": "Phase is the current lifecycle phase of the namespace. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/\n\n", "type": "string" } }, @@ -6743,7 +7395,7 @@ "type": "string" }, "operator": { - "description": "Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.", + "description": "Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.\n\n", "type": "string" }, "values": { @@ -6786,7 +7438,7 @@ "properties": { "configSource": { "$ref": "#/definitions/io.k8s.api.core.v1.NodeConfigSource", - "description": "Deprecated. If specified, the source of the node's configuration. The DynamicKubeletConfig feature gate must be enabled for the Kubelet to use this field. This field is deprecated as of 1.22: https://git.k8s.io/enhancements/keps/sig-node/281-dynamic-kubelet-configuration" + "description": "Deprecated: Previously used to specify the source of the node's configuration for the DynamicKubeletConfig feature. This feature is removed from Kubelets as of 1.24 and will be fully removed in 1.26." }, "externalID": { "description": "Deprecated. Not all kubelets will set this field. Remove field after 1.13. see: https://issues.k8s.io/61966", @@ -6877,7 +7529,7 @@ "description": "Set of ids/uuids to uniquely identify the node. More info: https://kubernetes.io/docs/concepts/nodes/node/#info" }, "phase": { - "description": "NodePhase is the recently observed lifecycle phase of the node. More info: https://kubernetes.io/docs/concepts/nodes/node/#phase The field is never populated, and now is deprecated.", + "description": "NodePhase is the recently observed lifecycle phase of the node. More info: https://kubernetes.io/docs/concepts/nodes/node/#phase The field is never populated, and now is deprecated.\n\n", "type": "string" }, "volumesAttached": { @@ -6909,7 +7561,7 @@ "type": "string" }, "containerRuntimeVersion": { - "description": "ContainerRuntime Version reported by the node through runtime remote API (e.g. docker://1.5.0).", + "description": "ContainerRuntime Version reported by the node through runtime remote API (e.g. containerd://1.4.2).", "type": "string" }, "kernelVersion": { @@ -7025,11 +7677,11 @@ }, "spec": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeSpec", - "description": "Spec defines a specification of a persistent volume owned by the cluster. Provisioned by an administrator. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes" + "description": "spec defines a specification of a persistent volume owned by the cluster. Provisioned by an administrator. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes" }, "status": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeStatus", - "description": "Status represents the current information/status for the persistent volume. Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes" + "description": "status represents the current information/status for the persistent volume. Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes" } }, "type": "object", @@ -7058,11 +7710,11 @@ }, "spec": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeClaimSpec", - "description": "Spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" + "description": "spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" }, "status": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeClaimStatus", - "description": "Status represents the current information/status of a persistent volume claim. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" + "description": "status represents the current information/status of a persistent volume claim. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" } }, "type": "object", @@ -7079,18 +7731,18 @@ "properties": { "lastProbeTime": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", - "description": "Last time we probed the condition." + "description": "lastProbeTime is the time we probed the condition." }, "lastTransitionTime": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", - "description": "Last time the condition transitioned from one status to another." + "description": "lastTransitionTime is the time the condition transitioned from one status to another." }, "message": { - "description": "Human-readable message indicating details about last transition.", + "description": "message is the human-readable message indicating details about last transition.", "type": "string" }, "reason": { - "description": "Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports \"ResizeStarted\" that means the underlying persistent volume is being resized.", + "description": "reason is a unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports \"ResizeStarted\" that means the underlying persistent volume is being resized.", "type": "string" }, "status": { @@ -7114,7 +7766,7 @@ "type": "string" }, "items": { - "description": "A list of persistent volume claims. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims", + "description": "items is a list of persistent volume claims. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeClaim" }, @@ -7145,7 +7797,7 @@ "description": "PersistentVolumeClaimSpec describes the common attributes of storage devices and allows a Source for provider-specific attributes", "properties": { "accessModes": { - "description": "AccessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1", + "description": "accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1", "items": { "type": "string" }, @@ -7153,22 +7805,22 @@ }, "dataSource": { "$ref": "#/definitions/io.k8s.api.core.v1.TypedLocalObjectReference", - "description": "This field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. If the AnyVolumeDataSource feature gate is enabled, this field will always have the same contents as the DataSourceRef field." + "description": "dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. If the AnyVolumeDataSource feature gate is enabled, this field will always have the same contents as the DataSourceRef field." }, "dataSourceRef": { "$ref": "#/definitions/io.k8s.api.core.v1.TypedLocalObjectReference", - "description": "Specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any local object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the DataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, both fields (DataSource and DataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. There are two important differences between DataSource and DataSourceRef: * While DataSource only allows two specific types of objects, DataSourceRef\n allows any non-core object, as well as PersistentVolumeClaim objects.\n* While DataSource ignores disallowed values (dropping them), DataSourceRef\n preserves all values, and generates an error if a disallowed value is\n specified.\n(Alpha) Using this field requires the AnyVolumeDataSource feature gate to be enabled." + "description": "dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any local object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the DataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, both fields (DataSource and DataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. There are two important differences between DataSource and DataSourceRef: * While DataSource only allows two specific types of objects, DataSourceRef\n allows any non-core object, as well as PersistentVolumeClaim objects.\n* While DataSource ignores disallowed values (dropping them), DataSourceRef\n preserves all values, and generates an error if a disallowed value is\n specified.\n(Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled." }, "resources": { "$ref": "#/definitions/io.k8s.api.core.v1.ResourceRequirements", - "description": "Resources represents the minimum resources the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources" + "description": "resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources" }, "selector": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", - "description": "A label query over volumes to consider for binding." + "description": "selector is a label query over volumes to consider for binding." }, "storageClassName": { - "description": "Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1", + "description": "storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1", "type": "string" }, "volumeMode": { @@ -7176,7 +7828,7 @@ "type": "string" }, "volumeName": { - "description": "VolumeName is the binding reference to the PersistentVolume backing this claim.", + "description": "volumeName is the binding reference to the PersistentVolume backing this claim.", "type": "string" } }, @@ -7186,21 +7838,28 @@ "description": "PersistentVolumeClaimStatus is the current status of a persistent volume claim.", "properties": { "accessModes": { - "description": "AccessModes contains the actual access modes the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1", + "description": "accessModes contains the actual access modes the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1", "items": { "type": "string" }, "type": "array" }, + "allocatedResources": { + "additionalProperties": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity" + }, + "description": "allocatedResources is the storage resource within AllocatedResources tracks the capacity allocated to a PVC. It may be larger than the actual capacity when a volume expansion operation is requested. For storage quota, the larger value from allocatedResources and PVC.spec.resources is used. If allocatedResources is not set, PVC.spec.resources alone is used for quota calculation. If a volume expansion capacity request is lowered, allocatedResources is only lowered if there are no expansion operations in progress and if the actual volume capacity is equal or lower than the requested capacity. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.", + "type": "object" + }, "capacity": { "additionalProperties": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity" }, - "description": "Represents the actual resources of the underlying volume.", + "description": "capacity represents the actual resources of the underlying volume.", "type": "object" }, "conditions": { - "description": "Current Condition of persistent volume claim. If underlying persistent volume is being resized then the Condition will be set to 'ResizeStarted'.", + "description": "conditions is the current Condition of persistent volume claim. If underlying persistent volume is being resized then the Condition will be set to 'ResizeStarted'.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeClaimCondition" }, @@ -7209,7 +7868,11 @@ "x-kubernetes-patch-strategy": "merge" }, "phase": { - "description": "Phase represents the current phase of PersistentVolumeClaim.", + "description": "phase represents the current phase of PersistentVolumeClaim.\n\n", + "type": "string" + }, + "resizeStatus": { + "description": "resizeStatus stores status of resize operation. ResizeStatus is not set by default but when expansion is complete resizeStatus is set to empty string by resize controller or kubelet. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.", "type": "string" } }, @@ -7236,11 +7899,11 @@ "description": "PersistentVolumeClaimVolumeSource references the user's PVC in the same namespace. This volume finds the bound PV and mounts that volume for the pod. A PersistentVolumeClaimVolumeSource is, essentially, a wrapper around another type of volume that is owned by someone else (the system).", "properties": { "claimName": { - "description": "ClaimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims", + "description": "claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims", "type": "string" }, "readOnly": { - "description": "Will force the ReadOnly setting in VolumeMounts. Default false.", + "description": "readOnly Will force the ReadOnly setting in VolumeMounts. Default false.", "type": "boolean" } }, @@ -7257,7 +7920,7 @@ "type": "string" }, "items": { - "description": "List of persistent volumes. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes", + "description": "items is a list of persistent volumes. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolume" }, @@ -7288,7 +7951,7 @@ "description": "PersistentVolumeSpec is the specification of a persistent volume.", "properties": { "accessModes": { - "description": "AccessModes contains all ways the volume can be mounted. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes", + "description": "accessModes contains all ways the volume can be mounted. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes", "items": { "type": "string" }, @@ -7296,73 +7959,73 @@ }, "awsElasticBlockStore": { "$ref": "#/definitions/io.k8s.api.core.v1.AWSElasticBlockStoreVolumeSource", - "description": "AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore" + "description": "awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore" }, "azureDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.AzureDiskVolumeSource", - "description": "AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod." + "description": "azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod." }, "azureFile": { "$ref": "#/definitions/io.k8s.api.core.v1.AzureFilePersistentVolumeSource", - "description": "AzureFile represents an Azure File Service mount on the host and bind mount to the pod." + "description": "azureFile represents an Azure File Service mount on the host and bind mount to the pod." }, "capacity": { "additionalProperties": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity" }, - "description": "A description of the persistent volume's resources and capacity. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity", + "description": "capacity is the description of the persistent volume's resources and capacity. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity", "type": "object" }, "cephfs": { "$ref": "#/definitions/io.k8s.api.core.v1.CephFSPersistentVolumeSource", - "description": "CephFS represents a Ceph FS mount on the host that shares a pod's lifetime" + "description": "cephFS represents a Ceph FS mount on the host that shares a pod's lifetime" }, "cinder": { "$ref": "#/definitions/io.k8s.api.core.v1.CinderPersistentVolumeSource", - "description": "Cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md" + "description": "cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md" }, "claimRef": { "$ref": "#/definitions/io.k8s.api.core.v1.ObjectReference", - "description": "ClaimRef is part of a bi-directional binding between PersistentVolume and PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName is the authoritative bind between PV and PVC. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding" + "description": "claimRef is part of a bi-directional binding between PersistentVolume and PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName is the authoritative bind between PV and PVC. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding" }, "csi": { "$ref": "#/definitions/io.k8s.api.core.v1.CSIPersistentVolumeSource", - "description": "CSI represents storage that is handled by an external CSI driver (Beta feature)." + "description": "csi represents storage that is handled by an external CSI driver (Beta feature)." }, "fc": { "$ref": "#/definitions/io.k8s.api.core.v1.FCVolumeSource", - "description": "FC represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod." + "description": "fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod." }, "flexVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.FlexPersistentVolumeSource", - "description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin." + "description": "flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin." }, "flocker": { "$ref": "#/definitions/io.k8s.api.core.v1.FlockerVolumeSource", - "description": "Flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running" + "description": "flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running" }, "gcePersistentDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.GCEPersistentDiskVolumeSource", - "description": "GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk" + "description": "gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk" }, "glusterfs": { "$ref": "#/definitions/io.k8s.api.core.v1.GlusterfsPersistentVolumeSource", - "description": "Glusterfs represents a Glusterfs volume that is attached to a host and exposed to the pod. Provisioned by an admin. More info: https://examples.k8s.io/volumes/glusterfs/README.md" + "description": "glusterfs represents a Glusterfs volume that is attached to a host and exposed to the pod. Provisioned by an admin. More info: https://examples.k8s.io/volumes/glusterfs/README.md" }, "hostPath": { "$ref": "#/definitions/io.k8s.api.core.v1.HostPathVolumeSource", - "description": "HostPath represents a directory on the host. Provisioned by a developer or tester. This is useful for single-node development and testing only! On-host storage is not supported in any way and WILL NOT WORK in a multi-node cluster. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath" + "description": "hostPath represents a directory on the host. Provisioned by a developer or tester. This is useful for single-node development and testing only! On-host storage is not supported in any way and WILL NOT WORK in a multi-node cluster. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath" }, "iscsi": { "$ref": "#/definitions/io.k8s.api.core.v1.ISCSIPersistentVolumeSource", - "description": "ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin." + "description": "iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin." }, "local": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalVolumeSource", - "description": "Local represents directly-attached storage with node affinity" + "description": "local represents directly-attached storage with node affinity" }, "mountOptions": { - "description": "A list of mount options, e.g. [\"ro\", \"soft\"]. Not validated - mount will simply fail if one is invalid. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options", + "description": "mountOptions is the list of mount options, e.g. [\"ro\", \"soft\"]. Not validated - mount will simply fail if one is invalid. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options", "items": { "type": "string" }, @@ -7370,43 +8033,43 @@ }, "nfs": { "$ref": "#/definitions/io.k8s.api.core.v1.NFSVolumeSource", - "description": "NFS represents an NFS mount on the host. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs" + "description": "nfs represents an NFS mount on the host. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs" }, "nodeAffinity": { "$ref": "#/definitions/io.k8s.api.core.v1.VolumeNodeAffinity", - "description": "NodeAffinity defines constraints that limit what nodes this volume can be accessed from. This field influences the scheduling of pods that use this volume." + "description": "nodeAffinity defines constraints that limit what nodes this volume can be accessed from. This field influences the scheduling of pods that use this volume." }, "persistentVolumeReclaimPolicy": { - "description": "What happens to a persistent volume when released from its claim. Valid options are Retain (default for manually created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). Recycle must be supported by the volume plugin underlying this PersistentVolume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming", + "description": "persistentVolumeReclaimPolicy defines what happens to a persistent volume when released from its claim. Valid options are Retain (default for manually created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). Recycle must be supported by the volume plugin underlying this PersistentVolume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming\n\n", "type": "string" }, "photonPersistentDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.PhotonPersistentDiskVolumeSource", - "description": "PhotonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine" + "description": "photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine" }, "portworxVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.PortworxVolumeSource", - "description": "PortworxVolume represents a portworx volume attached and mounted on kubelets host machine" + "description": "portworxVolume represents a portworx volume attached and mounted on kubelets host machine" }, "quobyte": { "$ref": "#/definitions/io.k8s.api.core.v1.QuobyteVolumeSource", - "description": "Quobyte represents a Quobyte mount on the host that shares a pod's lifetime" + "description": "quobyte represents a Quobyte mount on the host that shares a pod's lifetime" }, "rbd": { "$ref": "#/definitions/io.k8s.api.core.v1.RBDPersistentVolumeSource", - "description": "RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md" + "description": "rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md" }, "scaleIO": { "$ref": "#/definitions/io.k8s.api.core.v1.ScaleIOPersistentVolumeSource", - "description": "ScaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes." + "description": "scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes." }, "storageClassName": { - "description": "Name of StorageClass to which this persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.", + "description": "storageClassName is the name of StorageClass to which this persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.", "type": "string" }, "storageos": { "$ref": "#/definitions/io.k8s.api.core.v1.StorageOSPersistentVolumeSource", - "description": "StorageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod More info: https://examples.k8s.io/volumes/storageos/README.md" + "description": "storageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod More info: https://examples.k8s.io/volumes/storageos/README.md" }, "volumeMode": { "description": "volumeMode defines if a volume is intended to be used with a formatted filesystem or to remain in raw block state. Value of Filesystem is implied when not included in spec.", @@ -7414,7 +8077,7 @@ }, "vsphereVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.VsphereVirtualDiskVolumeSource", - "description": "VsphereVolume represents a vSphere volume attached and mounted on kubelets host machine" + "description": "vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine" } }, "type": "object" @@ -7423,15 +8086,15 @@ "description": "PersistentVolumeStatus is the current status of a persistent volume.", "properties": { "message": { - "description": "A human-readable message indicating details about why the volume is in this state.", + "description": "message is a human-readable message indicating details about why the volume is in this state.", "type": "string" }, "phase": { - "description": "Phase indicates if a volume is available, bound to a claim, or released by a claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase", + "description": "phase indicates if a volume is available, bound to a claim, or released by a claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase\n\n", "type": "string" }, "reason": { - "description": "Reason is a brief CamelCase string that describes any failure and is meant for machine parsing and tidy display in the CLI.", + "description": "reason is a brief CamelCase string that describes any failure and is meant for machine parsing and tidy display in the CLI.", "type": "string" } }, @@ -7441,11 +8104,11 @@ "description": "Represents a Photon Controller persistent disk resource.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "pdID": { - "description": "ID that identifies Photon Controller persistent disk", + "description": "pdID is the ID that identifies Photon Controller persistent disk", "type": "string" } }, @@ -7516,10 +8179,10 @@ }, "namespaceSelector": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", - "description": "A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means \"this pod's namespace\". An empty selector ({}) matches all namespaces. This field is beta-level and is only honored when PodAffinityNamespaceSelector feature is enabled." + "description": "A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means \"this pod's namespace\". An empty selector ({}) matches all namespaces." }, "namespaces": { - "description": "namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means \"this pod's namespace\"", + "description": "namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means \"this pod's namespace\".", "items": { "type": "string" }, @@ -7674,6 +8337,19 @@ } ] }, + "io.k8s.api.core.v1.PodOS": { + "description": "PodOS defines the OS parameters of a pod.", + "properties": { + "name": { + "description": "Name is the name of the operating system. The currently supported values are linux and windows. Additional value may be defined in future and can be one of: https://github.com/opencontainers/runtime-spec/blob/master/config.md#platform-specific-configuration Clients should expect to handle additional values and treat unrecognized values in this field as os: null", + "type": "string" + } + }, + "required": [ + "name" + ], + "type": "object" + }, "io.k8s.api.core.v1.PodReadinessGate": { "description": "PodReadinessGate contains the reference to a pod condition", "properties": { @@ -7691,16 +8367,16 @@ "description": "PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext. Field values of container.securityContext take precedence over field values of PodSecurityContext.", "properties": { "fsGroup": { - "description": "A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:\n\n1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw----\n\nIf unset, the Kubelet will not modify the ownership and permissions of any volume.", + "description": "A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:\n\n1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw----\n\nIf unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows.", "format": "int64", "type": "integer" }, "fsGroupChangePolicy": { - "description": "fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are \"OnRootMismatch\" and \"Always\". If not specified, \"Always\" is used.", + "description": "fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are \"OnRootMismatch\" and \"Always\". If not specified, \"Always\" is used. Note that this field cannot be set when spec.os.name is windows.", "type": "string" }, "runAsGroup": { - "description": "The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container.", + "description": "The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.", "format": "int64", "type": "integer" }, @@ -7709,20 +8385,20 @@ "type": "boolean" }, "runAsUser": { - "description": "The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container.", + "description": "The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.", "format": "int64", "type": "integer" }, "seLinuxOptions": { "$ref": "#/definitions/io.k8s.api.core.v1.SELinuxOptions", - "description": "The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container." + "description": "The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows." }, "seccompProfile": { "$ref": "#/definitions/io.k8s.api.core.v1.SeccompProfile", - "description": "The seccomp options to use by the containers in this pod." + "description": "The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows." }, "supplementalGroups": { - "description": "A list of groups applied to the first process run in each container, in addition to the container's primary GID. If unspecified, no groups will be added to any container.", + "description": "A list of groups applied to the first process run in each container, in addition to the container's primary GID. If unspecified, no groups will be added to any container. Note that this field cannot be set when spec.os.name is windows.", "items": { "format": "int64", "type": "integer" @@ -7730,7 +8406,7 @@ "type": "array" }, "sysctls": { - "description": "Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch.", + "description": "Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.Sysctl" }, @@ -7738,7 +8414,7 @@ }, "windowsOptions": { "$ref": "#/definitions/io.k8s.api.core.v1.WindowsSecurityContextOptions", - "description": "The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence." + "description": "The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux." } }, "type": "object" @@ -7773,7 +8449,7 @@ "description": "Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy." }, "dnsPolicy": { - "description": "Set DNS policy for the pod. Defaults to \"ClusterFirst\". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'.", + "description": "Set DNS policy for the pod. Defaults to \"ClusterFirst\". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'.\n\n", "type": "string" }, "enableServiceLinks": { @@ -7781,7 +8457,7 @@ "type": "boolean" }, "ephemeralContainers": { - "description": "List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource. This field is alpha-level and is only honored by servers that enable the EphemeralContainers feature.", + "description": "List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource. This field is beta-level and available on clusters that haven't disabled the EphemeralContainers feature gate.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.EphemeralContainer" }, @@ -7815,7 +8491,7 @@ "type": "string" }, "imagePullSecrets": { - "description": "ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod", + "description": "ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference" }, @@ -7844,15 +8520,19 @@ "type": "object", "x-kubernetes-map-type": "atomic" }, + "os": { + "$ref": "#/definitions/io.k8s.api.core.v1.PodOS", + "description": "Specifies the OS of the containers in the pod. Some pod and container fields are restricted if this is set.\n\nIf the OS field is set to linux, the following fields must be unset: -securityContext.windowsOptions\n\nIf the OS field is set to windows, following fields must be unset: - spec.hostPID - spec.hostIPC - spec.securityContext.seLinuxOptions - spec.securityContext.seccompProfile - spec.securityContext.fsGroup - spec.securityContext.fsGroupChangePolicy - spec.securityContext.sysctls - spec.shareProcessNamespace - spec.securityContext.runAsUser - spec.securityContext.runAsGroup - spec.securityContext.supplementalGroups - spec.containers[*].securityContext.seLinuxOptions - spec.containers[*].securityContext.seccompProfile - spec.containers[*].securityContext.capabilities - spec.containers[*].securityContext.readOnlyRootFilesystem - spec.containers[*].securityContext.privileged - spec.containers[*].securityContext.allowPrivilegeEscalation - spec.containers[*].securityContext.procMount - spec.containers[*].securityContext.runAsUser - spec.containers[*].securityContext.runAsGroup This is a beta field and requires the IdentifyPodOS feature" + }, "overhead": { "additionalProperties": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity" }, - "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md This field is beta-level as of Kubernetes v1.18, and is only honored by servers that enable the PodOverhead feature.", + "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md", "type": "object" }, "preemptionPolicy": { - "description": "PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.", + "description": "PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset.", "type": "string" }, "priority": { @@ -7872,11 +8552,11 @@ "type": "array" }, "restartPolicy": { - "description": "Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy", + "description": "Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy\n\n", "type": "string" }, "runtimeClassName": { - "description": "RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the \"legacy\" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class This is a beta feature as of Kubernetes v1.14.", + "description": "RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the \"legacy\" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class", "type": "string" }, "schedulerName": { @@ -7961,14 +8641,14 @@ "x-kubernetes-patch-strategy": "merge" }, "containerStatuses": { - "description": "The list has one entry per container in the manifest. Each entry is currently the output of `docker inspect`. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status", + "description": "The list has one entry per container in the manifest. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.ContainerStatus" }, "type": "array" }, "ephemeralContainerStatuses": { - "description": "Status for any ephemeral containers that have run in this pod. This field is alpha-level and is only populated by servers that enable the EphemeralContainers feature.", + "description": "Status for any ephemeral containers that have run in this pod. This field is beta-level and available on clusters that haven't disabled the EphemeralContainers feature gate.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.ContainerStatus" }, @@ -7994,7 +8674,7 @@ "type": "string" }, "phase": { - "description": "The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The conditions array, the reason and message fields, and the individual container status arrays contain more detail about the pod's status. There are five possible phase values:\n\nPending: The pod has been accepted by the Kubernetes system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. Succeeded: All containers in the pod have terminated in success, and will not be restarted. Failed: All containers in the pod have terminated, and at least one container has terminated in failure. The container either exited with non-zero status or was terminated by the system. Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod.\n\nMore info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase", + "description": "The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The conditions array, the reason and message fields, and the individual container status arrays contain more detail about the pod's status. There are five possible phase values:\n\nPending: The pod has been accepted by the Kubernetes system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. Succeeded: All containers in the pod have terminated in success, and will not be restarted. Failed: All containers in the pod have terminated, and at least one container has terminated in failure. The container either exited with non-zero status or was terminated by the system. Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod.\n\nMore info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase\n\n", "type": "string" }, "podIP": { @@ -8011,7 +8691,7 @@ "x-kubernetes-patch-strategy": "merge" }, "qosClass": { - "description": "The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info: https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md", + "description": "The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info: https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md\n\n", "type": "string" }, "reason": { @@ -8115,7 +8795,7 @@ "type": "integer" }, "protocol": { - "description": "Protocol is the protocol of the service port of which status is recorded here The supported values are: \"TCP\", \"UDP\", \"SCTP\"", + "description": "Protocol is the protocol of the service port of which status is recorded here The supported values are: \"TCP\", \"UDP\", \"SCTP\"\n\n", "type": "string" } }, @@ -8129,15 +8809,15 @@ "description": "PortworxVolumeSource represents a Portworx volume resource.", "properties": { "fsType": { - "description": "FSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "volumeID": { - "description": "VolumeID uniquely identifies a Portworx volume", + "description": "volumeID uniquely identifies a Portworx volume", "type": "string" } }, @@ -8170,13 +8850,17 @@ "properties": { "exec": { "$ref": "#/definitions/io.k8s.api.core.v1.ExecAction", - "description": "One and only one of the following should be specified. Exec specifies the action to take." + "description": "Exec specifies the action to take." }, "failureThreshold": { "description": "Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1.", "format": "int32", "type": "integer" }, + "grpc": { + "$ref": "#/definitions/io.k8s.api.core.v1.GRPCAction", + "description": "GRPC specifies an action involving a GRPC port. This is a beta field and requires enabling GRPCContainerProbe feature gate." + }, "httpGet": { "$ref": "#/definitions/io.k8s.api.core.v1.HTTPGetAction", "description": "HTTPGet specifies the http request to perform." @@ -8198,7 +8882,7 @@ }, "tcpSocket": { "$ref": "#/definitions/io.k8s.api.core.v1.TCPSocketAction", - "description": "TCPSocket specifies an action involving a TCP port. TCP hooks not yet supported" + "description": "TCPSocket specifies an action involving a TCP port." }, "terminationGracePeriodSeconds": { "description": "Optional duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this value overrides the value provided by the pod spec. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate. Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.", @@ -8217,12 +8901,12 @@ "description": "Represents a projected volume source", "properties": { "defaultMode": { - "description": "Mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", + "description": "defaultMode are the mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", "format": "int32", "type": "integer" }, "sources": { - "description": "list of volume projections", + "description": "sources is the list of volume projections", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.VolumeProjection" }, @@ -8235,27 +8919,27 @@ "description": "Represents a Quobyte mount that lasts the lifetime of a pod. Quobyte volumes do not support ownership management or SELinux relabeling.", "properties": { "group": { - "description": "Group to map volume access to Default is no group", + "description": "group to map volume access to Default is no group", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false.", + "description": "readOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false.", "type": "boolean" }, "registry": { - "description": "Registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes", + "description": "registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes", "type": "string" }, "tenant": { - "description": "Tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin", + "description": "tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin", "type": "string" }, "user": { - "description": "User to map volume access to Defaults to serivceaccount user", + "description": "user to map volume access to Defaults to serivceaccount user", "type": "string" }, "volume": { - "description": "Volume is a string that references an already created Quobyte volume by name.", + "description": "volume is a string that references an already created Quobyte volume by name.", "type": "string" } }, @@ -8269,38 +8953,38 @@ "description": "Represents a Rados Block Device mount that lasts the lifetime of a pod. RBD volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd", + "description": "fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd", "type": "string" }, "image": { - "description": "The rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "keyring": { - "description": "Keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "monitors": { - "description": "A collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "items": { "type": "string" }, "type": "array" }, "pool": { - "description": "The rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "SecretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it" + "description": "secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it" }, "user": { - "description": "The rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" } }, @@ -8314,38 +8998,38 @@ "description": "Represents a Rados Block Device mount that lasts the lifetime of a pod. RBD volumes support ownership management and SELinux relabeling.", "properties": { "fsType": { - "description": "Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd", + "description": "fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd", "type": "string" }, "image": { - "description": "The rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "keyring": { - "description": "Keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "monitors": { - "description": "A collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "items": { "type": "string" }, "type": "array" }, "pool": { - "description": "The rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" }, "readOnly": { - "description": "ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "SecretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it" + "description": "secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it" }, "user": { - "description": "The rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", + "description": "user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it", "type": "string" } }, @@ -8704,43 +9388,43 @@ "description": "ScaleIOPersistentVolumeSource represents a persistent ScaleIO volume", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Default is \"xfs\"", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Default is \"xfs\"", "type": "string" }, "gateway": { - "description": "The host address of the ScaleIO API Gateway.", + "description": "gateway is the host address of the ScaleIO API Gateway.", "type": "string" }, "protectionDomain": { - "description": "The name of the ScaleIO Protection Domain for the configured storage.", + "description": "protectionDomain is the name of the ScaleIO Protection Domain for the configured storage.", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretReference", - "description": "SecretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail." + "description": "secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail." }, "sslEnabled": { - "description": "Flag to enable/disable SSL communication with Gateway, default false", + "description": "sslEnabled is the flag to enable/disable SSL communication with Gateway, default false", "type": "boolean" }, "storageMode": { - "description": "Indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.", + "description": "storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.", "type": "string" }, "storagePool": { - "description": "The ScaleIO Storage Pool associated with the protection domain.", + "description": "storagePool is the ScaleIO Storage Pool associated with the protection domain.", "type": "string" }, "system": { - "description": "The name of the storage system as configured in ScaleIO.", + "description": "system is the name of the storage system as configured in ScaleIO.", "type": "string" }, "volumeName": { - "description": "The name of a volume already created in the ScaleIO system that is associated with this volume source.", + "description": "volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source.", "type": "string" } }, @@ -8755,43 +9439,43 @@ "description": "ScaleIOVolumeSource represents a persistent ScaleIO volume", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Default is \"xfs\".", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Default is \"xfs\".", "type": "string" }, "gateway": { - "description": "The host address of the ScaleIO API Gateway.", + "description": "gateway is the host address of the ScaleIO API Gateway.", "type": "string" }, "protectionDomain": { - "description": "The name of the ScaleIO Protection Domain for the configured storage.", + "description": "protectionDomain is the name of the ScaleIO Protection Domain for the configured storage.", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "SecretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail." + "description": "secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail." }, "sslEnabled": { - "description": "Flag to enable/disable SSL communication with Gateway, default false", + "description": "sslEnabled Flag enable/disable SSL communication with Gateway, default false", "type": "boolean" }, "storageMode": { - "description": "Indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.", + "description": "storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.", "type": "string" }, "storagePool": { - "description": "The ScaleIO Storage Pool associated with the protection domain.", + "description": "storagePool is the ScaleIO Storage Pool associated with the protection domain.", "type": "string" }, "system": { - "description": "The name of the storage system as configured in ScaleIO.", + "description": "system is the name of the storage system as configured in ScaleIO.", "type": "string" }, "volumeName": { - "description": "The name of a volume already created in the ScaleIO system that is associated with this volume source.", + "description": "volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source.", "type": "string" } }, @@ -8820,11 +9504,11 @@ "description": "A scoped-resource selector requirement is a selector that contains values, a scope name, and an operator that relates the scope name and values.", "properties": { "operator": { - "description": "Represents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.", + "description": "Represents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.\n\n", "type": "string" }, "scopeName": { - "description": "The name of the scope that the selector applies to.", + "description": "The name of the scope that the selector applies to.\n\n", "type": "string" }, "values": { @@ -8849,7 +9533,7 @@ "type": "string" }, "type": { - "description": "type indicates which kind of seccomp profile will be applied. Valid options are:\n\nLocalhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied.", + "description": "type indicates which kind of seccomp profile will be applied. Valid options are:\n\nLocalhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied.\n\n", "type": "string" } }, @@ -8901,7 +9585,7 @@ "type": "object" }, "type": { - "description": "Used to facilitate programmatic handling of secret data.", + "description": "Used to facilitate programmatic handling of secret data. More info: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types", "type": "string" } }, @@ -8989,7 +9673,7 @@ "description": "Adapts a secret into a projected volume.\n\nThe contents of the target Secret's Data field will be presented in a projected volume as files using the keys in the Data field as the file names. Note that this is identical to a secret volume source without the default mode.", "properties": { "items": { - "description": "If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", + "description": "items if unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.KeyToPath" }, @@ -9000,7 +9684,7 @@ "type": "string" }, "optional": { - "description": "Specify whether the Secret or its key must be defined", + "description": "optional field specify whether the Secret or its key must be defined", "type": "boolean" } }, @@ -9010,11 +9694,11 @@ "description": "SecretReference represents a Secret Reference. It has enough information to retrieve secret in any namespace", "properties": { "name": { - "description": "Name is unique within a namespace to reference a secret resource.", + "description": "name is unique within a namespace to reference a secret resource.", "type": "string" }, "namespace": { - "description": "Namespace defines the space within which the secret name must be unique.", + "description": "namespace defines the space within which the secret name must be unique.", "type": "string" } }, @@ -9025,23 +9709,23 @@ "description": "Adapts a Secret into a volume.\n\nThe contents of the target Secret's Data field will be presented in a volume as files using the keys in the Data field as the file names. Secret volumes support ownership management and SELinux relabeling.", "properties": { "defaultMode": { - "description": "Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", + "description": "defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.", "format": "int32", "type": "integer" }, "items": { - "description": "If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", + "description": "items If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.KeyToPath" }, "type": "array" }, "optional": { - "description": "Specify whether the Secret or its keys must be defined", + "description": "optional field specify whether the Secret or its keys must be defined", "type": "boolean" }, "secretName": { - "description": "Name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret", + "description": "secretName is the name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret", "type": "string" } }, @@ -9051,27 +9735,27 @@ "description": "SecurityContext holds security configuration that will be applied to a container. Some fields are present in both SecurityContext and PodSecurityContext. When both are set, the values in SecurityContext take precedence.", "properties": { "allowPrivilegeEscalation": { - "description": "AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN", + "description": "AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows.", "type": "boolean" }, "capabilities": { "$ref": "#/definitions/io.k8s.api.core.v1.Capabilities", - "description": "The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime." + "description": "The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows." }, "privileged": { - "description": "Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false.", + "description": "Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows.", "type": "boolean" }, "procMount": { - "description": "procMount denotes the type of proc mount to use for the containers. The default is DefaultProcMount which uses the container runtime defaults for readonly paths and masked paths. This requires the ProcMountType feature flag to be enabled.", + "description": "procMount denotes the type of proc mount to use for the containers. The default is DefaultProcMount which uses the container runtime defaults for readonly paths and masked paths. This requires the ProcMountType feature flag to be enabled. Note that this field cannot be set when spec.os.name is windows.", "type": "string" }, "readOnlyRootFilesystem": { - "description": "Whether this container has a read-only root filesystem. Default is false.", + "description": "Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows.", "type": "boolean" }, "runAsGroup": { - "description": "The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.", + "description": "The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.", "format": "int64", "type": "integer" }, @@ -9080,21 +9764,21 @@ "type": "boolean" }, "runAsUser": { - "description": "The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.", + "description": "The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.", "format": "int64", "type": "integer" }, "seLinuxOptions": { "$ref": "#/definitions/io.k8s.api.core.v1.SELinuxOptions", - "description": "The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence." + "description": "The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows." }, "seccompProfile": { "$ref": "#/definitions/io.k8s.api.core.v1.SeccompProfile", - "description": "The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options." + "description": "The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows." }, "windowsOptions": { "$ref": "#/definitions/io.k8s.api.core.v1.WindowsSecurityContextOptions", - "description": "The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence." + "description": "The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux." } }, "type": "object" @@ -9159,7 +9843,7 @@ "description": "Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" }, "secrets": { - "description": "Secrets is the list of secrets allowed to be used by pods running using this ServiceAccount. More info: https://kubernetes.io/docs/concepts/configuration/secret", + "description": "Secrets is a list of the secrets in the same namespace that pods running using this ServiceAccount are allowed to use. Pods are only limited to this list if this service account has a \"kubernetes.io/enforce-mountable-secrets\" annotation set to \"true\". This field should not be used to find auto-generated service account token secrets for use outside of pods. Instead, tokens can be requested directly using the TokenRequest API, or service account token secrets can be manually created. More info: https://kubernetes.io/docs/concepts/configuration/secret", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.ObjectReference" }, @@ -9216,16 +9900,16 @@ "description": "ServiceAccountTokenProjection represents a projected service account token volume. This projection can be used to insert a service account token into the pods runtime filesystem for use against APIs (Kubernetes API Server or otherwise).", "properties": { "audience": { - "description": "Audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver.", + "description": "audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver.", "type": "string" }, "expirationSeconds": { - "description": "ExpirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes.", + "description": "expirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes.", "format": "int64", "type": "integer" }, "path": { - "description": "Path is the path relative to the mount point of the file to project the token into.", + "description": "path is the path relative to the mount point of the file to project the token into.", "type": "string" } }, @@ -9273,7 +9957,7 @@ "description": "ServicePort contains information on service's port.", "properties": { "appProtocol": { - "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", + "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", "type": "string" }, "name": { @@ -9291,7 +9975,7 @@ "type": "integer" }, "protocol": { - "description": "The IP protocol for this port. Supports \"TCP\", \"UDP\", and \"SCTP\". Default is TCP.", + "description": "The IP protocol for this port. Supports \"TCP\", \"UDP\", and \"SCTP\". Default is TCP.\n\n", "type": "string" }, "targetPort": { @@ -9308,7 +9992,7 @@ "description": "ServiceSpec describes the attributes that a user creates on a service.", "properties": { "allocateLoadBalancerNodePorts": { - "description": "allocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer. Default is \"true\". It may be set to \"false\" if the cluster load-balancer does not rely on NodePorts. If the caller requests specific NodePorts (by specifying a value), those requests will be respected, regardless of this field. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type. This field is beta-level and is only honored by servers that enable the ServiceLBNodePortControl feature.", + "description": "allocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer. Default is \"true\". It may be set to \"false\" if the cluster load-balancer does not rely on NodePorts. If the caller requests specific NodePorts (by specifying a value), those requests will be respected, regardless of this field. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type.", "type": "boolean" }, "clusterIP": { @@ -9316,7 +10000,7 @@ "type": "string" }, "clusterIPs": { - "description": "ClusterIPs is a list of IP addresses assigned to this service, and are usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be empty) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are \"None\", empty string (\"\"), or a valid IP address. Setting this to \"None\" makes a \"headless service\" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. If this field is not specified, it will be initialized from the clusterIP field. If this field is specified, clients must ensure that clusterIPs[0] and clusterIP have the same value.\n\nUnless the \"IPv6DualStack\" feature gate is enabled, this field is limited to one value, which must be the same as the clusterIP field. If the feature gate is enabled, this field may hold a maximum of two entries (dual-stack IPs, in either order). These IPs must correspond to the values of the ipFamilies field. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies", + "description": "ClusterIPs is a list of IP addresses assigned to this service, and are usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be empty) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are \"None\", empty string (\"\"), or a valid IP address. Setting this to \"None\" makes a \"headless service\" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. If this field is not specified, it will be initialized from the clusterIP field. If this field is specified, clients must ensure that clusterIPs[0] and clusterIP have the same value.\n\nThis field may hold a maximum of two entries (dual-stack IPs, in either order). These IPs must correspond to the values of the ipFamilies field. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies", "items": { "type": "string" }, @@ -9335,7 +10019,7 @@ "type": "string" }, "externalTrafficPolicy": { - "description": "externalTrafficPolicy denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. \"Local\" preserves the client source IP and avoids a second hop for LoadBalancer and Nodeport type services, but risks potentially imbalanced traffic spreading. \"Cluster\" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading.", + "description": "externalTrafficPolicy denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. \"Local\" preserves the client source IP and avoids a second hop for LoadBalancer and Nodeport type services, but risks potentially imbalanced traffic spreading. \"Cluster\" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading.\n\n", "type": "string" }, "healthCheckNodePort": { @@ -9348,7 +10032,7 @@ "type": "string" }, "ipFamilies": { - "description": "IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service, and is gated by the \"IPv6DualStack\" feature gate. This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail. This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service. Valid values are \"IPv4\" and \"IPv6\". This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to \"headless\" services. This field will be wiped when updating a Service to type ExternalName.\n\nThis field may hold a maximum of two entries (dual-stack families, in either order). These families must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.", + "description": "IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service. This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail. This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service. Valid values are \"IPv4\" and \"IPv6\". This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to \"headless\" services. This field will be wiped when updating a Service to type ExternalName.\n\nThis field may hold a maximum of two entries (dual-stack families, in either order). These families must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.", "items": { "type": "string" }, @@ -9356,7 +10040,7 @@ "x-kubernetes-list-type": "atomic" }, "ipFamilyPolicy": { - "description": "IPFamilyPolicy represents the dual-stack-ness requested or required by this Service, and is gated by the \"IPv6DualStack\" feature gate. If there is no value provided, then this field will be set to SingleStack. Services can be \"SingleStack\" (a single IP family), \"PreferDualStack\" (two IP families on dual-stack configured clusters or a single IP family on single-stack clusters), or \"RequireDualStack\" (two IP families on dual-stack configured clusters, otherwise fail). The ipFamilies and clusterIPs fields depend on the value of this field. This field will be wiped when updating a service to type ExternalName.", + "description": "IPFamilyPolicy represents the dual-stack-ness requested or required by this Service. If there is no value provided, then this field will be set to SingleStack. Services can be \"SingleStack\" (a single IP family), \"PreferDualStack\" (two IP families on dual-stack configured clusters or a single IP family on single-stack clusters), or \"RequireDualStack\" (two IP families on dual-stack configured clusters, otherwise fail). The ipFamilies and clusterIPs fields depend on the value of this field. This field will be wiped when updating a service to type ExternalName.", "type": "string" }, "loadBalancerClass": { @@ -9364,7 +10048,7 @@ "type": "string" }, "loadBalancerIP": { - "description": "Only applies to Service Type: LoadBalancer LoadBalancer will get created with the IP specified in this field. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature.", + "description": "Only applies to Service Type: LoadBalancer. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature. Deprecated: This field was under-specified and its meaning varies across implementations, and it cannot support dual-stack. As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version.", "type": "string" }, "loadBalancerSourceRanges": { @@ -9401,7 +10085,7 @@ "x-kubernetes-map-type": "atomic" }, "sessionAffinity": { - "description": "Supports \"ClientIP\" and \"None\". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies", + "description": "Supports \"ClientIP\" and \"None\". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies\n\n", "type": "string" }, "sessionAffinityConfig": { @@ -9409,7 +10093,7 @@ "description": "sessionAffinityConfig contains the configurations of session affinity." }, "type": { - "description": "type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. \"ClusterIP\" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual construction of an Endpoints object or EndpointSlice objects. If clusterIP is \"None\", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a virtual IP. \"NodePort\" builds on ClusterIP and allocates a port on every node which routes to the same endpoints as the clusterIP. \"LoadBalancer\" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the same endpoints as the clusterIP. \"ExternalName\" aliases this service to the specified externalName. Several other fields do not apply to ExternalName services. More info: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types", + "description": "type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. \"ClusterIP\" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual construction of an Endpoints object or EndpointSlice objects. If clusterIP is \"None\", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a virtual IP. \"NodePort\" builds on ClusterIP and allocates a port on every node which routes to the same endpoints as the clusterIP. \"LoadBalancer\" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the same endpoints as the clusterIP. \"ExternalName\" aliases this service to the specified externalName. Several other fields do not apply to ExternalName services. More info: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types\n\n", "type": "string" } }, @@ -9452,23 +10136,23 @@ "description": "Represents a StorageOS persistent volume resource.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.ObjectReference", - "description": "SecretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted." + "description": "secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted." }, "volumeName": { - "description": "VolumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace.", + "description": "volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace.", "type": "string" }, "volumeNamespace": { - "description": "VolumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to \"default\" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.", + "description": "volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to \"default\" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.", "type": "string" } }, @@ -9478,23 +10162,23 @@ "description": "Represents a StorageOS persistent volume resource.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "readOnly": { - "description": "Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", + "description": "readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.", "type": "boolean" }, "secretRef": { "$ref": "#/definitions/io.k8s.api.core.v1.LocalObjectReference", - "description": "SecretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted." + "description": "secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted." }, "volumeName": { - "description": "VolumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace.", + "description": "volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace.", "type": "string" }, "volumeNamespace": { - "description": "VolumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to \"default\" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.", + "description": "volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to \"default\" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.", "type": "string" } }, @@ -9539,7 +10223,7 @@ "description": "The node this Taint is attached to has the \"effect\" on any pod that does not tolerate the Taint.", "properties": { "effect": { - "description": "Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.", + "description": "Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.\n\n", "type": "string" }, "key": { @@ -9565,7 +10249,7 @@ "description": "The pod this Toleration is attached to tolerates any taint that matches the triple using the matching operator .", "properties": { "effect": { - "description": "Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.", + "description": "Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.\n\n", "type": "string" }, "key": { @@ -9573,7 +10257,7 @@ "type": "string" }, "operator": { - "description": "Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.", + "description": "Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.\n\n", "type": "string" }, "tolerationSeconds": { @@ -9631,16 +10315,21 @@ "description": "LabelSelector is used to find matching pods. Pods that match this label selector are counted to determine the number of pods in their corresponding topology domain." }, "maxSkew": { - "description": "MaxSkew describes the degree to which pods may be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 1/1/0: | zone1 | zone2 | zone3 | | P | P | | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 1/1/1; scheduling it onto zone1(zone2) would make the ActualSkew(2-0) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.", + "description": "MaxSkew describes the degree to which pods may be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. The global minimum is the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 2/2/1: In this case, the global minimum is 1. | zone1 | zone2 | zone3 | | P P | P P | P | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.", + "format": "int32", + "type": "integer" + }, + "minDomains": { + "description": "MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats \"global minimum\" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result, when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule.\n\nFor example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same labelSelector spread as 2/2/2: | zone1 | zone2 | zone3 | | P P | P P | P P | The number of domains is less than 5(MinDomains), so \"global minimum\" is treated as 0. In this situation, new pod with the same labelSelector cannot be scheduled, because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, it will violate MaxSkew.\n\nThis is an alpha field and requires enabling MinDomainsInPodTopologySpread feature gate.", "format": "int32", "type": "integer" }, "topologyKey": { - "description": "TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each as a \"bucket\", and try to put balanced number of pods into each bucket. It's a required field.", + "description": "TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each as a \"bucket\", and try to put balanced number of pods into each bucket. We define a domain as a particular instance of a topology. Also, we define an eligible domain as a domain whose nodes match the node selector. e.g. If TopologyKey is \"kubernetes.io/hostname\", each Node is a domain of that topology. And, if TopologyKey is \"topology.kubernetes.io/zone\", each zone is a domain of that topology. It's a required field.", "type": "string" }, "whenUnsatisfiable": { - "description": "WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint. - DoNotSchedule (default) tells the scheduler not to schedule it. - ScheduleAnyway tells the scheduler to schedule the pod in any location,\n but giving higher precedence to topologies that would help reduce the\n skew.\nA constraint is considered \"Unsatisfiable\" for an incoming pod if and only if every possible node assigment for that pod would violate \"MaxSkew\" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P | P | P | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it *more* imbalanced. It's a required field.", + "description": "WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint. - DoNotSchedule (default) tells the scheduler not to schedule it. - ScheduleAnyway tells the scheduler to schedule the pod in any location,\n but giving higher precedence to topologies that would help reduce the\n skew.\nA constraint is considered \"Unsatisfiable\" for an incoming pod if and only if every possible node assignment for that pod would violate \"MaxSkew\" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P | P | P | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it *more* imbalanced. It's a required field.\n\n", "type": "string" } }, @@ -9679,123 +10368,123 @@ "properties": { "awsElasticBlockStore": { "$ref": "#/definitions/io.k8s.api.core.v1.AWSElasticBlockStoreVolumeSource", - "description": "AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore" + "description": "awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore" }, "azureDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.AzureDiskVolumeSource", - "description": "AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod." + "description": "azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod." }, "azureFile": { "$ref": "#/definitions/io.k8s.api.core.v1.AzureFileVolumeSource", - "description": "AzureFile represents an Azure File Service mount on the host and bind mount to the pod." + "description": "azureFile represents an Azure File Service mount on the host and bind mount to the pod." }, "cephfs": { "$ref": "#/definitions/io.k8s.api.core.v1.CephFSVolumeSource", - "description": "CephFS represents a Ceph FS mount on the host that shares a pod's lifetime" + "description": "cephFS represents a Ceph FS mount on the host that shares a pod's lifetime" }, "cinder": { "$ref": "#/definitions/io.k8s.api.core.v1.CinderVolumeSource", - "description": "Cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md" + "description": "cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md" }, "configMap": { "$ref": "#/definitions/io.k8s.api.core.v1.ConfigMapVolumeSource", - "description": "ConfigMap represents a configMap that should populate this volume" + "description": "configMap represents a configMap that should populate this volume" }, "csi": { "$ref": "#/definitions/io.k8s.api.core.v1.CSIVolumeSource", - "description": "CSI (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature)." + "description": "csi (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature)." }, "downwardAPI": { "$ref": "#/definitions/io.k8s.api.core.v1.DownwardAPIVolumeSource", - "description": "DownwardAPI represents downward API about the pod that should populate this volume" + "description": "downwardAPI represents downward API about the pod that should populate this volume" }, "emptyDir": { "$ref": "#/definitions/io.k8s.api.core.v1.EmptyDirVolumeSource", - "description": "EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir" + "description": "emptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir" }, "ephemeral": { "$ref": "#/definitions/io.k8s.api.core.v1.EphemeralVolumeSource", - "description": "Ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.\n\nUse this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity\n tracking are needed,\nc) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through\n a PersistentVolumeClaim (see EphemeralVolumeSource for more\n information on the connection between this volume type\n and PersistentVolumeClaim).\n\nUse PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod.\n\nUse CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information.\n\nA pod can use both types of ephemeral volumes and persistent volumes at the same time.\n\nThis is a beta feature and only available when the GenericEphemeralVolume feature gate is enabled." + "description": "ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.\n\nUse this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity\n tracking are needed,\nc) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through\n a PersistentVolumeClaim (see EphemeralVolumeSource for more\n information on the connection between this volume type\n and PersistentVolumeClaim).\n\nUse PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod.\n\nUse CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information.\n\nA pod can use both types of ephemeral volumes and persistent volumes at the same time." }, "fc": { "$ref": "#/definitions/io.k8s.api.core.v1.FCVolumeSource", - "description": "FC represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod." + "description": "fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod." }, "flexVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.FlexVolumeSource", - "description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin." + "description": "flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin." }, "flocker": { "$ref": "#/definitions/io.k8s.api.core.v1.FlockerVolumeSource", - "description": "Flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running" + "description": "flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running" }, "gcePersistentDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.GCEPersistentDiskVolumeSource", - "description": "GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk" + "description": "gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk" }, "gitRepo": { "$ref": "#/definitions/io.k8s.api.core.v1.GitRepoVolumeSource", - "description": "GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container." + "description": "gitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container." }, "glusterfs": { "$ref": "#/definitions/io.k8s.api.core.v1.GlusterfsVolumeSource", - "description": "Glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md" + "description": "glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md" }, "hostPath": { "$ref": "#/definitions/io.k8s.api.core.v1.HostPathVolumeSource", - "description": "HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath" + "description": "hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath" }, "iscsi": { "$ref": "#/definitions/io.k8s.api.core.v1.ISCSIVolumeSource", - "description": "ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md" + "description": "iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md" }, "name": { - "description": "Volume's name. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names", + "description": "name of the volume. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names", "type": "string" }, "nfs": { "$ref": "#/definitions/io.k8s.api.core.v1.NFSVolumeSource", - "description": "NFS represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs" + "description": "nfs represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs" }, "persistentVolumeClaim": { "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeClaimVolumeSource", - "description": "PersistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" + "description": "persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims" }, "photonPersistentDisk": { "$ref": "#/definitions/io.k8s.api.core.v1.PhotonPersistentDiskVolumeSource", - "description": "PhotonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine" + "description": "photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine" }, "portworxVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.PortworxVolumeSource", - "description": "PortworxVolume represents a portworx volume attached and mounted on kubelets host machine" + "description": "portworxVolume represents a portworx volume attached and mounted on kubelets host machine" }, "projected": { "$ref": "#/definitions/io.k8s.api.core.v1.ProjectedVolumeSource", - "description": "Items for all in one resources secrets, configmaps, and downward API" + "description": "projected items for all in one resources secrets, configmaps, and downward API" }, "quobyte": { "$ref": "#/definitions/io.k8s.api.core.v1.QuobyteVolumeSource", - "description": "Quobyte represents a Quobyte mount on the host that shares a pod's lifetime" + "description": "quobyte represents a Quobyte mount on the host that shares a pod's lifetime" }, "rbd": { "$ref": "#/definitions/io.k8s.api.core.v1.RBDVolumeSource", - "description": "RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md" + "description": "rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md" }, "scaleIO": { "$ref": "#/definitions/io.k8s.api.core.v1.ScaleIOVolumeSource", - "description": "ScaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes." + "description": "scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes." }, "secret": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretVolumeSource", - "description": "Secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret" + "description": "secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret" }, "storageos": { "$ref": "#/definitions/io.k8s.api.core.v1.StorageOSVolumeSource", - "description": "StorageOS represents a StorageOS volume attached and mounted on Kubernetes nodes." + "description": "storageOS represents a StorageOS volume attached and mounted on Kubernetes nodes." }, "vsphereVolume": { "$ref": "#/definitions/io.k8s.api.core.v1.VsphereVirtualDiskVolumeSource", - "description": "VsphereVolume represents a vSphere volume attached and mounted on kubelets host machine" + "description": "vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine" } }, "required": [ @@ -9860,7 +10549,7 @@ "properties": { "required": { "$ref": "#/definitions/io.k8s.api.core.v1.NodeSelector", - "description": "Required specifies hard node constraints that must be met." + "description": "required specifies hard node constraints that must be met." } }, "type": "object" @@ -9870,19 +10559,19 @@ "properties": { "configMap": { "$ref": "#/definitions/io.k8s.api.core.v1.ConfigMapProjection", - "description": "information about the configMap data to project" + "description": "configMap information about the configMap data to project" }, "downwardAPI": { "$ref": "#/definitions/io.k8s.api.core.v1.DownwardAPIProjection", - "description": "information about the downwardAPI data to project" + "description": "downwardAPI information about the downwardAPI data to project" }, "secret": { "$ref": "#/definitions/io.k8s.api.core.v1.SecretProjection", - "description": "information about the secret data to project" + "description": "secret information about the secret data to project" }, "serviceAccountToken": { "$ref": "#/definitions/io.k8s.api.core.v1.ServiceAccountTokenProjection", - "description": "information about the serviceAccountToken data to project" + "description": "serviceAccountToken is information about the serviceAccountToken data to project" } }, "type": "object" @@ -9891,19 +10580,19 @@ "description": "Represents a vSphere volume resource.", "properties": { "fsType": { - "description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", + "description": "fsType is filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.", "type": "string" }, "storagePolicyID": { - "description": "Storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName.", + "description": "storagePolicyID is the storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName.", "type": "string" }, "storagePolicyName": { - "description": "Storage Policy Based Management (SPBM) profile name.", + "description": "storagePolicyName is the storage Policy Based Management (SPBM) profile name.", "type": "string" }, "volumePath": { - "description": "Path that identifies vSphere volume vmdk", + "description": "volumePath is the path that identifies vSphere volume vmdk", "type": "string" } }, @@ -9957,7 +10646,7 @@ "description": "Endpoint represents a single logical \"backend\" implementing a service.", "properties": { "addresses": { - "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", + "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", "items": { "type": "string" }, @@ -10037,7 +10726,7 @@ "description": "EndpointPort represents a Port used by an EndpointSlice", "properties": { "appProtocol": { - "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", + "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", "type": "string" }, "name": { @@ -10061,7 +10750,7 @@ "description": "EndpointSlice represents a subset of the endpoints that implement a service. For a given service there may be multiple EndpointSlice objects, selected by labels, which must be joined to produce the full set of endpoints.", "properties": { "addressType": { - "description": "addressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Represents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name.", + "description": "addressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Represents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name.\n\n", "type": "string" }, "apiVersion": { @@ -10158,7 +10847,7 @@ "description": "Endpoint represents a single logical \"backend\" implementing a service.", "properties": { "addresses": { - "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", + "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", "items": { "type": "string" }, @@ -10234,7 +10923,7 @@ "description": "EndpointPort represents a Port used by an EndpointSlice", "properties": { "appProtocol": { - "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", + "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", "type": "string" }, "name": { @@ -11065,7 +11754,7 @@ "type": "object" }, "io.k8s.api.flowcontrol.v1beta1.ResourcePolicyRule": { - "description": "ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) least one member of namespaces matches the request.", + "description": "ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) either (d1) the request does not specify a namespace (i.e., `Namespace==\"\"`) and clusterScope is true or (d2) the request specifies a namespace and least one member of namespaces matches the request's namespace.", "properties": { "apiGroups": { "description": "`apiGroups` is a list of matching API groups and may not be empty. \"*\" matches all API groups and, if present, must be the only entry. Required.", @@ -11177,6 +11866,555 @@ ], "type": "object" }, + "io.k8s.api.flowcontrol.v1beta2.FlowDistinguisherMethod": { + "description": "FlowDistinguisherMethod specifies the method of a flow distinguisher.", + "properties": { + "type": { + "description": "`type` is the type of flow distinguisher method The supported types are \"ByUser\" and \"ByNamespace\". Required.", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.FlowSchema": { + "description": "FlowSchema defines the schema of a group of flows. Note that a flow is made up of a set of inbound API requests with similar attributes and is identified by a pair of strings: the name of the FlowSchema and a \"flow distinguisher\".", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", + "description": "`metadata` is the standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + }, + "spec": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchemaSpec", + "description": "`spec` is the specification of the desired behavior of a FlowSchema. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" + }, + "status": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchemaStatus", + "description": "`status` is the current status of a FlowSchema. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" + } + }, + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.FlowSchemaCondition": { + "description": "FlowSchemaCondition describes conditions for a FlowSchema.", + "properties": { + "lastTransitionTime": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", + "description": "`lastTransitionTime` is the last time the condition transitioned from one status to another." + }, + "message": { + "description": "`message` is a human-readable message indicating details about last transition.", + "type": "string" + }, + "reason": { + "description": "`reason` is a unique, one-word, CamelCase reason for the condition's last transition.", + "type": "string" + }, + "status": { + "description": "`status` is the status of the condition. Can be True, False, Unknown. Required.", + "type": "string" + }, + "type": { + "description": "`type` is the type of the condition. Required.", + "type": "string" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.FlowSchemaList": { + "description": "FlowSchemaList is a list of FlowSchema objects.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "items": { + "description": "`items` is a list of FlowSchemas.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + }, + "type": "array" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", + "description": "`metadata` is the standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + } + }, + "required": [ + "items" + ], + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchemaList", + "version": "v1beta2" + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.FlowSchemaSpec": { + "description": "FlowSchemaSpec describes how the FlowSchema's specification looks like.", + "properties": { + "distinguisherMethod": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowDistinguisherMethod", + "description": "`distinguisherMethod` defines how to compute the flow distinguisher for requests that match this schema. `nil` specifies that the distinguisher is disabled and thus will always be the empty string." + }, + "matchingPrecedence": { + "description": "`matchingPrecedence` is used to choose among the FlowSchemas that match a given request. The chosen FlowSchema is among those with the numerically lowest (which we take to be logically highest) MatchingPrecedence. Each MatchingPrecedence value must be ranged in [1,10000]. Note that if the precedence is not specified, it will be set to 1000 as default.", + "format": "int32", + "type": "integer" + }, + "priorityLevelConfiguration": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationReference", + "description": "`priorityLevelConfiguration` should reference a PriorityLevelConfiguration in the cluster. If the reference cannot be resolved, the FlowSchema will be ignored and marked as invalid in its status. Required." + }, + "rules": { + "description": "`rules` describes which requests will match this flow schema. This FlowSchema matches a request if and only if at least one member of rules matches the request. if it is an empty slice, there will be no requests matching the FlowSchema.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PolicyRulesWithSubjects" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + } + }, + "required": [ + "priorityLevelConfiguration" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.FlowSchemaStatus": { + "description": "FlowSchemaStatus represents the current state of a FlowSchema.", + "properties": { + "conditions": { + "description": "`conditions` is a list of the current states of FlowSchema.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchemaCondition" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "type" + ], + "x-kubernetes-list-type": "map" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.GroupSubject": { + "description": "GroupSubject holds detailed information for group-kind subject.", + "properties": { + "name": { + "description": "name is the user group that matches, or \"*\" to match all user groups. See https://github.com/kubernetes/apiserver/blob/master/pkg/authentication/user/user.go for some well-known group names. Required.", + "type": "string" + } + }, + "required": [ + "name" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.LimitResponse": { + "description": "LimitResponse defines how to handle requests that can not be executed right now.", + "properties": { + "queuing": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.QueuingConfiguration", + "description": "`queuing` holds the configuration parameters for queuing. This field may be non-empty only if `type` is `\"Queue\"`." + }, + "type": { + "description": "`type` is \"Queue\" or \"Reject\". \"Queue\" means that requests that can not be executed upon arrival are held in a queue until they can be executed or a queuing limit is reached. \"Reject\" means that requests that can not be executed upon arrival are rejected. Required.", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object", + "x-kubernetes-unions": [ + { + "discriminator": "type", + "fields-to-discriminateBy": { + "queuing": "Queuing" + } + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.LimitedPriorityLevelConfiguration": { + "description": "LimitedPriorityLevelConfiguration specifies how to handle requests that are subject to limits. It addresses two issues:\n * How are requests for this priority level limited?\n * What should be done with requests that exceed the limit?", + "properties": { + "assuredConcurrencyShares": { + "description": "`assuredConcurrencyShares` (ACS) configures the execution limit, which is a limit on the number of requests of this priority level that may be exeucting at a given time. ACS must be a positive number. The server's concurrency limit (SCL) is divided among the concurrency-controlled priority levels in proportion to their assured concurrency shares. This produces the assured concurrency value (ACV) --- the number of requests that may be executing at a time --- for each such priority level:\n\n ACV(l) = ceil( SCL * ACS(l) / ( sum[priority levels k] ACS(k) ) )\n\nbigger numbers of ACS mean more reserved concurrent requests (at the expense of every other PL). This field has a default value of 30.", + "format": "int32", + "type": "integer" + }, + "limitResponse": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.LimitResponse", + "description": "`limitResponse` indicates what to do with requests that can not be executed right now" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.NonResourcePolicyRule": { + "description": "NonResourcePolicyRule is a predicate that matches non-resource requests according to their verb and the target non-resource URL. A NonResourcePolicyRule matches a request if and only if both (a) at least one member of verbs matches the request and (b) at least one member of nonResourceURLs matches the request.", + "properties": { + "nonResourceURLs": { + "description": "`nonResourceURLs` is a set of url prefixes that a user should have access to and may not be empty. For example:\n - \"/healthz\" is legal\n - \"/hea*\" is illegal\n - \"/hea\" is legal but matches nothing\n - \"/hea/*\" also matches nothing\n - \"/healthz/*\" matches all per-component health checks.\n\"*\" matches all non-resource urls. if it is present, it must be the only entry. Required.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + }, + "verbs": { + "description": "`verbs` is a list of matching verbs and may not be empty. \"*\" matches all verbs. If it is present, it must be the only entry. Required.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + } + }, + "required": [ + "verbs", + "nonResourceURLs" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.PolicyRulesWithSubjects": { + "description": "PolicyRulesWithSubjects prescribes a test that applies to a request to an apiserver. The test considers the subject making the request, the verb being requested, and the resource to be acted upon. This PolicyRulesWithSubjects matches a request if and only if both (a) at least one member of subjects matches the request and (b) at least one member of resourceRules or nonResourceRules matches the request.", + "properties": { + "nonResourceRules": { + "description": "`nonResourceRules` is a list of NonResourcePolicyRules that identify matching requests according to their verb and the target non-resource URL.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.NonResourcePolicyRule" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + }, + "resourceRules": { + "description": "`resourceRules` is a slice of ResourcePolicyRules that identify matching requests according to their verb and the target resource. At least one of `resourceRules` and `nonResourceRules` has to be non-empty.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.ResourcePolicyRule" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + }, + "subjects": { + "description": "subjects is the list of normal user, serviceaccount, or group that this rule cares about. There must be at least one member in this slice. A slice that includes both the system:authenticated and system:unauthenticated user groups matches every request. Required.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.Subject" + }, + "type": "array", + "x-kubernetes-list-type": "atomic" + } + }, + "required": [ + "subjects" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration": { + "description": "PriorityLevelConfiguration represents the configuration of a priority level.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", + "description": "`metadata` is the standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + }, + "spec": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationSpec", + "description": "`spec` is the specification of the desired behavior of a \"request-priority\". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" + }, + "status": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationStatus", + "description": "`status` is the current status of a \"request-priority\". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" + } + }, + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationCondition": { + "description": "PriorityLevelConfigurationCondition defines the condition of priority level.", + "properties": { + "lastTransitionTime": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", + "description": "`lastTransitionTime` is the last time the condition transitioned from one status to another." + }, + "message": { + "description": "`message` is a human-readable message indicating details about last transition.", + "type": "string" + }, + "reason": { + "description": "`reason` is a unique, one-word, CamelCase reason for the condition's last transition.", + "type": "string" + }, + "status": { + "description": "`status` is the status of the condition. Can be True, False, Unknown. Required.", + "type": "string" + }, + "type": { + "description": "`type` is the type of the condition. Required.", + "type": "string" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationList": { + "description": "PriorityLevelConfigurationList is a list of PriorityLevelConfiguration objects.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "items": { + "description": "`items` is a list of request-priorities.", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + }, + "type": "array" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", + "description": "`metadata` is the standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + } + }, + "required": [ + "items" + ], + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfigurationList", + "version": "v1beta2" + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationReference": { + "description": "PriorityLevelConfigurationReference contains information that points to the \"request-priority\" being used.", + "properties": { + "name": { + "description": "`name` is the name of the priority level configuration being referenced Required.", + "type": "string" + } + }, + "required": [ + "name" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationSpec": { + "description": "PriorityLevelConfigurationSpec specifies the configuration of a priority level.", + "properties": { + "limited": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.LimitedPriorityLevelConfiguration", + "description": "`limited` specifies how requests are handled for a Limited priority level. This field must be non-empty if and only if `type` is `\"Limited\"`." + }, + "type": { + "description": "`type` indicates whether this priority level is subject to limitation on request execution. A value of `\"Exempt\"` means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract from the capacity made available to other priority levels. A value of `\"Limited\"` means that (a) requests of this priority level _are_ subject to limits and (b) some of the server's limited capacity is made available exclusively to this priority level. Required.", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object", + "x-kubernetes-unions": [ + { + "discriminator": "type", + "fields-to-discriminateBy": { + "limited": "Limited" + } + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationStatus": { + "description": "PriorityLevelConfigurationStatus represents the current state of a \"request-priority\".", + "properties": { + "conditions": { + "description": "`conditions` is the current state of \"request-priority\".", + "items": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationCondition" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "type" + ], + "x-kubernetes-list-type": "map" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.QueuingConfiguration": { + "description": "QueuingConfiguration holds the configuration parameters for queuing", + "properties": { + "handSize": { + "description": "`handSize` is a small positive number that configures the shuffle sharding of requests into queues. When enqueuing a request at this priority level the request's flow identifier (a string pair) is hashed and the hash value is used to shuffle the list of queues and deal a hand of the size specified here. The request is put into one of the shortest queues in that hand. `handSize` must be no larger than `queues`, and should be significantly smaller (so that a few heavy flows do not saturate most of the queues). See the user-facing documentation for more extensive guidance on setting this field. This field has a default value of 8.", + "format": "int32", + "type": "integer" + }, + "queueLengthLimit": { + "description": "`queueLengthLimit` is the maximum number of requests allowed to be waiting in a given queue of this priority level at a time; excess requests are rejected. This value must be positive. If not specified, it will be defaulted to 50.", + "format": "int32", + "type": "integer" + }, + "queues": { + "description": "`queues` is the number of queues for this priority level. The queues exist independently at each apiserver. The value must be positive. Setting it to 1 effectively precludes shufflesharding and thus makes the distinguisher method of associated flow schemas irrelevant. This field has a default value of 64.", + "format": "int32", + "type": "integer" + } + }, + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.ResourcePolicyRule": { + "description": "ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) either (d1) the request does not specify a namespace (i.e., `Namespace==\"\"`) and clusterScope is true or (d2) the request specifies a namespace and least one member of namespaces matches the request's namespace.", + "properties": { + "apiGroups": { + "description": "`apiGroups` is a list of matching API groups and may not be empty. \"*\" matches all API groups and, if present, must be the only entry. Required.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + }, + "clusterScope": { + "description": "`clusterScope` indicates whether to match requests that do not specify a namespace (which happens either because the resource is not namespaced or the request targets all namespaces). If this field is omitted or false then the `namespaces` field must contain a non-empty list.", + "type": "boolean" + }, + "namespaces": { + "description": "`namespaces` is a list of target namespaces that restricts matches. A request that specifies a target namespace matches only if either (a) this list contains that target namespace or (b) this list contains \"*\". Note that \"*\" matches any specified namespace but does not match a request that _does not specify_ a namespace (see the `clusterScope` field for that). This list may be empty, but only if `clusterScope` is true.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + }, + "resources": { + "description": "`resources` is a list of matching resources (i.e., lowercase and plural) with, if desired, subresource. For example, [ \"services\", \"nodes/status\" ]. This list may not be empty. \"*\" matches all resources and, if present, must be the only entry. Required.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + }, + "verbs": { + "description": "`verbs` is a list of matching verbs and may not be empty. \"*\" matches all verbs and, if present, must be the only entry. Required.", + "items": { + "type": "string" + }, + "type": "array", + "x-kubernetes-list-type": "set" + } + }, + "required": [ + "verbs", + "apiGroups", + "resources" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.ServiceAccountSubject": { + "description": "ServiceAccountSubject holds detailed information for service-account-kind subject.", + "properties": { + "name": { + "description": "`name` is the name of matching ServiceAccount objects, or \"*\" to match regardless of name. Required.", + "type": "string" + }, + "namespace": { + "description": "`namespace` is the namespace of matching ServiceAccount objects. Required.", + "type": "string" + } + }, + "required": [ + "namespace", + "name" + ], + "type": "object" + }, + "io.k8s.api.flowcontrol.v1beta2.Subject": { + "description": "Subject matches the originator of a request, as identified by the request authentication system. There are three ways of matching an originator; by user, group, or service account.", + "properties": { + "group": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.GroupSubject", + "description": "`group` matches based on user group name." + }, + "kind": { + "description": "`kind` indicates which one of the other fields is non-empty. Required", + "type": "string" + }, + "serviceAccount": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.ServiceAccountSubject", + "description": "`serviceAccount` matches ServiceAccounts." + }, + "user": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.UserSubject", + "description": "`user` matches based on username." + } + }, + "required": [ + "kind" + ], + "type": "object", + "x-kubernetes-unions": [ + { + "discriminator": "kind", + "fields-to-discriminateBy": { + "group": "Group", + "serviceAccount": "ServiceAccount", + "user": "User" + } + } + ] + }, + "io.k8s.api.flowcontrol.v1beta2.UserSubject": { + "description": "UserSubject holds detailed information for user-kind subject.", + "properties": { + "name": { + "description": "`name` is the username that matches, or \"*\" to match all usernames. Required.", + "type": "string" + } + }, + "required": [ + "name" + ], + "type": "object" + }, "io.k8s.api.networking.v1.HTTPIngressPath": { "description": "HTTPIngressPath associates a path with a backend. Incoming urls matching the path are forwarded to the backend.", "properties": { @@ -11367,7 +12605,7 @@ "type": "string" }, "scope": { - "description": "Scope represents if this refers to a cluster or namespace scoped resource. This may be set to \"Cluster\" (default) or \"Namespace\". Field can be enabled with IngressClassNamespacedParams feature gate.", + "description": "Scope represents if this refers to a cluster or namespace scoped resource. This may be set to \"Cluster\" (default) or \"Namespace\".", "type": "string" } }, @@ -11532,6 +12770,10 @@ "spec": { "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicySpec", "description": "Specification of the desired behavior for this NetworkPolicy." + }, + "status": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicyStatus", + "description": "Status is the current state of the NetworkPolicy. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" } }, "type": "object", @@ -11689,6 +12931,25 @@ ], "type": "object" }, + "io.k8s.api.networking.v1.NetworkPolicyStatus": { + "description": "NetworkPolicyStatus describe the current state of the NetworkPolicy.", + "properties": { + "conditions": { + "description": "Conditions holds an array of metav1.Condition that describe the state of the NetworkPolicy. Current service state", + "items": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Condition" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "type" + ], + "x-kubernetes-list-type": "map", + "x-kubernetes-patch-merge-key": "type", + "x-kubernetes-patch-strategy": "merge" + } + }, + "type": "object" + }, "io.k8s.api.networking.v1.ServiceBackendPort": { "description": "ServiceBackendPort is the service port being referenced.", "properties": { @@ -11738,7 +12999,7 @@ }, "overhead": { "$ref": "#/definitions/io.k8s.api.node.v1.Overhead", - "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see\n https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/\nThis field is in beta starting v1.18 and is only honored by servers that enable the PodOverhead feature." + "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see\n https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/" }, "scheduling": { "$ref": "#/definitions/io.k8s.api.node.v1.Scheduling", @@ -11814,129 +13075,6 @@ }, "type": "object" }, - "io.k8s.api.node.v1alpha1.Overhead": { - "description": "Overhead structure represents the resource overhead associated with running a pod.", - "properties": { - "podFixed": { - "additionalProperties": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity" - }, - "description": "PodFixed represents the fixed resource overhead associated with running a pod.", - "type": "object" - } - }, - "type": "object" - }, - "io.k8s.api.node.v1alpha1.RuntimeClass": { - "description": "RuntimeClass defines a class of container runtime supported in the cluster. The RuntimeClass is used to determine which container runtime is used to run all containers in a pod. RuntimeClasses are (currently) manually defined by a user or cluster provisioner, and referenced in the PodSpec. The Kubelet is responsible for resolving the RuntimeClassName reference before running the pod. For more details, see https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - }, - "spec": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClassSpec", - "description": "Specification of the RuntimeClass More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status" - } - }, - "required": [ - "spec" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.node.v1alpha1.RuntimeClassList": { - "description": "RuntimeClassList is a list of RuntimeClass objects.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is a list of schema objects.", - "items": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "node.k8s.io", - "kind": "RuntimeClassList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.node.v1alpha1.RuntimeClassSpec": { - "description": "RuntimeClassSpec is a specification of a RuntimeClass. It contains parameters that are required to describe the RuntimeClass to the Container Runtime Interface (CRI) implementation, as well as any other components that need to understand how the pod will be run. The RuntimeClassSpec is immutable.", - "properties": { - "overhead": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.Overhead", - "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md This field is beta-level as of Kubernetes v1.18, and is only honored by servers that enable the PodOverhead feature." - }, - "runtimeHandler": { - "description": "RuntimeHandler specifies the underlying runtime and configuration that the CRI implementation will use to handle pods of this class. The possible values are specific to the node & CRI configuration. It is assumed that all handlers are available on every node, and handlers of the same name are equivalent on every node. For example, a handler called \"runc\" might specify that the runc OCI runtime (using native Linux containers) will be used to run the containers in a pod. The RuntimeHandler must be lowercase, conform to the DNS Label (RFC 1123) requirements, and is immutable.", - "type": "string" - }, - "scheduling": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.Scheduling", - "description": "Scheduling holds the scheduling constraints to ensure that pods running with this RuntimeClass are scheduled to nodes that support it. If scheduling is nil, this RuntimeClass is assumed to be supported by all nodes." - } - }, - "required": [ - "runtimeHandler" - ], - "type": "object" - }, - "io.k8s.api.node.v1alpha1.Scheduling": { - "description": "Scheduling specifies the scheduling constraints for nodes supporting a RuntimeClass.", - "properties": { - "nodeSelector": { - "additionalProperties": { - "type": "string" - }, - "description": "nodeSelector lists labels that must be present on nodes that support this RuntimeClass. Pods using this RuntimeClass can only be scheduled to a node matched by this selector. The RuntimeClass nodeSelector is merged with a pod's existing nodeSelector. Any conflicts will cause the pod to be rejected in admission.", - "type": "object", - "x-kubernetes-map-type": "atomic" - }, - "tolerations": { - "description": "tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass.", - "items": { - "$ref": "#/definitions/io.k8s.api.core.v1.Toleration" - }, - "type": "array", - "x-kubernetes-list-type": "atomic" - } - }, - "type": "object" - }, "io.k8s.api.node.v1beta1.Overhead": { "description": "Overhead structure represents the resource overhead associated with running a pod.", "properties": { @@ -11971,7 +13109,7 @@ }, "overhead": { "$ref": "#/definitions/io.k8s.api.node.v1beta1.Overhead", - "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md This field is beta-level as of Kubernetes v1.18, and is only honored by servers that enable the PodOverhead feature." + "description": "Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md" }, "scheduling": { "$ref": "#/definitions/io.k8s.api.node.v1beta1.Scheduling", @@ -12398,8 +13536,7 @@ }, "selector": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", - "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace.", - "x-kubernetes-patch-strategy": "replace" + "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace." } }, "type": "object" @@ -12950,7 +14087,7 @@ "type": "array" }, "verbs": { - "description": "Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. '*' represents all verbs.", + "description": "Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '*' represents all verbs.", "items": { "type": "string" }, @@ -13154,398 +14291,6 @@ "type": "object", "x-kubernetes-map-type": "atomic" }, - "io.k8s.api.rbac.v1alpha1.AggregationRule": { - "description": "AggregationRule describes how to locate ClusterRoles to aggregate into the ClusterRole", - "properties": { - "clusterRoleSelectors": { - "description": "ClusterRoleSelectors holds a list of selectors which will be used to find ClusterRoles and create the rules. If any of the selectors match, then the ClusterRole's permissions will be added", - "items": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector" - }, - "type": "array" - } - }, - "type": "object" - }, - "io.k8s.api.rbac.v1alpha1.ClusterRole": { - "description": "ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRole, and will no longer be served in v1.22.", - "properties": { - "aggregationRule": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.AggregationRule", - "description": "AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be stomped by the controller." - }, - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata." - }, - "rules": { - "description": "Rules holds all the PolicyRules for this ClusterRole", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.PolicyRule" - }, - "type": "array" - } - }, - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.ClusterRoleBinding": { - "description": "ClusterRoleBinding references a ClusterRole, but not contain it. It can reference a ClusterRole in the global namespace, and adds who information via Subject. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRoleBinding, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata." - }, - "roleRef": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleRef", - "description": "RoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error." - }, - "subjects": { - "description": "Subjects holds references to the objects the role applies to.", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Subject" - }, - "type": "array" - } - }, - "required": [ - "roleRef" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.ClusterRoleBindingList": { - "description": "ClusterRoleBindingList is a collection of ClusterRoleBindings. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRoleBindings, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is a list of ClusterRoleBindings", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard object's metadata." - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBindingList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.ClusterRoleList": { - "description": "ClusterRoleList is a collection of ClusterRoles. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRoles, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is a list of ClusterRoles", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard object's metadata." - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.PolicyRule": { - "description": "PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.", - "properties": { - "apiGroups": { - "description": "APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.", - "items": { - "type": "string" - }, - "type": "array" - }, - "nonResourceURLs": { - "description": "NonResourceURLs is a set of partial urls that a user should have access to. *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as \"pods\" or \"secrets\") or non-resource URL paths (such as \"/api\"), but not both.", - "items": { - "type": "string" - }, - "type": "array" - }, - "resourceNames": { - "description": "ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed.", - "items": { - "type": "string" - }, - "type": "array" - }, - "resources": { - "description": "Resources is a list of resources this rule applies to. '*' represents all resources.", - "items": { - "type": "string" - }, - "type": "array" - }, - "verbs": { - "description": "Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. '*' represents all verbs.", - "items": { - "type": "string" - }, - "type": "array" - } - }, - "required": [ - "verbs" - ], - "type": "object" - }, - "io.k8s.api.rbac.v1alpha1.Role": { - "description": "Role is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 Role, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata." - }, - "rules": { - "description": "Rules holds all the PolicyRules for this Role", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.PolicyRule" - }, - "type": "array" - } - }, - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.RoleBinding": { - "description": "RoleBinding references a role, but does not contain it. It can reference a Role in the same namespace or a ClusterRole in the global namespace. It adds who information via Subjects and namespace information by which namespace it exists in. RoleBindings in a given namespace only have effect in that namespace. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 RoleBinding, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata." - }, - "roleRef": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleRef", - "description": "RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error." - }, - "subjects": { - "description": "Subjects holds references to the objects the role applies to.", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Subject" - }, - "type": "array" - } - }, - "required": [ - "roleRef" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.RoleBindingList": { - "description": "RoleBindingList is a collection of RoleBindings Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 RoleBindingList, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is a list of RoleBindings", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard object's metadata." - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBindingList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.RoleList": { - "description": "RoleList is a collection of Roles. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 RoleList, and will no longer be served in v1.22.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is a list of Roles", - "items": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard object's metadata." - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "rbac.authorization.k8s.io", - "kind": "RoleList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.rbac.v1alpha1.RoleRef": { - "description": "RoleRef contains information that points to the role being used", - "properties": { - "apiGroup": { - "description": "APIGroup is the group for the resource being referenced", - "type": "string" - }, - "kind": { - "description": "Kind is the type of resource being referenced", - "type": "string" - }, - "name": { - "description": "Name is the name of resource being referenced", - "type": "string" - } - }, - "required": [ - "apiGroup", - "kind", - "name" - ], - "type": "object" - }, - "io.k8s.api.rbac.v1alpha1.Subject": { - "description": "Subject contains a reference to the object or user identities a role binding applies to. This can either hold a direct API object reference, or a value for non-objects such as user and group names.", - "properties": { - "apiVersion": { - "description": "APIVersion holds the API group and version of the referenced subject. Defaults to \"v1\" for ServiceAccount subjects. Defaults to \"rbac.authorization.k8s.io/v1alpha1\" for User and Group subjects.", - "type": "string" - }, - "kind": { - "description": "Kind of object being referenced. Values defined by this API group are \"User\", \"Group\", and \"ServiceAccount\". If the Authorizer does not recognized the kind value, the Authorizer should report an error.", - "type": "string" - }, - "name": { - "description": "Name of the object being referenced.", - "type": "string" - }, - "namespace": { - "description": "Namespace of the referenced object. If the object kind is non-namespace, such as \"User\" or \"Group\", and this value is not empty the Authorizer should report an error.", - "type": "string" - } - }, - "required": [ - "kind", - "name" - ], - "type": "object" - }, "io.k8s.api.scheduling.v1.PriorityClass": { "description": "PriorityClass defines mapping from a priority class name to the priority integer value. The value can be any valid integer.", "properties": { @@ -13570,7 +14315,7 @@ "description": "Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" }, "preemptionPolicy": { - "description": "PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.", + "description": "PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset.", "type": "string" }, "value": { @@ -13626,86 +14371,6 @@ } ] }, - "io.k8s.api.scheduling.v1alpha1.PriorityClass": { - "description": "DEPRECATED - This group version of PriorityClass is deprecated by scheduling.k8s.io/v1/PriorityClass. PriorityClass defines mapping from a priority class name to the priority integer value. The value can be any valid integer.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "description": { - "description": "description is an arbitrary string that usually provides guidelines on when this priority class should be used.", - "type": "string" - }, - "globalDefault": { - "description": "globalDefault specifies whether this PriorityClass should be considered as the default priority for pods that do not have any priority class. Only one PriorityClass can be marked as `globalDefault`. However, if more than one PriorityClasses exists with their `globalDefault` field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.", - "type": "boolean" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - }, - "preemptionPolicy": { - "description": "PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.", - "type": "string" - }, - "value": { - "description": "The value of this priority class. This is the actual priority that pods receive when they have the name of this class in their pod spec.", - "format": "int32", - "type": "integer" - } - }, - "required": [ - "value" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.scheduling.v1alpha1.PriorityClassList": { - "description": "PriorityClassList is a collection of priority classes.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "items is the list of PriorityClasses", - "items": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "scheduling.k8s.io", - "kind": "PriorityClassList", - "version": "v1alpha1" - } - ] - }, "io.k8s.api.storage.v1.CSIDriver": { "description": "CSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster. Kubernetes attach detach controller uses this object to determine whether attach is required. Kubelet uses this object to determine whether pod information needs to be passed on mount. CSIDriver objects are non-namespaced.", "properties": { @@ -13781,7 +14446,7 @@ "type": "boolean" }, "fsGroupPolicy": { - "description": "Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details. This field is beta, and is only honored by servers that enable the CSIVolumeFSGroupPolicy feature gate.\n\nThis field is immutable.\n\nDefaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.", + "description": "Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details.\n\nThis field is immutable.\n\nDefaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.", "type": "string" }, "podInfoOnMount": { @@ -13793,7 +14458,7 @@ "type": "boolean" }, "storageCapacity": { - "description": "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.\n\nThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.\n\nAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.\n\nThis field is immutable.\n\nThis is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.", + "description": "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.\n\nThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.\n\nAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.\n\nThis field was immutable in Kubernetes <= 1.22 and now is mutable.", "type": "boolean" }, "tokenRequests": { @@ -13929,6 +14594,89 @@ ], "type": "object" }, + "io.k8s.api.storage.v1.CSIStorageCapacity": { + "description": "CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.\n\nFor example this can express things like: - StorageClass \"standard\" has \"1234 GiB\" available in \"topology.kubernetes.io/zone=us-east1\" - StorageClass \"localssd\" has \"10 GiB\" available in \"kubernetes.io/hostname=knode-abc123\"\n\nThe following three cases all imply that no capacity is available for a certain combination: - no object exists with suitable topology and storage class name - such an object exists, but the capacity is unset - such an object exists, but the capacity is zero\n\nThe producer of these objects can decide which approach is more suitable.\n\nThey are consumed by the kube-scheduler when a CSI driver opts into capacity-aware scheduling with CSIDriverSpec.StorageCapacity. The scheduler compares the MaximumVolumeSize against the requested size of pending volumes to filter out unsuitable nodes. If MaximumVolumeSize is unset, it falls back to a comparison against the less precise Capacity. If that is also unset, the scheduler assumes that capacity is insufficient and tries some other node.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "capacity": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThe semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable." + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "maximumVolumeSize": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", + "description": "MaximumVolumeSize is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThis is defined since CSI spec 1.4.0 as the largest size that may be used in a CreateVolumeRequest.capacity_range.required_bytes field to create a volume with the same parameters as those in GetCapacityRequest. The corresponding value in the Kubernetes API is ResourceRequirements.Requests in a volume claim." + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", + "description": "Standard object's metadata. The name has no particular meaning. It must be be a DNS subdomain (dots allowed, 253 characters). To ensure that there are no conflicts with other CSI drivers on the cluster, the recommendation is to use csisc-, a generated name, or a reverse-domain name which ends with the unique CSI driver name.\n\nObjects are namespaced.\n\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + }, + "nodeTopology": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", + "description": "NodeTopology defines which nodes have access to the storage for which capacity was reported. If not set, the storage is not accessible from any node in the cluster. If empty, the storage is accessible from all nodes. This field is immutable." + }, + "storageClassName": { + "description": "The name of the StorageClass that the reported capacity applies to. It must meet the same requirements as the name of a StorageClass object (non-empty, DNS subdomain). If that object no longer exists, the CSIStorageCapacity object is obsolete and should be removed by its creator. This field is immutable.", + "type": "string" + } + }, + "required": [ + "storageClassName" + ], + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + ] + }, + "io.k8s.api.storage.v1.CSIStorageCapacityList": { + "description": "CSIStorageCapacityList is a collection of CSIStorageCapacity objects.", + "properties": { + "apiVersion": { + "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", + "type": "string" + }, + "items": { + "description": "Items is the list of CSIStorageCapacity objects.", + "items": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "name" + ], + "x-kubernetes-list-type": "map" + }, + "kind": { + "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", + "type": "string" + }, + "metadata": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", + "description": "Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" + } + }, + "required": [ + "items" + ], + "type": "object", + "x-kubernetes-group-version-kind": [ + { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacityList", + "version": "v1" + } + ] + }, "io.k8s.api.storage.v1.StorageClass": { "description": "StorageClass describes the parameters for a class of storage for which PersistentVolumes can be dynamically provisioned.\n\nStorageClasses are non-namespaced; the name of the storage class according to etcd is in ObjectMeta.Name.", "properties": { @@ -14209,241 +14957,8 @@ }, "type": "object" }, - "io.k8s.api.storage.v1alpha1.CSIStorageCapacity": { - "description": "CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.\n\nFor example this can express things like: - StorageClass \"standard\" has \"1234 GiB\" available in \"topology.kubernetes.io/zone=us-east1\" - StorageClass \"localssd\" has \"10 GiB\" available in \"kubernetes.io/hostname=knode-abc123\"\n\nThe following three cases all imply that no capacity is available for a certain combination: - no object exists with suitable topology and storage class name - such an object exists, but the capacity is unset - such an object exists, but the capacity is zero\n\nThe producer of these objects can decide which approach is more suitable.\n\nThey are consumed by the kube-scheduler if the CSIStorageCapacity beta feature gate is enabled there and a CSI driver opts into capacity-aware scheduling with CSIDriver.StorageCapacity.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "capacity": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", - "description": "Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThe semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable and treated like zero capacity." - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "maximumVolumeSize": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", - "description": "MaximumVolumeSize is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThis is defined since CSI spec 1.4.0 as the largest size that may be used in a CreateVolumeRequest.capacity_range.required_bytes field to create a volume with the same parameters as those in GetCapacityRequest. The corresponding value in the Kubernetes API is ResourceRequirements.Requests in a volume claim." - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object's metadata. The name has no particular meaning. It must be be a DNS subdomain (dots allowed, 253 characters). To ensure that there are no conflicts with other CSI drivers on the cluster, the recommendation is to use csisc-, a generated name, or a reverse-domain name which ends with the unique CSI driver name.\n\nObjects are namespaced.\n\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - }, - "nodeTopology": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", - "description": "NodeTopology defines which nodes have access to the storage for which capacity was reported. If not set, the storage is not accessible from any node in the cluster. If empty, the storage is accessible from all nodes. This field is immutable." - }, - "storageClassName": { - "description": "The name of the StorageClass that the reported capacity applies to. It must meet the same requirements as the name of a StorageClass object (non-empty, DNS subdomain). If that object no longer exists, the CSIStorageCapacity object is obsolete and should be removed by its creator. This field is immutable.", - "type": "string" - } - }, - "required": [ - "storageClassName" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.storage.v1alpha1.CSIStorageCapacityList": { - "description": "CSIStorageCapacityList is a collection of CSIStorageCapacity objects.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is the list of CSIStorageCapacity objects.", - "items": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - }, - "type": "array", - "x-kubernetes-list-map-keys": [ - "name" - ], - "x-kubernetes-list-type": "map" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacityList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.storage.v1alpha1.VolumeAttachment": { - "description": "VolumeAttachment captures the intent to attach or detach the specified volume to/from the specified node.\n\nVolumeAttachment objects are non-namespaced.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", - "description": "Standard object metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - }, - "spec": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachmentSpec", - "description": "Specification of the desired attach/detach volume behavior. Populated by the Kubernetes system." - }, - "status": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachmentStatus", - "description": "Status of the VolumeAttachment request. Populated by the entity completing the attach or detach operation, i.e. the external-attacher." - } - }, - "required": [ - "spec" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.storage.v1alpha1.VolumeAttachmentList": { - "description": "VolumeAttachmentList is a collection of VolumeAttachment objects.", - "properties": { - "apiVersion": { - "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", - "type": "string" - }, - "items": { - "description": "Items is the list of VolumeAttachments", - "items": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - }, - "type": "array" - }, - "kind": { - "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", - "type": "string" - }, - "metadata": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", - "description": "Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata" - } - }, - "required": [ - "items" - ], - "type": "object", - "x-kubernetes-group-version-kind": [ - { - "group": "storage.k8s.io", - "kind": "VolumeAttachmentList", - "version": "v1alpha1" - } - ] - }, - "io.k8s.api.storage.v1alpha1.VolumeAttachmentSource": { - "description": "VolumeAttachmentSource represents a volume that should be attached. Right now only PersistenVolumes can be attached via external attacher, in future we may allow also inline volumes in pods. Exactly one member can be set.", - "properties": { - "inlineVolumeSpec": { - "$ref": "#/definitions/io.k8s.api.core.v1.PersistentVolumeSpec", - "description": "inlineVolumeSpec contains all the information necessary to attach a persistent volume defined by a pod's inline VolumeSource. This field is populated only for the CSIMigration feature. It contains translated fields from a pod's inline VolumeSource to a PersistentVolumeSpec. This field is alpha-level and is only honored by servers that enabled the CSIMigration feature." - }, - "persistentVolumeName": { - "description": "Name of the persistent volume to attach.", - "type": "string" - } - }, - "type": "object" - }, - "io.k8s.api.storage.v1alpha1.VolumeAttachmentSpec": { - "description": "VolumeAttachmentSpec is the specification of a VolumeAttachment request.", - "properties": { - "attacher": { - "description": "Attacher indicates the name of the volume driver that MUST handle this request. This is the name returned by GetPluginName().", - "type": "string" - }, - "nodeName": { - "description": "The node that the volume should be attached to.", - "type": "string" - }, - "source": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachmentSource", - "description": "Source represents the volume that should be attached." - } - }, - "required": [ - "attacher", - "source", - "nodeName" - ], - "type": "object" - }, - "io.k8s.api.storage.v1alpha1.VolumeAttachmentStatus": { - "description": "VolumeAttachmentStatus is the status of a VolumeAttachment request.", - "properties": { - "attachError": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeError", - "description": "The last error encountered during attach operation, if any. This field must only be set by the entity completing the attach operation, i.e. the external-attacher." - }, - "attached": { - "description": "Indicates the volume is successfully attached. This field must only be set by the entity completing the attach operation, i.e. the external-attacher.", - "type": "boolean" - }, - "attachmentMetadata": { - "additionalProperties": { - "type": "string" - }, - "description": "Upon successful attach, this field is populated with any information returned by the attach operation that must be passed into subsequent WaitForAttach or Mount calls. This field must only be set by the entity completing the attach operation, i.e. the external-attacher.", - "type": "object" - }, - "detachError": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeError", - "description": "The last error encountered during detach operation, if any. This field must only be set by the entity completing the detach operation, i.e. the external-attacher." - } - }, - "required": [ - "attached" - ], - "type": "object" - }, - "io.k8s.api.storage.v1alpha1.VolumeError": { - "description": "VolumeError captures an error encountered during a volume operation.", - "properties": { - "message": { - "description": "String detailing the error encountered during Attach or Detach operation. This string maybe logged, so it should not contain sensitive information.", - "type": "string" - }, - "time": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", - "description": "Time the error was encountered." - } - }, - "type": "object" - }, "io.k8s.api.storage.v1beta1.CSIStorageCapacity": { - "description": "CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.\n\nFor example this can express things like: - StorageClass \"standard\" has \"1234 GiB\" available in \"topology.kubernetes.io/zone=us-east1\" - StorageClass \"localssd\" has \"10 GiB\" available in \"kubernetes.io/hostname=knode-abc123\"\n\nThe following three cases all imply that no capacity is available for a certain combination: - no object exists with suitable topology and storage class name - such an object exists, but the capacity is unset - such an object exists, but the capacity is zero\n\nThe producer of these objects can decide which approach is more suitable.\n\nThey are consumed by the kube-scheduler if the CSIStorageCapacity beta feature gate is enabled there and a CSI driver opts into capacity-aware scheduling with CSIDriver.StorageCapacity.", + "description": "CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.\n\nFor example this can express things like: - StorageClass \"standard\" has \"1234 GiB\" available in \"topology.kubernetes.io/zone=us-east1\" - StorageClass \"localssd\" has \"10 GiB\" available in \"kubernetes.io/hostname=knode-abc123\"\n\nThe following three cases all imply that no capacity is available for a certain combination: - no object exists with suitable topology and storage class name - such an object exists, but the capacity is unset - such an object exists, but the capacity is zero\n\nThe producer of these objects can decide which approach is more suitable.\n\nThey are consumed by the kube-scheduler when a CSI driver opts into capacity-aware scheduling with CSIDriverSpec.StorageCapacity. The scheduler compares the MaximumVolumeSize against the requested size of pending volumes to filter out unsuitable nodes. If MaximumVolumeSize is unset, it falls back to a comparison against the less precise Capacity. If that is also unset, the scheduler assumes that capacity is insufficient and tries some other node.", "properties": { "apiVersion": { "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources", @@ -14451,7 +14966,7 @@ }, "capacity": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.api.resource.Quantity", - "description": "Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThe semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable and treated like zero capacity." + "description": "Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.\n\nThe semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable." }, "kind": { "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", @@ -15075,6 +15590,19 @@ "x-kubernetes-preserve-unknown-fields": { "description": "x-kubernetes-preserve-unknown-fields stops the API server decoding step from pruning fields which are not specified in the validation schema. This affects fields recursively, but switches back to normal pruning behaviour if nested properties or additionalProperties are specified in the schema. This can either be true or undefined. False is forbidden.", "type": "boolean" + }, + "x-kubernetes-validations": { + "description": "x-kubernetes-validations describes a list of validation rules written in the CEL expression language. This field is an alpha-level. Using this field requires the feature gate `CustomResourceValidationExpressions` to be enabled.", + "items": { + "$ref": "#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.ValidationRule" + }, + "type": "array", + "x-kubernetes-list-map-keys": [ + "rule" + ], + "x-kubernetes-list-type": "map", + "x-kubernetes-patch-merge-key": "rule", + "x-kubernetes-patch-strategy": "merge" } }, "type": "object" @@ -15115,6 +15643,23 @@ ], "type": "object" }, + "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.ValidationRule": { + "description": "ValidationRule describes a validation rule written in the CEL expression language.", + "properties": { + "message": { + "description": "Message represents the message displayed when validation fails. The message is required if the Rule contains line breaks. The message must not contain line breaks. If unset, the message is \"failed rule: {Rule}\". e.g. \"must be a URL with the host matching spec.host\"", + "type": "string" + }, + "rule": { + "description": "Rule represents the expression which will be evaluated by CEL. ref: https://github.com/google/cel-spec The Rule is scoped to the location of the x-kubernetes-validations extension in the schema. The `self` variable in the CEL expression is bound to the scoped value. Example: - Rule scoped to the root of a resource with a status subresource: {\"rule\": \"self.status.actual <= self.spec.maxDesired\"}\n\nIf the Rule is scoped to an object with properties, the accessible properties of the object are field selectable via `self.field` and field presence can be checked via `has(self.field)`. Null valued fields are treated as absent fields in CEL expressions. If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map are accessible via `self[mapKey]`, map containment can be checked via `mapKey in self` and all entries of the map are accessible via CEL macros and functions such as `self.all(...)`. If the Rule is scoped to an array, the elements of the array are accessible via `self[i]` and also by macros and functions. If the Rule is scoped to a scalar, `self` is bound to the scalar value. Examples: - Rule scoped to a map of objects: {\"rule\": \"self.components['Widget'].priority < 10\"} - Rule scoped to a list of integers: {\"rule\": \"self.values.all(value, value >= 0 && value < 100)\"} - Rule scoped to a string value: {\"rule\": \"self.startsWith('kube')\"}\n\nThe `apiVersion`, `kind`, `metadata.name` and `metadata.generateName` are always accessible from the root of the object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.\n\nUnknown data preserved in custom resources via x-kubernetes-preserve-unknown-fields is not accessible in CEL expressions. This includes: - Unknown field values that are preserved by object schemas with x-kubernetes-preserve-unknown-fields. - Object properties where the property schema is of an \"unknown type\". An \"unknown type\" is recursively defined as:\n - A schema with no type and x-kubernetes-preserve-unknown-fields set to true\n - An array where the items schema is of an \"unknown type\"\n - An object where the additionalProperties schema is of an \"unknown type\"\n\nOnly property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible. Accessible property names are escaped according to the following rules when accessed in the expression: - '__' escapes to '__underscores__' - '.' escapes to '__dot__' - '-' escapes to '__dash__' - '/' escapes to '__slash__' - Property names that exactly match a CEL RESERVED keyword escape to '__{keyword}__'. The keywords are:\n\t \"true\", \"false\", \"null\", \"in\", \"as\", \"break\", \"const\", \"continue\", \"else\", \"for\", \"function\", \"if\",\n\t \"import\", \"let\", \"loop\", \"package\", \"namespace\", \"return\".\nExamples:\n - Rule accessing a property named \"namespace\": {\"rule\": \"self.__namespace__ > 0\"}\n - Rule accessing a property named \"x-prop\": {\"rule\": \"self.x__dash__prop > 0\"}\n - Rule accessing a property named \"redact__d\": {\"rule\": \"self.redact__underscores__d > 0\"}\n\nEquality on arrays with x-kubernetes-list-type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:\n - 'set': `X + Y` performs a union where the array positions of all elements in `X` are preserved and\n non-intersecting elements in `Y` are appended, retaining their partial order.\n - 'map': `X + Y` performs a merge where the array positions of all keys in `X` are preserved but the values\n are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with\n non-intersecting keys are appended, retaining their partial order.", + "type": "string" + } + }, + "required": [ + "rule" + ], + "type": "object" + }, "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.WebhookClientConfig": { "description": "WebhookClientConfig contains the information to make a TLS connection with the webhook.", "properties": { @@ -15534,6 +16079,11 @@ "kind": "DeleteOptions", "version": "v1" }, + { + "group": "autoscaling", + "kind": "DeleteOptions", + "version": "v2" + }, { "group": "autoscaling", "kind": "DeleteOptions", @@ -15609,6 +16159,11 @@ "kind": "DeleteOptions", "version": "v1beta1" }, + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "DeleteOptions", + "version": "v1beta2" + }, { "group": "imagepolicy.k8s.io", "kind": "DeleteOptions", @@ -15788,7 +16343,7 @@ "type": "string" }, "selfLink": { - "description": "selfLink is a URL representing this object. Populated by the system. Read-only.\n\nDEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release.", + "description": "Deprecated: selfLink is a legacy read-only field that is no longer populated by the system.", "type": "string" } }, @@ -15823,7 +16378,7 @@ }, "time": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", - "description": "Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'" + "description": "Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over." } }, "type": "object" @@ -15844,7 +16399,7 @@ "type": "object" }, "clusterName": { - "description": "The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request.", + "description": "Deprecated: ClusterName is a legacy field that was always cleared by the system and never used; it will be removed completely in 1.25.\n\nThe name in the go struct is changed to help clients detect accidental use.", "type": "string" }, "creationTimestamp": { @@ -15869,7 +16424,7 @@ "x-kubernetes-patch-strategy": "merge" }, "generateName": { - "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.\n\nIf this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).\n\nApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", + "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.\n\nIf this field is specified and the generated name exists, the server will return a 409.\n\nApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", "type": "string" }, "generation": { @@ -15913,7 +16468,7 @@ "type": "string" }, "selfLink": { - "description": "SelfLink is a URL representing this object. Populated by the system. Read-only.\n\nDEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release.", + "description": "Deprecated: selfLink is a legacy read-only field that is no longer populated by the system.", "type": "string" }, "uid": { @@ -15931,7 +16486,7 @@ "type": "string" }, "blockOwnerDeletion": { - "description": "If true, AND if the owner has the \"foregroundDeletion\" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs \"delete\" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.", + "description": "If true, AND if the owner has the \"foregroundDeletion\" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs \"delete\" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.", "type": "boolean" }, "controller": { @@ -16201,6 +16756,11 @@ "kind": "WatchEvent", "version": "v1" }, + { + "group": "autoscaling", + "kind": "WatchEvent", + "version": "v2" + }, { "group": "autoscaling", "kind": "WatchEvent", @@ -16276,6 +16836,11 @@ "kind": "WatchEvent", "version": "v1beta1" }, + { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "WatchEvent", + "version": "v1beta2" + }, { "group": "imagepolicy.k8s.io", "kind": "WatchEvent", @@ -17446,6 +18011,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -17506,6 +18078,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "object name and auth scope, such as for teams and projects", "in": "path", @@ -17849,6 +18428,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -18064,6 +18650,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -18135,6 +18728,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -18442,6 +19042,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -18657,6 +19264,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -18728,6 +19342,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -19035,6 +19656,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -19250,6 +19878,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -19321,6 +19956,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -19628,6 +20270,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -19843,6 +20492,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -19914,6 +20570,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -20221,6 +20884,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -20436,6 +21106,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -20507,6 +21184,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -20638,6 +21322,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -20709,6 +21400,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -21016,6 +21714,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -21231,6 +21936,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -21302,6 +22014,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -21477,6 +22196,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "name of the Binding", "in": "path", @@ -21652,6 +22378,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -21723,6 +22456,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -21777,6 +22517,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "name of the Eviction", "in": "path", @@ -21925,7 +22672,7 @@ "uniqueItems": true }, { - "description": "Redirect the standard error stream of the pod for this call. Defaults to true.", + "description": "Redirect the standard error stream of the pod for this call.", "in": "query", "name": "stderr", "type": "boolean", @@ -21939,7 +22686,7 @@ "uniqueItems": true }, { - "description": "Redirect the standard output stream of the pod for this call. Defaults to true.", + "description": "Redirect the standard output stream of the pod for this call.", "in": "query", "name": "stdout", "type": "boolean", @@ -22816,6 +23563,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -22887,6 +23641,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -23194,6 +23955,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -23409,6 +24177,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -23480,6 +24255,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -23787,6 +24569,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -24002,6 +24791,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -24073,6 +24869,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -24204,6 +25007,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -24275,6 +25085,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -24406,6 +25223,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -24477,6 +25301,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -24784,6 +25615,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -24999,6 +25837,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -25070,6 +25915,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -25201,6 +26053,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -25272,6 +26131,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -25579,6 +26445,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -25794,6 +26667,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -25865,6 +26745,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -26172,6 +27059,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -26387,6 +27281,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -26458,6 +27359,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -26512,6 +27420,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "name of the TokenRequest", "in": "path", @@ -26595,6 +27510,127 @@ } }, "/api/v1/namespaces/{namespace}/services": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete collection of Service", + "operationId": "deleteCoreV1CollectionNamespacedService", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "core_v1" + ], + "x-kubernetes-action": "deletecollection", + "x-kubernetes-group-version-kind": { + "group": "", + "kind": "Service", + "version": "v1" + } + }, "get": { "consumes": [ "*/*" @@ -26742,6 +27778,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -26839,13 +27882,13 @@ "200": { "description": "OK", "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + "$ref": "#/definitions/io.k8s.api.core.v1.Service" } }, "202": { "description": "Accepted", "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + "$ref": "#/definitions/io.k8s.api.core.v1.Service" } }, "401": { @@ -26957,6 +28000,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -27028,6 +28078,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -27683,6 +28740,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -27754,6 +28818,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -27955,6 +29026,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -28026,6 +29104,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -28080,6 +29165,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "name of the Namespace", "in": "path", @@ -28233,6 +29325,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -28304,6 +29403,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -28603,6 +29709,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -28810,6 +29923,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -28881,6 +30001,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -29512,6 +30639,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -29583,6 +30717,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -29993,6 +31134,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -30200,6 +31348,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -30271,6 +31426,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -30394,6 +31556,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -30465,6 +31634,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -36614,6 +37790,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -36821,6 +38004,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -36892,6 +38082,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -37191,6 +38388,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -37398,6 +38602,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -37469,6 +38680,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -38294,6 +39512,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -38501,6 +39726,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -38572,6 +39804,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -38695,6 +39934,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -38766,6 +40012,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -39361,6 +40614,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -39568,6 +40828,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -39639,6 +40906,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -39762,6 +41036,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -39833,6 +41114,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -40769,6 +42057,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -40984,6 +42279,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -41055,6 +42357,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -41362,6 +42671,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -41577,6 +42893,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -41648,6 +42971,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -41779,6 +43109,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -41850,6 +43187,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -42157,6 +43501,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -42372,6 +43723,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -42443,6 +43801,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -42574,6 +43939,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -42645,6 +44017,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -42776,6 +44155,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -42847,6 +44233,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -43154,6 +44547,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -43369,6 +44769,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -43440,6 +44847,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -43571,6 +44985,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -43642,6 +45063,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -43773,6 +45201,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -43844,6 +45279,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -44151,6 +45593,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -44366,6 +45815,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -44437,6 +45893,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -44568,6 +46031,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -44639,6 +46109,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -44770,6 +46247,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -44841,6 +46325,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -46968,6 +48459,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "If 'true', then the output is pretty printed.", "in": "query", @@ -47116,6 +48614,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "object name and auth scope, such as for teams and projects", "in": "path", @@ -47206,6 +48711,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "If 'true', then the output is pretty printed.", "in": "query", @@ -47288,6 +48800,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "If 'true', then the output is pretty printed.", "in": "query", @@ -47370,6 +48889,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "If 'true', then the output is pretty printed.", "in": "query", @@ -47882,6 +49408,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -48097,6 +49630,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -48168,6 +49708,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -48299,6 +49846,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -48370,6 +49924,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -48765,6 +50326,1337 @@ } ] }, + "/apis/autoscaling/v2/": { + "get": { + "consumes": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "description": "get available resources", + "operationId": "getAutoscalingV2APIResources", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ] + } + }, + "/apis/autoscaling/v2/horizontalpodautoscalers": { + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind HorizontalPodAutoscaler", + "operationId": "listAutoscalingV2HorizontalPodAutoscalerForAllNamespaces", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete collection of HorizontalPodAutoscaler", + "operationId": "deleteAutoscalingV2CollectionNamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "deletecollection", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind HorizontalPodAutoscaler", + "operationId": "listAutoscalingV2NamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "post": { + "consumes": [ + "*/*" + ], + "description": "create a HorizontalPodAutoscaler", + "operationId": "createAutoscalingV2NamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "post", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + } + }, + "/apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete a HorizontalPodAutoscaler", + "operationId": "deleteAutoscalingV2NamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "delete", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "read the specified HorizontalPodAutoscaler", + "operationId": "readAutoscalingV2NamespacedHorizontalPodAutoscaler", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "name of the HorizontalPodAutoscaler", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update the specified HorizontalPodAutoscaler", + "operationId": "patchAutoscalingV2NamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace the specified HorizontalPodAutoscaler", + "operationId": "replaceAutoscalingV2NamespacedHorizontalPodAutoscaler", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + } + }, + "/apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status": { + "get": { + "consumes": [ + "*/*" + ], + "description": "read status of the specified HorizontalPodAutoscaler", + "operationId": "readAutoscalingV2NamespacedHorizontalPodAutoscalerStatus", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "name of the HorizontalPodAutoscaler", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update status of the specified HorizontalPodAutoscaler", + "operationId": "patchAutoscalingV2NamespacedHorizontalPodAutoscalerStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace status of the specified HorizontalPodAutoscaler", + "operationId": "replaceAutoscalingV2NamespacedHorizontalPodAutoscalerStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.autoscaling.v2.HorizontalPodAutoscaler" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + } + }, + "/apis/autoscaling/v2/watch/horizontalpodautoscalers": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of HorizontalPodAutoscaler. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchAutoscalingV2HorizontalPodAutoscalerListForAllNamespaces", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/autoscaling/v2/watch/namespaces/{namespace}/horizontalpodautoscalers": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of HorizontalPodAutoscaler. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchAutoscalingV2NamespacedHorizontalPodAutoscalerList", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/autoscaling/v2/watch/namespaces/{namespace}/horizontalpodautoscalers/{name}": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch changes to an object of kind HorizontalPodAutoscaler. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", + "operationId": "watchAutoscalingV2NamespacedHorizontalPodAutoscaler", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "autoscaling_v2" + ], + "x-kubernetes-action": "watch", + "x-kubernetes-group-version-kind": { + "group": "autoscaling", + "kind": "HorizontalPodAutoscaler", + "version": "v2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "name of the HorizontalPodAutoscaler", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, "/apis/autoscaling/v2beta1/": { "get": { "consumes": [ @@ -49178,6 +52070,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -49393,6 +52292,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -49464,6 +52370,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -49595,6 +52508,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -49666,6 +52586,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -50474,6 +53401,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -50689,6 +53623,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -50760,6 +53701,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -50891,6 +53839,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -50962,6 +53917,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -51914,6 +54876,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -52129,6 +55098,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -52200,6 +55176,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -52331,6 +55314,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -52402,6 +55392,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -52709,6 +55706,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -52924,6 +55928,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -52995,6 +56006,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -53126,6 +56144,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -53197,6 +56222,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -54362,6 +57394,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -54577,6 +57616,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -54648,6 +57694,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -54779,6 +57832,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -54850,6 +57910,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -55572,6 +58639,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -55779,6 +58853,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -55850,6 +58931,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -55973,6 +59061,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -56044,6 +59139,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -56167,6 +59269,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -56238,6 +59347,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -56952,6 +60068,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -57167,6 +60290,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -57238,6 +60368,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -58079,6 +61216,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -58294,6 +61438,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -58365,6 +61516,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -59173,6 +62331,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -59388,6 +62553,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -59459,6 +62631,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -60300,6 +63479,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -60515,6 +63701,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -60586,6 +63779,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -61394,6 +64594,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -61609,6 +64816,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -61680,6 +64894,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -62402,6 +65623,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -62609,6 +65837,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -62680,6 +65915,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -62803,6 +66045,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -62874,6 +66123,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -63173,6 +66429,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -63380,6 +66643,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -63451,6 +66721,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -63574,6 +66851,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -63645,6 +66929,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -64143,6 +67434,2111 @@ } ] }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/": { + "get": { + "consumes": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "description": "get available resources", + "operationId": "getFlowcontrolApiserverV1beta2APIResources", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ] + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete collection of FlowSchema", + "operationId": "deleteFlowcontrolApiserverV1beta2CollectionFlowSchema", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "deletecollection", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind FlowSchema", + "operationId": "listFlowcontrolApiserverV1beta2FlowSchema", + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchemaList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "post": { + "consumes": [ + "*/*" + ], + "description": "create a FlowSchema", + "operationId": "createFlowcontrolApiserverV1beta2FlowSchema", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "post", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete a FlowSchema", + "operationId": "deleteFlowcontrolApiserverV1beta2FlowSchema", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "delete", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "read the specified FlowSchema", + "operationId": "readFlowcontrolApiserverV1beta2FlowSchema", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "name of the FlowSchema", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update the specified FlowSchema", + "operationId": "patchFlowcontrolApiserverV1beta2FlowSchema", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace the specified FlowSchema", + "operationId": "replaceFlowcontrolApiserverV1beta2FlowSchema", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}/status": { + "get": { + "consumes": [ + "*/*" + ], + "description": "read status of the specified FlowSchema", + "operationId": "readFlowcontrolApiserverV1beta2FlowSchemaStatus", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "name of the FlowSchema", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update status of the specified FlowSchema", + "operationId": "patchFlowcontrolApiserverV1beta2FlowSchemaStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace status of the specified FlowSchema", + "operationId": "replaceFlowcontrolApiserverV1beta2FlowSchemaStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.FlowSchema" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete collection of PriorityLevelConfiguration", + "operationId": "deleteFlowcontrolApiserverV1beta2CollectionPriorityLevelConfiguration", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "deletecollection", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind PriorityLevelConfiguration", + "operationId": "listFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfigurationList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "post": { + "consumes": [ + "*/*" + ], + "description": "create a PriorityLevelConfiguration", + "operationId": "createFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "post", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete a PriorityLevelConfiguration", + "operationId": "deleteFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "delete", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "read the specified PriorityLevelConfiguration", + "operationId": "readFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "name of the PriorityLevelConfiguration", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update the specified PriorityLevelConfiguration", + "operationId": "patchFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace the specified PriorityLevelConfiguration", + "operationId": "replaceFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}/status": { + "get": { + "consumes": [ + "*/*" + ], + "description": "read status of the specified PriorityLevelConfiguration", + "operationId": "readFlowcontrolApiserverV1beta2PriorityLevelConfigurationStatus", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "name of the PriorityLevelConfiguration", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update status of the specified PriorityLevelConfiguration", + "operationId": "patchFlowcontrolApiserverV1beta2PriorityLevelConfigurationStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace status of the specified PriorityLevelConfiguration", + "operationId": "replaceFlowcontrolApiserverV1beta2PriorityLevelConfigurationStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.flowcontrol.v1beta2.PriorityLevelConfiguration" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + } + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/flowschemas": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of FlowSchema. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchFlowcontrolApiserverV1beta2FlowSchemaList", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/flowschemas/{name}": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch changes to an object of kind FlowSchema. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", + "operationId": "watchFlowcontrolApiserverV1beta2FlowSchema", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "watch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "FlowSchema", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "name of the FlowSchema", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/prioritylevelconfigurations": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of PriorityLevelConfiguration. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchFlowcontrolApiserverV1beta2PriorityLevelConfigurationList", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/prioritylevelconfigurations/{name}": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch changes to an object of kind PriorityLevelConfiguration. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", + "operationId": "watchFlowcontrolApiserverV1beta2PriorityLevelConfiguration", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "flowcontrolApiserver_v1beta2" + ], + "x-kubernetes-action": "watch", + "x-kubernetes-group-version-kind": { + "group": "flowcontrol.apiserver.k8s.io", + "kind": "PriorityLevelConfiguration", + "version": "v1beta2" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "name of the PriorityLevelConfiguration", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, "/apis/internal.apiserver.k8s.io/": { "get": { "consumes": [ @@ -64470,6 +69866,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -64677,6 +70080,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -64748,6 +70158,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -64871,6 +70288,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -64942,6 +70366,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -65537,6 +70968,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -65744,6 +71182,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -65815,6 +71260,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -66233,6 +71685,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -66448,6 +71907,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -66519,6 +71985,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -66650,6 +72123,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -66721,6 +72201,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -67028,6 +72515,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -67243,6 +72737,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -67314,6 +72815,229 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "networking_v1" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "networking.k8s.io", + "kind": "NetworkPolicy", + "version": "v1" + } + } + }, + "/apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status": { + "get": { + "consumes": [ + "*/*" + ], + "description": "read status of the specified NetworkPolicy", + "operationId": "readNetworkingV1NamespacedNetworkPolicyStatus", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "networking_v1" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "networking.k8s.io", + "kind": "NetworkPolicy", + "version": "v1" + } + }, + "parameters": [ + { + "description": "name of the NetworkPolicy", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update status of the specified NetworkPolicy", + "operationId": "patchNetworkingV1NamespacedNetworkPolicyStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "networking_v1" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "networking.k8s.io", + "kind": "NetworkPolicy", + "version": "v1" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace status of the specified NetworkPolicy", + "operationId": "replaceNetworkingV1NamespacedNetworkPolicyStatus", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.networking.v1.NetworkPolicy" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -68734,6 +74458,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -68941,6 +74672,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -69012,6 +74750,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -69280,846 +75025,6 @@ } ] }, - "/apis/node.k8s.io/v1alpha1/": { - "get": { - "consumes": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "description": "get available resources", - "operationId": "getNodeV1alpha1APIResources", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ] - } - }, - "/apis/node.k8s.io/v1alpha1/runtimeclasses": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of RuntimeClass", - "operationId": "deleteNodeV1alpha1CollectionRuntimeClass", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind RuntimeClass", - "operationId": "listNodeV1alpha1RuntimeClass", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClassList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a RuntimeClass", - "operationId": "createNodeV1alpha1RuntimeClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - } - }, - "/apis/node.k8s.io/v1alpha1/runtimeclasses/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a RuntimeClass", - "operationId": "deleteNodeV1alpha1RuntimeClass", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified RuntimeClass", - "operationId": "readNodeV1alpha1RuntimeClass", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the RuntimeClass", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified RuntimeClass", - "operationId": "patchNodeV1alpha1RuntimeClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified RuntimeClass", - "operationId": "replaceNodeV1alpha1RuntimeClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.node.v1alpha1.RuntimeClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - } - }, - "/apis/node.k8s.io/v1alpha1/watch/runtimeclasses": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of RuntimeClass. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchNodeV1alpha1RuntimeClassList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/node.k8s.io/v1alpha1/watch/runtimeclasses/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind RuntimeClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchNodeV1alpha1RuntimeClass", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "node_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "node.k8s.io", - "kind": "RuntimeClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the RuntimeClass", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, "/apis/node.k8s.io/v1beta1/": { "get": { "consumes": [ @@ -70414,6 +75319,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -70621,6 +75533,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -70692,6 +75611,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -71295,6 +76221,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -71510,6 +76443,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -71581,6 +76521,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -71712,6 +76659,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -71783,6 +76737,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -72591,6 +77552,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -72806,6 +77774,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -72877,6 +77852,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -73008,6 +77990,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -73079,6 +78068,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -73489,6 +78485,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -73696,6 +78699,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -73767,6 +78777,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -74719,6 +79736,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -74926,6 +79950,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -74997,6 +80028,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -75296,6 +80334,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -75503,6 +80548,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -75574,6 +80626,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -75881,6 +80940,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -76096,6 +81162,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -76167,6 +81240,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -76474,6 +81554,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -76689,6 +81776,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -76760,6 +81854,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -78194,3775 +83295,6 @@ } ] }, - "/apis/rbac.authorization.k8s.io/v1alpha1/": { - "get": { - "consumes": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "description": "get available resources", - "operationId": "getRbacAuthorizationV1alpha1APIResources", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ] - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of ClusterRoleBinding", - "operationId": "deleteRbacAuthorizationV1alpha1CollectionClusterRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind ClusterRoleBinding", - "operationId": "listRbacAuthorizationV1alpha1ClusterRoleBinding", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBindingList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a ClusterRoleBinding", - "operationId": "createRbacAuthorizationV1alpha1ClusterRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a ClusterRoleBinding", - "operationId": "deleteRbacAuthorizationV1alpha1ClusterRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified ClusterRoleBinding", - "operationId": "readRbacAuthorizationV1alpha1ClusterRoleBinding", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the ClusterRoleBinding", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified ClusterRoleBinding", - "operationId": "patchRbacAuthorizationV1alpha1ClusterRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified ClusterRoleBinding", - "operationId": "replaceRbacAuthorizationV1alpha1ClusterRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/clusterroles": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of ClusterRole", - "operationId": "deleteRbacAuthorizationV1alpha1CollectionClusterRole", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind ClusterRole", - "operationId": "listRbacAuthorizationV1alpha1ClusterRole", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRoleList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a ClusterRole", - "operationId": "createRbacAuthorizationV1alpha1ClusterRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/clusterroles/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a ClusterRole", - "operationId": "deleteRbacAuthorizationV1alpha1ClusterRole", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified ClusterRole", - "operationId": "readRbacAuthorizationV1alpha1ClusterRole", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the ClusterRole", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified ClusterRole", - "operationId": "patchRbacAuthorizationV1alpha1ClusterRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified ClusterRole", - "operationId": "replaceRbacAuthorizationV1alpha1ClusterRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.ClusterRole" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of RoleBinding", - "operationId": "deleteRbacAuthorizationV1alpha1CollectionNamespacedRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind RoleBinding", - "operationId": "listRbacAuthorizationV1alpha1NamespacedRoleBinding", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBindingList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a RoleBinding", - "operationId": "createRbacAuthorizationV1alpha1NamespacedRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a RoleBinding", - "operationId": "deleteRbacAuthorizationV1alpha1NamespacedRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified RoleBinding", - "operationId": "readRbacAuthorizationV1alpha1NamespacedRoleBinding", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the RoleBinding", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified RoleBinding", - "operationId": "patchRbacAuthorizationV1alpha1NamespacedRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified RoleBinding", - "operationId": "replaceRbacAuthorizationV1alpha1NamespacedRoleBinding", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBinding" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of Role", - "operationId": "deleteRbacAuthorizationV1alpha1CollectionNamespacedRole", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind Role", - "operationId": "listRbacAuthorizationV1alpha1NamespacedRole", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a Role", - "operationId": "createRbacAuthorizationV1alpha1NamespacedRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a Role", - "operationId": "deleteRbacAuthorizationV1alpha1NamespacedRole", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified Role", - "operationId": "readRbacAuthorizationV1alpha1NamespacedRole", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the Role", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified Role", - "operationId": "patchRbacAuthorizationV1alpha1NamespacedRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified Role", - "operationId": "replaceRbacAuthorizationV1alpha1NamespacedRole", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.Role" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - } - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/rolebindings": { - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind RoleBinding", - "operationId": "listRbacAuthorizationV1alpha1RoleBindingForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleBindingList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/roles": { - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind Role", - "operationId": "listRbacAuthorizationV1alpha1RoleForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.rbac.v1alpha1.RoleList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterrolebindings": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of ClusterRoleBinding. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1ClusterRoleBindingList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterrolebindings/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind ClusterRoleBinding. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchRbacAuthorizationV1alpha1ClusterRoleBinding", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the ClusterRoleBinding", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterroles": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of ClusterRole. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1ClusterRoleList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterroles/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind ClusterRole. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchRbacAuthorizationV1alpha1ClusterRole", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "ClusterRole", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the ClusterRole", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/rolebindings": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of RoleBinding. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1NamespacedRoleBindingList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/rolebindings/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind RoleBinding. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchRbacAuthorizationV1alpha1NamespacedRoleBinding", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the RoleBinding", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/roles": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of Role. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1NamespacedRoleList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/roles/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind Role. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchRbacAuthorizationV1alpha1NamespacedRole", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the Role", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/rolebindings": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of RoleBinding. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1RoleBindingListForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "RoleBinding", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/rbac.authorization.k8s.io/v1alpha1/watch/roles": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of Role. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchRbacAuthorizationV1alpha1RoleListForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "rbacAuthorization_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "rbac.authorization.k8s.io", - "kind": "Role", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, "/apis/scheduling.k8s.io/": { "get": { "consumes": [ @@ -82290,6 +83622,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -82497,6 +83836,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -82568,6 +83914,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -82836,846 +84189,6 @@ } ] }, - "/apis/scheduling.k8s.io/v1alpha1/": { - "get": { - "consumes": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "description": "get available resources", - "operationId": "getSchedulingV1alpha1APIResources", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ] - } - }, - "/apis/scheduling.k8s.io/v1alpha1/priorityclasses": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of PriorityClass", - "operationId": "deleteSchedulingV1alpha1CollectionPriorityClass", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind PriorityClass", - "operationId": "listSchedulingV1alpha1PriorityClass", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClassList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a PriorityClass", - "operationId": "createSchedulingV1alpha1PriorityClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - } - }, - "/apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a PriorityClass", - "operationId": "deleteSchedulingV1alpha1PriorityClass", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified PriorityClass", - "operationId": "readSchedulingV1alpha1PriorityClass", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the PriorityClass", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified PriorityClass", - "operationId": "patchSchedulingV1alpha1PriorityClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified PriorityClass", - "operationId": "replaceSchedulingV1alpha1PriorityClass", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.scheduling.v1alpha1.PriorityClass" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - } - }, - "/apis/scheduling.k8s.io/v1alpha1/watch/priorityclasses": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of PriorityClass. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchSchedulingV1alpha1PriorityClassList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/scheduling.k8s.io/v1alpha1/watch/priorityclasses/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind PriorityClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchSchedulingV1alpha1PriorityClass", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "scheduling_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "scheduling.k8s.io", - "kind": "PriorityClass", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the PriorityClass", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, "/apis/storage.k8s.io/": { "get": { "consumes": [ @@ -84003,6 +84516,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -84210,6 +84730,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -84281,6 +84808,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -84580,6 +85114,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -84787,6 +85328,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -84858,6 +85406,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -84896,6 +85451,731 @@ } } }, + "/apis/storage.k8s.io/v1/csistoragecapacities": { + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind CSIStorageCapacity", + "operationId": "listStorageV1CSIStorageCapacityForAllNamespaces", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacityList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete collection of CSIStorageCapacity", + "operationId": "deleteStorageV1CollectionNamespacedCSIStorageCapacity", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "deletecollection", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "list or watch objects of kind CSIStorageCapacity", + "operationId": "listStorageV1NamespacedCSIStorageCapacity", + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacityList" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "list", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "post": { + "consumes": [ + "*/*" + ], + "description": "create a CSIStorageCapacity", + "operationId": "createStorageV1NamespacedCSIStorageCapacity", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "post", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + } + }, + "/apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities/{name}": { + "delete": { + "consumes": [ + "*/*" + ], + "description": "delete a CSIStorageCapacity", + "operationId": "deleteStorageV1NamespacedCSIStorageCapacity", + "parameters": [ + { + "in": "body", + "name": "body", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", + "in": "query", + "name": "gracePeriodSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", + "in": "query", + "name": "orphanDependents", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", + "in": "query", + "name": "propagationPolicy", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "202": { + "description": "Accepted", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "delete", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "get": { + "consumes": [ + "*/*" + ], + "description": "read the specified CSIStorageCapacity", + "operationId": "readStorageV1NamespacedCSIStorageCapacity", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "get", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "name of the CSIStorageCapacity", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + } + ], + "patch": { + "consumes": [ + "application/json-patch+json", + "application/merge-patch+json", + "application/strategic-merge-patch+json", + "application/apply-patch+yaml" + ], + "description": "partially update the specified CSIStorageCapacity", + "operationId": "patchStorageV1NamespacedCSIStorageCapacity", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, + { + "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", + "in": "query", + "name": "force", + "type": "boolean", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "patch", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "put": { + "consumes": [ + "*/*" + ], + "description": "replace the specified CSIStorageCapacity", + "operationId": "replaceStorageV1NamespacedCSIStorageCapacity", + "parameters": [ + { + "in": "body", + "name": "body", + "required": true, + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + { + "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", + "in": "query", + "name": "dryRun", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", + "in": "query", + "name": "fieldManager", + "type": "string", + "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + } + ], + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "201": { + "description": "Created", + "schema": { + "$ref": "#/definitions/io.k8s.api.storage.v1.CSIStorageCapacity" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "put", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + } + }, "/apis/storage.k8s.io/v1/storageclasses": { "delete": { "consumes": [ @@ -85157,6 +86437,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -85364,6 +86651,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -85435,6 +86729,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -85734,6 +87035,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -85941,6 +87249,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -86012,6 +87327,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -86135,6 +87457,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -86206,6 +87535,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -86704,6 +88040,363 @@ } ] }, + "/apis/storage.k8s.io/v1/watch/csistoragecapacities": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchStorageV1CSIStorageCapacityListForAllNamespaces", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/storage.k8s.io/v1/watch/namespaces/{namespace}/csistoragecapacities": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch individual changes to a list of CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead.", + "operationId": "watchStorageV1NamespacedCSIStorageCapacityList", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "watchlist", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, + "/apis/storage.k8s.io/v1/watch/namespaces/{namespace}/csistoragecapacities/{name}": { + "get": { + "consumes": [ + "*/*" + ], + "description": "watch changes to an object of kind CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", + "operationId": "watchStorageV1NamespacedCSIStorageCapacity", + "produces": [ + "application/json", + "application/yaml", + "application/vnd.kubernetes.protobuf", + "application/json;stream=watch", + "application/vnd.kubernetes.protobuf;stream=watch" + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" + } + }, + "401": { + "description": "Unauthorized" + } + }, + "schemes": [ + "https" + ], + "tags": [ + "storage_v1" + ], + "x-kubernetes-action": "watch", + "x-kubernetes-group-version-kind": { + "group": "storage.k8s.io", + "kind": "CSIStorageCapacity", + "version": "v1" + } + }, + "parameters": [ + { + "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", + "in": "query", + "name": "allowWatchBookmarks", + "type": "boolean", + "uniqueItems": true + }, + { + "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", + "in": "query", + "name": "continue", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", + "in": "query", + "name": "fieldSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", + "in": "query", + "name": "labelSelector", + "type": "string", + "uniqueItems": true + }, + { + "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", + "in": "query", + "name": "limit", + "type": "integer", + "uniqueItems": true + }, + { + "description": "name of the CSIStorageCapacity", + "in": "path", + "name": "name", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "object name and auth scope, such as for teams and projects", + "in": "path", + "name": "namespace", + "required": true, + "type": "string", + "uniqueItems": true + }, + { + "description": "If 'true', then the output is pretty printed.", + "in": "query", + "name": "pretty", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersion", + "type": "string", + "uniqueItems": true + }, + { + "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", + "in": "query", + "name": "resourceVersionMatch", + "type": "string", + "uniqueItems": true + }, + { + "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", + "in": "query", + "name": "timeoutSeconds", + "type": "integer", + "uniqueItems": true + }, + { + "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", + "in": "query", + "name": "watch", + "type": "boolean", + "uniqueItems": true + } + ] + }, "/apis/storage.k8s.io/v1/watch/storageclasses": { "get": { "consumes": [ @@ -87164,1907 +88857,6 @@ } ] }, - "/apis/storage.k8s.io/v1alpha1/": { - "get": { - "consumes": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "description": "get available resources", - "operationId": "getStorageV1alpha1APIResources", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ] - } - }, - "/apis/storage.k8s.io/v1alpha1/csistoragecapacities": { - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind CSIStorageCapacity", - "operationId": "listStorageV1alpha1CSIStorageCapacityForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacityList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of CSIStorageCapacity", - "operationId": "deleteStorageV1alpha1CollectionNamespacedCSIStorageCapacity", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind CSIStorageCapacity", - "operationId": "listStorageV1alpha1NamespacedCSIStorageCapacity", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacityList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a CSIStorageCapacity", - "operationId": "createStorageV1alpha1NamespacedCSIStorageCapacity", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - } - }, - "/apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a CSIStorageCapacity", - "operationId": "deleteStorageV1alpha1NamespacedCSIStorageCapacity", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified CSIStorageCapacity", - "operationId": "readStorageV1alpha1NamespacedCSIStorageCapacity", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the CSIStorageCapacity", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified CSIStorageCapacity", - "operationId": "patchStorageV1alpha1NamespacedCSIStorageCapacity", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified CSIStorageCapacity", - "operationId": "replaceStorageV1alpha1NamespacedCSIStorageCapacity", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.CSIStorageCapacity" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - } - }, - "/apis/storage.k8s.io/v1alpha1/volumeattachments": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete collection of VolumeAttachment", - "operationId": "deleteStorageV1alpha1CollectionVolumeAttachment", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Status" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "deletecollection", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "list or watch objects of kind VolumeAttachment", - "operationId": "listStorageV1alpha1VolumeAttachment", - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachmentList" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "list", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "post": { - "consumes": [ - "*/*" - ], - "description": "create a VolumeAttachment", - "operationId": "createStorageV1alpha1VolumeAttachment", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "post", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - } - }, - "/apis/storage.k8s.io/v1alpha1/volumeattachments/{name}": { - "delete": { - "consumes": [ - "*/*" - ], - "description": "delete a VolumeAttachment", - "operationId": "deleteStorageV1alpha1VolumeAttachment", - "parameters": [ - { - "in": "body", - "name": "body", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.", - "in": "query", - "name": "gracePeriodSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the \"orphan\" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.", - "in": "query", - "name": "orphanDependents", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.", - "in": "query", - "name": "propagationPolicy", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "202": { - "description": "Accepted", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "delete", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "get": { - "consumes": [ - "*/*" - ], - "description": "read the specified VolumeAttachment", - "operationId": "readStorageV1alpha1VolumeAttachment", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "get", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "name of the VolumeAttachment", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - } - ], - "patch": { - "consumes": [ - "application/json-patch+json", - "application/merge-patch+json", - "application/strategic-merge-patch+json", - "application/apply-patch+yaml" - ], - "description": "partially update the specified VolumeAttachment", - "operationId": "patchStorageV1alpha1VolumeAttachment", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - }, - { - "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", - "in": "query", - "name": "force", - "type": "boolean", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "patch", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "put": { - "consumes": [ - "*/*" - ], - "description": "replace the specified VolumeAttachment", - "operationId": "replaceStorageV1alpha1VolumeAttachment", - "parameters": [ - { - "in": "body", - "name": "body", - "required": true, - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - { - "description": "When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed", - "in": "query", - "name": "dryRun", - "type": "string", - "uniqueItems": true - }, - { - "description": "fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.", - "in": "query", - "name": "fieldManager", - "type": "string", - "uniqueItems": true - } - ], - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "201": { - "description": "Created", - "schema": { - "$ref": "#/definitions/io.k8s.api.storage.v1alpha1.VolumeAttachment" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "put", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - } - }, - "/apis/storage.k8s.io/v1alpha1/watch/csistoragecapacities": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchStorageV1alpha1CSIStorageCapacityListForAllNamespaces", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/storage.k8s.io/v1alpha1/watch/namespaces/{namespace}/csistoragecapacities": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchStorageV1alpha1NamespacedCSIStorageCapacityList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/storage.k8s.io/v1alpha1/watch/namespaces/{namespace}/csistoragecapacities/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchStorageV1alpha1NamespacedCSIStorageCapacity", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "CSIStorageCapacity", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the CSIStorageCapacity", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "object name and auth scope, such as for teams and projects", - "in": "path", - "name": "namespace", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/storage.k8s.io/v1alpha1/watch/volumeattachments": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch individual changes to a list of VolumeAttachment. deprecated: use the 'watch' parameter with a list operation instead.", - "operationId": "watchStorageV1alpha1VolumeAttachmentList", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "watchlist", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, - "/apis/storage.k8s.io/v1alpha1/watch/volumeattachments/{name}": { - "get": { - "consumes": [ - "*/*" - ], - "description": "watch changes to an object of kind VolumeAttachment. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.", - "operationId": "watchStorageV1alpha1VolumeAttachment", - "produces": [ - "application/json", - "application/yaml", - "application/vnd.kubernetes.protobuf", - "application/json;stream=watch", - "application/vnd.kubernetes.protobuf;stream=watch" - ], - "responses": { - "200": { - "description": "OK", - "schema": { - "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.WatchEvent" - } - }, - "401": { - "description": "Unauthorized" - } - }, - "schemes": [ - "https" - ], - "tags": [ - "storage_v1alpha1" - ], - "x-kubernetes-action": "watch", - "x-kubernetes-group-version-kind": { - "group": "storage.k8s.io", - "kind": "VolumeAttachment", - "version": "v1alpha1" - } - }, - "parameters": [ - { - "description": "allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.", - "in": "query", - "name": "allowWatchBookmarks", - "type": "boolean", - "uniqueItems": true - }, - { - "description": "The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\".\n\nThis field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.", - "in": "query", - "name": "continue", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.", - "in": "query", - "name": "fieldSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "A selector to restrict the list of returned objects by their labels. Defaults to everything.", - "in": "query", - "name": "labelSelector", - "type": "string", - "uniqueItems": true - }, - { - "description": "limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.\n\nThe server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.", - "in": "query", - "name": "limit", - "type": "integer", - "uniqueItems": true - }, - { - "description": "name of the VolumeAttachment", - "in": "path", - "name": "name", - "required": true, - "type": "string", - "uniqueItems": true - }, - { - "description": "If 'true', then the output is pretty printed.", - "in": "query", - "name": "pretty", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersion", - "type": "string", - "uniqueItems": true - }, - { - "description": "resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.\n\nDefaults to unset", - "in": "query", - "name": "resourceVersionMatch", - "type": "string", - "uniqueItems": true - }, - { - "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", - "in": "query", - "name": "timeoutSeconds", - "type": "integer", - "uniqueItems": true - }, - { - "description": "Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.", - "in": "query", - "name": "watch", - "type": "boolean", - "uniqueItems": true - } - ] - }, "/apis/storage.k8s.io/v1beta1/": { "get": { "consumes": [ @@ -89478,6 +89270,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ @@ -89693,6 +89492,13 @@ "type": "string", "uniqueItems": true }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true + }, { "description": "Force is going to \"force\" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.", "in": "query", @@ -89764,6 +89570,13 @@ "name": "fieldManager", "type": "string", "uniqueItems": true + }, + { + "description": "fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.", + "in": "query", + "name": "fieldValidation", + "type": "string", + "uniqueItems": true } ], "produces": [ diff --git a/api-ref-assets/config/fields.yaml b/api-ref-assets/config/fields.yaml index 9f6e23f939..71cffef523 100644 --- a/api-ref-assets/config/fields.yaml +++ b/api-ref-assets/config/fields.yaml @@ -6,6 +6,7 @@ - initContainers - imagePullSecrets - enableServiceLinks + - os - name: Volumes fields: - volumes @@ -49,11 +50,9 @@ - securityContext - name: Beta level fields: + - ephemeralContainers - preemptionPolicy - overhead - - name: Alpha level - fields: - - ephemeralContainers - name: Deprecated fields: - serviceAccount @@ -154,6 +153,7 @@ - timeoutSeconds - failureThreshold - successThreshold + - grpc - definition: io.k8s.api.core.v1.SecurityContext field_categories: @@ -225,6 +225,9 @@ - stdin - stdinOnce - tty + - name: Security context + fields: + - securityContext - name: Not allowed fields: - ports @@ -232,7 +235,6 @@ - lifecycle - livenessProbe - readinessProbe - - securityContext - startupProbe - definition: io.k8s.api.core.v1.ReplicationControllerSpec @@ -313,6 +315,7 @@ - revisionHistoryLimit - volumeClaimTemplates - minReadySeconds + - persistentVolumeClaimRetentionPolicy - definition: io.k8s.api.apps.v1.StatefulSetUpdateStrategy field_categories: @@ -393,12 +396,16 @@ - completedIndexes - conditions - uncountedTerminatedPods + - name: Alpha level + fields: + - ready - definition: io.k8s.api.batch.v1.CronJobSpec field_categories: - fields: - jobTemplate - schedule + - timeZone - concurrencyPolicy - startingDeadlineSeconds - suspend @@ -421,6 +428,22 @@ - value - periodSeconds +- definition: io.k8s.api.autoscaling.v2.HorizontalPodAutoscalerSpec + field_categories: + - fields: + - maxReplicas + - scaleTargetRef + - minReplicas + - behavior + - metrics + +- definition: io.k8s.api.autoscaling.v2.HPAScalingPolicy + field_categories: + - fields: + - type + - value + - periodSeconds + - definition: io.k8s.api.core.v1.ServiceSpec field_categories: - fields: diff --git a/api-ref-assets/config/toc.yaml b/api-ref-assets/config/toc.yaml index 5c4efa8982..78475b137f 100644 --- a/api-ref-assets/config/toc.yaml +++ b/api-ref-assets/config/toc.yaml @@ -23,7 +23,7 @@ parts: - PodSpec - Container - EphemeralContainer - - Handler + - LifecycleHandler - NodeAffinity - PodAffinity - PodAntiAffinity @@ -60,6 +60,9 @@ parts: - name: HorizontalPodAutoscaler group: autoscaling version: v1 + - name: HorizontalPodAutoscaler + group: autoscaling + version: v2 - name: HorizontalPodAutoscaler group: autoscaling version: v2beta2 @@ -121,7 +124,7 @@ parts: version: v1 - name: CSIStorageCapacity group: storage.k8s.io - version: v1beta1 + version: v1 - name: Authentication Resources chapters: - name: ServiceAccount @@ -217,10 +220,10 @@ parts: version: v1 - name: FlowSchema group: flowcontrol.apiserver.k8s.io - version: v1beta1 + version: v1beta2 - name: PriorityLevelConfiguration group: flowcontrol.apiserver.k8s.io - version: v1beta1 + version: v1beta2 - name: Binding group: "" version: v1 diff --git a/assets/scss/_base.scss b/assets/scss/_base.scss index 4113b49bee..97ce27fe28 100644 --- a/assets/scss/_base.scss +++ b/assets/scss/_base.scss @@ -810,11 +810,10 @@ section#cncf { } } -.td-search { - header > .header-filler { - height: $hero-padding-top; - background-color: black; - } +// Header filler size adjustment + +.header-hero.filler { + height: $hero-padding-top; } // Docs specific @@ -859,17 +858,6 @@ section#cncf { /* DOCUMENTATION */ -body.td-documentation { - header > .header-filler { - height: $hero-padding-top; - background-color: black; - } - /* Special case for if an announcement is active */ - header section#announcement ~ .header-filler { - display: none; - } -} - // nav-tabs and tab-content .nav-tabs { border-bottom: none !important; diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index e7f0902346..db0263d991 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -7,7 +7,7 @@ $announcement-size-adjustment: 8px; } main { - img { + *:not(figure) > img { max-width: 100%; } @@ -26,6 +26,10 @@ $announcement-size-adjustment: 8px; } } +.header-hero #quickstartButton.button { + margin-top: 1em; +} + section { .main-section { @media only screen and (min-width: 1024px) { @@ -34,8 +38,11 @@ section { } } -.td-outer { - padding: 0 !important; +body { + header + .td-outer { + min-height: 50vh; + height: auto; + } } @@ -215,8 +222,14 @@ footer { padding: 1rem !important; min-height: initial !important; - .footer__links { - width: 100%; + > div, > p { + max-width: 95%; + @media only screen and (min-width: 768px) { + max-width: calc(min(80rem,90vw)); // avoid spreading too wide + } + } + + > .footer__links { margin: auto; padding-bottom: 1rem; @@ -226,6 +239,8 @@ footer { } @media only screen and (min-width: 768px) { + max-width: calc(min(60rem,90vw)); // avoid spreading too wide + nav { display: flex; flex-direction: row; @@ -298,6 +313,12 @@ footer { padding-top: 1.5rem !important; top: 5rem !important; + @supports (position: sticky) { + position: sticky !important; + height: calc(100vh - 10rem); + overflow-y: auto; + } + #TableOfContents { padding-top: 1rem; } @@ -308,52 +329,102 @@ main { .td-content>table td { word-break: break-word; } + + table.no-word-break td, + table.no-word-break code { + word-break: normal; + } } // blockquotes and callouts -.td-content, body { - blockquote.callout { +body { + .alert { + // Override Docsy styles padding: 0.4rem 0.4rem 0.4rem 1rem; - border: 1px solid #eee; - border-left-width: 0.5em; + border-top: 1px solid #eee; + border-bottom: 1px solid #eee; + border-right: 1px solid #eee; + border-radius: 0.25em; + border-left-width: 0.5em; // fallback in case calc() is missing background: #fff; color: #000; margin-top: 0.5em; margin-bottom: 0.5em; } - blockquote.callout { - border-radius: calc(1em/3); + // Set minimum width and radius for alert color + .alert { + border-left-width: calc(max(0.5em, 4px)); + border-top-left-radius: calc(max(0.5em, 4px)); + border-bottom-left-radius: calc(max(0.5em, 4px)); } - .callout.caution { + .alert.callout.caution { border-left-color: #f0ad4e; } - - .callout.note { + .alert.callout.note { border-left-color: #428bca; } - - .callout.warning { + .alert.callout.warning { border-left-color: #d9534f; } + .alert.third-party-content { + border-left-color: #444; + } - h1:first-of-type + blockquote.callout { + h1:first-of-type + .alert.callout { margin-top: 1.5em; } } -.deprecation-warning { - padding: 20px; - margin: 20px 0; +// Special color for third party content disclaimers +.alert.third-party-content { border-left-color: #222 }; + +// Highlight disclaimer when targeted as a fragment + +#third-party-content-disclaimer { + color: #000; + background: #f8f9fa; + transition: all 0.5s ease; +} + +@keyframes disclaimer-highlight { + from { background: #f8f922; color: #000; } + 50% { background: #f8f944; color: #000; } + to { background: #f8f9cb; color: #000; } +} + +#third-party-content-disclaimer:target { + color: #000; + animation: disclaimer-highlight 1.25s ease; + background: #f8f9cb; +} + +.deprecation-warning, .pageinfo.deprecation-warning { + padding: clamp(10px, 2vmin, 20px); + margin: clamp(10px, 1vh, 20px) 0; background-color: #faf5b6; color: #000; } +.deprecation-warning.outdated-blog, .pageinfo.deprecation-warning.outdated-blog { + background-color: $blue; + color: $white; +} + body.td-home .deprecation-warning, body.td-blog .deprecation-warning, body.td-documentation .deprecation-warning { border-radius: 3px; } +.deprecation-warning p:only-child { + margin-bottom: 0; +} + +.td-documentation .td-content > .highlight { + max-width: initial; + width: 100%; +} + body.td-home #deprecation-warning { max-width: 1000px; margin-top: 2.5rem; @@ -495,7 +566,8 @@ main.content { } } -/* COMMUNITY */ +/* COMMUNITY legacy styles */ +/* Leave these in place until localizations are caught up */ .newcommunitywrapper { .news { @@ -524,34 +596,6 @@ main.content { } } -/* ANNOUNCEMENTS */ -section#fp-announcement ~ .header-hero { - padding: $announcement-size-adjustment 0; - - > div { - margin-top: $announcement-size-adjustment; - margin-bottom: $announcement-size-adjustment; - } - - h1, h2, h3, h4, h5 { - margin: $announcement-size-adjustment 0; - } -} - -section#announcement ~ .header-hero { - padding: #{$announcement-size-adjustment / 2} 0; - - > div { - margin-top: #{$announcement-size-adjustment / 2}; - margin-bottom: #{$announcement-size-adjustment / 2}; - padding-bottom: #{$announcement-size-adjustment / 2}; - } - - h1, h2, h3, h4, h5 { - margin: #{$announcement-size-adjustment / 2} 0; - } -} - /* DOCUMENTATION */ /* Don't show lead text */ @@ -577,64 +621,141 @@ body.td-documentation { @media print { /* Do not print announcements */ - #announcement, section#announcement, #fp-announcement, section#fp-announcement { + #announcement { display: none; } } -#announcement, #fp-announcement { +#announcement { > * { color: inherit; - background: inherit; + background: transparent; } a { color: inherit; - border-bottom: 1px solid #fff; + text-decoration: underline; } a:hover { color: inherit; - border-bottom: none; + text-decoration: initial; } } -#announcement { - padding-top: 105px; - padding-bottom: 25px; -} - .header-hero { padding-top: 40px; } -/* Extra announcement height only for landscape viewports */ -@media (min-aspect-ratio: 8/9) { - #fp-announcement { - min-height: 25vh; - } -} +#announcement { + // default background is blue; overrides are possible + color: #fff; -#fp-announcement aside { - padding-top: 115px; - padding-bottom: 25px; -} - -.announcement { - .content { + .announcement-main { + margin-left: auto; + margin-right: auto; margin-bottom: 0px; + + // for padding-top see _size.scss + padding-bottom: calc(max(2em, 2rem)); + + max-width: calc(min(1200px - 8em, 80vw)); } - > p { - .gridPage #announcement .content p, - .announcement > h4, - .announcement > h3 { - color: #ffffff; + h1, h2, h3, h4, h5, h6, p * { + color: inherit; /* defaults to white */ + background: transparent; + + img.event-logo { + display: inline-block; + max-height: calc(min(80px, 8em)); + max-width: calc(min(240px, 33vw)); + float: right; } } } +#announcement + .header-hero { + padding-top: 2em; +} + +// Extra padding for anything except wide viewports +@media (min-width: 992px) { + #announcement aside { // more specific + .announcement-main { + padding-top: calc(max(8em, 8rem)); + } + } +} + +@media (max-width: 768px) { + #announcement { + padding-top: 4rem; + padding-bottom: 4rem; + .announcement-main, aside .announcement-main { + padding-top: calc(min(2rem,2em)); + } + } +} + +@media (max-width: 480px) { + #announcement { + padding-bottom: 0.5em; + } + #announcement aside { + h1, h2, h3, h4, h5, h6 { + img.event-logo { + margin-left: auto; + margin-right: auto; + margin-bottom: 0.75em; + display: block; + max-height: initial; + max-width: calc(min(calc(100vw - 2em), 240px)); + float: initial; + } + } + } +} + +#announcement + .header-hero.filler { + display: none; +} + +@media (min-width: 768px) { + #announcement + .header-hero { + display: none; + } +} + +figure { + > figcaption { + padding-top: 1em; + margin-bottom: 3em; + } +} + +// Clamp size for release logos +figure.release-logo { + > figcaption { + font-size: 1.8em; + } + > img { + max-width: 100%; + max-height: calc(max(40em,min(80vh,70em))); + height: auto; + width: auto; + } +} + + +// Match Docsy-imposed max width on text body +@media (min-width: 1200px) { + body.td-blog main .td-content > figure { + max-width: 80%; + } +} + .td-content { table code { background-color: inherit !important; @@ -642,3 +763,93 @@ body.td-documentation { font-size: inherit !important; } } + +/* Force size constraints on figures */ +figure { + &.diagram-small img { + max-height: clamp(20mm,12em,80vh); + margin-left: auto; + margin-right: auto; + display: block; + } + &.diagram-medium img { + max-height: clamp(25mm,20em,80vh); + margin-left: auto; + margin-right: auto; + display: block; + } + &.diagram-large img { + max-width: clamp(0vw, 95vw, 100%); + max-height: calc(80vh - 8rem); + } + + figure + noscript > *{ + max-width: calc(max(100%, 100vw)); + } +} + +@media only screen and (min-width: 768px) { + figure { + &.diagram-small, &.diagram-medium { + max-width: 80%; + } + &.diagram-large { + max-width: 100%; + width: 100%; + } + &.diagram-small img { + max-width: clamp(30rem, 45ch, 100mm); + } + &.diagram-medium img { + max-width: clamp(50rem, 20ch, 160mm); + } + &.diagram-large img { + max-width: clamp(25vw, 95vw, 100%); + max-height: calc(100vh - 10rem); + } + } + figure + noscript > * { + max-width: 80%; + } +} + +// Indent definition lists +dl { + padding-left: 1.5em; + + // Add vertical space before definitions + > *:not(dt) + dt, dt:first-child { + margin-top: 1.5em; + } +} + +.release-details { + padding-left: 2em; + + > :not(p) { + font-size: 1.125em; + } + + .release-inline-heading, .release-inline-value { + display: inline-block + } + + .release-inline-value { + padding-left: 0.25em; + } + + p { + margin-top: 1em; + margin-bottom: 1em; + } +} + +.no-js .mermaid { + display: none; +} + +div.alert > em.javascript-required { + display: inline-block; + min-height: 1.5em; + margin: calc(max(4em, ( 8vh + 4em ) / 2)) 0 0.25em 0; +} diff --git a/assets/scss/_reset.scss b/assets/scss/_reset.scss old mode 100755 new mode 100644 diff --git a/assets/scss/_size.scss b/assets/scss/_size.scss index 9f0b7f655f..14d56201c3 100644 --- a/assets/scss/_size.scss +++ b/assets/scss/_size.scss @@ -18,3 +18,11 @@ section, line-height: $vendor-strip-height; font-size: $vendor-strip-font-size; } + +#announcement { + min-height: $hero-padding-top; + + .announcement-main { + padding-top: calc(max(8em, 8rem, #{$hero-padding-top} / 3)); + } +} diff --git a/assets/scss/_skin.scss b/assets/scss/_skin.scss old mode 100755 new mode 100644 diff --git a/cloudbuild.yaml b/cloudbuild.yaml index 61b5adc5f4..542d58016c 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -8,18 +8,21 @@ options: substitution_option: ALLOW_LOOSE steps: # It's fine to bump the tag to a recent version, as needed - - name: "gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4" - entrypoint: make + - name: "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55" + entrypoint: 'bash' env: - DOCKER_CLI_EXPERIMENTAL=enabled - TAG=$_GIT_TAG - BASE_REF=$_PULL_BASE_REF args: - - container-image + - -c + - | + gcloud auth configure-docker \ + && make container-push substitutions: # _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and # can be used as a substitution _GIT_TAG: "12345" # _PULL_BASE_REF will contain the ref that was pushed to to trigger this build - - # a branch like 'master' or 'release-0.2', or a tag like 'v0.2'. - _PULL_BASE_REF: "master" + # a branch like 'main' or 'release-0.2', or a tag like 'v0.2'. + _PULL_BASE_REF: "main" diff --git a/config.toml b/config.toml index 517d083062..4b030df417 100644 --- a/config.toml +++ b/config.toml @@ -121,7 +121,8 @@ id = "UA-00000000-0" [params] copyright_k8s = "The Kubernetes Authors" -copyright_linux = "Copyright © 2020 The Linux Foundation ®." +copyright_linux = "Copyright © 2020 The Linux Foundation ®." + # privacy_policy = "https://policies.google.com/privacy" # First one is picked as the Twitter card image if not set on page. @@ -137,10 +138,10 @@ time_format_default = "January 02, 2006 at 3:04 PM PST" description = "Production-Grade Container Orchestration" showedit = true -latest = "v1.22" +latest = "v1.24" -fullversion = "v1.22.0" -version = "v1.22" +fullversion = "v1.24.0" +version = "v1.24" githubbranch = "main" docsbranch = "main" deprecated = false @@ -153,10 +154,10 @@ githubWebsiteRaw = "raw.githubusercontent.com/kubernetes/website" # GitHub repository link for editing a page and opening issues. github_repo = "https://github.com/kubernetes/website" -#Searching +# Searching k8s_search = true -#The following search parameters are specific to Docsy's implementation. Kubernetes implementes its own search-related partials and scripts. +# The following search parameters are specific to Docsy's implementation. Kubernetes implementes its own search-related partials and scripts. # Google Custom Search Engine ID. Remove or comment out to disable search. #gcs_engine_id = "011737558837375720776:fsdu1nryfng" @@ -177,51 +178,53 @@ js = [ ] [[params.versions]] -fullversion = "v1.22.0" -version = "v1.22" -githubbranch = "v1.22.0" +fullversion = "v1.24.0" +version = "v1.24" +githubbranch = "v1.24.0" docsbranch = "main" url = "https://kubernetes.io" [[params.versions]] -fullversion = "v1.21.4" +fullversion = "v1.23.6" +version = "v1.23" +githubbranch = "v1.23.6" +docsbranch = "release-1.23" +url = "https://v1-23.docs.kubernetes.io" + +[[params.versions]] +fullversion = "v1.22.9" +version = "v1.22" +githubbranch = "v1.22.9" +docsbranch = "release-1.22" +url = "https://v1-22.docs.kubernetes.io" + +[[params.versions]] +fullversion = "v1.21.12" version = "v1.21" -githubbranch = "v1.21.4" +githubbranch = "v1.21.12" docsbranch = "release-1.21" url = "https://v1-21.docs.kubernetes.io" [[params.versions]] -fullversion = "v1.20.10" +fullversion = "v1.20.15" version = "v1.20" -githubbranch = "v1.20.10" +githubbranch = "v1.20.15" docsbranch = "release-1.20" url = "https://v1-20.docs.kubernetes.io" -[[params.versions]] -fullversion = "v1.19.14" -version = "v1.19" -githubbranch = "v1.19.14" -docsbranch = "release-1.19" -url = "https://v1-19.docs.kubernetes.io" - -[[params.versions]] -fullversion = "v1.18.20" -version = "v1.18" -githubbranch = "v1.18.20" -docsbranch = "release-1.18" -url = "https://v1-18.docs.kubernetes.io" - # User interface configuration [params.ui] # Enable to show the side bar menu in its compact state. sidebar_menu_compact = false +# Show expand/collapse icon for sidebar sections. +sidebar_menu_foldable = true # https://github.com/gohugoio/hugo/issues/8918#issuecomment-903314696 sidebar_cache_limit = 1 -# Set to true to disable breadcrumb navigation. +# Set to true to disable breadcrumb navigation. breadcrumb_disable = false -# Set to true to hide the sidebar search box (the top nav search box will still be displayed if search is enabled) +# Set to true to hide the sidebar search box (the top nav search box will still be displayed if search is enabled) sidebar_search_disable = false -# Set to false if you don't want to display a logo (/assets/icons/logo.svg) in the top nav bar +# Set to false if you don't want to display a logo (/assets/icons/logo.svg) in the top nav bar navbar_logo = true # Set to true to disable the About link in the site footer footer_about_disable = false @@ -242,50 +245,50 @@ no = 'Sorry to hear that. Please + +--> + + + +
+

+

+
+

Bevorstehende Veranstaltungen

+ {{< upcoming-events >}} +
+
+ +
+
+
+

Globale Gemeinschaft

+Mit mehr als 150 Treffen auf der ganzen Welt, Tendenz steigend, solltest du deine lokalen Kube-Leute finden. Wenn keins in der Nähe ist, nimm die Sache in die Hand und gründe dein eigenes. +
+ +
+EIN MEETUP FINDEN +
+ + +
+
+ + + + +
+

+

+

Aktuelle Neuigkeiten

+ +
+
+ +
+ +
+



+
+ + diff --git a/content/de/docs/concepts/cluster-administration/_index.md b/content/de/docs/concepts/cluster-administration/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/cluster-administration/addons.md b/content/de/docs/concepts/cluster-administration/addons.md index abf15e453f..77f098198b 100644 --- a/content/de/docs/concepts/cluster-administration/addons.md +++ b/content/de/docs/concepts/cluster-administration/addons.md @@ -21,17 +21,17 @@ Die Add-Ons in den einzelnen Kategorien sind alphabetisch sortiert - Die Reihenf * [ACI](https://www.github.com/noironetworks/aci-containers) bietet Container-Networking und Network-Security mit Cisco ACI. * [Calico](https://docs.projectcalico.org/latest/introduction/) ist ein Networking- und Network-Policy-Provider. Calico unterstützt eine Reihe von Networking-Optionen, damit Du die richtige für deinen Use-Case wählen kannst. Dies beinhaltet Non-Overlaying and Overlaying-Networks mit oder ohne BGP. Calico nutzt die gleiche Engine um Network-Policies für Hosts, Pods und (falls Du Istio & Envoy benutzt) Anwendungen auf Service-Mesh-Ebene durchzusetzen. -* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) vereint Flannel und Calico um Networking- und Network-Policies bereitzustellen. +* [Canal](https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel) vereint Flannel und Calico um Networking- und Network-Policies bereitzustellen. * [Cilium](https://github.com/cilium/cilium) ist ein L3 Network- and Network-Policy-Plugin welches das transparent HTTP/API/L7-Policies durchsetzen kann. Sowohl Routing- als auch Overlay/Encapsulation-Modes werden uterstützt. Außerdem kann Cilium auf andere CNI-Plugins aufsetzen. -* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) ermöglicht das nahtlose Verbinden von Kubernetes mit einer Reihe an CNI-Plugins wie z.B. Calico, Canal, Flannel, Romana, oder Weave. -* [Contiv](http://contiv.github.io) bietet konfigurierbares Networking (Native L3 auf BGP, Overlay mit vxlan, Klassisches L2, Cisco-SDN/ACI) für verschiedene Anwendungszwecke und auch umfangreiches Policy-Framework. Das Contiv-Projekt ist vollständig [Open Source](http://github.com/contiv). Der [installer](http://github.com/contiv/install) bietet sowohl kubeadm als auch nicht-kubeadm basierte Installationen. +* [CNI-Genie](https://github.com/cni-genie/CNI-Genie) ermöglicht das nahtlose Verbinden von Kubernetes mit einer Reihe an CNI-Plugins wie z.B. Calico, Canal, Flannel, Romana, oder Weave. +* [Contiv](https://contivpp.io/) bietet konfigurierbares Networking (Native L3 auf BGP, Overlay mit vxlan, Klassisches L2, Cisco-SDN/ACI) für verschiedene Anwendungszwecke und auch umfangreiches Policy-Framework. Das Contiv-Projekt ist vollständig [Open Source](http://github.com/contiv). Der [installer](http://github.com/contiv/install) bietet sowohl kubeadm als auch nicht-kubeadm basierte Installationen. * [Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), basierend auf [Tungsten Fabric](https://tungsten.io), ist eine Open Source, multi-Cloud Netzwerkvirtualisierungs- und Policy-Management Plattform. Contrail und Tungsten Fabric sind mit Orechstratoren wie z.B. Kubernetes, OpenShift, OpenStack und Mesos integriert und bieten Isolationsmodi für Virtuelle Maschinen, Container (bzw. Pods) und Bare Metal workloads. * [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) ist ein Overlay-Network-Provider der mit Kubernetes genutzt werden kann. * [Knitter](https://github.com/ZTE/Knitter/) ist eine Network-Lösung die Mehrfach-Network in Kubernetes ermöglicht. -* [Multus](https://github.com/Intel-Corp/multus-cni) ist ein Multi-Plugin für Mehrfachnetzwerk-Unterstützung um alle CNI-Plugins (z.B. Calico, Cilium, Contiv, Flannel), zusätzlich zu SRIOV-, DPDK-, OVS-DPDK- und VPP-Basierten Workloads in Kubernetes zu unterstützen. -* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) bietet eine Integration zwischen VMware NSX-T und einem Orchestator wie z.B. Kubernetes. Außerdem bietet es eine Integration zwischen NSX-T und Containerbasierten CaaS/PaaS-Plattformen wie z.B. Pivotal Container Service (PKS) und OpenShift. +* [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) ist ein Multi-Plugin für Mehrfachnetzwerk-Unterstützung um alle CNI-Plugins (z.B. Calico, Cilium, Contiv, Flannel), zusätzlich zu SRIOV-, DPDK-, OVS-DPDK- und VPP-Basierten Workloads in Kubernetes zu unterstützen. +* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/index.html) Container Plug-in (NCP) bietet eine Integration zwischen VMware NSX-T und einem Orchestator wie z.B. Kubernetes. Außerdem bietet es eine Integration zwischen NSX-T und Containerbasierten CaaS/PaaS-Plattformen wie z.B. Pivotal Container Service (PKS) und OpenShift. * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) ist eine SDN-Plattform die Policy-Basiertes Networking zwischen Kubernetes Pods und nicht-Kubernetes Umgebungen inklusive Sichtbarkeit und Security-Monitoring bereitstellt. -* [Romana](http://romana.io) ist eine Layer 3 Network-Lösung für Pod-Netzwerke welche auch die [NetworkPolicy API](/docs/concepts/services-networking/network-policies/) unterstützt. Details zur Installation als kubeadm Add-On sind [hier](https://github.com/romana/romana/tree/master/containerize) verfügbar. +* [Romana](https://github.com/romana/romana) ist eine Layer 3 Network-Lösung für Pod-Netzwerke welche auch die [NetworkPolicy API](/docs/concepts/services-networking/network-policies/) unterstützt. Details zur Installation als kubeadm Add-On sind [hier](https://github.com/romana/romana/tree/master/containerize) verfügbar. * [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) bietet Networking and Network-Policies und arbeitet auf beiden Seiten der Network-Partition ohne auf eine externe Datenbank angwiesen zu sein. ## Service-Discovery @@ -52,5 +52,3 @@ Die Add-Ons in den einzelnen Kategorien sind alphabetisch sortiert - Die Reihenf Es gibt einige weitere Add-Ons die in dem abgekündigten [cluster/addons](https://git.k8s.io/kubernetes/cluster/addons)-Verzeichnis dokumentiert sind. Add-Ons die ordentlich gewartet werden dürfen gerne hier aufgezählt werden. Wir freuen uns auf PRs! - - diff --git a/content/de/docs/concepts/configuration/_index.md b/content/de/docs/concepts/configuration/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/containers/_index.md b/content/de/docs/concepts/containers/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/containers/images.md b/content/de/docs/concepts/containers/images.md index d142405f06..68bc356bc0 100644 --- a/content/de/docs/concepts/containers/images.md +++ b/content/de/docs/concepts/containers/images.md @@ -16,7 +16,7 @@ Die `image` Eigenschaft eines Containers unterstüzt die gleiche Syntax wie die ## Aktualisieren von Images -Die Standardregel für das Herunterladen von Images ist `IfNotPresent`, dies führt dazu, dass das Kubelet Images überspringt, die bereits auf einem Node vorliegen. +Die Standardregel für das Herunterladen von Images ist `IfNotPresent`, dies führt dazu, dass das Image wird nur heruntergeladen wenn es noch nicht lokal verfügbar ist. Wenn sie stattdessen möchten, dass ein Image immer forciert heruntergeladen wird, können sie folgendes tun: diff --git a/content/de/docs/concepts/extend-kubernetes/_index.md b/content/de/docs/concepts/extend-kubernetes/_index.md index 5e389601ea..a537b745ad 100644 --- a/content/de/docs/concepts/extend-kubernetes/_index.md +++ b/content/de/docs/concepts/extend-kubernetes/_index.md @@ -1,4 +1,4 @@ --- -title: "Kubernets erweitern" +title: "Kubernetes erweitern" weight: 110 --- diff --git a/content/de/docs/concepts/overview/_index.md b/content/de/docs/concepts/overview/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/overview/what-is-kubernetes.md b/content/de/docs/concepts/overview/what-is-kubernetes.md index 66b79d6928..c9b49eed2a 100644 --- a/content/de/docs/concepts/overview/what-is-kubernetes.md +++ b/content/de/docs/concepts/overview/what-is-kubernetes.md @@ -15,7 +15,7 @@ Diese Seite ist eine Übersicht über Kubernetes. Kubernetes ist eine portable, erweiterbare Open-Source-Plattform zur Verwaltung von containerisierten Arbeitslasten und Services, die sowohl die deklarative Konfiguration als auch die Automatisierung erleichtert. -Es hat einen großes, schnell wachsendes Ökosystem. Kubernetes Dienstleistungen, Support und Tools sind weit verbreitet. +Es hat ein großes, schnell wachsendes Ökosystem. Kubernetes Dienstleistungen, Support und Tools sind weit verbreitet. Google hat das Kubernetes-Projekt 2014 als Open-Source-Projekt zur Verfügung gestellt. Kubernetes baut auf anderthalb Jahrzehnten Erfahrung auf, die Google mit der Ausführung von Produktions-Workloads in großem Maßstab hat, kombiniert mit den besten Ideen und Praktiken der Community. @@ -50,18 +50,18 @@ für Managementtools zu bieten, den Status von Kontrollpunkten zu ermitteln. Darüber hinaus basiert die [Kubernetes-Steuerungsebene](/docs/concepts/overview/components/) auf den gleichen APIs, die Entwicklern und Anwendern zur Verfügung stehen. Benutzer können ihre eigenen Controller, wie z.B. -[Scheduler](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/devel/scheduler.md), mit +[Scheduler](https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md), mit ihren [eigenen APIs](/docs/concepts/api-extension/custom-resources/) schreiben, die von einem universellen [Kommandozeilen-Tool](/docs/user-guide/kubectl-overview/) angesprochen werden können. -Dieses [Design](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md) hat es einer Reihe anderer Systeme ermöglicht, auf Kubernetes aufzubauen. +Dieses [Design](https://git.k8s.io/design-proposals-archive/architecture/architecture.md) hat es einer Reihe anderer Systeme ermöglicht, auf Kubernetes aufzubauen. ## Was Kubernetes nicht ist Kubernetes ist kein traditionelles, allumfassendes PaaS (Plattform als ein Service) System. Da Kubernetes nicht auf Hardware-, sondern auf Containerebene arbeitet, bietet es einige allgemein anwendbare Funktionen, die PaaS-Angeboten gemeinsam sind, wie Bereitstellung, Skalierung, Lastausgleich, Protokollierung und Überwachung. -Kubernetes ist jedoch nicht monolithisch, und diese Standardlösungen sind optional und modular etweiterbar. +Kubernetes ist jedoch nicht monolithisch, und diese Standardlösungen sind optional und modular erweiterbar. Kubernetes liefert die Bausteine für den Aufbau von Entwicklerplattformen, bewahrt aber die Wahlmöglichkeiten und Flexibilität der Benutzer, wo es wichtig ist. @@ -79,7 +79,7 @@ Kubernetes: Cluster-Speichersysteme (z.B. Ceph) als eingebaute Dienste. Solche Komponenten können auf Kubernetes laufen und/oder von Anwendungen, die auf Kubernetes laufen, über portable Mechanismen wie den Open Service Broker angesprochen werden. -* Bietet keine Konfigurationssprache bzw. kein Konfigurationssystem (z.B.[jsonnet](https://github.com/google/jsonnet)). +* Bietet keine Konfigurationssprache bzw. kein Konfigurationssystem (z.B. [jsonnet](https://github.com/google/jsonnet)). Es bietet eine deklarative API, die von beliebigen Formen deklarativer Spezifikationen angesprochen werden kann. * Bietet keine umfassenden Systeme zur Maschinenkonfiguration, Wartung, Verwaltung oder Selbstheilung. @@ -135,17 +135,17 @@ Zusammenfassung der Container-Vorteile: * **Dev und Ops Trennung der Bedenken**: Erstellen Sie Anwendungscontainer-Images nicht zum Deployment-, sondern zum Build-Releasezeitpunkt und entkoppeln Sie so Anwendungen von der Infrastruktur. -* **Überwachbarkeit** +* **Überwachbarkeit**: Nicht nur Informationen und Metriken auf Betriebssystemebene werden angezeigt, sondern auch der Zustand der Anwendung und andere Signale. * **Umgebungskontinuität in Entwicklung, Test und Produktion**: Läuft auf einem Laptop genauso wie in der Cloud. -* **Cloud- und OS-Distribution portabilität**: +* **Cloud- und OS-Distribution-Portabilität**: Läuft auf Ubuntu, RHEL, CoreOS, On-Prem, Google Kubernetes Engine und überall sonst. * **Anwendungsorientiertes Management**: Erhöht den Abstraktionsgrad vom Ausführen eines Betriebssystems auf virtueller Hardware bis zum Ausführen einer Anwendung auf einem Betriebssystem unter Verwendung logischer Ressourcen. -* **Locker gekoppelte, verteilte, elastische, freie [micro-services](https://martinfowler.com/articles/microservices.html)**: +* **Locker gekoppelte, verteilte, elastische, freie [Microservices](https://martinfowler.com/articles/microservices.html)**: Anwendungen werden in kleinere, unabhängige Teile zerlegt und können dynamisch bereitgestellt und verwaltet werden -- nicht ein monolithischer Stack, der auf einer großen Single-Purpose-Maschine läuft. * **Ressourcenisolierung**: diff --git a/content/de/docs/concepts/policy/_index.md b/content/de/docs/concepts/policy/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/services-networking/_index.md b/content/de/docs/concepts/services-networking/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/storage/_index.md b/content/de/docs/concepts/storage/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index 956190e6c7..eb3dcd971f 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -248,7 +248,7 @@ einige Einschränkungen: Eintrag zur Liste `metadata.finalizers` hinzugefügt werden. - Pod-Updates dürfen keine Felder ändern, die Ausnahmen sind `spec.containers[*].image`, - `spec.initContainers[*].image`,` spec.activeDeadlineSeconds` oder + `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` oder `spec.tolerations`. Für `spec.tolerations` kannnst du nur neue Einträge hinzufügen. - Für `spec.activeDeadlineSeconds` sind nur zwei Änderungen erlaubt: diff --git a/content/de/docs/contribute/_index.md b/content/de/docs/contribute/_index.md index db7c6687ab..e1ddd56875 100644 --- a/content/de/docs/contribute/_index.md +++ b/content/de/docs/contribute/_index.md @@ -1,6 +1,6 @@ --- content_type: concept -title: Zur Kubernets-Dokumentation beitragen +title: Zur Kubernetes-Dokumentation beitragen linktitle: Mitmachen main_menu: true weight: 80 diff --git a/content/de/docs/contribute/localization.md b/content/de/docs/contribute/localization.md index d40f941776..828865d5f5 100644 --- a/content/de/docs/contribute/localization.md +++ b/content/de/docs/contribute/localization.md @@ -22,30 +22,30 @@ Da Mitwirkende nicht ihren eigenen Pull Request freigeben können, brauchst du m Alle Lokalisierungsteams müssen sich mit ihren eigenen Ressourcen selbst tragen. Die Kubernetes-Website ist gerne bereit, deine Arbeit zu beherbergen, aber es liegt an dir, sie zu übersetzen. -### Finden deinen Zwei-Buchstaben-Sprachcode +### Ermittlung deines Zwei-Buchstaben-Sprachcodes Rufe den [ISO 639-1 Standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) auf und finde deinen Zwei-Buchstaben-Ländercode zur Lokalisierung. Zum Beispiel ist der Zwei-Buchstaben-Code für Korea `ko`. -### Duplizieren und klonen des Repositories +### Duplizieren und Klonen des Repositories -Als erstes [erstells du dir deine eigenes Duplikat](/docs/contribute/new-content/new-content/#fork-the-repo) vom [kubernetes/website] Repository. +Als erstes [erstellst du dir deine eigenes Duplikat](/docs/contribute/new-content/new-content/#fork-the-repo) vom [kubernetes/website] Repository. -Dann klonst du das Duplikat und `cd` hinein: +Dann klonst du das Duplikat und wechselst in das neu erstellte Verzeichnis: ```shell git clone https://github.com//website cd website ``` -### Eröffne ein Pull Request +### Eröffnen eines Pull Requests Als nächstes [eröffnest du einen Pull Request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) um eine Lokalisierung zum `kubernetes/website` Repository hinzuzufügen. -Der PR muss die [minimalen Inhaltsanforderungen](#mindestanforderungen) erfüllen bevor dieser genehmigt werden kann. +Der PR muss die [minimalen Inhaltsanforderungen](#mindestanforderungen) erfüllen, bevor dieser genehmigt werden kann. -Wie der PR für eine neue Lokalisierung aussieht kannst du dir an dem PR für die [Französische Dokumentation](https://github.com/kubernetes/website/pull/12548) ansehen. +Wie der PR für eine neue Lokalisierung aussieht, kannst du dir an dem PR für die [Französische Dokumentation](https://github.com/kubernetes/website/pull/12548) ansehen. -### Trete der Kubernetes GitHub Organisation bei +### Tritt der Kubernetes GitHub Organisation bei Sobald du eine Lokalisierungs-PR eröffnet hast, kannst du Mitglied der Kubernetes GitHub Organisation werden. Jede Person im Team muss einen eigenen [Antrag auf Mitgliedschaft in der Organisation](https://github.com/kubernetes/org/issues/new/choose) im `kubernetes/org`-Repository erstellen. @@ -94,9 +94,9 @@ contentDir = "content/de" weight = 3 ``` -Wenn du deinem Block einen Parameter `weight` zuweist, suche den Sprachblock mit dem höchsten Gewicht und addiere 1 zu diesem Wert. +Wenn du deinem Block einen Parameter `weight` zuweist, suche den Sprachblock mit dem höchsten Gewicht und addiere 1 zu diesem Wert. -Weitere Informationen zu Hugos Multilingualen Support findest du unter "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)" auf in der Hugo Dokumentation. +Weitere Informationen zu Hugos multilingualem Support findest du unter "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)" auf in der Hugo Dokumentation. ### Neuen Lokalisierungsordner erstellen @@ -213,7 +213,7 @@ Die neueste Version ist {{< latest-version >}}, so dass der neueste Versionszwei ### Seitenverlinkung in der Internationalisierung -Lokalisierungen müssen den Inhalt von [`i18n/de.toml`](https://github.com/kubernetes/website/blob/master/i18n/en.toml) in einer neuen sprachspezifischen Datei enthalten. Als Beispiel: `i18n/de.toml`. +Lokalisierungen müssen den Inhalt von [`i18n/de.toml`](https://github.com/kubernetes/website/blob/main/i18n/en.toml) in einer neuen sprachspezifischen Datei enthalten. Als Beispiel: `i18n/de.toml`. Füge eine neue Lokalisierungsdatei zu `i18n/` hinzu. Zum Beispiel mit Deutsch (`de`): @@ -278,7 +278,7 @@ Die Teams müssen den lokalisierten Inhalt in demselben Versionszweig zusammenf Ein Genehmiger muss einen Entwicklungszweig aufrechterhalten, indem er seinen Quellzweig auf dem aktuellen Stand hält und Merge-Konflikte auflöst. Je länger ein Entwicklungszweig geöffnet bleibt, desto mehr Wartung erfordert er in der Regel. Ziehe in Betracht, regelmäßig Entwicklungszweige zusammenzuführen und neue zu eröffnen, anstatt einen extrem lang laufenden Entwicklungszweig zu unterhalten. -Zu Beginn jedes Team-Meilensteins ist es hilfreich, ein Problem [Vergleich der Upstream-Änderungen](https://github.com/kubernetes/website/blob/master/scripts/upstream_changes.py) zwischen dem vorherigen Entwicklungszweig und dem aktuellen Entwicklungszweig zu öffnen. +Zu Beginn jedes Team-Meilensteins ist es hilfreich, ein Problem [Vergleich der Upstream-Änderungen](https://github.com/kubernetes/website/blob/main/scripts/upstream_changes.py) zwischen dem vorherigen Entwicklungszweig und dem aktuellen Entwicklungszweig zu öffnen. Während nur Genehmiger einen neuen Entwicklungszweig eröffnen und Pull-Anfragen zusammenführen können, kann jeder eine Pull-Anfrage für einen neuen Entwicklungszweig eröffnen. Es sind keine besonderen Genehmigungen erforderlich. @@ -301,5 +301,3 @@ Sobald eine Lokalisierung die Anforderungen an den Arbeitsablauf und die Mindest - Die Sprachauswahl auf der Website aktivieren - Die Verfügbarkeit der Lokalisierung über die Kanäle der [Cloud Native Computing Foundation](https://www.cncf.io/about/) (CNCF), einschließlich des [Kubernetes Blogs](https://kubernetes.io/blog/) veröffentlichen. - - diff --git a/content/de/docs/contribute/participate/_index.md b/content/de/docs/contribute/participate/_index.md new file mode 100644 index 0000000000..5b39657764 --- /dev/null +++ b/content/de/docs/contribute/participate/_index.md @@ -0,0 +1,94 @@ +--- +title: Bei SIG Docs mitmachen +content_type: concept +weight: 60 +card: + name: contribute + weight: 60 +--- + + + +Die SIG Docs ist eine der [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (Fachspezifischen Interessengruppen) innerhalb des Kubernetes-Projekts, die sich auf das Schreiben, Aktualisieren und Pflegen der Dokumentation für Kubernetes als Ganzes konzentriert. Weitere Informationen über die SIG findest du unter SIG Docs im [GitHub Repository der Community](https://github.com/kubernetes/community/tree/master/sig-docs). + +SIG Docs begrüßt Inhalte und Bewertungen von allen Mitwirkenden. Jeder kann einen +Pull Request (PR) eröffnen, und jeder ist willkommen, Fragen zum Inhalt zu stellen oder Kommentare +zu laufenden Pull Requests abzugeben. + +Du kannst dich ausserdem als [Member](/de/docs/contribute/participate/roles-and-responsibilities/#member), +[Reviewer](/de/docs/contribute/participate/roles-and-responsibilities/#reviewer), oder +[Approver](/de/docs/contribute/participate/roles-and-responsibilities/#approver) beteiligen. +Diese Rollen erfordern einen erweiterten Zugriff und bringen bestimmte Verantwortlichkeiten zur Genehmigung und Bestätigung von Änderungen mit sich. +Unter [community-membership](https://github.com/kubernetes/community/blob/master/community-membership.md) findest du weitere Informationen darüber, wie die Mitgliedschaft in der Kubernetes-Community funktioniert. + +Der Rest dieses Dokuments umreißt einige spezielle Vorgehensweisen dieser Rollen innerhalb von SIG Docs, die für die Pflege eines der öffentlichsten Aushängeschilder von Kubernetes verantwortlich ist - die Kubernetes-Website und die Dokumentation. + + +## SIG Docs Vorstand + +Jede SIG, auch die SIG Docs, wählt ein oder mehrere SIG-Mitglieder, die als +Vorstand fungieren. Sie sind die Kontaktstellen zwischen der SIG Docs und anderen Teilen der +der Kubernetes-Organisation. Sie benötigen umfassende Kenntnisse über die Struktur +des Kubernetes-Projekts als Ganzes und wie SIG Docs darin arbeitet. Hier findest alle weiteren Informationen zu den aktuellen Vorsitzenden und der [Leitung](https://github.com/kubernetes/community/tree/master/sig-docs#leadership). + +## SIG Docs-Teams und Automatisierung + +Die Automatisierung in SIG Docs stützt sich auf zwei verschiedene Mechanismen: +GitHub-Teams und OWNERS-Dateien. + +### GitHub Teams + +Es gibt zwei Kategorien von SIG Docs [Teams](https://github.com/orgs/kubernetes/teams?query=sig-docs) auf GitHub: + +- `@sig-docs-{language}-owners` sind Genehmiger und Verantwortliche +- `@sig-docs-{language}-reviewers` sind Reviewer + +Jede Gruppe kann in GitHub-Kommentaren mit ihrem `@name` referenziert werden, um mit allen Mitgliedern dieser Gruppe zu kommunizieren. + +Manchmal überschneiden sich Prow- und GitHub-Teams, ohne eine genaue Übereinstimmung. Für die Zuordnung von Issues, Pull-Requests und zur Unterstützung von PR-Genehmigungen verwendet die +Automatisierung die Informationen aus den `OWNERS`-Dateien. + +### OWNERS Dateien und Front-Matter + +Das Kubernetes-Projekt verwendet ein Automatisierungstool namens prow für die Automatisierung im Zusammenhang mit GitHub-Issues und Pull-Requests. +Das [Kubernetes-Website-Repository](https://github.com/kubernetes/website) verwendet zwei [prow-Plugins](https://github.com/kubernetes/test-infra/tree/master/prow/plugins): + +- blunderbuss +- approve + +Diese beiden Plugins nutzen die +[OWNERS](https://github.com/kubernetes/website/blob/main/OWNERS) und +[OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS_ALIASES) +Dateien auf der obersten Ebene des GitHub-Repositorys `kubernetes/website`, um zu steuern +wie prow innerhalb des Repositorys arbeitet. + +Eine OWNERS-Datei enthält eine Liste von Personen, die SIG Docs-Reviewer und +Genehmiger sind. OWNERS-Dateien können auch in Unterverzeichnissen existieren und bestimmen, wer +Dateien in diesem Unterverzeichnis und seinen Unterverzeichnissen als Gutachter oder +Genehmiger bestätigen darf. Weitere Informationen über OWNERS-Dateien im Allgemeinen findest du unter +[OWNERS](https://github.com/kubernetes/community/blob/master/contributors/guide/owners.md). + +Außerdem kann eine einzelne Markdown-Datei in ihrem Front-Matter (Vorspann) Reviewer und Genehmiger auflisten. +Entweder durch Auflistung einzelner GitHub-Benutzernamen oder GitHub-Gruppen. + +Die Kombination aus OWNERS-Dateien und Front-Matter in Markdown-Dateien bestimmt, welche Empfehlungen PR-Eigentümer von automatisierten Systemen erhalten, und wen sie um eine technische und redaktionelle Überprüfung ihres PRs bitten sollen. +## So funktioniert das Zusammenführen + +Wenn ein Pull Request mit der Branch (Ast) zusammengeführt wird, in dem der Inhalt bereitgestellt werden soll, wird dieser Inhalt auf http://kubernetes.io veröffentlicht. Um sicherzustellen, dass die Qualität der veröffentlichten Inhalte hoch ist, beschränken wir das Zusammenführen von Pull Requests auf +SIG Docs Freigabeberechtigte. So funktioniert es: + +- Wenn eine Pull-Anfrage sowohl das `lgtm`- als auch das `approve`-Label hat, kein `hold`-Label hat, + und alle Tests bestanden sind, wird der Pull Request automatisch zusammengeführt. +- Jedes Kubernetes-Mitglied kann das `lgtm`-Label hinzufügen, indem es einen `/lgtm`-Kommentar hinzufügt. +- Mitglieder der Kubernetes-Organisation und SIG Docs-Genehmiger können kommentieren, um das automatische Zusammenführen eines Pull Requests zu verhindern (durch Hinzufügen eines `/hold`-Kommentars + kann ein vorheriger `/lgtm`-Kommentar zurückgehalten werden). +- Nur SIG Docs-Genehmiger können einen Pull Request zusammenführen indem sie einen `/approve` Kommentar hinzufügen. + Einige Genehmiger übernehmen auch weitere spezielle Rollen, wie zum Beispiel [PR Wrangler](/docs/contribute/participate/pr-wranglers/) oder [SIG Docs Vorsitzende](#sig-docs-chairperson). + +## {{% heading "whatsnext" %}} + +Weitere Informationen über die Mitarbeit an der Kubernetes-Dokumentation findest du unter: + +- [Neue Inhalte beisteuern](/docs/contribute/new-content/overview/) +- [Inhalte überprüfen](/docs/contribute/review/reviewing-prs) +- [Styleguide für die Dokumentation](/docs/contribute/style/) diff --git a/content/de/docs/contribute/participate/pr-wranglers.md b/content/de/docs/contribute/participate/pr-wranglers.md new file mode 100644 index 0000000000..1a184fcf65 --- /dev/null +++ b/content/de/docs/contribute/participate/pr-wranglers.md @@ -0,0 +1,81 @@ +--- +title: PR Wranglers +content_type: concept +weight: 20 +--- + + + +SIG Docs [approvers](/docs/contribute/participate/roles-and-responsibilities/#approvers) übernehmen einwöchige Schichten um die [Pull Requests](https://github.com/kubernetes/website/wiki/PR-Wranglers) des Repositories zu verwalten. + +Dieser Abschnitt behandelt die Aufgaben eines PR-Wranglers. Weitere Informationen über gute Reviews findest du unter [Überprüfen von Änderungen](/docs/contribute/review/). + + +## Aufgaben + +Tägliche Aufgaben in einer einwöchigen Schicht als PR Wrangler: + +- Sortiere und kennzeichne täglich eingehende Probleme. Siehe [Einstufung und Kategorisierung von Problemen](/docs/contribute/review/for-approvers/#triage-and-categorize-issues) für Richtlinien, wie SIG Docs Metadaten verwendet. +- Überprüfe [offene Pull Requests](https://github.com/kubernetes/website/pulls) auf Qualität und Einhaltung der [Style](/docs/contribute/style/style-guide/) und [Content](/docs/contribute/style/content-guide/) Leitfäden. + - Beginne mit den kleinsten PRs (`size/XS`) und ende mit den größten (`size/XXL`). Überprüfe so viele PRs, wie du kannst. +- Achte darauf, dass die PR-Autoren den [CLA](https://github.com/kubernetes/community/blob/master/CLA.md) unterschreiben. + - Verwende [dieses](https://github.com/zparnold/k8s-docs-pr-botherer) Skript, um diejenigen, die den CLA noch nicht unterschrieben haben, daran zu erinnern, dies zu tun. +- Gib Feedback zu den Änderungen und bitte die Mitglieder anderer SIGs um technische Überprüfung. + - Gib inline Vorschläge für die vorgeschlagenen inhaltlichen Änderungen in den PR ein. + - Wenn du den Inhalt überprüfen musst, kommentiere den PR und bitte um weitere Details. + - Vergebe das/die entsprechende(n) `sig/`-Label. + - Falls nötig, weise die Reviever aus dem Block `revievers:` im Vorspann der Datei zu. +- Benutze den Kommentar `/approve`, um einen PR zum Zusammenführen zu genehmigen. Führe den PR zusammen, wenn er inhaltlich und technisch einwandfrei ist. + - PRs sollten einen `/lgtm`-Kommentar von einem anderen Mitglied haben, bevor sie zusammengeführt werden. + - Erwäge, technisch korrekte Inhalte zu akzeptieren, die nicht den [Stilrichtlinien](/docs/contribute/style/style-guide/) entsprechen. Eröffne ein neues Thema mit dem Label `good first issue`, um Stilprobleme anzusprechen. + +### Hilfreiche GitHub-Anfragen für Wranglers + +Die folgenden Anfragen sind beim Wrangling hilfreich. +Wenn du diese Anfragen abgearbeitet hast, ist die verbleibende Liste der zu prüfenden PRs meist klein. +Diese Anfragen schließen Lokalisierungs-PRs aus. Alle Anfragen beziehen sich auf den `main`-Branch, außer der letzten. + +- [Kein CLA, nicht zusammenfürbar](https://github.com/kubernetes/website/pulls?q=is%3Aopen+ist%3Apr+label%3A%22cncf-cla%3A+no%22+-label%3A%22do-not-merge%2Fwork-in-progress%22+-label%3A%22do-not-merge%2Fhold%22+label%3Alanguage%2Fen): + Erinnere den Beitragenden daran, den CLA zu unterschreiben. Wenn sowohl der Bot als auch ein Mensch sie daran erinnert haben, schließe + den PR und erinnere die Autoren daran, dass sie ihn erneut öffnen können, nachdem sie den CLA unterschrieben haben. + **Überprüfe keine PRs, deren Autoren den CLA nicht unterschrieben haben!** +- [Benötigt LGTM](https://github.com/kubernetes/website/pulls?q=is%3Aopen+ist%3Apr+-label%3A%22cncf-cla%3A+kein%22+-label%3Ado-not-merge%2Fwork-in-progress+-label%3Ado-not-merge%2Fhold+label%3Alanguage%2Fen+-label%3Algtm): + Listet PRs auf, die ein LGTM von einem Mitglied benötigen. Wenn der PR eine technische Überprüfung benötigt, schalte einen der vom Bot vorgeschlagenen Reviewer ein. Wenn der Inhalt überarbeitet werden muss, füge Vorschläge und Feedback in-line hinzu. +- [Hat LGTM, braucht die Zustimmung von Docs](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+-label%3Ado-not-merge%2Fwork-in-progress+-label%3Ado-not-merge%2Fhold+label%3Alanguage%2Fen+label%3Algtm+): + Listet PRs auf, die einen `/approve`-Kommentar benötigen, um zusammengeführt zu werden. +- [Quick Wins](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+base%3Amain+-label%3A%22do-not-merge%2Fwork-in-progress%22+-label%3A%22do-not-merge%2Fhold%22+label%3A%22cncf-cla%3A+yes%22+label%3A%22size%2FXS%22+label%3A%22language%2Fen%22): Listet PRs gegen den Hauptzweig auf, die nicht eindeutig blockiert sind. (ändere "XS" in der Größenbezeichnung, wenn du dich durch die PRs arbeitest [XS, S, M, L, XL, XXL]). +- [Nicht gegen den `main`-Branch](https://github.com/kubernetes/website/pulls?q=is%3Aopen+ist%3Apr+label%3Alanguage%2Fen+-base%3Amain): Wenn der PR gegen einen `dev-`Ast gerichtet ist, ist er für eine kommende Veröffentlichung. Weise diesen dem [Docs Release Manager](https://github.com/kubernetes/sig-release/tree/master/release-team#kubernetes-release-team-roles) zu: `/assign @`. Wenn der PR gegen einen alten Ast gerichtet ist, hilf dem Autor herauszufinden, ob er auf den richtigen Ast gerichtet ist. + +### Hilfreiche Prow-Befehle für Wranglers + +``` +# Englisches Label hinzufügen +/language en + +# füge dem PR ein Squash-Label hinzu, wenn es mehr als einen Commit gibt +/label tide/merge-method-squash + +# einen PR ueber Prow neu betiteln (z.B. als Work-in-Progress [WIP]) +/retitle [WIP] +``` + +### Wann sind Pull Requests zu schließen + +Reviews und Genehmigungen sind ein Mittel, um unsere PR-Warteschlange kurz und aktuell zu halten. Ein weiteres Mittel ist das Schließen. + +PRs werden geschlossen, wenn: +- Der Autor den CLA seit zwei Wochen nicht unterschrieben hat. + + Die Autoren können den PR wieder öffnen, nachdem sie den CLA unterschrieben haben. Dies ist ein risikoarmer Weg, um sicherzustellen, dass nichts zusammengeführt wird, ohne dass ein CLA unterzeichnet wurde. + +- Der Autor hat seit Zwei oder mehr Wochen nicht auf Kommentare oder Feedback geantwortet. + +Hab keine Angst, Pull Requests zu schließen. Mitwirkende können sie leicht wieder öffnen und die laufenden Arbeiten fortsetzen. Oft ist es die Nachricht über die Schließung, die einen Autor dazu anspornt, seinen Beitrag wieder aufzunehmen und zu beenden. + +Um eine Pull-Anfrage zu schließen, hinterlasse einen `/close`-Kommentar zu dem PR. + +{{< note >}} + +Der [`k8s-ci-robot`](https://github.com/k8s-ci-robot) Bot markiert Themen nach 90 Tagen Inaktivität als veraltet. Nach weiteren 30 Tagen markiert er Issues als faul und schließt sie. PR-Beauftragte sollten Themen nach 14-30 Tagen Inaktivität schließen. + +{{< /note >}} diff --git a/content/de/docs/contribute/participate/roles-and-responsibilities.md b/content/de/docs/contribute/participate/roles-and-responsibilities.md new file mode 100644 index 0000000000..66bf0c7891 --- /dev/null +++ b/content/de/docs/contribute/participate/roles-and-responsibilities.md @@ -0,0 +1,227 @@ +--- +title: Rollen und Verantwortlichkeiten +content_type: concept +weight: 10 +--- + +<!-- overview --> + +Jeder kann zu Kubernetes beitragen. Wenn deine Beiträge zu SIG Docs wachsen, kannst du dich für verschiedene Stufen der Mitgliedschaft in der Community bewerben. +Diese Rollen ermöglichen es dir, mehr Verantwortung innerhalb der Gemeinschaft zu übernehmen. +Jede Rolle erfordert mehr Zeit und Engagement. Die Rollen sind: + +- Jeder: kann regelmäßig zur Kubernetes-Dokumentation beitragen +- Member: können Issues zuweisen und einstufen und Pull Requests unverbindlich prüfen +- Reviewer: können die Überprüfung von Dokumentations-Pull-Requests leiten und für die Qualität einer Änderung bürgen +- Approver: können die Überprüfung von Dokumentations- und Merge-Änderungen leiten + +<!-- body --> + +## Jeder + +Jeder mit einem GitHub-Konto kann zu Kubernetes beitragen. SIG Docs heißt alle neuen Mitwirkenden willkommen! + +Jeder kann: + +- Ein Problem in einem beliebigen [Kubernetes](https://github.com/kubernetes/) + Repository, einschließlich + [`kubernetes/website`](https://github.com/kubernetes/website) melden +- Unverbindliches Feedback zu einem Pull Request geben +- Zu einer Lokalisierung beitragen +- Verbesserungen auf [Slack](https://slack.k8s.io/) oder der + [SIG docs mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) vorschlagen. + +Nach dem [Signieren des CLA](/docs/contribute/new-content/overview/#sign-the-cla) kann jeder auch: + +- eine Pull-Anfrage öffnen, um bestehende Inhalte zu verbessern, neue Inhalte hinzuzufügen oder einen Blogbeitrag oder eine Fallstudie zu schreiben +- Diagramme, Grafiken und einbettbare Screencasts und Videos erstellen + +Weitere Informationen findest du unter [neue Inhalte beisteuern](/docs/contribute/new-content/). + +## Member + +Ein Member (Mitglied) ist jemand, der bereits mehrere Pull Requests an +`kubernetes/website` eingereicht hat. Mitglieder sind ein Teil der +[Kubernetes GitHub Organisation](https://github.com/kubernetes). + +Member können: + +- Alles tun, was unter [Jeder](#jeder) aufgeführt ist +- Den Kommentar `/lgtm` verwenden, um einem Pull Request das Label LGTM (looks good to me) hinzuzufügen + + {{< note >}} + Die Verwendung von `/lgtm` löst eine Automatisierung aus. Wenn du eine unverbindliche + Zustimmung geben willst, funktioniert der Kommentar "LGTM" auch! + {{< /note >}} + +- Verwende den Kommentar `/hold`, um das Zusammenführen eines Pull Requests zu blockieren. +- Benutze den Kommentar `/assign`, um einem Pull Request einen Reviewer zuzuweisen. +- Unverbindliche Überprüfung von Pull Requests +- Nutze die Automatisierung, um Issues zu sortieren und zu kategorisieren +- Neue Funktionen dokumentieren + +### Mitglied werden + +Du kannst ein Mitglied werden, nachdem du mindestens 5 substantielle Pull Requests eingereicht hast und die anderen +[Anforderungen](https://github.com/kubernetes/community/blob/master/community-membership.md#member) erforderst: + +1. Finde zwei [Reviewer](#reviewers) oder [Approver](#approvers), die deine Mitgliedschaft [sponsern](/docs/contribute/advanced#sponsor-a-new-contributor). + + Bitte um Sponsoring im [#sig-docs channel on Slack](https://kubernetes.slack.com) oder auf der + [SIG Docs Mailingliste](https://groups.google.com/forum/#!forum/kubernetes-sig-docs). + + {{< note >}} + Schicke keine direkte E-Mail oder Slack-Direktnachricht an ein einzelnes + SIG Docs-Mitglied. Du musst das Sponsoring beantragen, bevor du deine Bewerbung einreichst. + {{< /note >}} + +1. Eröffne ein GitHub-Issue im + [`kubernetes/org`](https://github.com/kubernetes/org/) Repository. Verwende dabei das + **Organization Membership Request** issue template. + +1. Informiere deine Sponsoren über das GitHub-Issue. Du kannst entweder: + - Ihren GitHub-Benutzernamen in deinem Issue (`@<GitHub-Benutzername>`) erwähnen + - Ihnen den Issue-Link über Slack oder per E-Mail senden. + + Die Sponsoren werden deine Anfrage mit einer "+1"-Stimme genehmigen. Sobald deine Sponsoren + genehmigen, fügt dich ein Kubernetes-GitHub-Admin als Mitglied hinzu. + Herzlichen Glückwunsch! + + Wenn dein Antrag auf Mitgliedschaft nicht angenommen wird, erhältst du eine Rückmeldung. + Nachdem du dich mit dem Feedback auseinandergesetzt hast, kannst du dich erneut bewerben. + +1. Nimm die Einladung zur Kubernetes GitHub Organisation in deinem E-Mail-Konto an. + + {{< note >}} + GitHub sendet die Einladung an die Standard-E-Mail-Adresse in deinem Konto. + {{< /note >}} + +## Reviewer + +Reviewer (Gutachteren) sind dafür verantwortlich, offene Pull Requests zu überprüfen. Anders als bei den Mitgliedern +musst du auf das Feedback der Prüfer eingehen. Reviewer sind Mitglieder des +[@kubernetes/sig-docs-{language}-reviews](https://github.com/orgs/kubernetes/teams?query=sig-docs) +GitHub-Teams. + +Gutachteren können: + +- Alles tun, was unter [Jeder](#jeder) und [Member](#member) aufgeführt ist +- Pull Requests überprüfen und verbindliches Feedback geben + + {{< note >}} + Um unverbindliches Feedback zu geben, stellst du deinen Kommentaren eine Formulierung wie "Optional:" voran. + {{< /note >}} + +- Bearbeite benutzerseitige Zeichenfolgen im Code +- Verbessere Code-Kommentare + +### Zuweisung von Reviewern zu Pull Requests + +Die Automatisierung weist allen Pull Requests Reviewer zu. Du kannst eine +Review von einer bestimmten Person anfordern, indem du einen Kommentar schreibst: `/assign +[@_github_handle]`. + +Wenn der zugewiesene Prüfer den PR nicht kommentiert hat, kann ein anderer Prüfer +einspringen. Du kannst bei Bedarf auch technische Prüfer zuweisen. + +### Verwendung von `/lgtm` + +LGTM steht für "Looks good to me" und zeigt an, dass ein Pull Request +technisch korrekt und bereit zum Zusammenführen ist. Alle PRs brauchen einen `/lgtm` Kommentar von einem +Reviewer und einen `/approve` Kommentar von einem Approver, um zusammengeführt zu werden. + +Ein `/lgtm`-Kommentar vom Reviewer ist verbindlich und löst eine Automatisierung aus, die das `lgtm`-Label hinzufügt. + +### Reviewer werden + +Wenn du die +[Anforderungen](https://github.com/kubernetes/community/blob/master/community-membership.md#reviewer) erfüllst, +kannst du ein SIG Docs-Reviewer werden. Reviewer in anderen SIGs müssen sich gesondert für den Reviewer-Status in SIG Docs bewerben. + +So bewirbst du dich: + +1. Eröffne einen Pull Request, in dem du deinen GitHub-Benutzernamen in einen Abschnitt der + [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS) Datei + im `kubernetes/website` Repository hinzufügt. + + {{< note >}} + Wenn du dir nicht sicher bist, wo du dich hinzufügen sollst, füge dich zu `sig-docs-de-reviews` hinzu. + {{< /note >}} + +1. Weise den PR einem oder mehreren SIG-Docs-Genehmigern zu (Benutzernamen, die unter + `sig-docs-{language}-owners` aufgelisted sind). + Wenn der PR genehmigt wurde, fügt dich ein SIG Docs-Lead dem entsprechenden GitHub-Team hinzu. Sobald du hinzugefügt bist, + wird [@k8s-ci-robot](https://github.com/kubernetes/test-infra/tree/master/prow#bots-home) + dich als Reviewer für neue Pull Requests vorschlagen und zuweisen. + + +## Approver + + +Approver (Genehmiger) prüfen und genehmigen Pull Requests zum Zusammenführen. Genehmigende sind Mitglieder des +[@kubernetes/sig-docs-{language}-owners](https://github.com/orgs/kubernetes/teams/?query=sig-docs) +GitHub-Teams. + +Genehmigende können Folgendes tun: + +- Alles, was unter [Jeder](#jeder), [Member](#member) und [Reviewer](#reviewes) aufgeführt ist +- Inhalte von Mitwirkenden veröffentlichen, indem sie Pull Requests mit dem Kommentar `/approve` genehmigen und zusammenführen +- Verbesserungen für den Style Guide vorschlagen +- Verbesserungsvorschläge für Docs-Tests einbringen +- Verbesserungsvorschläge für die Kubernetes-Website oder andere Tools machen + +Wenn der PR bereits einen `/lgtm` hat, oder wenn der Genehmigende ebenfalls mit +`/lgtm` kommentiert, wird der PR automatisch zusammengeführt. Ein SIG Docs-Genehmiger sollte nur ein +`/lgtm` für eine Änderung hinterlassen, die keine weitere technische Überprüfung erfordert. + +### Pull Requests genehmigen + +Genehmiger und SIG Docs-Leads sind die Einzigen, die Pull Requests +in das Website-Repository aufnehmen. Damit sind bestimmte Verantwortlichkeiten verbunden. + +- Genehmigende können den Befehl `/approve` verwenden, der PRs in das Repository einfügt. + + {{< warning >}} + Ein unvorsichtiges Zusammenführen kann die Website lahmlegen, also sei dir sicher, dass du es auch so meinst, wenn du etwas zusammenführst. + {{< /warning >}} + +- Vergewissere dich, dass die vorgeschlagenen Änderungen den + [Beitragsrichtlinien](/docs/contribute/style/content-guide/#contributing-content) entsprechen. + + Wenn du jemals eine Frage hast oder dir bei etwas nicht sicher bist, fordere einfach Hilfe an, um eine zusätzliche Überprüfung zu erhalten. + +- Vergewissere dich, dass die Netlify-Tests erfolgreich sind, bevor du einen PR mittels `/approve` genehmigst. + + <img src="/images/docs/contribute/netlify-pass.png" width="75%" alt="Netlify-Tests müssen vor der Freigabe bestanden werden" /> + +- Besuche die Netlify-Seitenvorschau für den PR, um sicherzustellen, dass alles gut aussieht, bevor du es genehmigst. + +- Nimm am [PR Wrangler Rotationsplan](https://github.com/kubernetes/website/wiki/PR-Wranglers) + für wöchentliche Rotationen teil. SIG Docs erwartet von allen Genehmigern, dass sie an dieser + Rotation teilnehmen. Siehe [PR-Wranglers](/docs/contribute/participate/pr-wranglers/). + für weitere Details. + +### Approver werden + +Wenn du die [Anforderungen](https://github.com/kubernetes/community/blob/master/community-membership.md#approver) erfüllst, +kannst du ein SIG Docs Approver werden. Genehmigende in anderen SIGs müssen sich separat für den Approver-Status in SIG Docs bewerben. + +So bewirbst du dich: + +1. Eröffne eine Pull-Anfrage, in der du dich in einem Abschnitt der + [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS) + Datei im `kubernetes/website` Repository hinzuzufügen. + + {{< note >}} + Wenn du dir nicht sicher bist, wo du dich hinzufügen sollst, füge dich zu `sig-docs-de-owners` hinzu. + {{< /note >}} + +2. Weise den PR einem oder mehreren aktuellen SIG Docs Genehmigern zu. + +Wenn der PR genehmigt wurde, fügt dich ein SIG Docs-Lead dem entsprechenden GitHub-Team hinzu. Sobald du hinzugefügt bist, +wird [@k8s-ci-robot](https://github.com/kubernetes/test-infra/tree/master/prow#bots-home) +dich als Reviewer für neue Pull Requests vorschlagen und zuweisen. + +## {{% heading "whatsnext" %}} + +- Erfahre mehr über [PR-Wrangling](/de/docs/contribute/participate/pr-wranglers/), eine Rolle, die alle Genehmiger im Wechsel übernehmen. diff --git a/content/de/docs/reference/_index.md b/content/de/docs/reference/_index.md index 7ffa9c04c9..9df5aa53ba 100644 --- a/content/de/docs/reference/_index.md +++ b/content/de/docs/reference/_index.md @@ -56,6 +56,6 @@ Offiziell unterstützte Clientbibliotheken: ## Design Dokumentation -Ein Archiv der Designdokumente für Kubernetes-Funktionalität. Gute Ansatzpunkte sind [Kubernetes Architektur](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md) und [Kubernetes Design Übersicht](https://git.k8s.io/community/contributors/design-proposals). +Ein Archiv der Designdokumente für Kubernetes-Funktionalität. Gute Ansatzpunkte sind [Kubernetes Architektur](https://git.k8s.io/design-proposals-archive/architecture/architecture.md) und [Kubernetes Design Übersicht](https://git.k8s.io/community/contributors/design-proposals). diff --git a/content/de/docs/reference/glossary/etcd.md b/content/de/docs/reference/glossary/etcd.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/index.md b/content/de/docs/reference/glossary/index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-apiserver.md b/content/de/docs/reference/glossary/kube-apiserver.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-controller-manager.md b/content/de/docs/reference/glossary/kube-controller-manager.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-scheduler.md b/content/de/docs/reference/glossary/kube-scheduler.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kubelet.md b/content/de/docs/reference/glossary/kubelet.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/kubectl/_index.md b/content/de/docs/reference/kubectl/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/kubectl/cheatsheet.md b/content/de/docs/reference/kubectl/cheatsheet.md index bce7eeefb3..d2472b1c2a 100644 --- a/content/de/docs/reference/kubectl/cheatsheet.md +++ b/content/de/docs/reference/kubectl/cheatsheet.md @@ -322,7 +322,7 @@ Ausgabeformat | Beschreibung ### Kubectl Ausgabe Ausführlichkeit und Debugging -Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v ` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) beschrieben. +Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) beschrieben. Ausführlichkeit | Beschreibung --------------| ----------- diff --git a/content/de/docs/setup/minikube.md b/content/de/docs/setup/minikube.md index 643348ee89..35de917a65 100644 --- a/content/de/docs/setup/minikube.md +++ b/content/de/docs/setup/minikube.md @@ -424,7 +424,7 @@ export no_proxy=$no_proxy,$(minikube ip) Minikube verwendet [libmachine](https://github.com/docker/machine/tree/master/libmachine) zur Bereitstellung von VMs, und [kubeadm](https://github.com/kubernetes/kubeadm) um einen Kubernetes-Cluster in Betrieb zu nehmen. -Weitere Informationen zu Minikube finden Sie im [Vorschlag](https://git.k8s.io/community/contributors/design-proposals/cluster-lifecycle/local-cluster-ux.md). +Weitere Informationen zu Minikube finden Sie im [Vorschlag](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/local-cluster-ux.md). ## Zusätzliche Links diff --git a/content/de/docs/setup/release/_index.md b/content/de/docs/setup/release/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/setup/release/building-from-source.md b/content/de/docs/setup/release/building-from-source.md index 76879df995..14c9ccee92 100644 --- a/content/de/docs/setup/release/building-from-source.md +++ b/content/de/docs/setup/release/building-from-source.md @@ -27,6 +27,6 @@ cd kubernetes make release ``` -Mehr Informationen zum Release-Prozess finden Sie im kubernetes/kubernetes [`build`](http://releases.k8s.io/{{< param "githubbranch" >}}/build/) Verzeichnis. +Mehr Informationen zum Release-Prozess finden Sie im kubernetes/kubernetes [`build`](http://releases.k8s.io/master/build/) Verzeichnis. diff --git a/content/de/docs/tasks/access-application-cluster/_index.md b/content/de/docs/tasks/access-application-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/access-kubernetes-api/_index.md b/content/de/docs/tasks/access-kubernetes-api/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/administer-cluster/_index.md b/content/de/docs/tasks/administer-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/configure-pod-container/_index.md b/content/de/docs/tasks/configure-pod-container/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/debug-application-cluster/_index.md b/content/de/docs/tasks/debug-application-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/federation/_index.md b/content/de/docs/tasks/federation/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/inject-data-application/_index.md b/content/de/docs/tasks/inject-data-application/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/job/_index.md b/content/de/docs/tasks/job/_index.md index ab0432135f..0497a5cfc6 100644 --- a/content/de/docs/tasks/job/_index.md +++ b/content/de/docs/tasks/job/_index.md @@ -1,5 +1,6 @@ --- title: "Jobs ausführen" +description: Führen Sie Jobs mit paralleler Verarbeitung aus. weight: 50 --- diff --git a/content/de/docs/tasks/manage-daemon/_index.md b/content/de/docs/tasks/manage-daemon/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/run-application/_index.md b/content/de/docs/tasks/run-application/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/run-application/horizontal-pod-autoscale.md b/content/de/docs/tasks/run-application/horizontal-pod-autoscale.md index cd120285f1..31019744a7 100644 --- a/content/de/docs/tasks/run-application/horizontal-pod-autoscale.md +++ b/content/de/docs/tasks/run-application/horizontal-pod-autoscale.md @@ -11,7 +11,7 @@ weight: 90 <!-- overview --> -Der Horizontal Pod Autoscaler skaliert automatisch die Anzahl der Pods eines Replication Controller, Deployment oder Replikat Set basierend auf der beobachteten CPU-Auslastung (oder, mit Unterstützung von [benutzerdefinierter Metriken](https://git.k8s.io/community/contributors/design-proposals/instrumentation/custom-metrics-api.md), von der Anwendung bereitgestellten Metriken). Beachte, dass die horizontale Pod Autoskalierung nicht für Objekte gilt, die nicht skaliert werden können, z. B. DaemonSets. +Der Horizontal Pod Autoscaler skaliert automatisch die Anzahl der Pods eines Replication Controller, Deployment oder Replikat Set basierend auf der beobachteten CPU-Auslastung (oder, mit Unterstützung von [benutzerdefinierter Metriken](https://git.k8s.io/design-proposals-archive/instrumentation/custom-metrics-api.md), von der Anwendung bereitgestellten Metriken). Beachte, dass die horizontale Pod Autoskalierung nicht für Objekte gilt, die nicht skaliert werden können, z. B. DaemonSets. Der Horizontal Pod Autoscaler ist als Kubernetes API-Ressource und einem Controller implementiert. Die Ressource bestimmt das Verhalten des Controllers. @@ -46,7 +46,7 @@ Das Verwenden von Metriken aus Heapster ist seit der Kubernetes Version 1.11 ver Siehe [Unterstützung der Metrik APIs](#unterstützung-der-metrik-apis) für weitere Details. -Der Autoscaler greift über die Scale Sub-Ressource auf die entsprechenden skalierbaren Controller (z.B. Replication Controller, Deployments und Replika Sets) zu. Scale ist eine Schnittstelle, mit der Sie die Anzahl der Replikate dynamisch einstellen und jeden ihrer aktuellen Zustände untersuchen können. Weitere Details zu der Scale Sub-Ressource findest du [hier](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#scale-subresource). +Der Autoscaler greift über die Scale Sub-Ressource auf die entsprechenden skalierbaren Controller (z.B. Replication Controller, Deployments und Replika Sets) zu. Scale ist eine Schnittstelle, mit der Sie die Anzahl der Replikate dynamisch einstellen und jeden ihrer aktuellen Zustände untersuchen können. Weitere Details zu der Scale Sub-Ressource findest du [hier](https://git.k8s.io/design-proposals-archive/autoscaling/horizontal-pod-autoscaler.md#scale-subresource). ### Details zum Algorithmus @@ -90,7 +90,7 @@ Die aktuelle stabile Version, die nur die Unterstützung für die automatische S Die Beta-Version, welche die Skalierung des Speichers und benutzerdefinierte Metriken unterstützt, befindet sich unter `autoscaling/v2beta2`. Die in `autoscaling/v2beta2` neu eingeführten Felder bleiben bei der Arbeit mit `autoscaling/v1` als Anmerkungen erhalten. -Weitere Details über das API Objekt kann unter dem [HorizontalPodAutoscaler Objekt](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object) gefunden werden. +Weitere Details über das API Objekt kann unter dem [HorizontalPodAutoscaler Objekt](https://git.k8s.io/design-proposals-archive/autoscaling/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object) gefunden werden. ## Unterstützung des Horizontal Pod Autoscaler in kubectl @@ -100,7 +100,7 @@ Das auflisten der Autoskalierer geschieht über `kubectl get hpa` und eine detai Letzendlich können wir einen Autoskalierer mit `kubectl delete hpa` löschen. Zusätzlich gibt es einen speziellen Befehl `kubectl autoscale` zur einfachen Erstellung eines Horizontal Pod Autoscalers. -Wenn du beispielsweise `kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80` ausführst, wird ein Autoskalierer für den Replication Set *foo* erstellt, wobei die Ziel-CPU-Auslastung auf `80%` und die Anzahl der Replikate zwischen 2 und 5 gesetzt wird. +Wenn du beispielsweise `kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80` ausführst, wird ein Autoskalierer für den ReplicaSet *foo* erstellt, wobei die Ziel-CPU-Auslastung auf `80%` und die Anzahl der Replikate zwischen 2 und 5 gesetzt wird. Die Detaildokumentation von `kubectl autoscale` kann [hier](/docs/reference/generated/kubectl/kubectl-commands/#autoscale) gefunden werden. ## Autoskalieren während rollierender Updates @@ -166,7 +166,7 @@ Standardmäßig ruft der HorizontalPodAutoscaler Controller Metriken aus einer R ## {{% heading "whatsnext" %}} -* Design Dokument [Horizontal Pod Autoscaling](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md). +* Design Dokument [Horizontal Pod Autoscaling](https://git.k8s.io/design-proposals-archive/autoscaling/horizontal-pod-autoscaler.md). * kubectl autoscale Befehl: [kubectl autoscale](/docs/reference/generated/kubectl/kubectl-commands/#autoscale). * Verwenden des [Horizontal Pod Autoscaler](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/). diff --git a/content/de/docs/tasks/service-catalog/_index.md b/content/de/docs/tasks/service-catalog/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/tls/_index.md b/content/de/docs/tasks/tls/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/tools/_index.md b/content/de/docs/tasks/tools/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tutorials/_index.md b/content/de/docs/tutorials/_index.md index e00382c893..3ab69b6073 100644 --- a/content/de/docs/tutorials/_index.md +++ b/content/de/docs/tutorials/_index.md @@ -50,6 +50,8 @@ Bevor Sie die einzelnen Lernprogramme durchgehen, möchten Sie möglicherweise e * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Services * [Source IP verwenden](/docs/tutorials/services/source-ip/) diff --git a/content/de/docs/tutorials/hello-minikube.md b/content/de/docs/tutorials/hello-minikube.md index a1bf6dd493..2137c5fdd9 100644 --- a/content/de/docs/tutorials/hello-minikube.md +++ b/content/de/docs/tutorials/hello-minikube.md @@ -105,7 +105,7 @@ Der Pod führt einen Container basierend auf dem bereitgestellten Docker-Image a hello-node-5f76cf6ccf-br9b5 1/1 Running 0 1m ``` -4. Cluster Events anzigen: +4. Cluster Events anzeigen: ```shell kubectl get events diff --git a/content/de/docs/tutorials/kubernetes-basics/_index.html b/content/de/docs/tutorials/kubernetes-basics/_index.html index 8bf1c1dc38..8cb49dce31 100644 --- a/content/de/docs/tutorials/kubernetes-basics/_index.html +++ b/content/de/docs/tutorials/kubernetes-basics/_index.html @@ -46,7 +46,7 @@ card: </div> <div id="basics-modules" class="content__modules"> - <h2>Kubernets Grundlagen Module</h2> + <h2>Kubernetes Grundlagen Module</h2> <div class="row"> <div class="col-md-12"> <div class="row"> diff --git a/content/de/includes/default-storage-class-prereqs.md b/content/de/includes/default-storage-class-prereqs.md index 3865e644b7..1ca065cc14 100644 --- a/content/de/includes/default-storage-class-prereqs.md +++ b/content/de/includes/default-storage-class-prereqs.md @@ -1 +1 @@ -Sie benötigen entweder einen dynamischen PersistentVolume-Anbieter mit einer [Standard-Speicherklasse](/docs/concepts/storage/storage-classes/), oder Sie selbst stellen statische [PersistentVolumes](/docs/user-guide/persistent-volumes/#provisioning) bereit, um die [PersistentVolumeClaims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) zu erfüllen, die hier verwendet werden. \ No newline at end of file +Sie benötigen entweder einen dynamischen PersistentVolume-Anbieter mit einer [Standard-Speicherklasse](/docs/concepts/storage/storage-classes/), oder Sie selbst stellen statische [PersistentVolumes](/docs/concepts/storage/persistent-volumes/#provisioning) bereit, um die [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) zu erfüllen, die hier verwendet werden. \ No newline at end of file diff --git a/content/zh/includes/partner-style.css b/content/de/includes/partner-style.css similarity index 100% rename from content/zh/includes/partner-style.css rename to content/de/includes/partner-style.css diff --git a/content/de/partners/_index.html b/content/de/partners/_index.html index e99b03b62a..ecf3e2542b 100644 --- a/content/de/partners/_index.html +++ b/content/de/partners/_index.html @@ -1,91 +1,53 @@ --- title: Partner bigheader: Kubernetes Partner -abstract: Entwicklung des Kubernetes-Ökosystems. +abstract: Erweiterung des Kubernetes-Ökosystems. class: gridPage cid: partners --- <section id="users"> - <main> - <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, dynamische Codebasis zu schaffen, die ein Spektrum von aufeinander abgestimmten Plattformen unterstützt.</h5> - <div class="col-container"> - <div class="col-nav"> - <center> - <h5> - <b>Kubernetes zertifizierte Service Provider</b> - </h5> - <br>Geprüfte Service Provider mit großer Erfahrung, die Unternehmen bei der erfolgreichen Einführung von Kubernetes unterstützen. - <br><br><br> - <button id="kcsp" class="button" onClick="updateSrc(this.id)">KCSP-Partner anzeigen</button> - <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden? - </center> - </div> - <div class="col-nav"> - <center> - <h5> - <b>Kubernetes-Distributionen, gehostete Plattformen und zertifizierte Installateure</b> - </h5>Software-Konformität stellt sicher, dass die Kubernetes-Versionen aller Hersteller die erforderlichen APIs unterstützen. - <br><br><br> - <button id="conformance" class="button" onClick="updateSrc(this.id)">Zertifizierte Partner anzeigen</button> - <br><br>Interessiert daran, <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes zertifiziert</a> zu werden? - </center> - </div> - <div class="col-nav"> - <center> - <h5><b>Kubernetes Training Partner</b></h5> - <br>Geprüfte Schulungsanbieter, die über umfassende Erfahrung in Cloud Native Technologietrainings verfügen. - <br><br><br><br> - <button id="ktp" class="button" onClick="updateSrc(this.id)">KTP Partner anzeigen</button> - <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden? - </center> - </div> - </div> -<script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script> -<script type="text/javascript"> - - var defaultLink = "https://landscape.cncf.io/category=kubernetes-certified-service-provider&format=card-mode&grouping=category&embed=yes"; - var firstLink = "https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&format=card-mode&grouping=category&embed=yes"; - var secondLink = "https://landscape.cncf.io/category=kubernetes-training-partner&format=card-mode&grouping=category&embed=yes"; - - function updateSrc(buttonId) { - if (buttonId == "kcsp") { - $("#landscape").attr("src",defaultLink); - window.location.hash = "#kcsp"; - } - if (buttonId == "conformance") { - $("#landscape").attr("src",firstLink); - window.location.hash = "#conformance"; - } - if (buttonId == "ktp") { - $("#landscape").attr("src",secondLink); - window.location.hash = "#ktp"; - } - } - - // Automatically load the correct iframe based on the URL fragment - document.addEventListener('DOMContentLoaded', function() { - var showContent = "kcsp"; - if (window.location.hash) { - console.log('hash is:', window.location.hash.substring(1)); - showContent = window.location.hash.substring(1); - } - updateSrc(showContent); - }); -</script> -<body> - <div id="frameHolder"> - <iframe id="landscape" frameBorder="0" scrolling="no" style="width: 1px; min-width: 100%" src=""></iframe> - <script src="https://landscape.cncf.io/iframeResizer.js"></script> + <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, lebendige Codebasis zu schaffen, die ein Spektrum von ergänzenden Plattformen unterstützt.</h5> + <div class="col-container"> + <div class="col-nav"> + <center> + <h5> + <b>Kubernetes-zertifizierte Service-Anbieter</b> + </h5> + <br>Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen. + <br><br><br> + <button class="button landscape-trigger landscape-default" data-landscape-types="kubernetes-certified-service-provider" id="kcsp">KCSP Partner anzeigen</button> + <br><br>Interessiert daran, ein + <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden? + </center> + </div> + <div class="col-nav"> + <center> + <h5> + <b>Zertifizierte Kubernetes-Distributionen, gehostete Plattformen und Installationssysteme</b> + </h5>Die Softwarekonformität stellt sicher, dass die Kubernetes-Version eines jeden Anbieters die erforderlichen APIs unterstützt. + <br><br><br> + <button class="button landscape-trigger" data-landscape-types="certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer" id="conformance">Konforme Partner anzeigen</button> + <br><br>Interessiert daran, + <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes Zertifiziert</a> zu werden? + </center> + </div> + <div class="col-nav"> + <center> + <h5> + <b>Kubernetes Schulungspartner</b> + </h5> + <br>Geprüfte Schulungsanbieter mit umfassender Erfahrung in der Weiterbildung im Bereich Cloud Native Technology. + <br><br><br> + <button class="button landscape-trigger" data-landscape-types="kubernetes-training-partner" id="ktp">KTP Partner anzeigen</button> + <br><br>Interessiert daran, ein + <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden? + </center> + </div> </div> -</body> - </main> + {{< cncf-landscape helpers=true >}} </section> <style> {{< include "partner-style.css" >}} -</style> - -<script> - {{< include "partner-script.js" >}} -</script> +</style> \ No newline at end of file diff --git a/content/de/training/_index.html b/content/de/training/_index.html new file mode 100644 index 0000000000..6ba21e7861 --- /dev/null +++ b/content/de/training/_index.html @@ -0,0 +1,137 @@ +--- +title: Schulungen +bigheader: Kubernetes Schulungen und Zertifizierungen +abstract: Schulungsprogramme, Zertifizierungen und Partner. +layout: basic +cid: training +class: training +--- + +<section class="call-to-action"> + <div class="main-section"> + <div class="call-to-action" id="cta-certification"> + <div class="cta-text"> + <h2>Gestalte deine Cloud Native Karriere</h2> + <p>Kubernetes ist das Herzstück der Cloud Native-Bewegung. Mit den Schulungen und Zertifizierungen der Linux Foundation und unserer Schulungspartner kannst Du in deine Karriere investieren, Kubernetes lernen und deine Cloud Native-Projekte zum Erfolg führen.</p> + </div> + <div class="logo-certification cta-image" id="logo-kcnf"> + <img src="/images/training/kubernetes-kcnf-white.svg" /> + </div> + <div class="logo-certification cta-image" id="logo-cka"> + <img src="/images/training/kubernetes-cka-white.svg"/> + </div> + <div class="logo-certification cta-image" id="logo-ckad"> + <img src="/images/training/kubernetes-ckad-white.svg"/> + </div> + <div class="logo-certification cta-image" id="logo-cks"> + <img src="/images/training/kubernetes-cks-white.svg"/> + </div> + </div> + </div> +</section> + +<section> + <div class="main-section padded"> + <center> + <h2>Nimm an einen kostenlosen Kurs bei edX teil</h2> + </center> + <div class="col-container"> + <div class="col-nav"> + <center> + <h5> + <b>Einführung in Kubernetes <br>  </b> + </h5> + <p>Möchtest Du Kubernetes lernen? Erfahre alles über dieses leistungsstarke System zur Verwaltung von Containeranwendungen.</p> + <br> + <a href="https://www.edx.org/course/introduction-to-kubernetes" target="_blank" class="button">Zum Kurs</a> + </center> + </div> + <div class="col-nav"> + <center> + <h5> + <b>Einführung in Cloud-Infrastruktur Technologien</b> + </h5> + <p>Lerne die Grundlagen für den Aufbau und die Verwaltung von Cloud-Technologien direkt von der Linux Foundation, dem Marktführer im Bereich Open Source.</p> + <br> + <a href="https://www.edx.org/course/introduction-to-cloud-infrastructure-technologies" target="_blank" class="button">Zum Kurs</a> + </center> + </div> + <div class="col-nav"> + <center> + <h5> + <b>Einführung in Linux</b> + </h5> + <p>Du hast nie Linux gelernt? Willst du eine Auffrischung? Erarbeite dir gute Linux-Kenntnisse über die grafische Oberfläche und die Kommandozeile der wichtigsten Linux-Distributionen.</p> + <br> + <a href="https://www.edx.org/course/introduction-to-linux" target="_blank" class="button">Zum Kurs</a> + </center> + </div> + </div> +</section> + +<div class="padded lighter-gray-bg"> + <div class="main-section two-thirds-centered"> + <center> + <h2>Mit der Linux Foundation lernen</h2> + <p>Die Linux Foundation bietet Kurse für alle Aspekte der Entwicklung und des Betriebs von Kubernetes-Anwendungen an, die entweder von Lehrkräften geleitet werden oder zum Selbststudium geeignet sind.</p> + <br/><br/> + <a href="https://training.linuxfoundation.org/training/course-catalog/?_sft_technology=kubernetes" target="_blank" class="button">Kurse anzeigen</a> + </center> + </div> +</div> + +<section id="get-certified"> + <div class="main-section padded"> + <h2>Werde Kubernetes zertifiziert</h2> + <div class="col-container"> + <div class="col-nav"> + <h5> + <b>Kubernetes and Cloud Native Associate (KCNA)</b> + </h5> + <p>Die Prüfung zum Kubernetes and Cloud Native Associate (KCNA) weist die grundlegenden Kenntnisse und Fähigkeiten eines Benutzers in Kubernetes und dem breiteren Cloud Native-Ökosystem nach.</p> + <p>Ein zertifizierter KCNA bestätigt konzeptionelles Wissen über das gesamte Cloud Native Ecosystem, mit besonderem Fokus auf Kubernetes.</p> + <br> + <a href="https://training.linuxfoundation.org/certification/kubernetes-cloud-native-associate/" target="_blank" class="button">Zur Zertifizierung</a> + </div> + <div class="col-nav"> + <h5> + <b>Certified Kubernetes Application Developer (CKAD)</b> + </h5> + <p>Die Prüfung zum Certified Kubernetes Application Developer (Zertifizierter Kubernetes-Anwendungsentwickler) bescheinigt, dass Teilnehmer Cloud Native-Anwendungen für Kubernetes entwerfen, erstellen, konfigurieren und bereitstellen können.</p> + <p>Ein CKAD kann Anwendungsressourcen definieren und zentrale Elemente verwenden, um skalierbare Anwendungen und Tools in Kubernetes zu erstellen, zu überwachen und Fehler zu beheben.</p> + <br> + <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-application-developer-ckad/" target="_blank" class="button">Zur Zertifizierung</a> + </div> + <div class="col-nav"> + <h5> + <b>Certified Kubernetes Administrator (CKA)</b> + </h5> + <p>Das Certified Kubernetes Administrator (CKA)-Programm garantiert, dass CKAs die Fähigkeiten, das Wissen und die Kompetenz besitzen, um die Aufgaben eines Kubernetes-Administrators zu erfüllen.</p> + <p>Ein zertifizierter Kubernetes-Administrator hat nachgewiesen, dass er in der Lage ist, grundlegende Installationen durchzuführen sowie Kubernetes-Cluster in einer Produktionsumgebung zu konfigurieren und zu verwalten.</p> + <br> + <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/" target="_blank" class="button">Zur Zertifizierung</a> + </div> + <div class="col-nav"> + <h5> + <b>Certified Kubernetes Security Specialist (CKS)</b> + </h5> + <p>Das Programm Certified Kubernetes Security Specialist (CKS) bietet die Gewissheit, dass der Zertifikatsinhaber mit einem breiten Spektrum an Best Practices vertraut ist und diese beherrscht. Die CKS-Zertifizierung umfasst Fähigkeiten zur Sicherung von Container-basierten Anwendungen und Kubernetes-Plattformen während der Erstellung, Bereitstellung und Laufzeit.</p> + <p><em>Kandidaten für den CKS müssen über eine aktuelle Zertifizierung als Certified Kubernetes Administrator (CKA) verfügen, um nachzuweisen, dass sie über ausreichende Kubernetes-Kenntnisse verfügen, bevor sie sich für den CKS anmelden.</em></p> + <br> + <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-security-specialist/" target="_blank" class="button">Zur Zertifizierung</a> + </div> + </div> + </div> +</section> + +<div class="padded lighter-gray-bg"> + <div class="main-section two-thirds-centered"> + <center> + <h2>Kubernetes Schulungspartner</h2> + <p>Unser Netzwerk von Kubernetes-Schulungspartnern bietet Schulungsangebote für Kubernetes- und Cloud Native-Projekte.</p> + </center> + </div> + <div class="main-section landscape-section"> + {{< cncf-landscape helpers=false category="kubernetes-training-partner" >}} + </div> +</div> diff --git a/content/en/_index.html b/content/en/_index.html index db4c966102..fbdd5d5720 100644 --- a/content/en/_index.html +++ b/content/en/_index.html @@ -16,7 +16,7 @@ It groups containers that make up an application into logical units for easy man {{% blocks/feature image="scalable" %}} #### Planet Scale -Designed on the same principles that allows Google to run billions of containers a week, Kubernetes can scale without increasing your ops team. +Designed on the same principles that allow Google to run billions of containers a week, Kubernetes can scale without increasing your operations team. {{% /blocks/feature %}} @@ -43,12 +43,12 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise <button id="desktopShowVideoButton" onclick="kub.showVideo()">Watch Video</button> <br> <br> - <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Attend KubeCon North America on October 11-15, 2021</a> + <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america" button id="desktopKCButton">Attend KubeCon North America on October 24-28, 2022</a> <br> <br> <br> <br> - <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22" button id="desktopKCButton">Attend KubeCon Europe on May 17-20, 2022</a> + <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/" button id="desktopKCButton">Attend KubeCon Europe on April 17-21, 2023</a> </div> <div id="videoPlayer"> <iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe> diff --git a/content/en/blog/_posts/2015-03-00-Welcome-To-Kubernetes-Blog.md b/content/en/blog/_posts/2015-03-00-Welcome-To-Kubernetes-Blog.md index 54e4fbe4ad..45502e0072 100644 --- a/content/en/blog/_posts/2015-03-00-Welcome-To-Kubernetes-Blog.md +++ b/content/en/blog/_posts/2015-03-00-Welcome-To-Kubernetes-Blog.md @@ -1,9 +1,13 @@ --- -title: Welcome to the Kubernetes Blog! +title: Welcome to the Kubernetes Blog! date: 2015-03-20 slug: welcome-to-kubernetes-blog url: /blog/2015/03/Welcome-To-Kubernetes-Blog +evergreen: true --- + +**Author:** Kit Merker (Google) + Welcome to the new Kubernetes Blog. Follow this blog to learn about the Kubernetes Open Source project. We plan to post release notes, how-to articles, events, and maybe even some off topic fun here from time to time. @@ -25,6 +29,3 @@ To start things off, here's a roundup of recent Kubernetes posts from other site Happy cloud computing! - - - - Kit Merker - Product Manager, Google Cloud Platform diff --git a/content/en/blog/_posts/2015-04-00-Faster-Than-Speeding-Latte.md b/content/en/blog/_posts/2015-04-00-Faster-Than-Speeding-Latte.md index fd6118b1c5..90a11d2582 100644 --- a/content/en/blog/_posts/2015-04-00-Faster-Than-Speeding-Latte.md +++ b/content/en/blog/_posts/2015-04-00-Faster-Than-Speeding-Latte.md @@ -1,8 +1,11 @@ --- -title: " Faster than a speeding Latte " +title: "Faster than a speeding Latte" date: 2015-04-06 slug: faster-than-speeding-latte url: /blog/2015/04/Faster-Than-Speeding-Latte +evergreen: true --- + Check out Brendan Burns racing Kubernetes. -[![Check out Brendan Burns racing Kubernetes](https://img.youtube.com/vi/7vZ9dRKRMyc/0.jpg)](https://www.youtube.com/watch?v=?7vZ9dRKRMyc) + +{{< youtube id="7vZ9dRKRMyc" title="Latte vs. Kubernetes setup - which is faster?">}} diff --git a/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0160.md b/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0160.md index abe97435eb..86dcc36bb9 100644 --- a/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0160.md +++ b/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0160.md @@ -1,8 +1,9 @@ --- -title: " Kubernetes Release: 0.16.0 " +title: "Kubernetes Release: 0.16.0" date: 2015-05-11 slug: kubernetes-release-0160 url: /blog/2015/05/Kubernetes-Release-0160 +evergreen: true --- Release Notes: diff --git a/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0170.md b/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0170.md index 29d86abe6d..0859edc059 100644 --- a/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0170.md +++ b/content/en/blog/_posts/2015-05-00-Kubernetes-Release-0170.md @@ -1,8 +1,9 @@ --- -title: " Kubernetes Release: 0.17.0 " +title: "Kubernetes Release: 0.17.0" date: 2015-05-15 slug: kubernetes-release-0170 url: /blog/2015/05/Kubernetes-Release-0170 +evergreen: true --- Release Notes: diff --git a/content/en/blog/_posts/2015-07-00-Announcing-First-Kubernetes-Enterprise.md b/content/en/blog/_posts/2015-07-00-Announcing-First-Kubernetes-Enterprise.md index 98b4c49d67..ae0b3f8f39 100644 --- a/content/en/blog/_posts/2015-07-00-Announcing-First-Kubernetes-Enterprise.md +++ b/content/en/blog/_posts/2015-07-00-Announcing-First-Kubernetes-Enterprise.md @@ -1,9 +1,10 @@ --- -title: " Announcing the First Kubernetes Enterprise Training Course " +title: "Announcing the First Kubernetes Enterprise Training Course" date: 2015-07-08 slug: announcing-first-kubernetes-enterprise url: /blog/2015/07/Announcing-First-Kubernetes-Enterprise --- + At Google we rely on Linux application containers to run our core infrastructure. Everything from Search to Gmail runs in containers.  In fact, we like containers so much that even our Google Compute Engine VMs run in containers!  Because containers are critical to our business, we have been working with the community on many of the basic container technologies (from cgroups to Docker’s LibContainer) and even decided to build the next generation of Google’s container scheduling technology, Kubernetes, in the open. diff --git a/content/en/blog/_posts/2015-11-00-Kubernetes-1-1-Performance-Upgrades-Improved-Tooling-And-A-Growing-Community.md b/content/en/blog/_posts/2015-11-00-Kubernetes-1-1-Performance-Upgrades-Improved-Tooling-And-A-Growing-Community.md index 754f6319e0..953504df01 100644 --- a/content/en/blog/_posts/2015-11-00-Kubernetes-1-1-Performance-Upgrades-Improved-Tooling-And-A-Growing-Community.md +++ b/content/en/blog/_posts/2015-11-00-Kubernetes-1-1-Performance-Upgrades-Improved-Tooling-And-A-Growing-Community.md @@ -1,9 +1,13 @@ --- -title: " Kubernetes 1.1 Performance upgrades, improved tooling and a growing community " +title: "Kubernetes 1.1 Performance upgrades, improved tooling and a growing community" date: 2015-11-09 slug: kubernetes-1-1-performance-upgrades-improved-tooling-and-a-growing-community url: /blog/2015/11/Kubernetes-1-1-Performance-Upgrades-Improved-Tooling-And-A-Growing-Community +evergreen: true --- + +**Author:** David Aronchick (Google) + Since the Kubernetes 1.0 release in July, we’ve seen tremendous adoption by companies building distributed systems to manage their container clusters. We’re also been humbled by the rapid growth of the community who help make Kubernetes better everyday. We have seen commercial offerings such as Tectonic by CoreOS and RedHat Atomic Host emerge to deliver deployment and support of Kubernetes. And a growing ecosystem has added Kubernetes support including tool vendors such as Sysdig and Project Calico. With the help of hundreds of contributors, we’re proud to announce the availability of Kubernetes 1.1, which offers major performance upgrades, improved tooling, and new features that make applications even easier to build and deploy. @@ -50,4 +54,3 @@ As we mentioned above, we would love your help: But, most of all, just let us know how you are transforming your business using Kubernetes, and how we can help you do it even faster. Thank you for your support! - - David Aronchick, Senior Product Manager for Kubernetes and Google Container Engine diff --git a/content/en/blog/_posts/2016-02-00-Kubecon-Eu-2016-Kubernetes-Community-In.md b/content/en/blog/_posts/2016-02-00-Kubecon-Eu-2016-Kubernetes-Community-In.md index 38ee6129fd..c0995e1c14 100644 --- a/content/en/blog/_posts/2016-02-00-Kubecon-Eu-2016-Kubernetes-Community-In.md +++ b/content/en/blog/_posts/2016-02-00-Kubecon-Eu-2016-Kubernetes-Community-In.md @@ -1,37 +1,41 @@ --- -title: " KubeCon EU 2016: Kubernetes Community in London " +title: "KubeCon EU 2016: Kubernetes Community in London" date: 2016-02-24 slug: kubecon-eu-2016-kubernetes-community-in url: /blog/2016/02/Kubecon-Eu-2016-Kubernetes-Community-In +evergreen: true --- -KubeCon EU 2016 is the inaugural [European Kubernetes](http://kubernetes.io/) community conference that follows on the American launch in November 2015. KubeCon is fully dedicated to education and community engagement for[Kubernetes](http://kubernetes.io/) enthusiasts, production users and the surrounding ecosystem. +**Author:** Sarah Novotny (Google) + +KubeCon EU 2016 is the inaugural European Kubernetes community conference that follows on the American launch in November 2015. KubeCon is fully dedicated to education and community engagement for [Kubernetes](/) enthusiasts, production users and the surrounding ecosystem. Come join us in London and hang out with hundreds from the Kubernetes community and experience a wide variety of deep technical expert talks and use cases. Don’t miss these great speaker sessions at the conference: -* “Kubernetes Hardware Hacks: Exploring the Kubernetes API Through Knobs, Faders, and Sliders” by Ian Lewis and Brian Dorsey, Developer Advocate, Google -* [http://sched.co/6Bl3](http://sched.co/6Bl3) +* “Kubernetes Hardware Hacks: Exploring the Kubernetes API Through Knobs, Faders, and Sliders” by Ian Lewis and Brian Dorsey, Developer Advocate, Google [https://sched.co/6Bl3](http://sched.co/6Bl3) -* “rktnetes: what's new with container runtimes and Kubernetes” by Jonathan Boulle, Developer and Team Lead at CoreOS -* [http://sched.co/6BY7](http://sched.co/6BY7) +* “rktnetes: what's new with container runtimes and Kubernetes” by Jonathan Boulle, Developer and Team Lead at CoreOS [https://sched.co/6BY7](http://sched.co/6BY7) -* “Kubernetes Documentation: Contributing, fixing issues, collecting bounties” by John Mulhausen, Lead Technical Writer, Google -* [http://sched.co/6BUP](http://sched.co/6BUP)  -* “[What is OpenStack's role in a Kubernetes world?](https://kubeconeurope2016.sched.org/event/6BYC/what-is-openstacks-role-in-a-kubernetes-world?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” By Thierry Carrez, Director of Engineering, OpenStack Foundation -* http://sched.co/6BYC -* “A Practical Guide to Container Scheduling” by Mandy Waite, Developer Advocate, Google -* [http://sched.co/6BZa](http://sched.co/6BZa) +* “Kubernetes Documentation: Contributing, fixing issues, collecting bounties” by John Mulhausen, Lead Technical Writer, Google [https://sched.co/6BUP](http://sched.co/6BUP)  -* “[Kubernetes in Production in The New York Times newsroom](https://kubeconeurope2016.sched.org/event/67f2/kubernetes-in-production-in-the-new-york-times-newsroom?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” Eric Lewis, Web Developer, New York Times -* [http://sched.co/67f2](http://sched.co/67f2) -* “[Creating an Advanced Load Balancing Solution for Kubernetes with NGINX](https://kubeconeurope2016.sched.org/event/6Bc9/creating-an-advanced-load-balancing-solution-for-kubernetes-with-nginx?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” by Andrew Hutchings, Technical Product Manager, NGINX -* http://sched.co/6Bc9 -* And many more http://kubeconeurope2016.sched.org/ +* “[What is OpenStack's role in a Kubernetes world?](https://kubeconeurope2016.sched.org/event/6BYC/what-is-openstacks-role-in-a-kubernetes-world?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” By Thierry Carrez, Director of Engineering, OpenStack Foundation [https://sched.co/6BYC](http://sched.co/6BYC) + +* “A Practical Guide to Container Scheduling” by Mandy Waite, Developer Advocate, Google [https://sched.co/6BZa](http://sched.co/6BZa) + +* “[Kubernetes in Production in The New York Times newsroom](https://kubeconeurope2016.sched.org/event/67f2/kubernetes-in-production-in-the-new-york-times-newsroom?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” Eric Lewis, Web Developer, New York Times [https://sched.co/67f2](http://sched.co/67f2) + +* “[Creating an Advanced Load Balancing Solution for Kubernetes with NGINX](https://kubeconeurope2016.sched.org/event/6Bc9/creating-an-advanced-load-balancing-solution-for-kubernetes-with-nginx?iframe=yes&w=i:0;&sidebar=yes&bg=no#?iframe=yes&w=i:100;&sidebar=yes&bg=no)” by Andrew Hutchings, Technical Product Manager, NGINX [https://sched.co/6Bc9](https://sched.co/6Bc9) + +…and many more https://kubeconeurope2016.sched.org/ -Get your KubeCon EU [tickets here](https://ti.to/kubecon/kubecon-eu-2016). +~Get your KubeCon EU [tickets here](https://ti.to/kubecon/kubecon-eu-2016)~. Venue Location: CodeNode * 10 South Pl, London, United Kingdom Accommodations: [hotels](https://skillsmatter.com/contact-us#hotels) Website: [kubecon.io](https://www.kubecon.io/) Twitter: [@KubeConio](https://twitter.com/kubeconio) #KubeCon + Google is a proud Diamond sponsor of KubeCon EU 2016. Come to London next month, March 10th & 11th, and visit booth #13 to learn all about Kubernetes, Google Container Engine (GKE) and Google Cloud Platform! - -_KubeCon is organized by KubeAcademy, LLC, a community-driven group of developers focused on the education of developers and the promotion of Kubernetes._ - --* Sarah Novotny, Kubernetes Community Manager, Google diff --git a/content/en/blog/_posts/2016-03-00-Kubernetes-1-2-Even-More-Performance-Upgrades-Plus-Easier-Application-Deployment-And-Management-.md b/content/en/blog/_posts/2016-03-00-Kubernetes-1-2-Even-More-Performance-Upgrades-Plus-Easier-Application-Deployment-And-Management-.md index c63763d4e5..bc9083bc0d 100644 --- a/content/en/blog/_posts/2016-03-00-Kubernetes-1-2-Even-More-Performance-Upgrades-Plus-Easier-Application-Deployment-And-Management-.md +++ b/content/en/blog/_posts/2016-03-00-Kubernetes-1-2-Even-More-Performance-Upgrades-Plus-Easier-Application-Deployment-And-Management-.md @@ -1,17 +1,20 @@ --- -title: " Kubernetes 1.2: Even more performance upgrades, plus easier application deployment and management " +title: "Kubernetes 1.2: Even more performance upgrades, plus easier application deployment and management" date: 2016-03-17 slug: kubernetes-1.2-even-more-performance-upgrades-plus-easier-application-deployment-and-management url: /blog/2016/03/Kubernetes-1-2-Even-More-Performance-Upgrades-Plus-Easier-Application-Deployment-And-Management +evergreen: true --- -Today we released Kubernetes 1.2. This release represents significant improvements for large organizations building distributed systems. Now with over 680 unique contributors to the project, this release represents our largest yet. +**Author:** David Aronchick (Google) + +Today the Kubernetes project released Kubernetes 1.2. This release represents significant improvements for large organizations building distributed systems. Now with over 680 unique contributors to the project, this release represents our largest yet. From the beginning, our mission has been to make building distributed systems easy and accessible for all. With the Kubernetes 1.2 release we’ve made strides towards our goal by increasing scale, decreasing latency and overall simplifying the way applications are deployed and managed. Now, developers at organizations of all sizes can build production scale apps more easily than ever before.  -### What’s new:  +## What’s new - **Significant scale improvements**. Increased cluster scale by 400% to 1,000 nodes and 30,000 containers per cluster. -- **Simplified application deployment and management**.  +- **Simplified application deployment and management**. - Dynamic Configuration (via the ConfigMap API) enables applications to pull their configuration when they run rather than packaging it in at build time.  - Turnkey Deployments (via the Beta Deployment API) let you declare your application and Kubernetes will do the rest. It handles versioning, multiple simultaneous rollouts, aggregating status across all pods, maintaining application availability and rollback.  @@ -28,15 +31,15 @@ From the beginning, our mission has been to make building distributed systems ea - **And many more**. For a complete list of updates, see the [release notes on github](https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0).  -#### Community  +## Community -All these improvements would not be possible without our enthusiastic and global community. The momentum is astounding. We’re seeing over 400 pull requests per week, a 50% increase since the previous 1.1 release. There are meetups and conferences discussing Kubernetes nearly every day, on top of the 85 Kubernetes related [meetup groups](http://www.meetup.com/topics/kubernetes/) around the world. We’ve also seen significant participation in the community in the form of Special Interest Groups, with 18 active SIGs that cover topics from AWS and OpenStack to big data and scalability, to get involved [join or start a new SIG](https://github.com/kubernetes/kubernetes/wiki/Special-Interest-Groups-(SIGs)). Lastly, we’re proud that Kubernetes is the first project to be accepted to the Cloud Native Computing Foundation (CNCF), read more about the announcement [here](https://cncf.io/news/announcement/2016/03/cloud-native-computing-foundation-accepts-kubernetes-first-hosted-projec-0).  +All these improvements would not be possible without our enthusiastic and global community. The momentum is astounding. We’re seeing over 400 pull requests per week, a 50% increase since the previous 1.1 release. There are meetups and conferences discussing Kubernetes nearly every day, on top of the 85 Kubernetes related [meetup groups](http://www.meetup.com/topics/kubernetes/) around the world. We’ve also seen significant participation in the community in the form of Special Interest Groups, with 18 active SIGs that cover topics from AWS and OpenStack to big data and scalability, to get involved [join or start a new SIG](https://github.com/kubernetes/kubernetes/wiki/Special-Interest-Groups-(SIGs)). Lastly, we’re proud that Kubernetes is the first project to be accepted to the Cloud Native Computing Foundation (CNCF), read more about the announcement [here](https://cncf.io/news/announcement/2016/03/cloud-native-computing-foundation-accepts-kubernetes-first-hosted-projec-0). -#### Documentation  +## Documentation -With Kubernetes 1.2 comes a relaunch of our website at [kubernetes.io](http://kubernetes.io/). We’ve slimmed down the docs contribution process so that all you have to do is fork/clone and send a PR. And the site works the same whether you’re staging it on your laptop, on github.io, or viewing it in production. It’s a pure GitHub Pages project; no scripts, no plugins.  +With Kubernetes 1.2 comes a relaunch of our website at [kubernetes.io](http://kubernetes.io/). We’ve slimmed down the docs contribution process so that all you have to do is fork/clone and send a PR. And the site works the same whether you’re staging it on your laptop, on github.io, or viewing it in production. It’s a pure GitHub Pages project; no scripts, no plugins. @@ -48,7 +51,7 @@ To entice you even further to contribute, we’re also announcing our new bounty -#### Roadmap  +## Roadmap All of our work is done in the open, to learn the latest about the project j[oin the weekly community meeting](https://groups.google.com/forum/#!forum/kubernetes-community-video-chat) or [watch a recorded hangout](https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ). In keeping with our major release schedule of every three to four months, here are just a few items that are in development for [next release and beyond](https://github.com/kubernetes/kubernetes/wiki/Release-1.3):  @@ -64,7 +67,7 @@ Kubernetes 1.2 is available for download at [get.k8s.io](http://get.k8s.io/) and -#### Connect  +## Connect We’d love to hear from you and see you participate in this growing community:  @@ -73,8 +76,7 @@ We’d love to hear from you and see you participate in this growing community:& -  Connect with the community on [Slack](http://slack.kubernetes.io/)  - Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for latest updates  -Thank you for your support!  +Thank you for your support! - - _David Aronchick, Senior Product Manager for Kubernetes, Google_ diff --git a/content/en/blog/_posts/2016-04-00-Kubernetes-Network-Policy-APIs.md b/content/en/blog/_posts/2016-04-00-Kubernetes-Network-Policy-APIs.md index 7289f57669..b2a5561dd9 100644 --- a/content/en/blog/_posts/2016-04-00-Kubernetes-Network-Policy-APIs.md +++ b/content/en/blog/_posts/2016-04-00-Kubernetes-Network-Policy-APIs.md @@ -131,7 +131,7 @@ In this example, the ‘ **tenant-a** ’ namespace would get policy ‘ **pol1* -Today, [Romana](http://romana.io/), [OpenShift](https://www.openshift.com/), [OpenContrail](http://www.opencontrail.org/) and [Calico](http://projectcalico.org/) support network policies applied to namespaces and pods. Cisco and VMware are working on implementations as well. Both Romana and Calico demonstrated these capabilities with Kubernetes 1.2 recently at KubeCon. You can watch their presentations here: [Romana](https://www.youtube.com/watch?v=f-dLKtK6qCs) ([slides](http://www.slideshare.net/RomanaProject/kubecon-london-2016-ronana-cloud-native-sdn)), [Calico](https://www.youtube.com/watch?v=p1zfh4N4SX0) ([slides](http://www.slideshare.net/kubecon/kubecon-eu-2016-secure-cloudnative-networking-with-project-calico)).  +Today, Romana, OpenShift, OpenContrail and Calico support network policies applied to namespaces and pods. Cisco and VMware are working on implementations as well. Both Romana and Calico demonstrated these capabilities with Kubernetes 1.2 recently at KubeCon. You can watch their presentations here: [Romana](https://www.youtube.com/watch?v=f-dLKtK6qCs) ([slides](http://www.slideshare.net/RomanaProject/kubecon-london-2016-ronana-cloud-native-sdn)), [Calico](https://www.youtube.com/watch?v=p1zfh4N4SX0) ([slides](http://www.slideshare.net/kubecon/kubecon-eu-2016-secure-cloudnative-networking-with-project-calico)).  diff --git a/content/en/blog/_posts/2016-07-00-Kubernetes-1-3-Bridging-Cloud-Native-And-Enterprise-Workloads.md b/content/en/blog/_posts/2016-07-00-Kubernetes-1-3-Bridging-Cloud-Native-And-Enterprise-Workloads.md index 13e682c0a8..bc86fa5eeb 100644 --- a/content/en/blog/_posts/2016-07-00-Kubernetes-1-3-Bridging-Cloud-Native-And-Enterprise-Workloads.md +++ b/content/en/blog/_posts/2016-07-00-Kubernetes-1-3-Bridging-Cloud-Native-And-Enterprise-Workloads.md @@ -1,9 +1,13 @@ --- -title: " Kubernetes 1.3: Bridging Cloud Native and Enterprise Workloads " +title: "Kubernetes 1.3: Bridging Cloud Native and Enterprise Workloads" date: 2016-07-06 slug: kubernetes-1.3-bridging-cloud-native-and-enterprise-workloads url: /blog/2016/07/Kubernetes-1-3-Bridging-Cloud-Native-And-Enterprise-Workloads +evergreen: true --- + +**Author:** Aparna Sinha, Google + Nearly two years ago, when we officially kicked off the Kubernetes project, we wanted to simplify distributed systems management and provide the core technology required to everyone. The community’s response to this effort has blown us away. Today, thousands of customers, partners and developers are running clusters in production using Kubernetes and have joined the cloud native revolution.  Thanks to the help of over 800 contributors, we are pleased to announce today the availability of Kubernetes 1.3, our most robust and feature-rich release to date. @@ -14,7 +18,7 @@ Product highlights in Kubernetes 1.3 include the ability to bridge services acro -**What’s new:** +## What’s new - **Increased scale and automation** - Customers want to scale their services up and down automatically in response to application demand. In 1.3 we have made it easier to autoscale clusters up and down while doubling the maximum number of nodes per cluster. Customers no longer need to think about cluster size, and can allow the underlying cluster to respond to demand. @@ -31,13 +35,13 @@ Product highlights in Kubernetes 1.3 include the ability to bridge services acro - **Updated Kubernetes dashboard UI** - Customers can now use the Kubernetes open source dashboard for the majority of interactions with their clusters, rather than having to use the CLI. The updated UI lets users control, edit and create all workload resources (including Deployments and PetSets). - And many more. For a complete list of updates, see the [_release notes on GitHub_](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.0). -**Community** +## Community We could not have achieved this milestone without the tireless effort of countless people that are part of the Kubernetes community. We have [19 different Special Interest Groups](https://github.com/kubernetes/community/blob/master/README.md#special-interest-groups-sig), and over 100 meetups around the world. Kubernetes is a community project, built in the open, and it truly would not be possible without the over 233 person-years of effort the community has put in to date. Woot! -**Availability** +## Availability Kubernetes 1.3 is available for download at [get.k8s.io](http://get.k8s.io/) and via the open source repository hosted on [GitHub](http://github.com/kubernetes/kubernetes). To get started with Kubernetes try our [Hello World app](/docs/hellonode/). @@ -47,7 +51,7 @@ To learn the latest about the project, we encourage everyone to [join the weekly -**Connect** +## Connect We’d love to hear from you and see you participate in this growing community: @@ -58,8 +62,4 @@ We’d love to hear from you and see you participate in this growing community: -Thank you for your support!  - - - --- Aparna Sinha, Product Manager, Google +Thank you for your support! diff --git a/content/en/blog/_posts/2016-09-00-High-Performance-Network-Policies-Kubernetes.md b/content/en/blog/_posts/2016-09-00-High-Performance-Network-Policies-Kubernetes.md index bc21d1edaa..c8c452bc67 100644 --- a/content/en/blog/_posts/2016-09-00-High-Performance-Network-Policies-Kubernetes.md +++ b/content/en/blog/_posts/2016-09-00-High-Performance-Network-Policies-Kubernetes.md @@ -65,7 +65,7 @@ Network policies are an exciting feature, which the Kubernetes community has wor -There are only a few policy-capable networking backends available for Kubernetes today: [Romana](http://romana.io/), [Calico](http://projectcalico.org/), and [Canal](https://github.com/tigera/canal); with [Weave](http://www.weave.works/) indicating support in the near future. Red Hat’s OpenShift includes network policy features as well. +There are only a few policy-capable networking backends available for Kubernetes today: Romana, [Calico](http://projectcalico.org/), and [Canal](https://github.com/tigera/canal); with [Weave](http://www.weave.works/) indicating support in the near future. Red Hat’s OpenShift includes network policy features as well. @@ -100,7 +100,7 @@ This is because during a typical network performance benchmark, there’s no app - Hardware: Two servers with Intel Core i5-5250U CPUs (2 core, 2 threads per core) running at 1.60GHz, 16GB RAM and 512GB SSD. NIC: Intel Ethernet Connection I218-V (rev 03) - Ubuntu 14.04.5 - Kubernetes 1.3 for data collection (verified samples on [v1.4.0-beta.5](http://v1.4.0-beta.5/)) -- [Romana v0.9.3.1](https://github.com/romana/romana) +- Romana v0.9.3.1 - Client and server load test [software](https://github.com/paninetworks/testing-tools) For the tests we had a client pod send 2,000 HTTP requests to a server pod. HTTP requests were sent by the client pod at a rate that ensured that neither the server nor network ever saturated. We also made sure each request started a new TCP session by disabling persistent connections (i.e. HTTP [keep-alive](https://en.wikipedia.org/wiki/HTTP_persistent_connection)). We ran each test with different response sizes and measured the average request duration time (how long does it take to complete a request of that size). Finally, we repeated each set of measurements with different policy configurations. @@ -189,4 +189,4 @@ These tests were performed using Romana as the backend policy provider and other -If you wish to try it for yourself, we invite you to check out [Romana](http://romana.io/). In our [GitHub repo](https://github.com/romana/romana) you can find an easy to use installer, which works with AWS, Vagrant VMs or any other servers. You can use it to quickly get you started with a Romana powered Kubernetes or OpenStack cluster. +If you wish to try it for yourself, we invite you to check out Romana. In our GitHub repo you can find an easy to use installer, which works with AWS, Vagrant VMs or any other servers. You can use it to quickly get you started with a Romana powered Kubernetes or OpenStack cluster. diff --git a/content/en/blog/_posts/2017-06-00-Kubernetes-1-7-Security-Hardening-Stateful-Application-Extensibility-Updates.md b/content/en/blog/_posts/2017-06-00-Kubernetes-1-7-Security-Hardening-Stateful-Application-Extensibility-Updates.md index e8f15ec6b9..e80f557011 100644 --- a/content/en/blog/_posts/2017-06-00-Kubernetes-1-7-Security-Hardening-Stateful-Application-Extensibility-Updates.md +++ b/content/en/blog/_posts/2017-06-00-Kubernetes-1-7-Security-Hardening-Stateful-Application-Extensibility-Updates.md @@ -19,7 +19,7 @@ Security: - [Node authorizer](/docs/reference/access-authn-authz/node/) and admission control plugin are new additions that restrict kubelet’s access to secrets, pods and other objects based on its node. - [Encryption for Secrets](/docs/tasks/administer-cluster/encrypt-data/), and other resources in etcd, is now available as alpha.  - [Kubelet TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/) now supports client and server certificate rotation. -- [Audit logs](/docs/tasks/debug-application-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit. +- [Audit logs](/docs/tasks/debug/debug-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit. Stateful workloads: diff --git a/content/en/blog/_posts/2017-08-00-High-Performance-Networking-With-Ec2.md b/content/en/blog/_posts/2017-08-00-High-Performance-Networking-With-Ec2.md index b735c501e7..181af0aa58 100644 --- a/content/en/blog/_posts/2017-08-00-High-Performance-Networking-With-Ec2.md +++ b/content/en/blog/_posts/2017-08-00-High-Performance-Networking-With-Ec2.md @@ -9,7 +9,7 @@ url: /blog/2017/08/High-Performance-Networking-With-Ec2 One of the most popular platforms for running Kubernetes is Amazon Web Services’ Elastic Compute Cloud (AWS EC2). With more than a decade of experience delivering IaaS, and expanding over time to include a rich set of services with easy to consume APIs, EC2 has captured developer mindshare and loyalty worldwide. -When it comes to networking, however, EC2 has some limits that hinder performance and make deploying Kubernetes clusters to production unnecessarily complex. The preview release of [Romana v2.0](http://romana.io/), a network and security automation solution for Cloud Native applications, includes features that address some well known network issues when running Kubernetes in EC2. +When it comes to networking, however, EC2 has some limits that hinder performance and make deploying Kubernetes clusters to production unnecessarily complex. The preview release of Romana v2.0, a network and security automation solution for Cloud Native applications, includes features that address some well known network issues when running Kubernetes in EC2. ## Traditional VPC Networking Performance Roadblocks @@ -40,7 +40,7 @@ Whether you were interested in advanced networking for traffic isolation or runn The way to avoid running out of VPC routes is to use them sparingly by making them forward pod traffic for multiple instances. From a networking perspective, what that means is that the VPC route needs to forward to a router, which can then forward traffic on to the final destination instance. -[Romana](http://romana.io/) is a CNI network provider that configures routes on the host to forward pod network traffic without an overlay. Since inter-node routes are installed on hosts, no VPC routes are necessary at all. However, when the VPC is split into subnets for an HA deployment across zones, VPC routes are necessary. +Romana is a CNI network provider that configures routes on the host to forward pod network traffic without an overlay. Since inter-node routes are installed on hosts, no VPC routes are necessary at all. However, when the VPC is split into subnets for an HA deployment across zones, VPC routes are necessary. Fortunately, inter-node routes on hosts allows them to act as a network router and forward traffic inbound from another zone just as it would for traffic from local pods. This makes any Kubernetes node configured by Romana able to accept inbound pod traffic from other zones and forward it to the proper destination node on the subnet. @@ -73,8 +73,5 @@ When using Romana v2.0, native VPC networking is now available for clusters of a ![](https://archive.org/download/hpc-ec2-vpc-2/hpc-ec2-vpc-2.png) -The preview release of Romana v2.0 is available [here](http://romana.io/preview). We welcome comments and feedback so we can make EC2 deployments of Kubernetes as fast and reliable as possible. - - -- _Juergen Brendel and Chris Marino, co-founders of Pani Networks, sponsor of the Romana project_ diff --git a/content/en/blog/_posts/2017-09-00-Kubernetes-18-Security-Workloads-And.md b/content/en/blog/_posts/2017-09-00-Kubernetes-18-Security-Workloads-And.md index d8f09da298..e2f031f8a5 100644 --- a/content/en/blog/_posts/2017-09-00-Kubernetes-18-Security-Workloads-And.md +++ b/content/en/blog/_posts/2017-09-00-Kubernetes-18-Security-Workloads-And.md @@ -3,8 +3,10 @@ title: " Kubernetes 1.8: Security, Workloads and Feature Depth " date: 2017-09-29 slug: kubernetes-18-security-workloads-and url: /blog/2017/09/Kubernetes-18-Security-Workloads-And +evergreen: true --- -_Editor's note: today's post is by Aparna Sinha, Group Product Manager, Kubernetes, Google; Ihor Dvoretskyi, Developer Advocate, CNCF; Jaice Singer DuMars, Kubernetes Ambassador, Microsoft; and Caleb Miles, Technical Program Manager, CoreOS on the latest release of Kubernetes 1.8._ + +**Authors:** Kubernetes v1.8 release team We’re pleased to announce the delivery of Kubernetes 1.8, our third release this year. Kubernetes 1.8 represents a snapshot of many exciting enhancements and refinements underway. In addition to functional improvements, we’re increasing project-wide focus on maturing [process](https://github.com/kubernetes/sig-release), formalizing [architecture](https://github.com/kubernetes/community/tree/master/sig-architecture), and strengthening Kubernetes’ [governance model](https://github.com/kubernetes/community/tree/master/community/elections/2017). The evolution of mature processes clearly signals that sustainability is a driving concern, and helps to ensure that Kubernetes is a viable and thriving project far into the future. @@ -50,7 +52,7 @@ The [Release team](https://github.com/kubernetes/features/blob/master/release-1. As the Kubernetes community has grown, our release process has become an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid clip. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. -## User Highlights +## User highlights According to [Redmonk](http://redmonk.com/fryan/2017/09/10/cloud-native-technologies-in-the-fortune-100/), 54 percent of Fortune 100 companies are running Kubernetes in some form with adoption coming from every sector across the world. Recent user stories from the community include: @@ -91,3 +93,6 @@ The simplest way to get involved with Kubernetes is by joining one of the many [ - Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for latest updates - Chat with the community on [Slack](http://slack.k8s.io/). - [Share your Kubernetes story.](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform) + + +_Editor's note: this announcement was authored by Aparna Sinha (Google), Ihor Dvoretskyi (CNCF), Jaice Singer DuMars (Microsoft), and Caleb Miles (CoreOS)._ diff --git a/content/en/blog/_posts/2017-11-00-Containerd-Container-Runtime-Options-Kubernetes.md b/content/en/blog/_posts/2017-11-00-Containerd-Container-Runtime-Options-Kubernetes.md index 2a105c6396..7bfcebe705 100644 --- a/content/en/blog/_posts/2017-11-00-Containerd-Container-Runtime-Options-Kubernetes.md +++ b/content/en/blog/_posts/2017-11-00-Containerd-Container-Runtime-Options-Kubernetes.md @@ -4,7 +4,12 @@ date: 2017-11-02 slug: containerd-container-runtime-options-kubernetes url: /blog/2017/11/Containerd-Container-Runtime-Options-Kubernetes --- - **_Editor's note: Today's post is by Lantao Liu, Software Engineer at Google, and Mike Brown, Open Source Developer Advocate at IBM._** + +**Authors:** Lantao Liu (Google), and Mike Brown (IBM) + +_Update: Kubernetes support for Docker via `dockershim` is now deprecated. +For more information, read the [deprecation notice](/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation). +You can also discuss the deprecation via a dedicated [GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917)._ A _container runtime_ is software that executes containers and manages container images on a node. Today, the most widely known container runtime is [Docker](https://www.docker.com/), but there are other container runtimes in the ecosystem, such as [rkt](https://coreos.com/rkt/), [containerd](https://containerd.io/), and [lxd](https://linuxcontainers.org/lxd/). Docker is by far the most common container runtime used in production Kubernetes environments, but Docker’s smaller offspring, containerd, may prove to be a better option. This post describes using containerd with Kubernetes. diff --git a/content/en/blog/_posts/2017-12-00-Kubernetes-19-Workloads-Expanded-Ecosystem.md b/content/en/blog/_posts/2017-12-00-Kubernetes-19-Workloads-Expanded-Ecosystem.md index cb73d95162..cefcf21452 100644 --- a/content/en/blog/_posts/2017-12-00-Kubernetes-19-Workloads-Expanded-Ecosystem.md +++ b/content/en/blog/_posts/2017-12-00-Kubernetes-19-Workloads-Expanded-Ecosystem.md @@ -1,9 +1,13 @@ --- -title: " Kubernetes 1.9: Apps Workloads GA and Expanded Ecosystem " +title: "Kubernetes 1.9: Apps Workloads GA and Expanded Ecosystem" date: 2017-12-15 slug: kubernetes-19-workloads-expanded-ecosystem url: /blog/2017/12/Kubernetes-19-Workloads-Expanded-Ecosystem +evergreen: true --- + +**Authors:** Kubernetes v1.9 release team + We’re pleased to announce the delivery of Kubernetes 1.9, our fourth and final release this year. Today’s release continues the evolution of an increasingly rich feature set, more robust stability, and even greater community contributions. As the fourth release of the year, it gives us an opportunity to look back at the progress made in key areas. Particularly notable is the advancement of the Apps Workloads API to stable. This removes any reservations potential adopters might have had about the functional stability required to run mission-critical workloads. Another big milestone is the beta release of Windows support, which opens the door for many Windows-specific applications and workloads to run in Kubernetes, significantly expanding the implementation scenarios and enterprise readiness of Kubernetes. @@ -87,7 +91,7 @@ For recorded sessions from the largest Kubernetes gathering, [KubeCon + CloudNat ## Webinar -Join members of the Kubernetes 1.9 release team on **January 9th from 10am-11am PT** to learn about the major features in this release as they demo some of the highlights in the areas of Windows and Docker support, storage, admission control, and the workloads API. [Register here](https://zoom.us/webinar/register/WN_oVjQMwyzQFOmWsfVzDsa2A). +Join members of the Kubernetes 1.9 release team on **January 9th from 10am-11am PT** to learn about the major features in this release as they demo some of the highlights in the areas of Windows and Docker support, storage, admission control, and the workloads API. [~Register here~](https://zoom.us/webinar/register/WN_oVjQMwyzQFOmWsfVzDsa2A). ## Get involved: diff --git a/content/en/blog/_posts/2017-12-00-Using-Ebpf-In-Kubernetes.md b/content/en/blog/_posts/2017-12-00-Using-Ebpf-In-Kubernetes.md index 09dd0068f5..cc896a4948 100644 --- a/content/en/blog/_posts/2017-12-00-Using-Ebpf-In-Kubernetes.md +++ b/content/en/blog/_posts/2017-12-00-Using-Ebpf-In-Kubernetes.md @@ -117,7 +117,7 @@ To achieve the best possible isolation, each function call would have to happen By using Landlock, we could isolate function calls from each other within the same container, making a temporary file created by one function call inaccessible to the next function call, for example. Integration between Landlock and technologies like Kubernetes-based serverless frameworks would be a ripe area for further exploration. ## Auditing kubectl-exec with eBPF -In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug-application-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster. +In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug/debug-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster. While these events log Kubernetes events, they don't currently provide the level of visibility that some may require. For example, while we can see that someone has used `kubectl exec` to enter a container, we are not able to see what commands were executed in that session. With eBPF one can attach a BPF program that would record any commands executed in the `kubectl exec` session and pass those commands to a user-space program that logs those events. We could then play that session back and know the exact sequence of events that took place. ## Learn more about eBPF diff --git a/content/en/blog/_posts/2018-03-26-kubernetes-1-10-stabilizing-storage-security-networking.md b/content/en/blog/_posts/2018-03-26-kubernetes-1-10-stabilizing-storage-security-networking.md index 922ec9f706..904b2d34e9 100644 --- a/content/en/blog/_posts/2018-03-26-kubernetes-1-10-stabilizing-storage-security-networking.md +++ b/content/en/blog/_posts/2018-03-26-kubernetes-1-10-stabilizing-storage-security-networking.md @@ -1,13 +1,9 @@ --- -title: 'Kubernetes 1.10: Stabilizing Storage, Security, and Networking ' -author: kbarnard -tags: +title: 'Kubernetes 1.10: Stabilizing Storage, Security, and Networking' date: 2018-03-26 modified_time: '2018-03-27T11:01:39.569-07:00' -blogger_id: tag:blogger.com,1999:blog-112706738355446097.post-6519705795358457586 -blogger_orig_url: https://kubernetes.io/blog/2018/03/26/kubernetes-1.10-stabilizing-storage-security-networking/ slug: kubernetes-1.10-stabilizing-storage-security-networking -date: 2018-03-26 +evergreen: true --- ***Editor's note: today's post is by the [1.10 Release diff --git a/content/en/blog/_posts/2018-06-26-kubernetes-1-11-release-announcement.md b/content/en/blog/_posts/2018-06-26-kubernetes-1-11-release-announcement.md index a5d9f8b4d0..4d3007f290 100644 --- a/content/en/blog/_posts/2018-06-26-kubernetes-1-11-release-announcement.md +++ b/content/en/blog/_posts/2018-06-26-kubernetes-1-11-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.11: In-Cluster Load Balancing and CoreDNS Plugin Graduate to General Availability' date: 2018-06-27 slug: kubernetes-1.11-release-announcement +evergreen: true --- **Author**: Kubernetes 1.11 [Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.11/release_team.md) diff --git a/content/en/blog/_posts/2018-07-11-dynamic-kubelet-configuration.md b/content/en/blog/_posts/2018-07-11-dynamic-kubelet-configuration.md index ed3914f3e5..4f5f548cee 100644 --- a/content/en/blog/_posts/2018-07-11-dynamic-kubelet-configuration.md +++ b/content/en/blog/_posts/2018-07-11-dynamic-kubelet-configuration.md @@ -6,6 +6,8 @@ date: 2018-07-11 **Author**: Michael Taufen (Google) +**Editor’s note: The feature has been removed in the version 1.24 after deprecation in 1.22.** + **Editor’s note: this post is part of a [series of in-depth articles](https://kubernetes.io/blog/2018/06/27/kubernetes-1.11-release-announcement/) on what’s new in Kubernetes 1.11** ## Why Dynamic Kubelet Configuration? diff --git a/content/en/blog/_posts/2018-07-18-11-ways-not-to-get-hacked.md b/content/en/blog/_posts/2018-07-18-11-ways-not-to-get-hacked.md index 329d81cf56..cf72ba30f3 100644 --- a/content/en/blog/_posts/2018-07-18-11-ways-not-to-get-hacked.md +++ b/content/en/blog/_posts/2018-07-18-11-ways-not-to-get-hacked.md @@ -66,7 +66,7 @@ There are plenty of [good examples](https://docs.bitnami.com/kubernetes/how-to/c Incorrect or excessively permissive RBAC policies are a security threat in case of a compromised pod. Maintaining least privilege, and continuously reviewing and improving RBAC rules, should be considered part of the "technical debt hygiene" that teams build into their development lifecycle. -[Audit Logging](/docs/tasks/debug-application-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation's security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started. +[Audit Logging](/docs/tasks/debug/debug-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation's security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started. For read requests such as get, list, and watch, only the request object is saved in the audit logs; the response object is not. For requests involving sensitive data such as Secret and ConfigMap, only the metadata is exported. For all other requests, both request and response objects are saved in audit logs. diff --git a/content/en/blog/_posts/2018-08-03-make-kubernetes-production-grade-anywhere.md b/content/en/blog/_posts/2018-08-03-make-kubernetes-production-grade-anywhere.md index 329b2c4de7..00416256e0 100644 --- a/content/en/blog/_posts/2018-08-03-make-kubernetes-production-grade-anywhere.md +++ b/content/en/blog/_posts/2018-08-03-make-kubernetes-production-grade-anywhere.md @@ -174,7 +174,7 @@ Cluster-distributed stateful services (e.g., Cassandra) can benefit from splitti ## Other considerations -[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location. +[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location. Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git. diff --git a/content/en/blog/_posts/2018-09-27-kubernetes-1-12-release-announcement.md b/content/en/blog/_posts/2018-09-27-kubernetes-1-12-release-announcement.md index b24a6cbe09..1f847a38ca 100644 --- a/content/en/blog/_posts/2018-09-27-kubernetes-1-12-release-announcement.md +++ b/content/en/blog/_posts/2018-09-27-kubernetes-1-12-release-announcement.md @@ -2,6 +2,7 @@ layout: blog title: 'Kubernetes 1.12: Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability' date: 2018-09-27 +evergreen: true --- **Author**: The 1.12 [Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.12/release_team.md) diff --git a/content/en/blog/_posts/2018-10-01-health-checking-grpc.md b/content/en/blog/_posts/2018-10-01-health-checking-grpc.md index 21eb668dc2..7fbde70524 100644 --- a/content/en/blog/_posts/2018-10-01-health-checking-grpc.md +++ b/content/en/blog/_posts/2018-10-01-health-checking-grpc.md @@ -6,6 +6,10 @@ date: 2018-10-01 **Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google) +**Update (December 2021):** _Kubernetes now has built-in gRPC health probes starting in v1.23. +To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe). +This article was originally written about an external tool to achieve the same task._ + [gRPC](https://grpc.io) is on its way to becoming the lingua franca for communication between cloud-native microservices. If you are deploying gRPC applications to Kubernetes today, you may be wondering about the best way to diff --git a/content/en/blog/_posts/2018-10-16-kubernetes-2018-north-american-contributor-summit.md b/content/en/blog/_posts/2018-10-16-kubernetes-2018-north-american-contributor-summit.md index 095e47a73d..13158529a2 100644 --- a/content/en/blog/_posts/2018-10-16-kubernetes-2018-north-american-contributor-summit.md +++ b/content/en/blog/_posts/2018-10-16-kubernetes-2018-north-american-contributor-summit.md @@ -1,13 +1,12 @@ --- -layout: "Blog" +layout: blog title: "Kubernetes 2018 North American Contributor Summit" -date: 2018-10-16 +date: 2018-10-16 +evergreen: true --- **Authors:** -[Bob Killen][bob] (University of Michigan) -[Sahdev Zala][sahdev] (IBM), -[Ihor Dvoretskyi][ihor] (CNCF) +[Bob Killen][bob] (University of Michigan), [Sahdev Zala][sahdev] (IBM), [Ihor Dvoretskyi][ihor] (CNCF) The 2018 North American Kubernetes Contributor Summit to be hosted right before @@ -22,32 +21,34 @@ Unlike previous Contributor Summits, the event now spans two-days with a more relaxed ‘hallway’ track and general Contributor get-together to be hosted from 5-8pm on Sunday December 9th at the [Garage Lounge and Gaming Hall][garage], just a short walk away from the Convention Center. There, contributors can enjoy -billiards, bowling, trivia and more; accompanied by a variety of food and drink. +billiards, bowling, trivia and more; accompanied by a variety of food and drink. Things pick up the following day, Monday the 10th with three separate tracks: -### New Contributor Workshop: +### New contributor workshop A half day workshop aimed at getting new and first time contributors onboarded and comfortable with working within the Kubernetes Community. Staying for the duration is required; this is not a workshop you can drop into. -### Current Contributor Track: +### Current contributor track Reserved for those that are actively engaged with the development of the project; the Current Contributor Track includes Talks, Workshops, Birds of a Feather, Unconferences, Steering Committee Sessions, and more! Keep an eye on the [schedule in GitHub][schedule] as content is frequently being updated. -### Docs Sprint: -SIG-Docs will have a curated list of issues and challenges to be tackled closer +### Docs sprint + +SIG Docs will have a curated list of issues and challenges to be tackled closer to the event date. -## To Register: +## How To Register {#to-register} + To register for the Contributor Summit, see the [Registration section of the Event Details in GitHub][register]. Please note that registrations are being reviewed. If you select the “Current Contributor Track” and are not an active contributor, you will be asked to attend the New Contributor Workshop, or asked to be put on a waitlist. With thousands of contributors and only 300 spots, we -need to make sure the right folks are in the room. +need to make sure the right folks are in the room. If you have any questions or concerns, please don’t hesitate to reach out to the Contributor Summit Events Team at community@kubernetes.io. diff --git a/content/en/blog/_posts/2018-11-08-kubernetes-docs-update-i18n.md b/content/en/blog/_posts/2018-11-08-kubernetes-docs-update-i18n.md index 38fd24372c..4c9af11a9e 100644 --- a/content/en/blog/_posts/2018-11-08-kubernetes-docs-update-i18n.md +++ b/content/en/blog/_posts/2018-11-08-kubernetes-docs-update-i18n.md @@ -41,7 +41,7 @@ These repo labels let reviewers filter for PRs and issues by language. For examp ### Team review -L10n teams can now review and approve their own PRs. For example, review and approval permissions for English are [assigned in an OWNERS file](https://github.com/kubernetes/website/blob/master/content/en/OWNERS) in the top subfolder for English content. +L10n teams can now review and approve their own PRs. For example, review and approval permissions for English are [assigned in an OWNERS file](https://github.com/kubernetes/website/blob/main/content/en/OWNERS) in the top subfolder for English content. Adding `OWNERS` files to subdirectories lets localization teams review and approve changes without requiring a rubber stamp approval from reviewers who may lack fluency. diff --git a/content/en/blog/_posts/2018-12-03-kubernetes-1-13-release-announcement.md b/content/en/blog/_posts/2018-12-03-kubernetes-1-13-release-announcement.md index 8aba0dc232..3e76486764 100644 --- a/content/en/blog/_posts/2018-12-03-kubernetes-1-13-release-announcement.md +++ b/content/en/blog/_posts/2018-12-03-kubernetes-1-13-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.13: Simplified Cluster Management with Kubeadm, Container Storage Interface (CSI), and CoreDNS as Default DNS are Now Generally Available' date: 2018-12-03 slug: kubernetes-1-13-release-announcement +evergreen: true --- **Author**: The 1.13 [Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.13/release_team.md) diff --git a/content/en/blog/_posts/2018-12-04-kubeadm-ga-release.md b/content/en/blog/_posts/2018-12-04-kubeadm-ga-release.md index 8601a7b94d..7ce4106076 100644 --- a/content/en/blog/_posts/2018-12-04-kubeadm-ga-release.md +++ b/content/en/blog/_posts/2018-12-04-kubeadm-ga-release.md @@ -2,6 +2,7 @@ layout: blog title: Production-Ready Kubernetes Cluster Creation with kubeadm date: 2018-12-04 +evergreen: true --- **Authors**: Lucas Käldström (CNCF Ambassador) and Luc Perkins (CNCF Developer Advocate) diff --git a/content/en/blog/_posts/2019-02-11-runc-CVE-2019-5736.md b/content/en/blog/_posts/2019-02-11-runc-CVE-2019-5736.md index 84482daf79..027cc2e9bf 100644 --- a/content/en/blog/_posts/2019-02-11-runc-CVE-2019-5736.md +++ b/content/en/blog/_posts/2019-02-11-runc-CVE-2019-5736.md @@ -1,17 +1,20 @@ --- title: Runc and CVE-2019-5736 date: 2019-02-11 +evergreen: false # mentions PodSecurityPolicy --- +Authors: Kubernetes Product Security Committee + This morning [a container escape vulnerability in runc was announced](https://www.openwall.com/lists/oss-security/2019/02/11/2). We wanted to provide some guidance to Kubernetes users to ensure everyone is safe and secure. -## What Is Runc? +## What is runc? Very briefly, runc is the low-level tool which does the heavy lifting of spawning a Linux container. Other tools like Docker, Containerd, and CRI-O sit on top of runc to deal with things like data formatting and serialization, but runc is at the heart of all of these systems. Kubernetes in turn sits on top of those tools, and so while no part of Kubernetes itself is vulnerable, most Kubernetes installations are using runc under the hood. -### What Is The Vulnerability? +### What is the vulnerability? While full details are still embargoed to give people time to patch, the rough version is that when running a process as root (UID 0) inside a container, that process can exploit a bug in runc to gain root privileges on the host running the container. This then allows them unlimited access to the server as well as any other containers on that server. @@ -19,13 +22,14 @@ If the process inside the container is either trusted (something you know is not The most common source of risk is attacker-controller container images, such as unvetted images from public repositories. -### What Should I Do? +### What should i do? As with all security issues, the two main options are to mitigate the vulnerability or upgrade your version of runc to one that includes the fix. As the exploit requires UID 0 within the container, a direct mitigation is to ensure all your containers are running as a non-0 user. This can be set within the container image, or via your pod specification: ```yaml +--- apiVersion: v1 kind: Pod metadata: @@ -39,6 +43,7 @@ spec: This can also be enforced globally using a PodSecurityPolicy: ```yaml +--- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: @@ -89,7 +94,7 @@ We don't have specific confirmation that Docker for Mac and Docker for Windows a If you are unable to upgrade Docker, the Rancher team has provided backports of the fix for many older versions at [github.com/rancher/runc-cve](https://github.com/rancher/runc-cve). -## Getting More Information +## Getting more information If you have any further questions about how this vulnerability impacts Kubernetes, please join us at [discuss.kubernetes.io](https://discuss.kubernetes.io/). diff --git a/content/en/blog/_posts/2019-03-20-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits.md b/content/en/blog/_posts/2019-03-20-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits.md index 37b16daf4c..99a8dcd886 100644 --- a/content/en/blog/_posts/2019-03-20-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits.md +++ b/content/en/blog/_posts/2019-03-20-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits.md @@ -1,13 +1,13 @@ --- title: A Look Back and What's in Store for Kubernetes Contributor Summits date: 2019-03-20 +layout: blog +evergreen: true --- **Authors:** Paris Pittman (Google), Jonas Rosland (VMware) -**tl;dr** - [click here] for Barcelona Contributor Summit information. - {{<figure width="600" src="/images/blog/2019-03-14-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits/celebrationsig.jpg" caption="Seattle Contributor Summit">}} As our contributing community grows in great numbers, with more than 16,000 contributors this year across 150+ GitHub repositories, it’s important to provide face to face connections for our large distributed teams to have opportunities for collaboration and learning. In [Contributor Experience], our methodology with planning events is a lot like our documentation; we build from personas -- interests, skills, and motivators to name a few. This way we ensure there is valuable content and learning for everyone. @@ -28,13 +28,14 @@ We build the contributor summits around you: These personas combined with ample feedback from previous events, produce the altogether experience that welcomed over 600 contributors in Copenhagen (May), Shanghai(November), and Seattle(December) in 2018. Seattle's event drew over 300+ contributors, equal to Shanghai and Copenhagen combined, for the 6th contributor event in Kubernetes history. In true Kubernetes fashion, we expect another record breaking year of attendance. We've pre-ordered 900+ [contributor patches], a tradition, and we are looking forward to giving them to you! -With that said... +With that said… + **Save the Dates:** Barcelona: May 19th (evening) and 20th (all day) Shanghai: June 24th (all day) San Diego: November 18th, 19th, and activities in KubeCon/CloudNativeCon week -In an effort of continual improvement, here's what to expect from us this year: +In an effort of continual improvement, here's what to expect from us this year: * Large new contributor workshops and contributor socials at all three events expected to break previous attendance records * A multiple track event in San Diego for all contributor types including workshops, birds of a feather, lightning talks and more @@ -42,7 +43,8 @@ In an effort of continual improvement, here's what to expect from us this year: * [An event website]! * Follow along with updates: kubernetes-dev@googlegroups.com is our main communication hub as always; however, we will also blog here, our [Thursday Kubernetes Community Meeting], [twitter], SIG meetings, event site, discuss.kubernetes.io, and #contributor-summit on Slack. * Opportunities to get involved: We still have 2019 roles available! -Reach out to Contributor Experience via community@kubernetes.io, stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex). +Reach out to Contributor Experience via community@kubernetes.io, stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex). + {{<figure width="600" src="/images/blog/2019-03-14-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits/unconference.jpg" caption="Unconference voting">}} @@ -51,11 +53,11 @@ Reach out to Contributor Experience via community@kubernetes.io, stop by a Wedne Our 2018 crew 🥁 Jorge Castro, Paris Pittman, Bob Killen, Jeff Sica, Megan Lehn, Guinevere Saenger, Josh Berkus, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Lindsey Tulloch, Zach Corleissen, Tim Pepper, Ihor Dvoretskyi, Nancy Mohamed, Chris Short, Mario Loria, Jason DeTiberus, Sahdev Zala, Mithra Raja -And an introduction to our 2019 crew (a thanks in advance ;) )... -Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster +And an introduction to our 2019 crew (a thanks in advance ;) )… +Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster -## Relive Seattle Contributor Summit +## Relive Seattle Contributor Summit 📈 80% growth rate since the Austin 2017 December event @@ -81,15 +83,11 @@ Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, 📸 Pictures (special thanks to [rdodev]) -Garage Pic -Reg Desk - {{<figure width="600" src="/images/blog/2019-03-14-A-Look-Back-And-Whats-In-Store-For-Kubernetes-Contributor-Summits/grouppicseatle.JPG" caption="Some of the group in Seattle">}} “I love Contrib Summit! The intros and deep dives during KubeCon were a great extension of Contrib Summit. Y'all did an excellent job in the morning to level set expectations and prime everyone.” -- julianv “great work! really useful and fun!” - coffeepac -[click here]: https://events.linuxfoundation.org/events/contributor-summit-europe-2019/ [Contributor Experience]: https://github.com/kubernetes/community/tree/master/sig-contributor-experience [Subproject OWNERs]: https://github.com/kubernetes/community/blob/master/community-membership.md [Chair or Tech Lead]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance.md diff --git a/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md b/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md index 448309ba3a..e021bbf1db 100644 --- a/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md +++ b/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md @@ -2,6 +2,7 @@ title: 'Kubernetes 1.14: Production-level support for Windows Nodes, Kubectl Updates, Persistent Local Volumes GA' date: 2019-03-25 slug: kubernetes-1-14-release-announcement +evergreen: true --- **Authors:** The 1.14 [Release Team](https://bit.ly/k8s114-team) diff --git a/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md b/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md index 12dd2d79a3..bed35dabb5 100644 --- a/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md +++ b/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md @@ -144,10 +144,9 @@ ways to address it. - Specifically add an iptables rule to drop the packets that are marked as *INVALID*, so it won’t reach to client pod and cause harm. -The fix is drafted (https://github.com/kubernetes/kubernetes/pull/74840), but -unfortunately it didn’t catch the v1.14 release window. However, for the users -that are affected by this bug, there is a way to mitigate the problem by applying -the following rule in your cluster. +The [fix](https://github.com/kubernetes/kubernetes/pull/74840) is available in v1.15+. +However, for the users that are affected by this bug, there is a way to mitigate the +problem by applying the following rule in your cluster. ```yaml apiVersion: extensions/v1beta1 diff --git a/content/en/blog/_posts/2019-04-26-latest-on-localization.md b/content/en/blog/_posts/2019-04-26-latest-on-localization.md index 8e6176a8ce..98f1d74706 100644 --- a/content/en/blog/_posts/2019-04-26-latest-on-localization.md +++ b/content/en/blog/_posts/2019-04-26-latest-on-localization.md @@ -8,7 +8,7 @@ date: 2019-04-26 Last year we optimized the Kubernetes website for [hosting multilingual content](/blog/2018/11/08/kubernetes-docs-updates-international-edition/). Contributors responded by adding multiple new localizations: as of April 2019, Kubernetes docs are partially available in nine different languages, with six added in 2019 alone. You can see a list of available languages in the language selector at the top of each page. -By _partially available_, I mean that localizations are ongoing projects. They range from mostly complete ([Chinese docs for 1.12](https://v1-12.docs.kubernetes.io/zh/)) to brand new (1.14 docs in [Portuguese](https://kubernetes.io/pt/)). If you're interested in helping an existing localization, read on! +By _partially available_, I mean that localizations are ongoing projects. They range from mostly complete ([Chinese docs for 1.12](https://v1-12.docs.kubernetes.io/zh-cn/)) to brand new (1.14 docs in [Portuguese](https://kubernetes.io/pt/)). If you're interested in helping an existing localization, read on! ## What is a localization? diff --git a/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md b/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md index 41c5fbcf36..bf8417673b 100644 --- a/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md +++ b/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md @@ -2,6 +2,7 @@ title: "Join us for the 2019 KubeCon Diversity Lunch & Hack" date: 2019-05-02 slug: kubecon-diversity-lunch-and-hack +evergreen: false --- **Authors:** Kiran Oliver, Podcast Producer, The New Stack @@ -36,4 +37,4 @@ To make this all possible, we need you. Yes, you, to register. As much as we lov We look forward to seeing you! -_Special thanks to [Leah Petersen](https://www.linkedin.com/in/leahstunts/), [Sarah Conway](https://www.linkedin.com/in/sarah-conway-6166151/) and [Paris Pittman](https://www.linkedin.com/in/parispittman/) for their help in editing this post._ +_Special thanks to [Leah Petersen](https://www.linkedin.com/in/leahstunts/), [Sarah Conway](https://www.linkedin.com/in/sarah-conway-6166151/) and [Paris Pittman](https://www.linkedin.com/in/parispittman/) for their help in editing this post._ diff --git a/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md b/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md index 3561203548..83dfb6dd50 100644 --- a/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md +++ b/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: "Kubernetes 1.15: Extensibility and Continuous Improvement" date: 2019-06-19 slug: kubernetes-1-15-release-announcement +evergreen: true --- **Authors:** The 1.15 [Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.15/release_team.md) diff --git a/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md b/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md index 350fbb1735..62ea9c8337 100644 --- a/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md +++ b/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md @@ -3,19 +3,18 @@ layout: blog title: "Contributor Summit San Diego Registration Open!" date: 2019-09-24 slug: san-diego-contributor-summit +evergreen: true --- -**Authors: Paris Pittman (Google), Jeffrey Sica (Red Hat), Jonas Rosland (VMware)** - - +**Authors:** Paris Pittman (Google), Jeffrey Sica (Red Hat), Jonas Rosland (VMware) [Contributor Summit San Diego 2019 Event Page] -Registration is now open and in record time, we’ve hit capacity for the -*new contributor workshop* session of the event! Waitlist is now available. +In record time, we’ve hit capacity for the *new contributor workshop* session of +the event! **Sunday, November 17** Evening Contributor Celebration: -[QuartYard]* +[QuartYard]† Address: 1301 Market Street, San Diego, CA 92101 Time: 6:00PM - 9:00PM @@ -68,7 +67,7 @@ Check out past blogs on [persona building around our events] and the [Barcelona ![Group Picture in 2018](/images/blog/2019-09-24-san-diego-contributor-summit/IMG_2588.JPG) -*=QuartYard has a huge stage! Want to perform something in front of your contributor peers? Reach out to us! community@kubernetes.io +†=QuartYard has a huge stage! Want to perform something in front of your contributor peers? Reach out to us! community@kubernetes.io diff --git a/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md b/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md index e78b6a0ec7..f0f2835513 100644 --- a/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md +++ b/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md @@ -5,13 +5,7 @@ date: 2019-10-10 slug: contributor-summit-san-diego-schedule --- - -Authors: Josh Berkus (Red Hat), Paris Pittman (Google), Jonas Rosland (VMware) - -tl;dr A week ago we announced that [registration is open][reg] for the contributor -summit , and we're now live with [the full Contributor Summit schedule!][schedule] -Grab your spot while tickets are still available. There is currently a waitlist -for new contributor workshop. ([Register here!][reg]) +**Authors:** Josh Berkus (Red Hat), Paris Pittman (Google), Jonas Rosland (VMware) There are many great sessions planned for the Contributor Summit, spread across five rooms of current contributor content in addition to the new contributor @@ -32,7 +26,7 @@ While the schedule contains difficult decisions in every timeslot, we've picked a few below to give you a taste of what you'll hear, see, and participate in, at the summit: -* **[Vision]**: SIG-Architecture will be sharing their vision of where we're going +* **[Vision]**: SIG Architecture will be sharing their vision of where we're going with Kubernetes development for the next year and beyond. * **[Security]**: Tim Allclair and CJ Cullen will present on the current state of Kubernetes security. In another security talk, Vallery Lancey will lead a @@ -47,7 +41,7 @@ the summit: one, or at least pass one. * **[End Users]**: Several end users from the CNCF partner ecosystem, invited by Cheryl Hung, will hold a Q&A with contributors to strengthen our feedback loop. -* **[Docs]**: As always, SIG-Docs will run a three-hour contributing-to-documentation +* **[Docs]**: As always, SIG Docs will run a three-hour contributing-to-documentation workshop. We're also giving out awards to contributors who distinguished themselves in 2019, diff --git a/content/en/blog/_posts/2019-12-09-kubernetes-1.17-release-announcement.md b/content/en/blog/_posts/2019-12-09-kubernetes-1.17-release-announcement.md index 983d7ba31e..6fb460acfb 100644 --- a/content/en/blog/_posts/2019-12-09-kubernetes-1.17-release-announcement.md +++ b/content/en/blog/_posts/2019-12-09-kubernetes-1.17-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: "Kubernetes 1.17: Stability" date: 2019-12-09T13:00:00-08:00 slug: kubernetes-1-17-release-announcement +evergreen: true --- **Authors:** [Kubernetes 1.17 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.17/release_team.md) diff --git a/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md b/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md index ae05bd8b9c..11e3ccc412 100644 --- a/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md +++ b/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md @@ -5,11 +5,28 @@ date: 2020-02-18 slug: Contributor-Summit-Amsterdam-Schedule-Announced --- -**Authors:** Jeffrey Sica (Red Hat), Amanda Katona (VMware) +**Authors:** Jeffrey Sica (Red Hat), Amanda Katona (VMware) -tl;dr [Registration is open](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/) and the [schedule is live](https://kcseu2020.sched.com/) so register now and we’ll see you in Amsterdam! +![Contributor Summit](/images/blog/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced/contribsummit.jpg) -## Kubernetes Contributor Summit +Hello everyone and Happy 2020! It’s hard to believe that KubeCon EU 2020 is less than six weeks away, and with that another contributor summit! This year we have the pleasure of being in Amsterdam in early spring, so be sure to pack some warmer clothing. This summit looks to be exciting with a lot of fantastic community-driven content. We received **26** submissions from the CFP. From that, the events team selected **12** sessions. Each of the sessions falls into one of four categories: + +* Community +* Contributor Improvement +* Sustainability +* In-depth Technical + +On top of the presentations, there will be a dedicated Docs Sprint as well as the New Contributor Workshop 101 and 201 Sessions. All told, we will have five separate rooms of content throughout the day on Monday. Please **[see the full schedule](https://kcseu2020.sched.com/)** to see what sessions you’d be interested in. We hope between the content provided and the inevitable hallway track, everyone has a fun and enriching experience. + +Speaking of fun, the social Sunday night should be a blast! We’re hosting this summit’s social close to the conference center, at [ZuidPool](https://www.zuid-pool.nl/en/). There will be games, bingo, and unconference sign-up throughout the evening. It should be a relaxed way to kick off the week. + +[~Registration is open~](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/)! Space is limited so it’s always a good idea to register early. + +If you have any questions, reach out to the [Amsterdam Team](https://github.com/kubernetes/community/tree/master/events/2020/03-contributor-summit#team) on Slack in the [#contributor-summit](https://kubernetes.slack.com/archives/C7J893413) channel. + +Hope to see you there! + +## Kubernetes Contributor Summit schedule **Sunday, March 29, 2020** @@ -25,21 +42,3 @@ tl;dr [Registration is open](https://events.linuxfoundation.org/kubernetes-contr - Address: [Europaplein 24, 1078 GZ Amsterdam, Netherlands](https://www.google.com/search?q=kubecon+amsterdam+2020&oq=kubecon+amste&aqs=chrome.0.35i39j69i57j0l4j69i61l2.3957j1j4&sourceid=chrome&ie=UTF-8&ibp=htl;events&rciv=evn&sa=X&ved=2ahUKEwiZoLvQ0dvnAhVST6wKHScBBZ8Q5bwDMAB6BAgSEAE#) - Time: 09:00 - 17:00 (Breakfast at 08:00) -![Contributor Summit](/images/blog/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced/contribsummit.jpg) - -Hello everyone and Happy 2020! It’s hard to believe that KubeCon EU 2020 is less than six weeks away, and with that another contributor summit! This year we have the pleasure of being in Amsterdam in early spring, so be sure to pack some warmer clothing. This summit looks to be exciting with a lot of fantastic community-driven content. We received **26** submissions from the CFP. From that, the events team selected **12** sessions. Each of the sessions falls into one of four categories: - -* Community -* Contributor Improvement -* Sustainability -* In-depth Technical - -On top of the presentations, there will be a dedicated Docs Sprint as well as the New Contributor Workshop 101 and 201 Sessions. All told, we will have five separate rooms of content throughout the day on Monday. Please **[see the full schedule](https://kcseu2020.sched.com/)** to see what sessions you’d be interested in. We hope between the content provided and the inevitable hallway track, everyone has a fun and enriching experience. - -Speaking of fun, the social Sunday night should be a blast! We’re hosting this summit’s social close to the conference center, at [ZuidPool](https://www.zuid-pool.nl/en/). There will be games, bingo, and unconference sign-up throughout the evening. It should be a relaxed way to kick off the week. - -[Registration is open](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/)! Space is limited so it’s always a good idea to register early. - -If you have any questions, reach out to the [Amsterdam Team](https://github.com/kubernetes/community/tree/master/events/2020/03-contributor-summit#team) on Slack in the [#contributor-summit](https://kubernetes.slack.com/archives/C7J893413) channel. - -Hope to see you there! diff --git a/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md b/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md index 1996b6201e..10d96ec351 100644 --- a/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md +++ b/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md @@ -3,13 +3,14 @@ layout: blog title: Contributor Summit Amsterdam Postponed date: 2020-03-04 slug: Contributor-Summit-Delayed +evergreen: true --- -**Authors:** Dawn Foster (VMware), Jorge Castro (VMware) +**Authors:** Dawn Foster (VMware), Jorge Castro (VMware) The CNCF has announced that [KubeCon + CloudNativeCon EU has been delayed](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/attend/novel-coronavirus-update/) until July/August of 2020. As a result the Contributor Summit planning team is weighing options for how to proceed. Here’s the current plan: - There will be an in-person Contributor Summit as planned when KubeCon + CloudNativeCon is rescheduled. -- We are looking at options for having additional virtual contributor activities in the meantime. +- We are looking at options for having additional virtual contributor activities in the meantime. We will communicate via this blog and the usual communications channels on the final plan. Please bear with us as we adapt when we get more information. Thank you for being patient as the team pivots to bring you a great Contributor Summit! \ No newline at end of file diff --git a/content/en/blog/_posts/2020-03-25-kubernetes-1.18-release-announcement.md b/content/en/blog/_posts/2020-03-25-kubernetes-1.18-release-announcement.md index d4fb5dc7df..a65924b174 100644 --- a/content/en/blog/_posts/2020-03-25-kubernetes-1.18-release-announcement.md +++ b/content/en/blog/_posts/2020-03-25-kubernetes-1.18-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.18: Fit & Finish' date: 2020-03-25 slug: kubernetes-1-18-release-announcement +evergreen: true --- **Authors:** [Kubernetes 1.18 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md) diff --git a/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md b/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md index dfed13ca43..eab08ed299 100644 --- a/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md +++ b/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md @@ -67,7 +67,7 @@ Let's see an example of a cluster to understand this API. As the feature name "PodTopologySpread" implies, the basic usage of this feature is to run your workload with an absolute even manner (maxSkew=1), or relatively even manner (maxSkew>=2). See the [official -document](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) +document](/docs/concepts/scheduling-eviction/topology-spread-constraints/) for more details. In addition to this basic usage, there are some advanced usage examples that diff --git a/content/en/blog/_posts/2020-05-21-wsl2-dockerdesktop-k8s.md b/content/en/blog/_posts/2020-05-21-wsl2-dockerdesktop-k8s.md index 0a84075aaf..cfc0cc53fc 100644 --- a/content/en/blog/_posts/2020-05-21-wsl2-dockerdesktop-k8s.md +++ b/content/en/blog/_posts/2020-05-21-wsl2-dockerdesktop-k8s.md @@ -360,7 +360,7 @@ So let's fix the issue by installing the missing package: sudo apt install -y conntrack ``` -![minikube-install-conntrack](/images/blog/2020-05-21-wsl2-dockerdesktop-k8s/wsl2-minikube-install conntrack.png) +![minikube-install-conntrack](/images/blog/2020-05-21-wsl2-dockerdesktop-k8s/wsl2-minikube-install-conntrack.png) Let's try to launch it again: diff --git a/content/en/blog/_posts/2020-08-26-kubernetes-release-1.19.md b/content/en/blog/_posts/2020-08-26-kubernetes-release-1.19.md index 6f7a4c4096..39559ffe4c 100644 --- a/content/en/blog/_posts/2020-08-26-kubernetes-release-1.19.md +++ b/content/en/blog/_posts/2020-08-26-kubernetes-release-1.19.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.19: Accentuate the Paw-sitive' date: 2020-08-26 slug: kubernetes-release-1.19-accentuate-the-paw-sitive +evergreen: true --- **Authors:** [Kubernetes 1.19 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.19/release_team.md) diff --git a/content/en/blog/_posts/2020-09-03-warnings/index.md b/content/en/blog/_posts/2020-09-03-warnings/index.md index 50576c0329..a5cfb9f710 100644 --- a/content/en/blog/_posts/2020-09-03-warnings/index.md +++ b/content/en/blog/_posts/2020-09-03-warnings/index.md @@ -3,9 +3,10 @@ layout: blog title: "Warning: Helpful Warnings Ahead" date: 2020-09-03 slug: warnings +evergreen: true --- -**Author**: Jordan Liggitt (Google) +**Author**: [Jordan Liggitt](https://github.com/liggitt) (Google) As Kubernetes maintainers, we're always looking for ways to improve usability while preserving compatibility. As we develop features, triage bugs, and answer support questions, we accumulate information that would be helpful for Kubernetes users to know. @@ -59,7 +60,7 @@ so we added two administrator-facing tools to help track use of deprecated APIs Starting in Kubernetes v1.19, when a request is made to a deprecated REST API endpoint, an `apiserver_requested_deprecated_apis` gauge metric is set to `1` in the kube-apiserver process. This metric has labels for the API `group`, `version`, `resource`, and `subresource`, -and a `removed_version` label that indicates the Kubernetes release in which the API will no longer be served. +and a `removed_release` label that indicates the Kubernetes release in which the API will no longer be served. This is an example query using `kubectl`, [prom2json](https://github.com/prometheus/prom2json), and [jq](https://stedolan.github.io/jq/) to determine which deprecated APIs have been requested @@ -168,7 +169,7 @@ You can also find that information through the following Prometheus query, which returns information about requests made to deprecated APIs which will be removed in v1.22: ```promql -apiserver_requested_deprecated_apis{removed_version="1.22"} * on(group,version,resource,subresource) +apiserver_requested_deprecated_apis{removed_release="1.22"} * on(group,version,resource,subresource) group_right() apiserver_request_total ``` @@ -176,7 +177,7 @@ group_right() apiserver_request_total Metrics are a fast way to check whether deprecated APIs are being used, and at what rate, but they don't include enough information to identify particular clients or API objects. -Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/) +Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug/debug-cluster/audit/) for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`. Administrators can use those audit events to identify specific clients or objects that need to be updated. @@ -327,7 +328,3 @@ A couple areas we're looking at next are warning about [known problematic values we cannot reject outright for compatibility reasons, and warning about use of deprecated fields or field values (like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)). I'm excited to see progress in this area, continuing to make it easier to use Kubernetes. - ---- - -_[Jordan Liggitt](https://twitter.com/liggitt) is a software engineer at Google, and helps lead Kubernetes authentication, authorization, and API efforts._ \ No newline at end of file diff --git a/content/en/blog/_posts/2020-09-16-gsoc‘20 -building-operators-for-cluster-addons.md b/content/en/blog/_posts/2020-09-16-gsoc20-building-operators-for-cluster-addons.md similarity index 100% rename from content/en/blog/_posts/2020-09-16-gsoc‘20 -building-operators-for-cluster-addons.md rename to content/en/blog/_posts/2020-09-16-gsoc20-building-operators-for-cluster-addons.md diff --git a/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md b/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md index 0fab185f98..dc4b97db2a 100644 --- a/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md +++ b/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md @@ -70,7 +70,7 @@ To correct the latter issue, we now employ a "hunt and peck" approach to removin ### 1. Upgrade to kubernetes 1.18 and make use of Pod Topology Spread Constraints While this seems like it could have been the perfect solution, at the time of writing Kubernetes 1.18 was unavailable on the two most common managed Kubernetes services in public cloud, EKS and GKE. -Furthermore, [pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) were still a [beta feature in 1.18](https://v1-18.docs.kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) which meant that it [wasn't guaranteed to be available in managed clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#kubernetes_feature_choices) even when v1.18 became available. +Furthermore, [pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) were still a beta feature in 1.18 which meant that it [wasn't guaranteed to be available in managed clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#kubernetes_feature_choices) even when v1.18 became available. The entire endeavour was concerningly reminiscent of checking [caniuse.com](https://caniuse.com/) when Internet Explorer 8 was still around. ### 2. Deploy a statefulset _per zone_. diff --git a/content/en/blog/_posts/2020-10-12-steering-committee-results.md b/content/en/blog/_posts/2020-10-12-steering-committee-results.md index 2acbb6d6e4..b47d9674f3 100644 --- a/content/en/blog/_posts/2020-10-12-steering-committee-results.md +++ b/content/en/blog/_posts/2020-10-12-steering-committee-results.md @@ -29,7 +29,7 @@ They join continuing members Christoph Blecker ([@cblecker](https://github.com/c * Josh Berkus ([@jberkus](https://github.com/jberkus)), Red Hat * Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community: * Aaron Crickenberger ([@spiffxp](https://github.com/spiffxp)), Google - * and Lachlan Evenson([@lachie8e)](https://github.com/lachie8e)), Microsoft + * and Lachlan Evenson([@lachie83)](https://github.com/lachie83)), Microsoft * And thank you to all the candidates who came forward to run for election. As [Jorge Castro put it](https://twitter.com/castrojo/status/1315718627639820288?s=20): we are spoiled with capable, kind, and selfless volunteers who put the needs of the project first. ## Get Involved with the Steering Committee diff --git a/content/en/blog/_posts/2020-11-02-remembering-dan-kohn.md b/content/en/blog/_posts/2020-11-02-remembering-dan-kohn.md index b8ffb8686a..579567750a 100644 --- a/content/en/blog/_posts/2020-11-02-remembering-dan-kohn.md +++ b/content/en/blog/_posts/2020-11-02-remembering-dan-kohn.md @@ -3,6 +3,7 @@ layout: blog title: "Remembering Dan Kohn" date: 2020-11-02 slug: remembering-dan-kohn +evergreen: true --- **Author**: The Kubernetes Steering Committee diff --git a/content/en/blog/_posts/2020-11-18-cloud-native-security-for-your-cluster/index.md b/content/en/blog/_posts/2020-11-18-cloud-native-security-for-your-cluster/index.md index 900f8b07ee..212f66bf37 100644 --- a/content/en/blog/_posts/2020-11-18-cloud-native-security-for-your-cluster/index.md +++ b/content/en/blog/_posts/2020-11-18-cloud-native-security-for-your-cluster/index.md @@ -18,9 +18,9 @@ The paper attempts to _not_ focus on any specific [cloud native project](https:/ ## Kubernetes native security controls When using Kubernetes as a workload orchestrator, some of the security controls this version of the whitepaper recommends are: -* [Pod Security Policies](/docs/concepts/policy/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster +* [Pod Security Policies](/docs/concepts/security/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster * [Resource requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits): Apply requests (soft constraint) and limits (hard constraint) for shared resources such as memory and CPU -* [Audit log analysis](/docs/tasks/debug-application-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events +* [Audit log analysis](/docs/tasks/debug/debug-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events * [Control plane authentication and certificate root of trust](/docs/concepts/architecture/control-plane-node-communication/): Enable mutual TLS authentication with a trusted CA for communication within the cluster * [Secrets management](/docs/concepts/configuration/secret/): Integrate with a built-in or external secrets store diff --git a/content/en/blog/_posts/2020-12-02-dockershim-faq.md b/content/en/blog/_posts/2020-12-02-dockershim-faq.md index edcab9fe53..4711207b1c 100644 --- a/content/en/blog/_posts/2020-12-02-dockershim-faq.md +++ b/content/en/blog/_posts/2020-12-02-dockershim-faq.md @@ -3,15 +3,19 @@ layout: blog title: "Dockershim Deprecation FAQ" date: 2020-12-02 slug: dockershim-faq -aliases: [ '/dockershim' ] --- + +_**Update**: There is a [newer version](/blog/2022/02/17/dockershim-faq/) of this article available._ + This document goes over some frequently asked questions regarding the Dockershim deprecation announced as a part of the Kubernetes v1.20 release. For more detail on the deprecation of Docker as a container runtime for Kubernetes kubelets, and what that means, check out the blog post [Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/). +Also, you can read [check whether Dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/) to check whether it does. + ### Why is dockershim being deprecated? Maintaining dockershim has become a heavy burden on the Kubernetes maintainers. @@ -28,7 +32,7 @@ as cgroups v2 and user namespaces are being implemented in these newer CRI runtimes. Removing support for the dockershim will allow further development in those areas. -[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim +[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim ### Can I still use Docker in Kubernetes 1.20? @@ -42,9 +46,11 @@ startup if using Docker as the runtime. Given the impact of this change, we are using an extended deprecation timeline. It will not be removed before Kubernetes 1.22, meaning the earliest release without -dockershim would be 1.23 in late 2021. We will be working closely with vendors -and other ecosystem groups to ensure a smooth transition and will evaluate things -as the situation evolves. +dockershim would be 1.23 in late 2021. +_Update_: removal of dockershim is scheduled for Kubernetes v1.24, see +[Dockershim Removal Kubernetes Enhancement Proposal][drkep]. +We will be working closely with vendors and other ecosystem groups to ensure a smooth transition and will evaluate +things as the situation evolves. ### Can I still use dockershim after it is removed from Kubernetes? @@ -149,7 +155,7 @@ runtime where possible. Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the -former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the +former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from dockercli to crictl](/docs/reference/tools/map-crictl-dockercli/)) and for the latter you can use newer container build options like [img], [buildah], [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. diff --git a/content/en/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md b/content/en/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md index 944704967b..68831d8aba 100644 --- a/content/en/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md +++ b/content/en/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md @@ -3,10 +3,18 @@ layout: blog title: "Don't Panic: Kubernetes and Docker" date: 2020-12-02 slug: dont-panic-kubernetes-and-docker +evergreen: true +--- + +**Update:** _Kubernetes support for Docker via `dockershim` is now removed. +For more information, read the [removal FAQ](/dockershim). +You can also discuss the deprecation via a dedicated [GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917)._ + --- **Authors:** Jorge Castro, Duffie Cooley, Kat Cosgrove, Justin Garrison, Noah Kantrowitz, Bob Killen, Rey Lejano, Dan “POP” Papandrea, Jeffrey Sica, Davanum “Dims” Srinivas + Kubernetes is [deprecating Docker](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation) as a container runtime after v1.20. @@ -24,7 +32,7 @@ shouldn’t, use Docker as a development tool anymore. Docker is still a useful tool for building containers, and the images that result from running `docker build` can still run in your Kubernetes cluster. -If you’re using a managed Kubernetes service like GKE, EKS, or AKS (which [defaults to containerd](https://github.com/Azure/AKS/releases/tag/2020-11-16)) you will need to +If you’re using a managed Kubernetes service like AKS, EkS or GKE, you will need to make sure your worker nodes are using a supported container runtime before Docker support is removed in a future version of Kubernetes. If you have node customizations you may need to update them based on your environment and runtime @@ -33,8 +41,8 @@ testing and planning. If you’re rolling your own clusters, you will also need to make changes to avoid your clusters breaking. At v1.20, you will get a deprecation warning for Docker. -When Docker runtime support is removed in a future release (currently planned -for the 1.22 release in late 2021) of Kubernetes it will no longer be supported +When Docker runtime support is removed in a future release (<del>currently planned +for the 1.22 release in late 2021</del>) of Kubernetes it will no longer be supported and you will need to switch to one of the other compliant container runtimes, like containerd or CRI-O. Just make sure that the runtime you choose supports the docker daemon configurations you currently use (e.g. logging). @@ -101,4 +109,4 @@ questions regardless of experience level or complexity! Our goal is to make sure everyone is educated as much as possible on the upcoming changes. We hope this has answered most of your questions and soothed some anxieties! ❤️ -Looking for more answers? Check out our accompanying [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/). +Looking for more answers? Check out our accompanying [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/) _(updated February 2022)_. diff --git a/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md b/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md index deb459c4be..295c8d7d36 100644 --- a/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md +++ b/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.20: The Raddest Release' date: 2020-12-08 slug: kubernetes-1-20-release-announcement +evergreen: true --- **Authors:** [Kubernetes 1.20 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.20/release_team.md) @@ -31,7 +32,7 @@ The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin. -Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/). +Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/). ### Beta: API Priority and Fairness diff --git a/content/en/blog/_posts/2021-04-06-PodSecurityPolicy-Past-Present-and-Future.md b/content/en/blog/_posts/2021-04-06-PodSecurityPolicy-Past-Present-and-Future.md index c65069460a..73043a67e0 100644 --- a/content/en/blog/_posts/2021-04-06-PodSecurityPolicy-Past-Present-and-Future.md +++ b/content/en/blog/_posts/2021-04-06-PodSecurityPolicy-Past-Present-and-Future.md @@ -21,7 +21,7 @@ Until then, PSP is still PSP. There will be at least a year during which the new ## What is PodSecurityPolicy? -[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) is a built-in [admission controller](/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) that allows a cluster administrator to control security-sensitive aspects of the Pod specification. +[PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) is a built-in [admission controller](/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) that allows a cluster administrator to control security-sensitive aspects of the Pod specification. First, one or more PodSecurityPolicy resources are created in a cluster to define the requirements Pods must meet. Then, RBAC rules are created to control which PodSecurityPolicy applies to a given pod. If a pod meets the requirements of its PSP, it will be admitted to the cluster as usual. In some cases, PSP can also modify Pod fields, effectively creating new defaults for those fields. If a Pod does not meet the PSP requirements, it is rejected, and cannot run. diff --git a/content/en/blog/_posts/2021-04-08-kubernetes-release-1.21.md b/content/en/blog/_posts/2021-04-08-kubernetes-release-1.21.md index ed0da32f25..b22eeeb027 100644 --- a/content/en/blog/_posts/2021-04-08-kubernetes-release-1.21.md +++ b/content/en/blog/_posts/2021-04-08-kubernetes-release-1.21.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.21: Power to the Community' date: 2021-04-08 slug: kubernetes-1-21-release-announcement +evergreen: true --- **Authors:** [Kubernetes 1.21 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.21/release-team.md) diff --git a/content/en/blog/_posts/2021-04-19-introducing-indexed-jobs.md b/content/en/blog/_posts/2021-04-19-introducing-indexed-jobs.md index 990dd6308b..e0233d69a6 100644 --- a/content/en/blog/_posts/2021-04-19-introducing-indexed-jobs.md +++ b/content/en/blog/_posts/2021-04-19-introducing-indexed-jobs.md @@ -62,7 +62,7 @@ spec: Note that completion mode is an alpha feature in the 1.21 release. To be able to use it in your cluster, make sure to enable the `IndexedJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on the -[API server](docs/reference/command-line-tools-reference/kube-apiserver/) and +[API server](/docs/reference/command-line-tools-reference/kube-apiserver/) and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/). When you run the example, you will see that each of the three created Pods gets a diff --git a/content/en/blog/_posts/2021-04-22-gateway-api/index.md b/content/en/blog/_posts/2021-04-22-gateway-api/index.md index d9c798a5b1..145554803d 100644 --- a/content/en/blog/_posts/2021-04-22-gateway-api/index.md +++ b/content/en/blog/_posts/2021-04-22-gateway-api/index.md @@ -30,15 +30,15 @@ This led to design principles that allow the Gateway API to improve upon Ingress The Gateway API introduces a few new resource types: -- **[GatewayClasses](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.GatewayClass)** are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes. -- **[Gateways](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.Gateway)** are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs. -- **Routes** are not a single resource, but represent many different protocol-specific Route resources. The [HTTPRoute](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.HTTPRoute) has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are [TCPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TCPRoute), [UDPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.UDPRoute), and [TLSRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TLSRoute) which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future. +- **[GatewayClasses](https://gateway-api.sigs.k8s.io/concepts/api-overview/#gatewayclass)** are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes. +- **[Gateways](https://gateway-api.sigs.k8s.io/concepts/api-overview/#gateway)** are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs. +- **Routes** are not a single resource, but represent many different protocol-specific Route resources. The [HTTPRoute](https://gateway-api.sigs.k8s.io/concepts/api-overview/#httproute) has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are [TCPRoutes](https://gateway-api.sigs.k8s.io/concepts/api-overview/#tcproute-and-udproute), [UDPRoutes](https://gateway-api.sigs.k8s.io/concepts/api-overview/#tcproute-and-udproute), and [TLSRoutes](https://gateway-api.sigs.k8s.io/concepts/api-overview/#gateway) which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future. ![The resources of the Gateway API](gateway-api-resources.png) ### Gateway Controller Implementations -The good news is that although Gateway is in [Alpha](https://github.com/kubernetes-sigs/gateway-api/releases), there are already several [Gateway controller implementations](https://gateway-api.sigs.k8s.io/references/implementations/) that you can run. Since it’s a standardized spec, the following example could be run on any of them and should function the exact same way. Check out [getting started](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see how to install and use one of these Gateway controllers. +The good news is that although Gateway is in [Alpha](https://github.com/kubernetes-sigs/gateway-api/releases), there are already several [Gateway controller implementations](https://gateway-api.sigs.k8s.io/implementations/) that you can run. Since it’s a standardized spec, the following example could be run on any of them and should function the exact same way. Check out [getting started](https://gateway-api.sigs.k8s.io/v1alpha1/guides/getting-started/) to see how to install and use one of these Gateway controllers. ## Getting Hands-on with the Gateway API @@ -134,7 +134,7 @@ spec: So we have two HTTPRoutes matching and routing traffic to different Services. You might be wondering, where are these Services accessible? Through which networks or IPs are they exposed? -How Routes are exposed to clients is governed by [Route binding](https://gateway-api.sigs.k8s.io/concepts/api-overview/#route-binding), which describes how Routes and Gateways create a bidirectional relationship between each other. When Routes are bound to a Gateway it means their collective routing rules are configured on the underlying load balancers or proxies and the Routes are accessible through the Gateway. Thus, a Gateway is a logical representation of a networking data plane that can be configured through Routes. +How Routes are exposed to clients is governed by [Route binding](https://gateway-api.sigs.k8s.io/concepts/api-overview/#route-resources), which describes how Routes and Gateways create a bidirectional relationship between each other. When Routes are bound to a Gateway it means their collective routing rules are configured on the underlying load balancers or proxies and the Routes are accessible through the Gateway. Thus, a Gateway is a logical representation of a networking data plane that can be configured through Routes. ![How Routes bind with Gateways](route-binding.png ) @@ -192,6 +192,6 @@ When you put it all together, you have a single load balancing infrastructure th There are many resources to check out to learn more. -* Check out the [user guides](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see what use-cases can be addressed. -* Try out one of the [existing Gateway controllers ](https://gateway-api.sigs.k8s.io/references/implementations/) +* Check out the [user guides](https://gateway-api.sigs.k8s.io/v1alpha1/guides/getting-started/) to see what use-cases can be addressed. +* Try out one of the [existing Gateway controllers ](https://gateway-api.sigs.k8s.io/implementations/) * Or [get involved](https://gateway-api.sigs.k8s.io/contributing/community/) and help design and influence the future of Kubernetes service networking! diff --git a/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md b/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md index 1f6403b301..c868b1bd5c 100644 --- a/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md +++ b/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md @@ -108,7 +108,7 @@ metadata: uid: 93a37fed-23e3-45e8-b6ee-b2521db81638 ``` -In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and put it into a read-only state. The deletion timestamp signals that the object can only be read, with the exception of removing the finalizer key updates. In other words, the deletion will not be complete until we edit the object and remove the finalizer. +In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and blocked removal of the object from etcd. The deletion timestamp signals that deletion was requested, but the deletion will not be complete until we edit the object and remove the finalizer. Here's a demonstration of using the `patch` command to remove finalizers. If we want to delete an object, we can simply patch it on the command line to remove the finalizers. In this way, the deletion that was running in the background will complete and the object will be deleted. When we attempt to `get` that configmap, it will be gone. @@ -190,9 +190,10 @@ kubectl get configmap No resources found in default namespace. ``` -To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=false option for `kubectl delete` to delete an object and orphan its children. +To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=orphan option for `kubectl delete` to delete an object and orphan its children. *Update: starting with kubectl v1.20, the default for cascade is `background`.* -In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=false, the parent is deleted but the child still exists: + +In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=orphan, the parent is deleted but the child still exists: ``` kubectl get configmap @@ -200,7 +201,7 @@ NAME DATA AGE mymap-child 0 13m8s mymap-parent 0 13m8s -kubectl delete --cascade=false configmap/mymap-parent +kubectl delete --cascade=orphan configmap/mymap-parent configmap "mymap-parent" deleted kubectl get configmap diff --git a/content/en/blog/_posts/2021-08-04-kubernetes-release-1.22.md b/content/en/blog/_posts/2021-08-04-kubernetes-release-1.22.md index 9a196f7fba..4189d1a4c8 100644 --- a/content/en/blog/_posts/2021-08-04-kubernetes-release-1.22.md +++ b/content/en/blog/_posts/2021-08-04-kubernetes-release-1.22.md @@ -3,6 +3,7 @@ layout: blog title: 'Kubernetes 1.22: Reaching New Peaks' date: 2021-08-04 slug: kubernetes-1-22-release-announcement +evergreen: true --- **Authors:** [Kubernetes 1.22 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.22/release-team.md) diff --git a/content/en/blog/_posts/2021-08-06-server-side-apply-ga.md b/content/en/blog/_posts/2021-08-06-server-side-apply-ga.md index eca57a561e..01403715cc 100644 --- a/content/en/blog/_posts/2021-08-06-server-side-apply-ga.md +++ b/content/en/blog/_posts/2021-08-06-server-side-apply-ga.md @@ -121,7 +121,7 @@ deploymentApplyConfig.Spec.Template.Spec.WithContainers(corev1ac.Container(). ) // apply -applied, err := deploymentClient.Apply(ctx, extractedDeployment, metav1.ApplyOptions{FieldManager: fieldMgr}) +applied, err := deploymentClient.Apply(ctx, deploymentApplyConfig, metav1.ApplyOptions{FieldManager: fieldMgr}) ``` For developers using Custom Resource Definitions (CRDs), the Kubebuilder apply support will provide the same capabilities. Documentation will be included in the Kubebuilder book when available. diff --git a/content/en/blog/_posts/2021-08-30-volume-populators-alpha.md b/content/en/blog/_posts/2021-08-30-volume-populators-alpha.md index 4f3a408584..a632415f50 100644 --- a/content/en/blog/_posts/2021-08-30-volume-populators-alpha.md +++ b/content/en/blog/_posts/2021-08-30-volume-populators-alpha.md @@ -96,15 +96,16 @@ out. First install the volume-data-source-validator controller. ```terminal -kubectl apply -f https://github.com/kubernetes-csi/volume-data-source-validator/blob/master/deploy/kubernetes/rbac-data-source-validator.yaml -kubectl apply -f https://github.com/kubernetes-csi/volume-data-source-validator/blob/master/deploy/kubernetes/setup-data-source-validator.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/master/client/config/crd/populator.storage.k8s.io_volumepopulators.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/master/deploy/kubernetes/rbac-data-source-validator.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/master/deploy/kubernetes/setup-data-source-validator.yaml ``` Next install the example populator. ```terminal -kubectl apply -f https://github.com/kubernetes-csi/lib-volume-populator/blob/master/example/hello-populator/crd.yaml -kubectl apply -f https://github.com/kubernetes-csi/lib-volume-populator/blob/master/example/hello-populator/deploy.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/master/example/hello-populator/crd.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/master/example/hello-populator/deploy.yaml ``` Create an instance of the `Hello` CR, with some text. diff --git a/content/en/blog/_posts/2021-09-03-api-server-tracing.md b/content/en/blog/_posts/2021-09-03-api-server-tracing.md index fc98a68d23..344ff8fa46 100644 --- a/content/en/blog/_posts/2021-09-03-api-server-tracing.md +++ b/content/en/blog/_posts/2021-09-03-api-server-tracing.md @@ -42,7 +42,7 @@ samplingRatePerMillion: 10000 ### Enabling Etcd Tracing -Add `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=0.0.0.0:4317`, `--experimental-distributed-tracing-service-name=etcd` flags to etcd to enable tracing. Note that this traces every request, so it will probably generate a lot of traces if you enable it. +Add `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=0.0.0.0:4317`, `--experimental-distributed-tracing-service-name=etcd` flags to etcd to enable tracing. Note that this traces every request, so it will probably generate a lot of traces if you enable it. Required etcd version is [v3.5+](https://etcd.io/docs/v3.5/op-guide/monitoring/#distributed-tracing). ### Example Trace: List Nodes diff --git a/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md b/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md index c04df56284..64d47ae657 100644 --- a/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md +++ b/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md @@ -28,7 +28,7 @@ metadata: name: shared-cache spec: accessModes: - - ReadWriteMany # Allow many pods to access shared-cache simultaneously. + - ReadWriteMany # Allow many nodes to access shared-cache simultaneously. resources: requests: storage: 1Gi @@ -255,7 +255,7 @@ The minimum required versions are: ## What’s next? -As part of the beta graduation for this feature, SIG Storage plans to update the Kubenetes scheduler to support pod preemption in relation to ReadWriteOncePod storage. +As part of the beta graduation for this feature, SIG Storage plans to update the Kubernetes scheduler to support pod preemption in relation to ReadWriteOncePod storage. This means if two pods request a PersistentVolumeClaim with ReadWriteOncePod, the pod with highest priority will gain access to the PersistentVolumeClaim and any pod with lower priority will be preempted from the node and be unable to access the PersistentVolumeClaim. ## How can I learn more? diff --git a/content/en/blog/_posts/2021-09-27-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-09-27-SIG-Node-Spotlight/index.md new file mode 100644 index 0000000000..88c9fc890b --- /dev/null +++ b/content/en/blog/_posts/2021-09-27-SIG-Node-Spotlight/index.md @@ -0,0 +1,72 @@ +--- +layout: blog +title: "Spotlight on SIG Node" +date: 2021-09-27 +slug: sig-node-spotlight-2021 +--- + +**Author:** Dewan Ahmed, Red Hat + +## Introduction + +In Kubernetes, a _Node_ is a representation of a single machine in your cluster. [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved. + +## A summary of our conversation + +### Could you tell us a little about what SIG Node does? + +SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. + +### "SIG Node is a vertical SIG" could you explain a bit more? + +EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code. + +Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector. + +### How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG? + +SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority. + +### As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work? + +EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result. + +In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues. + +### Why should new and existing contributors consider joining SIG Node? + +SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you the foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster. + +SIG Node maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. + +### What do you do to help new contributors get started? + +EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start. + +I always encourage new contributors to work on things that they have some investment in already. In SIG Node, that might mean volunteering to help fix a bug that you have personally been affected by, or helping to triage bugs you care about by priority. + +To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver. + +[Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. + +### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? + +SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. + +The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms. + +Finally, we are always looking for feedback on how best to run your workload. Come and explain specifics of it, and what features in SIG Node components may help to run it better. + +### What are you getting positive feedback on, and what’s coming up next for SIG Node? + +EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on. + +We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies. + +### Any closing thoughts/resources you’d like to share? + +SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info. + +## Wrap Up + +SIG Node hosted a [KubeCon + CloudNativeCon Europe 2021 talk](https://www.youtube.com/watch?v=z5aY4e2RENA) with an intro and deep dive to their awesome SIG. Join the SIG's meetings to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream Node team as a contributor! \ No newline at end of file diff --git a/content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md b/content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md new file mode 100644 index 0000000000..42e9186692 --- /dev/null +++ b/content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md @@ -0,0 +1,243 @@ +--- +layout: blog +title: "How to Handle Data Duplication in Data-Heavy Kubernetes Environments" +date: 2021-09-29 +slug: how-to-handle-data-duplication-in-data-heavy-kubernetes-environments +--- + +**Authors:** +Augustinas Stirbis (CAST AI) + +## Why Duplicate Data? + +It’s convenient to create a copy of your application with a copy of its state for each team. +For example, you might want a separate database copy to test some significant schema changes +or develop other disruptive operations like bulk insert/delete/update... + +**Duplicating data takes a lot of time.** That’s because you need first to download +all the data from a source block storage provider to compute and then send +it back to a storage provider again. There’s a lot of network traffic and CPU/RAM used in this process. +Hardware acceleration by offloading certain expensive operations to dedicated hardware is +**always a huge performance boost**. It reduces the time required to complete an operation by orders +of magnitude. + +## Volume Snapshots to the rescue + +Kubernetes introduced [VolumeSnapshots](/docs/concepts/storage/volume-snapshots/) as alpha in 1.12, +beta in 1.17, and the Generally Available version in 1.20. +VolumeSnapshots use specialized APIs from storage providers to duplicate volume of data. + +Since data is already in the same storage device (array of devices), duplicating data is usually +a metadata operation for storage providers with local snapshots (majority of on-premise storage providers). +All you need to do is point a new disk to an immutable snapshot and only +save deltas (or let it do a full-disk copy). As an operation that is inside the storage back-end, +it’s much quicker and usually doesn’t involve sending traffic over the network. +Public Clouds storage providers under the hood work a bit differently. They save snapshots +to Object Storage and then copy back from Object storage to Block storage when "duplicating" disk. +Technically there is a lot of Compute and network resources spent on Cloud providers side, +but from Kubernetes user perspective VolumeSnapshots work the same way whether is it local or +remote snapshot storage provider and no Compute and Network resources are involved in this operation. + +## Sounds like we have our solution, right? + +Actually, VolumeSnapshots are namespaced, and Kubernetes protects namespaced data from +being shared between tenants (Namespaces). This Kubernetes limitation is a conscious design +decision so that a Pod running in a different namespace can’t mount another application’s +[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVC). + +One way around it would be to create multiple volumes with duplicate data in one namespace. +However, you could easily reference the wrong copy. + +So the idea is to separate teams/initiatives by namespaces to avoid that and generally +limit access to the production namespace. + +## Solution? Creating a Golden Snapshot externally + +Another way around this design limitation is to create Snapshot externally (not through Kubernetes). +This is also called pre-provisioning a snapshot manually. Next, I will import it +as a multi-tenant golden snapshot that can be used for many namespaces. Below illustration will be +for AWS EBS (Elastic Block Storage) and GCE PD (Persistent Disk) services. + +### High-level plan for preparing the Golden Snapshot + +1. Identify Disk (EBS/Persistent Disk) that you want to clone with data in the cloud provider +2. Make a Disk Snapshot (in cloud provider console) +3. Get Disk Snapshot ID + +### High-level plan for cloning data for each team + +1. Create Namespace “sandbox01” +2. Import Disk Snapshot (ID) as VolumeSnapshotContent to Kubernetes +3. Create VolumeSnapshot in the Namespace "sandbox01" mapped to VolumeSnapshotContent +4. Create the PersistentVolumeClaim from VolumeSnapshot +5. Install Deployment or StatefulSet with PVC + +## Step 1: Identify Disk + +First, you need to identify your golden source. In my case, it’s a PostgreSQL database +on PersistentVolumeClaim “postgres-pv-claim” in the “production” namespace. + +```terminal +kubectl -n <namespace> get pvc <pvc-name> -o jsonpath='{.spec.volumeName}' +``` + +The output will look similar to: +``` +pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d90 +``` + +## Step 2: Prepare your golden source + +You need to do this once or every time you want to refresh your golden data. + +### Make a Disk Snapshot + +Go to AWS EC2 or GCP Compute Engine console and search for an EBS volume +(on AWS) or Persistent Disk (on GCP), that has a label matching the last output. +In this case I saw: `pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d9`. + +Click on Create snapshot and give it a name. You can do it in Console manually, +in AWS CloudShell / Google Cloud Shell, or in the terminal. To create a snapshot in the +terminal you must have the AWS CLI tool (`aws`) or Google's CLI (`gcloud`) +installed and configured. + +Here’s the command to create snapshot on GCP: + +```terminal +gcloud compute disks snapshot <cloud-disk-id> --project=<gcp-project-id> --snapshot-names=<set-new-snapshot-name> --zone=<availability-zone> --storage-location=<region> +``` +{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png" alt="Screenshot of a terminal showing volume snapshot creation on GCP" title="GCP snapshot creation" >}} + + +GCP identifies the disk by its PVC name, so it’s direct mapping. In AWS, you need to +find volume by the CSIVolumeName AWS tag with PVC name value first that will be used for snapshot creation. + +{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png" alt="Screenshot of AWS web console, showing EBS volume identification" title="Identify disk ID on AWS" >}} + +Mark done Volume (volume-id) ```vol-00c7ecd873c6fb3ec``` and ether create EBS snapshot in AWS Console, or use ```aws cli```. + +```terminal +aws ec2 create-snapshot --volume-id '<volume-id>' --description '<set-new-snapshot-name>' --tag-specifications 'ResourceType=snapshot' +``` + +## Step 3: Get your Disk Snapshot ID + +In AWS, the command above will output something similar to: +```terminal +"SnapshotId": "snap-09ed24a70bc19bbe4" +``` + +If you’re using the GCP cloud, you can get the snapshot ID from the gcloud command by querying for the snapshot’s given name: + +```terminal +gcloud compute snapshots --project=<gcp-project-id> describe <new-snapshot-name> | grep id: +``` +You should get similar output to: +``` +id: 6645363163809389170 +``` + +## Step 4: Create a development environment for each team + +Now I have my Golden Snapshot, which is immutable data. Each team will get a copy +of this data, and team members can modify it as they see fit, given that a new EBS/persistent +disk will be created for each team. + +Below I will define a manifest for each namespace. To save time, you can replace +the namespace name (such as changing “sandbox01” → “sandbox42”) using tools +such as `sed` or `yq`, with Kubernetes-aware templating tools like +[Kustomize](/docs/tasks/manage-kubernetes-objects/kustomization/), +or using variable substitution in a CI/CD pipeline. + +Here's an example manifest: + +```yaml +--- +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshotContent +metadata: + name: postgresql-orders-db-sandbox01 + namespace: sandbox01 +spec: + deletionPolicy: Retain + driver: pd.csi.storage.gke.io + source: + snapshotHandle: 'gcp/projects/staging-eu-castai-vt5hy2/global/snapshots/6645363163809389170' + volumeSnapshotRef: + kind: VolumeSnapshot + name: postgresql-orders-db-snap + namespace: sandbox01 +--- +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: postgresql-orders-db-snap + namespace: sandbox01 +spec: + source: + volumeSnapshotContentName: postgresql-orders-db-sandbox01 +``` + +In Kubernetes, VolumeSnapshotContent (VSC) objects are not namespaced. +However, I need a separate VSC for each different namespace to use, so the +`metadata.name` of each VSC must also be different. To make that straightfoward, +I used the target namespace as part of the name. + +Now it’s time to replace the driver field with the CSI (Container Storage Interface) driver +installed in your K8s cluster. Major cloud providers have CSI driver for block storage that +support VolumeSnapshots but quite often CSI drivers are not installed by default, consult +with your Kubernetes provider. + +That manifest above defines a VSC that works on GCP. +On AWS, driver and SnashotHandle values might look like: + +```YAML + driver: ebs.csi.aws.com + source: + snapshotHandle: "snap-07ff83d328c981c98" +``` + +At this point, I need to use the *Retain* policy, so that the CSI driver doesn’t try to +delete my manually created EBS disk snapshot. + +For GCP, you will have to build this string by hand - add a full project ID and snapshot ID. +For AWS, it’s just a plain snapshot ID. + +VSC also requires specifying which VolumeSnapshot (VS) will use it, so VSC and VS are +referencing each other. + +Now I can create PersistentVolumeClaim from VS above. It’s important to set this first: + + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: postgres-pv-claim + namespace: sandbox01 +spec: + dataSource: + kind: VolumeSnapshot + name: postgresql-orders-db-snap + apiGroup: snapshot.storage.k8s.io + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 21Gi +``` + +If default StorageClass has [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) policy, +then the actual Cloud Disk will be created from the Golden Snapshot only when some Pod bounds that PVC. + +Now I assign that PVC to my Pod (in my case, it’s Postgresql) as I would with any other PVC. + +```terminal +kubectl -n <namespace> get volumesnapshotContent,volumesnapshot,pvc,pod +``` + +Both VS and VSC should be *READYTOUSE* true, PVC bound, and the Pod (from Deployment or StatefulSet) running. + +**To keep on using data from my Golden Snapshot, I just need to repeat this for the +next namespace and voilà! No need to waste time and compute resources on the duplication process.** diff --git a/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md new file mode 100644 index 0000000000..e32d7bcb72 --- /dev/null +++ b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md @@ -0,0 +1,417 @@ +--- +layout: blog +title: A Closer Look at NSA/CISA Kubernetes Hardening Guidance +date: 2021-10-05 +slug: nsa-cisa-kubernetes-hardening-guidance +--- + +**Authors:** Jim Angel (Google), Pushkar Joglekar (VMware), and Savitha +Raghunathan (Red Hat) + +{{% alert title="Disclaimer" %}} +The open source tools listed in this article are to serve as examples only +and are in no way a direct recommendation from the Kubernetes community or authors. +{{% /alert %}} + +## Background + +USA's National Security Agency (NSA) and the Cybersecurity and Infrastructure +Security Agency (CISA) +released, "[Kubernetes Hardening Guidance](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF)" +on August 3rd, 2021. The guidance details threats to Kubernetes environments +and provides secure configuration guidance to minimize risk. + +The following sections of this blog correlate to the sections in the NSA/CISA guidance. +Any missing sections are skipped because of limited opportunities to add +anything new to the existing content. + +_Note_: This blog post is not a substitute for reading the guide. Reading the published +guidance is recommended before proceeding as the following content is +complementary. + +## Introduction and Threat Model + +Note that the threats identified as important by the NSA/CISA, or the intended audience of this guidance, may be different from the threats that other enterprise users of Kubernetes consider important. This section +is still useful for organizations that care about data, resource theft and +service unavailability. + +The guidance highlights the following three sources of compromises: + +- Supply chain risks +- Malicious threat actors +- Insider threats (administrators, users, or cloud service providers) + +The [threat model](https://en.wikipedia.org/wiki/Threat_model) tries to take a step back and review threats that not only +exist within the boundary of a Kubernetes cluster but also include the underlying +infrastructure and surrounding workloads that Kubernetes does not manage. + +For example, when a workload outside the cluster shares the same physical +network, it has access to the kubelet and to control plane components: etcd, controller manager, scheduler and API +server. Therefore, the guidance recommends having network level isolation +separating Kubernetes clusters from other workloads that do not need connectivity +to Kubernetes control plane nodes. Specifically, scheduler, controller-manager, +etcd only need to be accessible to the API server. Any interactions with Kubernetes +from outside the cluster can happen by providing access to API server port. + +List of ports and protocols for each of these components are +defined in [Ports and Protocols](/docs/reference/ports-and-protocols/) +within the Kubernetes documentation. + +> Special note: kube-scheduler and kube-controller-manager uses different ports than the ones mentioned in the guidance + +The [Threat modelling](https://cnsmap.netlify.app/threat-modelling) section +from the CNCF [Cloud Native Security Whitepaper + Map](https://github.com/cncf/tag-security/tree/main/security-whitepaper) +provides another perspective on approaching threat modelling Kubernetes, from a +cloud native lens. + +## Kubernetes Pod security + +Kubernetes by default does not guarantee strict workload isolation between pods +running in the same node in a cluster. However, the guidance provides several +techniques to enhance existing isolation and reduce the attack surface in case of a +compromise. + +### "Non-root" containers and "rootless" container engines + +Several best practices related to basic security principle of least privilege +i.e. provide only the permissions are needed; no more, no less, are worth a +second look. + +The guide recommends setting non-root user at build time instead of relying on +setting `runAsUser` at runtime in your Pod spec. This is a good practice and provides +some level of defense in depth. For example, if the container image is built with user `10001` +and the Pod spec misses adding the `runAsuser` field in its `Deployment` object. In this +case there are certain edge cases that are worth exploring for awareness: + +1. Pods can fail to start, if the user defined at build time is different from + the one defined in pod spec and some files are as a result inaccessible. +2. Pods can end up sharing User IDs unintentionally. This can be problematic + even if the User IDs are non-zero in a situation where a container escape to + host file system is possible. Once the attacker has access to the host file + system, they get access to all the file resources that are owned by other + unrelated pods that share the same UID. +3. Pods can end up sharing User IDs, with other node level processes not managed + by Kubernetes e.g. node level daemons for auditing, vulnerability scanning, + telemetry. The threat is similar to the one above where host file system + access can give attacker full access to these node level daemons without + needing to be root on the node. + +However, none of these cases will have as severe an impact as a container +running as root being able to escape as a root user on the host, which can provide +an attacker with complete control of the worker node, further allowing lateral +movement to other worker or control plane nodes. + +Kubernetes 1.22 introduced +an [alpha feature](/docs/tasks/administer-cluster/kubelet-in-userns/) +that specifically reduces the impact of such a control plane component running +as root user to a non-root user through user namespaces. + +That ([alpha stage](/docs/reference/command-line-tools-reference/feature-gates/#feature-stages)) support for user namespaces / rootless mode is available with +the following container runtimes: + +- [Docker Engine](https://docs.docker.com/engine/security/rootless/) +- [Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics) + +Some distributions support running in rootless mode, like the following: + +- [kind](https://kind.sigs.k8s.io/docs/user/rootless/) +- [k3s](https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental) +- [Usernetes](https://github.com/rootless-containers/usernetes) + +### Immutable container filesystems + +The NSA/CISA Kubernetes Hardening Guidance highlights an often overlooked feature `readOnlyRootFileSystem`, with a +working example in [Appendix B](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=42). This example limits execution and tampering of +containers at runtime. Any read/write activity can then be limited to few +directories by using `tmpfs` volume mounts. + +However, some applications that modify the container filesystem at runtime, like exploding a WAR or JAR file at container startup, +could face issues when enabling this feature. To avoid this issue, consider making minimal changes to the filesystem at runtime +when possible. + +### Building secure container images + +Kubernetes Hardening Guidance also recommends running a scanner at deploy time as an admission controller, +to prevent vulnerable or misconfigured pods from running in the cluster. +Theoretically, this sounds like a good approach but there are several caveats to +consider before this can be implemented in practice: + +- Depending on network bandwidth, available resources and scanner of choice, + scanning for vulnerabilities for an image can take an indeterminate amount of + time. This could lead to slower or unpredictable pod start up times, which + could result in spikes of unavailability when apps are serving peak load. +- If the policy that allows or denies pod startup is made using incorrect or + incomplete data it could result in several false positive or false negative + outcomes like the following: + - inside a container image, the `openssl` package is detected as vulnerable. However, + the application is written in Golang and uses the Go `crypto` package for TLS. Therefore, this vulnerability + is not in the code execution path and as such has minimal impact if it + remains unfixed. + - A vulnerability is detected in the `openssl` package for a Debian base image. + However, the upstream Debian community considers this as a Minor impact + vulnerability and as a result does not release a patch fix for this + vulnerability. The owner of this image is now stuck with a vulnerability that + cannot be fixed and a cluster that does not allow the image to run because + of predefined policy that does not take into account whether the fix for a + vulnerability is available or not + - A Golang app is built on top of a [distroless](https://github.com/GoogleContainerTools/distroless) + image, but it is compiled with a Golang version that uses a vulnerable [standard library](https://pkg.go.dev/std). + The scanner has + no visibility into golang version but only on OS level packages. So it + allows the pod to run in the cluster in spite of the image containing an + app binary built on vulnerable golang. + +To be clear, relying on vulnerability scanners is absolutely a good idea but +policy definitions should be flexible enough to allow: + +- Creation of exception lists for images or vulnerabilities through labelling +- Overriding the severity with a risk score based on impact of a vulnerability +- Applying the same policies at build time to catch vulnerable images with + fixable vulnerabilities before they can be deployed into Kubernetes clusters + +Special considerations like offline vulnerability database fetch, may also be +needed, if the clusters run in an air-gapped environment and the scanners +require internet access to update the vulnerability database. + +### Pod Security Policies + +Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) +API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/), +but some of the guidance in this section will still apply for the next few years, until cluster operators +upgrade their clusters to newer Kubernetes versions. + +The Kubernetes project is working on a replacement for PodSecurityPolicy. +Kubernetes v1.22 includes an alpha feature called [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +that is intended to allow enforcing a minimum level of isolation between pods. + +The built-in isolation levels for Pod Security Admission are derived +from [Pod Security Standards](/docs/concepts/security/pod-security-standards/), which is a superset of all the components mentioned in Table I [page 10](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=17) of +the guidance. + +Information about migrating from PodSecurityPolicy to the Pod Security +Admission feature is available +in +[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/). + +One important behavior mentioned in the guidance that remains the same between +Pod Security Policy and its replacement is that enforcing either of them does +not affect pods that are already running. With both PodSecurityPolicy and Pod Security Admission, +the enforcement happens during the pod creation +stage. + +### Hardening container engines + +Some container workloads are less trusted than others but may need to run in the +same cluster. In those cases, running them on dedicated nodes that include +hardened container runtimes that provide stricter pod isolation boundaries can +act as a useful security control. + +Kubernetes supports +an API called [RuntimeClass](/docs/concepts/containers/runtime-class/) that is +stable / GA (and, therefore, enabled by default) stage as of Kubernetes v1.20. +RuntimeClass allows you to ensure that Pods requiring strong isolation are scheduled onto +nodes that can offer it. + +Some third-party projects that you can use in conjunction with RuntimeClass are: + +- [kata containers](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-runtime-class-for-kata-containers) +- [gvisor](https://gvisor.dev/docs/user_guide/containerd/quick_start/) + +As discussed here and in the guidance, many features and tooling exist in and around +Kubernetes that can enhance the isolation boundaries between +pods. Based on relevant threats and risk posture, you should pick and choose +between them, instead of trying to apply all the recommendations. Having said that, cluster +level isolation i.e. running workloads in dedicated clusters, remains the strictest workload +isolation mechanism, in spite of improvements mentioned earlier here and in the guide. + +## Network Separation and Hardening + +Kubernetes Networking can be tricky and this section focuses on how to secure +and harden the relevant configurations. The guide identifies the following as key +takeaways: +- Using NetworkPolicies to create isolation between resources, +- Securing the control plane +- Encrypting traffic and sensitive data + +### Network Policies + +Network policies can be created with the help of network plugins. In order to +make the creation and visualization easier for users, Cilium supports +a [web GUI tool](https://editor.cilium.io). That web GUI lets you create Kubernetes +NetworkPolicies (a generic API that nevertheless requires a compatible CNI plugin), +and / or Cilium network policies (CiliumClusterwideNetworkPolicy and CiliumNetworkPolicy, +which only work in clusters that use the Cilium CNI plugin). +You can use these APIs to restrict network traffic between pods, and therefore minimize the +attack vector. + +Another scenario that is worth exploring is the usage of external IPs. Some +services, when misconfigured, can create random external IPs. An attacker can take +advantage of this misconfiguration and easily intercept traffic. This vulnerability +has been reported +in [CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/). +Using [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) +can mitigate this vulnerability by preventing the services from using random +external IPs. [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) +only allows creation of services that don't require external IPs or whose +external IPs are within the range specified by the administrator. + +> CVE-2020-8554 - Kubernetes API server in all versions allow an attacker +> who is able to create a ClusterIP service and set the `spec.externalIPs` field, +> to intercept traffic to that IP address. Additionally, an attacker who is able to +> patch the `status` (which is considered a privileged operation and should not +> typically be granted to users) of a LoadBalancer service can set the +> `status.loadBalancer.ingress.ip` to similar effect. + +### Resource Policies + +In addition to configuring ResourceQuotas and limits, consider restricting how many process +IDs (PIDs) a given Pod can use, and also to reserve some PIDs for node-level use to avoid +resource exhaustion. More details to apply these limits can be +found in [Process ID Limits And Reservations](/docs/concepts/policy/pid-limiting/). + +### Control Plane Hardening + +In the next section, the guide covers control plane hardening. It is worth +noting that +from [Kubernetes 1.20](https://github.com/kubernetes/kubernetes/issues/91506), +insecure port from API server, has been removed. + +### Etcd + +As a general rule, the etcd server should be configured to only trust +certificates assigned to the API server. It limits the attack surface and prevents a +malicious attacker from gaining access to the cluster. It might be beneficial to +use a separate CA for etcd, as it by default trusts all the certificates issued +by the root CA. + +### Kubeconfig Files + +In addition to specifying the token and certificates directly, `.kubeconfig` +supports dynamic retrieval of temporary tokens using auth provider plugins. +Beware of the possibility of malicious +shell [code execution](https://banzaicloud.com/blog/kubeconfig-security/) in a +`kubeconfig` file. Once attackers gain access to the cluster, they can steal ssh +keys/secrets or more. + +### Secrets +Kubernetes [Secrets](/docs/concepts/configuration/secret/) is the native way of managing secrets as a Kubernetes +API object. However, in some scenarios such as a desire to have a single source of truth for all app secrets, irrespective of whether they run on Kubernetes or not, secrets can be managed loosely coupled with +Kubernetes and consumed by pods through side-cars or init-containers with minimal usage of Kubernetes Secrets API. + +[External secrets providers](https://github.com/external-secrets/kubernetes-external-secrets) +and [csi-secrets-store](https://github.com/kubernetes-sigs/secrets-store-csi-driver) +are some of these alternatives to Kubernetes Secrets + +## Log Auditing + +The NSA/CISA guidance stresses monitoring and alerting based on logs. The key points +include logging at the host level, application level, and on the cloud. When +running Kubernetes in production, it's important to understand who's +responsible, and who's accountable, for each layer of logging. + +### Kubernetes API auditing + +One area that deserves more focus is what exactly should alert or be logged. The +document outlines a sample policy in [Appendix L: Audit Policy](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=55) that logs all +RequestResponse's including metadata and request / response bodies. While helpful for a demo, it may not be practical for production. + +Each organization needs to evaluate their +own threat model and build an audit policy that complements or helps troubleshooting incident response. Think +about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug/debug-cluster/audit/#audit-policy). +It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. + +Audit logging configurations can also be tested with +kind following these [instructions](https://kind.sigs.k8s.io/docs/user/auditing). + +### Streaming logs and auditing + +Logging is important for threat and anomaly detection. As the document outlines, +it's a best practice to scan and alert on logs as close to real time as possible +and to protect logs from tampering if a compromise occurs. It's important to +reflect on the various levels of logging and identify the critical areas such as +API endpoints. + +Kubernetes API audit logging can stream to a webhook and there's an example in [Appendix N: Webhook configuration](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=58). Using a webhook could be a method that +stores logs off cluster and/or centralizes all audit logs. Once logs are +centrally managed, look to enable alerting based on critical events. Also ensure +you understand what the baseline is for normal activities. + +### Alert identification + +While the guide stressed the importance of notifications, there is not a blanket +event list to alert from. The alerting requirements vary based on your own +requirements and threat model. Examples include the following events: + +- Changes to the `securityContext` of a Pod +- Updates to admission controller configs +- Accessing certain files / URLs + +### Additional logging resources + +- [Seccomp Security Profiles and You: A Practical Guide - Duffie Cooley](https://www.youtube.com/watch?v=OPuu8wsu2Zc) +- [TGI Kubernetes 119: Gatekeeper and OPA](https://www.youtube.com/watch?v=ZJgaGJm9NJE) +- [Abusing The Lack of Kubernetes Auditing Policies](https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies/) +- [Enable seccomp for all workloads with a new v1.22 alpha feature](https://kubernetes.io/blog/2021/08/25/seccomp-default/) +- [This Week in Cloud Native: Auditing / Pod Security](https://www.twitch.tv/videos/1147889860) + +## Upgrading and Application Security practices + +Kubernetes releases three times per year, so upgrade-related toil is a common problem for +people running production clusters. In addition to this, operators must +regularly upgrade the underlying node's operating system and running +applications. This is a best practice to ensure continued support and to reduce +the likelihood of bugs or vulnerabilities. + +Kubernetes supports the three most recent stable releases. While each Kubernetes +release goes through a large number of tests before being published, some +teams aren't comfortable running the latest stable release until some time has +passed. No matter what version you're running, ensure that patch upgrades +happen frequently or automatically. More information can be found in +the [version skew](/releases/version-skew-policy/) policy +pages. + +When thinking about how you'll manage node OS upgrades, consider ephemeral +nodes. Having the ability to destroy and add nodes allows your team to respond +quicker to node issues. In addition, having deployments that tolerate node +instability (and a culture that encourages frequent deployments) allows for +easier cluster upgrades. + +Additionally, it's worth reiterating from the guidance that periodic +vulnerability scans and penetration tests can be performed on the various system +components to proactively look for insecure configurations and vulnerabilities. + +### Finding release & security information + +To find the most recent Kubernetes supported versions, refer to +[https://k8s.io/releases](https://k8s.io/releases), which includes minor versions. It's good to stay up to date with +your minor version patches. + +If you're running a managed Kubernetes offering, look for their release +documentation and find their various security channels. + +Subscribe to +the [Kubernetes Announce mailing list](https://groups.google.com/g/kubernetes-announce). +The Kubernetes Announce mailing list is searchable for terms such +as "[Security Advisories](https://groups.google.com/g/kubernetes-announce/search?q=%5BSecurity%20Advisory%5D)". +You can set up alerts and email notifications as long as you know what key +words to alert on. + +## Conclusion + +In summary, it is fantastic to see security practitioners sharing this +level of detailed guidance in public. This guidance further highlights +Kubernetes going mainstream and how securing Kubernetes clusters and the +application containers running on Kubernetes continues to need attention and focus of +practitioners. Only a few weeks after the guidance was published, an open source +tool [kubescape](https://github.com/armosec/kubescape) to validate cluster +against this guidance became available. + +This tool can be a great starting point to check the current state of your +clusters, after which you can use the information in this blog post and in the guidance to assess +where improvements can be made. + +Finally, it is worth reiterating that not all controls in this guidance will +make sense for all practitioners. The best way to know which controls matter is +to rely on the threat model of your own Kubernetes environment. + +_A special shout out and thanks to Rory McCune (@raesene) for his inputs to this blog post_ diff --git a/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md new file mode 100644 index 0000000000..624075fb4e --- /dev/null +++ b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md @@ -0,0 +1,142 @@ +--- +layout: blog +title: "Introducing ClusterClass and Managed Topologies in Cluster API" +date: 2021-10-08 +slug: capi-clusterclass-and-managed-topologies +--- + +**Author:** Fabrizio Pandini (VMware) + +The [Cluster API community](https://cluster-api.sigs.k8s.io/) is happy to announce the implementation of *ClusterClass and Managed Topologies*, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way. + +## A little bit of context… + +Before getting into the details, let's take a step back and look at the history of Cluster API. + +The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting. + +With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more. + +Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs, documentation, and on extensive test signals which inform Kubernetes releases. + +With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users. + +Enter ClusterClass and Managed Topologies, tada! + +## ClusterClass + +As the name suggests, ClusterClass and managed topologies are built in two parts. + +The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. + +![Defining a ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg) + +ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape. + +```yaml +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: ClusterClass +metadata: + name: my-amazing-cluster-class +spec: + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlaneTemplate + name: high-availability-control-plane + machineInfrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachineTemplate + name: control-plane-machine + workers: + machineDeployments: + - class: type1-workers + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: type1-bootstrap + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachineTemplate + name: type1-machine + - class: type2-workers + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: type2-bootstrap + infrastructure: + ref: + kind: DockerMachineTemplate + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + name: type2-machine + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerClusterTemplate + name: cluster-infrastructure + +``` + +The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios. + +## Managed Topologies + +Managed Topologies let you put the power of ClusterClass into action. + +Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster. + +![Create a Cluster with ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg) + +Here is an example: + +```yaml +--- +apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + metadata: + name: my-amazing-cluster + namespace: bar + spec: + topology: # define a managed topology + class: my-amazing-cluster-class # use the ClusterClass mentioned earlier + version: v1.21.2 + controlPlane: + replicas: 3 + workers: + machineDeployments: + - class: type1-workers + name: big-pool-of-machines + replicas: 5 + - class: type2-workers + name: small-pool-of-machines + replicas: 1 +``` + +But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology. + +All the power of Cluster API, extensibility, lifecycle automation, stability, all the features required for managing an enterprise grade Kubernetes cluster on the infrastructure provider of your choice are now at your fingertips: you can create your Cluster, add new machines, upgrade to the next Kubernetes version, and all from a single place. + +It is just as simple as it looks! + +## What’s next + +While the amazing Cluster API community is working hard to deliver the first version of ClusterClass and managed topologies later this year, we are already looking forward to what comes next for the project and its ecosystem. + +There are a lot of great ideas and opportunities ahead! + +We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on. + +Stay tuned for what comes next, and if you have any questions, comments or suggestions: + +* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T) +* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents +* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time +* Check out the [ClusterClass quick-start](https://cluster-api.sigs.k8s.io/user/quick-start.html) for the Docker provider (CAPD) in the Cluster API book. +* _UPDATE_: Check out the [ClusterClass experimental feature](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/index.html) documentation in the Cluster API book. diff --git a/content/en/blog/_posts/2021-10-18-kpng-specialized-proxiers.md b/content/en/blog/_posts/2021-10-18-kpng-specialized-proxiers.md new file mode 100644 index 0000000000..2c60c12f3f --- /dev/null +++ b/content/en/blog/_posts/2021-10-18-kpng-specialized-proxiers.md @@ -0,0 +1,241 @@ +--- +layout: blog +title: "Use KPNG to Write Specialized kube-proxiers" +date: 2021-10-18 +slug: use-kpng-to-write-specialized-kube-proxiers +--- + +**Author**: Lars Ekman (Ericsson) + +The post will show you how to create a specialized service kube-proxy +style network proxier using Kubernetes Proxy NG +[kpng](https://github.com/kubernetes-sigs/kpng) without interfering +with the existing kube-proxy. The kpng project aims at renewing the +the default Kubernetes Service implementation, the "kube-proxy". An +important feature of kpng is that it can be used as a library to +create proxiers outside K8s. While this is useful for CNI-plugins that +replaces the kube-proxy it also opens the possibility for anyone to +create a proxier for a special purpose. + + +## Define a service that uses a specialized proxier + +``` +apiVersion: v1 +kind: Service +metadata: + name: kpng-example + labels: + service.kubernetes.io/service-proxy-name: kpng-example +spec: + clusterIP: None + ipFamilyPolicy: RequireDualStack + externalIPs: + - 10.0.0.55 + - 1000::55 + selector: + app: kpng-alpine + ports: + - port: 6000 +``` + +If the `service.kubernetes.io/service-proxy-name` label is defined the +`kube-proxy` will ignore the service. A custom controller can watch +services with the label set to it's own name, "kpng-example" in +this example, and setup specialized load-balancing. + +The `service.kubernetes.io/service-proxy-name` label is [not +new](https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioservice-proxy-name), +but so far is has been quite hard to write a specialized proxier. + +The common use for a specialized proxier is assumed to be handling +external traffic for some use-case not supported by K8s. In that +case `ClusterIP` is not needed, so we use a "headless" service in this +example. + + +## Specialized proxier using kpng + +A [kpng](https://github.com/kubernetes-sigs/kpng) based proxier +consists of the `kpng` controller handling all the K8s api related +functions, and a "backend" implementing the load-balancing. The +backend can be linked with the `kpng` controller binary or be a +separate program communicating with the controller using gRPC. + +``` +kpng kube --service-proxy-name=kpng-example to-api +``` + +This starts the `kpng` controller and tell it to watch only services +with the "kpng-example" service proxy name. The "to-api" parameter +will open a gRPC server for backends. + +You can test this yourself outside your cluster. Please see the example +below. + +Now we start a backend that simply prints the updates from the +controller. + +``` +$ kubectl apply -f kpng-example.yaml +$ kpng-json | jq # (this is the backend) +{ + "Service": { + "Namespace": "default", + "Name": "kpng-example", + "Type": "ClusterIP", + "IPs": { + "ClusterIPs": {}, + "ExternalIPs": { + "V4": [ + "10.0.0.55" + ], + "V6": [ + "1000::55" + ] + }, + "Headless": true + }, + "Ports": [ + { + "Protocol": 1, + "Port": 6000, + "TargetPort": 6000 + } + ] + }, + "Endpoints": [ + { + "IPs": { + "V6": [ + "1100::202" + ] + }, + "Local": true + }, + { + "IPs": { + "V4": [ + "11.0.2.2" + ] + }, + "Local": true + }, + { + "IPs": { + "V4": [ + "11.0.1.2" + ] + } + }, + { + "IPs": { + "V6": [ + "1100::102" + ] + } + } + ] +} +``` + +A real backend would use some mechanism to load-balance traffic from +the external IPs to the endpoints. + + + +## Writing a backend + +The `kpng-json` backend looks like this: + +```go +package main +import ( + "os" + "encoding/json" + "sigs.k8s.io/kpng/client" +) +func main() { + client.Run(jsonPrint) +} +func jsonPrint(items []*client.ServiceEndpoints) { + enc := json.NewEncoder(os.Stdout) + for _, item := range items { + _ = enc.Encode(item) + } +} +``` + +(yes, that is the entire program) + +A real backend would of course be much more complex, but this +illustrates how `kpng` let you focus on load-balancing. + +You can have several backends connected to a `kpng` controller, so +during development or debug it can be useful to let something like the +`kpng-json` backend run in parallel with your real backend. + + +## Example + + +The complete example can be found [here](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec). + +As an example we implement an "all-ip" backend. It direct all traffic +for the externalIPs to a local endpoint, regardless of ports and upper +layer protocols. There is a +[KEP](https://github.com/kubernetes/enhancements/pull/2611) for this +function and this example is a much simplified version. + +To direct all traffic from an external address to a local POD [only +one iptables rule is +needed](https://github.com/kubernetes/enhancements/pull/2611#issuecomment-895061013), +for instance; + +``` +ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202 +``` + +As you can see the addresses are in the call to the backend and all it +have to do is: + +* Extract the addresses with `Local: true` +* Setup iptables rules for the `ExternalIPs` + +A script doing that may look like: + +``` +xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0]) +podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]') +ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip +``` + +Assuming the JSON output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!). + + +As this is an example we make it really simple for ourselves by using +a minor variation of the `kpng-json` backend above. Instead of just +printing, a program is called and the JSON output is passed as `stdin` +to that program. The backend can be tested stand-alone: + +``` +CALLOUT=jq kpng-callout +``` + +Where `jq` can be replaced with your own program or script. A script +may look like the example above. For more info and the complete +example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec). + + +## Summary + +While [kpng](https://github.com/kubernetes-sigs/kpng) is in early +stage of development this post wants to show how you may build your +own specialized K8s proxiers in the future. The only thing your +applications need to do is to add the +`service.kubernetes.io/service-proxy-name` label in the Service +manifest. + +It is a tedious process to get new features into the `kube-proxy` and +it is not unlikely that they will be rejected, so to write a +specialized proxier may be the only option. diff --git a/content/en/blog/_posts/2021-11-08-steering-committee-results-2021.md b/content/en/blog/_posts/2021-11-08-steering-committee-results-2021.md new file mode 100644 index 0000000000..922824e390 --- /dev/null +++ b/content/en/blog/_posts/2021-11-08-steering-committee-results-2021.md @@ -0,0 +1,56 @@ +--- +layout: blog +title: "Announcing the 2021 Steering Committee Election Results" +date: 2021-11-08 +slug: steering-committee-results-2021 +--- + +**Author**: Kaslin Fields + +The [2021 Steering Committee Election](https://github.com/kubernetes/community/tree/master/events/elections/2021) is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2021. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community. + +This community body is significant since it oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their [charter](https://github.com/kubernetes/steering/blob/master/charter.md). + +## Results + +Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle): + +* **Christoph Blecker ([@cblecker](https://github.com/cblecker)), Red Hat** +* **Stephen Augustus ([@justaugustus](https://github.com/justaugustus)), Cisco** +* **Paris Pittman ([@parispittman](https://github.com/parispittman)), Apple** +* **Tim Pepper ([@tpepper](https://github.com/tpepper)), VMware** + +They join continuing members: + +* **Davanum Srinivas ([@dims](https://github.com/dims)), VMware** +* **Jordan Liggitt ([@liggitt](https://github.com/liggitt)), Google** +* **Bob Killen ([@mrbobbytables](https://github.com/mrbobbytables)), Google** + +Paris Pittman and Christoph Blecker are returning Steering Committee Members. + +## Big Thanks + +Thank you and congratulations on a successful election to this round’s election officers: + +* Alison Dowdney, ([@alisondy](https://github.com/alisondy)) +* Noah Kantrowitz ([@coderanger](https://github.com/coderanger)) +* Josh Berkus ([@jberkus](https://github.com/jberkus)) + +Special thanks to Arnaud Meukam ([@ameukam](https://github.com/ameukam)), k8s-infra liaison, who enabled our voting software on community-owned infrastructure. + +Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community: + +* Derek Carr ([@derekwaynecarr](https://github.com/derekwaynecarr)) +* Nikhita Raghunath ([@nikhita](https://github.com/nikhita)) + +And thank you to all the candidates who came forward to run for election. + +## Get Involved with the Steering Committee + +This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee [backlog items](https://github.com/kubernetes/steering/projects/1) and weigh in by filing an issue or creating a PR against their [repo](https://github.com/kubernetes/steering). They have an open meeting on [the first Monday at 9:30am PT of every month](https://github.com/kubernetes/steering) and regularly attend Meet Our Contributors. They can also be contacted at their public mailing list steering@kubernetes.io. + +You can see what the Steering Committee meetings are all about by watching past meetings on the [YouTube Playlist](https://www.youtube.com/playlist?list=PL69nYSiGNLP1yP1B_nd9-drjoxp0Q14qM). + +--- + +_This post was written by the [Upstream Marketing Working Group](https://github.com/kubernetes/community/tree/master/communication/marketing-team#contributor-marketing). If you want to write stories about the Kubernetes community, learn more about us._ diff --git a/content/en/blog/_posts/2021-11-09-non-root-containers-and-devices.md b/content/en/blog/_posts/2021-11-09-non-root-containers-and-devices.md new file mode 100644 index 0000000000..2dcaf952dd --- /dev/null +++ b/content/en/blog/_posts/2021-11-09-non-root-containers-and-devices.md @@ -0,0 +1,238 @@ +--- +layout: blog +title: 'Non-root Containers And Devices' +date: 2021-11-09 +slug: non-root-containers-and-devices +--- + +**Author:** Mikko Ylinen (Intel) + +The user/group ID related security settings in Pod's `securityContext` trigger a problem when users want to +deploy containers that use accelerator devices (via [Kubernetes Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)) on Linux. In this blog +post I talk about the problem and describe the work done so far to address it. It's not meant to be a long story about getting the [k/k issue](https://github.com/kubernetes/kubernetes/issues/92211) fixed. + +Instead, this post aims to raise awareness of the issue and to highlight important device use-cases too. This is needed as Kubernetes works on new related features such as support for user namespaces. + +## Why non-root containers can't use devices and why it matters +One of the key security principles for running containers in Kubernetes is the +principle of least privilege. The Pod/container `securityContext` specifies the config +options to set, e.g., Linux capabilities, MAC policies, and user/group ID values to achieve this. + +Furthermore, the cluster admins are supported with tools like [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) (deprecated) or +[Pod Security Admission](/docs/concepts/security/pod-security-admission/) (alpha) to enforce the desired security settings for pods that are being deployed in +the cluster. These settings could, for instance, require that containers must be `runAsNonRoot` or +that they are forbidden from running with root's group ID in `runAsGroup` or `supplementalGroups`. + +In Kubernetes, the kubelet builds the list of [`Device`](https://pkg.go.dev/k8s.io/cri-api@v0.22.1/pkg/apis/runtime/v1#Device) resources to be made available to a container +(based on inputs from the Device Plugins) and the list is included in the CreateContainer CRI message +sent to the CRI container runtime. Each `Device` contains little information: host/container device +paths and the desired devices cgroups permissions. + +The [OCI Runtime Spec for Linux Container Configuration](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md) +expects that in addition to the devices cgroup fields, more detailed information about the devices +must be provided: + +```yaml +{ + "type": "<string>", + "path": "<string>", + "major": <int64>, + "minor": <int64>, + "fileMode": <uint32>, + "uid": <uint32>, + "gid": <uint32> +}, +``` + +The CRI container runtimes (containerd, CRI-O) are responsible for obtaining this information +from the host for each `Device`. By default, the runtimes copy the host device's user and group IDs: + +- `uid` (uint32, OPTIONAL) - id of device owner in the container namespace. +- `gid` (uint32, OPTIONAL) - id of device group in the container namespace. + +Similarly, the runtimes prepare other mandatory `config.json` sections based on the CRI fields, +including the ones defined in `securityContext`: `runAsUser`/`runAsGroup`, which become part of the POSIX +platforms user structure via: + +- `uid` (int, REQUIRED) specifies the user ID in the container namespace. +- `gid` (int, REQUIRED) specifies the group ID in the container namespace. +- `additionalGids` (array of ints, OPTIONAL) specifies additional group IDs in the container namespace to be added to the process. + +However, the resulting `config.json` triggers a problem when trying to run containers with +both devices added and with non-root uid/gid set via `runAsUser`/`runAsGroup`: the container user process +has no permission to use the device even when its group id (gid, copied from host) was permissive to +non-root groups. This is because the container user does not belong to that host group (e.g., via `additionalGids`). + +Being able to run applications that use devices as non-root user is normal and expected to work so that +the security principles can be met. Therefore, several alternatives were considered to get the gap filled with what the PodSec/CRI/OCI supports today. + +## What was done to solve the issue? +You might have noticed from the problem definition that it would at least be possible to workaround +the problem by manually adding the device gid(s) to `supplementalGroups`, or in +the case of just one device, set `runAsGroup` to the device's group id. However, this is problematic because the device gid(s) may have +different values depending on the nodes' distro/version in the cluster. For example, with GPUs the following commands for different distros and versions return different gids: + +Fedora 33: +``` +$ ls -l /dev/dri/ +total 0 +drwxr-xr-x. 2 root root 80 19.10. 10:21 by-path +crw-rw----+ 1 root video 226, 0 19.10. 10:42 card0 +crw-rw-rw-. 1 root render 226, 128 19.10. 10:21 renderD128 +$ grep -e video -e render /etc/group +video:x:39: +render:x:997: +``` + +Ubuntu 20.04: +``` +$ ls -l /dev/dri/ +total 0 +drwxr-xr-x 2 root root 80 19.10. 17:36 by-path +crw-rw---- 1 root video 226, 0 19.10. 17:36 card0 +crw-rw---- 1 root render 226, 128 19.10. 17:36 renderD128 +$ grep -e video -e render /etc/group +video:x:44: +render:x:133: +``` + +Which number to choose in your `securityContext`? Also, what if the `runAsGroup`/`runAsUser` values cannot be hard-coded because +they are automatically assigned during pod admission time via external security policies? + +Unlike volumes with `fsGroup`, the devices have no official notion of `deviceGroup`/`deviceUser` that the CRI runtimes (or kubelet) +would be able to use. We considered using container annotations set by the device plugins (e.g., `io.kubernetes.cri.hostDeviceSupplementalGroup/`) to get custom OCI `config.json` uid/gid values. +This would have required changes to all existing device plugins which was not ideal. + +Instead, a solution that is *seamless* to end-users without getting the device plugin vendors involved was preferred. The selected approach was +to re-use `runAsUser` and `runAsGroup` values in `config.json` for devices: + +```yaml +{ + "type": "c", + "path": "/dev/foo", + "major": 123, + "minor": 4, + "fileMode": 438, + "uid": <runAsUser>, + "gid": <runAsGroup> +}, +``` + +With `runc` OCI runtime (in non-rootless mode), the device is created (`mknod(2)`) in +the container namespace and the ownership is changed to `runAsUser`/`runAsGroup` using `chmod(2)`. + +{{< note >}} +[Rootless mode](/docs/tasks/administer-cluster/kubelet-in-userns/) and devices is not supported. +{{</note>}} +Having the ownership updated in the container namespace is justified as the user process is the only one accessing the device. Only `runAsUser`/`runAsGroup` +are taken into account, and, e.g., the `USER` setting in the container is currently ignored. + +While it is likely that the "faulty" deployments (i.e., non-root `securityContext` + devices) do not exist, to be absolutely sure no +deployments break, an opt-in config entry in both containerd and CRI-O to enable the new behavior was added. The following: + +`device_ownership_from_security_context (bool)` + +defaults to `false` and must be enabled to use the feature. + +## See non-root containers using devices after the fix +To demonstrate the new behavior, let's use a Data Plane Development Kit (DPDK) application using hardware accelerators, Kubernetes CPU manager, and HugePages as an example. The cluster runs containerd with: + +```toml +[plugins] + [plugins."io.containerd.grpc.v1.cri"] + device_ownership_from_security_context = true +``` + +or CRI-O with: + +```toml +[crio.runtime] +device_ownership_from_security_context = true +``` + +and the `Guaranteed` QoS Class Pod that runs DPDK's crypto-perf test utility with this YAML: + +```yaml +... +metadata: + name: qat-dpdk +spec: + securityContext: + runAsUser: 1000 + runAsGroup: 2000 + fsGroup: 3000 + containers: + - name: crypto-perf + image: intel/crypto-perf:devel + ... + resources: + requests: + cpu: "3" + memory: "128Mi" + qat.intel.com/generic: '4' + hugepages-2Mi: "128Mi" + limits: + cpu: "3" + memory: "128Mi" + qat.intel.com/generic: '4' + hugepages-2Mi: "128Mi" + ... +``` + +To verify the results, check the user and group ID that the container runs as: + +``` +$ kubectl exec -it qat-dpdk -c crypto-perf -- id +``` + +They are set to non-zero values as expected: + +``` +uid=1000 gid=2000 groups=2000,3000 +``` + +Next, check the device node permissions (`qat.intel.com/generic` exposes `/dev/vfio/` devices) are accessible to `runAsUser`/`runAsGroup`: + +``` +$ kubectl exec -it qat-dpdk -c crypto-perf -- ls -la /dev/vfio +total 0 +drwxr-xr-x 2 root root 140 Sep 7 10:55 . +drwxr-xr-x 7 root root 380 Sep 7 10:55 .. +crw------- 1 1000 2000 241, 0 Sep 7 10:55 58 +crw------- 1 1000 2000 241, 2 Sep 7 10:55 60 +crw------- 1 1000 2000 241, 10 Sep 7 10:55 68 +crw------- 1 1000 2000 241, 11 Sep 7 10:55 69 +crw-rw-rw- 1 1000 2000 10, 196 Sep 7 10:55 vfio +``` + +Finally, check the non-root container is also allowed to create HugePages: + +``` +$ kubectl exec -it qat-dpdk -c crypto-perf -- ls -la /dev/hugepages/ +``` + +`fsGroup` gives a `runAsUser` writable HugePages emptyDir mountpoint: + +``` +total 0 +drwxrwsr-x 2 root 3000 0 Sep 7 10:55 . +drwxr-xr-x 7 root root 380 Sep 7 10:55 .. +``` + +## Help us test it and provide feedback! +The functionality described here is expected to help with cluster security and the configurability of device permissions. To allow +non-root containers to use devices requires cluster admins to opt-in to the functionality by setting +`device_ownership_from_security_context = true`. To make it a default setting, please test it and provide your feedback (via SIG-Node meetings or issues)! +The flag is available in CRI-O v1.22 release and queued for containerd v1.6. + +More work is needed to get it *properly* supported. It is known to work with `runc` but it also needs to be made to function +with other OCI runtimes too, where applicable. For instance, Kata Containers supports device passthrough and allows it to make devices +available to containers in VM sandboxes too. + +Moreover, the additional challenge comes with support of user names and devices. This problem is still [open](https://github.com/kubernetes/enhancements/pull/2101) +and requires more brainstorming. + +Finally, it needs to be understood whether `runAsUser`/`runAsGroup` are enough or if device specific settings similar to `fsGroups` are needed in PodSpec/CRI v2. + +## Thanks +My thanks goes to Mike Brown (IBM, containerd), Peter Hunt (Redhat, CRI-O), and Alexander Kanevskiy (Intel) for providing all the feedback and good conversations. diff --git a/content/en/blog/_posts/2021-11-12-are-you-ready-for-dockershim-removal/index.md b/content/en/blog/_posts/2021-11-12-are-you-ready-for-dockershim-removal/index.md new file mode 100644 index 0000000000..74d2d526be --- /dev/null +++ b/content/en/blog/_posts/2021-11-12-are-you-ready-for-dockershim-removal/index.md @@ -0,0 +1,71 @@ +--- +layout: blog +title: "Dockershim removal is coming. Are you ready?" +date: 2021-11-12 +slug: are-you-ready-for-dockershim-removal +--- + +**Authors:** Sergey Kanzhelev, Google. With reviews from Davanum Srinivas, Elana Hashman, Noah Kantrowitz, Rey Lejano. + +{{% alert color="info" title="Poll closed" %}} +This poll closed on January 7, 2022. +{{% /alert %}} + +Last year we [announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation) +that Kubernetes' dockershim component (which provides a built-in integration for +Docker Engine) is deprecated. + +_Update: There's a [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/) +with more information, and you can also discuss the deprecation via a dedicated +[GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917)._ + +Our current plan is to remove dockershim from the Kubernetes codebase soon. +We are looking for feedback from you whether you are ready for dockershim +removal and to ensure that you are ready when the time comes. + +<del>Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8</del> + +The dockershim component that enables Docker as a Kubernetes container runtime is +being deprecated in favor of runtimes that directly use the [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) +created for Kubernetes. Many Kubernetes users have migrated to +other container runtimes without problems. However we see that dockershim is +still very popular. You may see some public numbers in recent [Container Report](https://www.datadoghq.com/container-report/#8) from DataDog. +Some Kubernetes hosting vendors just recently enabled other runtimes support +(especially for Windows nodes). And we know that many third party tools vendors +are still not ready: [migrating telemetry and security agents](/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents/#telemetry-and-security-agent-vendors). + +At this point, we believe that there is feature parity between Docker and the +other runtimes. Many end-users have used our [migration guide](/docs/tasks/administer-cluster/migrating-from-dockershim/) +and are running production workload using these different runtimes. The plan of +record today is that dockershim will be removed in version 1.24, slated for +release around April of next year. For those developing or running alpha and +beta versions, dockershim will be removed in December at the beginning of the +1.24 release development cycle. + +There is only one month left to give us feedback. We want you to tell us how +ready you are. + +<del>We are collecting opinions through this survey: https://forms.gle/svCJmhvTv78jGdSx8</del> +To better understand preparedness for the dockershim removal, our survey is +asking the version of Kubernetes you are currently using, and an estimate of +when you think you will adopt Kubernetes 1.24. All the aggregated information +on dockershim removal readiness will be published. +Free form comments will be reviewed by SIG Node leadership. If you want to +discuss any details of migrating from dockershim, report bugs or adoption +blockers, you can use one of the SIG Node contact options any time: +https://github.com/kubernetes/community/tree/master/sig-node#contact + +Kubernetes is a mature project. This deprecation is another +step in the effort to get away from permanent beta features and providing more +stability and compatibility guarantees. With the migration from dockershim you +will get more flexibility and choice of container runtime features as well as +less dependencies of your apps on specific underlying technology. Please take +time to review the [dockershim migration documentation](/docs/tasks/administer-cluster/migrating-from-dockershim/) +and consult your Kubernetes hosting vendor (if you have one) what container runtime options are available for you. +Read up [container runtime documentation with instructions on how to use containerd and CRI-O](/docs/setup/production-environment/container-runtimes/#container-runtimes) +to help prepare you when you're ready to upgrade to 1.24. CRI-O, containerd, and +Docker with [Mirantis cri-dockerd](https://github.com/Mirantis/cri-dockerd) are +not the only container runtime options, we encourage you to explore the [CNCF landscape on container runtimes](https://landscape.cncf.io/card-mode?category=container-runtime&grouping=category) +in case another suits you better. + +Thank you! diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-high.svg b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-high.svg new file mode 100644 index 0000000000..a11906c96b --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-high.svg @@ -0,0 +1,2 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no" ?> +<svg xmlns="http://www.w3.org/2000/svg" width="114.501ex" height="2.262ex" viewBox="0 -750 50609.3 1000" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" style=""><defs><path id="MJX-536-TEX-I-1D45A" d="M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-I-1D452" d="M39 168Q39 225 58 272T107 350T174 402T244 433T307 442H310Q355 442 388 420T421 355Q421 265 310 237Q261 224 176 223Q139 223 138 221Q138 219 132 186T125 128Q125 81 146 54T209 26T302 45T394 111Q403 121 406 121Q410 121 419 112T429 98T420 82T390 55T344 24T281 -1T205 -11Q126 -11 83 42T39 168ZM373 353Q367 405 305 405Q272 405 244 391T199 357T170 316T154 280T149 261Q149 260 169 260Q282 260 327 284T373 353Z"></path><path id="MJX-536-TEX-I-1D45C" d="M201 -11Q126 -11 80 38T34 156Q34 221 64 279T146 380Q222 441 301 441Q333 441 341 440Q354 437 367 433T402 417T438 387T464 338T476 268Q476 161 390 75T201 -11ZM121 120Q121 70 147 48T206 26Q250 26 289 58T351 142Q360 163 374 216T388 308Q388 352 370 375Q346 405 306 405Q243 405 195 347Q158 303 140 230T121 120Z"></path><path id="MJX-536-TEX-I-1D45F" d="M21 287Q22 290 23 295T28 317T38 348T53 381T73 411T99 433T132 442Q161 442 183 430T214 408T225 388Q227 382 228 382T236 389Q284 441 347 441H350Q398 441 422 400Q430 381 430 363Q430 333 417 315T391 292T366 288Q346 288 334 299T322 328Q322 376 378 392Q356 405 342 405Q286 405 239 331Q229 315 224 298T190 165Q156 25 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 114 189T154 366Q154 405 128 405Q107 405 92 377T68 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-I-1D466" d="M21 287Q21 301 36 335T84 406T158 442Q199 442 224 419T250 355Q248 336 247 334Q247 331 231 288T198 191T182 105Q182 62 196 45T238 27Q261 27 281 38T312 61T339 94Q339 95 344 114T358 173T377 247Q415 397 419 404Q432 431 462 431Q475 431 483 424T494 412T496 403Q496 390 447 193T391 -23Q363 -106 294 -155T156 -205Q111 -205 77 -183T43 -117Q43 -95 50 -80T69 -58T89 -48T106 -45Q150 -45 150 -87Q150 -107 138 -122T115 -142T102 -147L99 -148Q101 -153 118 -160T152 -167H160Q177 -167 186 -165Q219 -156 247 -127T290 -65T313 -9T321 21L315 17Q309 13 296 6T270 -6Q250 -11 231 -11Q185 -11 150 11T104 82Q103 89 103 113Q103 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-N-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path id="MJX-536-TEX-I-210E" d="M137 683Q138 683 209 688T282 694Q294 694 294 685Q294 674 258 534Q220 386 220 383Q220 381 227 388Q288 442 357 442Q411 442 444 415T478 336Q478 285 440 178T402 50Q403 36 407 31T422 26Q450 26 474 56T513 138Q516 149 519 151T535 153Q555 153 555 145Q555 144 551 130Q535 71 500 33Q466 -10 419 -10H414Q367 -10 346 17T325 74Q325 90 361 192T398 345Q398 404 354 404H349Q266 404 205 306L198 293L164 158Q132 28 127 16Q114 -11 83 -11Q69 -11 59 -2T48 16Q48 30 121 320L195 616Q195 629 188 632T149 637H128Q122 643 122 645T124 664Q129 683 137 683Z"></path><path id="MJX-536-TEX-I-1D456" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-I-1D454" d="M311 43Q296 30 267 15T206 0Q143 0 105 45T66 160Q66 265 143 353T314 442Q361 442 401 394L404 398Q406 401 409 404T418 412T431 419T447 422Q461 422 470 413T480 394Q480 379 423 152T363 -80Q345 -134 286 -169T151 -205Q10 -205 10 -137Q10 -111 28 -91T74 -71Q89 -71 102 -80T116 -111Q116 -121 114 -130T107 -144T99 -154T92 -162L90 -164H91Q101 -167 151 -167Q189 -167 211 -155Q234 -144 254 -122T282 -75Q288 -56 298 -13Q311 35 311 43ZM384 328L380 339Q377 350 375 354T369 368T359 382T346 393T328 402T306 405Q262 405 221 352Q191 313 171 233T151 117Q151 38 213 38Q269 38 323 108L331 118L384 328Z"></path><path id="MJX-536-TEX-N-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path id="MJX-536-TEX-N-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path id="MJX-536-TEX-I-1D45D" d="M23 287Q24 290 25 295T30 317T40 348T55 381T75 411T101 433T134 442Q209 442 230 378L240 387Q302 442 358 442Q423 442 460 395T497 281Q497 173 421 82T249 -10Q227 -10 210 -4Q199 1 187 11T168 28L161 36Q160 35 139 -51T118 -138Q118 -144 126 -145T163 -148H188Q194 -155 194 -157T191 -175Q188 -187 185 -190T172 -194Q170 -194 161 -194T127 -193T65 -192Q-5 -192 -24 -194H-32Q-39 -187 -39 -183Q-37 -156 -26 -148H-6Q28 -147 33 -136Q36 -130 94 103T155 350Q156 355 156 364Q156 405 131 405Q109 405 94 377T71 316T59 280Q57 278 43 278H29Q23 284 23 287ZM178 102Q200 26 252 26Q282 26 310 49T356 107Q374 141 392 215T411 325V331Q411 405 350 405Q339 405 328 402T306 393T286 380T269 365T254 350T243 336T235 326L232 322Q232 321 229 308T218 264T204 212Q178 106 178 102Z"></path><path id="MJX-536-TEX-I-1D451" d="M366 683Q367 683 438 688T511 694Q523 694 523 686Q523 679 450 384T375 83T374 68Q374 26 402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487H491Q506 153 506 145Q506 140 503 129Q490 79 473 48T445 8T417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157Q33 205 53 255T101 341Q148 398 195 420T280 442Q336 442 364 400Q369 394 369 396Q370 400 396 505T424 616Q424 629 417 632T378 637H357Q351 643 351 645T353 664Q358 683 366 683ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path id="MJX-536-TEX-I-1D460" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path id="MJX-536-TEX-I-1D450" d="M34 159Q34 268 120 355T306 442Q362 442 394 418T427 355Q427 326 408 306T360 285Q341 285 330 295T319 325T330 359T352 380T366 386H367Q367 388 361 392T340 400T306 404Q276 404 249 390Q228 381 206 359Q162 315 142 235T121 119Q121 73 147 50Q169 26 205 26H209Q321 26 394 111Q403 121 406 121Q410 121 419 112T429 98T420 83T391 55T346 25T282 0T202 -11Q127 -11 81 37T34 159Z"></path><path id="MJX-536-TEX-I-1D45B" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-I-1D461" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path id="MJX-536-TEX-I-1D44E" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path id="MJX-536-TEX-N-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path id="MJX-536-TEX-N-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path id="MJX-536-TEX-I-1D462" d="M21 287Q21 295 30 318T55 370T99 420T158 442Q204 442 227 417T250 358Q250 340 216 246T182 105Q182 62 196 45T238 27T291 44T328 78L339 95Q341 99 377 247Q407 367 413 387T427 416Q444 431 463 431Q480 431 488 421T496 402L420 84Q419 79 419 68Q419 43 426 35T447 26Q469 29 482 57T512 145Q514 153 532 153Q551 153 551 144Q550 139 549 130T540 98T523 55T498 17T462 -8Q454 -10 438 -10Q372 -10 347 46Q345 45 336 36T318 21T296 6T267 -6T233 -11Q189 -11 155 7Q103 38 103 113Q103 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-536-TEX-I-1D459" d="M117 59Q117 26 142 26Q179 26 205 131Q211 151 215 152Q217 153 225 153H229Q238 153 241 153T246 151T248 144Q247 138 245 128T234 90T214 43T183 6T137 -11Q101 -11 70 11T38 85Q38 97 39 102L104 360Q167 615 167 623Q167 626 166 628T162 632T157 634T149 635T141 636T132 637T122 637Q112 637 109 637T101 638T95 641T94 647Q94 649 96 661Q101 680 107 682T179 688Q194 689 213 690T243 693T254 694Q266 694 266 686Q266 675 193 386T118 83Q118 81 118 75T117 65V59Z"></path><path id="MJX-536-TEX-N-A0" d=""></path><path id="MJX-536-TEX-I-1D434" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path id="MJX-536-TEX-I-1D44F" d="M73 647Q73 657 77 670T89 683Q90 683 161 688T234 694Q246 694 246 685T212 542Q204 508 195 472T180 418L176 399Q176 396 182 402Q231 442 283 442Q345 442 383 396T422 280Q422 169 343 79T173 -11Q123 -11 82 27T40 150V159Q40 180 48 217T97 414Q147 611 147 623T109 637Q104 637 101 637H96Q86 637 83 637T76 640T73 647ZM336 325V331Q336 405 275 405Q258 405 240 397T207 376T181 352T163 330L157 322L136 236Q114 150 114 114Q114 66 138 42Q154 26 178 26Q211 26 245 58Q270 81 285 114T318 219Q336 291 336 325Z"></path><path id="MJX-536-TEX-I-1D440" d="M289 629Q289 635 232 637Q208 637 201 638T194 648Q194 649 196 659Q197 662 198 666T199 671T201 676T203 679T207 681T212 683T220 683T232 684Q238 684 262 684T307 683Q386 683 398 683T414 678Q415 674 451 396L487 117L510 154Q534 190 574 254T662 394Q837 673 839 675Q840 676 842 678T846 681L852 683H948Q965 683 988 683T1017 684Q1051 684 1051 673Q1051 668 1048 656T1045 643Q1041 637 1008 637Q968 636 957 634T939 623Q936 618 867 340T797 59Q797 55 798 54T805 50T822 48T855 46H886Q892 37 892 35Q892 19 885 5Q880 0 869 0Q864 0 828 1T736 2Q675 2 644 2T609 1Q592 1 592 11Q592 13 594 25Q598 41 602 43T625 46Q652 46 685 49Q699 52 704 61Q706 65 742 207T813 490T848 631L654 322Q458 10 453 5Q451 4 449 3Q444 0 433 0Q418 0 415 7Q413 11 374 317L335 624L267 354Q200 88 200 79Q206 46 272 46H282Q288 41 289 37T286 19Q282 3 278 1Q274 0 267 0Q265 0 255 0T221 1T157 2Q127 2 95 1T58 0Q43 0 39 2T35 11Q35 13 38 25T43 40Q45 46 65 46Q135 46 154 86Q158 92 223 354T289 629Z"></path><path id="MJX-536-TEX-N-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path id="MJX-536-TEX-N-2217" d="M229 286Q216 420 216 436Q216 454 240 464Q241 464 245 464T251 465Q263 464 273 456T283 436Q283 419 277 356T270 286L328 328Q384 369 389 372T399 375Q412 375 423 365T435 338Q435 325 425 315Q420 312 357 282T289 250L355 219L425 184Q434 175 434 161Q434 146 425 136T401 125Q393 125 383 131T328 171L270 213Q283 79 283 63Q283 53 276 44T250 35Q231 35 224 44T216 63Q216 80 222 143T229 213L171 171Q115 130 110 127Q106 124 100 124Q87 124 76 134T64 161Q64 166 64 169T67 175T72 181T81 188T94 195T113 204T138 215T170 230T210 250L74 315Q65 324 65 338Q65 353 74 363T98 374Q106 374 116 368T171 328L229 286Z"></path><path id="MJX-536-TEX-I-1D439" d="M48 1Q31 1 31 11Q31 13 34 25Q38 41 42 43T65 46Q92 46 125 49Q139 52 144 61Q146 66 215 342T285 622Q285 629 281 629Q273 632 228 634H197Q191 640 191 642T193 659Q197 676 203 680H742Q749 676 749 669Q749 664 736 557T722 447Q720 440 702 440H690Q683 445 683 453Q683 454 686 477T689 530Q689 560 682 579T663 610T626 626T575 633T503 634H480Q398 633 393 631Q388 629 386 623Q385 622 352 492L320 363H375Q378 363 398 363T426 364T448 367T472 374T489 386Q502 398 511 419T524 457T529 475Q532 480 548 480H560Q567 475 567 470Q567 467 536 339T502 207Q500 200 482 200H470Q463 206 463 212Q463 215 468 234T473 274Q473 303 453 310T364 317H309L277 190Q245 66 245 60Q245 46 334 46H359Q365 40 365 39T363 19Q359 6 353 0H336Q295 2 185 2Q120 2 86 2T48 1Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g data-mml-node="math"><g data-mml-node="mi"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(878, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(1344, 0)"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(2222, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(2707, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(3158, 0)"><use xlink:href="#MJX-536-TEX-I-1D466"></use></g><g data-mml-node="mo" transform="translate(3648, 0)"><use xlink:href="#MJX-536-TEX-N-2E"></use></g><g data-mml-node="mi" transform="translate(4092.7, 0)"><use xlink:href="#MJX-536-TEX-I-210E"></use></g><g data-mml-node="mi" transform="translate(4668.7, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mi" transform="translate(5013.7, 0)"><use xlink:href="#MJX-536-TEX-I-1D454"></use></g><g data-mml-node="mi" transform="translate(5490.7, 0)"><use xlink:href="#MJX-536-TEX-I-210E"></use></g><g data-mml-node="mo" transform="translate(6344.4, 0)"><use xlink:href="#MJX-536-TEX-N-3D"></use></g><g data-mml-node="mo" transform="translate(7400.2, 0)"><use xlink:href="#MJX-536-TEX-N-28"></use></g><g data-mml-node="mi" transform="translate(7789.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D45D"></use></g><g data-mml-node="mi" transform="translate(8292.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(8777.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D451"></use></g><g data-mml-node="mo" transform="translate(9297.2, 0)"><use xlink:href="#MJX-536-TEX-N-2E"></use></g><g data-mml-node="mi" transform="translate(9741.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D460"></use></g><g data-mml-node="mi" transform="translate(10210.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45D"></use></g><g data-mml-node="mi" transform="translate(10713.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(11179.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D450"></use></g><g data-mml-node="mo" transform="translate(11612.9, 0)"><use xlink:href="#MJX-536-TEX-N-2E"></use></g><g data-mml-node="mi" transform="translate(12057.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D450"></use></g><g data-mml-node="mi" transform="translate(12490.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(12975.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D45B"></use></g><g data-mml-node="mi" transform="translate(13575.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(13936.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D44E"></use></g><g data-mml-node="mi" transform="translate(14465.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mi" transform="translate(14810.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D45B"></use></g><g data-mml-node="mi" transform="translate(15410.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(15876.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(16327.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D460"></use></g><g data-mml-node="mo" transform="translate(16796.6, 0)"><use xlink:href="#MJX-536-TEX-N-5B"></use></g><g data-mml-node="mi" transform="translate(17074.6, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mo" transform="translate(17419.6, 0)"><use xlink:href="#MJX-536-TEX-N-5D"></use></g><g data-mml-node="mo" transform="translate(17697.6, 0)"><use xlink:href="#MJX-536-TEX-N-2E"></use></g><g data-mml-node="mi" transform="translate(18142.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(18593.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(19059.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D460"></use></g><g data-mml-node="mi" transform="translate(19528.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(20013.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D462"></use></g><g data-mml-node="mi" transform="translate(20585.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(21036.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D450"></use></g><g data-mml-node="mi" transform="translate(21469.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(21935.2, 0)"><use xlink:href="#MJX-536-TEX-I-1D460"></use></g><g data-mml-node="mo" transform="translate(22404.2, 0)"><use xlink:href="#MJX-536-TEX-N-2E"></use></g><g data-mml-node="mi" transform="translate(22848.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D459"></use></g><g data-mml-node="mi" transform="translate(23146.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mi" transform="translate(23491.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(24369.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mi" transform="translate(24714.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(25075.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D460"></use></g><g data-mml-node="mo" transform="translate(25544.9, 0)"><use xlink:href="#MJX-536-TEX-N-5B"></use></g><g data-mml-node="mi" transform="translate(25822.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(26700.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(27166.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(28044.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(28529.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(28980.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D466"></use></g><g data-mml-node="mo" transform="translate(29470.9, 0)"><use xlink:href="#MJX-536-TEX-N-5D"></use></g><g data-mml-node="mtext" transform="translate(29748.9, 0)"><use xlink:href="#MJX-536-TEX-N-A0"></use></g><g data-mml-node="mi" transform="translate(29998.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(30483.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mtext" transform="translate(30934.9, 0)"><use xlink:href="#MJX-536-TEX-N-A0"></use></g><g data-mml-node="mi" transform="translate(31184.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45B"></use></g><g data-mml-node="mi" transform="translate(31784.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(32269.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D451"></use></g><g data-mml-node="mi" transform="translate(32789.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(33255.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D434"></use></g><g data-mml-node="mi" transform="translate(34005.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D459"></use></g><g data-mml-node="mi" transform="translate(34303.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D459"></use></g><g data-mml-node="mi" transform="translate(34601.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(35086.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D450"></use></g><g data-mml-node="mi" transform="translate(35519.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D44E"></use></g><g data-mml-node="mi" transform="translate(36048.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(36409.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D44E"></use></g><g data-mml-node="mi" transform="translate(36938.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D44F"></use></g><g data-mml-node="mi" transform="translate(37367.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D459"></use></g><g data-mml-node="mi" transform="translate(37665.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(38131.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D440"></use></g><g data-mml-node="mi" transform="translate(39182.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D452"></use></g><g data-mml-node="mi" transform="translate(39648.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45A"></use></g><g data-mml-node="mi" transform="translate(40526.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(41011.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(41462.9, 0)"><use xlink:href="#MJX-536-TEX-I-1D466"></use></g><g data-mml-node="mo" transform="translate(41952.9, 0)"><use xlink:href="#MJX-536-TEX-N-29"></use></g><g data-mml-node="mo" transform="translate(42564.1, 0)"><use xlink:href="#MJX-536-TEX-N-2217"></use></g><g data-mml-node="mi" transform="translate(43286.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(43647.3, 0)"><use xlink:href="#MJX-536-TEX-I-210E"></use></g><g data-mml-node="mi" transform="translate(44223.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g><g data-mml-node="mi" transform="translate(44674.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(45159.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(45520.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(45881.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D459"></use></g><g data-mml-node="mi" transform="translate(46179.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D456"></use></g><g data-mml-node="mi" transform="translate(46524.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D45B"></use></g><g data-mml-node="mi" transform="translate(47124.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D454"></use></g><g data-mml-node="mi" transform="translate(47601.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D439"></use></g><g data-mml-node="mi" transform="translate(48350.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D44E"></use></g><g data-mml-node="mi" transform="translate(48879.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D450"></use></g><g data-mml-node="mi" transform="translate(49312.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D461"></use></g><g data-mml-node="mi" transform="translate(49673.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D45C"></use></g><g data-mml-node="mi" transform="translate(50158.3, 0)"><use xlink:href="#MJX-536-TEX-I-1D45F"></use></g></g></g></svg> \ No newline at end of file diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-min.svg b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-min.svg new file mode 100644 index 0000000000..f9711a641c --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/container-memory-min.svg @@ -0,0 +1,87 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!-- Generated by CodeCogs with dvisvgm 2.9.1 --> +<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='389.559851pt' height='13.50934pt' viewBox='-.239051 -.240635 389.559851 13.50934'> +<defs> +<path id='g1-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/> +<path id='g1-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/> +<path id='g1-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/> +<path id='g0-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/> +<path id='g0-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/> +<path id='g0-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/> +<path id='g0-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/> +<path id='g0-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/> +<path id='g0-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/> +<path id='g0-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g0-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g0-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/> +<path id='g0-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/> +<path id='g0-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/> +<path id='g0-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/> +<path id='g0-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/> +<path id='g0-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/> +<path id='g0-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/> +<path id='g0-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/> +</defs> +<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -64.41)'> +<use x='56.413267' y='65.753425' xlink:href='#g0-109'/> +<use x='66.652534' y='65.753425' xlink:href='#g0-101'/> +<use x='72.077974' y='65.753425' xlink:href='#g0-109'/> +<use x='82.317241' y='65.753425' xlink:href='#g0-111'/> +<use x='87.944679' y='65.753425' xlink:href='#g0-114'/> +<use x='93.545152' y='65.753425' xlink:href='#g0-121'/> +<use x='99.681804' y='65.753425' xlink:href='#g0-58'/> +<use x='102.933465' y='65.753425' xlink:href='#g0-109'/> +<use x='113.172732' y='65.753425' xlink:href='#g0-105'/> +<use x='117.166164' y='65.753425' xlink:href='#g0-110'/> +<use x='127.474599' y='65.753425' xlink:href='#g1-61'/> +<use x='139.90008' y='65.753425' xlink:href='#g0-112'/> +<use x='145.775223' y='65.753425' xlink:href='#g0-111'/> +<use x='151.402661' y='65.753425' xlink:href='#g0-100'/> +<use x='157.485354' y='65.753425' xlink:href='#g0-58'/> +<use x='160.737015' y='65.753425' xlink:href='#g0-115'/> +<use x='166.251021' y='65.753425' xlink:href='#g0-112'/> +<use x='172.126164' y='65.753425' xlink:href='#g0-101'/> +<use x='177.551604' y='65.753425' xlink:href='#g0-99'/> +<use x='182.589592' y='65.753425' xlink:href='#g0-58'/> +<use x='185.841254' y='65.753425' xlink:href='#g0-99'/> +<use x='190.879242' y='65.753425' xlink:href='#g0-111'/> +<use x='196.50668' y='65.753425' xlink:href='#g0-110'/> +<use x='203.494285' y='65.753425' xlink:href='#g0-116'/> +<use x='207.721445' y='65.753425' xlink:href='#g0-97'/> +<use x='213.866389' y='65.753425' xlink:href='#g0-105'/> +<use x='217.859822' y='65.753425' xlink:href='#g0-110'/> +<use x='224.847427' y='65.753425' xlink:href='#g0-101'/> +<use x='230.272867' y='65.753425' xlink:href='#g0-114'/> +<use x='235.873341' y='65.753425' xlink:href='#g0-115'/> +<use x='241.387347' y='65.753425' xlink:href='#g1-91'/> +<use x='244.639008' y='65.753425' xlink:href='#g0-105'/> +<use x='248.63244' y='65.753425' xlink:href='#g1-93'/> +<use x='251.884101' y='65.753425' xlink:href='#g0-58'/> +<use x='255.135763' y='65.753425' xlink:href='#g0-114'/> +<use x='260.736236' y='65.753425' xlink:href='#g0-101'/> +<use x='266.161676' y='65.753425' xlink:href='#g0-115'/> +<use x='271.675682' y='65.753425' xlink:href='#g0-111'/> +<use x='277.303119' y='65.753425' xlink:href='#g0-117'/> +<use x='283.965559' y='65.753425' xlink:href='#g0-114'/> +<use x='289.566032' y='65.753425' xlink:href='#g0-99'/> +<use x='294.604021' y='65.753425' xlink:href='#g0-101'/> +<use x='300.029461' y='65.753425' xlink:href='#g0-115'/> +<use x='305.543467' y='65.753425' xlink:href='#g0-58'/> +<use x='308.795128' y='65.753425' xlink:href='#g0-114'/> +<use x='314.395601' y='65.753425' xlink:href='#g0-101'/> +<use x='319.821042' y='65.753425' xlink:href='#g0-113'/> +<use x='325.440198' y='65.753425' xlink:href='#g0-117'/> +<use x='332.102638' y='65.753425' xlink:href='#g0-101'/> +<use x='337.528078' y='65.753425' xlink:href='#g0-115'/> +<use x='343.042083' y='65.753425' xlink:href='#g0-116'/> +<use x='347.269243' y='65.753425' xlink:href='#g0-115'/> +<use x='352.783249' y='65.753425' xlink:href='#g1-91'/> +<use x='356.03491' y='65.753425' xlink:href='#g0-109'/> +<use x='366.274177' y='65.753425' xlink:href='#g0-101'/> +<use x='371.699617' y='65.753425' xlink:href='#g0-109'/> +<use x='381.938884' y='65.753425' xlink:href='#g0-111'/> +<use x='387.566322' y='65.753425' xlink:href='#g0-114'/> +<use x='393.166795' y='65.753425' xlink:href='#g0-121'/> +<use x='399.303447' y='65.753425' xlink:href='#g1-93'/> +</g> +</svg> \ No newline at end of file diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/index.md b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/index.md new file mode 100644 index 0000000000..3085aecda2 --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/index.md @@ -0,0 +1,118 @@ +--- +layout: blog +title: 'Quality-of-Service for Memory Resources' +date: 2021-11-26 +slug: qos-memory-resources +--- + +**Authors:** Tim Xu (Tencent Cloud) + +Kubernetes v1.22, released in August 2021, introduced a new alpha feature that improves how Linux nodes implement memory resource requests and limits. + +In prior releases, Kubernetes did not support memory quality guarantees. +For example, if you set container resources as follows: +``` +apiVersion: v1 +kind: Pod +metadata: + name: example +spec: + containers: + - name: nginx + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "64Mi" + cpu: "500m" +``` +`spec.containers[].resources.requests`(e.g. cpu, memory) is designed for scheduling. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node. + +`spec.containers[].resources.limits` is passed to the container runtime when the kubelet starts a container. CPU is considered a "compressible" resource. If your app starts hitting your CPU limits, Kubernetes starts throttling your container, giving your app potentially worse performance. However, it won’t be terminated. That is what "compressible" means. + +In cgroup v1, and prior to this feature, the container runtime never took into account and effectively ignored spec.containers[].resources.requests["memory"]. This is unlike CPU, in which the container runtime consider both requests and limits. Furthermore, memory actually can't be compressed in cgroup v1. Because there is no way to throttle memory usage, if a container goes past its memory limit it will be terminated by the kernel with an OOM (Out of Memory) kill. + +Fortunately, cgroup v2 brings a new design and implementation to achieve full protection on memory. The new feature relies on cgroups v2 which most current operating system releases for Linux already provide. With this experimental feature, [quality-of-service for pods and containers](/docs/tasks/configure-pod-container/quality-service-pod/) extends to cover not just CPU time but memory as well. + +## How does it work? +Memory QoS uses the memory controller of cgroup v2 to guarantee memory resources in Kubernetes. Memory requests and limits of containers in pod are used to set specific interfaces `memory.min` and `memory.high` provided by the memory controller. When `memory.min` is set to memory requests, memory resources are reserved and never reclaimed by the kernel; this is how Memory QoS ensures the availability of memory for Kubernetes pods. And if memory limits are set in the container, this means that the system needs to limit container memory usage, Memory QoS uses `memory.high` to throttle workload approaching it's memory limit, ensuring that the system is not overwhelmed by instantaneous memory allocation. + +![](./memory-qos-cal.svg) + +The following table details the specific functions of these two parameters and how they correspond to Kubernetes container resources. + +<table> + <tr> + <th style="text-align:center">File</th> + <th style="text-align:center">Description</th> + </tr> + <tr> + <td>memory.min</td> + <td><code>memory.min</code> specifies a minimum amount of memory the cgroup must always retain, i.e., memory that can never be reclaimed by the system. If the cgroup's memory usage reaches this low limit and can’t be increased, the system OOM killer will be invoked. + <br> + <br> + <i>We map it to the container's memory request</i> + </td> + </tr> + <tr> + <td>memory.high</td> + <td><code>memory.high</code> is the memory usage throttle limit. This is the main mechanism to control a cgroup's memory use. If a cgroup's memory use goes over the high boundary specified here, the cgroup’s processes are throttled and put under heavy reclaim pressure. The default is max, meaning there is no limit. + <br> + <br> + <i>We use a formula to calculate <code>memory.high</code>, depending on container's memory limit or node allocatable memory (if container's memory limit is empty) and a throttling factor. Please refer to the KEP for more details on the formula.</i> + </td> + </tr> +</table> + +When container memory requests are made, kubelet passes `memory.min` to the back-end CRI runtime (possibly containerd, cri-o) via the `Unified` field in CRI during container creation. The `memory.min` in container level cgroup will be set to: + +![](./container-memory-min.svg) +<sub>i: the i<sup>th</sup> container in one pod</sub> + +Since the `memory.min` interface requires that the ancestor cgroup directories are all set, the pod and node cgroup directories need to be set correctly. + +`memory.min` in pod level cgroup: +![](./pod-memory-min.svg) +<sub>i: the i<sup>th</sup> container in one pod</sub> + +`memory.min` in node level cgroup: +![](./node-memory-min.svg) +<sub>i: the i<sup>th</sup> pod in one node, j: the j<sup>th</sup> container in one pod</sub> + +Kubelet will manage the cgroup hierarchy of the pod level and node level cgroups directly using runc libcontainer library, while container cgroup limits are managed by the container runtime. + +For memory limits, in addition to the original way of limiting memory usage, Memory QoS adds an additional feature of throttling memory allocation. A throttling factor is introduced as a multiplier (default is 0.8). If the result of multiplying memory limits by the factor is greater than memory requests, kubelet will set `memory.high` to the value and use `Unified` via CRI. And if the container does not specify memory limits, kubelet will use node allocatable memory instead. The `memory.high` in container level cgroup is set to: + +![](./container-memory-high.svg) +<sub>i: the i<sup>th</sup> container in one pod</sub> + +This can can help improve stability when pod memory usage increases, ensuring that memory is throttled as it approaches the memory limit. + +## How do I use it? +Here are the prerequisites for enabling Memory QoS on your Linux node, some of these are related to [Kubernetes support for cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2). + +1. Kubernetes since v1.22 +2. [runc](https://github.com/opencontainers/runc) since v1.0.0-rc93; [containerd](https://containerd.io/) since 1.4; [cri-o](https://cri-o.io/) since 1.20 +3. Linux kernel minimum version: 4.15, recommended version: 5.2+ +4. Linux image with cgroupv2 enabled or enabling cgroupv2 unified_cgroup_hierarchy manually + +OCI runtimes such as runc and crun already support cgroups v2 [`Unified`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#unified), and Kubernetes CRI has also made the desired changes to support passing [`Unified`](https://github.com/kubernetes/kubernetes/pull/102578). However, CRI Runtime support is required as well. Memory QoS in Alpha phase is designed to support containerd and cri-o. Related PR [Feature: containerd-cri support LinuxContainerResources.Unified #5627](https://github.com/containerd/containerd/pull/5627) has been merged and will be released in containerd 1.6. CRI-O [implement kube alpha features for 1.22 #5207](https://github.com/cri-o/cri-o/pull/5207) is still in WIP. + +With those prerequisites met, you can enable the memory QoS feature gate (see [Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/)). + +## How can I learn more? + +You can find more details as follows: +- [Support Memory QoS with cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos/#readme) +- [cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2/#readme) + +## How do I get involved? +You can reach SIG Node by several means: +- Slack: [#sig-node](https://kubernetes.slack.com/messages/sig-node) +- [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-node) +- [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fnode) + +You can also contact me directly: +- GitHub / Slack: @xiaoxubeii +- Email: xiaoxubeii@gmail.com \ No newline at end of file diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/memory-qos-cal.svg b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/memory-qos-cal.svg new file mode 100644 index 0000000000..545ee77b29 --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/memory-qos-cal.svg @@ -0,0 +1 @@ +<svg version="1.1" viewBox="0.0 0.0 982.8398950131234 228.2782152230971" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="p.0"><path d="m0 0l982.8399 0l0 228.27821l-982.8399 0l0 -228.27821z" clip-rule="nonzero"/></clipPath><g clip-path="url(#p.0)"><path fill="#ffffff" d="m0 0l982.8399 0l0 228.27821l-982.8399 0z" fill-rule="evenodd"/><path fill="#cfe2f3" d="m75.144356 2.3385792l245.51181 0l0 224.18898l-245.51181 0z" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m75.144356 2.3385792l245.51181 0l0 224.18898l-245.51181 0z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m-2.1023622 63.115482l479.43307 0" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" stroke-dasharray="8.0,3.0" d="m-2.1023622 63.115482l479.43307 0" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m330.11023 12.847766l314.96063 0l0 42.015747l-314.96063 0z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m-43.90813 27.41732l556.126 0l0 42.015747l-556.126 0z" fill-rule="evenodd"/><path fill="#000000" d="m8.274501 54.33732l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.191696 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm4.144822 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm15.540802 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm7.722946 -1.46875l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm0.94892883 -1.421875l1.625 -0.25q0.125 0.96875 0.75 1.5q0.625 0.515625 1.75 0.515625q1.125 0 1.671875 -0.453125q0.546875 -0.46875 0.546875 -1.09375q0 -0.546875 -0.484375 -0.875q-0.328125 -0.21875 -1.671875 -0.546875q-1.8125 -0.46875 -2.515625 -0.796875q-0.6875 -0.328125 -1.046875 -0.90625q-0.359375 -0.59375 -0.359375 -1.3125q0 -0.640625 0.296875 -1.1875q0.296875 -0.5625 0.8125 -0.921875q0.375 -0.28125 1.03125 -0.46875q0.671875 -0.203125 1.421875 -0.203125q1.140625 0 2.0 0.328125q0.859375 0.328125 1.265625 0.890625q0.421875 0.5625 0.578125 1.5l-1.609375 0.21875q-0.109375 -0.75 -0.640625 -1.171875q-0.515625 -0.421875 -1.46875 -0.421875q-1.140625 0 -1.625 0.375q-0.46875 0.375 -0.46875 0.875q0 0.3125 0.1875 0.578125q0.203125 0.265625 0.640625 0.4375q0.234375 0.09375 1.4375 0.421875q1.75 0.453125 2.4375 0.75q0.6875 0.296875 1.078125 0.859375q0.390625 0.5625 0.390625 1.40625q0 0.828125 -0.484375 1.546875q-0.46875 0.71875 -1.375 1.125q-0.90625 0.390625 -2.046875 0.390625q-1.875 0 -2.875 -0.78125q-0.984375 -0.78125 -1.25 -2.328125zm10.4375 2.890625l0 -1.875l1.875 0l0 1.875l-1.875 0zm4.730179 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.1562462 -1.6875 2.9843712 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.6718712 0.671875 -0.6718712 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165798 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141342 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.931427 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281967 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.150177 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125zm8.171875 -3.484375l3.875 -13.8125l1.3125 0l-3.859375 13.8125l-1.328125 0zm6.4176865 -0.234375l0 -9.671875l1.46875 0l0 1.375q1.0625 -1.59375 3.078125 -1.59375q0.875 0 1.609375 0.3125q0.734375 0.3125 1.09375 0.828125q0.375 0.5 0.515625 1.203125q0.09375 0.453125 0.09375 1.59375l0 5.953125l-1.640625 0l0 -5.890625q0 -1.0 -0.203125 -1.484375q-0.1875 -0.5 -0.671875 -0.796875q-0.484375 -0.296875 -1.140625 -0.296875q-1.046875 0 -1.8125 0.671875q-0.75 0.65625 -0.75 2.515625l0 5.28125l-1.640625 0zm9.766342 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm15.563217 4.84375l0 -1.21875q-0.90625 1.4375 -2.703125 1.4375q-1.15625 0 -2.125 -0.640625q-0.96875 -0.640625 -1.5 -1.78125q-0.53125 -1.140625 -0.53125 -2.625q0 -1.453125 0.484375 -2.625q0.484375 -1.1875 1.4375 -1.8125q0.96875 -0.625 2.171875 -0.625q0.875 0 1.546875 0.375q0.6875 0.359375 1.109375 0.953125l0 -4.796875l1.640625 0l0 13.359375l-1.53125 0zm-5.171875 -4.828125q0 1.859375 0.78125 2.78125q0.78125 0.921875 1.84375 0.921875q1.078125 0 1.828125 -0.875q0.75 -0.890625 0.75 -2.6875q0 -1.984375 -0.765625 -2.90625q-0.765625 -0.9375 -1.890625 -0.9375q-1.078125 0 -1.8125 0.890625q-0.734375 0.890625 -0.734375 2.8125zm15.906967 1.71875l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm20.637146 4.578125q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.703125 0q-0.265625 -0.515625 -0.328125 -1.1875zm-0.140625 -3.671875q-0.890625 0.375 -2.671875 0.625q-1.015625 0.140625 -1.4375 0.328125q-0.421875 0.1875 -0.65625 0.53125q-0.21875 0.34375 -0.21875 0.78125q0 0.65625 0.5 1.09375q0.5 0.4375 1.453125 0.4375q0.9375 0 1.671875 -0.40625q0.75 -0.421875 1.09375 -1.140625q0.265625 -0.5625 0.265625 -1.640625l0 -0.609375zm4.156967 4.859375l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.144821 0l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm3.5823212 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm15.610092 1.296875l1.609375 0.21875q-0.265625 1.65625 -1.359375 2.609375q-1.078125 0.9375 -2.671875 0.9375q-1.984375 0 -3.1875 -1.296875q-1.203125 -1.296875 -1.203125 -3.71875q0 -1.578125 0.515625 -2.75q0.515625 -1.171875 1.578125 -1.75q1.0625 -0.59375 2.3125 -0.59375q1.578125 0 2.578125 0.796875q1.0 0.796875 1.28125 2.265625l-1.59375 0.234375q-0.234375 -0.96875 -0.8125 -1.453125q-0.578125 -0.5 -1.390625 -0.5q-1.234375 0 -2.015625 0.890625q-0.78125 0.890625 -0.78125 2.8125q0 1.953125 0.75 2.84375q0.75 0.875 1.953125 0.875q0.96875 0 1.609375 -0.59375q0.65625 -0.59375 0.828125 -1.828125zm9.328125 2.359375q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.703125 0q-0.265625 -0.515625 -0.328125 -1.1875zm-0.140625 -3.671875q-0.890625 0.375 -2.671875 0.625q-1.015625 0.140625 -1.4375 0.328125q-0.421875 0.1875 -0.65625 0.53125q-0.21875 0.34375 -0.21875 0.78125q0 0.65625 0.5 1.09375q0.5 0.4375 1.453125 0.4375q0.9375 0 1.671875 -0.40625q0.75 -0.421875 1.09375 -1.140625q0.265625 -0.5625 0.265625 -1.640625l0 -0.609375zm5.7038574 4.859375l-1.515625 0l0 -13.359375l1.640625 0l0 4.765625q1.046875 -1.296875 2.65625 -1.296875q0.890625 0 1.6875 0.359375q0.796875 0.359375 1.3125 1.015625q0.515625 0.640625 0.796875 1.5625q0.296875 0.921875 0.296875 1.96875q0 2.484375 -1.234375 3.84375q-1.21875 1.359375 -2.953125 1.359375q-1.703125 0 -2.6875 -1.4375l0 1.21875zm-0.015625 -4.90625q0 1.734375 0.484375 2.515625q0.765625 1.265625 2.09375 1.265625q1.078125 0 1.859375 -0.9375q0.78125 -0.9375 0.78125 -2.78125q0 -1.890625 -0.75 -2.796875q-0.75 -0.90625 -1.828125 -0.90625q-1.0625 0 -1.859375 0.9375q-0.78125 0.9375 -0.78125 2.703125zm8.844452 4.90625l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm10.816681 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm14.324646 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.2031097 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.85935974 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165817 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141296 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.9314575 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281921 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.1502075 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125zm13.933289 -14.625l0.421875 -1.296875q1.453125 0.515625 2.109375 0.890625q-0.171875 -1.65625 -0.1875 -2.265625l1.328125 0q-0.03125 0.890625 -0.21875 2.25q0.9375 -0.46875 2.15625 -0.875l0.421875 1.296875q-1.15625 0.390625 -2.265625 0.515625q0.546875 0.484375 1.5625 1.71875l-1.09375 0.78125q-0.53125 -0.734375 -1.25 -1.96875q-0.671875 1.28125 -1.1875 1.96875l-1.078125 -0.78125q1.0625 -1.296875 1.515625 -1.71875q-1.171875 -0.234375 -2.234375 -0.515625zm16.677917 9.4375l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm1.6051636 1.46875l0 -13.359375l1.640625 0l0 4.796875q1.140625 -1.328125 2.890625 -1.328125q1.078125 0 1.859375 0.421875q0.796875 0.421875 1.140625 1.171875q0.34375 0.75 0.34375 2.171875l0 6.125l-1.640625 0l0 -6.125q0 -1.234375 -0.53125 -1.796875q-0.53125 -0.5625 -1.515625 -0.5625q-0.71875 0 -1.359375 0.390625q-0.640625 0.375 -0.921875 1.015625q-0.265625 0.640625 -0.265625 1.78125l0 5.296875l-1.640625 0zm10.360107 0l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm5.6189575 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm12.875671 3.375l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm5.1832886 0l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm1.5582886 1.46875l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.1917114 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm4.1448364 0l0 -9.671875l1.46875 0l0 1.375q1.0625 -1.59375 3.078125 -1.59375q0.875 0 1.609375 0.3125q0.734375 0.3125 1.09375 0.828125q0.375 0.5 0.515625 1.203125q0.09375 0.453125 0.09375 1.59375l0 5.953125l-1.640625 0l0 -5.890625q0 -1.0 -0.203125 -1.484375q-0.1875 -0.5 -0.671875 -0.796875q-0.484375 -0.296875 -1.140625 -0.296875q-1.046875 0 -1.8125 0.671875q-0.75 0.65625 -0.75 2.515625l0 5.28125l-1.640625 0zm10.063232 0.796875l1.59375 0.234375q0.109375 0.75 0.5625 1.078125q0.609375 0.453125 1.671875 0.453125q1.140625 0 1.75 -0.453125q0.625 -0.453125 0.84375 -1.265625q0.125 -0.5 0.109375 -2.109375q-1.0625 1.265625 -2.671875 1.265625q-2.0 0 -3.09375 -1.4375q-1.09375 -1.4375 -1.09375 -3.453125q0 -1.390625 0.5 -2.5625q0.515625 -1.171875 1.453125 -1.796875q0.953125 -0.640625 2.25 -0.640625q1.703125 0 2.8125 1.375l0 -1.15625l1.515625 0l0 8.359375q0 2.265625 -0.46875 3.203125q-0.453125 0.9375 -1.453125 1.484375q-0.984375 0.546875 -2.453125 0.546875q-1.71875 0 -2.796875 -0.78125q-1.0625 -0.765625 -1.03125 -2.34375zm1.359375 -5.8125q0 1.90625 0.75 2.78125q0.765625 0.875 1.90625 0.875q1.125 0 1.890625 -0.859375q0.765625 -0.875 0.765625 -2.734375q0 -1.78125 -0.796875 -2.671875q-0.78125 -0.90625 -1.890625 -0.90625q-1.09375 0 -1.859375 0.890625q-0.765625 0.875 -0.765625 2.625zm14.902771 5.015625l0 -8.40625l-1.453125 0l0 -1.265625l1.453125 0l0 -1.03125q0 -0.96875 0.171875 -1.453125q0.234375 -0.640625 0.828125 -1.03125q0.59375 -0.390625 1.671875 -0.390625q0.6875 0 1.53125 0.15625l-0.25 1.4375q-0.5 -0.09375 -0.953125 -0.09375q-0.75 0 -1.0625 0.328125q-0.3125 0.3125 -0.3125 1.1875l0 0.890625l1.890625 0l0 1.265625l-1.890625 0l0 8.40625l-1.625 0zm11.105164 -1.1875q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.703125 0q-0.265625 -0.515625 -0.328125 -1.1875zm-0.140625 -3.671875q-0.890625 0.375 -2.671875 0.625q-1.015625 0.140625 -1.4375 0.328125q-0.421875 0.1875 -0.65625 0.53125q-0.21875 0.34375 -0.21875 0.78125q0 0.65625 0.5 1.09375q0.5 0.4375 1.453125 0.4375q0.9375 0 1.671875 -0.40625q0.75 -0.421875 1.09375 -1.140625q0.265625 -0.5625 0.265625 -1.640625l0 -0.609375zm10.516357 1.3125l1.609375 0.21875q-0.265625 1.65625 -1.359375 2.609375q-1.078125 0.9375 -2.671875 0.9375q-1.984375 0 -3.1875 -1.296875q-1.203125 -1.296875 -1.203125 -3.71875q0 -1.578125 0.515625 -2.75q0.515625 -1.171875 1.578125 -1.75q1.0625 -0.59375 2.3125 -0.59375q1.578125 0 2.578125 0.796875q1.0 0.796875 1.28125 2.265625l-1.59375 0.234375q-0.234375 -0.96875 -0.8125 -1.453125q-0.578125 -0.5 -1.390625 -0.5q-1.234375 0 -2.015625 0.890625q-0.78125 0.890625 -0.78125 2.8125q0 1.953125 0.75 2.84375q0.75 0.875 1.953125 0.875q0.96875 0 1.609375 -0.59375q0.65625 -0.59375 0.828125 -1.828125zm6.59375 2.078125l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm0.9957886 -3.375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281982 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m75.144356 96.874016l245.51181 0" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m75.144356 96.874016l245.51181 0" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m40.41995 137.18373l314.96063 0l0 42.015747l-314.96063 0z" fill-rule="evenodd"/><path fill="#000000" d="m127.06289 164.10373l0 -9.671875l1.4687576 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.6250076 0zm12.85331 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm15.297592 9.46875l0 -4.734375q-0.375 0.546875 -1.0625 0.90625q-0.6875 0.34375 -1.46875 0.34375q-1.71875 0 -2.96875 -1.375q-1.234375 -1.375 -1.234375 -3.765625q0 -1.46875 0.5 -2.625q0.515625 -1.15625 1.46875 -1.75q0.96875 -0.59375 2.109375 -0.59375q1.796875 0 2.828125 1.515625l0 -1.296875l1.46875 0l0 13.375l-1.640625 0zm-5.046875 -8.5625q0 1.859375 0.78125 2.796875q0.78125 0.9375 1.875 0.9375q1.046875 0 1.796875 -0.890625q0.765625 -0.890625 0.765625 -2.703125q0 -1.9375 -0.796875 -2.90625q-0.796875 -0.96875 -1.875 -0.96875q-1.0625 0 -1.8125 0.90625q-0.734375 0.90625 -0.734375 2.828125zm15.594467 4.859375l0 -1.421875q-1.125 1.640625 -3.0625 1.640625q-0.859375 0 -1.609375 -0.328125q-0.734375 -0.328125 -1.09375 -0.828125q-0.359375 -0.5 -0.5 -1.21875q-0.109375 -0.46875 -0.109375 -1.53125l0 -5.984375l1.640625 0l0 5.359375q0 1.28125 0.109375 1.734375q0.15625 0.640625 0.65625 1.015625q0.5 0.375 1.234375 0.375q0.734375 0 1.375 -0.375q0.65625 -0.390625 0.921875 -1.03125q0.265625 -0.65625 0.265625 -1.890625l0 -5.1875l1.640625 0l0 9.671875l-1.46875 0zm10.672592 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm8.485092 2.875l1.625 -0.25q0.125 0.96875 0.75 1.5q0.625 0.515625 1.75 0.515625q1.125 0 1.671875 -0.453125q0.546875 -0.46875 0.546875 -1.09375q0 -0.546875 -0.484375 -0.875q-0.328125 -0.21875 -1.671875 -0.546875q-1.8125 -0.46875 -2.515625 -0.796875q-0.6875 -0.328125 -1.046875 -0.90625q-0.359375 -0.59375 -0.359375 -1.3125q0 -0.640625 0.296875 -1.1875q0.296875 -0.5625 0.8125 -0.921875q0.375 -0.28125 1.03125 -0.46875q0.671875 -0.203125 1.421875 -0.203125q1.140625 0 2.0 0.328125q0.859375 0.328125 1.265625 0.890625q0.421875 0.5625 0.578125 1.5l-1.609375 0.21875q-0.109375 -0.75 -0.640625 -1.171875q-0.515625 -0.421875 -1.46875 -0.421875q-1.140625 0 -1.625 0.375q-0.46875 0.375 -0.46875 0.875q0 0.3125 0.1875 0.578125q0.203125 0.265625 0.640625 0.4375q0.234375 0.09375 1.4375 0.421875q1.75 0.453125 2.4375 0.75q0.6875 0.296875 1.078125 0.859375q0.390625 0.5625 0.390625 1.40625q0 0.828125 -0.484375 1.546875q-0.46875 0.71875 -1.375 1.125q-0.90625 0.390625 -2.046875 0.390625q-1.875 0 -2.875 -0.78125q-0.984375 -0.78125 -1.25 -2.328125zm13.5625 1.421875l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm0.94892883 -1.421875l1.625 -0.25q0.125 0.96875 0.75 1.5q0.625 0.515625 1.75 0.515625q1.125 0 1.671875 -0.453125q0.546875 -0.46875 0.546875 -1.09375q0 -0.546875 -0.484375 -0.875q-0.328125 -0.21875 -1.671875 -0.546875q-1.8125 -0.46875 -2.515625 -0.796875q-0.6875 -0.328125 -1.046875 -0.90625q-0.359375 -0.59375 -0.359375 -1.3125q0 -0.640625 0.296875 -1.1875q0.296875 -0.5625 0.8125 -0.921875q0.375 -0.28125 1.03125 -0.46875q0.671875 -0.203125 1.421875 -0.203125q1.140625 0 2.0 0.328125q0.859375 0.328125 1.265625 0.890625q0.421875 0.5625 0.578125 1.5l-1.609375 0.21875q-0.109375 -0.75 -0.640625 -1.171875q-0.515625 -0.421875 -1.46875 -0.421875q-1.140625 0 -1.625 0.375q-0.46875 0.375 -0.46875 0.875q0 0.3125 0.1875 0.578125q0.203125 0.265625 0.640625 0.4375q0.234375 0.09375 1.4375 0.421875q1.75 0.453125 2.4375 0.75q0.6875 0.296875 1.078125 0.859375q0.390625 0.5625 0.390625 1.40625q0 0.828125 -0.484375 1.546875q-0.46875 0.71875 -1.375 1.125q-0.90625 0.390625 -2.046875 0.390625q-1.875 0 -2.875 -0.78125q-0.984375 -0.78125 -1.25 -2.328125zm10.4375 2.890625l0 -1.875l1.875 0l0 1.875l-1.875 0zm4.730179 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165802 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141342 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.931427 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281967 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.150177 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m486.67453 63.115482l0 0c2.3263855 0 4.557495 0.9241524 6.2025146 2.5691566c1.644989 1.6450043 2.5691528 3.8761063 2.5691528 6.2024994l0 64.15748c0 4.844452 3.927185 8.771652 8.771637 8.771652l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 64.157486c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m486.67453 63.115482l0 0c2.3263855 0 4.557495 0.9241524 6.2025146 2.5691566c1.644989 1.6450043 2.5691528 3.8761063 2.5691528 6.2024994l0 64.15748c0 4.844452 3.927185 8.771652 8.771637 8.771652l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 64.157486c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m486.67453 63.115482l0 0c2.3263855 0 4.557495 0.9241524 6.2025146 2.5691566c1.644989 1.6450043 2.5691528 3.8761063 2.5691528 6.2024994l0 64.15748c0 4.844452 3.927185 8.771652 8.771637 8.771652l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 64.157486c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m335.5223 96.874016l0 0c2.3263855 0 4.557495 0.92414856 6.202484 2.5691528c1.6450195 1.6450043 2.5691833 3.8761063 2.5691833 6.2024994l0 47.27558c0 4.844452 3.927185 8.7716675 8.771637 8.7716675l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 47.27559c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m335.5223 96.874016l0 0c2.3263855 0 4.557495 0.92414856 6.202484 2.5691528c1.6450195 1.6450043 2.5691833 3.8761063 2.5691833 6.2024994l0 47.27558c0 4.844452 3.927185 8.7716675 8.771637 8.7716675l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 47.27559c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m335.5223 96.874016l0 0c2.3263855 0 4.557495 0.92414856 6.202484 2.5691528c1.6450195 1.6450043 2.5691833 3.8761063 2.5691833 6.2024994l0 47.27558c0 4.844452 3.927185 8.7716675 8.771637 8.7716675l0 0c-4.844452 0 -8.771637 3.9272003 -8.771637 8.771652l0 47.27559c0 4.844452 -3.9272156 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m256.7953 140.68504l314.96063 0l0 42.015747l-314.96063 0z" fill-rule="evenodd"/><path fill="#000000" d="m364.89343 167.60504l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165802 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141357 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.931427 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281952 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.150177 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125zm8.4739685 -3.71875l0 -1.875l1.875 0l0 1.875l-1.875 0zm4.7301636 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm15.540802 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm4.1448364 0l0 -9.671875l1.46875 0l0 1.375q1.0625 -1.59375 3.078125 -1.59375q0.875 0 1.609375 0.3125q0.734375 0.3125 1.09375 0.828125q0.375 0.5 0.515625 1.203125q0.09375 0.453125 0.09375 1.59375l0 5.953125l-1.640625 0l0 -5.890625q0 -1.0 -0.203125 -1.484375q-0.1875 -0.5 -0.671875 -0.796875q-0.484375 -0.296875 -1.140625 -0.296875q-1.046875 0 -1.8125 0.671875q-0.75 0.65625 -0.75 2.515625l0 5.28125l-1.640625 0z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m408.75592 123.808395l314.96063 0l0 42.015747l-314.96063 0z" fill-rule="evenodd"/><path fill="#000000" d="m514.2488 150.7284l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165771 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141357 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.9313965 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281982 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.1502075 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125zm8.473938 -3.71875l0 -1.875l1.875 0l0 1.875l-1.875 0zm4.7301636 0l0 -13.359375l1.640625 0l0 4.796875q1.140625 -1.328125 2.890625 -1.328125q1.078125 0 1.859375 0.421875q0.796875 0.421875 1.140625 1.171875q0.34375 0.75 0.34375 2.171875l0 6.125l-1.640625 0l0 -6.125q0 -1.234375 -0.53125 -1.796875q-0.53125 -0.5625 -1.515625 -0.5625q-0.71875 0 -1.359375 0.390625q-0.640625 0.375 -0.921875 1.015625q-0.265625 0.640625 -0.265625 1.78125l0 5.296875l-1.640625 0zm10.375732 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm3.8323364 0.796875l1.59375 0.234375q0.109375 0.75 0.5625 1.078125q0.609375 0.453125 1.671875 0.453125q1.140625 0 1.75 -0.453125q0.625 -0.453125 0.84375 -1.265625q0.125 -0.5 0.109375 -2.109375q-1.0625 1.265625 -2.671875 1.265625q-2.0 0 -3.09375 -1.4375q-1.09375 -1.4375 -1.09375 -3.453125q0 -1.390625 0.5 -2.5625q0.515625 -1.171875 1.453125 -1.796875q0.953125 -0.640625 2.25 -0.640625q1.703125 0 2.8125 1.375l0 -1.15625l1.515625 0l0 8.359375q0 2.265625 -0.46875 3.203125q-0.453125 0.9375 -1.453125 1.484375q-0.984375 0.546875 -2.453125 0.546875q-1.71875 0 -2.796875 -0.78125q-1.0625 -0.765625 -1.03125 -2.34375zm1.359375 -5.8125q0 1.90625 0.75 2.78125q0.765625 0.875 1.90625 0.875q1.125 0 1.890625 -0.859375q0.765625 -0.875 0.765625 -2.734375q0 -1.78125 -0.796875 -2.671875q-0.78125 -0.90625 -1.890625 -0.90625q-1.09375 0 -1.859375 0.890625q-0.765625 0.875 -0.765625 2.625zm9.328796 5.015625l0 -13.359375l1.640625 0l0 4.796875q1.140625 -1.328125 2.890625 -1.328125q1.078125 0 1.859375 0.421875q0.796875 0.421875 1.140625 1.171875q0.34375 0.75 0.34375 2.171875l0 6.125l-1.640625 0l0 -6.125q0 -1.234375 -0.53125 -1.796875q-0.53125 -0.5625 -1.515625 -0.5625q-0.71875 0 -1.359375 0.390625q-0.640625 0.375 -0.921875 1.015625q-0.265625 0.640625 -0.265625 1.78125l0 5.296875l-1.640625 0z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m637.8268 2.3385792l0 0c2.326355 0 4.557495 0.9241538 6.2025146 2.569158c1.6449585 1.6450038 2.5691528 3.8761086 2.5691528 6.202495l0 94.551186c0 4.844452 3.927185 8.771652 8.771606 8.771652l0 0c-4.8444214 0 -8.771606 3.9272003 -8.771606 8.771652l0 94.551186c0 4.844452 -3.927246 8.771652 -8.7716675 8.771652z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m637.8268 2.3385792l0 0c2.326355 0 4.557495 0.9241538 6.2025146 2.569158c1.6449585 1.6450038 2.5691528 3.8761086 2.5691528 6.202495l0 94.551186c0 4.844452 3.927185 8.771652 8.771606 8.771652l0 0c-4.8444214 0 -8.771606 3.9272003 -8.771606 8.771652l0 94.551186c0 4.844452 -3.927246 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m637.8268 2.3385792l0 0c2.326355 0 4.557495 0.9241538 6.2025146 2.569158c1.6449585 1.6450038 2.5691528 3.8761086 2.5691528 6.202495l0 94.551186c0 4.844452 3.927185 8.771652 8.771606 8.771652l0 0c-4.8444214 0 -8.771606 3.9272003 -8.771606 8.771652l0 94.551186c0 4.844452 -3.927246 8.771652 -8.7716675 8.771652" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m615.37006 93.42519l417.29132 0l0 42.015755l-417.29132 0z" fill-rule="evenodd"/><path fill="#000000" d="m668.64746 120.34519l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.1917114 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm4.1447754 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm15.5408325 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm7.7229614 -1.46875l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625zm0.9489136 -1.421875l1.625 -0.25q0.125 0.96875 0.75 1.5q0.625 0.515625 1.75 0.515625q1.125 0 1.671875 -0.453125q0.546875 -0.46875 0.546875 -1.09375q0 -0.546875 -0.484375 -0.875q-0.328125 -0.21875 -1.671875 -0.546875q-1.8125 -0.46875 -2.515625 -0.796875q-0.6875 -0.328125 -1.046875 -0.90625q-0.359375 -0.59375 -0.359375 -1.3125q0 -0.640625 0.296875 -1.1875q0.296875 -0.5625 0.8125 -0.921875q0.375 -0.28125 1.03125 -0.46875q0.671875 -0.203125 1.421875 -0.203125q1.140625 0 2.0 0.328125q0.859375 0.328125 1.265625 0.890625q0.421875 0.5625 0.578125 1.5l-1.609375 0.21875q-0.109375 -0.75 -0.640625 -1.171875q-0.515625 -0.421875 -1.46875 -0.421875q-1.140625 0 -1.625 0.375q-0.46875 0.375 -0.46875 0.875q0 0.3125 0.1875 0.578125q0.203125 0.265625 0.640625 0.4375q0.234375 0.09375 1.4375 0.421875q1.75 0.453125 2.4375 0.75q0.6875 0.296875 1.078125 0.859375q0.390625 0.5625 0.390625 1.40625q0 0.828125 -0.484375 1.546875q-0.46875 0.71875 -1.375 1.125q-0.90625 0.390625 -2.046875 0.390625q-1.875 0 -2.875 -0.78125q-0.984375 -0.78125 -1.25 -2.328125zm10.4375 2.890625l0 -1.875l1.875 0l0 1.875l-1.875 0zm4.7301636 0l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165833 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141296 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.9314575 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281982 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.1501465 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125zm8.171875 -3.484375l3.875 -13.8125l1.3125 0l-3.859375 13.8125l-1.328125 0zm6.4176636 -0.234375l0 -9.671875l1.46875 0l0 1.375q1.0625 -1.59375 3.078125 -1.59375q0.875 0 1.609375 0.3125q0.734375 0.3125 1.09375 0.828125q0.375 0.5 0.515625 1.203125q0.09375 0.453125 0.09375 1.59375l0 5.953125l-1.640625 0l0 -5.890625q0 -1.0 -0.203125 -1.484375q-0.1875 -0.5 -0.671875 -0.796875q-0.484375 -0.296875 -1.140625 -0.296875q-1.046875 0 -1.8125 0.671875q-0.75 0.65625 -0.75 2.515625l0 5.28125l-1.640625 0zm9.766357 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm15.563232 4.84375l0 -1.21875q-0.90625 1.4375 -2.703125 1.4375q-1.15625 0 -2.125 -0.640625q-0.96875 -0.640625 -1.5 -1.78125q-0.53125 -1.140625 -0.53125 -2.625q0 -1.453125 0.484375 -2.625q0.484375 -1.1875 1.4375 -1.8125q0.96875 -0.625 2.171875 -0.625q0.875 0 1.546875 0.375q0.6875 0.359375 1.109375 0.953125l0 -4.796875l1.640625 0l0 13.359375l-1.53125 0zm-5.171875 -4.828125q0 1.859375 0.78125 2.78125q0.78125 0.921875 1.84375 0.921875q1.078125 0 1.828125 -0.875q0.75 -0.890625 0.75 -2.6875q0 -1.984375 -0.765625 -2.90625q-0.765625 -0.9375 -1.890625 -0.9375q-1.078125 0 -1.8125 0.890625q-0.734375 0.890625 -0.734375 2.8125zm15.906982 1.71875l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm20.637146 4.578125q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.703125 0q-0.265625 -0.515625 -0.328125 -1.1875zm-0.140625 -3.671875q-0.890625 0.375 -2.671875 0.625q-1.015625 0.140625 -1.4375 0.328125q-0.421875 0.1875 -0.65625 0.53125q-0.21875 0.34375 -0.21875 0.78125q0 0.65625 0.5 1.09375q0.5 0.4375 1.453125 0.4375q0.9375 0 1.671875 -0.40625q0.75 -0.421875 1.09375 -1.140625q0.265625 -0.5625 0.265625 -1.640625l0 -0.609375zm4.1569214 4.859375l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.1448364 0l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm3.5823364 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm15.610107 1.296875l1.609314 0.21875q-0.265625 1.65625 -1.359314 2.609375q-1.078125 0.9375 -2.671875 0.9375q-1.984375 0 -3.1875 -1.296875q-1.203125 -1.296875 -1.203125 -3.71875q0 -1.578125 0.515625 -2.75q0.515625 -1.171875 1.578125 -1.75q1.0625 -0.59375 2.3125 -0.59375q1.578125 0 2.578125 0.796875q0.99993896 0.796875 1.281189 2.265625l-1.593689 0.234375q-0.234375 -0.96875 -0.8125 -1.453125q-0.578125 -0.5 -1.390625 -0.5q-1.234375 0 -2.015625 0.890625q-0.78125 0.890625 -0.78125 2.8125q0 1.953125 0.75 2.84375q0.75 0.875 1.953125 0.875q0.96875 0 1.609375 -0.59375q0.65625 -0.59375 0.828125 -1.828125zm9.328064 2.359375q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.703125 0q-0.265625 -0.515625 -0.328125 -1.1875zm-0.140625 -3.671875q-0.890625 0.375 -2.671875 0.625q-1.015625 0.140625 -1.4375 0.328125q-0.421875 0.1875 -0.65625 0.53125q-0.21875 0.34375 -0.21875 0.78125q0 0.65625 0.5 1.09375q0.5 0.4375 1.453125 0.4375q0.9375 0 1.671875 -0.40625q0.75 -0.421875 1.09375 -1.140625q0.265625 -0.5625 0.265625 -1.640625l0 -0.609375zm5.7038574 4.859375l-1.515625 0l0 -13.359375l1.640625 0l0 4.765625q1.046875 -1.296875 2.65625 -1.296875q0.890625 0 1.6875 0.359375q0.796875 0.359375 1.3125 1.015625q0.515625 0.640625 0.796875 1.5625q0.296875 0.921875 0.296875 1.96875q0 2.484375 -1.234375 3.84375q-1.21875 1.359375 -2.953125 1.359375q-1.703125 0 -2.6875 -1.4375l0 1.21875zm-0.015625 -4.90625q0 1.734375 0.484375 2.515625q0.765625 1.265625 2.09375 1.265625q1.078125 0 1.859375 -0.9375q0.78125 -0.9375 0.78125 -2.78125q0 -1.890625 -0.75 -2.796875q-0.75 -0.90625 -1.828125 -0.90625q-1.0625 0 -1.859375 0.9375q-0.78125 0.9375 -0.78125 2.703125zm8.844482 4.90625l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm10.816711 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm14.324646 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm22.165771 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141357 5.765625l0 -9.671875l1.46875 0l0 1.359375q0.453125 -0.71875 1.203125 -1.140625q0.765625 -0.4375 1.71875 -0.4375q1.078125 0 1.765625 0.453125q0.6875 0.4375 0.96875 1.234375q1.15625 -1.6875 2.984375 -1.6875q1.453125 0 2.21875 0.796875q0.78125 0.796875 0.78125 2.453125l0 6.640625l-1.640625 0l0 -6.09375q0 -0.984375 -0.15625 -1.40625q-0.15625 -0.4375 -0.578125 -0.703125q-0.421875 -0.265625 -0.984375 -0.265625q-1.015625 0 -1.6875 0.6875q-0.671875 0.671875 -0.671875 2.15625l0 5.625l-1.640625 0l0 -6.28125q0 -1.09375 -0.40625 -1.640625q-0.40625 -0.546875 -1.3125 -0.546875q-0.6875 0 -1.28125 0.359375q-0.59375 0.359375 -0.859375 1.0625q-0.25 0.703125 -0.25 2.03125l0 5.015625l-1.640625 0zm14.9313965 -4.84375q0 -2.6875 1.484375 -3.96875q1.25 -1.078125 3.046875 -1.078125q2.0 0 3.265625 1.3125q1.265625 1.296875 1.265625 3.609375q0 1.859375 -0.5625 2.9375q-0.5625 1.0625 -1.640625 1.65625q-1.0625 0.59375 -2.328125 0.59375q-2.03125 0 -3.28125 -1.296875q-1.25 -1.3125 -1.25 -3.765625zm1.6875 0q0 1.859375 0.796875 2.796875q0.8125 0.921875 2.046875 0.921875q1.21875 0 2.03125 -0.921875q0.8125 -0.9375 0.8125 -2.84375q0 -1.796875 -0.8125 -2.71875q-0.8125 -0.921875 -2.03125 -0.921875q-1.234375 0 -2.046875 0.921875q-0.796875 0.90625 -0.796875 2.765625zm9.281982 4.84375l0 -9.671875l1.46875 0l0 1.46875q0.5625 -1.03125 1.03125 -1.359375q0.484375 -0.328125 1.0625 -0.328125q0.828125 0 1.6875 0.53125l-0.5625 1.515625q-0.609375 -0.359375 -1.203125 -0.359375q-0.546875 0 -0.96875 0.328125q-0.421875 0.328125 -0.609375 0.890625q-0.28125 0.875 -0.28125 1.921875l0 5.0625l-1.625 0zm6.1501465 3.71875l-0.1875 -1.53125q0.546875 0.140625 0.9375 0.140625q0.546875 0 0.875 -0.1875q0.328125 -0.171875 0.546875 -0.5q0.15625 -0.25 0.5 -1.21875q0.046875 -0.140625 0.140625 -0.40625l-3.671875 -9.6875l1.765625 0l2.015625 5.59375q0.390625 1.078125 0.703125 2.25q0.28125 -1.125 0.671875 -2.203125l2.078125 -5.640625l1.640625 0l-3.6875 9.828125q-0.59375 1.609375 -0.921875 2.203125q-0.4375 0.8125 -1.0 1.1875q-0.5625 0.375 -1.34375 0.375q-0.484375 0 -1.0625 -0.203125z" fill-rule="nonzero"/></g></svg> \ No newline at end of file diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/node-memory-min.svg b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/node-memory-min.svg new file mode 100644 index 0000000000..7c03aafc28 --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/node-memory-min.svg @@ -0,0 +1,98 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!-- Generated by CodeCogs with dvisvgm 2.9.1 --> +<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='446.671233pt' height='30.306891pt' viewBox='-.239051 -.248234 446.671233 30.306891'> +<defs> +<path id='g0-88' d='M15.135243 16.737235L16.581818 12.911582H16.282939C15.816687 14.154919 14.54944 14.96787 13.174595 15.326526C12.923537 15.386301 11.75193 15.697136 9.456538 15.697136H2.247572L8.332752 8.5599C8.416438 8.464259 8.440349 8.428394 8.440349 8.368618C8.440349 8.344707 8.440349 8.308842 8.356663 8.18929L2.785554 .573848H9.336986C10.938979 .573848 12.026899 .74122 12.134496 .765131C12.780075 .860772 13.820174 1.06401 14.764633 1.661768C15.063512 1.853051 15.876463 2.391034 16.282939 3.359402H16.581818L15.135243 0H1.004234C.729265 0 .71731 .011955 .681445 .083686C.669489 .119552 .669489 .3467 .669489 .478207L6.993773 9.133748L.800996 16.390535C.681445 16.533998 .681445 16.593773 .681445 16.605729C.681445 16.737235 .789041 16.737235 1.004234 16.737235H15.135243Z'/> +<path id='g1-105' d='M2.375093-4.97335C2.375093-5.148692 2.247572-5.276214 2.064259-5.276214C1.857036-5.276214 1.625903-5.084932 1.625903-4.845828C1.625903-4.670486 1.753425-4.542964 1.936737-4.542964C2.14396-4.542964 2.375093-4.734247 2.375093-4.97335ZM1.211457-2.048319L.781071-.948443C.74122-.828892 .70137-.73325 .70137-.597758C.70137-.207223 1.004234 .079701 1.42665 .079701C2.199751 .079701 2.526526-1.036115 2.526526-1.139726C2.526526-1.219427 2.462765-1.243337 2.406974-1.243337C2.311333-1.243337 2.295392-1.187547 2.271482-1.107846C2.088169-.470237 1.761395-.143462 1.44259-.143462C1.346949-.143462 1.251308-.183313 1.251308-.398506C1.251308-.589788 1.307098-.73325 1.41071-.980324C1.490411-1.195517 1.570112-1.41071 1.657783-1.625903L1.904857-2.271482C1.976588-2.454795 2.072229-2.701868 2.072229-2.83736C2.072229-3.235866 1.753425-3.514819 1.346949-3.514819C.573848-3.514819 .239103-2.399004 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.239601 .494147-2.319303C.71731-3.076463 1.083935-3.291656 1.323039-3.291656C1.43462-3.291656 1.514321-3.251806 1.514321-3.028643C1.514321-2.948941 1.506351-2.83736 1.42665-2.598257L1.211457-2.048319Z'/> +<path id='g1-106' d='M3.291656-4.97335C3.291656-5.124782 3.172105-5.276214 2.980822-5.276214C2.741719-5.276214 2.534496-5.053051 2.534496-4.845828C2.534496-4.694396 2.654047-4.542964 2.84533-4.542964C3.084433-4.542964 3.291656-4.766127 3.291656-4.97335ZM1.625903 .398506C1.506351 .884682 1.115816 1.40274 .629639 1.40274C.502117 1.40274 .382565 1.370859 .366625 1.362889C.613699 1.243337 .645579 1.028144 .645579 .956413C.645579 .765131 .502117 .661519 .334745 .661519C.103611 .661519-.111582 .860772-.111582 1.123786C-.111582 1.42665 .183313 1.625903 .637609 1.625903C1.123786 1.625903 2.000498 1.323039 2.239601 .366625L2.956912-2.486675C2.980822-2.582316 2.996762-2.646077 2.996762-2.765629C2.996762-3.203985 2.646077-3.514819 2.183811-3.514819C1.338979-3.514819 .844832-2.399004 .844832-2.295392C.844832-2.223661 .900623-2.191781 .964384-2.191781C1.052055-2.191781 1.060025-2.215691 1.115816-2.335243C1.354919-2.885181 1.761395-3.291656 2.1599-3.291656C2.327273-3.291656 2.422914-3.180075 2.422914-2.917061C2.422914-2.805479 2.399004-2.693898 2.375093-2.582316L1.625903 .398506Z'/> +<path id='g3-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/> +<path id='g3-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/> +<path id='g3-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/> +<path id='g2-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/> +<path id='g2-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/> +<path id='g2-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/> +<path id='g2-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/> +<path id='g2-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/> +<path id='g2-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/> +<path id='g2-106' d='M4.184309-3.789788C4.23213-3.981071 4.23213-4.148443 4.23213-4.196264C4.23213-4.889664 3.718057-5.272229 3.180075-5.272229C1.972603-5.272229 1.327024-3.526775 1.327024-3.443088C1.327024-3.383313 1.374844-3.335492 1.446575-3.335492C1.542217-3.335492 1.554172-3.383313 1.613948-3.502864C2.092154-4.662516 2.689913-5.033126 3.144209-5.033126C3.395268-5.033126 3.526775-4.901619 3.526775-4.483188C3.526775-4.196264 3.490909-4.076712 3.443088-3.861519L2.307347 .645579C2.080199 1.530262 1.518306 2.199751 .860772 2.199751C.812951 2.199751 .561893 2.199751 .334745 2.080199C.621669 2.020423 .848817 1.793275 .848817 1.506351C.848817 1.315068 .705355 1.123786 .442341 1.123786C.131507 1.123786-.155417 1.3868-.155417 1.745455C-.155417 2.235616 .37061 2.438854 .860772 2.438854C1.685679 2.438854 2.773599 1.829141 3.072478 .633624L4.184309-3.789788ZM4.674471-7.460025C4.674471-7.758904 4.423412-7.854545 4.27995-7.854545C3.957161-7.854545 3.682192-7.543711 3.682192-7.280697C3.682192-7.10137 3.825654-6.886177 4.088667-6.886177C4.363636-6.886177 4.674471-7.149191 4.674471-7.460025Z'/> +<path id='g2-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g2-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g2-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/> +<path id='g2-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/> +<path id='g2-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/> +<path id='g2-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/> +<path id='g2-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/> +<path id='g2-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/> +<path id='g2-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/> +<path id='g2-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/> +</defs> +<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -66.444003)'> +<use x='56.413267' y='69.937791' xlink:href='#g2-109'/> +<use x='66.652534' y='69.937791' xlink:href='#g2-101'/> +<use x='72.077974' y='69.937791' xlink:href='#g2-109'/> +<use x='82.317241' y='69.937791' xlink:href='#g2-111'/> +<use x='87.944679' y='69.937791' xlink:href='#g2-114'/> +<use x='93.545152' y='69.937791' xlink:href='#g2-121'/> +<use x='99.681804' y='69.937791' xlink:href='#g2-58'/> +<use x='102.933465' y='69.937791' xlink:href='#g2-109'/> +<use x='113.172732' y='69.937791' xlink:href='#g2-105'/> +<use x='117.166164' y='69.937791' xlink:href='#g2-110'/> +<use x='127.474599' y='69.937791' xlink:href='#g3-61'/> +<use x='139.90008' y='58.580327' xlink:href='#g0-88'/> +<use x='147.092819' y='83.774682' xlink:href='#g1-105'/> +<use x='159.161195' y='58.580327' xlink:href='#g0-88'/> +<use x='165.85349' y='83.774682' xlink:href='#g1-106'/> +<use x='178.422309' y='69.937791' xlink:href='#g2-112'/> +<use x='184.297452' y='69.937791' xlink:href='#g2-111'/> +<use x='189.92489' y='69.937791' xlink:href='#g2-100'/> +<use x='196.007582' y='69.937791' xlink:href='#g3-91'/> +<use x='199.259244' y='69.937791' xlink:href='#g2-105'/> +<use x='203.252676' y='69.937791' xlink:href='#g3-93'/> +<use x='206.504337' y='69.937791' xlink:href='#g2-58'/> +<use x='209.755998' y='69.937791' xlink:href='#g2-115'/> +<use x='215.270004' y='69.937791' xlink:href='#g2-112'/> +<use x='221.145147' y='69.937791' xlink:href='#g2-101'/> +<use x='226.570587' y='69.937791' xlink:href='#g2-99'/> +<use x='231.608576' y='69.937791' xlink:href='#g2-58'/> +<use x='234.860237' y='69.937791' xlink:href='#g2-99'/> +<use x='239.898226' y='69.937791' xlink:href='#g2-111'/> +<use x='245.525663' y='69.937791' xlink:href='#g2-110'/> +<use x='252.513269' y='69.937791' xlink:href='#g2-116'/> +<use x='256.740429' y='69.937791' xlink:href='#g2-97'/> +<use x='262.885373' y='69.937791' xlink:href='#g2-105'/> +<use x='266.878805' y='69.937791' xlink:href='#g2-110'/> +<use x='273.866411' y='69.937791' xlink:href='#g2-101'/> +<use x='279.291851' y='69.937791' xlink:href='#g2-114'/> +<use x='284.892324' y='69.937791' xlink:href='#g2-115'/> +<use x='290.40633' y='69.937791' xlink:href='#g3-91'/> +<use x='293.657991' y='69.937791' xlink:href='#g2-106'/> +<use x='299.173487' y='69.937791' xlink:href='#g3-93'/> +<use x='302.425148' y='69.937791' xlink:href='#g2-58'/> +<use x='305.676809' y='69.937791' xlink:href='#g2-114'/> +<use x='311.277283' y='69.937791' xlink:href='#g2-101'/> +<use x='316.702723' y='69.937791' xlink:href='#g2-115'/> +<use x='322.216728' y='69.937791' xlink:href='#g2-111'/> +<use x='327.844166' y='69.937791' xlink:href='#g2-117'/> +<use x='334.506606' y='69.937791' xlink:href='#g2-114'/> +<use x='340.107079' y='69.937791' xlink:href='#g2-99'/> +<use x='345.145068' y='69.937791' xlink:href='#g2-101'/> +<use x='350.570508' y='69.937791' xlink:href='#g2-115'/> +<use x='356.084513' y='69.937791' xlink:href='#g2-58'/> +<use x='359.336175' y='69.937791' xlink:href='#g2-114'/> +<use x='364.936648' y='69.937791' xlink:href='#g2-101'/> +<use x='370.362088' y='69.937791' xlink:href='#g2-113'/> +<use x='375.981245' y='69.937791' xlink:href='#g2-117'/> +<use x='382.643684' y='69.937791' xlink:href='#g2-101'/> +<use x='388.069124' y='69.937791' xlink:href='#g2-115'/> +<use x='393.58313' y='69.937791' xlink:href='#g2-116'/> +<use x='397.81029' y='69.937791' xlink:href='#g2-115'/> +<use x='403.324295' y='69.937791' xlink:href='#g3-91'/> +<use x='406.575957' y='69.937791' xlink:href='#g2-109'/> +<use x='416.815224' y='69.937791' xlink:href='#g2-101'/> +<use x='422.240664' y='69.937791' xlink:href='#g2-109'/> +<use x='432.479931' y='69.937791' xlink:href='#g2-111'/> +<use x='438.107368' y='69.937791' xlink:href='#g2-114'/> +<use x='443.707842' y='69.937791' xlink:href='#g2-121'/> +<use x='449.844493' y='69.937791' xlink:href='#g3-93'/> +</g> +</svg> \ No newline at end of file diff --git a/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/pod-memory-min.svg b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/pod-memory-min.svg new file mode 100644 index 0000000000..84d3a940a0 --- /dev/null +++ b/content/en/blog/_posts/2021-11-26-memory-qos-cgroups-v2/pod-memory-min.svg @@ -0,0 +1,97 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!-- Generated by CodeCogs with dvisvgm 2.9.1 --> +<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='411.32491pt' height='36.683306pt' viewBox='-.239051 -.232683 411.32491 36.683306'> +<defs> +<path id='g0-88' d='M15.135243 16.737235L16.581818 12.911582H16.282939C15.816687 14.154919 14.54944 14.96787 13.174595 15.326526C12.923537 15.386301 11.75193 15.697136 9.456538 15.697136H2.247572L8.332752 8.5599C8.416438 8.464259 8.440349 8.428394 8.440349 8.368618C8.440349 8.344707 8.440349 8.308842 8.356663 8.18929L2.785554 .573848H9.336986C10.938979 .573848 12.026899 .74122 12.134496 .765131C12.780075 .860772 13.820174 1.06401 14.764633 1.661768C15.063512 1.853051 15.876463 2.391034 16.282939 3.359402H16.581818L15.135243 0H1.004234C.729265 0 .71731 .011955 .681445 .083686C.669489 .119552 .669489 .3467 .669489 .478207L6.993773 9.133748L.800996 16.390535C.681445 16.533998 .681445 16.593773 .681445 16.605729C.681445 16.737235 .789041 16.737235 1.004234 16.737235H15.135243Z'/> +<path id='g3-48' d='M3.897385-2.542466C3.897385-3.395268 3.809714-3.913325 3.5467-4.423412C3.196015-5.124782 2.550436-5.300125 2.11208-5.300125C1.107846-5.300125 .74122-4.550934 .629639-4.327771C.342715-3.745953 .326775-2.956912 .326775-2.542466C.326775-2.016438 .350685-1.211457 .73325-.573848C1.099875 .01594 1.689664 .167372 2.11208 .167372C2.494645 .167372 3.180075 .047821 3.57858-.74122C3.873474-1.315068 3.897385-2.024408 3.897385-2.542466ZM2.11208-.055791C1.841096-.055791 1.291158-.183313 1.123786-1.020174C1.036115-1.474471 1.036115-2.223661 1.036115-2.638107C1.036115-3.188045 1.036115-3.745953 1.123786-4.184309C1.291158-4.99726 1.912827-5.076961 2.11208-5.076961C2.383064-5.076961 2.933001-4.941469 3.092403-4.216189C3.188045-3.777833 3.188045-3.180075 3.188045-2.638107C3.188045-2.16787 3.188045-1.45056 3.092403-1.004234C2.925031-.167372 2.375093-.055791 2.11208-.055791Z'/> +<path id='g3-61' d='M5.826152-2.654047C5.945704-2.654047 6.105106-2.654047 6.105106-2.83736S5.913823-3.020672 5.794271-3.020672H.781071C.661519-3.020672 .470237-3.020672 .470237-2.83736S.629639-2.654047 .749191-2.654047H5.826152ZM5.794271-.964384C5.913823-.964384 6.105106-.964384 6.105106-1.147696S5.945704-1.331009 5.826152-1.331009H.749191C.629639-1.331009 .470237-1.331009 .470237-1.147696S.661519-.964384 .781071-.964384H5.794271Z'/> +<path id='g1-105' d='M2.375093-4.97335C2.375093-5.148692 2.247572-5.276214 2.064259-5.276214C1.857036-5.276214 1.625903-5.084932 1.625903-4.845828C1.625903-4.670486 1.753425-4.542964 1.936737-4.542964C2.14396-4.542964 2.375093-4.734247 2.375093-4.97335ZM1.211457-2.048319L.781071-.948443C.74122-.828892 .70137-.73325 .70137-.597758C.70137-.207223 1.004234 .079701 1.42665 .079701C2.199751 .079701 2.526526-1.036115 2.526526-1.139726C2.526526-1.219427 2.462765-1.243337 2.406974-1.243337C2.311333-1.243337 2.295392-1.187547 2.271482-1.107846C2.088169-.470237 1.761395-.143462 1.44259-.143462C1.346949-.143462 1.251308-.183313 1.251308-.398506C1.251308-.589788 1.307098-.73325 1.41071-.980324C1.490411-1.195517 1.570112-1.41071 1.657783-1.625903L1.904857-2.271482C1.976588-2.454795 2.072229-2.701868 2.072229-2.83736C2.072229-3.235866 1.753425-3.514819 1.346949-3.514819C.573848-3.514819 .239103-2.399004 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.239601 .494147-2.319303C.71731-3.076463 1.083935-3.291656 1.323039-3.291656C1.43462-3.291656 1.514321-3.251806 1.514321-3.028643C1.514321-2.948941 1.506351-2.83736 1.42665-2.598257L1.211457-2.048319Z'/> +<path id='g1-110' d='M1.594022-1.307098C1.617933-1.42665 1.697634-1.729514 1.721544-1.849066C1.833126-2.279452 1.833126-2.287422 2.016438-2.550436C2.279452-2.940971 2.654047-3.291656 3.188045-3.291656C3.474969-3.291656 3.642341-3.124284 3.642341-2.749689C3.642341-2.311333 3.307597-1.40274 3.156164-1.012204C3.052553-.749191 3.052553-.70137 3.052553-.597758C3.052553-.143462 3.427148 .079701 3.769863 .079701C4.550934 .079701 4.877709-1.036115 4.877709-1.139726C4.877709-1.219427 4.813948-1.243337 4.758157-1.243337C4.662516-1.243337 4.646575-1.187547 4.622665-1.107846C4.431382-.454296 4.096638-.143462 3.793773-.143462C3.666252-.143462 3.602491-.223163 3.602491-.406476S3.666252-.765131 3.745953-.964384C3.865504-1.267248 4.216189-2.183811 4.216189-2.630137C4.216189-3.227895 3.801743-3.514819 3.227895-3.514819C2.582316-3.514819 2.16787-3.124284 1.936737-2.82142C1.880946-3.259776 1.530262-3.514819 1.123786-3.514819C.836862-3.514819 .637609-3.331507 .510087-3.084433C.318804-2.709838 .239103-2.311333 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.223661 .526027-2.430884C.621669-2.82142 .765131-3.291656 1.099875-3.291656C1.307098-3.291656 1.354919-3.092403 1.354919-2.917061C1.354919-2.773599 1.315068-2.622167 1.251308-2.359153C1.235367-2.295392 1.115816-1.825156 1.083935-1.713574L.789041-.518057C.757161-.398506 .70934-.199253 .70934-.167372C.70934 .01594 .860772 .079701 .964384 .079701C1.107846 .079701 1.227397-.01594 1.283188-.111582C1.307098-.159402 1.370859-.430386 1.41071-.597758L1.594022-1.307098Z'/> +<path id='g4-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/> +<path id='g4-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/> +<path id='g4-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/> +<path id='g2-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/> +<path id='g2-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/> +<path id='g2-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/> +<path id='g2-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/> +<path id='g2-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/> +<path id='g2-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/> +<path id='g2-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g2-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/> +<path id='g2-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/> +<path id='g2-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/> +<path id='g2-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/> +<path id='g2-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/> +<path id='g2-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/> +<path id='g2-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/> +<path id='g2-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/> +<path id='g2-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/> +</defs> +<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -62.281577)'> +<use x='56.413267' y='73.369366' xlink:href='#g2-109'/> +<use x='66.652534' y='73.369366' xlink:href='#g2-101'/> +<use x='72.077974' y='73.369366' xlink:href='#g2-109'/> +<use x='82.317241' y='73.369366' xlink:href='#g2-111'/> +<use x='87.944679' y='73.369366' xlink:href='#g2-114'/> +<use x='93.545152' y='73.369366' xlink:href='#g2-121'/> +<use x='99.681804' y='73.369366' xlink:href='#g2-58'/> +<use x='102.933465' y='73.369366' xlink:href='#g2-109'/> +<use x='113.172732' y='73.369366' xlink:href='#g2-105'/> +<use x='117.166164' y='73.369366' xlink:href='#g2-110'/> +<use x='127.474599' y='73.369366' xlink:href='#g4-61'/> +<use x='145.965287' y='58.425345' xlink:href='#g1-110'/> +<use x='139.90008' y='62.011901' xlink:href='#g0-88'/> +<use x='141.682474' y='87.206256' xlink:href='#g1-105'/> +<use x='144.565614' y='87.206256' xlink:href='#g3-61'/> +<use x='151.15212' y='87.206256' xlink:href='#g3-48'/> +<use x='159.161195' y='73.369366' xlink:href='#g2-112'/> +<use x='165.036338' y='73.369366' xlink:href='#g2-111'/> +<use x='170.663775' y='73.369366' xlink:href='#g2-100'/> +<use x='176.746468' y='73.369366' xlink:href='#g2-58'/> +<use x='179.998129' y='73.369366' xlink:href='#g2-115'/> +<use x='185.512135' y='73.369366' xlink:href='#g2-112'/> +<use x='191.387278' y='73.369366' xlink:href='#g2-101'/> +<use x='196.812718' y='73.369366' xlink:href='#g2-99'/> +<use x='201.850707' y='73.369366' xlink:href='#g2-58'/> +<use x='205.102368' y='73.369366' xlink:href='#g2-99'/> +<use x='210.140357' y='73.369366' xlink:href='#g2-111'/> +<use x='215.767794' y='73.369366' xlink:href='#g2-110'/> +<use x='222.7554' y='73.369366' xlink:href='#g2-116'/> +<use x='226.982559' y='73.369366' xlink:href='#g2-97'/> +<use x='233.127504' y='73.369366' xlink:href='#g2-105'/> +<use x='237.120936' y='73.369366' xlink:href='#g2-110'/> +<use x='244.108542' y='73.369366' xlink:href='#g2-101'/> +<use x='249.533982' y='73.369366' xlink:href='#g2-114'/> +<use x='255.134455' y='73.369366' xlink:href='#g2-115'/> +<use x='260.648461' y='73.369366' xlink:href='#g4-91'/> +<use x='263.900122' y='73.369366' xlink:href='#g2-105'/> +<use x='267.893554' y='73.369366' xlink:href='#g4-93'/> +<use x='271.145216' y='73.369366' xlink:href='#g2-58'/> +<use x='274.396877' y='73.369366' xlink:href='#g2-114'/> +<use x='279.99735' y='73.369366' xlink:href='#g2-101'/> +<use x='285.42279' y='73.369366' xlink:href='#g2-115'/> +<use x='290.936796' y='73.369366' xlink:href='#g2-111'/> +<use x='296.564234' y='73.369366' xlink:href='#g2-117'/> +<use x='303.226673' y='73.369366' xlink:href='#g2-114'/> +<use x='308.827147' y='73.369366' xlink:href='#g2-99'/> +<use x='313.865135' y='73.369366' xlink:href='#g2-101'/> +<use x='319.290575' y='73.369366' xlink:href='#g2-115'/> +<use x='324.804581' y='73.369366' xlink:href='#g2-58'/> +<use x='328.056242' y='73.369366' xlink:href='#g2-114'/> +<use x='333.656716' y='73.369366' xlink:href='#g2-101'/> +<use x='339.082156' y='73.369366' xlink:href='#g2-113'/> +<use x='344.701312' y='73.369366' xlink:href='#g2-117'/> +<use x='351.363752' y='73.369366' xlink:href='#g2-101'/> +<use x='356.789192' y='73.369366' xlink:href='#g2-115'/> +<use x='362.303198' y='73.369366' xlink:href='#g2-116'/> +<use x='366.530357' y='73.369366' xlink:href='#g2-115'/> +<use x='372.044363' y='73.369366' xlink:href='#g4-91'/> +<use x='375.296024' y='73.369366' xlink:href='#g2-109'/> +<use x='385.535291' y='73.369366' xlink:href='#g2-101'/> +<use x='390.960731' y='73.369366' xlink:href='#g2-109'/> +<use x='401.199998' y='73.369366' xlink:href='#g2-111'/> +<use x='406.827436' y='73.369366' xlink:href='#g2-114'/> +<use x='412.427909' y='73.369366' xlink:href='#g2-121'/> +<use x='418.564561' y='73.369366' xlink:href='#g4-93'/> +</g> +</svg> \ No newline at end of file diff --git a/content/en/blog/_posts/2021-12-01-kubernetes-1.22-release-interview.md b/content/en/blog/_posts/2021-12-01-kubernetes-1.22-release-interview.md new file mode 100644 index 0000000000..532a1252a0 --- /dev/null +++ b/content/en/blog/_posts/2021-12-01-kubernetes-1.22-release-interview.md @@ -0,0 +1,343 @@ +--- +layout: blog +title: "Contribution, containers and cricket: the Kubernetes 1.22 release interview" +date: 2021-12-01 +--- + +**Author**: Craig Box (Google) + +The Kubernetes release train rolls on, and we look ahead to the release of 1.23 next week. [As is our tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the process that brought us the previous version. + +The release team for 1.22 was led by [Savitha Raghunathan](https://twitter.com/coffeeartgirl), who was, at the time, a Senior Platform Engineer at MathWorks. [I spoke to Savitha](https://kubernetespodcast.com/episode/157-kubernetes-1.22/) on the [Kubernetes Podcast from Google](https://kubernetespodcast.com/), the weekly<super>*</super> show covering the Kubernetes and Cloud Native ecosystem. + +Our release conversations shine a light on the team that puts together each Kubernetes release. Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/) so you catch the story of 1.23. + +And in case you're interested in why the show has been on a hiatus the last few weeks, all will be revealed in the next episode! + +*This transcript has been lightly edited and condensed for clarity.* + +--- + +**CRAIG BOX: Welcome to the show, Savitha.** + +SAVITHA RAGHUNATHAN: Hey, Craig. Thanks for having me on the show. How are you today? + +**CRAIG BOX: I'm very well, thank you. I've interviewed a lot of people on the show, and you're actually the first person who's asked that of me.** + +SAVITHA RAGHUNATHAN: I'm glad. It's something that I always do. I just want to make sure the other person is good and happy. + +**CRAIG BOX: That's very kind of you. Thank you for kicking off on a wonderful foot there. I want to ask first of all — you grew up in Chennai. My association with Chennai is the [Super Kings cricket team](https://en.wikipedia.org/wiki/Chennai_Super_Kings). Was cricket part of your upbringing?** + +SAVITHA RAGHUNATHAN: Yeah. Actually, a lot. My mom loves watching cricket. I have a younger brother, and when we were growing up, we used to play cricket on the terrace. Everyone surrounding me, my best friends — and even now, my partner — loves watching cricket, too. Cricket is a part of my life. + +I stopped watching it a while ago, but I still enjoy a good game. + +**CRAIG BOX: It's probably a bit harder in the US. Everything's in a different time zone. I find, with my cricket team being on the other side of the world, that it's a lot easier when they're playing near me, as opposed to trying to keep up with what they're doing when they're playing at 3:00 in the morning.** + +SAVITHA RAGHUNATHAN: That is actually one of the things that made me lose touch with cricket. I'm going to give you a piece of interesting information. I never supported Chennai Super Kings. I always supported [Royal Challengers of Bangalore](https://en.wikipedia.org/wiki/Royal_Challengers_Bangalore). + +I once went to the stadium, and it was a match between the Chennai Super Kings and the RCB. I was the only one who was cheering whenever the RCB hit a 6, or when they were scoring. I got the stares of thousands of people looking at me. I'm like, "what are you doing?" My friends are like, "you're going to get us killed! Just stop screaming!" + +**CRAIG BOX: I hear you. As a New Zealander in the UK, there are a lot of international cricket matches I've been to where I am one of the few people dressed in the full beige kit. But I have to ask, why an affiliation with a different team?** + +SAVITHA RAGHUNATHAN: I'm not sure. When the IPL came out, I really liked Virat Kohli. He was playing for RCB at that time, and I think pretty much that's it. + +**CRAIG BOX: Well, what I know about the Chennai Super Kings is that their coach is New Zealand's finest batsmen and [air conditioning salesman](https://www.youtube.com/watch?v=vSZAaUCAclw), [Stephen Fleming](https://en.wikipedia.org/wiki/Stephen_Fleming).** + +SAVITHA RAGHUNATHAN: Oh, really? + +**CRAIG BOX: Yeah, he's a dead ringer for the guy who played the [yellow Wiggle](https://s1.reutersmedia.net/resources/r/?m=02&d=20061130&t=2&i=153531&w=&fh=545px&fw=&ll=&pl=&sq=&r=153531) back in the day.** + +SAVITHA RAGHUNATHAN: Oh, interesting. I remember the name, but I cannot put the picture and the name together. I stopped watching cricket once I moved to the States. Then, all my focus was on studies and extracurriculars. I have always been an introvert. The campus — it was a new thing for me — they had international festivals. + +And every week, they'd have some kind of new thing going on, so I'd go check them out. I wouldn't participate, but I did go out and check them out. That was a big feat for me around that time because a lot of people — and still, even now, a lot of people — they kind of scare me. I don't know how to make a conversation with everyone. + +I'll just go and say, "hi, how are you? OK, I'm good. I'm just going to move on". And I'll just go to the next person. And after two hours, I'm out of that place. + +**CRAIG BOX: Perhaps a pleasant side effect of the last 12 months — a lot fewer gatherings of people.** + +SAVITHA RAGHUNATHAN: Could be that, but I'm so excited about KubeCon. But when I think about it, I'm like "oh my God. There's going to be a lot of people. What am I going to do? I'm going to meet all my friends over there". + +Sometimes I have social anxiety like, what's going to happen? + +**CRAIG BOX: What's going to happen is you're going to ask them how they are at the beginning, and they're immediately going to be set at ease.** + +SAVITHA RAGHUNATHAN: *laughs* I hope so. + +**CRAIG BOX: Let's talk a little bit, then, about your transition from India to the US. You did your undergraduate degree in computer science at the SSN College of Engineering. How did you end up at Arizona State?** + +SAVITHA RAGHUNATHAN: I always wanted to pursue higher studies when I was in India, and I didn't have the opportunity immediately. Once I graduated from my school there, I went and I worked for a couple of years. My aim was always to get out of there and come here, do my graduate studies. + +Eventually, I want to do a PhD. I have an idea of what I want to do. I always wanted to keep studying. If there's an option that I could just keep studying and not do work or anything of that sort, I'd just pick that other one — I'll just keep studying. + +But unfortunately, you need money and other things to live and sustain in this world. So I'm like, OK, I'll take a break from studies, and I will work for a while. + +**CRAIG BOX: The road to success is littered with dreams of PhDs. I have a lot of friends who thought that that was the path they were going to take, and they've had a beautiful career and probably aren't going to go back to study. Did you use the [Matlab](https://en.wikipedia.org/wiki/MATLAB) software at all while you were going through your schooling?** + +SAVITHA RAGHUNATHAN: No, unfortunately. That is a question that everyone asks. I have not used Matlab. I haven't used it even now. I don't use it for work. I didn't have any necessity for my school work. I didn't have anything to do with Matlab. I never analysed, or did data processing, or anything, with Matlab. So unfortunately, no. + +Everyone asks me like, you're working at [MathWorks](https://en.wikipedia.org/wiki/MathWorks). Have you used Matlab? I'm like, no. + +**CRAIG BOX: Fair enough. Nor have I. But it's been around since the late 1970s, so I imagine there are a lot of people who will have come across it at some point. Do you work with a lot of people who have been working on it that whole time?** + +SAVITHA RAGHUNATHAN: Kind of. Not all the time, but I get to meet some folks who work on the product itself. Most of my interactions are with the infrastructure team and platform engineering teams at MathWorks. One other interesting fact is that when I joined the company — MathWorks has an extensive internal curriculum for training and learning, which I really love. They have an "Intro to Matlab" course, and that's on my bucket of things to do. + +It was like 500 years ago. I added it, and I never got to it. I'm like, OK, maybe this year at least I want to get to it and I want to learn something new. My partner used Matlab extensively. He misses it right now at his current employer. And he's like, "you have the entire licence! You have access to the entire suite and you haven't used it?" I'm like, "no!" + +**CRAIG BOX: Well, I have bad news for the idea of you doing a PhD, I'm sorry.** + +SAVITHA RAGHUNATHAN: Another thing is that none of my family knew about the company MathWorks and Matlab. The only person who knew was my younger brother. He was so proud. He was like, "oh my God". + +When he was 12 years old, he started getting involved in robotics and all that stuff. That's how he got introduced to Matlab. He goes absolutely bananas for the swag. So all the t-shirts, all the hoodies — any swag that I get from MathWorks goes to him, without saying. + +Over the five, six years, the things that I've got — there was only one sweatshirt that I kept for myself. Everything else I've just given to him. And he cherishes it. He's the only one in my family who knew about Matlab and MathWorks. + +Now, everyone knows, because I'm working there. They were initially like, I don't even know that company name. Is it like Amazon? I'm like, no, we make software that can send people to the moon. And we also make software that can do amazing robotic surgeries and even make a car drive on its own. That's something that I take immense pride in. + +I know I don't directly work on the product, but I'm enabling the people who are creating the product. I'm really, really proud of that. + +**CRAIG BOX: I think Jeff Bezos is working on at least two out of three of those disciplines that you mentioned before, so it's maybe a little bit like Amazon. One thing I've always thought about Matlab is that, because it's called Matlab, it solves that whole problem where [Americans call it math, and the rest of the world call it maths](https://www.grammar.com/math_vs._maths). Why do Americans think there's only one math?** + +SAVITHA RAGHUNATHAN: Definitely. I had trouble — growing up in India, it's always British English. And I had so much trouble when I moved here. So many things changed. + +One of the things is maths. I always got used to writing maths, physics, and everything. + +**CRAIG BOX: They don't call it "physic" in the US, do they?** + +SAVITHA RAGHUNATHAN: No, no, they don't. Luckily, they don't. That still stays "physics". But math — I had trouble. It's maths. Even when you do the full abbreviations like mathematics and you are still calling it math, I'm like, mm. + +**CRAIG BOX: They can do the computer science abbreviation thing and call it math-7-S or whatever the number of letters is.** + +SAVITHA RAGHUNATHAN: Just like Kubernetes. K-8-s. + +**CRAIG BOX: Your path to Kubernetes is through MathWorks. They started out as a company making software which was distributed in a physical sense — boxed copies, if you will. I understand now there is a cloud version. Can I assume that that is where the two worlds intersect?** + +SAVITHA RAGHUNATHAN: Kind of. I have interaction with the team that supports Matlab on the cloud, but I don't get to work with them on a day-to-day basis. They use Docker containers, and they are building the platform using Kubernetes. So yeah, a little bit of that. + +**CRAIG BOX: So what exactly is the platform that you are engineering day to day?** + +SAVITHA RAGHUNATHAN: Providing Kubernetes as a platform, obviously — that goes without saying — to some of the internal development teams. In the future we might expand it to more teams within the company. That is a focus area right now, so that's what we are doing. In the process, we might even get to work with the people who are deploying Matlab on the cloud, which is exciting. + +**CRAIG BOX: Now, your path to contribution to Kubernetes, you've said before, was through [fixing a 404 error on the Kubernetes.io website](https://github.com/kubernetes/website/pull/15588). Do you remember what the page was?** + +SAVITHA RAGHUNATHAN: I do. I was going to something for work, and I came across this changelog. In Kubernetes there's a nice page — once you got to the release page, there would be a long list of changelogs. + +One of the things that I fixed was, the person who worked on the feature had changed their GitHub handle, and that wasn't reflected on this page. So that was my first. I got curious and clicked on the links. One of the links was the handle, and that went to a 404. And I was like "Yeah, I'll just fix that. They have done all the hard work. They can get the credit that's due". + +It was easy. It wasn't overwhelming for me to pick it up as my first issue. Before that I logged on around Kubernetes for about six to eight months without doing anything because it was just a lot. + +**CRAIG BOX: One of the other things that you said about your initial contribution is that you had to learn how to use Git. As a very powerful tool, I find Git is a high barrier to entry for even contributing code to a project. When you want to contribute a blog post or documentation or a fix like you did before, I find it almost impossible to think how a new user would come along and do that. What was your process? Do you think that there's anything we can do to make that barrier lower for new contributors?** + +SAVITHA RAGHUNATHAN: Of course. There are more and more tutorials available these days. There is a new contributor workshop. They actually have a [GitHub workflow section](https://www.kubernetes.dev/docs/guide/github-workflow/), [how to do a pull request](https://www.kubernetes.dev/docs/guide/pull-requests/) and stuff like that. I know a couple of folks from SIG Docs that are working on which Git commands that you need, or how to get to writing something small and getting it committed. But more tutorials or more links to intro to Git would definitely help. + +The thing is also, someone like a documentation writer — they don't actually want to know the entirety of Git. Honestly, it's an ocean. I don't know how to do it. Most of the time, I still ask for help even though I work with Git on a day to day basis. There are several articles and a lot of help is available already within the community. Maybe we could just add a couple more to [kubernetes.dev](https://kubernetes.dev/). That is an amazing site for all the new contributors and existing contributors who want to build code, who want to write documentation. + +We could just add a tutorial there like, "hey, don't know Git, you are new to Git? You just need to know these main things". + +**CRAIG BOX: I find it a shame, to be honest, that people need to use Git for that, by comparison to Wikipedia where you can come along, and even though it might be written in Markdown or something like it, it seems like the barrier is a lot lower. Similar to you, I always have to look up anything more complicated than the five or six Git commands that I use on a day to day basis. Even to do simple things, I basically just go and follow a recipe which I find on the internet.** + +SAVITHA RAGHUNATHAN: This is how I got introduced to one of the amazing mentors in Kubernetes. Everyone knows him by his handle, Dims. It was my second PR to the Kubernetes website, and I made a mistake. I destroyed the Git history. I could not push my reviews and comments — I addressed them. I couldn't push them back. + +My immediate thought was to delete it and recreate, do another pull request. But then I was like, "what happens to others who have already put effort into reviewing them?" I asked for help, and Dims was there. + +I would say I just got lucky he was there. And he was like, "OK, let me walk you through". We did troubleshooting through Slack messages. I copied and pasted all the errors. Every single command that he said, I copied and pasted. And then he was like, "OK, run this one. Try this one. And do this one". + +Finally, I got it fixed. So you know what I did? I went and I stored the command history somewhere local for the next time when I run into this problem. Luckily, I haven't. But I find the contributors so helpful. They are busy. They have a lot of things to do, but they take moments to stop and help someone who's new. + +That is also another part of the reason why I stay — I want to contribute more. It's mainly the community. It's the Kubernetes community. I know you asked me about Git, and I just took the conversation to the Kubernetes community. That's how my brain works. + +**CRAIG BOX: A lot of people in the community do that and think that's fantastic, obviously, people like Dims who are just floating around on Slack and seem to have endless time. I don't know how they do it.** + +SAVITHA RAGHUNATHAN: I really want to know the secret for endless time. If I only had 48 hours in a day. I would sleep for 16 hours, and I would use the rest of the time for doing the things that I want. + +**CRAIG BOX: If I had a chance to sleep up to 48 hours a day, I think it'd be a lot more than 16.** + +**Now, one of the areas that you've been contributing to Kubernetes is in the release team. In 1.18, you were a shadow for the docs role. You led that role in 1.19. And you were a release lead shadow for versions 1,20 and 1.21 before finally leading this release, 1.22, which we will talk about soon.** + +**How did you get involved? And how did you decide which roles to take as you went through that process?** + +SAVITHA RAGHUNATHAN: That is a topic I love to talk about. This was fresh when I started learning about Kubernetes and using Kubernetes at work. And I got so much help from the community, I got interested in contributing back. + +At the first KubeCon that I attended in 2018, in Seattle, they had a speed mentoring session. Now they call it "pod mentoring". I went to the session, and said, "hey, I want to contribute. I don't know where to start". And I got a lot of information on how to get started. + +One of the places was SIG Release and the release team. I came back and diligently attended all the SIG Release meetings for four to six months. And in between, I applied to the Kubernetes release team — 1.14 and 1.15. I didn't get through. So I took a little bit of a break, and I focused on doing some documentation work. Then I applied for 1.18. + +Since I was already working on some kinds of — not like full fledged "documentation" documentation, I still don't write. I eventually want to write something really nice and full fledged documentation like other awesome folks. + +**CRAIG BOX: You'll need a lot more than 48 hours in your day to do that.** + +SAVITHA RAGHUNATHAN: *laughing* That's how I applied for the docs role, because I know a little bit about the website. I've done a few pull requests and commits. That's how I got started. I applied for that one role, and I got selected for the 1.18 team. That's how my journey just took off. + +And the next release, I was leading the documentation team. And as everyone knows, the pandemic hit. It was one of the longest releases. I could lean back on the community. I would just wait for the release team meetings. + +It was my way of coping with the pandemic. It took my mind off. It was actually more than a release team, they were people. They were all people first, and we took care of each other. So it felt good. + +And then, I became a release lead shadow for 1.20 and 1.21 because I wanted to know more. I wanted to learn more. I wasn't ready. I still don't feel ready, but I have led 1.22. So if I could do it, anyone could do it. + +**CRAIG BOX: How much of this work is day job?** + +SAVITHA RAGHUNATHAN: I am lucky to be blessed with an awesome team. I do most of my work after work, but there have been times where I have to take meetings and attend to immediate urgent stuff. During the time of exception requests and stuff like that, I take a little bit of time from my work. + +My team has been wonderful: they support me in all possible ways, and the management as well. Other than the meetings, I don't do much of the work during the day job. It just takes my focus and attention away too much, and I end up having to spend a lot of time sitting in front of the computer, which I don't like. + +Before the pandemic I had a good work life balance. I'd just go to work at 7:00, 7:30, and I'd be back by 4 o'clock. I never touched my laptop ever again. I left all work behind when I came home. So right now, I'm still learning how to get through. + +I try to limit the amount of open source work that I do during work time. The release lead shadow and the release lead job — they require a lot of time, effort. So on average, I'd be spending two to three hours post work time on the release activities. + +**CRAIG BOX: Before the pandemic, everyone was worried that if we let people work from home, they wouldn't work enough. I think the opposite has actually happened, is that now we're worried that if we let people work from home, they will just get on the computer in the morning and you'll have to pry it out of their hands at midnight.** + +SAVITHA RAGHUNATHAN: Yeah, I think the productivity has increased at least twofold, I would say, for everyone, once they started working from home. + +**CRAIG BOX: But at the expense of work-life balance, though, because as you say, when you're sitting in the same chair in front of, perhaps, the same computer doing your MathWorks work and then your open source work, they kind of can blur into one perhaps?** + +SAVITHA RAGHUNATHAN: That is a challenge. I face it every day. But so many others are also facing it. I implemented a few little tricks to help me. When I used to come back home from work, the first thing I would do is remove my watch. That was an indication that OK, I'm done. + +That's the thing that I still do. I just remove my watch, and I just keep it right where my workstation is. And I just close the door so that I never look back. Even going past the room, I don't get a glimpse of my work office. I start implementing tiny little things like that to avoid burnout. + +I think I'm still facing a little bit of burnout. I don't know if I have fully recovered from it. I constantly feel like I need a vacation. And I could just take a vacation for like a month or two. If it's possible, I will just do it. + +**CRAIG BOX: I do hope that travel opens up for everyone as an opportunity because I know that, for a lot of people, it's not so much they've been working from home but they've been living at work. The idea of taking vacation effectively means, well, I've been stuck in the same place, if I've been under a lockdown. It's hard to justify that. It will be good as things improve worldwide for us to be able to start focusing more on mental health and perhaps getting away from the "everything room," as I sometimes call it.** + +SAVITHA RAGHUNATHAN: I'm totally looking forward to it. I hope that travel opens up and I could go home and I could meet my siblings and my aunt and my parents. + +**CRAIG BOX: Catch a cricket match?** + +SAVITHA RAGHUNATHAN: Yeah. Probably yes, if I have company and if there is anything interesting happening around the time. I don't mind going back to the Chepauk Stadium and catching a match or two. + +**CRAIG BOX: Let's turn now to the recently released [Kubernetes 1.22](https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/). Congratulations on the launch.** + +SAVITHA RAGHUNATHAN: Thank you. + +**CRAIG BOX: Each launch comes with a theme and a mascot or a logo. What is the theme for this release?** + +SAVITHA RAGHUNATHAN: The theme for the release is reaching new peaks. I am fascinated with a lot of space travel and chasing stars, the Milky Way. The best place to do that is over the top of a mountain. So that is the release logo, basically. It's a mountain — Mount Rainier. On top of that, there is a Kubernetes flag, and it's overlooking the Milky Way. + +It's also symbolic that with every release, that we are achieving something new, bigger, and better, and we are making the release awesome. So I just wanted to incorporate that into the team as to say, we are achieving new things with every release. That's the "reaching new peaks" theme. + +**CRAIG BOX: The last couple of releases have both been incrementally larger — as a result, perhaps, of the fact there are now only three releases per year rather than four. There were also changes to the process, where the work has been driven a lot more by the SIGs than by the release team having to go and ask the SIGs what was going on. What can you say about the size and scope of the 1.22 release?** + +SAVITHA RAGHUNATHAN: The 1.22 release is the largest release to date. We have 56 enhancements if I'm not wrong, and we have a good amount of features that's graduated as stable. You can now say that Kubernetes as a project has become more mature because you see new features coming in. At the same time, you see the features that weren't used getting deprecated — we have like three deprecations in this release. + +Aside from that fact, we also have a big team that's supporting one of the longest releases. This is the first official release cycle after the cadence KEP got approved. Officially, we are at four months, even though 1.19 was six months, and 1.21 was like 3 and 1/2 months, I think, this is the first one after the official KEP approval. + +**CRAIG BOX: What changes did you make to the process knowing that you had that extra month?** + +SAVITHA RAGHUNATHAN: One of the things the community had asked for is more time for development. We tried to incorporate that in the release schedule. We had about six weeks between the enhancements freeze and the code freeze. That's one. + +It might not be visible to everyone, but one of the things that I wanted to make sure of was the health of the team — since it was a long, long release, we had time to plan out, and not have everyone work during the weekends or during their evenings or time off. That actually helped everyone keep their sanity, and also in making good progress and delivering good results at the end of the release. That's one of the process improvements that I'd call out. + +We got better by making a post during the exception request process. Everyone works around the world. People from the UK start a little earlier than the people in the US East Coast. The West Coast starts three hours later than the East Coast. We used to make a post every Friday evening saying "hey, we actually received this many requests. We have addressed a number of them. We are waiting on a couple, or whatever. All the release team members are done for the day. We will see you around on Monday. Have a good weekend." Something like that. + +We set the expectations from the community as well. We understand things are really important and urgent, but we are done. This gave everyone their time back. They don't have to worry over the weekend thinking like, hey, what's happening? What's happening in the release? They could spend time with their family, or they could do whatever they want to do, like go on a hike, or just sit and watch TV. + +There have been weekends that I just did that. I just binge-watched a series. That's what I did. + +**CRAIG BOX: Any recommendations?** + +SAVITHA RAGHUNATHAN: I'm a big fan of Marvel, so I have watched the new [Loki](https://en.wikipedia.org/wiki/Loki_(TV_series)), which I really love. Loki is one of my favourite characters in Marvel. And I also liked [WandaVision](https://en.wikipedia.org/wiki/WandaVision). That was good, too. + +**CRAIG BOX: I've not seen Loki yet, but I've heard it described as the best series of Doctor Who in the last few years.** + +SAVITHA RAGHUNATHAN: Really? + +**CRAIG BOX: There must be an element of time-travelling in there if that's how people are describing it.** + +SAVITHA RAGHUNATHAN: You should really go and watch it whenever you have time. It's really amazing. I might go back and watch it again because I might have missed bits and pieces. That always happens in Marvel movies and the episodes; you need to watch them a couple of times to catch, "oh, this is how they relate". + +**CRAIG BOX: Yes, the mark of good media that you want to immediately go back and watch it again once you've seen it.** + +**Let's look now at some of the new features in Kubernetes 1.22. A couple of things that have graduated to general availability — server-side apply, external credential providers, a couple of new security features — the replacement for pod security policy has been announced, and seccomp is now available by default.** + +**Do you have any favourite features in 1.22 that you'd like to discuss?** + +SAVITHA RAGHUNATHAN: I have a lot of them. All my favourite features are related to security. OK, one of them is not security, but a major theme of my favourite KEPs is security. I'll start with the [default seccomp](https://github.com/kubernetes/enhancements/issues/2413). I think it will help make clusters secure by default, and may assist in preventing more vulnerabilities, which means less headaches for the cluster administrators. + +This is close to my heart because the base of the MathWorks platform is provisioning Kubernetes clusters. Knowing that they are secure by default will definitely provide me with some good sleep. And also, I'm paranoid about security most of the time. I'm super interested in making everything secure. It might get in the way of making the users of the platform angry because it's not usable in any way. + +My next one is [rootless Kubelet](https://github.com/kubernetes/enhancements/issues/2033). That feature's going to enable the cluster admin, the platform developers to deploy Kubernetes components to run in a user namespace. And I think that is also a great addition. + +Like you mention, the most awaited drop in for the PSP replacement is here. It's [pod admission control](https://github.com/kubernetes/enhancements/issues/2579). It lets cluster admins apply the pod security standards. And I think it's just not related to the cluster admins. I might have to go back and check on that. Anyone can probably use it — the developers and the admins alike. + +It also supports various modes, which is most welcome. There are times where you don't want to just cut the users off because they are trying to do something which is not securely correct. You just want to warn them, hey, this is what you are doing. This might just cause a security issue later, so you might want to correct it. But you just don't want to cut them off from using the platform, or them trying to attempt to do something — deploy their workload and get their day-to-day job done. That is something that I really like, that it also supports a warning mechanism. + +Another one which is not security is [node swap support](https://github.com/kubernetes/enhancements/issues/2400). Kubernetes didn't have support for swap before, but it is taken into consideration now. This is an alpha feature. With this, you can take advantage of the swap, which is provisioned on the Linux VMs. + +Some of the workloads — when they are deployed, they might need a lot of swap for the start-up — example, like Node and Java applications, which I just took out of their KEP user stories. So if anyone's interested, they can go and look in the KEP. That's useful. And it also increases the node stability and whatnot. So I think it's going to be beneficial for a lot of folks. + +We know how Java and containers work. I think it has gotten better, but five years ago, it was so hard to get a Java application to fit in a small container. It always needed a lot of memory, swap, and everything to start up and run. I think this will help the users and help the admins and keep the cost low, and it will tie into so many other things as well. I'm excited about that feature. + +Another feature that I want to just call out — I don't use Windows that much, but I just want to give a shout out to the folks who are doing an amazing job bringing all the Kubernetes features to Windows as well, to give a seamless experience. + +One of the things is [Windows privileged containers](https://github.com/kubernetes/enhancements/issues/1981). I think it went alpha this release. And that is a wonderful addition, if you ask me. It can take advantage of whatever that's happening on the Linux side. And they can also port it over and see, OK, I can now run Windows containers in a privileged mode. + +So whatever they are trying to achieve, they can do it. So that's a noteworthy mention. I need to give a shout out for the folks who work and make things happen in the Windows ecosystem as well. + +**CRAIG BOX: One of the things that's great about the release process is the continuity between groups and teams. There's always an emeritus advisor who was a lead from a previous release. One thing that I always ask when I do these interviews is, what is the advice that you give to the next person? When [we talked to Nabarun for the 1.21 interview](https://kubernetespodcast.com/episode/146-kubernetes-1.21/), he said that his advice to you would be "do, delegate, and defer". Figure out what you can do, figure out what you can ask other people to do, and figure out what doesn't need to be done. Were you able to take that advice on board?** + +SAVITHA RAGHUNATHAN: Yeah, you won't believe it. [I have it right here stuck to my monitor.](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/3) + +**CRAIG BOX: Next to your Git cheat sheet?** + +SAVITHA RAGHUNATHAN: *laughs* Absolutely. I just have it stuck there. I just took a look at it. + +**CRAIG BOX: Someone that you will have been able to delegate and defer to is Rey Lejano from Rancher Labs and SUSE, who is the release lead to be for 1.23.** + +SAVITHA RAGHUNATHAN: I want to tell Rey to beware of the team's mental health. Schedule in such a way that it avoids burnout. Check in, and make sure that everyone is doing good. If they need some kind of help, create a safe space where they can actually ask for help, if they want to step back, if they need someone to cover. + +I think that is most important. The releases are successful based on the thousands and thousands of contributors. But when it comes to a release team, you need to have a healthy team where people feel they are in a good place and they just want to make good contributions, which means they want to be heard. That's one thing that I want to tell Rey. + +Also collaborate and learn from each other. I constantly learn. I think the team was 39 folks, including me. Every day I learned something or the other, even starting from how to interact. + +Sometimes I have learned more leadership skills from my release lead shadows. They are awesome, and they are mature. I constantly learn from them, and I admire them a lot. + +It also helps to have good, strong individuals in the team who can step up and help when needed. For example, unfortunately, we lost one of our teammates after the start of the release cycle. That was tragic. His name was [Peeyush Gupta](https://github.com/cncf/memorials/blob/main/peeyush-gupta.md). He was an awesome and wonderful human — very warm. + +I didn't get more of a chance to interact with him. I had exchanged a few Slack messages, but I got his warm personality. I just want to take a couple of seconds to remember him. He was awesome. + +After we lost him, we had this strong person from the team step up and lead the communications, who had never been a part of the release team before at all. He was a shadow for the first time. His name is Jesse Butler. So he stepped up, and he just took it away. He ran the comms show for 1.22. + +That's what the community is about. You take care of team members, and the team will take care of you. So that's one other thing that I want to let Rey know, and maybe whoever — I think it's applicable overall. + +**CRAIG BOX: There's a link to a [family education fund for Peeyush Gupta](https://milaap.org/fundraisers/support-peeyush-gupta-family-education), which you can find in the show notes.** + +**Five releases in a row now you've been a member of the release team. Will you be putting your feet up now for 1.23?** + +SAVITHA RAGHUNATHAN: I am going to take a break for a while. In the future, I want to be contributing, if not the release team, the SIG Release and the release management effort. But right now, I have been there for five releases. And I feel like, OK, I just need a little bit of fresh air. + +And also the pandemic and the burnout has caught up, so I'm going to take a break from certain contributions. You will see me in the future. I will be around, but I might not be actively participating in the release team activities. I will be around the community. Anyone can reach out to me. They all know my Slack, so they can just reach out to me via Slack or Twitter. + +**CRAIG BOX: Yes, your Twitter handle is CoffeeArtGirl. Does that mean that you'll be spending some time working on your lattes?** + +SAVITHA RAGHUNATHAN: I am very bad at making lattes. The coffee art means that I used to [make art with coffee](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/1). You get instant coffee powder and just mix it with water. You get the colours, very beautiful brown colours. I used to make art using that. + +And I love coffee. So I just combined all the words together. And I had to come up with it in a span of one hour or so because I was joining this 'meet our contributors' panel. And Paris asked me, "do you have a Twitter handle?" I was planning to create one, but I didn't have the time. + +I'm like, well, let me just think what I could just come up with real quick. So I just came up with that. So that's the story behind my Twitter handle. Everyone's interested in it. You are not the first person you have asked me or mentioned about it. So many others are like, why coffee art? + +**CRAIG BOX: And you are also interested in art with perhaps other materials?** + +SAVITHA RAGHUNATHAN: Yes. My interests keep changing. I used to do pebble art. It's just collecting pebbles from wherever I go, and I used to paint on them. I used to use watercolour, but I want to come back to watercolour sometime. + +My recent interests are coloured pencils, which came back. When I was very young, I used to do a lot of coloured pencils. And then I switched to watercolours and oil painting. So I just go around in circles. + +One of the hobbies that I picked up during a pandemic is crochet. I made a scarf for Mother's Day. My mum and my dad were here last year. They got stuck because of the pandemic, and they couldn't go back home. So they stayed with me for 10 months. That is the jackpot that I had, that I got to spend so much time with my parents after I moved to the US. + +**CRAIG BOX: And they got rewarded with a scarf.** + +SAVITHA RAGHUNATHAN: Yeah. + +**CRAIG BOX: One to share between them.** + +SAVITHA RAGHUNATHAN: I started making a blanket for my dad. And it became so heavy, I might have to just pick up some lighter yarn. I still don't know the differences between different kinds of yarns, but I'm getting better. + +I started out because I wanted to make these little toys. They call them [amigurumi](https://en.wikipedia.org/wiki/Amigurumi) in the crochet world. I wanted to make them. That's why I started out. I'm trying. I made [a little cat](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/2) which doesn't look like a cat, but it is a cat. I have to tell everyone that it's a cat so that they don't mock me later, but. + +**CRAIG BOX: It's an artistic interpretation of a cat.** + +SAVITHA RAGHUNATHAN: It definitely is! + +--- + +_[Savitha Raghunathan](https://twitter.com/coffeeartgirl), now a Senior Software Engineer at Red Hat, served as the Kubernetes 1.22 release team lead._ + +_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._ diff --git a/content/en/blog/_posts/2021-12-07-kubernetes-release-1.23.md b/content/en/blog/_posts/2021-12-07-kubernetes-release-1.23.md new file mode 100644 index 0000000000..f0bce232dd --- /dev/null +++ b/content/en/blog/_posts/2021-12-07-kubernetes-release-1.23.md @@ -0,0 +1,189 @@ +--- +layout: blog +title: 'Kubernetes 1.23: The Next Frontier' +date: 2021-12-07 +slug: kubernetes-1-23-release-announcement +evergreen: true +--- + +**Authors:** [Kubernetes 1.23 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.23/release-team.md) + +We’re pleased to announce the release of Kubernetes 1.23, the last release of 2021! + +This release consists of 47 enhancements: 11 enhancements have graduated to stable, 17 enhancements are moving to beta, and 19 enhancements are entering alpha. Also, 1 feature has been deprecated. + +## Major Themes + +### Deprecation of FlexVolume + +FlexVolume is deprecated. The out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See [this doc](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) for more information. Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI. Users of FlexVolume should move their workloads to the CSI driver. + +### Deprecation of klog specific flags + +To simplify the code base, several [logging flags were marked as deprecated](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog) in Kubernetes 1.23. The code which implements them will be removed in a future release, so users of those need to start replacing the deprecated flags with some alternative solutions. + +### Software Supply Chain SLSA Level 1 Compliance in the Kubernetes Release Process + +Kubernetes releases now generate provenance attestation files describing the staging and release phases of the release process. Artifacts are now verified as they are handed over from one phase to the next. This final piece completes the work needed to comply with Level 1 of the [SLSA security framework](https://slsa.dev/) (Supply-chain Levels for Software Artifacts). + +### IPv4/IPv6 Dual-stack Networking graduates to GA + +[IPv4/IPv6 dual-stack networking](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack) graduates to GA. Since 1.21, Kubernetes clusters have been enabled to support dual-stack networking by default. In 1.23, the `IPv6DualStack` feature gate is removed. The use of dual-stack networking is not mandatory. Although clusters are enabled to support dual-stack networking, Pods and Services continue to default to single-stack. To use dual-stack networking Kubernetes nodes must have routable IPv4/IPv6 network interfaces, a dual-stack capable CNI network plugin must be used, Pods must be configured to be dual-stack and Services must have their `.spec.ipFamilyPolicy` field set to either `PreferDualStack` or `RequireDualStack`. + +### HorizontalPodAutoscaler v2 graduates to GA + +The HorizontalPodAutscaler `autoscaling/v2` stable API moved to GA in 1.23. The HorizontalPodAutoscaler `autoscaling/v2beta2` API has been deprecated. + +### Generic Ephemeral Volume feature graduates to GA + +The generic ephemeral volume feature moved to GA in 1.23. This feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. All StorageClass parameters for volume provisioning and all features supported with PersistentVolumeClaims are supported. + +### Skip Volume Ownership change graduates to GA + +The feature to configure volume permission and ownership change policy for Pods moved to GA in 1.23. This allows users to skip recursive permission changes on mount and speeds up the pod start up time. + +### Allow CSI drivers to opt-in to volume ownership and permission change graduates to GA + +The feature to allow CSI Drivers to declare support for fsGroup based permissions graduates to GA in 1.23. + +### PodSecurity graduates to Beta + +[PodSecurity](https://kubernetes.io/docs/concepts/security/pod-security-admission/) moves to Beta. `PodSecurity` replaces the deprecated `PodSecurityPolicy` admission controller. `PodSecurity` is an admission controller that enforces Pod Security Standards on Pods in a Namespace based on specific namespace labels that set the enforcement level. In 1.23, the `PodSecurity` feature gate is enabled by default. + +### Container Runtime Interface (CRI) v1 is default + +The Kubelet now supports the CRI `v1` API, which is now the project-wide default. +If a container runtime does not support the `v1` API, Kubernetes will fall back to the `v1alpha2` implementation. There is no intermediate action required by end-users, because `v1` and `v1alpha2` do not differ in their implementation. It is likely that `v1alpha2` will be removed in one of the future Kubernetes releases to be able to develop `v1`. + +### Structured logging graduate to Beta + +Structured logging reached its Beta milestone. Most log messages from kubelet and kube-scheduler have been converted. Users are encouraged to try out JSON output or parsing of the structured text format and provide feedback on possible solutions for the open issues, such as handling of multi-line strings in log values. + +### Simplified Multi-point plugin configuration for scheduler + +The kube-scheduler is adding a new, simplified config field for Plugins to allow multiple extension points to be enabled in one spot. The new `multiPoint` plugin field is intended to simplify most scheduler setups for administrators. Plugins that are enabled via `multiPoint` will automatically be registered for each individual extension point that they implement. For example, a plugin that implements Score and Filter extensions can be simultaneously enabled for both. This means entire plugins can be enabled and disabled without having to manually edit individual extension point settings. These extension points can now be abstracted away due to their irrelevance for most users. + +### CSI Migration updates + +CSI Migration enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding CSI driver. +If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. +After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface. +- CSI Migration feature is turned on by default but stays in Beta for GCE PD, AWS EBS, and Azure Disk in 1.23. +- CSI Migration is introduced as an Alpha feature for Ceph RBD and Portworx in 1.23. + +### Expression language validation for CRD is alpha + +Expression language validation for CRD is in alpha starting in 1.23. If the `CustomResourceValidationExpressions` feature gate is enabled, custom resources will be validated by validation rules using the [Common Expression Language (CEL)](https://github.com/google/cel-spec). + +### Server Side Field Validation is Alpha + +If the `ServerSideFieldValidation` feature gate is enabled starting 1.23, users will receive warnings from the server when they send Kubernetes objects in the request that contain unknown or duplicate fields. Previously unknown fields and all but the last duplicate fields would be dropped by the server. + +With the feature gate enabled, we also introduce the `fieldValidation` query parameter so that users can specify the desired behavior of the server on a per request basis. Valid values for the `fieldValidation` query parameter are: + +- Ignore (default when feature gate is disabled, same as pre-1.23 behavior of dropping/ignoring unkonwn fields) +- Warn (default when feature gate is enabled). +- Strict (this will fail the request with an Invalid Request error) + +### OpenAPI v3 is Alpha + +If the `OpenAPIV3` feature gate is enabled starting 1.23, users will be able to request the OpenAPI v3.0 spec for all Kubernetes types. OpenAPI v3 aims to be fully transparent and includes support for a set of fields that are dropped when publishing OpenAPI v2: `default`, `nullable`, `oneOf`, `anyOf`. A separate spec is published per Kubernetes group version (at the `$cluster/openapi/v3/apis/<group>/<version>` endpoint) for improved performance and discovery, for all group versions can be found at the `$cluster/openapi/v3` path. + +## Other Updates + +### Graduated to Stable + +- [IPv4/IPv6 Dual-Stack Support](https://github.com/kubernetes/enhancements/issues/563) +- [Skip Volume Ownership Change](https://github.com/kubernetes/enhancements/issues/695) +- [TTL After Finished Controller](https://github.com/kubernetes/enhancements/issues/592) +- [Config FSGroup Policy in CSI Driver object](https://github.com/kubernetes/enhancements/issues/1682) +- [Generic Ephemeral Inline Volumes](https://github.com/kubernetes/enhancements/issues/1698) +- [Defend Against Logging Secrets via Static Analysis](https://github.com/kubernetes/enhancements/issues/1933) +- [Namespace Scoped Ingress Class Parameters](https://github.com/kubernetes/enhancements/issues/2365) +- [Reducing Kubernetes Build Maintenance](https://github.com/kubernetes/enhancements/issues/2420) +- [Graduate HPA API to GA](https://github.com/kubernetes/enhancements/issues/2702) + + +### Major Changes + +- [Priority and Fairness for API Server Requests](https://github.com/kubernetes/enhancements/issues/1040) + +### Release Notes + +Check out the full details of the Kubernetes 1.23 release in our [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md). + +### Availability + +Kubernetes 1.23 is available for download on [GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.23.0). To get started with Kubernetes, check out these [interactive tutorials](https://kubernetes.io/docs/tutorials/) or run local Kubernetes clusters using Docker container “nodes” with [kind](https://kind.sigs.k8s.io/). You can also easily install 1.23 using [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). + +### Release Team + +This release was made possible by a very dedicated group of individuals, who came together as a team to deliver technical content, documentation, code, and a host of other components that go into every Kubernetes release. + +A huge thank you to the release lead Rey Lejano for leading us through a successful release cycle, and to everyone else on the release team for supporting each other, and working so hard to deliver the 1.23 release for the community. + +### Release Theme and Logo + +**Kubernetes 1.23: The Next Frontier** + +{{< figure src="/images/blog/2021-12-07-kubernetes-release-1.23/kubernetes-1.23.png" alt="" class="release-logo" >}} + + +"The Next Frontier" theme represents the new and graduated enhancements in 1.23, Kubernetes' history of Star Trek references, and the growth of community members in the release team. + +Kubernetes has a history of Star Trek references. The original codename for Kubernetes within Google is Project 7, a reference to Seven of Nine from Star Trek Voyager. And of course Borg was the name for the predecessor to Kubernetes. "The Next Frontier" theme continues the Star Trek references. "The Next Frontier" is a fusion of two Star Trek titles, Star Trek V: The Final Frontier and Star Trek the Next Generation. + +"The Next Frontier" represents a line in the SIG Release charter, "Ensure there is a consistent group of community members in place to support the release process across time." With each release team, we grow the community with new release team members and for many it's their first contribution in their open source frontier. + +Reference: https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/ +Reference: https://github.com/kubernetes/community/blob/master/sig-release/charter.md + +The Kubernetes 1.23 release logo continues with the theme's Star Trek reference. Every star is a helm from the Kubernetes logo. The ship represents the collective teamwork of the release team. + +Rey Lejano designed the logo. + +### User Highlights + +- [Findings of the latest CNCF End User Technology Radar](https://www.cncf.io/announcements/2021/09/22/cncf-end-user-technology-radar-provides-insights-into-devsecops/) were themed around DevSecOps. Check out the [Radar Page](https://radar.cncf.io/) for the full details and findings. +- Learn about how [end user Aegon Life India migrated core processes from its traditional monolith to a microservice-based architecture](https://www.cncf.io/case-studies/aegon-life-india/) in its effort to transform into a leading digital service company. +- Utilizing multiple cloud native projects, [Seagate engineered edgerX to run Real-time Analytics at the Edge](https://www.cncf.io/case-studies/seagate/). +- Check out how [Zambon worked with SparkFabrik to develop 16 websites, with cloud native technologies, to enable stakeholders to easily update content while maintaining a consistent brand identity](https://www.cncf.io/case-studies/zambon/). +- Using Kubernetes, [InfluxData was able to deliver on the promise of multi-cloud, multi-region service availability](https://www.cncf.io/case-studies/influxdata/) by creating a true cloud abstraction layer that allows for the seamless delivery of InfluxDB as a single application to multiple global clusters across three major cloud providers. + + +### Ecosystem Updates + +- [KubeCon + CloudNativeCon NA 2021](https://www.cncf.io/events/kubecon-cloudnativecon-north-america-2021/) was held in October 2021, both online and in person. All talks are [now available on-demand](https://www.youtube.com/playlist?list=PLj6h78yzYM2Nd1U4RMhv7v88fdiFqeYAP) for anyone that would like to catch up! +- [Kubernetes and Cloud Native Essentials Training and KCNA Certification are now generally available for enrollment and scheduling](https://www.cncf.io/announcements/2021/11/18/kubernetes-and-cloud-native-essentials-training-and-kcna-certification-now-available/). Additionally, a new online training course, [Kubernetes and Cloud Native Essentials (LFS250)](https://www.cncf.io/announcements/2021/10/13/entry-level-kubernetes-certification-to-help-advance-cloud-careers/), has been released to both prepare individuals for entry-level cloud roles and to sit for the KCNA exam. +- [New resources are now available from the Inclusive Naming Initiative](https://www.cncf.io/announcements/2021/10/13/inclusive-naming-initiative-announces-new-community-resources-for-a-more-inclusive-future/), including an Inclusive Strategies for Open Source (LFC103) course, Language Evaluation Framework, and Implementation Path. + + +### Project Velocity + +The [CNCF K8s DevStats](https://k8s.devstats.cncf.io/d/12/dashboards?orgId=1&refresh=15m) project aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects. This includes everything from individual contributions to the number of companies that are contributing, and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem. + +In the v1.23 release cycle, which ran for 16 weeks (August 23 to December 7), we saw contributions from [1032 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.22.0%20-%20now&var-metric=contributions) and [1084 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.22.0%20-%20now&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All&var-repo_name=kubernetes%2Fkubernetes). + +### Event Update + +- [KubeCon + CloudNativeCon China 2021](https://www.lfasiallc.com/kubecon-cloudnativecon-open-source-summit-china/) is happening this month from December 9 - 11. After taking a break last year, the event will be virtual this year and includes 105 sessions. Check out the event schedule [here](https://www.lfasiallc.com/kubecon-cloudnativecon-open-source-summit-china/program/schedule/). +- KubeCon + CloudNativeCon Europe 2022 will take place in Valencia, Spain, May 4 – 7, 2022! You can find more information about the conference and registration on the [event site](https://events.linuxfoundation.org/archive/2021/kubecon-cloudnativecon-europe/). +- Kubernetes Community Days has upcoming events scheduled in Pakistan, Brazil, Chengdu, and in Australia. + +### Upcoming Release Webinar + +Join members of the Kubernetes 1.23 release team on January 4, 2022 to learn about the major features of this release, as well as deprecations and removals to help plan for upgrades. For more information and registration, visit the [event page](https://community.cncf.io/e/mrey9h/) on the CNCF Online Programs site. + +### Get Involved + +The simplest way to get involved with Kubernetes is by joining one of the many [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly [community meeting](https://github.com/kubernetes/community/tree/master/communication), and through the channels below: + +- Find out more about contributing to Kubernetes at the [Kubernetes Contributors](https://www.kubernetes.dev/) website +- Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for the latest updates +- Join the community discussion on [Discuss](https://discuss.kubernetes.io/) +- Join the community on [Slack](http://slack.k8s.io/) +- Post questions (or answer questions) on [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes) +- Share your Kubernetes [story](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform) +- Read more about what’s happening with Kubernetes on the [blog](https://kubernetes.io/blog/) +- Learn more about the [Kubernetes Release Team](https://github.com/kubernetes/sig-release/tree/master/release-team) + diff --git a/content/en/blog/_posts/2021-12-08-dual-stack-networking-ga.md b/content/en/blog/_posts/2021-12-08-dual-stack-networking-ga.md new file mode 100644 index 0000000000..4955bf3c0a --- /dev/null +++ b/content/en/blog/_posts/2021-12-08-dual-stack-networking-ga.md @@ -0,0 +1,56 @@ +--- +layout: blog +title: 'Kubernetes 1.23: Dual-stack IPv4/IPv6 Networking Reaches GA' +date: 2021-12-08 +slug: dual-stack-networking-ga +--- + +**Author:** Bridget Kromhout (Microsoft) + +"When will Kubernetes have IPv6?" This question has been asked with increasing frequency ever since alpha support for IPv6 was first added in k8s v1.9. While Kubernetes has supported IPv6-only clusters since v1.18, migration from IPv4 to IPv6 was not yet possible at that point. At long last, [dual-stack IPv4/IPv6 networking](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack/) has reached general availability (GA) in Kubernetes v1.23. + +What does dual-stack networking mean for you? Let’s take a look… + + +## Service API updates + +[Services](/docs/concepts/services-networking/service/) were single-stack before 1.20, so using both IP families meant creating one Service per IP family. The user experience was simplified in 1.20, when Services were re-implemented to allow both IP families, meaning a single Service can handle both IPv4 and IPv6 workloads. Dual-stack load balancing is possible between services running any combination of IPv4 and IPv6. + +The Service API now has new fields to support dual-stack, replacing the single ipFamily field. +* You can select your choice of IP family by setting `ipFamilyPolicy` to one of three options: SingleStack, PreferDualStack, or RequireDualStack. A service can be changed between single-stack and dual-stack (within some limits). +* Setting `ipFamilies` to a list of families assigned allows you to set the order of families used. +* `clusterIPs` is inclusive of the previous `clusterIP` but allows for multiple entries, so it’s no longer necessary to run duplicate services, one in each of the two IP families. Instead, you can assign cluster IP addresses in both IP families. + +Note that Pods are also dual-stack. For a given pod, there is no possibility of setting multiple IP addresses in the same family. + + +## Default behavior remains single-stack + + +Starting in 1.20 with the re-implementation of dual-stack services as alpha, the underlying networking for Kubernetes has included dual-stack whether or not a cluster was configured with the feature flag to enable dual-stack. + +Kubernetes 1.23 removed that feature flag as part of graduating the feature to stable. Dual-stack networking is always available if you want to configure it. You can set your cluster network to operate as single-stack IPv4, as single-stack IPv6, or as dual-stack IPv4/IPv6. + +While Services are set according to what you configure, Pods default to whatever the CNI plugin sets. If your CNI plugin assigns single-stack IPs, you will have single-stack unless `ipFamilyPolicy` specifies PreferDualStack or RequireDualStack. If your CNI plugin assigns dual-stack IPs, `pod.status.PodIPs` defaults to dual-stack. + +Even though dual-stack is possible, it is not mandatory to use it. Examples in the documentation show the variety possible in [dual-stack service configurations](/docs/concepts/services-networking/dual-stack/#dual-stack-service-configuration-scenarios). + + +## Try dual-stack right now + +While upstream Kubernetes now supports [dual-stack networking](/docs/concepts/services-networking/dual-stack/) as a GA or stable feature, each provider’s support of dual-stack Kubernetes may vary. Nodes need to be provisioned with routable IPv4/IPv6 network interfaces. Pods need to be dual-stack. The [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) is what assigns the IP addresses to the Pods, so it's the network plugin being used for the cluster that needs to support dual-stack. Some Container Network Interface (CNI) plugins support dual-stack, as does kubenet. + +Ecosystem support of dual-stack is increasing; you can create [dual-stack clusters with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/), try a [dual-stack cluster locally with KIND](https://kind.sigs.k8s.io/docs/user/configuration/#ip-family), and deploy dual-stack clusters in cloud providers (after checking docs for CNI or kubenet availability). + +## Get involved with SIG Network + +SIG-Network wants to learn from community experiences with dual-stack networking to find out more about evolving needs and your use cases. The [SIG-network update video from KubeCon NA 2021](https://www.youtube.com/watch?v=uZ0WLxpmBbY&list=PLj6h78yzYM2Nd1U4RMhv7v88fdiFqeYAP&index=4) summarizes the SIG’s recent updates, including dual-stack going to stable in 1.23. + +The current SIG-Network [KEPs](https://github.com/orgs/kubernetes/projects/10) and [issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fnetwork) on GitHub illustrate the SIG’s areas of emphasis. The [dual-stack API server](https://github.com/kubernetes/enhancements/issues/2438) is one place to consider contributing. + +[SIG-Network meetings](https://github.com/kubernetes/community/tree/master/sig-network#meetings) are a friendly, welcoming venue for you to connect with the community and share your ideas. Looking forward to hearing from you! + +## Acknowledgments + +The dual-stack networking feature represents the work of many Kubernetes contributors. Thanks to all who contributed code, experience reports, documentation, code reviews, and everything in between. Bridget Kromhout details this community effort in [Dual-Stack Networking in Kubernetes](https://containerjournal.com/features/dual-stack-networking-in-kubernetes/). KubeCon keynotes by Tim Hockin & Khaled (Kal) Henidak in 2019 ([The Long Road to IPv4/IPv6 Dual-stack Kubernetes](https://www.youtube.com/watch?v=o-oMegdZcg4)) and by Lachlan Evenson in 2021 ([And Here We Go: Dual-stack Networking in Kubernetes](https://www.youtube.com/watch?v=lVrt8F2B9CM)) talk about the dual-stack journey, spanning five years and a great many lines of code. + diff --git a/content/en/blog/_posts/2021-12-09-pod-security-admission-beta.md b/content/en/blog/_posts/2021-12-09-pod-security-admission-beta.md new file mode 100644 index 0000000000..f1eec3ef76 --- /dev/null +++ b/content/en/blog/_posts/2021-12-09-pod-security-admission-beta.md @@ -0,0 +1,782 @@ +--- +layout: blog +title: 'Kubernetes 1.23: Pod Security Graduates to Beta' +date: 2021-12-09 +slug: pod-security-admission-beta +--- + +**Authors:** Jim Angel (Google), Lachlan Evenson (Microsoft) + +With the release of Kubernetes v1.23, [Pod Security admission](/docs/concepts/security/pod-security-admission/) has now entered beta. Pod Security is a [built-in](/docs/reference/access-authn-authz/admission-controllers/) admission controller that evaluates pod specifications against a predefined set of [Pod Security Standards](/docs/concepts/security/pod-security-standards/) and determines whether to `admit` or `deny` the pod from running. + +Pod Security is the successor to [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) which was deprecated in the v1.21 release, and will be removed in Kubernetes v1.25. In this article, we cover the key concepts of Pod Security along with how to use it. We hope that cluster administrators and developers alike will use this new mechanism to enforce secure defaults for their workloads. + +## Why Pod Security + +The overall aim of Pod Security is to let you isolate workloads. You can run a cluster that runs different workloads and, without adding extra third-party tooling, implement controls that require Pods for a workload to restrict their own privileges to a defined bounding set. + +Pod Security overcomes key shortcomings of Kubernetes' existing, but deprecated, PodSecurityPolicy (PSP) mechanism: + + * Policy authorization model — challenging to deploy with controllers. + * Risks around switching — a lack of dry-run/audit capabilities made it hard to enable PodSecurityPolicy. + * Inconsistent and Unbounded API — the large configuration surface and evolving constraints led to a complex and confusing API. + +The shortcomings of PSP made it very difficult to use which led the community to reevaluate whether or not a better implementation could achieve the same goals. One of those goals was to provide an out-of-the-box solution to apply security best practices. Pod Security ships with predefined Pod Security levels that a cluster administrator can configure to meet the desired security posture. + +It's important to note that Pod Security doesn't have complete feature parity with the deprecated PodSecurityPolicy. Specifically, it doesn't have the ability to mutate or change Kubernetes resources to auto-remediate a policy violation on behalf of the user. Additionally, it doesn't provide fine-grained control over each allowed field and value within a pod specification or any other Kubernetes resource that you may wish to evaluate. If you need more fine-grained policy control then take a look at these [other](/docs/concepts/security/pod-security-standards/#faq) projects which support such use cases. + +Pod Security also adheres to Kubernetes best practices of declarative object management by denying resources that violate the policy. This requires resources to be updated in source repositories, and tooling to be updated prior to being deployed to Kubernetes. + +## How Does Pod Security Work? + +Pod Security is a built-in [admission controller](/docs/reference/access-authn-authz/admission-controllers/) starting with Kubernetes v1.22, but can also be run as a standalone [webhook](/docs/concepts/security/pod-security-admission/#webhook). Admission controllers function by intercepting requests in the Kubernetes API server prior to persistence to storage. They can either `admit` or `deny` a request. In the case of Pod Security, pod specifications will be evaluated against a configured policy in the form of a Pod Security Standard. This means that security sensitive fields in a pod specification will only be allowed to have [specific](h/docs/concepts/security/pod-security-standards/#profile-details) values. + +## Configuring Pod Security + +### Pod Security Standards + +In order to use Pod Security we first need to understand [Pod Security Standards](/docs/concepts/security/pod-security-standards/). These standards define three different policy levels that range from permissive to restrictive. These levels are as follows: + * `privileged` — open and unrestricted + * `baseline` — Covers known privilege escalations while minimizing restrictions + * `restricted` — Highly restricted, hardening against known and unknown privilege escalations. May cause compatibility issues + +Each of these policy levels define which fields are restricted within a pod specification and the allowed values. Some of the fields restricted by these policies include: + * `spec.securityContext.sysctls` + * `spec.hostNetwork` + * `spec.volumes[*].hostPath` + * `spec.containers[*].securityContext.privileged` + +Policy levels are applied via labels on Namespace resources, which allows for granular per-namespace policy selection. The AdmissionConfiguration in the API server can also be configured to set cluster-wide default levels and exemptions. + +### Policy modes + +Policies are applied in a specific mode. Multiple modes (with different policy levels) can be set on the same namespace. Here is a list of modes: + * `enforce` — Any Pods that violate the policy will be rejected + * `audit` — Violations will be recorded as an annotation in the audit logs, but don't affect whether the pod is allowed. + * `warn` — Violations will send a warning message back to the user, but don't affect whether the pod is allowed. + +In addition to modes you can also pin the policy to a specific version (for example v1.22). Pinning to a specific version allows the behavior to remain consistent if the policy definition changes in future Kubernetes releases. + +## Hands on demo + +### Prerequisites + +- [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) +- [kubectl](/docs/tasks/tools/) +- [Docker](https://docs.docker.com/get-docker/) or [Podman](https://podman.io/getting-started/installation) container runtime & CLI + +### Deploy a kind cluster + +```shell +kind create cluster --image kindest/node:v1.23.0 +``` + +It might take a while to start and once it's started it might take a minute or so before the node becomes ready. + +```shell +kubectl cluster-info --context kind-kind +``` + +Wait for the node STATUS to become ready. + +```shell +kubectl get nodes +``` + +The output is similar to this: + +``` +NAME STATUS ROLES AGE VERSION +kind-control-plane Ready control-plane,master 54m v1.23.0 +``` + +### Confirm Pod Security is enabled + +The best way to [confirm the API's default enabled plugins](/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default) is to check the Kubernetes API container's help arguments. + +```shell +kubectl -n kube-system exec kube-apiserver-kind-control-plane -it -- kube-apiserver -h | grep "default enabled ones" +``` + +The output is similar to this: + +``` +... + --enable-admission-plugins strings +admission plugins that should be enabled in addition +to default enabled ones (NamespaceLifecycle, LimitRanger, +ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, +DefaultTolerationSeconds, DefaultStorageClass, +StorageObjectInUseProtection, PersistentVolumeClaimResize, +RuntimeClass, CertificateApproval, CertificateSigning, +CertificateSubjectRestriction, DefaultIngressClass, +MutatingAdmissionWebhook, ValidatingAdmissionWebhook, +ResourceQuota). +... +``` + +`PodSecurity` is listed in the group of default enabled admission plugins. + +If using a cloud provider, or if you don't have access to the API server, the best way to check would be to run a quick end-to-end test: + +```shell +kubectl create namespace verify-pod-security +kubectl label namespace verify-pod-security pod-security.kubernetes.io/enforce=restricted +# The following command does NOT create a workload (--dry-run=server) +kubectl -n verify-pod-security run test --dry-run=server --image=busybox --privileged +kubectl delete namespace verify-pod-security +``` + +The output is similar to this: + +``` +Error from server (Forbidden): pods "test" is forbidden: violates PodSecurity "restricted:latest": privileged (container "test" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +``` + +### Configure Pod Security + +Policies are applied to a namespace via labels. These labels are as follows: + * `pod-security.kubernetes.io/<MODE>: <LEVEL>` (required to enable pod security) + * `pod-security.kubernetes.io/<MODE>-version: <VERSION>` (*optional*, defaults to latest) + +A specific version can be supplied for each enforcement mode. The version pins the policy to the version that was shipped as part of the Kubernetes release. Pinning to a specific Kubernetes version allows for deterministic policy behavior while allowing flexibility for future updates to Pod Security Standards. The possible <MODE(S)> are `enforce`, `audit` and `warn`. + +### When to use `warn`? + +The typical uses for `warn` are to get ready for a future change where you want to enforce a different policy. The most two common cases would be: + +* `warn` at the same level but a different version (e.g. pin `enforce` to *restricted+v1.23* and `warn` at *restricted+latest*) +* `warn` at a stricter level (e.g. `enforce` baseline, `warn` restricted) + +It's not recommended to use `warn` for the exact same level+version of the policy as `enforce`. In the admission sequence, if `enforce` fails, the entire sequence fails before evaluating the `warn`. + +First, create a namespace called `verify-pod-security` if not created earlier. For the demo, `--overwrite` is used when labeling to allow repurposing a single namespace for multiple examples. + +```shell +kubectl create namespace verify-pod-security +``` + +### Deploy demo workloads + +Each workload represents a higher level of security that would not pass the profile that comes after it. + +For the following examples, use the `busybox` container runs a `sleep` command for 1 million seconds (≅11 days) or until deleted. Pod Security is not interested in which container image you chose, but rather the Pod level settings and their implications for security. + +### Privileged level and workload + +For the privileged pod, use the [privileged policy](/docs/concepts/security/pod-security-standards/#privileged). This allows the process inside a container to gain new processes (also known as "privilege escalation") and can be dangerous if untrusted. + +First, let's apply a restricted Pod Security level for a test. + +```shell +# enforces a "restricted" security policy and audits on restricted +kubectl label --overwrite ns verify-pod-security \ + pod-security.kubernetes.io/enforce=restricted \ + pod-security.kubernetes.io/audit=restricted +``` + +Next, try to deploy a privileged workload in the namespace. + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-privileged +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: true +EOF +``` + +The output is similar to this: + +``` +Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +``` + +Now let's apply the privileged Pod Security level and try again. + +```shell +# enforces a "privileged" security policy and warns / audits on baseline +kubectl label --overwrite ns verify-pod-security \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/warn=baseline \ + pod-security.kubernetes.io/audit=baseline +``` + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-privileged +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: true +EOF +``` + +The output is similar to this: + +``` +pod/busybox-privileged created +``` + +We can run `kubectl -n verify-pod-security get pods` to verify it is running. Clean up with: + +```shell +kubectl -n verify-pod-security delete pod busybox-privileged +``` + +### Baseline level and workload + +The [baseline policy](/docs/concepts/security/pod-security-standards/#baseline) demonstrates sensible defaults while preventing common container exploits. + +Let's revert back to a restricted Pod Security level for a quick test. + +```shell +# enforces a "restricted" security policy and audits on restricted +kubectl label --overwrite ns verify-pod-security \ + pod-security.kubernetes.io/enforce=restricted \ + pod-security.kubernetes.io/audit=restricted +``` + +Apply the workload. + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-baseline +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - NET_BIND_SERVICE + - CHOWN +EOF +``` + +The output is similar to this: + +``` +Error from server (Forbidden): error when creating "STDIN": pods "busybox-baseline" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +``` + +Let's apply the baseline Pod Security level and try again. + +```shell +# enforces a "baseline" security policy and warns / audits on restricted +kubectl label --overwrite ns verify-pod-security \ + pod-security.kubernetes.io/enforce=baseline \ + pod-security.kubernetes.io/warn=restricted \ + pod-security.kubernetes.io/audit=restricted +``` + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-baseline +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - NET_BIND_SERVICE + - CHOWN +EOF +``` + +The output is similar to the following. Note that the warnings match the error message from the test above, but the pod is still successfully created. + +``` +Warning: would violate PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +pod/busybox-baseline created +``` + +Remember, we set the `verify-pod-security` namespace to `warn` based on the restricted profile. We can run `kubectl -n verify-pod-security get pods` to verify it is running. Clean up with: + +```shell +kubectl -n verify-pod-security delete pod busybox-baseline +``` + +### Restricted level and workload + +The [restricted policy](/docs/concepts/security/pod-security-standards/#restricted) requires rejection of all privileged parameters. It is the most secure with a trade-off for complexity. +The restricted policy allows containers to add the `NET_BIND_SERVICE` capability only. + +While we've already tested restricted as a blocking function, let's try to get something running that meets all the criteria. + +First we need to reapply the restricted profile, for the last time. + +```shell +# enforces a "restricted" security policy and audits on restricted +kubectl label --overwrite ns verify-pod-security \ + pod-security.kubernetes.io/enforce=restricted \ + pod-security.kubernetes.io/audit=restricted +``` + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-restricted +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - NET_BIND_SERVICE +EOF +``` + +The output is similar to this: + +``` +Error from server (Forbidden): error when creating "STDIN": pods "busybox-restricted" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +``` + +This is because the restricted profile explicitly requires that certain values are set to the most secure parameters. + +By requiring explicit values, manifests become more declarative and your entire security model can shift left. With the `restricted` level of enforcement, a company could audit their cluster's compliance based on permitted manifests. + +Let's fix each warning resulting in the following file: + +```shell +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-restricted +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + add: + - NET_BIND_SERVICE +EOF +``` + +The output is similar to this: + +``` +pod/busybox-restricted created +``` + +Run `kubectl -n verify-pod-security get pods` to verify it is running. The output is similar to this: + +``` +NAME READY STATUS RESTARTS AGE +busybox-restricted 0/1 CreateContainerConfigError 0 2m26s +``` + +Let's figure out why the container is not starting with `kubectl -n verify-pod-security describe pod busybox-restricted`. The output is similar to this: + +``` +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning Failed 2m29s (x8 over 3m55s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "busybox-restricted_verify-pod-security(a4c6a62d-2166-41a9-b288-20df17cf5c90)", container: busybox) +``` + +To solve this, set the effective UID (`runAsUser`) to a non-zero (root) value or use the `nobody` UID (65534). + +```shell +# delete the original pod +kubectl -n verify-pod-security delete pod busybox-restricted + +# create the pod again with new runAsUser +cat <<EOF | kubectl -n verify-pod-security apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-restricted +spec: + securityContext: + runAsUser: 65534 + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + add: + - NET_BIND_SERVICE +EOF +``` + +Run `kubectl -n verify-pod-security get pods` to verify it is running. The output is similar to this: + +``` +NAME READY STATUS RESTARTS AGE +busybox-restricted 1/1 Running 0 25s +``` + +Clean up the demo (restricted pod and namespace) with: + +```shell +kubectl delete namespace verify-pod-security +``` + +At this point, if you wanted to dive deeper into linux permissions or what is permitted for a certain container, exec into the control plane and play around with `containerd` and `crictl inspect`. + +```shell +# if using docker, shell into the control plane +docker exec -it kind-control-plane bash + +# list running containers +crictl ps + +# inspect each one by container ID +crictl inspect <CONTAINER ID> +``` + +### Applying a cluster-wide policy + +In addition to applying labels to namespaces to configure policy you can also configure cluster-wide policies and exemptions using the AdmissionConfiguration resource. + +Using this resource, policy definitions are applied cluster-wide by default and any policy that is applied via namespace labels will take precedence. + +There is no runtime configurable API for the `AdmissionConfiguration` configuration file so a cluster administrator would need to specify a path to the file below via the `--admission-control-config-file` flag on the API server. + +In the following resource we are enforcing the baseline policy and warning and auditing the baseline policy. We are also making the kube-system namespace exempt from this policy. + +It's not recommended to alter control plane / clusters after install, so let's build a new cluster with a default policy on all namespaces. + +First, delete the current cluster. + +```shell +kind delete cluster +``` + +Create a Pod Security configuration that `enforce` and `audit` baseline policies while using a restricted profile to `warn` the end user. + +```shell +cat <<EOF > pod-security.yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: AdmissionConfiguration +plugins: +- name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1beta1 + kind: PodSecurityConfiguration + defaults: + enforce: "baseline" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + # Array of authenticated usernames to exempt. + usernames: [] + # Array of runtime class names to exempt. + runtimeClasses: [] + # Array of namespaces to exempt. + namespaces: [kube-system] +EOF +``` + +For additional options, check out the official [_standards admission controller_](/docs/tasks/configure-pod-container/enforce-standards-admission-controller/#configure-the-admission-controller) docs. + +We now have a default baseline policy. Next pass it to the kind configuration to enable the `--admission-control-config-file` API server argument and pass the policy file. To pass a file to a kind cluster, use a configuration file to pass additional setup instructions. Kind uses `kubeadm` to provision the cluster and the configuration file has the ability to pass `kubeadmConfigPatches` for further customization. In our case, the local file is mounted into the control plane node as `/etc/kubernetes/policies/pod-security.yaml` which is then mounted into the `apiServer` container. We also pass the `--admission-control-config-file` argument pointing to the policy's location. + +```shell +cat <<EOF > kind-config.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: +- role: control-plane + kubeadmConfigPatches: + - | + kind: ClusterConfiguration + apiServer: + # enable admission-control-config flag on the API server + extraArgs: + admission-control-config-file: /etc/kubernetes/policies/pod-security.yaml + # mount new file / directories on the control plane + extraVolumes: + - name: policies + hostPath: /etc/kubernetes/policies + mountPath: /etc/kubernetes/policies + readOnly: true + pathType: "DirectoryOrCreate" + # mount the local file on the control plane + extraMounts: + - hostPath: ./pod-security.yaml + containerPath: /etc/kubernetes/policies/pod-security.yaml + readOnly: true +EOF +``` + +Create a new cluster using the kind configuration file defined above. + +```shell +kind create cluster --image kindest/node:v1.23.0 --config kind-config.yaml +``` + +Let's look at the default namespace. + +```shell +kubectl describe namespace default +``` + +The output is similar to this: + +``` +Name: default +Labels: kubernetes.io/metadata.name=default +Annotations: <none> +Status: Active + +No resource quota. + +No LimitRange resource. +``` + +Let's create a new namespace and see if the labels apply there. + +```shell +kubectl create namespace test-defaults +kubectl describe namespace test-defaults +``` + +Same. + +``` +Name: test-defaults +Labels: kubernetes.io/metadata.name=test-defaults +Annotations: <none> +Status: Active + +No resource quota. + +No LimitRange resource. +``` + +Can a privileged workload be deployed? + +```shell +cat <<EOF | kubectl -n test-defaults apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-privileged +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + securityContext: + allowPrivilegeEscalation: true +EOF +``` + +Hmm... yep. The default `warn` level is working at least. + +``` +Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") +pod/busybox-privileged created +``` + +Let's delete the pod with `kubectl -n test-defaults delete pod/busybox-privileged`. + +Is my config even working? + +```shell +# if using docker, shell into the control plane +docker exec -it kind-control-plane bash + +# cat out the file we mounted +cat /etc/kubernetes/policies/pod-security.yaml + +# check the api server logs +cat /var/log/containers/kube-apiserver*.log + +# check the api server config +cat /etc/kubernetes/manifests/kube-apiserver.yaml +``` + +**UPDATE:** The baseline policy permits `allowPrivilegeEscalation`. While I cannot see the Pod Security default levels of enforcement, they are there. Let's try to provide a manifest that violates the baseline by requesting hostNetwork access. + +```shell +# delete the original pod +kubectl -n test-defaults delete pod busybox-privileged + +cat <<EOF | kubectl -n test-defaults apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: busybox-privileged +spec: + containers: + - name: busybox + image: busybox + args: + - sleep + - "1000000" + hostNetwork: true +EOF +``` + +The output is similar to this: + +``` +Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true) +``` + +#### Yes!!! It worked! 🎉🎉🎉 {#it-worked} + +I later found out, another way to check if things are operating as intended is to check the raw API server metrics endpoint. + +Run the following command: + +``` +kubectl get --raw /metrics | grep pod_security_evaluations_total +``` + +The output is similar to this: + +``` +# HELP pod_security_evaluations_total [ALPHA] Number of policy evaluations that occurred, not counting ignored or exempt requests. +# TYPE pod_security_evaluations_total counter +pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 2 +pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="privileged",policy_version="latest",request_operation="create",resource="pod",subresource=""} 0 +pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="privileged",policy_version="latest",request_operation="update",resource="pod",subresource=""} 0 +pod_security_evaluations_total{decision="deny",mode="audit",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 1 +pod_security_evaluations_total{decision="deny",mode="enforce",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 1 +pod_security_evaluations_total{decision="deny",mode="warn",policy_level="restricted",policy_version="latest",request_operation="create",resource="controller",subresource=""} 2 +pod_security_evaluations_total{decision="deny",mode="warn",policy_level="restricted",policy_version="latest",request_operation="create",resource="pod",subresource=""} 2 +``` + +A monitoring tool could ingest these metrics too for reporting, assessments, or measuring trends. + +## Clean up + +When finished, delete the kind cluster. + +```shell +kind delete cluster +``` + +## Auditing + +Auditing is another way to track what policies are being enforced in your cluster. To set up auditing with kind, review the official docs for [enabling auditing](https://kind.sigs.k8s.io/docs/user/auditing/). As of [version 1.11](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.11.md#sig-auth), Kubernetes audit logs include two annotations that indicate whether or not a request was authorized (`authorization.k8s.io/decision`) and the reason for the decision (`authorization.k8s.io/reason`). Audit events can be streamed to a webhook for monitoring, tracking, or alerting. + +The audit events look similar to the following: + +```yaml +{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","pod-security.kubernetes.io/audit":"allowPrivilegeEscalation != false (container \"busybox\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"busybox\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"busybox\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"busybox\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}} +``` + +Auditing is also a good first step in evaluating your cluster's current compliance with Pod Security. The Kubernetes Enhancement Proposal (KEP) hints at a future where `baseline` [could be the default for unlabeled namespaces](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md#rollout-of-baseline-by-default-for-unlabeled-namespaces). + + +Example `audit-policy.yaml` configuration tuned for Pod Security events: + +``` +apiVersion: audit.k8s.io/v1 +kind: Policy +rules: +- level: RequestResponse + resources: + - group: "" # core API group + resources: ["pods", "pods/ephemeralcontainers", "podtemplates", "replicationcontrollers"] + - group: "apps" + resources: ["daemonsets", "deployments", "replicasets", "statefulsets"] + - group: "batch" + resources: ["cronjobs", "jobs"] + verbs: ["create", "update"] + omitStages: + - "RequestReceived" + - "ResponseStarted" + - "Panic" +``` + +Once auditing is enabled, look at the configured local file if using `--audit-log-path` or the destination of a webhook if using `--audit-webhook-config-file`. + +If using a file (`--audit-log-path`), run `cat /PATH/TO/API/AUDIT.log | grep "is forbidden:"` to see all rejected workloads audited. + +## PSP migrations + +If you're already using PSP, SIG Auth has created a guide and [published the steps to migrate off of PSP](/docs/tasks/configure-pod-container/migrate-from-psp/). + +To summarize the process: +- Update all existing PSPs to be non-mutating +- Apply Pod Security policies in `warn` or `audit` mode +- Upgrade Pod Security policies to `enforce` mode +- Remove `PodSecurityPolicy` from `--enable-admission-plugins` + +Listed as "optional future extensions" and currently out of scope, SIG Auth has kicked around the idea of providing a tool to assist with migrations. More [details in the KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md#automated-psp-migration-tooling). + +## Wrap up + +Pod Security is a promising new feature that provides an out-of-the-box way to allow users to improve the security posture of their workloads. Like any new enhancement that has matured to beta, we ask that you try it out, provide feedback, or share your experience via either raising a Github issue or joining SIG Auth community meetings. It's our hope that Pod Security will be deployed on every cluster in our ongoing pursuit as a community to make Kubernetes security a priority. + +For a step by step guide on how to enable "baseline" Pod Security Standards with Pod Security Admission feature please refer to these dedicated [tutorials](/docs/tutorials/security/) that cover the configuration needed at cluster level and namespace level. + +## Additional resources + +- [Official Pod Security Docs](/docs/concepts/security/pod-security-admission/) +- [Enforce Pod Security Standards with Namespace Labels](/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/) +- [Enforce Pod Security Standards by Configuring the Built-in Admission Controller](/docs/tasks/configure-pod-container/enforce-standards-admission-controller/) +- [Official Kubernetes Enhancement Proposal](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md) (KEP) +- [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) +- [Hands on with Kubernetes Pod Security](https://medium.com/@LachlanEvenson/hands-on-with-kubernetes-pod-security-admission-b6cac495cd11) \ No newline at end of file diff --git a/content/en/blog/_posts/2021-12-10-csi-migration-status.md b/content/en/blog/_posts/2021-12-10-csi-migration-status.md new file mode 100644 index 0000000000..0fa192c2cb --- /dev/null +++ b/content/en/blog/_posts/2021-12-10-csi-migration-status.md @@ -0,0 +1,135 @@ +--- +layout: blog +title: "Kubernetes 1.23: Kubernetes In-Tree to CSI Volume Migration Status Update" +date: 2021-12-10 +slug: storage-in-tree-to-csi-migration-status-update +--- + +**Author:** Jiawei Wang (Google) + +The Kubernetes in-tree storage plugin to [Container Storage Interface (CSI)](/blog/2019/01/15/container-storage-interface-ga/) migration infrastructure has already been [beta](/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/) since v1.17. CSI migration was introduced as alpha in Kubernetes v1.14. + +Since then, SIG Storage and other Kubernetes special interest groups are working to ensure feature stability and compatibility in preparation for GA. +This article is intended to give a status update to the feature as well as changes between Kubernetes 1.17 and 1.23. In addition, I will also cover the future roadmap for the CSI migration feature GA for each storage plugin. + +## Quick recap: What is CSI Migration, and why migrate? + +The Container Storage Interface (CSI) was designed to help Kubernetes replace its existing, in-tree storage driver mechanisms - especially vendor specific plugins. +Kubernetes support for the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md#README) has been +[generally available](/blog/2019/01/15/container-storage-interface-ga/) since Kubernetes v1.13. +Support for using CSI drivers was introduced to make it easier to add and maintain new integrations between Kubernetes and storage backend technologies. Using CSI drivers allows for for better maintainability (driver authors can define their own release cycle and support lifecycle) and reduce the opportunity for vulnerabilities (with less in-tree code, the risks of a mistake are reduced, and cluster operators can select only the storage drivers that their cluster requires). + +As more CSI Drivers were created and became production ready, SIG Storage group wanted all Kubernetes users to benefit from the CSI model. However, we cannot break API compatibility with the existing storage API types. The solution we came up with was CSI migration: a feature that translates in-tree APIs to equivalent CSI APIs and delegates operations to a replacement CSI driver. + +The CSI migration effort enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding [CSI driver](https://kubernetes-csi.github.io/docs/introduction.html) from the storage backend. +If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. Existing `StorageClass`, `PersistentVolume` and `PersistentVolumeClaim` objects should continue to work. +When a Kubernetes cluster administrator updates a cluster to enable CSI migration, existing workloads that utilize PVCs which are backed by in-tree storage plugins will continue to function as they always have. +However, behind the scenes, Kubernetes hands control of all storage management operations (previously targeting in-tree drivers) to CSI drivers. + +For example, suppose you are a `kubernetes.io/gce-pd` user, after CSI migration, you can still use `kubernetes.io/gce-pd` to provision new volumes, mount existing GCE-PD volumes or delete existing volumes. All existing API/Interface will still function correctly. However, the underlying function calls are all going through the [GCE PD CSI driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) instead of the in-tree Kubernetes function. + +This enables a smooth transition for end users. Additionally as storage plugin developers, we can reduce the burden of maintaining the in-tree storage plugins and eventually remove them from the core Kubernetes binary. + +## What has been changed, and what's new? + +Building on the work done in Kubernetes v1.17 and earlier, the releases since then have +made a series of changes: + +### New feature gates + +The Kubernetes v1.21 release deprecated the `CSIMigration{provider}Complete` feature flags, and stopped honoring them. In their place came new feature flags named `InTreePlugin{vendor}Unregister`, that replace the old feature flag and retain all the functionality that `CSIMigration{provider}Complete` provided. + +`CSIMigration{provider}Complete` was introduced before as a supplementary feature gate once CSI migration is enabled on all of the nodes. This flag unregisters the in-tree storage plugin you specify with the `{provider}` part of the flag name. + +When you enable that feature gate, then instead of using the in-tree driver code, your cluster directly selects and uses the relevant CSI driver. This happens without any check for whether CSI migration is enabled on the node, or whether you have in fact deployed that CSI driver. + +While this feature gate is a great helper, SIG Storage (and, I'm sure, lots of cluster operators) also wanted a feature gate that lets you disable an in-tree storage plugin, even without also enabling CSI migration. For example, you might want to disable the EBS storage plugin on a GCE cluster, because EBS volumes are specific to a different vendor's cloud (AWS). + +To make this possible, Kubernetes v1.21 introduced a new feature flag set: `InTreePlugin{vendor}Unregister`. + +`InTreePlugin{vendor}Unregister` is a standalone feature gate that can be enabled and disabled independently from CSI Migration. When enabled, the component will not register the specific in-tree storage plugin to the supported list. If the cluster operator only enables this flag, end users will get an error from PVC saying it cannot find the plugin when the plugin is used. The cluster operator may want to enable this regardless of CSI Migration if they do not want to support the legacy in-tree APIs and only support CSI moving forward. + +### Observability + +Kubernetes v1.21 introduced [metrics](https://github.com/kubernetes/kubernetes/issues/98279) for tracking CSI migration. +You can use these metrics to observe how your cluster is using storage services and whether access to that storage is using the legacy in-tree driver or its CSI-based replacement. + +| Components | Metrics | Notes | +| -------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Kube-Controller-Manager | storage_operation_duration_seconds | A new label `migrated` is added to the metric to indicate whether this storage operation is a CSI migration operation(string value `true` for enabled and `false` for not enabled). | +| Kubelet | csi_operations_seconds | The new metric exposes labels including `driver_name`, `method_name`, `grpc_status_code` and `migrated`. The meaning of these labels is identical to `csi_sidecar_operations_seconds`. | +| CSI Sidecars(provisioner, attacher, resizer) | csi_sidecar_operations_seconds | A new label `migrated` is added to the metric to indicate whether this storage operation is a CSI migration operation(string value `true` for enabled and `false` for not enabled). | + +### Bug fixes and feature improvement + +We have fixed numerous bugs like dangling attachment, garbage collection, incorrect topology label through the help of our beta testers. + +### Cloud Provider && Cluster Lifecycle Collaboration + +SIG Storage has been working closely with SIG Cloud Provider and SIG Cluster Lifecycle on the rollout of CSI migration. + +If you are a user of a managed Kubernetes service, check with your provider if anything needs to be done. In many cases, the provider will manage the migration and no additional work is required. + +If you use a distribution of Kubernetes, check its official documentation for information about support for this feature. For the CSI Migration feature graduation to GA, SIG Storage and SIG Cluster Lifecycle are collaborating towards making the migration mechanisms available in tooling (such as kubeadm) as soon as they're available in Kubernetes itself. + +## What is the timeline / status? {#timeline-and-status} + +The current and targeted releases for each individual driver is shown in the table below: + +| Driver | Alpha | Beta (in-tree deprecated) | Beta (on-by-default) | GA | Target "in-tree plugin" removal | +| ---------------- | ----- | ------------------------- | -------------------- | ------------- | ------------------------------- | +| AWS EBS | 1.14 | 1.17 | 1.23 | 1.24 (Target) | 1.26 (Target) | +| GCE PD | 1.14 | 1.17 | 1.23 | 1.24 (Target) | 1.26 (Target) | +| OpenStack Cinder | 1.14 | 1.18 | 1.21 | 1.24 (Target) | 1.26 (Target) | +| Azure Disk | 1.15 | 1.19 | 1.23 | 1.24 (Target) | 1.26 (Target) | +| Azure File | 1.15 | 1.21 | 1.24 (Target) | 1.25 (Target) | 1.27 (Target) | +| vSphere | 1.18 | 1.19 | 1.24 (Target) | 1.25 (Target) | 1.27 (Target) | +| Ceph RBD | 1.23 | +| Portworx | 1.23 | + +The following storage drivers will not have CSI migration support. The ScaleIO driver was already removed; the others are deprecated and will be removed from core Kubernetes. + +| Driver | Deprecated | Code Removal | +| --------- | ---------- | ------------- | +| ScaleIO | 1.16 | 1.22 | +| Flocker | 1.22 | 1.25 (Target) | +| Quobyte | 1.22 | 1.25 (Target) | +| StorageOS | 1.22 | 1.25 (Target) | + +## What's next? + +With more CSI drivers graduating to GA, we hope to soon mark the overall CSI Migration feature as GA. We are expecting cloud provider in-tree storage plugins code removal to happen by Kubernetes v1.26 and v1.27. + +## What should I do as a user? + +Note that all new features for the Kubernetes storage system (such as volume snapshotting) will only be added to the CSI interface. Therefore, if you are starting up a new cluster, creating stateful applications for the first time, or require these new features we recommend using CSI drivers natively (instead of the in-tree volume plugin API). Follow the [updated user guides for CSI drivers](https://kubernetes-csi.github.io/docs/drivers.html) and use the new CSI APIs. + +However, if you choose to roll a cluster forward or continue using specifications with the legacy volume APIs, CSI Migration will ensure we continue to support those deployments with the new CSI drivers. However, if you want to leverage new features like snapshot, it will require a manual migration to re-import an existing intree PV as a CSI PV. + +## How do I get involved? + +The Kubernetes Slack channel [#csi-migration](https://kubernetes.slack.com/messages/csi-migration) along with any of the standard [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams. + +This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. We offer a huge thank you to the contributors who stepped up these last quarters to help move the project forward: + +* Michelle Au (msau42) +* Jan Šafránek (jsafrane) +* Hemant Kumar (gnufied) + +Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution to the CSI migration feature: + +* Andy Zhang (andyzhangz) +* Divyen Patel (divyenpatel) +* Deep Debroy (ddebroy) +* Humble Devassy Chirammal (humblec) +* Jing Xu (jingxu97) +* Jordan Liggitt (liggitt) +* Matthew Cary (mattcary) +* Matthew Wong (wongma7) +* Neha Arora (nearora-msft) +* Oksana Naumov (trierra) +* Saad Ali (saad-ali) +* Tim Bannister (sftim) +* Xing Yang (xing-yang) + +Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors. diff --git a/content/en/blog/_posts/2021-12-15-prevent-persistentvolume-leaks-when-deleting-out-of-order.md b/content/en/blog/_posts/2021-12-15-prevent-persistentvolume-leaks-when-deleting-out-of-order.md new file mode 100644 index 0000000000..c57296026f --- /dev/null +++ b/content/en/blog/_posts/2021-12-15-prevent-persistentvolume-leaks-when-deleting-out-of-order.md @@ -0,0 +1,199 @@ +--- +layout: blog +title: "Kubernetes 1.23: Prevent PersistentVolume leaks when deleting out of order" +date: 2021-12-15T10:00:00-08:00 +slug: kubernetes-1-23-prevent-persistentvolume-leaks-when-deleting-out-of-order +--- + +**Author:** Deepak Kinni (VMware) + +[PersistentVolume](/docs/concepts/storage/persistent-volumes/) (or PVs for short) are +associated with [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy). +The Reclaim Policy is used to determine the actions that need to be taken by the storage +backend on deletion of the PV. +Where the reclaim policy is `Delete`, the expectation is that the storage backend +releases the storage resource that was allocated for the PV. In essence, the reclaim +policy needs to honored on PV deletion. + +With the recent Kubernetes v1.23 release, an alpha feature lets you configure your +cluster to behave that way and honor the configured reclaim policy. + + +## How did reclaim work in previous Kubernetes releases? + +[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#Introduction) (or PVC for short) is +a request for storage by a user. A PV and PVC are considered [Bound](/docs/concepts/storage/persistent-volumes/#Binding) +if there is a newly created PV or a matching PV is found. The PVs themselves are +backed by a volume allocated by the storage backend. + +Normally, if the volume is to be deleted, then the expectation is to delete the +PVC for a bound PV-PVC pair. However, there are no restrictions to delete a PV +prior to deleting a PVC. + +First, I'll demonstrate the behavior for clusters that are running an older version of Kubernetes. + +#### Retrieve an PVC that is bound to a PV + +Retrieve an existing PVC `example-vanilla-block-pvc` +``` +kubectl get pvc example-vanilla-block-pvc +``` +The following output shows the PVC and it's `Bound` PV, the PV is shown under the `VOLUME` column: +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +example-vanilla-block-pvc Bound pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO example-vanilla-block-sc 19s +``` + +#### Delete PV + +When I try to delete a bound PV, the cluster blocks and the `kubectl` tool does +not return back control to the shell; for example: + +``` +kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da +``` + +``` +persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted +^C +``` + +Retrieving the PV: +``` +kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da +``` + +It can be observed that the PV is in `Terminating` state +``` +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO Delete Terminating default/example-vanilla-block-pvc example-vanilla-block-sc 2m23s +``` + +#### Delete PVC + +``` +kubectl delete pvc example-vanilla-block-pvc +``` + +The following output is seen if the PVC gets successfully deleted: +``` +persistentvolumeclaim "example-vanilla-block-pvc" deleted +``` + +The PV object from the cluster also gets deleted. When attempting to retrieve the PV +it will be observed that the PV is no longer found: + +``` +kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da +``` + +``` +Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found +``` + +Although the PV is deleted the underlying storage resource is not deleted, and +needs to be removed manually. + +To sum it up, the reclaim policy associated with the Persistent Volume is currently +ignored under certain circumstance. For a `Bound` PV-PVC pair the ordering of PV-PVC +deletion determines whether the PV reclaim policy is honored. The reclaim policy +is honored if the PVC is deleted first, however, if the PV is deleted prior to +deleting the PVC then the reclaim policy is not exercised. As a result of this behavior, +the associated storage asset in the external infrastructure is not removed. + +## PV reclaim policy with Kubernetes v1.23 + +The new behavior ensures that the underlying storage object is deleted from the backend when users attempt to delete a PV manually. + +#### How to enable new behavior? + +To make use of the new behavior, you must have upgraded your cluster to the v1.23 release of Kubernetes. +You need to make sure that you are running the CSI [`external-provisioner`](https://github.com/kubernetes-csi/external-provisioner) version `4.0.0`, or later. +You must also enable the `HonorPVReclaimPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the +`external-provisioner` and for the `kube-controller-manager`. + +If you're not using a CSI driver to integrate with your storage backend, the fix isn't +available. The Kubernetes project doesn't have a current plan to fix the bug for in-tree +storage drivers: the future of those in-tree drivers is deprecation and migration to CSI. + +#### How does it work? + +The new behavior is achieved by adding a finalizer `external-provisioner.volume.kubernetes.io/finalizer` on new and existing PVs, the finalizer is only removed after the storage from backend is deleted. + +An example of a PV with the finalizer, notice the new finalizer in the finalizers list + +``` +kubectl get pv pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 -o yaml +``` + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + annotations: + pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com + creationTimestamp: "2021-11-17T19:28:56Z" + finalizers: + - kubernetes.io/pv-protection + - external-provisioner.volume.kubernetes.io/finalizer + name: pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 + resourceVersion: "194711" + uid: 087f14f2-4157-4e95-8a70-8294b039d30e +spec: + accessModes: + - ReadWriteOnce + capacity: + storage: 1Gi + claimRef: + apiVersion: v1 + kind: PersistentVolumeClaim + name: example-vanilla-block-pvc + namespace: default + resourceVersion: "194677" + uid: a7b7e3ba-f837-45ba-b243-dec7d8aaed53 + csi: + driver: csi.vsphere.vmware.com + fsType: ext4 + volumeAttributes: + storage.kubernetes.io/csiProvisionerIdentity: 1637110610497-8081-csi.vsphere.vmware.com + type: vSphere CNS Block Volume + volumeHandle: 2dacf297-803f-4ccc-afc7-3d3c3f02051e + persistentVolumeReclaimPolicy: Delete + storageClassName: example-vanilla-block-sc + volumeMode: Filesystem +status: + phase: Bound +``` + +The presence of the finalizer prevents the PV object from being removed from the +cluster. As stated previously, the finalizer is only removed from the PV object +after it is successfully deleted from the storage backend. To learn more about +finalizers, please refer to [Using Finalizers to Control Deletion](/blog/2021/05/14/using-finalizers-to-control-deletion/). + +#### What about CSI migrated volumes? + +The fix is applicable to CSI migrated volumes as well. However, when the feature +`HonorPVReclaimPolicy` is enabled on 1.23, and CSI Migration is disabled, the finalizer +is removed from the PV object if it exists. + +### Some caveats + +1. The fix is applicable only to CSI volumes and migrated volumes. In-tree volumes will exhibit older behavior. +2. The fix is introduced as an alpha feature in the [external-provisioner](https://github.com/kubernetes-csi/external-provisioner) under the feature gate `HonorPVReclaimPolicy`. The feature is disabled by default, and needs to be enabled explicitly. + +### References + +* [KEP-2644](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2644-honor-pv-reclaim-policy) +* [Volume leak issue](https://github.com/kubernetes-csi/external-provisioner/issues/546) + +### How do I get involved? + +The Kubernetes Slack channel [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams. + +Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution: + +* Jan Šafránek (jsafrane) +* Xing Yang (xing-yang) +* Matthew Wong (wongma7) + +Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors. \ No newline at end of file diff --git a/content/en/blog/_posts/2021-12-16-StatefulSet-PVC-Auto-Deletion.md b/content/en/blog/_posts/2021-12-16-StatefulSet-PVC-Auto-Deletion.md new file mode 100644 index 0000000000..a436694640 --- /dev/null +++ b/content/en/blog/_posts/2021-12-16-StatefulSet-PVC-Auto-Deletion.md @@ -0,0 +1,103 @@ +--- +layout: blog +title: 'Kubernetes 1.23: StatefulSet PVC Auto-Deletion (alpha)' +date: 2021-12-16 +slug: kubernetes-1-23-statefulset-pvc-auto-deletion +--- + +**Author:** Matthew Cary (Google) + +Kubernetes v1.23 introduced a new, alpha-level policy for +[StatefulSets](/docs/concepts/workloads/controllers/statefulset/) that controls the lifetime of +[PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/) (PVCs) generated from the +StatefulSet spec template for cases when they should be deleted automatically when the StatefulSet +is deleted or pods in the StatefulSet are scaled down. + +## What problem does this solve? +A StatefulSet spec can include Pod and PVC templates. When a replica is first created, the +Kubernetes control plane creates a PVC for that replica if one does not already exist. The behavior +before Kubernetes v1.23 was that the control plane never cleaned up the PVCs created for +StatefulSets - this was left up to the cluster administrator, or to some add-on automation that +you’d have to find, check suitability, and deploy. The common pattern for managing PVCs, either +manually or through tools such as Helm, is that the PVCs are tracked by the tool that manages them, +with explicit lifecycle. Workflows that use StatefulSets must determine on their own what PVCs are +created by a StatefulSet and what their lifecycle should be. + +Before this new feature, when a StatefulSet-managed replica disappears, either because the +StatefulSet is reducing its replica count, or because its StatefulSet is deleted, the PVC and its +backing volume remains and must be manually deleted. While this behavior is appropriate when the +data is critical, in many cases the persistent data in these PVCs is either temporary, or can be +reconstructed from another source. In those cases, PVCs and their backing volumes remaining after +their StatefulSet or replicas have been deleted are not necessary, incur cost, and require manual +cleanup. + +## The new StatefulSet PVC retention policy + +If you enable the alpha feature, a StatefulSet spec includes a PersistentVolumeClaim retention +policy. This is used to control if and when PVCs created from a StatefulSet’s `volumeClaimTemplate` +are deleted. This first iteration of the retention policy contains two situations where PVCs may be +deleted. + +The first situation is when the StatefulSet resource is deleted (which implies that all replicas are +also deleted). This is controlled by the `whenDeleted` policy. The second situation, controlled by +`whenScaled` is when the StatefulSet is scaled down, which removes some but not all of the replicas +in a StatefulSet. In both cases the policy can either be `Retain`, where the corresponding PVCs are +not touched, or `Delete`, which means that PVCs are deleted. The deletion is done with a normal +[object deletion](/docs/concepts/architecture/garbage-collection/), so that, for example, all +retention policies for the underlying PV are respected. + +This policy forms a matrix with four cases. I’ll walk through and give an example for each one. + + * **`whenDeleted` and `whenScaled` are both `Retain`.** This matches the existing behavior for + StatefulSets, where no PVCs are deleted. This is also the default retention policy. It’s + appropriate to use when data on StatefulSet volumes may be irreplaceable and should only be + deleted manually. + + * **`whenDeleted` is `Delete` and `whenScaled` is `Retain`.** In this case, PVCs are deleted only when + the entire StatefulSet is deleted. If the StatefulSet is scaled down, PVCs are not touched, + meaning they are available to be reattached if a scale-up occurs with any data from the previous + replica. This might be used for a temporary StatefulSet, such as in a CI instance or ETL + pipeline, where the data on the StatefulSet is needed only during the lifetime of the + StatefulSet lifetime, but while the task is running the data is not easily reconstructible. Any + retained state is needed for any replicas that scale down and then up. + + * **`whenDeleted` and `whenScaled` are both `Delete`.** PVCs are deleted immediately when their + replica is no longer needed. Note this does not include when a Pod is deleted and a new version + rescheduled, for example when a node is drained and Pods need to migrate elsewhere. The PVC is + deleted only when the replica is no longer needed as signified by a scale-down or StatefulSet + deletion. This use case is for when data does not need to live beyond the life of its + replica. Perhaps the data is easily reconstructable and the cost savings of deleting unused PVCs + is more important than quick scale-up, or perhaps that when a new replica is created, any data + from a previous replica is not usable and must be reconstructed anyway. + + * **`whenDeleted` is `Retain` and `whenScaled` is `Delete`.** This is similar to the previous case, + when there is little benefit to keeping PVCs for fast reuse during scale-up. An example of a + situation where you might use this is an Elasticsearch cluster. Typically you would scale that + workload up and down to match demand, whilst ensuring a minimum number of replicas (for example: + 3). When scaling down, data is migrated away from removed replicas and there is no benefit to + retaining those PVCs. However, it can be useful to bring the entire Elasticsearch cluster down + temporarily for maintenance. If you need to take the Elasticsearch system offline, you can do + this by temporarily deleting the StatefulSet, and then bringing the Elasticsearch cluster back + by recreating the StatefulSet. The PVCs holding the Elasticsearch data will still exist and the + new replicas will automatically use them. + +Visit the +[documentation](/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-policies) to +see all the details. + +## What’s next? + +Enable the feature and try it out! Enable the `StatefulSetAutoDeletePVC` feature gate on a cluster, +then create a StatefulSet using the new policy. Test it out and tell us what you think! + +I'm very curious to see if this owner reference mechanism works well in practice. For example, we +realized there is no mechanism in Kubernetes for knowing who set a reference, so it’s possible that +the StatefulSet controller may fight with custom controllers that set their own +references. Fortunately, maintaining the existing retention behavior does not involve any new owner +references, so default behavior will be compatible. + +Please tag any issues you report with the label `sig/apps` and assign them to Matthew Cary +([@mattcary](https://github.com/mattcary) at GitHub). + +Enjoy! + diff --git a/content/en/blog/_posts/2021-12-17-security-profiles-operator-v0.4.0/index.md b/content/en/blog/_posts/2021-12-17-security-profiles-operator-v0.4.0/index.md new file mode 100644 index 0000000000..39dfb8cb52 --- /dev/null +++ b/content/en/blog/_posts/2021-12-17-security-profiles-operator-v0.4.0/index.md @@ -0,0 +1,148 @@ +--- +layout: blog +title: "What's new in Security Profiles Operator v0.4.0" +date: 2021-12-17 +slug: security-profiles-operator +--- + +**Authors:** Jakub Hrozek, Juan Antonio Osorio, Paulo Gomes, Sascha Grunert + +--- + +The [Security Profiles Operator (SPO)](https://sigs.k8s.io/security-profiles-operator) +is an out-of-tree Kubernetes enhancement to make the management of +[seccomp](https://en.wikipedia.org/wiki/Seccomp), +[SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) and +[AppArmor](https://en.wikipedia.org/wiki/AppArmor) profiles easier and more +convenient. We're happy to announce that we recently [released +v0.4.0](https://github.com/kubernetes-sigs/security-profiles-operator/releases/tag/v0.4.0) +of the operator, which contains a ton of new features, fixes and usability +improvements. + +## What's new + +It has been a while since the last +[v0.3.0](https://github.com/kubernetes-sigs/security-profiles-operator/releases/tag/v0.3.0) +release of the operator. We added new features, fine-tuned existing ones and +reworked our documentation in 290 commits over the past half year. + +One of the highlights is that we're now able to record seccomp and SELinux +profiles using the operators [log enricher](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#log-enricher-based-recording). +This allows us to reduce the dependencies required for profile recording to have +[auditd](https://linux.die.net/man/8/auditd) or +[syslog](https://en.wikipedia.org/wiki/Syslog) (as fallback) running on the +nodes. All profile recordings in the operator work in the same way by using the +`ProfileRecording` CRD as well as their corresponding [label +selectors](/docs/concepts/overview/working-with-objects/labels). The log +enricher itself can be also used to gather meaningful insights about seccomp and +SELinux messages of a node. Checkout the [official +documentation](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#using-the-log-enricher) +to learn more about it. + +### seccomp related improvements + +Beside the log enricher based recording we now offer an alternative to record +seccomp profiles by utilizing [ebpf](https://ebpf.io). This optional feature can +be enabled by setting `enableBpfRecorder` to `true`. This results in running a +dedicated container, which ships a custom bpf module on every node to collect +the syscalls for containers. It even supports older Kernel versions which do not +expose the [BPF Type Format (BTF)](https://www.kernel.org/doc/html/latest/bpf/btf.html) per +default as well as the `amd64` and `arm64` architectures. Checkout +[our documentation](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#ebpf-based-recording) +to see it in action. By the way, we now add the seccomp profile architecture of +the recorder host to the recorded profile as well. + +We also graduated the seccomp profile API from `v1alpha1` to `v1beta1`. This +aligns with our overall goal to stabilize the CRD APIs over time. The only thing +which has changed is that the seccomp profile type `Architectures` now points to +`[]Arch` instead of `[]*Arch`. + +### SELinux enhancements + +Managing SELinux policies (an equivalent to using `semodule` that +you would normally call on a single server) is not done by SPO +itself, but by another container called selinuxd to provide better +isolation. This release switched to using selinuxd containers from +a personal repository to images located under [our team's quay.io +repository](https://quay.io/organization/security-profiles-operator). +The selinuxd repository has moved as well to [the containers GitHub +organization](https://github.com/containers/selinuxd). + +Please note that selinuxd links dynamically to `libsemanage` and mounts the +SELinux directories from the nodes, which means that the selinuxd container +must be running the same distribution as the cluster nodes. SPO defaults +to using CentOS-8 based containers, but we also build Fedora based ones. +If you are using another distribution and would like us to add support for +it, please file [an issue against selinuxd](https://github.com/containers/selinuxd/issues). + +#### Profile Recording + +This release adds support for recording of SELinux profiles. +The recording itself is managed via an instance of a `ProfileRecording` Custom +Resource as seen in an +[example](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/examples/profilerecording-selinux-logs.yaml) +in our repository. From the user's point of view it works pretty much the same +as recording of seccomp profiles. + +Under the hood, to know what the workload is doing SPO installs a special +permissive policy called [selinuxrecording](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/deploy/base/profiles/selinuxrecording.cil) +on startup which allows everything and logs all AVCs to `audit.log`. +These AVC messages are scraped by the log enricher component and when +the recorded workload exits, the policy is created. + +#### `SELinuxProfile` CRD graduation + +An `v1alpha2` version of the `SelinuxProfile` object has been introduced. This +removes the raw Common Intermediate Language (CIL) from the object itself and +instead adds a simple policy language to ease the writing and parsing +experience. + +Alongside, a `RawSelinuxProfile` object was also introduced. This contains a +wrapped and raw representation of the policy. This was intended for folks to be +able to take their existing policies into use as soon as possible. However, on +validations are done here. + +### AppArmor support + +This version introduces the initial support for AppArmor, allowing users to load and +unload AppArmor profiles into cluster nodes by using the new [AppArmorProfile](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/deploy/base/crds/apparmorprofile.yaml) CRD. + +To enable AppArmor support use the [enableAppArmor feature gate](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/examples/config.yaml#L10) switch of your SPO configuration. +Then use our [apparmor example](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/examples/apparmorprofile.yaml) to deploy your first profile across your cluster. + +### Metrics + +The operator now exposes metrics, which are described in detail in +our new [metrics documentation](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#using-metrics). +We decided to secure the metrics retrieval process by using +[kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy), while we ship an +additional `spo-metrics-client` cluster role (and binding) to retrieve the +metrics from within the cluster. If you're using +[OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift), +then we provide an out of the box working +[`ServiceMonitor`](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#automatic-servicemonitor-deployment) +to access the metrics. + +#### Debuggability and robustness + +Beside all those new features, we decided to restructure parts of the Security +Profiles Operator internally to make it better to debug and more robust. For +example, we now maintain an internal [gRPC](https://grpc.io) API to communicate +within the operator across different features. We also improved the performance +of the log enricher, which now caches results for faster retrieval of the log +data. The operator can be put into a more [verbose log mode](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#set-logging-verbosity) +by setting `verbosity` from `0` to `1`. + +We also print the used `libseccomp` and `libbpf` versions on startup, as well as +expose CPU and memory profiling endpoints for each container via the +[`enableProfiling` option](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#enable-cpu-and-memory-profiling). +Dedicated liveness and startup probes inside of the operator daemon will now +additionally improve the life cycle of the operator. + +## Conclusion + +Thank you for reading this update. We're looking forward to future enhancements +of the operator and would love to get your feedback about the latest release. +Feel free to reach out to us via the Kubernetes slack +[#security-profiles-operator](https://kubernetes.slack.com/messages/security-profiles-operator) +for any feedback or question. diff --git a/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/index.md b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/index.md new file mode 100644 index 0000000000..977a65c147 --- /dev/null +++ b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/index.md @@ -0,0 +1,108 @@ +--- +layout: blog +title: "Using Admission Controllers to Detect Container Drift at Runtime" +date: 2021-12-21 +slug: admission-controllers-for-container-drift +--- + +**Author:** Saifuding Diliyaer (Box) +{{< figure src="intro-illustration.png" alt="Introductory illustration" attr="Illustration by Munire Aireti" >}} + +At Box, we use Kubernetes (K8s) to manage hundreds of micro-services that enable Box to stream data at a petabyte scale. When it comes to the deployment process, we run [kube-applier](https://github.com/box/kube-applier) as part of the GitOps workflows with declarative configuration and automated deployment. Developers declare their K8s apps manifest into a Git repository that requires code reviews and automatic checks to pass, before any changes can get merged and applied inside our K8s clusters. With `kubectl exec` and other similar commands, however, developers are able to directly interact with running containers and alter them from their deployed state. This interaction could then subvert the change control and code review processes that are enforced in our CI/CD pipelines. Further, it allows such impacted containers to continue receiving traffic long-term in production. + +To solve this problem, we developed our own K8s component called [kube-exec-controller](https://github.com/box/kube-exec-controller) along with its corresponding [kubectl plugin](https://github.com/box/kube-exec-controller#kubectl-pi). They function together in detecting and terminating potentially mutated containers (caused by interactive kubectl commands), as well as revealing the interaction events directly to the target Pods for better visibility. + +## Admission control for interactive kubectl commands +Once a request is sent to K8s, it needs to be authenticated and authorized by the API server to proceed. Additionally, K8s has a separate layer of protection called [admission controllers](/docs/reference/access-authn-authz/admission-controllers/), which can intercept the request before an object is persisted in *etcd*. There are various predefined admission controls compiled into the API server binary (e.g. ResourceQuota to enforce hard resource usage limits per namespace). Besides, there are two dynamic admission controls named [MutatingAdmissionWebhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) and [ValidatingAdmissionWebhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), used for mutating or validating K8s requests respectively. The latter is what we adopted to detect container drift at runtime caused by interactive kubectl commands. This whole process can be divided into three steps as explained in detail below. + +### 1. Admit interactive kubectl command requests +First of all, we needed to enable a validating webhook that sends qualified requests to *kube-exec-controller*. To add the new validation mechanism applying to interactive kubectl commands specifically, we configured the webhook’s rules with resources as `[pods/exec, pods/attach]`, and operations as `CONNECT`. These rules tell the cluster's API server that all `exec` and `attach` requests should be subject to our admission control webhook. In the ValidatingAdmissionWebhook that we configured, we specified a `service` reference (could also be replaced with `url` that gives the location of the webhook) and `caBundle` to allow validating its X.509 certificate, both under the `clientConfig` stanza. + +Here is a short example of what our ValidatingWebhookConfiguration object looks like: +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingWebhookConfiguration +metadata: + name: example-validating-webhook-config +webhooks: + - name: validate-pod-interaction.example.com + sideEffects: None + rules: + - apiGroups: ["*"] + apiVersions: ["*"] + operations: ["CONNECT"] + resources: ["pods/exec", "pods/attach"] + failurePolicy: Fail + clientConfig: + service: + # reference to kube-exec-controller service deployed inside the K8s cluster + name: example-service + namespace: kube-exec-controller + path: "/admit-pod-interaction" + caBundle: "{{VALUE}}" # PEM encoded CA bundle to validate kube-exec-controller's certificate + admissionReviewVersions: ["v1", "v1beta1"] +``` + +### 2. Label the target Pod with potentially mutated containers +Once a request of `kubectl exec` comes in, *kube-exec-controller* makes an internal note to label the associated Pod. The added labels mean that we can not only query all the affected Pods, but also enable the security mechanism to retrieve previously identified Pods, in case the controller service itself gets restarted. + +The admission control process cannot directly modify the targeted in its admission response. This is because the `pods/exec` request is against a subresource of the Pod API, and the API kind for that subresource is `PodExecOptions`. As a result, there is a separate process in *kube-exec-controller* that patches the labels asynchronously. The admission control always permits the `exec` request, then acts as a client of the K8s API to label the target Pod and to log related events. Developers can check whether their Pods are affected or not using `kubectl` or similar tools. For example: + +``` +$ kubectl get pod --show-labels +NAME READY STATUS RESTARTS AGE LABELS +test-pod 1/1 Running 0 2s box.com/podInitialInteractionTimestamp=1632524400,box.com/podInteractorUsername=username-1,box.com/podTTLDuration=1h0m0s + +$ kubectl describe pod test-pod +... +Events: +Type Reason Age    From                            Message +----       ------       ----   ----                            ------- +Warning PodInteraction 5s admission-controller-service Pod was interacted with 'kubectl exec' command by user 'username-1' initially at time 2021-09-24 16:00:00 -0800 PST +Warning PodInteraction 5s admission-controller-service Pod will be evicted at time 2021-09-24 17:00:00 -0800 PST (in about 1h0m0s). +``` + +### 3. Evict the target Pod after a predefined period +As you can see in the above event messages, the affected Pod is not evicted immediately. At times, developers might have to get into their running containers necessarily for debugging some live issues. Therefore, we define a time to live (TTL) of affected Pods based on the environment of clusters they are running. In particular, we allow a longer time in our dev clusters as it is more common to run `kubectl exec` or other interactive commands for active development. + +For our production clusters, we specify a lower time limit so as to avoid the impacted Pods serving traffic abidingly. The *kube-exec-controller* internally sets and tracks a timer for each Pod that matches the associated TTL. Once the timer is up, the controller evicts that Pod using K8s API. The eviction (rather than deletion) is to ensure service availability, since the cluster respects any configured [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) (PDB). Let's say if a user has defined *x* number of Pods as critical in their PDB, the eviction (as requested by *kube-exec-controller*) does not continue when the target workload has fewer than *x* Pods running. + +Here comes a sequence diagram of the entire workflow mentioned above: + + <!-- Mermaid Live Editor link - https://mermaid-js.github.io/mermaid-live-editor/edit/#pako:eNp9kjFPAzEMhf-KlalIbWd0QpUQdGJB3JrFTUyJmjhHzncFof53nGtpqYTYEuu958-Wv4zLnkxjenofiB09BtwWTJbRSS6QCLCHu01ZPdJIMXdUYNZTGYOjRd4zlRvLHRYJLnTIArvbtozV83TbAnZhUcVUrkXo04OU2I6uKu99Cn0fMsNDZik5Rm3SHntYTrRYrabUBl4GBmt2w4acRKAPcrBcLq0Bl1NC9pYnoRouHZopX9RX9aotddJeADaf4DDGwFuQN4IRY_Ao9bunzVvOO13COeYCcR9j3k-OCQDP9KfgC8TJsFbZIHSxnGljzp1lgKs2v9HXugMBwe2WPHTZ94CvottB6Ap5eg2s9cBaUnrLVEP_Yp5ynrOf3fxPV2V1lBOhmZtEJWHweiFfldQa1SWyptGnAuAQxRrLB5UOna6P1j7o4ZhGykBzg4Pk9pPdz_-oOR3ZsXj4BjrP5rU--> + +![Sequence Diagram](/images/sequence_diagram.svg) + +## A new kubectl plugin for better user experience +Our admission controller component works great for solving the container drift issue we had on the platform. It is also able to submit all related Events to the target Pod that has been affected. However, K8s clusters don't retain Events very long (the default retention period is one hour). We need to provide other ways for developers to get their Pod interaction activity. A [kubectl plugin](/docs/tasks/extend-kubectl/kubectl-plugins/) is a perfect choice for us to expose this information. We named our plugin `kubectl pi` (short for `pod-interaction`) and provide two subcommands: `get` and `extend`. + +When the `get` subcommand is called, the plugin checks the metadata attached by our admission controller and transfers it to human-readable information. Here is an example output from running `kubectl pi get`: + +``` +$ kubectl pi get test-pod +POD-NAME INTERACTOR POD-TTL EXTENSION EXTENSION-REQUESTER EVICTION-TIME +test-pod  username-1  1h0m0s   /          /                    2021-09-24 17:00:00 -0800 PST +``` + +The plugin can also be used to extend the TTL for a Pod that is marked for future eviction. This is useful in case developers need extra time to debug ongoing issues. To achieve this, a developer uses the `kubectl pi extend` subcommand, where the plugin patches the relevant *annotations* for the given Pod. These *annotations* include the duration and username who made the extension request for transparency (displayed in the table returned from the `kubectl pi get` command). + +Correspondingly, there is another webhook defined in *kube-exec-controller* which admits valid annotation updates. Once admitted, those updates reset the eviction timer of the target Pod as requested. An example of requesting the extension from the developer side would be: + +``` +$ kubectl pi extend test-pod --duration=30m +Successfully extended the termination time of pod/test-pod with a duration=30m +  +$ kubectl pi get test-pod +POD-NAME  INTERACTOR  POD-TTL  EXTENSION  EXTENSION-REQUESTER  EVICTION-TIME +test-pod  username-1  1h0m0s   30m        username-2           2021-09-24 17:30:00 -0800 PST +``` + +## Future improvement +Although our admission controller service works great in handling interactive requests to a Pod, it could as well evict the Pod while the actual commands are no-op in these requests. For instance, developers sometimes run `kubectl exec` merely to check their service logs stored on hosts. Nevertheless, the target Pods would still get bounced despite the state of their containers not changing at all. One of the improvements here could be adding the ability to distinguish the commands that are passed to the interactive requests, so that no-op commands should not always force a Pod eviction. However, this becomes challenging when developers get a shell to a running container and execute commands inside the shell, since they will no longer be visible to our admission controller service. + +Another item worth pointing out here is the choice of using K8s *labels* and *annotations*. In our design, we decided to have all immutable metadata attached as *labels* for better enforcing the immutability in our admission control. Yet some of these metadata could fit better as *annotations*. For instance, we had a label with the key `box.com/podInitialInteractionTimestamp` used to list all affected Pods in *kube-exec-controller* code, although its value would be unlikely to query for. As a more ideal design in the K8s world, a single *label* could be preferable in our case for identification with other metadata applied as *annotations* instead. + +## Summary +With the power of admission controllers, we are able to secure our K8s clusters by detecting potentially mutated containers at runtime, and evicting their Pods without affecting service availability. We also utilize kubectl plugins to provide flexibility of the eviction time and hence, bringing a better and more self-independent experience to service owners. We are proud to announce that we have open-sourced the whole project for the community to leverage in their own K8s clusters. Any contribution is more than welcomed and appreciated. You can find this project hosted on GitHub at https://github.com/box/kube-exec-controller + +*Special thanks to Ayush Sobti and Ethan Goldblum for their technical guidance on this project.* diff --git a/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/intro-illustration.png b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/intro-illustration.png new file mode 100644 index 0000000000..c52b432b28 Binary files /dev/null and b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/intro-illustration.png differ diff --git a/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/workflow-diagram.svg b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/workflow-diagram.svg new file mode 100644 index 0000000000..746b862fbd --- /dev/null +++ b/content/en/blog/_posts/2021-12-21-admission-controllers-for-container-drift/workflow-diagram.svg @@ -0,0 +1,6902 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="1982px" height="1400px" viewBox="0 0 1982 1400" enable-background="new 0 0 1982 1400" xml:space="preserve"> <image id="image0" width="1982" height="1400" x="0" y="0" + href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAB74AAAV4CAYAAAA3xnIQAAABOmlDQ1BpY2MAACiRY2BgEkgsKMhh +YWBgyM0rKQpyd1KIiIxSYH/KwMUgxCDGwMggkJhcXOAYEOADVMIAo1HBt2tAdUBwWRdkFsfk1s72 +ffpeS0TOqDz6ulcBUz0K4EpJLU4G0n+AOCW5oKiEgYExAchWLi8pALFbgGyRIqCjgOwZIHY6hL0G +xE6CsA+A1YQEOQPZV4BsgeSMxBQg+wmQrZOEJJ6OxIbaC3aDj7uCh0uQgok3AceSCkpSK0pAtHN+ +QWVRZnpGiYIjMIRSFTzzkvV0FIwMjAwZGEDhDVH9WQwcjoxipxBildcZGCwCgYwYhFjITQaGXQEM +DNyzEWIaagwMvMBwO25akFiUCHcA4zeW4jRjIwibp4iBgfXH//+fZRkY2HcxMPwt+v//99z///8u +YWBgBpp5oBAAFZ9dxoUSB3gAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdw +nLpRPAAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAAXp6VFh0UmF3IHBy +b2ZpbGUgdHlwZSBpY2MAADiNrVPZbeQwDP1XFVsCb0rlyJINbP8NLHU4O5NJggTIgw2aong90ulv +a+nPgFJOMEAIhg4O1kDmCVi308VJSVyIQLMWrQTgp4a5AmAoaPGN40mGxs4OggoK0mDjvf4Vrsia +VsSFztTfKsuFs+aS/TjpkloFpTWSUPTo9BIsfSfjA6qJqbPxrmVXbJKiMXDyxcxseYDYPRiC+zxv +5nCwNtlaasn3eTJ/NNR2G+zJ4aZqBpIYwKoI+50gyGbLQfjOfG4DNfvEIRJ/Euip5/9cSIvkIFnC +jYBFjq/J+ynZvx/IRyd5D2MEErYrplOcrJhbmKKhvUum8V3sNKaTWjRIPN65d2+b/ljRxfeqTyBa +q+8vzipK9yFjU/ddbfe9p9ZUehnyqLoOjnJ8FJAyrYXrSzRnTT5+VzKEV+BlIDFRY+bHSBieS9cr +XwOMfVXkx1y8qztNw1H6lL0uMnlV2sAx/QNjKeG+t1FH5AAAAVt6VFh0UmF3IHByb2ZpbGUgdHlw +ZSB4bXAAACiRlVNLcoMwDN3rFD2CkWTJHIdie9eZLnv8PhlaJyHNTPEkEP3eB4W+Pj7pLS7NiWT3 +IpsXq56sWna1hVP8tt2aS+SkMttiat3YsmxH/Le6M3OiOQbB92jJRWtOnFSsOxo5SeYmik9ilpY2 +TnFAgTHcZMusSmoP+EcyOBRXnCQbMLuPi5ujiNuAcO6yyBqHO0kSRoDxXY8huIuvGAvaXrgGQKQn +l0dGqpbJxQSBdUhb4UIDw7MAvjCw4UQwhBF3g26PF5qiYF3BqCeyBqc27cYz/LOKzNkT0pZhYr9S +/xHwC7DlBObFsu14IcEf7y1AQglZkzYsKzBinxiv26LrHpae4GYUZ2cd9M92lZYPN/M5rsA9ndD0 +qvUfUld6RhpSFWuR/pS6Xrvouh3cb/fjuh63uz1rKdZbLZIjfPnfRYa+AfG11o/yanNeAACAAElE +QVR42uz9aXMcV5bm+z7bx5gHzABBcKbGrBw6q/uc2/eYdds1u2/6Q7dVtVVXVVZlZ5ZmiRJBgiRm +IObBx31eBMIJSJSogRIA5f+XGQJEICLcNwKUezy+1jLWWisAAAAAAAAAAAAAAK4p57I3AAAAAAAA +AAAAAACAn4LgGwAAAAAAAAAAAABwrRF8AwAAAAAAAAAAAACuNYJvAAAAAAAAAAAAAMC1RvANAAAA +AAAAAAAAALjWCL4BAAAAAAAAAAAAANcawTcAAAAAAAAAAAAA4Foj+AYAAAAAAAAAAAAAXGsE3wAA +AAAAAAAAAACAa43gGwAAAAAAAAAAAABwrRF8AwAAAAAAAAAAAACuNYJvAAAAAAAAAAAAAMC1RvAN +AAAAAAAAAAAAALjWCL4BAAAAAAAAAAAAANcawTcAAAAAAAAAAAAA4Foj+AYAAAAAAAAAAAAAXGsE +3wAAAAAAAAAAAACAa43gGwAAAAAAAAAAAABwrRF8AwAAAAAAAAAAAACuNYJvAAAAAAAAAAAAAMC1 +RvANAAAAAAAAAAAAALjWCL4BAAAAAAAAAAAAANcawTcAAAAAAAAAAAAA4Foj+AYAAAAAAAAAAAAA +XGsE3wAAAAAAAAAAAACAa43gGwAAAAAAAAAAAABwrRF8AwAAAAAAAAAAAACuNYJvAAAAAAAAAAAA +AMC1RvANAAAAAAAAAAAAALjWCL4BAAAAAAAAAAAAANcawTcAAAAAAAAAAAAA4FrzLnsDgB8rz3Ol +aaosyzQZTzSejJUkSfFnxhhJkud68gNfvu+rXC6rXC7L930ZY4rvAQAAAAAAAAAAAHB9EXzj2sqy +TFEUaTKZ6Pj4WMfHxxoMBppMJoqiSMYYua6rIAhUq9VUq9W0tLSk5eVlGWPkeR7BNwAAAAAAAAAA +APArQPCNa8FaK2ttEXbHUazReKTBYKh+v6e9/QMdHByq1+tpPJlqOp3IcVw5xqhULqleq6nZbGit +19doNFKr1VKpVFa5XFIQBAr8QI7rEIQDAAAAAAAAAAAA1xDBN66FNE01mUw0Ho+1vf1ET57uaH9/ +X51OR91uV9M40zTOlaS50swqzXIZaVbZ7Rr53uxWLvkqh74W2m2tra9rY31dN25s6ObNm6rVanJd +V47jXPbuAgAAAAAAAAAAAPgBCL5xLaRpqvF4rJOTE3308cf6t3/7d21vP9Hx8ZG63a78UkN+uSnX +K0uOLzm+jKwkK5slstlYeTJVGo+VJmMttJu6d+++3n7rbf32t3+nRqOpcrksxzgSuTcAAAAAAAAA +AABwrRB848qy1hZV3nt7e3q8va3Hj5/o0eMd7eye6nQoTW1DtlSRyg055aaMX5JxAhnXl+zL4DtL +x5I3lXUGyjTUMPG0fzKRHj1Rkku9/kB3bt/S5s1N3djYkOu6zAAHAAAAAAAAAAAArgljrbWXvRHA +q+R5rsPDQx0eHuqTTz7Vn/7t3/QfH3yoKAsVZaFytyonbMkJm3K9krygJMfxJONKxpFkZSRZm8vm +qWyWKIlGSuORbNKXSXpysqFajZJa9ZIe3L+r/8///X/p//ov/0VBGKhcKsv13MteBgAAAAAAAAAA +AACvQcU3rhxrrfIsV5zEOjw81JdffaWPPv5Ef/3gE330ySNVWjdUbW2p2thQZWFLldaWjDEyjiPp +uyq0rdJkojyZatJ9ocHRlxp1T3Xa7crJRur3+2q12rpxY0MLi4tyF12VvfJlLwcAAAAAAAAAAACA +1yD4xpWTJInG47G63a4++uhj/elPf9JXO/sap6Fa6++r1FhXubmuUm1Zfql5LvB+XVtyI8fxJc8q +rC3J2lxBua5ouK9keKje2OqDjz9XFE30u9/+Vn/4wx8UhuEsVKflOQAAAAAAAAAAAHBlEXzjyonj +WP1+XwcHh/rk00/1z//8rzoZJCovPlB784HC6pJK9WW5QUWOM5/D/f2CacfxZBxXjhvILzeVpVsa +HHyhYW7Un5zok8++1M6TR3IcV3fv3tXS0qJc1yX4BgAAAAAAAAAAAK4wgm9cOYPBQI8efalPP/1M +T54daBQ7yr2mvPKSys0N+aWGvLAux/V/+IMbIyMj4zqS68vxSirVV5RnsaK+o8lwT9PxiZ7v7umr +x48VhoEWFhbUarUue1kAAAAAAAAAAAAAfAuCb1w5p52OPv7kE/3vf/5XHXZiOeUV1YIFlZsbCquL +clxfxnHfyHMZGQXVRTleIMd1lcUjTcenOjrq6rPPvpDvubp//wHBNwAAAAAAAAAAAHCFEXzjysjz +XGma6vS0o8ePH+vjTz6RW9mQW7mhsHFD5caa/FLjzT6pMfJLDfmlumyWaNLbl3UOdNId6otHXyoM +PC20F3Tv3t3LXh4AAAAAAAAAAAAA34LgG1eCtVbD4VCj0UhHR0fq9gYajyNVq4Gq9RWVGqtyg+rP +uAVGblBVqbGuPIs1jkf6antH5dDX/fv3L3t5AAAAAAAAAAAAAHwHgm9cCdZajcdjHR0d6fj4RL1e +X5NppIoJFdbXVGqsyXF+3perG5RVbm7IcQONDj/S0e5zlXyj45MTWWslScaYy14qAAAAAAAAAAAA +AF/jXPYGAJJkc6t+v6+9vT0dHZ8oSqzcoCYnqMn1K3Lc8I3N9f42juPLLzUU1pZk3VBRnGk0Hmkw +GKrX62k6nSrP88teKgAAAAAAAAAAAABfQ8U3roQsz9Tr9fX8+QsdHXeU5L6C6rL8Ul2O6/8ildbG +8eUGZckYuV5ZxnGVZVbD0UgnxydqNBtyXVdBEFz2cgEAAAAAAAAAAAA4h+AbV4K1VoNBXwcHBzrt +9JVaT0FlUW5Yl/mZW5zPGceR65RkjCvXD2VcX1kujccTnZ6eyvVc1Wq1y14qAAAAAAAAAAAAAF9D +q3NcGVEUazAcaTyZKrOOvLAixw0k88u/TF0vkBeU5bi+4iRVrz9rdZ6m6WUvEwAAAAAAAAAAAICv +IfjGlWCt1TSKNBgOzoJv92y2dyDzSwffxshxfHl+RcYNFSepBoOBJpMJM74BAAAAAAAAAACAK4jg +G1dGnufK0kxZnstKMo47C71//vHeFxhJMo4cx5GMozy3StOU0BsAAAAAAAAAAAC4ogi+cTWZ4h9X +irX2sjcBAAAAAAAAAAAAwNcQfOPKcBxHruvKdRwZK9k8mwXNl5E1Wyub55LN5Rgj13XlOI6MuXph +PAAAAAAAAAAAAPC3juAbV4IxRmEYqlavqVwuyTW5smSsPItl7S/bYtxaqzxPlCYT2SyW77uq1Woq +lUqz9ucAAAAAAAAAAAAArhRSPFwZYRCoXquqUg7lmExpPAu+9QsH35KUZ4myZCqbJwp8T/V6XaVS +Sa7rXvYyAQAAAAAAAAAAAPgagm9cCcYY1WpVraysqN1qyDOZksmpsngom2e/yDZYmytPI6XxWFk8 +VZbGMspVKZe1uLioarVK8A0AAAAAAAAAAABcQQTfuBJcx1Wz2dTG+rqWFlrynUTx6FjpdCCbJ/ol +Bn3bPFOWRkqjodJ0ojxL5DpG1WpZy8vLqtVq8lzvspcKAAAAAAAAAAAAwNeQ4uFKMI5Rvd7Q+vq6 +lp+9UOhJWTRQlgyVJbMQ2hhXxvn5Kq7zLFEyHSganUhZJN9zVC6XVK/V1Ww2ZYyRMeaylwoAAAAA +AAAAAADA1xB840qYtzrP82UtLS2qXq8qDDwpnWo6OJBxA/mlhvxS42fbhiwZa9Lb1aS7o7Kfqn1z +Q3fu3NHi0gKBNwAAAAAAAAAAAHCFEXzjSjDGqFatqVQqaWV5WY16TYHvSflU0fBIxg1lHO/nDb7j +iab9PY1OvlJ7taWtG5u6d++uFhYIvgEAAAAAAAAAAICrjOAbV4bruXJcR+12W3fv3tHe3oFOhrk6 +wz2N0lTGceWFNTmuJ8cNZMwbGFFvrZJ4qDQaadrfVTo5kZKB2s0NPXxwXw8fPNDS4uJlLw0AAAAA +AAAAAACA70DwjSvFGKOFhbbee+9dyUr//tePdXz4mabDgdygorC6JC+syhhPxv3pwbeVVTLpnbU4 +f65s2pGrREuLTb3z9lt66623tLS4dNnLAgAAAAAAAAAAAOA7EHzjymk2m3r44IGqlYqOjo705ecf +aTrsKB0fadLbVVhdkCS5flmO48k47g94dCtrJWsz2SxRlkaaDg407uwomxwpdFOVW1Vtbqzq/v27 +un37ljyXXxMAAAAAAAAAAADgKiPRw5UTBIEajYZsbvX2O2+r3+9r+9mBTvpDne78m8rNGyo3b6hU +X5JfbikoN8/u+fo53DbPleeJ0ulA0fBQ0+GRosGeosG+mlVf927d0f07m/rNb36jhXZbnuvJOMz3 +BgAAAAAAAAAAAK4ygm9cOUEQqN1qq1Kp6P333pPruir/9T/073/+q57sfql40lOWTmXzVMbx5Jca +kozMa/NpK5unyrNEybij0ckTDU+fKJ92lUU91dobeu+dB/r////+u5ZXlrWwsCDHdWRe/8AAAAAA +AAAAAAAALhHBN64cY4xcz1XJKWl1dUVZlilNU0VRJGNyxXlFkQZK+jsaJj1Nui/k+qFcryTjejLG +lcx8/reVzXMpT2eV3tFIaTxWHvdlkq4a5VyNxSU1qut6cP+O3n3nLW3d2lK5XJbv+4TeAAAAAAAA +AAAAwDVA8I0ryxijVqulIAjkB75qtZpu3tzUF1/t6KsnL9TpPtXoNFecSUG5oaDckuNVZFxfxvUl +ayVZ2SxRnk6UJxPF466iaV8l39FCq67llYYe3L2thw/u6c7tLW1tbaler8vzPLnuD5kdDgAAAAAA +AAAAAOCyEHzjyjLGqF6vq16vq1qtamlxUbdv3VIY/m9NRj1lk4760UDJZCibD5TlI+VuWXL82c3a +2QPlkZRNZdOJbDSQEw1VCmpabrV179a6fvfb9/Sf//6P2tjYUBAECoLgsncdAAAAAAAAAAAAwA9A +8I1rIQgC1et1SdLvf/c7tVotHRwcqtvtqtvraTRJNJokmkSpoihRlKRyjJFjHAVBVeWwpUrJU6Uc +qFIOtNBuaXV1VWtra9rauql2u60gCOQ6VHkDAAAAAAAAAAAA142xdl4WC1xd1lrlWa4szxRFkaZR +pOFgoE6no5OTU+08e6Znz57r6OhInV5f/UFfvuvJcz3VGzUtLS5qeXlZN29u6ubNTS0tLqparapa +rSoMQ/m+L8+bXQfCXG8AAAAAAAAAAADgeqHiG1dGnueyuVUURxqPx5pMJppOp5pOp4qi+OzzSEmS +KI4jTSYT9Qd99Xt9HR4d6ujwSL1eT6PR7L6O68p1HU3GQ03GIw36PQ0HfZ2enKjeqKtUKqkUhgrC +ksIgVBgGsz8rhSqVyiqXS2f/XlIYhgTiAAAAAAAAAAAAwBVF8I0rI8sypWmqfr+vw8PDWfV2p6tO +tzP72Omq2+1pNBppNBpqMh5rGk0VTaeKokhRHCtNM+U2V57PGhkYGTmOkeu68j1HYamscqmksFRS +GM4C7UajoUajoWazqXarpYWFlhYXF7WwsKCFhQUtLS4pDENZawm/AQAAAAAAAAAAgCuI4BuXIksz +pVmqJEmUpqnieFbRPZlMdXJyot29Pe3v7evo5FTHxyc6PjnV8cmpOp2ehsOBRsOR4jhSmiXK0kyu +48g47lmVtyfH9WRlZXMrm2fKslR5ns6e3EqO68j3AvlB8DL0bre0tLig5aUFLS8vaXV5WaurKxoM +hhqNR0U7dN8PFAS+Aj+QcYyMMQTiAAAAAAAAAAAAwCVixjd+cXmeq9frqdPp6OjoSHv7Bzo8PFSn +01Wv19VwNNYkSjSZpppGuaZxqmmUaRrPbmlmlaZWmbWav3qNcWUcI8dxZRxXxnFmX7D2rIV6pjzP +JGtlZWVkZYzkGsn3Hfmeq9B3VA49lUNXpcBVKXRUKfmqVEqqVipaaC9oYXFBqysrWl/f0Pr6qkql +ksqlslzPvexlBQAAAAAAAAAAAP5mUfGNX5zNrXq9nnZ3d/Xo0Zf69LPP9ejLRzo8PNTx0ZHSzMoL +avJKNTl+VY5XlePX5AYNufWqAr8mL6zK8Upy3EDG9WcV18bRrPB6XoH9MhiX7Cz0tmcheBYrTyNl +SaQ0GiiLhxrGI/UnI+VxV1ncVxb1ZbNYspkcY3XjxoZu3tzSg/v39d77E1XKJdmmVRAEckXwDQAA +AAAAAAAAAFwWgm/87OZtzbvdrvb29rV/cDD7uH+g3YMj7e4d67CbahiXlLiLsp4nlepyyg05fkWu +X5UX1s4+nt2CilwvlOP6s+BbRvoe7catzWVtfhZ8z25JVFMaDZXGI2XxSJk/lvVqyt268mSiLIlk +81jdsStzOFCqHQ3GU714savVlRWtrq1qeWlZi4uLWlxcoO05AAAAAAAAAAAA8Auj1Tl+VtZaRVGk +6XSqR4++1L/+6U/6jw8+VLfbV6/X1yQxSlVWaqqyblnGLcvxy/L8stygPAu2HV/GDYqQ23FmM7yN +48kYR8Y40lml9/fYoNns77PK71n1d6I8S2TzVDZLZLO4CMbTZKosGSmLJ3LsWE4+laeJPEUKnERb +W1u6dWtLDx8+1Hvvvqf3339Xkgi/AQAAAAAAAAAAgF8QFd/4WVlr1R8MdHpyoi8efam//PVD/enf +/qwoSRVFmZywqXKzpnJrWaX6qsqNVfmlmozjy3F/hpenMTIys7Dcef3jZ2mkNB4rjYaadF9o0t/T +tNfXuH+geHSs49Oejk5OFcWJGo2GtrY2FQShwjCQM58zDgAAAAAAAAAAAOBnRfCNn1Wapnr81WP9 +xwcf6JPPvtTe0UDWbyms1lUOGvJKLQXltvzygvxSXW5QnlVyX5HQ2HE8uV44qyVvrskLKgqrCyo1 +byiddGTcSMedib56vKN2a0G1alXr62taX19XrVa77M0HAAAAAAAAAAAA/iYQfONnlWWZtp880T/9 +0//W9rMDjRJfuddSrX1b1cW78ivtWXW348k47uz2Ped1/xKM48j1S3K9cBZ611eUp7GyJFIej9Tf +/0jHhx/LZs9Vq1VUCj3leaZWq03wDQAAAAAAAAAAAPxCCL7xs8jSTFEcqdfr6eDgULt7++p0R3Jr +N1RZ2FTY2FBQW5YX1M4y7qsRdH+Tmc3rNpJxXDmSrFeS52fKwqqCwYr8yokiO9LuwakcfaRGo6G7 +d+9d9oYDAAAAAAAAAAAAfzMIvvGziJNY/X5fR0dH6nQ66vf6ihKjRnlR9eUH8sK6XK80C5WvGSMj +Oa5chQrry6rlsdLRno5OD9U/eaHbt29pMplc9mYCAAAAAAAAAAAAfzMIvvGziONY3W5X+/v76nR7 +Gk9jpbYir9RSpb2lopL6OjKzbTfGKKguyjieRnmi7slzRZ3nOjg40Gg0UpIkcl1XzhWZVw4AAAAA +AAAAAAD8WpHI4WcxHo+1v7+vx9tP1B9GcsKGwkpbrl85C40vewvfAKPZ7O9SY9ay3fWV5dJkGun0 +9FTdblfT6fSytxIAAAAAAAAAAAD41aPiGz+LyWSig4MDbW8/UX8UyQ0aCvwFeUFFV3ee9w9l5Hih +AjdQFNbkuL6slSaTqTrdrjqdjlzHVaVSuewNBQAAAAAAAAAAAH7VqPjGz2I8mWhv/0BPnjydVXwH +DQWVBbl+6bI37Y0yxpFx3Fnld1CVG9Y1iXLt7R1ob29fg+HgsjcRAAAAAAAAAAAA+NWj4hs/i8l4 +osODQz199lx5sCqntCa/siDH/3VWPxs3lFdqKqguaJpY7e7ta3GhqYWF9mVvGgAAAAAAAAAAAPCr +R8U33ihrrbI002g01vHJiQ4PDjWeJr/aiu854/rySw0FlWVNIqsXewd68WJX/X7/sjcNAAAAAAAA +AAAA+NWj4htvjLVWURRpOp2qP+hrNB4rimOV5cgvNeSXGnK84LI382fheoG8UlNhbUWTpK+9vUO1 +GxX1+7Q6BwAAAAAAAAAAAH5uBN94o6Io0nA41HAw0Hg8VhRHyuXKC+vySw0Zx73sTfxZGNdXUG4p +T2NNu30Nukdq1UP1CL4BAAAAAAAAAACAnx3BN96YPMvV7/e1u7urw+NTxankBTV5QUWOV5JxPBnz +ijtaKysra3MpzyTjyBhXxrnsTvxW1lrZPJPNUxnHl3FcmVfshOP48kp1BXmqadfTeDrVoD/QYDDQ +cDiU7/vyfV/Ope8TAAAAAAAAAAAA8OtD8I03Jre5Op2OdnZ2dHh0rDR3VaotyQ8bclzvlYHx+dA7 +T2PlWSrjGLluIOOEl7g3VtZKNs+UpVPlaSzXL8k1Jcl8s2rdcX15QVWSkXECJUmmyWSiwWCgbrer +Wq0mx3EIvgEAAAAAAAAAAICfAcE33pg8z9XtdvX06Y4ODk+U5J5KtWX55boc13/lfazNleeJ8jRW +Go2URkMZ11NQbsp3PMmYs8Dc/LCN+an7kqXKs1hZMlEyHSiNxgqrbZmKcxZ+X9we47hy/fKsItwL +ZeUoSlINBkMdHx/LWqswDOX7/o/cIgAAAAAAAAAAAADfhuAbb4y1VqenXe08e66DwxPFma+wtiIv +aMg4r36pZVmkZNJXPD7V6PSZxt1duUFZjeX7qi3dleuHcr3SLzob3NpcyaSjaHisyeBQk/6BotGp +mqtvqbn+rsKqL+M4MuZi9bYxkjGOvLCqoNKW8Tx1ekN99dVXyvNcjUZD5XL5Un9GAAAAAAAAAAAA +wK8RwTfemDzP1el29ez5Cx0cDaTyhsL6qrxS/VuD6zyNZ6F394W6e5+od/C5/FJDxjgq1ZYk1eW4 +gYx+ueBb1ioedzQ6faLB8RMNT59q0j+UjFG5uaGg3JSM/43gWzKz4NuvKqwuSl6iTm+ox48fq1ar +aXNz89J+NgAAAAAAAAAAAMCvGQOH8ZPlea4kSTQeT9Tv99U5PdVwNJE1gcLqkryw/q0V33kaK5n0 +FA0OFE86SqKh0mioZDpQPOkpS6ayefaL79OsxXlPybSjZNpXNB0pmfQUj08UT/rKkum33tcLKwqq +i7JuVZ3eSF893tbh0ZGiKPrF9wMAAAAAAAAAAAD4W0DwjZ8syzJFUaTxeKThcKDBoK9pFElOqLC2 +JC+syvmW4NtmsdJpV/HoSHkylpGVlCtLp0qmfWXJRNb+0sG3lc1iZfFQeTKW8lTGSHky0XR4omh0 +8u3BtzFy/YrC6qKsW1GnN9T29lMdHR4riuJfeD8AAAAAAAAAAACAvw0E3/jJ0jTVaDRSp9PRYDjS +dBopy6yMFygot+T5ZRnnVS81qzyNFE97ikfHstlUnufI8zzJpkrikbI0krX5L75PeZYoT6eyeSzX +NSoFvqRUyaSjeHyiPJl8yz2N3KCqoLIoeRX1BxO92N3TyWlHk8lUeZ7LWvuL7w8AAAAAAAAAAADw +a8aMb/xk0+lUBwcHevLkqYajqdywrsCvyvUrZ4G3Obu9ZG0uWas0nSqe9BWNuvI9qdxoyi+35Ieh +dPY9l8Vaq8APVGo05TYrUhDIJgMl41Nl9ZVX3scYyQ3KCqptJZMTJWmuyVkV/HA40Hg8VhAECoLg +0vYLAAAAAAAAAAAA+LUh+MZPNh5PZsH306cajCJ5YV0l05QXVGTMtzQVsFbW5sqSSMmkr3jSU6XZ +VKvdkl9uK3HLyq2VlaRLyb6tZK1831e9UlGt7Kk/yTWc9mfB93dVfPtlOa6vaVBTkuUajUYaDAYa +DgeaTCZyHIfgGwAAAAAAAAAAAHiDaHWOn2w8Hmt//0Db20/UG0xlgrr8cluuX3r1HaxVlk6UTLrK +k5E816hcClWulFWtVFUqV+T7l3xNxlnY7geBFtptbW7eUK1aUh6PFE86SuOxrM1e2YbdGFeOG8j1 +S3L9ihyvpGmU6uDgSPv7hxoOh5e7bwAAAAAAAAAAAMCvDME3frLxZKy9gwNtP91RbzCR8RryK205 +XvmV329llUYTTYfHypOhSoGnRrOlWq2ucrks3/fluO5l75YkKQwCLSwuamvrlurVkrJkrOm4pzQe +Kc+zomX7ecZIkpHrBgpKdZVqC4oSaXf/ULu7zwm+AQAAAAAAAAAAgDeM4Bs/mrVW1lqNxxMdHR3p ++fNdDcaRTNCQX16Q65e/7Y7K4qGi0YnyZKRyOdTS0rIajabCUijP87+9RfovzPcDtdotbWysq16t +yFUqm0yVJVPlyUR5lryi6tvIGCPHC+WXmwqqi5qm0t7+gZ4/31Wv17/s3QIAAAAAAAAAAAB+VZjx +jR8tz3KlWarxeKxet6dut6uqv6Zqqamw8u2tzq2s0nikeHis0Ey02K5rodZQkqRKkkRJctl79pJx +HHmepzAMVavX1Wi2ZCJPTh5r0j9QUG7JlJtynW9WqBs3kFdqKqwuaxzlerG7r2ajovv37132bgEA +AAAAAAAAAAC/KlejrBbXjrVWWZ4pSRJNJmP1ej31e13FaS6v1FRQbb+m4nukeHwsJx9rsV3Xvbt3 +tbS0KD/wL3vXLnCMeRl812pqtduqVcqSjTUdHCqZ9GXz9NX3dX35pab86oomsfRi71DPn79Ql4pv +AAAAAAAAAAAA4I2i4hs/irVWw8FQ3V5XJycdxUkm44Zy3LI8vyTHDWVeUQUta2XzVGk8UjQ6lYJA +7WZNt+/cUm6tev2BpOiyd+8lo1nbcsdRs9nU+tq63GCkQZxpODiUF1QUZsuvvKvjBfJLDYW1WHHn +VKNeX6cnJxoM+orjWK7jynEdmdlQcAAAAAAAAAAAAAA/EhXf+FFsbtXr9/Ts2TMdHp0oyY3CSkt+ +qSbHK8k4roy+FuhaKysra3Ml0VjRqCNlE7Vbdd29c0cLCwvy/at5LYbjOGo1m7q1tan1tWWFXq54 +dKA06svmr+7N7riB/HJT5caqMutpOByo2+1oOBxqOp0qSRNZay971wAAAAAAAAAAAIBr72qmjLjy +cpur1+vpxYsXOjntKM1dhZWm/LAqx/VlzDevqbA2V54nypKpsmSsLBnLNTW1GjXdvHlTj7efyPe+ +/pK8pGDYXPzEGKNWq6lyuaRMrvaPThUNj5Q2N5Vnr251bhxPXliT4/qyxlMUxRqNJhqNRppMJpIk +13XlOFx/AgAAAAAAAAAAAPwUBN/4Uay1mkwm6nZ7GgxHSnNXbtiQ45cl8+ogN89ipdFQ8fhUjlJV +yiXV6w3VGw1VqlWFYUmO48joLG629tJyb0k6X7BujFGz2VQQBBpPYvnOZ5oOT5VM+7JZLGvzWYX7 +ubbls08dGceV54Xyw7Icz1cUJTo9PVW73VYYhpe4gwAAAAAAAAAAAMCvA8E3fhRrrcbjsTqdjvqD +sVLrzYJvt/zq2d6S8ixRPOlrOjqRo0zlSkWNRmMWftdqCoJAjuOeJcZXpwW4tbPgu9FoaHl5Wf3B +UL6TaTo6VTwdKE9jyeayxvlae3cjY4yM8eT6ofygKscNFMWxOp2OwjBUo9G47N0DAAAAAAAAAAAA +rj2Cb/xo1lplWaY8PwupjXuh4vmb358rS6fK47E8x8ivVlWr1VQKQzmOI8cYyXzPJ78EpVJJjUZD +tVpVnuvK5plslirPU9k8k3Fevf3GzP5hjCMjR9ZKeZ7L5lZ5nl/2bgEAAAAAAAAAAADXHsE3fjRj +zCywdmZpr7VW31WpbfNceZYoSyOVPEclbxZ8B99o931Fqr3neb6Z7avv+SqXyyqFJXm+J3NWmW7z +THmeyTGOjNxvf7iz9THObN3mHwEAAAAAAAAAAAD8NKRu+NGMMXJdpwhvjfLvzKytzZVnmfIsles4 +KpdDlcoled756y+uVuh9fntcz1UYhrOW7K5bfN3aXLKvqdy2VvbssRzjyHXds+AcAAAAAAAAAAAA +wE9F8I0fbVbtPatcNub1Fd+zeddGMo6spDybBeH2LDS+mDVf3VDYysrm+exmNSsJf23l9iwgt8pl +jJHnebO1I/wGAAAAAAAAAAAAfjKCb/xoxsyCb9dxZjG1zWdVzfbbwu+zOdfOLPjOcqssz88Cc2kW +Ds9y71n2fUVnftvZbO7sbH+NcWSM85r55nZWFW7tuRbx/PoBAAAAAAAAAAAAbwLJG340Zz6r2hgZ +Mw+8v6vie/YPY2YvuyzLlF8Ivl95r8vezW+wdta2PUuz2bYbI2Ne17r85foYo6LVORXfAAAAAAAA +AAAAwE9H8I0f7WXlsjmLp/PvqPaW7Lzi2ziyVrOq6Sx7TfB9FVnZfNbufLYOZxXf3xXS27NZ4LJy +jJHrunKLOeEAAAAAAAAAAAAAfgqCb/xo5izAfdmy+zUV35rP+X4ZfM9u1yf4ttYqz2chdpblkpWM +HMl897xuq1mV+KxA/HylPBXfAAAAAAAAAAAAwE9F8I0fZR7aOo4jxxgZMwt2v7PduTHnqqOlLM++ +VvF9tUNga+3Z7WWArbPbLL9+XcX3N1udAwAAAAAAAAAAAPjpCL7xo32j4vuslfd33GNWGe24s8rp +LD+b8Z2/8luvGmvt2VzyV4X7r6v2fhmaG2PkOucr5QEAAAAAAAAAAAD8FCRv+NEuzPg29rurvc/d +R2cV39bmRfhdfF3maxHy1WmDbnN7Lqg3ct3Xtzgv7mvPZoJbO1s3l1bnAAAAAAAAAAAAwJviXfYG +4Pp6GXw7Z2H1y3be3/b9s4rv+YxvqyzPz7U611nhtJGZlUlfpdxbuT0/k9zIcVw537c03VrJzvZ1 +3urcMVx3AgAAAAAAAAAAALwJBN/40RxzFnrPq5ZtLqPZ/OtvMw+/c0l5ll8IvosI2UhXKvE+Y61V +mqazCnWjWdW2Y/Tavuz2ZbtzWSujswsG3O9XLQ4AAAAAAAAAAADgu1Fyih/NcZ2zGd9Gxsxal1v7 +3TO+jXFk5EhWyvPsrNX51Qu5Zy5WnM+qvXNlWTbbf8ectSv/Pg91NuNbVsYx8jyPGd8AAAAAAAAA +AADAG0Lyhh+taHVuHBnNWnmf1TZ/yx1m9zGOI6tZ8J3l2WvC8ktUbNYs4LbWKsuys5nkZhbgG0ev +rfjW2Yxvm0t2tgau687Wj3bnAAAAAAAAAAAAwE9G6oYf7eWMb3PWnTzXd/U5Nzqb8W1cWWuV51Z5 +9roq8cvcwQsfLrY6l2Tmbd6/V7fyefBtZ8G348p13cveQwAAAAAAAAAAAOBXgeAbP8qsxfc8+HYk +czbDWt+j1bk5a3We5bJXueJ7tsmFWVifvwy+jdGsz/n3rfg+a3VujFzPPat+Z8Y3AAAAAAAAAAAA +8FMRfONHM2Y+q/qs4luvqd42mrUGP2vvndtZiFzc58plwLNQ2+pl6H2h1bnjnM0s/x7sWSt4m8tI +cubV4gAAAAAAAAAAAAB+MoJv/Gjzam/HcaT5jO/vrPo+q/h2XFlriurp82G5ecVnV0Ux4/tse2eh +9/fbTiur/HzFt+uerRsAAAAAAAAAAACAn4rkDT/a+eDbSLMZ1rLfmnvPWoM7Mo4jKynPrbI8O7vf +/JuuZOR9oc25za0utG03r/s1ml0MYPP5jG/Jdd2iXTwAAAAAAAAAAACAn8a77A3A9eV5nsrlsgI/ +kDG58jSWzVJ9W/JtbS6bp8rTRI6x8jxXvufLcdxz33XVku95dfesSjsIArmeKxkpzzPlNpO12dn3 +Xdzw2VzvTHkaSTaX47hyHE+e58txHLmu+8M3BwAAAAAAAAAAAMA3UPGNH8UxjoIgULVaVRgGMsqV +pZHyLLlYwX2OtbnyNFKWTmWMFASBgrB0LgD+euNwq6vE8zyVSiUFgS8jqyxLZvt71q79G/PN7exi +gCydSsrluq78wJPreVR8AwAAAAAAAAAAAG8QFd/4UYxjFASharWaqtWyPGOKUDtPJsrSqRzHn7U1 +t5KsVZ7GSqKh0klXnmMVVquq16sqlcJzD6xXhN8/JAC33+i2biTpDQTMs4rvUKWwJM/z5Bgj2Uxp +MlYaj+S4gVw3KJ47zyKl0VDJpCtjU5VKocrlqoIgYL43AAAAAAAAAAAA8AYRfONHMcaoWq1ocXFR +7VZLYSmQUaZ02tOos6M8zxVU2vLLTeVnldHx6ESTzo4mvR2ttitaX13Xra0tNZvNN7BFZ4G3zYvb +y211ZBx3No/7J3AdV77vqVKtaGFhQYsLi/KdVNPOE9ksVam+qrC2rNlM8FzJpKdR55kmvRfyNdHq +6pI2Nzff0P4CAAAAAAAAAAAAmCP4xo9WqVQUBIHa7ZbKJV9GmZJpT+POc1k7C5u9Un3W7juZKhqd +atx7rsnptsorD3V7657u3Lmtdqv10zfGzmdqn80Rz1PNa8cd40oyMu5PDL692YzvaqWqhfaClpaX +FGWZJt0dJXEs47jyK23J5rJ5pnjS1bizo+HRF1pZrGttdUU3b95Uu0XwDQAAAAAAAAAAALxJBN/4 +UYwx8jxPxhgtLy/prYcPdHJyqsFE6k/GSgY7GsR9jU53ZPNENktkk75qQabW6qJu37qhhw/vn1V8 +N37i1lhlWaQ8jRSPe5oOjxSNTmaV3saRX26p0tpQubH2Rva71Wrp/oN76va6erHf1d5RV5PkRJMT +o3h8Oqs2t1Y2HsjNOmrVfG3dWNHD+7f0zjtvaWl5+bJ/fAAAAAAAAAAAAMCvCsE3fjRjjFzX1cry +iv74n/6T6vW6Pvr0S336+WMdnzzT4OixojiRMUaOIzXqFa0vL2hz46Hee+ctvfvOO9rauqlarf7T +NsRKWTJVOh1o1Hmq7t6nGh4/lnE8OY6ravumjPRGgm9Jardb+rvf/EbNRlN/+Y+PlH/4iXb3j9Q/ +PtFoPC2+r1IOtNCsanFtQW89vKP/9Pvf6cH9u1om+AYAAAAAAAAAAADeKIJv/GjGGBljtLi4oPff +f09ra6uSjHqdYyXjU+XTnuJpX57nyfU8NcNAt24s6t333tU7b7+le/fuanFxUcYYTafTH70dVlZ5 +GiuJhpoOjjQ62dbJ7mdy3VnwnWWxqou3fuxefuNPGvWG7t+/p42NdaVpokH/VGk0UB51NYkPZczs +PuVqWysLi7p796beeeuB/u437+vGjQ15Hr92AAAAAAAAAAAAwJtEAoefzPM8VSoVLS4u6u9+876q +lYqOjo80HA41Ho2K4LvVamljY0Mb6+taX19XuVwpQuI3xXE8BWFJ9VpNeZ4rz62MXhVf/xDmwnYa +xygIAllrdf/+fTmOowf376vT6ajX7xcXBFSrVS0tLWl5eVm3b91Ss9GQ67hvfJ8BAAAAAAAAAACA +v3UE3/jJPM9TtVpVqVRSrVbT22+/rSRJlCSxsiyT47hyXUe+78vzAwW+pyAI5Pv+G98W13UVhiWl +lYqSOFYcJzI/PfmWpCKwdpyzffE8PXz4QLdubSmOY8VxrDRNz1q7O/I8T57nyfd9BUGoMAjkuA7B +NwAAAAAAAAAAAPCGEXzjJ5tXOM8D4VqtJmut8ixXbnO5rlt8zy+zLa4cx5WMIxXP+eOf+1X3nO9P +uVxWuVyWtXZ2y62M83JfCbkBAAAAAAAAAACAnx/BN34WxphZdbM1v1jofdn7K0nGNd/4MwAAAAAA +AAAAAAA/L4Jv/Gz+FgLvr+8vAAAAAAAAAAAAgF+ec9kbAAAAAAAAAAAAAADAT0HwDQAAAAAAAAAA +AAC41gi+AQAAAAAAAAAAAADXGsE3AAAAAAAAAAAAAOBaI/gGAAAAAAAAAAAAAFxrBN8AAAAAAAAA +AAAAgGuN4BsAAAAAAAAAAAAAcK0RfAMAAAAAAAAAAAAArjWCbwAAAAAAAAAAAADAtUbwDQAAAAAA +AAAAAAC41gi+gW9hZSVZSZIx5rI3BwAAAAAAAAAAAMC3IPgGvqaIu1/m3gAAAAAAAAAAAACuMIJv +4HUo9gYAAAAAAAAAAACuNIJv4GvMuRsAAAAAAAAAAACAq4/gGwAAAAAAAAAAAABwrRF8AwAAAAAA +AAAAAACuNYJvAAAAAAAAAAAAAMC1RvANAAAAAAAAAAAAALjWCL4BAAAAAAAAAAAAANcawTeuDHPZ +GwAAAAAAAAAAAADgWiL4xhVjJBkZI1l72dsCAAAAAAAAAAAA4Dog+MaVQtU3AAAAAAAAAAAAgB+K +4BsAAAAAAAAAAAAAcK0RfAMAAAAAAAAAAAAArjWCbwAAAAAAAAAAAADAtUbwDQAAAAAAAAAAAAC4 +1gi+AQAAAAAAAAAAAADXGsE3AAAAAAAAAAAAAOBaI/gGAAAAAAAAAAAAAFxrBN8AAAAAAAAAAAAA +gGuN4BsAAAAAAAAAAAAAcK0RfAMAAAAAAAAAAAAArjWCbwAAAAAAAAAAAADAtUbwDQAAAAAAAAAA +AAC41gi+AQAAAAAAAAAAAADXGsE3AAAAAAAAAAAAAOBaI/gGAAAAAAAAAAAAAFxrBN+4Oswbehhz +9lhGkuzZn9of+CgXv9+YN7RxAAAAAAAAAAAAAN44gm9cLcboZcb8Q8Pqcw9z9r8fwxY3wm8AAAAA +AAAAAADgOiD4xpXhOI4815XjuLM/sFbW5t/z3lY2zyRZOa4rz/PkOOZlWG2trJ1H2q99qNk/bC4p +n22X58lxHMJvAAAAAAAAAAAA4AryLnsDAGlWTV0KQ9XrNVUrffUmibJkLJsl3yv8zrNUWTKRUaZK +uaSy2pKR4jiWtbNQ3Gax5LgyxpW+M8C2yrNEaTKVzR2Fvqdms6lSqSTH4VoRAAAAAAAAAAAA4Koh +xcOVYIxRqRSqXqupUinJc3JlyWQWVtvXVGlbqzyLlSYTGaWqlMtaaC+oUinLdRzJ5srzTHmWKM/T +b7Qwf+VDZomyZCplscLAV6PRULlcludxrQgAAAAAAAAAAABw1ZDi4Uowxqher2ttbVWHJz092z1W +NDxUXF9RGo/kBhU5jiczb4MuFa3Q8yxRMulp2t9X2RmruVTWarsp40h5lis1VTnZSKPTJwoqCwqq +bble+RvbkGeJ0nisdDpQGg1llMkPPFWqFbVaLVUrVSq+AQAAAAAAAAAAgCuI4BtXgmMctVotbW1t +ae/gWJ4+1aS/r3JjXcmkJy+sS34o91zwbWVl81R5Fiken2jSfaZSzdFCa1EP7t+S67oyxlF3mCjJ +BhocfqHqwm15QUWuV5J0sd15lkwVDY80HRwqjfpyjZ21X6/V1G63FQQBFd8AAAAAAAAAAADAFUSK +hyvBOLOK7/X1da2tLKle8eWbWDbpKx4dyXEDeaW6/LBa3CfPUmVppCweKh2fSnFPJbemtaWmHj58 +IM/z5Qe+nu8e6+B4qEHnqTw/VFBpzJ9Uxjiz1uc2VzzuaNrf06T3Qq6dqlGvanGhrXa7pXq9LmOM +zHfOBgcAAAAAAAAAAABwGQi+cSUYY1SpVGSt1c2bm3r33Xc0HI3Vmxj1Tz7XqPNMQbl9FlobSUZ5 +OlU86SuNBiq5E62tLOjunRu6e/eu7t69q2qtppXVVS18ta0PPvxc3W5Hcf+ZuvFQbliX45fluCXZ +PJbNYiXTgeLRibJpTzdvLOrOzd/qvXceanPzBoE3AAAAAAAAAAAAcIURfONKMMaoVCopDEJtbW3p +N++/L2ut/vofH+ngxWcaTVKF1UWF1YWzENoojceaDo+VRgNtbW7oxq2bevjgvh7cv6/79+9rfX1d +4/FEtVpNR4f7evzlRNPBM0WnO5ITyis15IUN5clYeTJRloyUJ2M5SrS6sKn/+7/8vf7uN+9pY2OD +4BsAAAAAAAAAAAC4wgi+cWU4jiM5UqPR1O3btyVJRkaB76rTG8o6oeSEsnYWQts8kMlLcpXozp3b +unv3rh4+fKD19TWVy2W5rqtKpaLbt7b0d7/5jdI0Va83UK8/VJRYWSeQNYEcOTK2JN9tK/CNKiVf +v//tb/TWw/taW1tXrVa77KUBAAAAAAAAAAAA8B2MtdZe9kYA502nU/X7ffV6PT1//kLPXzzX0fGp +er2heoORcpsrz61KYaBmvapmo6qNjQ3d2NjQysqKFhYW1G63lOe5sixTt9vT3t6e9vb3tb9/oIOD +Q3X7A43GkSbTSKUwUCn01WzUtbS4oOWlBd24cUObm5tqNhsKgkBBEFz2sgAAAAAAAAAAAAD4FgTf +uLKstRoMBhoMhjo+PtLu7q729/eVZZnyPFetVtPa2prW19fVbrfVbrcVhuG3Plae5Xr+4rlevHih +w8NDdTod9Xo91et11Wo1LS8v6+bNm7px44Y8z5PnebMqdAAAAAAAAAAAAABXGq3OcaUFQaBarVp8 +3m63Za2VtVZhGKrRaKjRaKhcLstzv/3lbIyRcYwajYastarX6xqPx5pOpwqCQKVSSbVaTa1WS57n +yXVc5noDAAAAAAAAAAAA1wQV37jS5iF3mqbK81xpmhZfcxxHrusWH40xrw2rszRTms0ey1qrLMtk +jCkex3Eced4sQCf4BgAAAAAAAAAAAK4Hgm8AAAAAAAAAAAAAwLXGAGMAAAAAAAAAAAAAwLVG8A0A +AAAAAAAAAAAAuNYIvgEAAAAAAAAAAAAA1xrBNwAAAAAAAAAAAADgWiP4BgAAAAAAAAAAAABcawTf +AAAAAAAAAAAAAIBrjeAbAAAAAAAAAAAAAHCtEXwDAAAAAAAAAAAAAK41gm8AAAAAAAAAAAAAwLVG +8A0AAAAAAAAAAAAAuNYIvgEAAAAAAAAAAAAA1xrBNwAAAAAAAAAAAADgWiP4BgAAAAAAAAAAAABc +awTfAAAAAAAAAAAAAIBrjeAbAAAAAAAAAAAAAHCtEXwDAAAAAAAAAAAAAK41gm8AAAAAAAAAAAAA +wLVG8A0AAAAAAAAAAAAAuNYIvgEAAAAAAAAAAAAA1xrBNwAAAAAAAAAAAADgWiP4BgAAAAAAAAAA +AABcawTfAAAAAAAAAAAAAIBrjeAbAAAAAAAAAAAAAHCtEXwDAAAAAAAAAAAAAK41gm8AAAAAAAAA +AAAAwLVG8A0AAAAAAAAAAAAAuNYIvgEAAAAAAAAAAAAA1xrBNwAAAAAAAAAAAADgWiP4BgAAAAAA +AAAAAABcawTfAAAAAAAAAAAAAIBrjeAbAAAAAAAAAAAAAHCteZe9AX8rkiRRkiSK41jWWmVZ9srv +c8zsWgTjGDmOI2OMPM+T67ryXE/GMTJmdsP3kySJoihSHMWSpNzm8jxPQRDI9325rsuaAgAAAAAA +AAAAANcYwfcvpNvtamdnRy9evFAcx5pOp8rzvPj6PHh1HEee58nzPJVKJZVKJTWbTbWaLTVbTZVK +JZVLZbmee9m7dG10u109fvxYOzs7yrJMSZKo2Wxqa2tLm5ubrCkAAAAAAAAAAABwzRF8/0K63a4+ +/fRTffTRRxqNRhoOh0rTVNI3Q2/f9xWGoRqNhhqNhlZXV7W1tSVJsk2rIAjkipD2++p2u/rkk0/0 +7//+70XV/fr6uqIoUr1WZ00BAAAAAAAAAACAa47g+xcyHo+1v7+vr776Sv1+X8PhUEmSFF+fB9+u +68p1Xfm+r0qlokqlona7rS+++ELLy8taW1vT6uqqVlZWtLS0pMXFxeL+eLXxeKy9vT09evRo1vI8 +jjUej7WxsaHhaKiwFF6ovgcAAAAAAAAAAABwvRB8/0Imk4lOTk60u7urXq+nfr9fVHxLL4Pr89Xf +ruvK8zyFYahSqaRKpaKtrS3dvn1bb7/9tn7zm9+o3WrLOObCY+CiyWSio6MjPX/+vAi+Pc/T6emp +ptOp0jSVtfayNxMAAAAAAAAAAADAj0Tw/QtJkkT9fl8nJyfq9XrqdDrKsky+78v3fTmOcyG4ttYq +z3MlSaIsy5TnuVzX1d7envb39zWdTlWtVrWysqJKpaJqpcqM6m+RJIl6vZ5OTk4URZHG47Hq9boG +g4HiOFaSJATfAAAAAAAAAAAAwDVG8H2JarWams2mWq2WarWaKpWKHMdRmqaK41j9fl/9fl/j8ViT +yUTj8Vj9fl87OztyHEfWWvX7fb311lt6++231Ww2L3uXAAAAAAAAAAAAAOAXR/B9iUqlklZXV3X7 +9m2trq5qdXVVrutqMploNBppZ2dHL1680NHRkbIs02AwUKfT0WAw0HQ61Xg81unpqay12ty8SfAN +AAAAAAAAAAAA4G8Swfcl8n1fzWZTN27cKGZ3h2FYhNrr6+s6ODjQ7u6udnd3tb+/X7RJHwwG2t3d +lbVWN2/e1LNnOwoCX7VaTeVy+ZXPl+e5sizTeDzWYDDQcDhUHMeK41g2t3Lc2VzxMAwVhuGshXq1 +qlKppMPDIx0fH2kymchaK2OMWq2WWq2WKpVKcZ+vs9aq1+up1+sVletRFKlcLp89fk2NRl31ev3C +ffIsV5ql6vf7GgwGmkwmxba6rivHceT7fjH/vFatqVqrKgiCNzLrfN5ifjKZqN/vazgcKkkSJXEi +SXJcR47jqFqtFutULpdVKpW+83EHg4GOjo50cnyiLM9krVUYhqpWq6pWq4rjWNPpVHEcK8sypWkq +3/fleZ6qlarqjboajUYx//1V+3p+/QaDQbF+8/nm8/XzPE+lUql4/nq9/sr1Gw6HOjo60vHRcbHN +pVKp+PmPxxONxyNNp9PZGiWJms2mms2mqtXqt742AAAAAAAAAAAAgDeF4PsSeZ6ner2u5eVl3bx5 +U/fu3VO9XleWZkrSRKPRSKPRSLu7u3r8+LEeP36sjz76SNPpVKPRSNIszH7y5Im++uorlctlra+v +vzL4ttYWLdRPT0+1s7Oj58+fazAYqN/vy1orz/MUBEHRfn1lZUUbGxtaWFjQ9vZjffDBBzo9PVWW +ZXIcR3fv3tW9e/e0vr6uVqv1ynAzz3IdHh7qyZMn2t/f18nJiQaDgZaWlrS0tKSNjQ3dunXrQvCd +Z7niZBYAv3jxQjs7Ozo5OVG/39doNCqC4POt4jc3N3XDuyHf9yXpJ4Xf87WajCc6PDos1mo0Gmk8 +HivPc/m+ryAItLa2ptXVVa2srGh5efm1wXev19NHH32kjz/+WFmWKcsy1et13bhxQ+vr6+p0Oup2 +u0VV//wigWq1quXlZd25c6cI/B3jvHKue57lStJEURRpd3dXz54909HRURHge54n3/dVrVYvrN/8 +z7++fr1eX5988ok++OCDYptbrZbu37+ve/fu6ejoSIeHh8UFGaPRSLdv39bdu3e1sbHxra8NAAAA +AAAAAAAA4E0h+L5EnuepXC6r3W5rZWVFN2/eVKvV+sb3HRwc6ObNm1pdXdVoNNLe3p5Go5EGg4Hi +ONaLFy+0vb2tdrutWrWmlZWV4r7z6t8kTdTr9dTtdvX06VN98cUX2t7eVqfTUa/XK8LcMAy1uLio +hYUF3bx5s6hy/vLLL/XBBx/o4OBAURTJGKPhcChjjFzHLQLzr8vyTIeHh/rss8+0vb2t/f19dbvd +osLddV0tLS1JuliR3u121el09Pnnn+uLL74o7jcYDOT7vnzfV6PR0MLCgpaWljSZTGSM0fLyclF9 +/UPNnz9NUx0fH+v4+Fg7Ozt69OiRtre3i+B4vlalUkmbm5va3NzU1tZWUU0dhqHCIHxlKN3v9/Xl +l1/qX//1X5WmaVEdfefOHd26dWtWWX18XIT80+lU1WpVtVpNN27c0HQ6lTFGCwsLWlhYUKVSkTFG +xphi+yeTibrdrrrdbrF+e3t76na76vV6CoJAvu+rXq9rYWFBi4uLGo/HkmaV7vMq9rnBoK9Hjx5d +2OalpSWNRiOlaard3d2iJX+v1yvCb9/3VS6XFYbhK1/XAAAAAAAAAAAAwJtC8H0NVCoVbWxsKM9z +bW9va2dnR9ZaDQYDRVGkbrer7e1tLS4uam1t7cJ94zguguQPPvhAn3zyiXZ2dnRwcKCjoyNFUaTp +dCprrVzXleu6qlQqKpfLWl5e1qNHj7S2tqadnZ0ieO71eppMJkWbbM/zVKvXtL6+fuG5rbXKskz7 ++/v65JNP9NVXX6nb7WoymajZbMp1XZXLZfmeL2utxuOxRqORnjx5oo8++kiff/659vf3dXBwcKEC +er6dQRAUbcYfP36szz//XPfv39c777yjt956qwiEv68oiopW8h9++KE+/vhj7ezs6Pj4WCcnJ4rj +WFEUFWvleZ6eP3+uZrOp9fV13bt3T/fv3y+qnV91IcB0OtXR0ZF2dnaUpqnSNNXBwUFxccB4PNZ4 +PFYURUXb8CAIFASBnj17pqdPn+rjjz/Wb3/7W/3+97/XjRs3igsB5vd99uyZPvzwQ3366affWL/p +dFq0Og/DsGg5//X1e+edd4r1m06nxUUA820+Pj7W6empHj16VLSyH41GRTv1hYWFoop9YWHhsn+F +AAAAAAAAAAAA8CtH8H0NVKuz2dXValVffvmlHj9+rMlkojRNNR6P1ev19PTpU62srOidd965cN84 +jtXr9bS7u6s///nP+p//83/qxYsXRbg6D6fnHMeRMUaO46jRaOjLL7/U6upq0eJ6MBjo8PBQvV5P +vu/LdV1Vq1Vtbm6+ctvTNNXe3p4+/fRTPX78WNPpVFmWaWtrq6iOdj23CL5PT0/1xRdf6B/+4R/0 +L//yL5pMJsW+zrd1vn3zYNbzPD1+/FgbGxs6PDxUtVrVwwcPJeeHrXMURUUb+D/96U/6x3/8R714 +8UJxHCtJkln1fJ5fWKv5bXFxUTs7O9rf31eSJFpZWX1l8D2ZTHR0dKTd3V0lSaI0TZXnuTzPk+M4 +xXOc/+i6rowx8n1fjx49UqPRUBRFWl9f1+LCYrEGk8mkCKP/8R//Uf/0T/9UrN98+1+1fo7j6MmT +J0Vlfblc1sOHD+W6s4r1efB9fpuNMXr27Jl83y+qwM9v99raWtHWPoqiy/4VAgAAAAAAAAAAwK8c +wfc1MK/OrdVqWlpa0ubmpo6Pj9XtdpXnuSaTSVGVPG9Zba2VJHU6HX366afFXOkXL15oOBwW1dJh +GCoMQxljZK1VkiQaDAYaDocajUY6PDzUeDxWtVpVtVot5l+Px2N1Oh09f/5ca2trxZxwaTYfOo5j +TSYTnZyc6OTkRJ1OR6PRSEmSFC2wl5eXtbS0pHK5rDRN9fz5c3388cf685//rK+++krHx8cKw1DV +alVhGCoIAoVhqCzLZK3VZDJRv9/XZDJRr9dTmqaq1+u6f/++dp7tqF6vq9FovHZ95/s0X6u//vWv ++uSTT4qW8vOW9OVyWaVSSY7jFGHvvOX86empnjx5ojRN1Wq1ziqxZ3PIz7cNn69xHMdFBfm8Kn2+ +r7VaTUEQFOF0r9dTv99XmqbF/s6D6mq1qvX1da0sr+jFixf6+OOP9Ze//EWPHj3S0dGRgiAo2um/ +av3mP+tut6s0TVWtVnXv3j09e/asWL88z7+xzfOw3vM8hWFYzF2fV8K32+3igg3P468ZAJdj/t+l +r/+7zV/+eW7zb/3+73osAPi1+K4OSee/5piXV5Uax7zye35ItyUAAF6FY3gAAIA354ee83/b+f7r +HusqIZG6RhzHUavV0sbGhnZ3d/X8+XPleV606O52u5pOp8X3W2t1dHSkv/71r/pf/+t/6fnz5xoM +BgqCQKurq9rY2NDi4qIWFxfluq6SJNF4PNajR4/05MkTjUYjjUYjTSYTWWtVLpeLKuEsyzQajbS/ +v6+9vT31+33lWV78Usyrp3d3d3V6eqrxeKw4juV5noIgULvd1o0bN7S2tqZKpaI4jvX48WP90z/9 +kz7++GPt7e0Vs6Q3NjaKkLzdbhcB7N7enr766iu9ePFCSZKo2+0W886/+OIL3bx5U2EYvnZd57Ox +j4+O9cEHH+gf/uEfdHBwoNFoVFS+t9vtYs08z9NwONRwONSTJ0+UZZniONbJyYmm06lWV1e1tbWl +SqWi9fX1C8H3q8yr+RuNhjY3N3Xr1i0tLCwoCAK5rqtPPvlEjx490snJiaIo0mQy0cHBgT777DOV +y2W5rqvFxUVtb2/rn//5n/XBBx9od3dXURRpYWFBGxsbWllZKX7W8/U7ODjQ48ePixbm3W5Xu7u7 +Raj+XetnrS1u5XJZrVZL9Xq9uEDg7t27Wl1dVb1eVxAEl/2rA+BvzPk3uKy1xZtkuc0v/P1VvIl2 +rsvG+fvzRhmAvzXzk9jzHx3HKT4vbvlZ5yDjyMrKOKa4kPb8/QEA+L6+fgwvzcLurx/Dz79+vivf +178GAACAbzp/rj///Hx34PNfm5/zzzO/63TOT/B9jTiOo3q9rrW1NS0uLqpUKslaW4Td8znO86ri +NE11eHiozz//XB999JEmk4lGo5HK5bIWFxd19+5dra+va319Xb7vazqdqt/vazweF9Xj0+lUk8mk +qPaez4d2HEeTyWQWGJ9Vn48nY/m+ryAINJ1OdXJyop2dHZ2enmo6nSrPc/m+r3q9roWFBa2trand +bivLMvX7fT19+lQffvhh0RI9z/MiPL5z547W19e1tramyWSiKIpUqVTU6/V0fHys8XiswWCg4+Nj +PXv2TNvb2yqXy1pbXfvONZ2v1Wg00v7Bvh49eqRPP/1USZJoOp2q1WppeXlZd+/e1d27d3Xv3j0F +QaBut6vT01MZYzQcDnV8fFxcfPD06VN99dVXarVaqlQq35i7/nVBEBQXNLz77rv6u7/7O62vr6tU +KhWzzKfTqdI01enpqbrdrk5OTvTll1+qXq9raWlJt27dKmZ7f/nll0V7+HK5rLW1Nd27d09ra2va +2NgoZn3X63X1+30dHBwU1d/Hx8d6/vy5Hj9+rFKppJWVlVdus+/7RceA5eVl3bx5U0tLS6rVamo0 +Grp9+7bW19fVbDa/18UHAPCmfCO8zq2yfNbpIstzZWmuLM+V57nSzCo/+1qeW0mzj9bmmt397E0z +O/+s+AQArj8z+4cpPj8Luo2R684+Oo6R67gyzsVROfNRPPM/c+UWY4bmJ8NX/UQYAHB1vPoYPpe1 +8+P2XFmWK8+t0iw/u2g1l82tbHEML1mbi2N4AAAAvTznP3dqPg+zXXd2vu8YI9dxLpzznx/xm5tc +jnVm1eDX6Jyf4Pua8X1fpVJpNhv7bAZzkiRyXVdxHBdtrPv9/jcqrvM8L1pPp2mq4XBYhJ7z9t3T +6VTD4bCYGz2v8HAcR57nFS2u52FmHMdFS/SdnR0tLCxoYWFBk8lEu7u7evz4sU5OTpSmqUqlktrt +tjY3N7W0tKRSWFIURep2u9rf29fh4aH6/X4xE/r8c3S7XWVZpl6vV4T6R0dHRev0+Ztv83bge3t7 +2tjYUJIm37me87UajUba3d1Vt9tVHMcyxqhUKmlxcVHvvfee/ut//a9aXl7W8vKyfN8vguL5mgVB +oJOTE52enqrf7+vJkydaWlp6begtSZVKRXfu3NEf/vAH3bt3T/fv39fi4qI8z5MxRvfv39doNCoq +7Y+OjjQcDrW3t6d2u621tTWtrKwUs9fn6zev0J9Xw+d5XmxzkiQ6OjpSHMcKgkBxHBevgX6/r93d +Xa2vryuOX71+jUZDN2/e1J07d3Tr1i1tbW3NfqZnr815dXmr1VK5XL7sXxsAfyMuVHqcD7yzXFmW +ahJlmkwTjaNU07NbUryJdvbm2fwx7NeqTs7eMLO8awbgV8LI6Oz/Z4G25DpGge8o9IxKgaswcOR7 +rlzXket6Zx9duY4r15tdEOu6rnInfxmCn52jSFf/KnAAwOX7tmP4PM+VppmmUarxNNEknh2/T6JM +SZpxDA8AAPAdXp7zz873jZF811HgGwWeUTl0FfiufM85K3h15Z2d58+zQdd1i2JYk88ujJdz9cNv +gu9rxBhzodLWdd2iYjnP8yL4nldQ7+7uam9vT51OR/1+X+VyuZjDnKZp0cb8+Pi4eKHOA3FJRfAq +qXiRzyu6gyBQkiSKokij0aiotDbGqFaraTwe6+DgQNvb2+p0OkrTVGEYamlpSbdv39bKyorCUqgk +TnR8fKydZztF8D2dTouZ3tKsbfp8HxzHKU5ohsOh4jgugm/HcZQkiXq9nvb394uQ/LvMg+/xeKzd +3V31ej2Nx2NVKhUFQaDl5WX99re/1f/4H/9DgR8oCGezt7MsUxLP9n88HivLMkVRpOPjY/X7fe3s +7Gh5eVlvvfXWa3+u1WpV9+/f13/7b/9Nq6urWl9fV71eL7ZvOBwWofTR0ZEkFes7n/G9tramg4MD +DQYDTSaTYv2MMcXFBYPB4ML6jUYjRVF0Yf3mr539/X11u10lSfyt2/zWW2/pv//3/67bt2/r3t17 +arfbxZVB81nf878YAeDndv4NrvlFYPlZZXeSpEqzTKPxVJ1BrN4gUn+UqD+OFSfpyyrwedXIWfX3 +vOr74kcA+PUwZ1eAO47kOLOT30poVC25qpc91SqeSoEr3/fkeS9vvu/LzVx5rqfcy4tjv+LcwTgy +7tU+EQYAXL4LIfVZ6P2NY/hJpO5gduuPEw1GiaZxevb+17lj+FnJ9zeO3S0V3wAA4G/R10JvxzUq ++Y4qoVGt5Khe8VQve/L92W2e/50/9z9fDDsvkp13fLvK4TfB96/AvO3AvPXgPCw9Pj5Wp9Mp2l4n +SSJjjMbjsU5PT5Vl2Tcea37feSj69XlJvu+r2Wwqz3P1+31lWabpdKrDw0M9fvy4aC8+D2Z3dnbU +7XaVpqmq1apWVlZ07949ra6uKgxDxUmsfr9fhN5pmipN0yKIHQwG2t/fV6fT+ca2zkP3+fzwedAR +x3HxZ/Pt/y7z9eh2u4qiSFmWyfd91Wo1tdttLS0taXlpuZhdOP9FztJMS0tLWl9f1+7url68eFFs +V7fbLdb+dVzXVaVSUavZUqPRULVaValUKta91WoVM7rn7e3n7dm73a6Oj491cHCgfr+vJEmUJEnx +ehgMBjo4OFCv1/vG886r9UejkeI4VpqmZyeXSbF+3zYby/d9tVotbW5uam1tTUvLS2o2m5f1KwAA +knRW3Z2dXciVKU5STaaJOv1InUGk7jBSdxCrP0o0niYaT2fVInmeKzv/ZpnOvQlnz9WH2Nf/NwUA +rhVz1t3JnF1k60kl3yj0HdXK8VkAPrvVyq6CwFcp9BX4gfzAn50U57Mg3FpbhN+5chlrrvSJMADg +arDWFqH3/Bh+GiU6HcTq9qfqDmJ1h7F6o1jjSapxlM6O4bPvewwvSRzHAwCAvzXOuS5v84vdpdA3 +KgeOauVU1dCoXpmd81fLnoLAVxjMil993y+C8PkYZM+bRcrGvhyBdhURfF8jL1s9za5szfO8aCdY +KpXk+35R0TsPc3u9XhFgzivDJRXtr1/1HPOQfB5qn//avI11rVbTixcvNJ1OFUWRjo6O9PjxYy0v +L2synWo8Huvw8FDPnz9Xv99XHMcKw1Crq6t68OCBVlZWFASBer1eMad7PjN8Xj09Dy/G4/Erq4az +LCuC8nlwO1+j+fbb15Tn5XmuyWSiXq+nwWBQtDkPw1CNRkPNZlOVSkWOO3tT8Ou/yNVqVUtLS2q3 +298IpeeP91OFYUnNZlO1Wu1CFbwkDYdDdTodHR0daTQaFes3D/273a4mk8n3Wr/5es3vn2WzE8lX +McYoCIIipJ9f7QMAl+FCpcjZfwfiJNV4GqvTj7S9O9T23kidQaLBePZmWZrOZ3zP5gLa2Ttls8fQ +y88vPI+u5sEcAPxYZh4LmNnnztnJsOfkKgdW5cCoVTVabRqttIwqpVBJKVCplCjMwuKC0QuPeTYT +zOa2uAocAICvu9Cx6azSe37hancY6cnuUNu7I532Yw0mqUbTVGmWK01nQXn+qmP4V1R3z/6V/xYB +AIC/NefeyZyf8ztGrrHy3VzlMFE5MFqsGa22jBYbriqlQNXyrNtzGIZnY2UuHlzNq7/n5/zS1Tvv +J/i+ZpIk0XQ6vVDNPL/yYt6y2lqrKIqK2dVpml54jHm4+V1twOePVyqV5LquGo2GarWaGo1GUdEx +Go2KOdHHx8fa3t7W1taWBoO++v2+jo+PdXx8XDxPpVLR6uqq7t27p+XlZYVhqDRNNR6P1ev1NJ1O +i306X70+r0R+FcdxitbrtVqtCIjnrcq/TyA7D6rPr9W84rter6tUKr3yF9c4RuVyWa3WrFJ7Hkqn +aarJZKLJZPJGgu8g8FWtVlWtVi/MdZ+H9oPBQN1uV9PpVFmWXdjW+YUS32f9JKler19cP9f51vvN +Xx/z1x0AXKZ5pcg8+J5ME3X7sfZPJnqyN9LnOwP1hqnGUaYoodchAHy72d+RgZcrdHO1q1ZpYmVy +qV5JlFYDZWeh9/wk2JiXx4zGmOIqcFeujHu1ToABAFfH14/hp9Es9D44PTuGfzZQZ5BoEmWaxhzD +AwAA/DSz4ynH2Nk5v5Nr3LTKUsmmjhq1RHmaqFRKLpzznzfvNpzbfNb2/Aoi+L5G5tXPJycnRSX3 +vDq5UqmoWq0qDMPie+fVv9ZaOY6jer2uarWqxcXFYi709+G67qzd9/KygiBQlmUaj8fa39/XkydP +NJ1O1el09OzZM+3s7Gh7e7uYlx3HsVzXVbVaVbPZ1OLiolZWVlSr1WYhff5yO+fhbLlcVq1WK+ZX +b2xsqFqtfq9trdVqWl5e1urqqu7evatq5fX3m8+Pmj///Bd3Pqv6u65WedX3zdvCv+ovhZ/ys/96 +ReP5Cu3z61cqlYr1W1tb08bGRjEz/HWq1aqWl5e1srKi+/fvq1arvZHtB4Cf27w9YpbN5gGe9iNt +7w70ZH+kF8dTDcaZpkmujC6HAPC95LlRLEfDaabDrpSkuZYbkbI0Vp69HJPzcjTSywtYz58IG2uK +PwcA4OtymxcX7XcGsbZ3h3qyN9KLo4kGo1TTOFea/fTnAQAAwIyVUZYbRXLUHedyzKzzzkqUKU+n +qqWlIn/6esY1P7c3xlzZTm8E39fIvMr65ORE3W5XcRwXlbfValX1er1oNXg+DJ0H35VKRUtLS3r4 +8KH++Mc/6ne/+933el5jjCqVisrlsrIs03A41NHRkT777DN5nqfJZCJjjNI01dOnT/XkyRMdHx+r +3+9rMpmo0WioXC4X87JXV1flOm5RTZzn+YVtDYJAjUZDq6ur+sMf/qC///u/1+rq6vfa1iAIVClX +VKlW1Gq1ValWvte6zp97/ks8f8Nu/qbdt3EcR57nfSP4nr8J+CaC71c9xvlw/etV3fP1W1lZ0e9/ +/3v9/d//vTY2Nr7Xc/m+r3K5rEqlooWFxe8dmAPAZZr/PTm7GChVmmXq9Kd6vDvSo+cD9UeZBuNU +aWZFnQgAfD9ZbpRbo2FklaSp+uNMWZLJt5kcG13oKDTv1mSMI9dxz/377ETYuMz6BgB8k83nI/0y +JensGH57d6TPd/rqj2fH8EnKMTwAAMCbZK2UZo6y3KpvreLEajRJZdNMgTIpT7/lnP/lbO95dubK +vXLn+wTf18A82BwOhzo8PNSTJ0+0v7+v8Xgsx3FUrVa1urpaVFJLswAzDMML7b7nbcM9z9PS0pLu +378v6buvxpiHCfMQOMsy1Wo1hWGopaWloiX2vGX5ixcv9MEHH2gwGOj09FTWWlUqFa2srGhtbU3t +drtoqy1JjjtrtT1vmT1vbz4P7mu1mm7evKmtra3vtZ0vK7A91WrV79WC2/O8C2s1X+/pdKrpdPqN +VvHnJUmiyWRy4fvmFyMEQfBGWoDHcazhYKjhcHihFfv8Z1wqlVQqlYqWkvP1S9NUlUpFN27c0L17 +937w+lWrFVqYA7g2cpsry3JNokzj8VSng1idfqzuINU0OZvnzTtmAPC9zcemprmRtY6yzFV/nKjr +T+XKufB36vmLRpPUlXFe/vu86vsqnQQDAC6ftfas2ttqGqUaTSJ1h7E6g1id4azSO0k5hgcAAPg5 +zM75jdLM0SR3ZOSoN45U96eyNrvQNXN+fj/vfjzP0Ywxcoxz5S52J/i+BtI0VRRFGg6H2t3d1ePH +j7W7u6vxeCzXdVWv17W5uakbN26o0WjIGFNUgc9ndOd5rjiONRqNNJ1O5XmeWq1W8SbVqxRVxVmu +LH/ZSnte/b24OKsI7nQ6iuNYURRpf39fkhRFkbrdrhzHUa1W0+bmpjY3N9VoNC48h+M4Rat23/fl +OE4ROg+HQ2VZpmq1qna7XVxR8irz6uqLswZnv3Sv4/t+sU/z8DhNU41GIw2HQ0VR9Or1yV/OUj8f +SnueV4TR8wr8n2IymajT7ajf7xczw+cXC5TLZVWr1eJihPn6RVGkwWCgLMtUqVReu37z6vGvdwkA +gOvCWqs0yzSeJjodROoPYw0ms5neWT4LbwAAP5y1RrlcJTIaR7m6w1ju/OJYo+KK7/nN83w5Tlqc +g1yVE18AwNVz/hi+O4jUHcQajFONp2fH8Je9gQAAAL9yuTWSPEWp1WiSqWOmkp3NmXGdl12P5x89 +z5t1dL7CF7sTfF8x59u1JklStBbvdrva29vTzs6Onj9/rtPTUyVJoiAI1G63dfv2bd28eVP1el3G +GJVKJdXrddVqtSJ8TZJE4/FYo9FIk8lESZKoXC5fqBY2xihLM2V5piiKiqrneRgahqHq9boqlYoW +FxfVbrd1fHysLMuKKu/xeFzMIzfGqNls6tatW7p58+Y3gm/PmwX0rVZL5XK5qCqfB/3j8VjT6bR4 +7jAMiwB3HtRmWVZsZ5IkRUDvuq5KpdJ3rrfjOCqVSmo0Ghdaxc/XajAYaDAYaDgcFlXWxhjlWa44 +idXv93V0dKTT01NNp9PiooNKpaJarXahuv27fuZZlilJkwst0vMsV5ql6vf72t/f1/HxcdFWPggC +VSoVtVottdtttdttlctlua5bXDjgum5xoUOe5SqVZ2F80XbyrANAnueaTCaKokhxHBch+PdZPwC4 +Cmxuz/4utRpPE/UGsXqjWKNpqijh7TIA+CmsjDI7a3s+ia36NpZrU7mO5LtWruvOjpO92bFymiZy +Xac4Vnccp5j7JV292V8AgMszO4bPNYlTdQeR+hzDAwAA/KLm5/xp7mgcWXXzSEapXMfKd2c5WxAE +8jxPQRDI9/3Zhe+5e6Hq+yrN+ib4vmLmYWcURTo9PdXp6ameP3+uJ0+eaHt7Wx9++KF6vV4RBFcq +Fa2vr+vhw4e6c+dOUcXdbDa1ubmp7e1tVSqVWdvBJJEkHR4e6qOPPlK73dbW1pbu3LmjRqMh3/Nl +HKPBcBb27u7uamdnR7u7u6rVaqrX61pbW9Pt27e1vLyshYUF3bx5U/1+X1mWFXPH55Ikkeu6ajQa +unXrlm7duqVms3lhf8ulkpaXl3Xz5k21Wi15njcLgZNEo9FI29vb+ud//meNRiPdvn1bW1tbZ5Uk +nuJ4Fjz3ej09efJET58+VZIkqtVqajQaun379muDZ2NMMYP88PBQ5XJZ0qy9+GAw0OHhoba3t/Xx +xx9rZWVFy8vL8n1f0+lU/X5fT58+1eeff66nT59qMBgUs9RXVla0urpatJ7/LlmWaTweq9PpqFqt +amFhobjoYDgc6vHjx/rggw/0+eefq9vtSpKq1apWVla0ubmpra0t3bp1S+12W77vy1qrOI5ljNHT +p0/1pz/9SVEU6fbt27p9+/aF9ZsH+9vb23r69Kmm02nxs759+7bu3Llz2b8SAPC9zKpFck2iVL1R +rPF0NtMbAPDmpGmucZYrMJlKfqRyYBUEQTE2yI99hUE4q/bOclnXFhf2AgBw3vzi1Xmr8/440WiS +KjnfVxMAAAC/iNxKcZpplOUKXKuSH6nkm+Kc3/d9xXFchODzIs6rOC6X4PuKyfJMcRxrPB5rb29P +T58+1SeffKIPP/xQX3zxhbrdrnq9XlFZUa/XdePGDb3zzju6e/du0dq81WzJ5larq6uqVCoyxiiO +Y8VxrOPjY3388ceSpD/+8Y9qtVpFJbBjnWKW+GeffaY///nP+uyzz7S2tqa1tTW9/fbbajabWl1d +LSrNO52Out2urLVF1bU0az/u+75arZZu3779yuC7VC5peXlFWZYVwW2e5xqPx0qSRDs7O/q3f/s3 +JUkix3G0srKiMAzlOq6iKFKn09GLFy/0f/7P/9Gf//xnJUmitbU13bhxQ77va319/TvX2xijer2u +Uqmk/f19VSoVSdJ4PNZkMtHR0ZGePHmiTz75RGmaFkH2YDDQycmJnj17pi+++ELPnz/XcDiUpGLm ++g8Nvk9PT9VutxXHsRzHKSroHz9+rI8++kiPHz9Wr9eTMUaVSqW4CGG+tgsLC8X6TadTxXGsZ8+e +FetnjNHa2lpRNR/HsTqdjvb39/WXv/xFf/7znzWZTLS2tqb19XW5rqu1tbXL/pUAgO/FWqsszzSN +Ug3Gyaw9IsE3ALwxVlKSWU2yTL4Slf1MYz+9EHyHQagkTeRlnrI8k2e9Yoarq6t3MgwAuHyzi1cz +DUaJxlGqLOUYHgAA4JdmrVWSWo3zTKGTq+xnKvt5cc4fhqGSJFGSJPI8r7iI8Spe7E7wfYnG47Ge +P3+uv/zlLzo6OtJXX32lIAiUJImm06n29/d1eHioZ8+e6enTpzo6OlKapvI8T41GQ5ubm7p7964e +PnyopaUlVSoVua47a3VeLqlpZwH1vCJ4MBio2+1qPB5rd3dX1s5mVB8fH2txcbFoUTCvNN/Z2dGj +R4+0u7sraTbbezweK01TGWO0sLCg27dv6/DwUE+fPpWkon12EATF7OmFhQUtLy+r2Wh+o3W253mq +1WZVzhsbG7p165aSJNFwONRgMFCn09Hjx4+LVurPnj2T7/sKgkCTyaTY1s8//1xfffVVMWOg1WoV +Ldpfp1wuF2H+5uambt26peFwqH6/r8FgoK+++kq+72tnZ0efffaZfN/XaDRSr9fTf/zHf2h/f1/j +8ViSVK/Xtby8rHv37unOnTtqt9uvff7JZKInT57oX/7lX/TkyRN9/PHH8n2/aLX+l7/8Rc+fPy9m +dtdqNS0vL+vBgwd6++23tbW1pcXFJa2vr+vWrVvF3PFut6tut6vt7W3leV7MiJ+v33Q61enpqTqd +TrF+kooq/SiKlGXZZf+aAMBr5XY2IsLmVkmWaxqlStJM+RU88AKA6yzLrJI0V2RSTSOj6TQtxuUk +SaIknY1qml/5PR/hM/872rhXo+0ZAODyvTyGz5WkmaZxqiTJlOccwwMAAPzSrDTrnpnmiuJUUZQp +imxxzj8/75+f82d5Jjd3L1zsbq29Eu3OCb4v0byN9cnJiSqVShFcn59bPZlMNBqNirnc89BybW1N +v/vd7/Sf//N/1ltvvaVms1mE3vM507VaTetr63rw4IGOj4/1+PFjRVGkyWSi4+NjjUYjHR4e6sMP +PywqxY0xmk6nxYztwWCgKIq0uLioNE2LbXcdV4uLi7p7965evHjxjcrmUqmkWq2mpaUlLS0tqdls +qlKtyPf8C9/neZ5KpZJarZZu3bql9957T1mW6enTp+p2u+r3+0qSRL1er2h77rpusU7z/RkMBur3 ++2o2m4qi6Af9HIIgUK1W08rySrE/z549K+Z8b29v6/T0tAjyHccpftHnwXuapvJ9X+VyWevr63rn +nXf08OFDLS4uvvb5R6ORHj9+rNPTU5VKpWLWeZqmSpJEp6enRRt513VVLpe1tram9957T7/5zW+0 +vLysRqOuW7du6d1331Wapnry5Ik6nY6Gw6HSdDYn/MmTJ/rXf/3Xb6zfvG37YDBQrVb7wesHAFdF +nltlaa4kzZVlOW+aAcAblp/NYo2VKYpzTae2OHcowu8kUZqmReh9/kLU+ZXgV+FEGABw+eZvlM6P +4dMs5+JVAACAy2Bn760maa44SRVFVtPpLEOKoqg41z8ffhcXul+xi90Jvi9Rr9dTv9/XixcvvtEO +wBgjx3HkOE7R1nweZtfrdd25c0e///3v9f/8f/8fNZoNVatVOY5T3H/+/csry3r48KEmk0lR9dvt +djWdTjUYDLS/v19U9c6Db2l28uE4ThFMO45ThLKu68o4ppgnvrGxoXq9rjAMi9De9301Gg2tra1p +cXFRjUbjG9XekuQ4TtF6e2trS7/5zW+UJImiKCoqnPv9vk5PT2dXkWTZhYB/vm6e5xXP6XmeKpWK +wjC8sCavYowpvn95ZVZF3e/3ZYwp5m53Oh0dHh5eWJv5m3jz561UKmo2m1pcXNSdO3f04MED3bp1 +S77v63UGg4HG47EODg6Kypj588z3zfM8lctl1Wo1tVot3blzR2+//bbu37+vUqmkIAh08+ZNvf/+ ++0rTWeVNr9crKuU7nc5r18913eLii0qloiAIruR8BgD4uvnfyVk+C7zTNFOWW/GeGQC8Wbm1splV +ajPFcarIyy5c+T0/AT5/8zzanQMAXm3+hmmW58rSWbU3x/AAAAC/PCspy61MlitOZuf8cewW5/tx +HCtN0+Jc//zF7vNzfkfOlaj6Jvi+As6H3q7rynGcorI7CAJVKhXV63U1m03duHFDm5ubevDggd55 +5x01mo1ZGO28+k2kRr2he/fuFX34G42Gnj9/rpOTE3U6naLidz6XW5pVa389aH3rrbf08OFDbW1t +FYH3/PEWFha0uLioxcVFTSYTDYdDhWFYtPxeWVmR7wffuQauM5snPQ/NK5WKFhYWdHx8rNPTU41G +I0VRVLQUt9YWazRrlz67IGBra0tvv/22Hj58qNXVVYVh+L1/Do16Qw8ePCgqtyuVip49e1bMME/T +VHEcX7goYF4FvrKyotu3b+v+/fv67W9/q6WlpSJMfh3f91WtVos292maKk3T4nVRq9VUrVa1vLys +ra0t3b17V3/4wx+K/ZsH2SsrK3rvvfcUBIHK5bLa7XaxfvPK/fPr53lesX71el21Wk2bm5vF+q2t +rb3yYgUAuIrOX2E4P+CaHbIBAN6Y+YVGNlWWJkqS7MLxa5ZlRbX3+au/AQD4NrP/VuTK7aza23IM +DwAAcAlsMYYmy1IlaaokyYtz/q+f7+e5LS52Lx7hCoTeEsH3pTvf7m9e5R0EQRH+VioVLS8vF/Ob +3333Xb3//vtaX19Xo9FQvVaXccy3vpiazabKlbI2NjbUarW0urqqzz//XF988YW2t7fV6/WKbZhX +fs+D34WFBa2trenmzZt699139d577+nmzZtqNptF8B34gRYWFrS0tKR2uy1jjKIoUhiGWllZ0f37 +988C2u8Ovh3X0fr6uhYXF7W0tKTFxUVtbGzos88+06NHj3R4eKh+v19cTTKfdR6GoarVqlZXV7W+ +vq779+/r3Xff1VtvvaWlpaUfFHw3m029/fbbunnzpmq1mmq1mtrttr788ktlWabxeFxUTc8r6tvt +ttbX1/XWW2/pD3/4g/74xz8WFwL4vv+9fsl93y9aw8/b2zuOcyH4XllZ0YMHD/THP/5R/+W//Bet +rKxoeXm5qJY3xmhtbU0LCwtaWV65sH5ffPGFDg4O1Ov1lOd58cak7/sKw1CVSkVra2taX1/XvXv3 +9N577+mdd97R0tISwTeA6+Xs6sIiBL/s7QGAX5miKk/ZWYuztKj0vjDr6xVtzwAAOO/lRasXPwIA +AOAS2HNdNbNMaZoojk2RJ83D7zybB9/ZlT3fJ/j+hSwsLOidd97ReDzWYDDQcDgsgua5eYA5b589 +rzpeXFzUysqK1tbWdPfuXd2+fVvNZlO+58v1vrui2PVclb2yAj/Q5uamHMdRtVpVu93WjRs31O/3 +NRwOFUVR8eZUpVIpKoZXVla0urqqW7du6caNG2q320UY6jiO5EjLy8t6//33NZ1O1e121ev1tLS0 +pPfee09vvfWW1tbWXhtAG2OKCveVlRUlSaJSqaR6va6VlRUdHR1pMBhoNBoVb6TN12heDb2ysqLN +zU3dunVLy8srqtWqcl1XCwsLev/999Xr9RTHsaIo0s2bN/X2229rYWGhmNs9X6swDHXr1i3lea56 +vV7s/3g8LlrGz39GKysrWllZ0a1bt4rQfH5BwPe9sqVSqejGjRu6ffv2N94olFRU089noN++fVvl +crloQT83X79sOdPd5K7CMCzmrM/Xb/66m+/D/DU2D9Ln27G8vKxarSbP84r1Ozo6Kv6Cu3Hjht5+ ++2212+2iPToAXKbiQMvOPxcF3wDwJlnJWsnaXJnNv9HS/NsC7/kx7VU8GQYAXK75cfvs2P3c5wAA +AP8ve+/ZJsdxpWnfkT6zvGsLR4AgKZLSuN2ZWTO7e1370/edlWalkaMkkiBh25vq8pU2It4PkZXd +AEESlAU1eWt6QIJdVRkms+K459T8WdHl/5RWKLmx+UVViFrZ/a8JeL9t9n4d+P4zsbuzy3//7/+d +hw8fkqYpWZZVvZyBKkj6auX3JjjZaDRoNBp0Oh1arRau42LZ1ht/vmVb9Pt9PM9UaN9/5z7zxbwK +BN90UG2qmTfVwJvP3fTpvildALC1tcU///M/c/fu3Uo6fVOpPhwOabfa36nyOgxDdnd2aTab7Ozs +8NFHH7Fer0nTlDzPq+vcyI1v5OA3suPtdtvMketi2za7O7v8r//1v3jw4EHlmGu32+zt7bG7s4vn +e7jOdS/ujWy47/vs7u7y0UcfsVgsKkmHzWdvkgg2/b37/T5RFJn/9h3WptFo8O677/Iv//Iv1Tg2 +vRCBKhGh3W5X/dK/SUY9DEO2t7arSvgPP/yQ9XpdrfXXzV8URTSbzes9Vs7f9vY2//N//s8qML9J +CNjf32dra8vsU7t+lNTU1PzluK7wLv+nKSUS1R/83jU1NTU11+gb0mcbWbObAe+vkzd/24zgmpqa +mpq3B10pNoHJXK2/M2pqampqampq/iJojVYaxSb4zUv2/k273/z623luq6NVfyb6gz79Qf8v9vlC +CFqtFq1Wi93d3T/qe/d6PXq9Hh999NEf5f1838f3/T/afG3m/u///u/feK663S7dbvePOk9fRxiG +3Llzh//8n/4zoy1TuR5F0e/9fp7n/VH322Z933T+ampqav5SaK2v+wK+pQevmpqamu81lSytMpLn +N6q9XzWC31bJs5qampqatwddyjTdTGCtqampqampqan5S7A5jylT9a0USvFadbe3PfhdB75rampq +ampq/rqoHGg1NTU1NX9MTDFe+Yz9BklzXWaJb/69pqampqbmW9FQKzbV1NTU1NTU1Pzl0FWLszIA +Xga+Xw14v+3J7nXgu6ampqampuavjrfz2FVTU1Pz/ce0YX3Z6H2d3JnS6q01gmtqampq3jJ0nbZa +U1NTU1NTU/OXp0x01xql9I0g+OsT3t9W6sB3TU1NTU1NzV8HGrSqXWY1NTU1fzJM1Nv842syvN92 +47empqam5u3jurc3dYvvmpqampqampq3gfJ89rrA9+t8AG+bL6AOfNfU1NTU1NR873npcKV17TSr +qamp+RNy8/H6OgP3df/+NhnBNTU1NTV/ea6/F3TVSkPrOom1pqampqampuZt4PtswteB75qaPyOu +69LpdOj3+xRFQZ7n9Pt9ms0mrudi2zZCiL/0ZdbU1NR8L3nJTfZ9Pp3V1NTUfA+5KX/+0t8r/dI/ +C7s+69bU1NTUvMwmb7WmpqampqampuYvjAbEVyXNvy8y51AHvmtq/qzs7OzwP/7H/6DT6SClRErJ +1tYWH3/8MaPRFo1GhOPUt2VNTU1NTU1NTc3byI2qPPhWY1ept9sYrqmpqampqampqampqampKdHG +6tebf9n89SvB77edOsJWU/NnZHd3l//9v/83/+W//Fe0Vmit8TyPMAwJggDbsrFs6y99mTU1NTU1 +NTU1NTVf4SUT91uyvb8vBnFNTU1NTU1NTU1NTU1NzX949Kttzb7l199im78OfNfU/BnxPA/P8+j1 +en/pS6mpqampqampqan5vXidefs2G701NTU1NTU1NTU1NTU1NTVvyverwvtV6tLSmpqampqampqa +mpqampqampqampqampqampqamprvNXXgu6ampqampqampqampqampqampqampqampqampqam4vtY +9V0Hvmtqampqampqampqamq+M68awN9Hg7impqampqampqampqampuYGr5j2Wuvvlb1fB75rampq +ampqampqampqvhMbm/d1BvD3ySCuqampqampqampqampqal5la/a9Rtb/223+evAd01NTU1NTU1N +TU1NTc0fxPfFAK6pqampqampqampqampqflmNK+3778PNn8d+K6pqampqampqampqampqampqamp +qampqampqamp+V5TB75rampqampqampqampqampqampqampqampqampqar7X1IHvmpqampqampqa +mpqampqampqampqampqampqamprvNXXgu6ampqampqampqampqampqampqampqampqampqbme43z +l76AbyPPc5IkIUkSptMps9mMLMsAsG2b4WBIf9AnDENcx8V27L/0JX+vkYXk9OyM8/Mz4jgGwLIs +ut0unU6HRqOB7/v4vv+XvtSamu9EmqbM53MWiwVxHBPHMXmeAyCEYDAYMBwOaTQaOI6DZdV5QTU1 +Nd+dwLdpBA6+Z2NbYFmCLJckaUGaSQqpyQuN0iDK12g0aI3nWvieTeDZuI6N61oIYX5LSsU6KVjF +OVJqNGAJqt+3bQvHFliWQCmN1po8V+RSkeUFea7ICoXWAKJ6X4RAIMqLEd95vH8OhDBjdWxBI3SJ +QgfHstBotFIkaUGSFaS5oigUeaHN+KqxvX5cloAodGgETjV3lvXK72rzh9IKrTWF1OVnKJKsIMsk +Smkzd5s5FALbMj+OYxEFDmFgl9f89Wht1i1JFcs4Y53Icq3+0LXR5bprrHI6XMci8Gw818a2BY5t +IQTV7yllfsx+VeSFpChe3kPmwv4Y11dTU1NTU1NT8/bx+jOoQIM5g2aSJM3LM6g5N/1pztYaz7EI +fIfAs3Acc3YrpCJNJWkuyQtFlpdnvrfibKYJPJtWw6MROtWZOssL4tT8SGnmTGlK2+Qvfc3fMI7I +pRG5r4xDEqd5OQ7KcfCWjuPPiS7/z/yJ0FiAsMB3bXzPxrUtbMcyNgiAMLbQTfsjywvyQiPly/bd +X4f9oRGAZZlnDJRD0xqpzF4CUW6379M4zZq7jqAZuTRCB0sY30MuJetEso5zpNIoZZ6tzcijETml +nSooCsUqzlnHGVKBVLqcrbf5OfGHzdnGBhfCVKjatsD3LALPecnPoze+CaVRSiGlJpeKopBkhfke +kpVv4q/lXvnuc+naglbDpdVwsYQFaAqpSFLJOi2MT0fqaz9OzV8tb33gO01TptMpl5eXPH78mMeP +HxPHMVprgiDggw8+4MMPP2Q0HBE1ojrw/QdSyIKnT5/ws5/9jOl0itYax3F45513uH//Pru7u/R6 +vTrwXfO9I45jjo6OeP78OZeXl1xeXrJer7EsE1j68MMP+fjjj9nf30drXe/xmpqa34tG4LI7DBl2 +A1wHPEcwW6SMpzFX84RVYoKmGz8AbAKeEs9x6bUc+i2PVsOn1XCxy2BpmkmOLlacXEiSrDDfzxZ0 +mx6Dtk8UuAS+g+NYFIVCFpJlnLFYZ8yXkqUsSIsM44+zqh8s2zjo9Nsb/LaEMZ5D32ZnELA/auC7 +5jqLQnI5WTGeKWbLgkVRoKREWBZCOKXz0XpN8FtjWYJu02V3ENKMXDxX4NrGMNqsi/nTGNyFVMRJ +wSrOWK4zJnNFkabGyYkw3hwhEMLGsi0816YR2GwPAnYGEb5rUb1heQ0G49SQyjg6LqcJh2eS1TpD +COsVQ/+7YoLpaI1GI4TGFtDwLbotm27TJ/QdwsAtjWmN0posk+SFJE5yVnHBci1ZxZI8y5EKEBZC +2K9c39u5f2pqampqampqvjsaSwgcG3MG7fvsbTUI3NKJXigupzGXU8liJVnIgkxKhOWYM+EfLeBg +HPm+ZzPoeAw6HlFgEwUucZJzNUu4mqcs15qikBSKtyCp1VxzI3S4vRWxP2pUfzdfJZyP14xnknWq +KOQmqdJ6C8+S5pqjwGF/q8HtretxLFYJ51drLqcF69QEZ68TjN+2cfzZp61KUEZLhNAIS2ADzcCm +23JoRS5h4BIGDqJMypZSkeWSNJMs1inzpQmArmSBkkVpuzrmz+/zPJd2mSXAtoy/QKARGDssLzCJ +FAjQxk78foz12u70XIetns/t7QaObRLCV4nk5HLJicxIc02uyt/r+9zebprfE7BOCo7PFxznCWmx +Scgube3vzVy86Xy9PG+WpbEsCFyLbtNl0Par+8R1LJTWaAV5XpAVijTLWa0LlrFiFSuWhSSX8iu2 ++l/PnH3bfOoyjmWx1fN5Z6+F65h9k2QF51cx55OYVVyQaEglCFEHv/+aeWsD3xtn42q14vTklCdP +n/DJJ5/w6aefslqtcByHZrNJGIbcvn2bTqdDoIK/9GV/75FScnh4yM9//nNOT08BcF2XOI7xfZ8g +CIii6C99mTU135k8z1ksFpyfn/Ps2TOePXtWJXdorSmKgkajged5dXJHTU3N700Y2Gz3Q+7uNgk9 +i9CHs8slQqWksSRLC7SSqBsVIQKFhSLyPbY7LntbDfrtkF7XB63JspzZKmW1Elw5EksrbMcm8Bx2 +y6Bqu6ym8FybvJBkuWK+TBnP1lx5mkuRU+SSpKw6l0pg2Q7CcrEsGywbIWzKtPy/9DS+hG2B51o0 +Q4vdQcB7t9s0QwdQpGnBc1ciVEKRSlZqTZHmWLaH5bhYllM6SG4kRmqNRmEJi07D4fYopN/1iXyH +wH85OK20OZPmhXEmLtcZ06XDbG7hWgpdZKySnDyXZIUug9822C6u0ES+w3bf4+HtFq3Q4aaBexOl +TXZ7LjWurRlPFiiZISwbsXHwXHs93hBjQAsUoLCFLvekQ7/tsd0P2epFNCOXZuRh20YtQKpN9VDO +YpUxX2bMloLxVKFlTpwVFEogVVFen2uMai2+2+XV1NTU1NTU1LyVaECXZ1BBMxTsDALev92mGZrz +cppKXngKoWJUJlnrlCLLXjqDaqzfM3Hx+jqMv0LhOQ69ps2tYUC35dNt+cwWCY7IyTNFmkhQBUpp +rCo4CH+Jc/0miBP6FvujiI/ud8z5W8PppY3KE5bLgkwXIAu0KgOa9ua8/nYcKK/HIdgbBnx4v1Nu +Dc3Z2EYVKctFQYZEqwIlhZl7y35rE4r/9HOmQCu0kggUAoVrWwSeIPBttnrGBum1ApoNj2bkmopg +BIWUlRrA1czmKhBMF+AuFEILCqUoZIbC+l7P8+aethxo+DbN0MGywBZGsW0V5yySArSF1BawCV6+ +zeMsbVutUKrAtSyGbZeHt5p4pZLd1SIhS2PG45xCa3Ktce3N77XwXAvXFkzmMcl6ztl5jCgUqgCp +LCzbAcsD/jqC35VPQKvyXpH4rk3oWXQaLtuDkJ1BSKu8TzzXKQPfiiQ3fqVVbPxE84XgaqZBFaAl +hZQU0tjqluUC1vfyXvkOs2m+Y5R5Ftu4DDsu799pE/g2aM1ynWMhWccxRa5Jc1X6hd72e6vmD+Gt +DHxvglBSSo6OjvjZv/+MX/ziFxwcHHBycoJt2/R6PbrdLmEY0mw2iaIIx3krh/O9QkrJZDLh6OiI +w8NDADzPY2tri3feeecleeiamu8TQRCws7MDwGKx4MWLF6xWq+rH8zyKomA8HvPDH/6Qbrdbf/nV +1NR8ZzzXptP02OoFNAOHRmiTpwmnVoLOp8hUUSQSWWYt27YgcCD0YLfb4N3bLe7fGhCFLo3Q5fJq +xmQ84+xswmoeI4qUYavB3nabna0OvZZPrxUQBjau62BbAqkUSmrizBhD82XM8emY43M4OZtwej5l +Oo9x/AaO38R2AyzHw8YvK5ZvPvuuA7RVofJLQdub1b4v/90fjnH2uY5FK4BR22G777M3imhGHmhN +mufk2YoiESxnK07XZywv57hhGy/sYnshtuNj3ZD60lqhVI62BL6j6DZttro+rYZPFHgvjXFjj0ql +kEqTZQXrVLKOM8bTFVezFkenVxwcXXB2OaXQDkrbOMpH2QG2UjT9Djv9gE4rvH7DV0eqIc0Lsqzg +arrA1gl5PDNr44bGYLUsU8HzprOnFFpLbKFwLUnkw63tDrd2Omz1IrrtkE7Lw3cdfM/CEqbiW2qQ +haKQkiSVxFnBOs44OZ9xcjHl7HLJ2XjF1TzD8QJsN8SyPeN84qt7R7803Osq92q3VP/4urG9bv+9 ++f55dU+++T796ud++2v/MuP909x7NTU1NTU1/3ExAU+FYwlavs2wZbHd89gbRjQbHmhI85wiX5Mn +gvU85jy9ZD1d4gYt3KBjznC2i7DftIXaq2cAE3xXMjfVrgoCx6cTaYZdl61+A8cquLzMEfkUnefk +SUGhbGw3wHb88pBgvfYzXs/vf464eR7ZBB8cK6TTdNkbNqr/lmcxByJGJZcUiSKLNQoXyw1ABNeq +VAiuE0a/2+f/Iee0l/6rkmiVY4vgK+Mo8oQjK0allxSJJo81UjtoN0C4IcbusG689x/bpvr9z8h/ +krNjWcWslUQVGVrlhJ4mdKHfCdnbbrO31aXTCui0ApqBi19K95trEki1adVl5LBXccFsGXM1WzOZ +rTg4XXB8NmOVahwvBDe83i/im/bL6xO7vz14+ub775s/4/pXNBqlMpTM8TyX/WHE/f0Ovu8SeDbz +ZcKzwwueHsxIpY2UDpIyeGn9Yev1JjbI6++P69e8fu+UFctoZJEi8wSdSyK/y6gXEHouvm/jWIrn +dkoRjykyQZ5bELSIAtjqGxUyz3FwRI7HimxxSJpaZh7wcbwIxxdgOYB941re/F74U9mdv18gXqGV +wqLAtRSeI9kdBNze6bAzatFrBXTbAaFn43mmNZvWpuVdUZgWb2lu1ADjRHJ2Oef0YsrZ5YKzqzVn +V2uE7eN4AZbtm4T1P4mf53V75035rrby6z5XVwk3Mk9RRYJyPSJ/yM4gIgpdAKbzNacXCkcu0UWB +zgVKCpPEb78qof/dxvTd99Wfy59W89ZGiqWUpGnKyckJP//5z/nXf/1XFosFy+WSdrttenq7Lr7v +02w2aUQNrDc+UNZ8HUopJpMJx8fHVeDb931u377NdDplvV7Xge+a7yVRGLG7u8ugP+DZs2fYts1q +teLi4oLLy0vyPGc2m7FcLul0Ovzwhz8Efl9p2Zqamv+o+I5Fu+Gy3QtpNVzaDZfF7IrAiiGfojJF +HisKbWFZNsJ18FxB27fY7Qoe3m7x4cMRdtlvOl7NWC6mnJ0es0o1olCMOhE/erfDD97dw/ddQt/F +tkzA2pj8BqWMXNp6nfK0A/0oheSCi+Mr1tMxXqOPpySeVoDAstxSFnxjuN88pG+yuOErTrOyN/T1 +8/KPUzW+UeRwbU07shh1HXb6AXujBu2Gj6m2yShin3wtODteoeIzlpMLwmILhMATGNlz7VYGqdYK +LQu0pQgcTa9pqg967Yhm4/XqQZthq7JaJS8KFsuYxWLFJ5+lxNMjTo8uUNIhUx4OIcqLEMqiHQi2 ++xHDXuMbxxonGUmScXgqsFRCGk/xdAfXctDCuiFH/0azVwb4C1y7IHAk7VBwfy/kHz7aYmfYJgxc +fM+t+qndXHJdOnq0Bqk1eZbz7MDhRavgM7FmPkvI1lO0boNwKofTdXXRy3Lx5b+8PKPlfrneO68a +vq/Zf99tA33l816WoPy6ffr6z/3m1940jvXNF/zJx/unuPdqampqamr+46LLitUCV9i0Q4tRy2Kn +Z6TOzRkU0qygSGbka8HlaYJOL1lPxwStLaPWY1kIy8K4Xd/MgQ83W+2Y61AyRxUpKEHgSDoRjNou +e8MIXaw5cFPsYobOCookp9C+kbm1zbneyLjy+jP863jTY8Tm/V4932iNVgWyyHEtTbfpsr/VrF62 +XEwIrBidXlUBYyVCXCHKik4TpCjf6qvX/MrnXs+bufhvPqd9+7n05t9txuEISbfpcuvGONbLKYGd +oKpxgCTAQZiEB+FwHff+mjPeH/msWi7C16zXdSbnm33Gd0NvbA+ZIwsTfHI9QdsX7A8ifvigyw8/ +uE3guwS+h+tY1zbIS5epqxZQSsFytebyas75eIouVpyfrigS45MWdqnuVVaM66/b41q/duF8mAsA +AIAASURBVIjfplal3/SeecPP0GVVqpIFKk/wbLg18vnHDwc0GyGtRsjZ5QyVTTk9WiFxSKSH0h6m +g5f1By3V9bV8Vzvm+tdfa3eUKghaK2SRUaQrVKGIXM1216fZ8AkCHy0zAjtFxpcUmUMhPVAuDU+z +0w+IAo/Ad5HZEk8syeZHpLlDWngo29jRluNjIcqEoq/xV3x1EW88p77DHHynuf3uwW/T077AEjme +JWm4BXe2Bvz9D4bcvz0gDDzTjkwIk8T/kq2+8ZUY/4RSmsMTnxdHisduQrzOOEqmCCcqk0Psr7HV +v2HNy+F8va35x7fVr+dyM9g3eP7dqPSWeUKRLtC+R8PT7I0aNCPjNwpceBZqLLVCFBItbZQ0PejR +9lc+6uv3k3jN737dNf9l/Wk1b2nguygKzs7OOD4+5tNPP+XFixdcXV2hlMJxHHq9Hu+88w4ff/wx +t2/fptEwQe86QPWHY1kWrVaL4XBImqaAkTrv9/u0Wi2CIKgr62u+l1i2RRiEeJ7HnTt3+PDDD0nT +lDzPuby8ZLlccnx8TKPR4P79+zx+/Jhut0u73a5lz2tqat4YIcCxBY4r8FwL33NwbBAUaJkic4nM +JQgb23KJXMHusM3dvTb37wzZ6jcJXIvZYsV8seTF4SmHx5dcXM3pdjqM9rvcvzVgb6tFr+UTJwmz +yYLVes1iFZMmGZ7v43oenVaDfrdFp+mxO2pjUTCZzDg6chlfKiBFZisKy0bYnjG8LAe0AFRlwG5+ +NpHQl/JTy0rqjSElLKvqqS3Kv//9KKOuWuE6Dt2mx1YvpNv0iAIH3zNSiJbQdNsRu1s9XhyGuJZE +pgsKP0JmLaTjYzseOIqqDzfX4xJobBtc28J1zM98sWQxXxKnGVmWkxcK13VwHIcwDGg1Q6LAx2p6 +hJ7F3b0+l+MBcbzkdBxzPl6TxgWxC4uFy4vjCzpNh72tDv1Og3YzwPM8XM8lSVJmswWz+ZLFKmG+ +THlxcMlytb5R3f3d59Ak3RvJyyi02RuF3NmOeGe/w+6wSavhoZUkS9YsV2tW64RCqmodfc/F81zC +wKcRBQQNj07To9PwiDwbG2WqOaSRU9NaI/Qm23+zd0zv+m/bO0KIMgPdur6JbvQ7qyQTTZkEb+SM ++Iqz5sZnCQtdZbzfUDjQ1w4psz9kVb3CZjXKqpKXr1XfuNY3Hy/CunGP6GrO2Nxz3zZesckOv3FN +ZWVXbRPV1NTU1NT8nuhNQELhOjbthsfWIKTb9gh9B98z/jBLaHrtiHSrx8HxJZ4lUfkKma+ReWIU +h2wHrb1vCIhc93bVKPPPSlbnKa0kMk+ReYzKBUJlRsnHEfiejecILCSoHC1zlMzR2jIJnjJH33TQ +bzIa+fqTlLlM8fpz/OZMVJ2xynOKEOX5SFTBlc3ZLUszZvMlJ+cTc0ZSirPzMbP5nDhek2c2Sllo +a2NvbGR/RZUAq8tz5s3zFkactjrfbRSrhLBMsuhL165fOpe+dN66MRvVa4QobRlQ0rSoyrKM2awc +hzJnvfOLMdP5nDiOKTILJW20Zfo0m3m5MV83z7ObcVZjuTn31o2xlGfVajzXywjqxpnxxvy8Xqbo +xlqVn3Rjvb76Gb/PGbI8O6sCQYFnF9iWYrvf5N5ekwe3h9zb67LVi5BSolTGcpmxXqes48TsU8C2 +LDzPwfddGmFAFAa47RBbKHxX8NnjMwQSWWTYyqyN3th35Rn8pfN7dW+VZ/nNvLy0X60bdsLNOdaV +eta1HaK/dvzX+ayvfsZmT76czCLzFK0sPFvTDG3aDZdOy2e1tnEtiS5WyMxBZjmFytHKx3bUjT1/ +/dGv/MNrbuzX2Aw375HN80dt9ujNeePl++01e6ecLfOnkihVgC6whMZzBa5j4TkWbvm80jJDSY2S +wlQ8C3BtgefYeK6D54BQOTKPkbmDLBRau2iZww07q0y3eL3t9oot9pLdV83DtW/g2g5U5b76pvW+ +Obcv76Xr5IQ3SHbSClSB5wm2+hH7I4/7t3rsbzXptwKUlhRZwjpOWa0T0iwr31dUtnrgu0RhSKsR +0Gl6tBsujdDBsZRpmybcG7Z66dMxC/XK81W9NF/VnFXfBa/uHW48k2/O2bfsxWr4r/7O9XMcYVXV +6eKG/4bqHnplvcvPV0qiZIaUOXkO8/mC49MLGqEHaK6maybTGXEck+UCKQGc68vVb7KfNs+SG9fM +jfu8ep7cHOeNOfrGe0t8jV3/Jvup5ut4KyOYeZ5zcHDAL37xC37zm99wdnZGkiREUUQURdy9e5e/ +//u/51/+5V/Y2toiDMPawfNHwrZttre3ee+992i324AJfN+/f5/RaESr1cLzvD/wU2pq/vwIIUyC +jCW4d+9eGcAIieOY4+NjiqJgOp1ydHTE48eP+d3vfsfdu3crZYmampqa3x/jPFOyKA3dAtux8Szo +RD5397r8w8d3uLXTo9+JKArJ6dklT58f89mTE54cnDOdx+zvbPPD925z99aQUa+FEDCdzjg5PePo +5IKj4wuuZkvarTatdpt7t3f44N073Nob0W238H2Pi/GMFwdHnF94xIVmXcQUmYvtRmg3qFw/xtlW +oJQsM2jlK86aGwa0ZYEoK9g3P8Ix0twbh9jvNWvGGPNsQbflsz1o0G542Nb1+9m2TafTwrJsBv1j +I5enUlSRIovEVBzIAq0LwDHXUxkg5lNuGmdFUXBxecXBwQGX4zmz5Zp1khGFEVHUYGvU49buiJ2t +PrZtE0UBO1s9Pnx4B9+x+cWnL7gYz0jinKVjI2ybzx8fcnU15t7egPfub3P/9hatVoOWbbNYLHn6 +/JAXhydcTNZcThLOphlXsxzb8UzFiPUaI+pbZq4yprWk1Qi4f3vI3743YGvQptUI0bpguVoxmy84 +ORtzfDYmSTLAxnJsuu0GnXaT0aDL9qhHt9PCsgSOY+PYpnrI7IuN0bxJjjCfudkzqsy+Nv3eFDez +l6tECWHmybJuOAwqB+X1PuSGo/C1edTf6JwQ1TwKyykrikylur6RPq+V+TxjmMryc3UZixflHnfK +Ci7j5KgcYq+M9+vuFSHsst+ajbaug+gbR+rmfb5uvJsxi40xbFnV+5lrA61rI7mmpqampub3wRwV +zTnGdXx67YDtQYt2w8e+oTJp2zbtdgvLthi8OMP3TM9nXZhzqJL59fn5tUlp5Vm0rPozn1lcn6/K +s1SRx6aKLbMoiqw8J6gbsZ9N8NOcI5Te9JsuuA6s3wy6vm7Em7OKeClgZ847m8rCG+9zI9Bn7AD7 +2g4QdnV+WScpx2cTPgtNJaiSOUcnl1yMZ6zjjKzwQN8IcADXARlVrYMq5+NmUO6lIMkmUGJdnyer +gOZm3C+dTV8+Z5mxWzfey/xpzmOSdZxycjbhs8e6Gsfx6ZjziynrdUomfTQOQoiXkg02QabrdX31 +81W1Ctc2lf2KTWWDZb8cnNHXZ079lff7urPfplLxer42fbKFZT7ju6lLvYKSKFngUBA6isiDu3sd +/vbDW9zbH9BpN3BcmyRNmC9WXI6nnJ5POL+YoAGlBb7v0W6FdNoRO6MBu9sDwsCn1QxxXYco9BFC +lfdWXu19lDD3bTUnm71zI3m7SijYJKDePDvbpU2wCbCZoLdSxWvmWN+YyxtTe0PJ6eZnVGf+jRqC +1miZIYsElQuUzK4TzUv7TeUJebqiSB2yNKFQPlqZ58kmGLkp/31VOvlVG6lKZkG8vOblHriZuKul +LMesqmDpJigsrLJaeDMm2y5tGqdKBN4klV/vxVfVGnSZVCTR2r6RhPJVG25jA2opqrWkmvtN4HCT +nHC9Rjftqdfe39bNZ5WxC28+J9WNZ/D1HPAVO7xKT6/uVcfYmJZbBjK/qWJ38+yXoAsC1+H2Tod/ ++MGI3VGHQbeJZQtWi4TlcmVs9fMx8/majYxEq9Wg22nS77bY2TLV4QJw7I2tbu5Js48VNjcLGcq5 +1RIlr+fs1WdI9Sy0rFLBwr5RPU5lAysl0XLzHXRjzl/Zi29iqyNML3dLG1vdPK9eDnqr8l64eW+a +NVcomZbJSnB+OeGzx4cEroXWisUy4fh0wmKVkmUOSjs3FrPcUa/uJ33j+7j67nnFDhc3ntebZP5N +cnw5p0oV13Okbtwfpa/Msq73p9lHm/cqQ+N1zPP35q0KfGutUVKRxAmHh4d88sknfPnll4zHY5Ik +od/vs7Ozw7vvvssPf/hD/u5v/+4bK72VUuR5jpSy+qlkg5SuXmvbNo7jYNs2tm1XX4Zfd41aa/I8 +r97bHMI0onTEWpaFZVnV+zqOYzZx+fo0TUmSBCUVSissYeG4ThVQLorCOMiVQkmFZVt4nodt29Xn +aqW/8trN9W8+69XrzvOcoiiq/ulSymrcN6+z2+1y+/ZtgiBAa43jOOzs7NDr9Yii6CuB75tSTDfn +ZbOe3zQvXzfXf+g8/zH3pNYarTR5Ya5lszZalQ9zS7x0HZs/XzcuWUiyPCPLspfG4zgOruu+9rVZ +ln1l3TzP+9o1l4UkzVKyLENrjZQS27ZxXde8xrKxbAslFXESk6WZuV+0qj7fdd2X9uFmDm7eM57n +4bobGSrxrXP4ur1xc/42c7i51tetpSzka695MzYp5cvXrfRr74/d3V0GgwEAT58+5fPPP2c2mzGf +z7m4uODp06f85je/wbIshsMhnU6n/qKpqan5/bkhv6RljpYZnh/QiSy2+yH39nv84ME+7Zb53p3P +VxyeXPLplwc8ORhzdrmgUIJGI+Lde9vsb/cIQ48izxlfTfjy8TO+fHrI0xenjCcLut0+3V6fXGoG +/Q7boz5BGNBuN9kZdRn1m/TaPmquWKUJMveM0a0kCtM+TKAQWuIg0aIAS95wmulKCltpgZbm4K8s +I09o2Q6WpRHCrTLcfx8vysZw9j2rCnw3I9OvW6nSIaEVYRDg+z79bosodHAsjaVz099pE/hWCmHf +yGKuKoqvs54BpFRMJlOePT/m8PSC8WTBbJnSarVoNtvcWiRoLBzXpdNu0gt8Bv0OlgDfdTg6vcIR +BessJ0l8tHCJ45jTM8l6HROFDqNeA9dzaUSK1Trm9OyCLx4/53SccDqOWRcuBSGW08CynZcrjL99 +s1UBfSE0jqVohQ63djr84P6O6R/nu1xNEs4vJxwdn/Ls4IyD4wvWcYawHGzHzOWg32EZ5ygsFBar +dUYhNUpfz9+mimbjhEMLs8eVqQS3dIEQ0lyPra7PjJsV1hZoC61tpHBuOC0Frg22JVBSo6VEyoK8 +yJGFRJTnBuuGxH8hJYWUCASObZXOaVF9nlSglADtolAIrUvHjVONR6CwhETYxtGJtTGkzbi1ttFS +oZWNsCy0KB0jpWPU4sZ4xdeMFwstHaTayKBuxqtfM94CWRRgWdivjFdqMyatLbRywHYRGiP/Zwn4 +Q5yXNTU1NTU1/2HZBGJMlWm3Hd44g/LKGdTH9z363TZR4ODaCnReJmCm2DI0lXboVwIh+oavKS8d ++AUCiS3M+QOh0EJiyYxCZLiiwBYSE1rVVQXpzQRErRWW0LgO+N4mUGACkkWRU0hp2im94ofTpVSu +iS1ppAbLdrFt42OzLbAsUQYZJUorpJLmNdIEDJXlYtnmZxMAybKCq9mKwxOJzE1C6sXlhOl8RZzm +FNou4wg3Kk+1rAKAQksE0pzNrK8Gq7UW5Y+FVhZYLspyTfLtJuERhYU25zOrACHL85us5mwzdqU3 +wT0HhF3thSxNGc8WHJyYoKQsMi7HU2bzFUmaU+CwkandzKvWEigrkeX1+jpCIiyFUUhS1bldI0z1 +OwKtHbRyrpM14UZlocaxwHIEWmpUeWYsihwpc4Qoz8hClEWEwrRqUgqlQUpQGoTtocv1snQpx19V +u3+XA+S1HaBlju1ImqHFsO1zb6/LBw922dvqUpS+zMlsyfHpBUcnlxwcX3J8Ojb2JBAGAd1Og0G3 +RZZrEA6DXpsgcE31t2MjqurJUr1LC3MGFyBEgbDN+qLLdday9A9erzHCBKWV7WI7HkI7VVB4U90p +kLi2wraN71KXgf1CFhSFxLItLHFzns28GptBoAtjPwnLQdvuSxLT5r0lnqOx7U1AflPLC44NvqPJ +HIlfgG3l2LaDbRfXVbiVZD3lNRVorbEs4/MU4tpPK5VGSg2Wg7Tc68BalSht7jULiWMVaKHKoOzm +2aLRWpjxYewOJc3etGxlgvw3niXVz4261pv/nZeCrDflnzfbadOyS6KVVSYFqBvXslEauw5+Cko7 +bOOzEPLGs5HyvhJoaSPlJsDoYtmqui6BwhEKy5Fl0NjsnUJKZKFu+K5Ftfaq2o+6DAbblNbY19ph ++obdaQtF6At2hi0+fHeHdjPEcx3SNGUynXF0fMrzwwteHJ1zNVuVCRUO3U6LQa/N9qiH1ALHdVms +U7JMVra6+ca5oS5RJajnRrVNmTU3tqtCCMXGl2CWo0wQVxub176RNCJwbHAsEyxWlrk/8qKgyPOX +/PyV7VrOpVYax7Gx7Zf9RFILlBJozHeM0Lq0029IMWiJjfmOFOWzHMs8WxQSjVEbKArJdL7g8Pgc +z1wkcZIxma5YxymZFKgqf0RX3znoArGZF6t8jiArm7/aT6rcU8pCadNaxLJdxEu+m02ygcTSEiEK +89y3b/jUykSD6ntM22hpV89mYRl/xR+mpPgfm7cq8J3nOXEcczm+5ODggCdPnnB6ekqSJARBwN7e +Hv/wD//Aj370I7a3t7+1p/dyueTq6orJZMJ8Pmc+n18HHLXG932CIKDdbjMcDun1egRBQBRFrw1y +bQLpaZpycX7B2fkZi8WC9XpNmqZVYC0MQ5rNJs1mk+FwyHA4JAiCaoxPnz7lyy+/ZLFYoJTCsiz2 +9vbY29tDa810OmU+n7Ner0mShGazyfb2Nv1+n9PTU87Ozliv19Vrd3d32d/fp9fr0Ww2iaLoq9cu +Fefn55ydnXF1dVVdd7vdptvtMhgM2N7eptlsIqWR8kmSBCkllmWRpimWZb02oLsJam7m5fzivHr/ +JEleOy/9fp/BYEAYhF9JXlBKURQFWZZxfn7O+fn5S/Nx8/0ajcZX5vmbEhe+K0oq0iwliRPOL865 +uLhgsVgQxzFpmlYB90ajQbfbpd/v0+l06HQ6r60SXiwXPH/+nOPjY+I4Jo5jwjBkNBoxGo3odrv0 +er3qtVprLi4uOTs7ZTwes1gsWK1W7O/vs7+/T7/fp9FovLTm88WcZ8+e8eLFC7LMBNkbjQa7u7vs +7e3RbDZpNBrM53MeP37M06dPq8SCKIqqa5nP59VYsywjz3OCICAMQ/q9PlvbW+zs7Hztvri5lkVR +VPtvNpt9ZS2DIKjWcjAYMBgMaDQaX1nL2WzG48ePefzkcXXNzWazun+m02nVp3u9XhPHMTs7O+zu +7tLv96v9Z1smWD4YDHj//fe5vLzk888/5+nTp6RpytHRET//+c9pNBo8ePCA3Z3d6ou7pqam5rui +1HXlqkWOLXL67Tbv39/hw/ducXd/SBi4rOOEq6sZR2djvnh2xuODCVezDG35NBuReX5HIb7v4tg2 +6/Wak5NzfvvpFxweXzKerlmlBdLOSMkYjGNOLxdsj2Z0Wg2cdgPPNY6DVjNgHidolYDKq4x2tEQq +ge9A4Gp8DwLXwXc3fcRLB1BekOcF6yRjuc6IM4mSLhoXZfvYrjn3CMsxFQS/R/WpJYxTJ/RtOs2A +XrtB4DtorUjTgjTLKApppL58jygM6HXbDPptMlxymSHztMr81WUW7Tex6bU9m6+YzNZcTFNmi5RF +6hHFKbGcsspgPIt5eG+PMPTxXJd2q0m/n9FsBviuwLaNJaW0RS4VSlskuSCXG4epMbCUhqxQpLki +KSCVDrlywAmw3cD0Z7zRu+7NMA5R17XwhUOr4dII3HLfmO+xy6spv/rkM371m0fM1wWLdYHUDpYb +4LgO8zThbK45mRQcnK3odxpIaYKws2VGJgWWEyBs51rSUurSsZeBzPAdTSOyaQQOvmvhuiajXimT +lJdm5idJc9ZZSpJqhG2q3MOGz6jXYNAJQBsDdrFYcjGOmc2WNPyAZiMiDEwioOPYXE3nTKdrbNui +123TaTcRAiwhWCc5k9mK2SJG6oIiN44xy/HRwqqMT9+FhmfTCCwC18f3BFJK1nHGOslYxZJlWlDI +G/LsOgeV4zvQDM14PdfCdc2+34w3yyRpLonTnHWakaSqGm/Q8Bl1I4bdsBrvcrHk4mrCdDqn4Yc0 +IjNe3/dwXZvJbM1svibOFIVyKYocyw2uK0E2spU1NTU1NTU1b4gJKNuWxkYR+g7tZkC/2yD0XSgL +WV49g4aBR6/TZNBrkeGSqQyVpyivTAa03Ou49yYgJPMysJOByrApaIY2rcicux1bYAlNkjikicvO +IKLfDvA9G9exXvaBl5XjWiuiwGZ3FLE17AMmCLRerZjO5qxWMWEYEIV+GXhwEAKSJCVOMparmMVy +TRLn+EGA7Yc0g5BWK6QVBYCp6EyShOUqZrWOWaeSOC5QwsXxQmzX+KaMH8fBtsB1bVzbAyWYB5sk +VnUdFiul5a2N6o0qcG1N4GoCTxB4NoHrYtuCjfBTXhTkhSROJKs4JU4lUrlIvKpqzrZtfF8QeRaB +a+G7Hr5TtnRCUxQ5eV6QZTmLVcJynZJlCqVtFJuKVgslzee6jgWWB56Fv/KwLMv0ot70Ua8k2jdt +f/IykJJjk9MMbBqhQ+C5uI6FbQtTEKUkSSpZxRnrJCcrBGkh0MLDdjcS1yaY0ogcBu2AQds1e0vl +rFcrJtMZ80VCGPhEoZFBdh0X27HJspQ0y1muEuaLmMU6QdiB6ZfshNhugA1lQqj7HZKWb1TKygIl +M4LQYm+rzXt3O+zv9IgCjzTLmM0WTGcLPn18xKPHx5xezlmsc5ZrVVXjurFiso45uSo4neY8OZiy +s9Vhf6fPzqjLOsnQXFcZa6UQQmMj8R1zfm8GDo4Fjm2uyxQQmbEvVwnrJCfNFGkOjheg3Airao11 +LUMc+Tb9VsCw7YI293EcJ8xmc2bzmCAMiEIP33dxHQfHtkiznCzLWcUZ86XZT8L2UU5oKoJL/2Yr +tOn0O9zZbbPVbxCV9qSwBGHgsbPV4/0Ht1ilsEoFhbaxbA/L9m7oQInSzoHxZMb4akleSBpRSLMR +llLYHkVRMJmvmc7WSOWQSwdEGYxXrklvFhLPhkZo0wzM3nRsgUAjpULKglWcsVonxKkkLQRZrrFd +s3csxzOV4MJ6KTHlT/OILhUPZPngUxmonNAXNAOb0PcqWXWpTEA0zwtjd6Y5qyRjnRbkUmC5IZbj +VwnQgWczaPuMug0soUqfQ8r4asp4siB0fRpRWK67h+95LOOc2SpnneYoISi0BZjEmdfbYdeJ46bV +mkM7cmlEDoFvbFrLsliu1nz2xTP+37//msk8Zb6WpIVV7lWbRZpxuZhzcpVzfJnw5Ysr0GatxrOY +NAfL8YwvYaOmJovyJ0OrFFcos+ah+c5xXRvbFkhpCsmy3NiuSVqwTnPWqa4q273AY9gK2epHWGWi +RBLHXF5NuBwvCVyXRhRVzyLfd5kv1kymc/Jc0uu26HVaJvgtBHkhmc5jpvM1ubKrRBm0h+WUreqU +xLEUjUDQ8B0Cz8N3BZZQrNZGEn42S5lmBUpqLEzLvMCzsSyTtGHbCUqX/hjrZRnyzbPasSSRZ9EI +jPS+awssSyMLk2SUZjlJmhMnGetEsk5ysLzyeRrcSKxXCAocoWgE5v18z8VzLGxLlDENszeTrDA+ +kSQlziRY5r3Me7oI2+O6nVlt338X3qrAd5ZlLJcrxuMxR0dHPHv2jIuLC/I8p9FosL+/zz/90z/x +0UcfsbOz861VpsYhfMKLFy9eChhvKkJN9Y4Jmt27dw9LWOiOCYi/ttpUSvI8Z71ec3h0yKNHjzg9 +PWU6nbJYLKoq3Ha7XQWSpZRGHtz1sGyLoih49uwZP/nJTzg/P6coCmzb5qOPPqoCeYeHh5ycnDCb +zZjNZgyHQz744APu3r3Lp59+ymeffcZsNqsqeT/88MMqu8uxndcGvqWSXFxc8Nlnn3FwcMDZ2RnT +6bQKCr7zzjsEvgk+FkVBnuckSUKWZeYhlOdGKtr6aoW9lJI0TVmtVhweHfLFF19wenpaVdBuqnE3 +87K1tcXdu3cJw7CqHrcd+6X3y7KM9XrN0dERjx494uTkpEoI2Mxzq9Wq3i/Pc6Iowvd8hP3HewgU +siCJE+aLOc+fP+eLL76oAvHL5bK6lsFgwO3bt7lz504VQH5d4HsTbP7Vr35VJWO0223u37/Pe++9 +h5Typddqrbm8NOv24sULzs/Pubq64uOPP66SEmzLfjnwPZ/z6NEj/v3f/531es16vabf7/PRRx/h +OA5KKXzfZz6f87vf/Y6f/OQnVXC61+vx4MEDHjx48JVA9Xq9roL6t27dQmlFv9/H8zyTHfe6yn2l +KQozh8fHxzx69Ijj42Om0ynT6bTaG5uA93A45P79+6atQRh9ZS3nizm/+/R3/N//+3+rax4Oh3z8 +8cc4jsPh4SHHx8ecn59X98/777/Pxx9/DBiVgGazaVQUbI9+v88HH3yAEIIsyzg7O+Pi4oKTkxOW +yyW3b99mNptRyAIHx5RB1tTU1HxnzEEdJRG6wBEZw67Ph+/u8o8/ekC73SAMPKZnC45OzvniyTFf +Pj/jyeEEqW38oEmz1aXRaBI1fHzfSGjlec7J2Tmff/6Ys/GCTDlIPDKdE8uUs6uY04sFe1tzLMum +2QhwHJtmI6TViPCnOSiJlJu+gEYSUaCxbOMk6jYduk2HTtPHcy0cy8iqLeOE9TphPE3ReUK6TpDK +oVAelltUEonWpudRWaHwpk4UgS6djprQt2i3fPqdsMryTtOUxXJFnue0W038wCeKAvrdNqNBh8lK +M13lRkauKCtpviXoDSZhK05SZssl4+mK8TRhsizwkhR3nTBbw9Us5vJqTuB7vHN7RLPbxHEc8kLR +jEJ8R+CU7b2kMvnneQFpGfgupKnUBZBSkxeKJJOkuSaVFoV2cWwf2wkQtl1WrLyZkXMt3w6+I2j6 +Nq3IpRG6BP5GsUdzOb7i1797xP/3458hvDbCbWN7TRzfwfEV1jxBiITQW9JquLQih1bDp90ImC4y +cmlhu76RcxOlk6+setYyBhnjuTbdKGK779Ns+LSiAEtAlhdkec58kTBfFlzNc9IkJU+yymAM2g57 +A49373QRmPvm7FyiEkkym9MJbbZ6gk4noNUICQKXp89XFOsEz/O4M3K5s981Gfm2xeVkyTO9IlnF +JEVRSiQqnErmT4HSOELQbTiMejbdpk+35ZHnOZeTJeOJ5FwZx4/MVenAshA6BZXiOjadKGK759Ns +BrQjH8sSJkmkKMe7UkzmBVmSkiWZ6f/p+Pgth92Bx3t3uwgKhC64uFTo7JL1ZE47sNnqNeh2fFrN +iDAMeH6YodKcIjW9zIrCwRFGecEWlvGE1XZxTU1NTU3NG6LL6mONJcxP6Ft0WgH9brOS5X3tGTT0 +6XeaDPotZmuLbJ0hi7RUwVGmMg3bVCmioZRRlkWGUClCJViioBNE7PQDOi2P0HfxbMFy7bBe2XTa +Eb2WT1gmMlb+l83Zr7y+KLC5td3kvXf6ZRWk5PISDq0Flzqn3w3odX2iMMB3TUX31WzBdJpzWmSs +pnPS5RJXtLBdScu32e012BlF2EJjCZgt4Pwi5qLIyNYxebxG4qJVy1SdbyqVtWdUbRwbC7CFReC5 +WFYZ+BYaLa5b9CgljVy8zPDQhJFNr2HTbbl0W371WkvAOklZxxlXszUXMiNdx0jtopSLtlws20Pg +EFgundCl03ToNF3akVONI80z1uuE5VJxqguy1YIsL5CFRa4cLMfDdly08spKehtbCCxsAs97aRxl +W/CyilaXilMF6Axbp1hktIIGOz2PTjsgCjzCwCHPCrK8YLaIubhKGBcJMlPIVCHxQDfZyFQLyyZ0 +XfaHAe/eaplkSZUzmcBzMUenKb2WR7/r0GyGhIFP4LvMlzaL5Zpza02yXJGtp1hOiHAibK8AyqA3 +lPaS/WZ2W5VwoY28fpHiuSH72x1+9IPbDLttwsAlTTPGkxmHR6f85tOn/Py3zxlPE4QbYTkRlm3O +6wiF0AlCr3hxMiPyBbd2uvzw/dvYtsUqLtCYoJLAVHoLDZaG0NUMWx7bfY/QswkDG4Eiz3KSLOXs +IuNcZsh0TZrnpKsMVTRx/ALHb1ZzsJEz9m2L3UHAw1tNczZHMp/NeXG4pIhj+k2PXtem1QoJA48o +8Jgv1yyWCZeTlGy9ZrKeYbkRwilMELKU1m73Q+7sRjy41WVn2DKBb8/FtmzCwGd/Z0iRpySZIs0U +UgmTSG7ZZdW6mX7bNgG0L5+syJYpiU4ZNF22Bmb9W42QdVrw/DAmWcTEuYXKLSSm0t0E6SRCGIW1 +QTNke+ATBS6R72DbkGUmMeRiLDmnQOUxWSFJ1wWOn+Nqha11WfHqbEph/xSP52uVsbKftECCTBAq +JWr6DNsO/Y5HM/RoNVyKvCDNC2Pbz9fMFxJdxKyXMVmqsWVWJeoA2F7AVrfBB+90cGyNhWY2X/C4 +mLKcLGj5MOqE9LserVaDdqvB8cWSwzPT/iCVFrkyNpjQ1tffPmVyjOMIGp5Fq2GuOQw8XMckis+X +az5/9JT/78f/TqY8hNfFdlvYvo3j+1iLFCFSPGdJ62hCu+HQbAS0GgGF1MQ52LYPtgtlQsKmOh6Z +oGWM5WpafsROz6Pd8mlGAb5rkxeSrChYLlPmy5jJQlJkKcssRQsXy/axfBh1Wnxwt43ngtAFi7nN +IzlhcbWk5TUYdiIzV82IVivi6KRApQWpSNnrd3nnThvPc02Sepzx9DAjW6csU4tcSqSWJvnfdipf +g7A0zcBmq+fQa/n0Wj62pbm8WjC+UujcYrkokNIkaXmOjR94uLZRXXMcc4+rzV4S5fcxORYFqBTH +UnQij1HXo9nwaAYOnmeRl4kt8+Wa2bzgSuTkyZoiXqMtH8uNsN3cJKg4nvn+1TmWq2kHAVt9j3bT +pxV5+J5t7q08Z7FMmC8Vk7mkSGMWSYIWAbYvcShbrgkbYYnSp/anPQn9tfFWBb5NoPqYx48fc3Z2 +xmq1QilFo9FgMBiwt7fHnTt32N7eptFovPY98jzn5OSEs9Mznj1/xpMnTzg4OODq6orpdEqappXU +8qa6+9kz83v7+/u89957fPDBB/T7fVzXxXGMxMQmKPbixQtevHjB8+fPefHiBZPJhNVqRRzHleRy +FEW0Wi36/T5HR0ccHR1x584d9vb2CIKA8/NzHj9+XPUVFkKwXq+5urpCKcXFxQXj8ZjVasVqtWJv +b492u02r1WI8HjMej7m4uGC1Wr0kYy2EIAgC+oP+V+ZFSsnp6Sm/+93vePLkSfX+AN1ut7zxTbXL +arXi/Pyck5OTKvA9Ho9Zr9eV1PdNyfbz8/NqPg4ODjg8POTq6orlcvnSvIRhSLvdptfr8eLFCw4P +D7l75y63bt9iZ2eHLMuqqvHnL55zcHBQve+r7+c4TjXPvV6vCnhu5nlnZwfg96r81tpksMbrmLPz +M54+fcrz58+rsU0mk6pieSML3mq1ODg44Msvv+TevXvcv3+f/f39Kpi7oSgKlssl4/GYs7Mzzs/P +iaIIpRSO4xAEAVtbW9fXojTn5+d8/vnnfPHFF4zHY2azWRUs3gTe+4N+FQyYz03F9yeffEKSJKRp +WqkCVDIjQrBarapg9CbZodFocHlpFBcmkwnT6ZTVamWymtOUKDIVh0dHR1xeXnJycsLdu3e5d+8e +3W63mu/NPTMej79yz1xeXrJcLlmtVlXFfBiGtFotOp0Ox8fHHB8fc/fOXXb3zHVv1jKOY05OTl66 +5pOTExaLBWdnZ1xeXnJxccFsNqvuH8/zKkWHVqv10r4Iw5C9vT0sy+LRo0e0Wi2m0ylZljGdTqv7 +4OLigna7XfW9r6mpqfkubLKxPc9mZ9RmazTgvXf2uLM/pNdpkEvF+GrK4ckFj56e8ejZOedXMZks +szvtAGF7RjqpzB0272kRBSGdTotlnFOsCtI8R+QZpCl5ngMay7avK5CVIsty49SR8lpeTxsZwXbD +o9XwGHZ8hl2fYcen1XBphi6ea2MJE/iOk5Qky5jO1oynay4nK04vV5xdrclVhpYZquyFrHFKma03 +C3pv5izwLCLPoR15hIGLbVkslivzs1gymy/J84z9vR2jeBMGjIY9dndG5CcLpotF1edbFrmRmiur +S67tcfGVT1daU0gjCVcoTHWvskBaxLnGXmUEnsU6zimkLluACFzXMRnyTinrVfZqMkaVwITxxSty +65t/FEY6DlHJK1Z9/RDXElxfnaqXh1H+nigrWTZ7z7JEKVFp1kFJk8EexwmW8rF0gSMkytZo+/rN +pdYUKifJJOtMsEogyRSZcrDdsJSzBEsUWELiWJJOJ6DXbDLoBGz1IwbdkEbo0Qg9hICiMApKy3XG +Ks7M3hmvuLhaMV/lzNcxNgG9psPd3Ta2ZZzPnYZN4GpG3ZBer82g16HVjGhEAY5ts1rOubwQOI7J +BG83HBqNkCgMaUUeeRaTJGvG84JkESMLaeQAbafq3eU7PqN+yIP9Jt2WcTzP50vW6xXn+ZoiS5FZ +ii4ElmPjucYR22+3GHRDtnoNRr2QRugThR6WVY63KFitU5Zxznhq7pXNeGerBEv79BrleIXGshTd +pkPgaPqtoBpvuxURhQGe55DEa67GVyyXBUlhstBtx696BNY2cU1NTU1NzXegkp/VBK5F6Lq0Gy5R +4OBUZ9Ali8XqK2fQMAwYDjrsbg0oztZMl+vyDJqhVNmy5EYPUXSOTYFj5TRCi26zTa/tsj1osj1o +0Gp4BJ6pNIuThCRJCQOfQa9Fr9smCgMsqzr8lfEgEzz2XItBJ+TubgdLmETSftuj03SYzjp02g3a +rYjA83DLwPd80WK+XDMatBh0G1xNFrTbLdqdNv1eh+1hm2Gvac6UAparNluDJpdXbZ4fnmMLyTKW +5ComjyWWE5hKZSukFflsDdqVLG6a5kSBqTSVmx7NskDJFMu2aEcWzTBi0PYY9gK2egHN0Kfd8PA9 +p2z7AnGSk2Y5s0XCeLbiarLmdLzmdBwjLE277dFpN9nqh2z1QzoNj1bk0AjtahxFUbCOU9Zxytao +y3i7x9l4wfHZjIurJZTBF4sGjdBha9DGtQWOLciLgujwDEsoVClbrsre6kLm2EJi2wXNwKbX7tBv +e4y6EaNeSKvhEwbGtioK06pnuUqZTDtczdecjdfmnLiWJEVOmi6xHKMS5Fim7/yd3XZZratYdH1a +kcP2oEGn2aDVatCIAgLfxXWdshoyYdRv0W4GdFoBk1nK1XxNHht7UQiBdsOyDY8o+9J/2y2zkbWW +WELhOSYo1e9G7Aw6RJGPY9tM4oTD4wt+9+g5h2dT5itJKh0cx8e1I7TlmpZZUPbXLdCZJiskzlVO +cLgg0+ecXMbk0sZ1TUDfD2wGnYBhL2DQCRh0jA0beDa+ayHQ5IUJLo16LcajLhfjOWeXc87Hc+Ic +4mxJJnPcoEwAKAdtWyawdne/iy00tlCs+hGthsuwF9FpNWi1m9U8+67LKjZVp+OrFp1mRLcdcjXP +mS4T0iRF2C627RJ4EVv9Brtb5n70y9aMQgg8z6XX7aBR5LmiKBSFhk21p7FRFUKA45h+zo6QRIFN +luX0e236vTaNMCCKfKbzFcv5jItzSZYXqEKjKbAdCFyXfjtg0A0ZdgNGvYBhNyDwXALfLu8RUzE9 +GrTZmXUZXy04Gy85Hy+Jc0FSxORpgeOGCILrftHoP3IAfKOWpnDQ2JYk8Cx6zSbdZqe8/pBeyyge +hL5rihcLSZJmLFcJq1XCycWc0/GC8TRmsZbM13M2KSsignbD4c5uB9+1cWxYrRoErqDd8Gi3G/S7 +LbrtFlEU0AhDNIL5cs1ynVOoawnrb/6eMYiNrW5xw1df/helSfOceJ2QI7Cq7wyFtnV1bxYSlJak +uWKVWqxS04IuyU1FO6IM++kCmxwhCtpth26rT7/tsdVvstWLaEY+UWQCxIU0tus6yVmtUybzmLPL +JWdXK2bLjPkqBxnTiixub7cIAxtbaFZxhOso2qGxwfu9zVz5REGAzlOuxhfMC03kGaW0TiukEYWk +uSTPM9JkhZikZIuUIsuN4obtlqogBbYr6LUj3tlvM+yYZ5lWBVkSc3mZIPM1qjBxrCjwGA46tJsB +nuuwWCUcX65wLFF9/6Cl+W5yoB159FsRvZbDoO0z6Po0Q4cwcHFtUcYgClZxwmKVMJ2tOL2cc3a5 +YLbImK0zkiTH8wMcR9Fp+vQ7HfqdgK1exLAX0IpMkozrWBSlGsE6zliuE6bzmNOLOWfjJbNVwTIp +SNOl8dcJk5hkWZs2HDVvylsV+F4ul7x48YJHjx4xHo8pigLP8+j1elUwczgc0m63q97Cr5KmKV98 +8QU//elP+eKLL3jx4gVnZ2ekpQO4KMyXmda6kln2fZ8wDBkMBvzLv/wLzWazCmbatk0cx0ynUz79 +9FN+/OMf86tf/YrZbMZ0OiVJkqr38uZBdTPQ+/jxY27dusXHH3/Mf/kv/4X779xnMplUAfHNa6+u +rnj8+HFVqb553011+u3bt9nd3WW9XpeV8csqgLjptRwEAaPR6CtzspEiPzk54dNPP+XJkycsFguK +omBnZ6cKuDqOg5SS2WzGyckJh4eHpGmKEIKzMyPrvuk3rbVmuVwymUyqquFf/vKXzGYzFosFSZJU +/Zw387IJcgZBwM7ODrdu3eKjjz7iv/7X/8rOzg7r9ZrZbMbnjz7nxz/+MT//+c9fmueb7wdUQWff +93n8+DGff/45H330Ef/8z//M1mgLYf3+kufz+ZzLy8uXxrZYLFgsFqRp+pU131xHo9Hg/v37PHz4 +kA8//JAf/vCHLwW+N6oBaZoymUw4OTmpEixc12U4HPLgwYPq95VWVeD70aNHlYR8q9Wi3W4TRRHd +bpe7d+9W/bwXiwWHh4dVcHijKpAkSVVNvpGv3wS5N0Fk27Y5PDwkDEPyPK/We6NGsFnDZrPJkydP ++M1vfsN/+2//jXarTafTAcxBMUkSZrMZX3zxBT/+8Y/56U9/WlV5x3FcXdcmCL8Zv+u6PH78mNu3 +b/ODH/yAf/qnf6pkxgGSJOHi4oLDw8PqfnYcp0rq2MjHb9ZoUxF+7949bt26VfUG3+D7PqPRiCgy +UvDdbpeLiwuyLCOOY66urqp7FagD3zU1Nb8Xm6qRyLcZjbYYbe1w59Y2u1t9HNdhOp8yvpry5bMT +PntyyuMXYxLpYnkNLCcA20fhILVlWgmXjjnXddjaGvDg/l0UDunRJct4hV0kyNxH6ALPs2mEvumJ +JiglAVNWq5gsl5UcslYK21Js9QIe3O6xv91mZ9hi1IvwXSOpaAmLTXsy892gSLKcOCkYTxf89NdP +Wa2fsogztN70K3Qoy2d409i3URHRRL7DoG3RbbkEro1Uitl8wcnZBVdXU6bzBUWeE4QhW1tDwjBg +ezRgf3eHyaKA06uqD5+paHdNzy696d340ipV/yQ2HdbKrHzLtrAst5RxcyiUJM2hUF+99k1bDNtE +m42MtigN27LXYPnWN+ZClL2gMH2csKuqDpPiqzfduW701qbqS16mAbOZZK0VWpjed0qajObNcDfV +2ZZt4Tourmf6IG6S5za/aCoSLBA2ubbQuSBfalZJgdSQKwfHs6vPFFriiJzIVbyzN+DDByN2Rh06 +rYB2wzdOmXIPmmtS5IUkzyXjyYqTiyuOTic8enbO6vkcS0e0Gw772x0cx8K1BduDFrtbPZbLFVEU +EIVBKQnnoJTk9Oyc54GRwncsjW1BpxmyvdWn2QxIkjVpvCbL51xNlxR5YWQDHd84aZTEc00v+Q/u +b5sM78jn6ETw7MUxq8WUJJZkaYHWAtfxiFzMeB/usbfVodMMaTUDPOd141XkuZEbPD6bcHw24dHT +M9brBZbyaDVs9rfaZryOxfagze5Wj8XiNlEYEEUBvu/hlLKkl+MJBwcunqOx0gJVgJL5dU++Ovhd +U1NTU1PzxujyDCUsIxfeb9p0Wx6B98oZdDJlNluQFwVheQaNAp/RsM/ezojp8gxxPkPmCUpm5Xm4 +VJcBtMpBFTgix3UKtvtN3rs34N07AzqtkG47wvdsbMtIYRdlsqrj2AS+R+AbmV3btl+9erRSRnGn +HbC31cFxBK4tyIYdbu8OSbMUz3XxPRfbtsp+wFQSzdPZiovxhPl8TadjfE7NpknoC32vrAI2dkCc +pMyXazqNL7F0zvHZjKtFSrxe43gN0ArH6tJtBezv9vEdC99zSNKEVsPHthSFMj16FTkUAse2GHZa +3NvvcHu7zd5Wh+1BC8+z8UoJ4E3AX0pJIRVpVhAnpmru/33ylDh5jmXBnd0G925vsTNssTtq02p6 +eLYw5zM0liUqGWdjH2XEScajJ8f8v198xnR6hdQmAGSJDt2mz/5OH9+zCTyXLMvMOEQZwFdFGcDP +sKSN6ypCV7I7bPH+/S3ev7dFM/RoRiaA75SJyUqbPt25VCSJkbp98uKc50fnHJzOOblYs1iucbzA +SEtbmk7TY2/UwXEw65t1ubU7ZB0n+J6D77lGyt4yfXQ3svDT2ZLbe0NOz6745e+eMp9PWa+XgMCy +HBwhULZjJN7fWG3KnKFd2/Qeb5YqUZ12o5TTt1mtEw6OLvjk02dMVgIpfNwgwA2auEGrTPi1yvuj +7N1NgdKSWWzx5GjF+aRgOl+TKRvPN0m1rYbDg9tdPrg/YnfYoBkalavNvSOErmSbkyQnzjLOLyY8 +eX7MsxcnHJxMOTybkWXr0v60KkUnxwrptHxubXdwbBvXERSFZH93xHqd4PubeXbM5zkWeV5QSMV8 +vub2/oTzyzmffH7Ib744ZLmMsewA7Qb4rsWg12Q06NBuhriuWyV0eK5Lt9MiCHzTm1zrynTViCpR +QgCO45b3u9nrhVQ0oqBKCnZdh5OzS45PLvBdsCiMCoUu8CyXlq94Z7/Nh+/ucHunSyN0aUYetrW5 +T0xfZqUUSVaQphmXV3Oevjjl2cEZh+dLjs+XxElsEnqEQKm8kvr/Y0a+TcF32S9ZCDxL02+GfPDO +gPfvD+m1jDJZGBpFDNexyr7uZcupQlLkRWmDXfH88IJHz864upqVVbQWSIdm6LC/3SMKjPR3USi2 +Rz3eu3+LIPBohAFB4JU+bIdlknNyMWU8jUkKrvuPv8HdYxLuTQ/2zfdDlXhRyqB7noOS5b1RNZfW +ZSs5CyybXAt0IShiiHMJQpPmDpZjVyomKIktchwrY39ryA8ebHNvv1cmwkTGdrUdI3WuzHdJLiV5 +rlisYo7PJ5yeTfni+QVfPL+AQtIMBLujNs2GX90fW/02D+/t4fumvV5QVrG7js1iMefFgUu61ri2 +xhaaZsNne6sHCNI0Jk1XZMWU6WJBkUks2zX+MK3QKscWLoNuxMO7I4a9kGYUsFqtOTg6I14vSeIV +eZbg+aaYY2+nz6Dbwvc8pvMVz48meA7EuS4VIySuZZQO72w3eO9en7t77fJZ7ZbzUkqTl/dCUZgk +kuUq4eT8ipPSpn/05JTlYo7vSDwh2B10+PDhDvdvj2g3jU/Ed+3y2WzmWSnzPZblBatVwvGZeb8v +D8Y8fjFmuUzM89k2rRK0sBBYb7C7aja8dYHvo6Mjnjx5wtXVVSVf3e/3uX37Njs7O/R6vdf3sC57 +CS+XS548ecJPf/pTvvzyS87OzhiPx1Vl7M2ewbrsI7MJ+jWbTba2tvjbv/1btre3q2DwZDLh4OCA +3/72t/zsZz/jl7/8ZRUI3Mh/Wzechpvgptaaq6srjo+PkVKyv7/PaDRiMplUldubgO7FxQWO41Tv +s7lOy7JYrVZVwBtMwHcTeD47OyMMzRdlr9fj/ffff+28xHHM2dkZz5494+DgoOpPnWVZVcXrum71 +vjevTwjBdDp9qeJbSslkMuHZs2f85je/4ac//Sm/+MUvqrFvgsGbw/jNudZaM5lMuLi4wLIsHj58 +iJKK6XTK4eEhv/3tb/npT3/Kv//7v3/jPJuqNROYvbq6qirUd3d3+fDDD6s1/y59mbXW1fs9efKE +Tz75pFpzVT7kNj2tb17Lpie5UorZbMZkMqmkw999990qGcIS1xXXcRwzmUyqSv0wDHn33XervaOk +qYgaj8ccHBxwcHBAkiSs12u63W4lHf/OO+8AVNewqX4+ODioxjUcDiv1BN/3EUKQpimz2Yyrq6uq +2n6jIGBZVpVQcdOgWi6XJEmC7/ucn5/z7Nkzer0eH330Eds729VrZrMZBwcH/O53v+NnP/sZP/nJ +T6r5A6oEkZtreTMh4PT0lPV6zdbWFj/60Y/MWroeWZYxn88Zj8fVNW8C8ptkmM39s8mYXK1WVeV7 +IYuX1nuzR7rdLtvb2wwGA46OjpBSslqtmM/nnJyccHx8TLvd/lpJ95qamppvwsiL+/S7be7d2eG9 +d+/T63ZpNSPyvODyasbT5yc8eX7K86MrTi9XuGEXN2piux7YNlLbZLliFWc0kk0fJtdUjz98h1zC +OjE9ghzfxS4l3pqRRyMyffwKaYLei8Wa2WJNmiuMo8PCtsC3YXsQ8f69Aff2e+yMuvQ7DbI8I8ty +lKqip6VDwS4l1ixmyzVnF1M+/+IZq0QitZFOt2xVBmjfLDN1U7EiSnnJYcen1woIfct8xy6WHB2f +cX55xXS6QGnF9vaWaRUSBGyNBtyer3hxMkZoiVQZWqbIIsVyPGOUf8UIf4Pnetlje3MesW+0ftHl +hWutyozcMrJd9VnevIX12ve9fu/yjEDpbLrRB9Jkj6sbwelX69bLvs4lSuhSRl2SF6o8T+ny4wSB +79HtNBkOuuQ6INc+wrGxXIFtaRB6U2hOUcqzp0IDsqwEsbEcC60kSkocR9IIoN9yuX+rw9/+4BY7 +oza2JbDExrBTlRNgU9Hgug67WzGjfsigHbBcLDk8lLhWTjuyGfYaeK6DV/YH398ZmH5fm2UrpyNJ +U5ON7UCem4zwPM/wPIdBr00UeqyXC+LVksvJCiEzijzHzgMcLzKGL5LQE4x6Ee/c6uOX/chMeyFz +/ojXmiLTOK5L5LkM2h73b3X5ux/ss7fde814dbVvTK9Fh3WcMOiEjLoBy+WSw6NTXKugGVhmvJ6D +57pmvMXrx5vlBaHv4ToCITaOdF3tEY1ZwpqampqamprvgDYV36HvMOy69No+oWejlGK+WHJ0fMrF +5YTpbIFUmu3RCCklYRgwGvTY39vm4HQG2sg+K5mipFEd2pwVlSoQOifwNO3AYn874sMHW/zw/X1z +9vDc0sekkKaRbeXfcBy7TGbd+DX1S9cOGtc21YmjQQPPMcEvAShpAs1ZZvpaA5WPzfNcPNc1PqL5 +gvU6JooiGlGIEBaFNAU5ovx917Fxy8KZIo2JVzOKLGO5XpPFyyqIaCNpRi5b/RaB7xIEHmcXV4S+ +a2SKtQl8Q45QAtdyGXZ83r/b58GdEbtbXbYGnVKtqlSfZFORa1V2iG3bZLlJxH324hjbtnhnv8nH +D4dsDzvsjjo4tkWem7FXHbnLc/0mQOvYNp5j8ez5Aa5VIAuNLoxcfCN0GfVbBKFP5HtcXE2JAg8b +BZhxaJEjpINWNqFr0W+73Nlp8fG72/z9R3fNORwT7C4KiVQKqyoGMXMqhKAVeTQDC0toVquEs8vY +qJoLgU1BI3AY9iI81/SgFYIqwFeUFc5KbYJvgrbbwPdddkd9dkddLrf7jK8mPH78hHmxRhU+eepj +WQ7aCdC2881SzTf3XBmMc2yIHJtm5NCMfBpRUFW+ruKU04sJTw/O0E4X7XZx/RaO38T2GqXKlSjP +uboKpmtVsC4kq0mOHielqePQCExyx6gf8u6dHn/z/g67o+YmdZm8TNC+XmPb9Jl3Ha62ezRDh8jV +ZFnK+cUFq7xAOS5F5mFZjpHtFopm6DLqNY0t4pm1qRIVCmnmWZoxaqDdbOB7DsWWZGfUYTpbMJvP +efJccSVjE+gUAq1zbCFwLKvyEW+qjrEEtmPjKgetTWDU+EzNZwSBZ15r28Zm8FxG/TZF6TvfrNnG +pp4vloSehWMZaXAtCywbmqFgux/w4FaXv3l/lzt7Azbm5+bZs7mPLcsy8+c4zOZLmpFD6EEhJeOr +GYsiRdk2MndMOyn9Onv7D3oslxLnCiUlnm/Tjmx2hiEP7/b5x49v47lWpUwnlUl42CTJ2LaF55pn +3KDbZNRrEHiC8WTCUx2TFxqpBbqIiDzBsNek1QiNDLclKLZ7RimvTI+/Vl3XhIGH79rYZZep64D/ +14z/xmNbaZBSUUiFlCZIv/E5O45Nq9VgOOizyiwy7aMtG9uzsGyj7y9KP4JU5icrACFLF4SFsB1Q +EiVzXEsSepqW73Bnp8UP39vlvXvbWJbAtsy+NkkiukrCcBxjk+a5ZNCN2OlHpFnKydk5aZrTCCwG +3ZBO29jrliXY3eoiZemz2NiumLmKQlPpLIRRWsvzDNuy6LZMG6/VckG8XjKZxRyKCTKLkU6A4+WA +Al3g2i69dsDd/T79TkjguYwdgUCxXC6I1yuKPMX3HaLQY9hvM+p3CAMf13NpNnxs21R8G/tfEfmC +Qdvlzk6THz4c8e6doUnaL/ud50WBVLryYWy+L/Isp99pMOiErNYrnr2Q6HyNbwd0Gxa3dxp8/O42 +P3h3r2zRIar4h5SbeTbv57lGoWDQa7A9aFDIgvOLCZfjGJSPKsLrwggc3riipObtCnyv12suLi44 +OjpiPp8jpSQIAra3t3nw4AFbW1u4jvu1r51OpxwdHVUy2pPJBCklzWaT0WjE9vY2URS9FBBbrVZV +VfHmS3kTdFRKVRXkP/nJT/j1r39dBbHb7TbdbpdWq0Wr1SKKoirouwncTafTlwKXT548od1uc3V1 +VVWebz5vs/mDwPTZjqKoqiK+d+8ee3t79Pt9sizDsqyq8lUpxXK55PT0tOoLvgkU3wxuHh8fc3V1 +xXq9roKbm4Dfppq+2Wx+6xrdDFI+fvyYH//4x/z617+ugs6b6vlNULbf71fB69VqxWQyYT6fE4ah +OTSUwdAsz3j8+DH/9m//xi9/+UuOj4/J85xOp1MFeZvNJo1GozrMTadTTk9PK3WAq6srDg8PefLk +CY8ePWI0GrG1tUUYhm+0/zbrnqYpT5484V//9V/59a9/zenpaSUD3mg0aLVadLtdut1uKRMaM5vN +uLy85PLykvl8zsHBAY1Go6ps7/cHDAZ9XM8kKOzu7vLixYtKSn9TwT+fz8vDeP7S3lytVmRZViU/ +rNdrzs/POT09ZbFYoLWurmMymZAkCVqbfvUbWfRms2n6zXse9tf0Od1I9W8qoXd2dmi325VB9Pz5 +c46OjlitTE+r2WzG6ekpT58+pdvtMhwOGfQHVR/7X/ziFxwcHJj+V+U9s1nLZrNZreVsNqv6a0sp +q3v56dOnfPbZZ2xvb78kAf/qum32ZbPZJIqiqn+853ncu3ePra0t2u32a/uub9j0Lr+8vOTw8LCS +9z85OeHo6Ijd3d0/w1Owpqbmr5Fet8ODB/cZDgZsbQ0ZDHpYwkgmTudLPvvygN98+oyjixWrFGy/ +heNFOG5gDBZhekNfTmIePb8gyTrsDFtGcm80ACAMGwwGQ04vrsgKyAp4eG+L3VGXRuST5xmX6zWn +ZxecnI+5GM/IVABWgOf7tBoB/U7IVr/B3rYJeDs2LMrA3NHxKUmaoxAIsekbFtHtthn2O6XRbxwL +lpCo0vDchGe/k9S5VgigEbmMehH9bkDgO0ilmE5nHB6fcHp+xWIZIyyb2SKmKMwZajjos1qntJvP +sC0FeYYqUmSRYEsjv2YuQ3Fteb5smOpSdlHJHJUnFGlugtJAw2vQb0fc2m7Rawe4jmUyosv+YXGa +k6QFeaGxXav8vi2rqTcB8TeZhU3/srLqYdOTS8nihoTajcD3JsgujLy6ZQlSrdBSs1hnrOKcJE2r +7P/RoMff/+1HNKKIySJjskhZp5o0h7SQJqs/l0glUNhlPz0X2zZV70ILtBIoaRy7jabPvf0uD+90 +uX97RLcdkaUpl+MJ46sJi+Wa5SpGKY1tW/i+x97OiFt720ZlqRmxuwM7o3O2BqaHZeBtziqVe7KK +7c/mS+aLJXGSkueSdZxyeDJmscrIC8n5eAnWmF6vS15IAt9nOOiSphlPjyYIkaNyWfaAT/FdgecK +WpFNI3IJA9MPPc9zlss1V5M555dXZNKl0A6dVsTdWwN+8GCPB3e26XWa5FnGyfiKy/GE5SpmsYyR +SuHYFp7rsLe7zd7uFoHv0WqGCGvI7pYZb7PhEwWvmmWiOs/P5gvmixVJkpLmBUlacHB8yXyVkhUC +hYXlWGU2uF0mQvxln3s1NTU1NTXfK8oeqMIqz6D9Bv1ORBi4SKWYzOYcnZxzen7FfBljWTazZUwh +NUHgMxz0WccZ7eYBNhKtclSRoYoEYTkolRs57CLDtSTDboP7+w3evTNge9gm8F0je7peM52vuJou +mM2XZWGCoNNqMBp06XdNX1TP8/hKy56vkdjNsozVOmY+X3B0csbxyQVSaaKoQbPZ5NbeFrf3trBt +mygKsS2LNMu5HF9xNZ1xfjFhMlvQiCKiKGI46LG3M6DXbbE1GvD+w3eIU8Xp5Yw8WZhgme0iixRV +BtCuL3JztjXnWyVtXNsl8i06TYdRv8n+do9Bt4Xn2KxWaw6Pzzg+OWcVJyhtpHybjZBmw7R8GvY6 +BFFAGHgMukY6emfUZXfUIwpd5KbQ5uiE49Pzqk9yEAQ0mw3azSaDfofhoGssFmHOVJa8jiJWp9HN +WDaJDFpVykGKAsvKEMpm1OvxwTtDHt4bsT3sYFsW88XS2H+zJdP5ivlije+ZhIBup83OVp9hv0Ov +2+KdO7ukheZqmnB5NSNTmkxmRslKvVzQkeem2Gm1XnN0csHJ6QWrdVL2hXW4c2ubd27vlBWYPoN+ +m1G/xbDfJEkLMq0oshXS9XFUDspDizdVDjLz4DmCRujQCBw819g7SioKKcnSnDiTJJnGsSxsyygu +WbZTSvdu1LYwqgLaJARrSyA2aglV8rKi1Qi4u9/lg3s97u716TR9iqIwrbBmCybTOZPZEqk0ge8T +RSE7WwO2Rz1832N3e4jj2EwWMWcXV8hiTk5GniywHB/bDcoArnxppEYaPyGOY45PLyt7VJXV2Hf2 +t7lza5t2M8TzPfq9DoNei1GvwXIVk+SCtMi5uJzwyWfPSNOE997ZIfBt/NJ/uVytODw65eDwhLTQ +JJmikFRJ1Xf2R9zZHTLs3/Dfl/ZClufMS5shywvyQnFyPuHscs46NUnRQggaYcDd/SF/84N73L8z +otuKkFKxWCyYL5ZMZkums6WxoQKfwPfYGvbY2Rrgui7bwz62ZTNb5sankOVkuiDPY7TcJC3ob5b8 +fvMHc7XHtCrQMqXX7vLunR4P7wzY3+kQBh5XkwmXl2OmsxnzRcxqHVfJLO1Wk729LfZ3TaC332ux +tz1gZ9hm1AuZLxIW6wyZx0iZ3/jczfRaCKFZLNcslkvWcUqWG/WyF0cXTObxtRpcuU7im+4eIRBY +SKWJM8UqLljFGXGSYglwXRP0/uiDd9FKMZ7GTJYpi7UkyyEpNGmWkeYJuaS01S2EZXq3Y5VqcJu+ +3jLDcWBvq8O7t9o8fGebUb+FRnN6dsl4fGXk2ldrsqzAti0c22Z7e8D+7jbtVosw8Nje6rM7umJ7 +0GS5TogCp7TNX+6/JoRguVwxX65Yr2MjY15Inh+eMV0krGLJxWSFtsY0Wy3297Zot5v0um3u3Nrh +8HyBayuzHkWCLBI818J3oR1ZtCKHKPBwbFPYsVrHTGYLLsYz5suYvFDX8nr65b2kywSKjXqA58De +qMn77wx5cLtf+sCMim2SmGfvxeWExSrGth1sx6HXMS1HQt8lSTMsYVTe0AWOrdndavPxD+7w/v1d +tgZtBHB2Puby8pLFYs1itSbNchzbwXFstkZ99ne3yvc087y3PWN7cMFsviDVkMkEVdhlmzllqr5r +W/+NeKsC35tg7iaYl+d5JYv97rvvMhqNjCTj17z28vKy6id8cnLCZDLBtm2CIODWrVv8zd/8DaPR +qKoC3vRYfv78eVUtvgnGSiWR0vx88cUX/Ou//itPnz6tgtmdTod3332Xu3fvsru7y3A4rIKgjx8/ +5pe//GVVZbpcLjk+Pubp06e02+0qUHuTTSXxRtp908e82Wxy+/btMnjax7IsGo0G8/mcJ0+eVJWp +mx7ei8WiCmyDkX7fyDVvAt9ZluF5Hr7vV9X0u7u7OI5DHMffuEYmwGj6ez99+pQf//jHfPnll8zn +c7Iso9ls0m63uXXrFu+//z7vvvtuVVF/dnbG48ePOTw8rJIPNnOc53n1fo8ePar6LLdaLR48eFDN +8/b2dvUAevr0Kb/61a+qMW2kw58+fcqjR4/QStPpdL5T4LuQBVma8eTJE/7t3/6NL7/8sgryhmHI +aDTi1q1b3L9/nzt37jCbzZjNZjx79qySrF8sFqxWK2zbZn9/n3v37pVB2UY155u+7bZtV0HuTeB7 +I9G9WCy4OL+oqu2Xy2V1cFiv15ydnVX9rYFKmnsT+JZSVpL7m6B3s9H8Rgn4jWR+q9Xi4cOH/O3f +/i17e3smWG7b/J//83+q+c/zvKouf/bsGcPhEMdx6Ha7PHv2jH/7t3/jk08+YT6fV73q33nnHe7d +u8fu7i67u7vVPXNwcMCvfvUrlsslWZZVcvvVWmpNu9X+xrUzmdYhW1tb9Pv9Krj+4MEDtre3vzXw +3e12uX37Nufn50wmk6/M83w+/5M9+2pqav666XY7vO97ZZWmTxAEzOcLptMVh8dnfP7lAb/47WNW +mY3wurhBG8dvYjmBkX3T2gS+pyu+eGEqUaLAo9sK2NkeMRz02d3d5sH9CeOrKZPpgslsyd7OkL3t +Lo3Q52K9Zjy+4vT8itOLCZdXM5zAxola+L5Ps2H6e20P2uxv92g3PIoiZzZf8uTZC375yafMlzFS +WdiOx7DfY2vY487tHVzHrtR4rDJz3bDpaH2dCf+t5/PSiSQQNAOb7UHEoB2aPl2FZDpbcHh8xunZ +mOU6x/UC5os1hZR02g2Gwz5SadrNCFsotCxQRWLkJl3jfNsotrxsCKkbl7DJKM+ReUKRZ6UvRuEK +h0G7w52dNoNOiGNbRj4tz0iSlCTNiTNFXmiEtnAsp8x6l28y+vLz1Y2s9rx0CuZl5VBe9k674UTU +VDLqwnawbA9t2SRSkwnNap2yilPiJCPwPWzHZms04D///Q959/5dXhyecnh0zvl4ztVszdVszUwl +5ElGnmukdlC42F5gqiYqmS0zv7pIifyQB7f7/OOP7tDvNum2G1xcXHJweMyjL59wfjnlYjxFSonn +uURRyA9/8B6e67CzPSqdlw12t47Z7jdoNjzCwC7XxZxpN33tQTKfLzk8PmUynROnRmrs+OSS+So3 +FQ9iSVbArf0d8ryg2WiyNexh2xadT59jUVDkCXZu9objeTQDl1bDphEaB2SaZqRpzmK55moyY3w1 +QTgRlhMShQ7v3B7xT3/3gH63TbfdMAmYRyd8/ugJ52Mz3qIoyrNYwI8+fIjj2OzvbtFshMbBOeqw +M2jgey5hYN+o4tfVXEtgvlhxdHzCZLpgnZj+6IenE2arjKww/eNtx0VYbrkXvsXhUlNTU1NTU/MS +Wm/kZlV5Bm0y7IYm+VJKZrNlFfhergoc12exTJCFJGxFDIc9lC7PoJZCS1PxLfMMy87NmUkrtMwQ +QjHqh3z8cIe7e322hh18z2O1XnM1nXFwdM6zA3PeNe3mbHa3h+T3C/zSJ+o6N925ovz/rw82ZVlu +1OxOz/nkt5/zye++oJCKbrfPaDRCa8Vo0KXbaVbVZ+vzCy7GY548PeSzL55xeHxOt9ej3+vz4P4t +wsBjNOiyvdUnDDwm8xW/+ewxRbYq5Wk9E/gu8pc6DG0k5auqXl1gWYrQF/RaHlv9iL3tPqN+k7zI +Wccxz14c8evffM54uigD3zaDfpfRoMut/W0c22YnML19B90mzWbI3laPve2OKbbIc84vLvntp4/4 +3WdfIJWgUNBsmsD99taI+3KfRiM00sDCwracUlVn069Yv7phyvN4GfRWBZocLQWWthkNGvzo/X3e +vTui2QxxbBPEPz455+D4gsOTS07PJ7SaIa1mxO29bVzHYtBr0+u06LSb5FJzfDrm+MRmtlJkeYoq +UrR+2Z9cFAWL5YrL8RWffv4lv/n0SybTJUo42I7Pf/qbD2hEAbf3RoShTxQFjAamT/tyuWa6kiTJ +CsdroGSBLiWAvz1kUN4zWuE4FlHg0gg8XMec3aUyvt4kLUhSRZJpAt/CKfeHZbulxPnNM6sog98W +GhtsXc61RmjT0qfZDHnn1oD//PEerUZAs+GzXq+5uppycHTC88MzDo/PyHJFp22KqbKsoBmF9Ptt +9nZGjIY9Ts8uefHigOViyTzOWSdzXL9RSnYXX1EKK6RktVpzNZny6Mvn/Pbzp1yOp0gtkNriH370 +PoHvEPq7BIFLpxWZee5HTGc+V4uCOM24uJxQZMavGvg2u1sddMO0Y12tYp48O+Dnv/oty1iyjCVZ +IYwEu+Xwj3/7Lt1WxLDfqqZto+iVZTmX4wlHx2esk4w4ybmcLDkfT0lSSS7NLzcaAe/cGvLPf3ef +djOi3QqN+uZ0xuHRKS+OLzg6viROc9qtBq1Wk/cf3qXRiNgedtnZGrA16nMxnnF0dMx8tmCeFiTx ++roj1x+56psy8K2KjH7b5wf3R3z47jb9dpMw8FgsFnz55BnPnh9xMZ5wOZ7guh6e77E9GvI36fu0 +GhGNRsSgZ6qS97babPdDZJ6yWqUURWwSlDZ+AV1qJgiBsASr9ZqT0zHjyYx1UhCnBSeXK6bzhCTX +FEq8gQ0mqj8KqSmUYhVLVklBnGQ4jkl0ajUafPzhQ+7e3uXw+JyD43NOzidcTVaMZ2tmi5QizUgz +idIOEsf4joTAFnYZ+BblnCV4kcetnRb/+Df32B526Hdb5HnG2fkFn37+mNPzSy7GU+J1gluqj33w +3jsIIPA9wjCg3+uwuz1mZ9Bg5mqaofMV98bGz7Jax5ycnnM5nrJKMuI45/BkzGQes0okeroiyQq2 +hn3iJMdzHPq9Nr7v8vmTU1xbI/MYlceoPMFxPULPpl3a6lHgY5dtOparhOlsweXVjHVcgOVV1fDX +67h5jG++e8x+8mzN3laTv/vBLjvDFp12hCVgvY4Zj6c8fnbAl08PObucloWFAbs7I+7d2WWr36GQ +EssuFf50gWtr9rfa/KeP73F7b0S/10Iryfn5BZ9+/gUnZ5dcXE5ZrWI83yS7vPfgLlorwsAnDAN6 +3TanW1N2hg0urzymK00apyjLRSu/LKjQta3/hrwVge/NwSxJEubzOfP5vJIw9n2f4XDI7du36ff7 +X9vbexO4nk6nLJfLqqf3zd7XDx8+ZG9vjzAMsW27kvPe3d1lZ2eH+XzOBx98wHA4xLZslsslq9WK +09NTzs7OqqC34zilzJCZvjzPq77CG7nrjfTypkp3vV5zeXn5UjX7Btd1GY1G9Pt9dnd3uX37Nvv7 ++0SbbMrhkP39fba2tmg2m/T7fU5PT2k2m3ieR1EU1dxdXl5ydnZWVUevVmtOT0959uwZk8mEPM/x +PI92u83W1haDwYBms1nJX38bJig5ZTabcnJywvn5ObPZrAqmb21t8fDhQx4+fMiDBw+4d+9eFcgd +j8eMRiNOTk6q/tH7+/tYlsXJyQknJycvVcpvel87pdxPnues1+uX5nlTuZ5lWfXfx+MxR0dHDIdD +8jz/1jFt2FTHb5IvNj3UNxLhOzs7/PCHP+Thw4fcunWLvb29qu93u92u9sFqtar+/vj4mEePHhGG +Ibu7u/i+X1V8dzqdqq/6Zv2WyyVxHLNcLrm4uODg8IDJZFLdD0BVyb9YLJhOp8xms6rf+sXFBVdX +VyRJApge1u12m16vR7PZxHa+WWo2DEP29va4f/8+H330ER9//DG7u7tlzxmLg4MDnj59ajIBy3vV +HDCvOD09JQxDgiCo1nOzlpvA+WYtN/L7N9fSsix8368q3pMkqar4B4MBaZa+9ppbrRa9Xo/BYMCt +W7e4c+cOw+HQSHM1TO/uW/u36PV635gE0Ww22dnZYXt7m2fPnr20Jza9yWtqamp+H7zyGRophWOb +ntFKa9ZJwnxuMv8Xy4RM+/iOwLE9sJxS9s1Co1AKZouM4zNJM3AYdAO6TSPFHIbGyHEdh3YrYtBb +M18sabeaNEKfJE05Ozvn8ZNnPH9xwHQ6J88LLE8jMO07Qt+lEXqEgUvgOcaQkNIkAJ1f8uz5AdN5 +jBYmQ38yj7mcrrlapFxOYhqNiBcnE9apRGJ6Qwthl1nPb96HSAiwBbg2NEJTtdFtB2X/upzpdMbp +6RlnF1PSXBCEhclQXsa0mg0c26ERhbRbIe1WRC5TtFAm+F1kOKowWdH69WF427bptpvs722hsGm2 +VoyWGX7QwA+b7G51uX9nxDu3Rwz7LVzXZrmKOT275NnBKeOrJVLbCMc1/aNt1xicSr/hFGhsW+C7 +giByaHgOoWeZKvpN4HtTWVJqh2mgkIpCapK0MMZlWrYusW1my4SD4yt+2/bYGbbZHrbxPYdOq0kY ++FiWTRT4DIdLpvM1s0XMYpWxWKUs45x1IlklOatYsU5j06/SNkFWWxR4HrQbLoNug+2hqUJCq+r8 +V+QSIQS+5yKVjeu45Rlak5Xf+VEU4nsurWbIoNcgDAJ8333JXzJfLDk9Pef0/JLDY9OvbrHOyJVF +VsDVLGOZGXnJYq1RIme+KkhSCRp8z6PVbNBuBLQij8UyQ5Aj8xjPsem1IgatgChwEQiWqzXj8ZTz +ywnLdUohBaFvE4Y+3XbIqN9iZ9TF91wEuhyvUbLZjNe2LRzHxS17rGW5keqMwgDPdWg3I/rd/5+9 +//yuI8myfMGfmbkWV+JCEJShMlJUZVe/1zO91nRX9x//ZnrqdVelzgzJoIK+WrgW88HcHQDJYJCR +WVk51Ti5kGCQgF83c3N3O2efvbev8xXrdlq22Ww5v5hyfnnFydmUk7Mpm11CXimyUrDYpGwSQV7b +1NJCmbqIqL3h5S3p+7u4i7u4i7u4i7t4dwhRI6kwhN6DToY+/Z6HaSiyLGO53nBxccXF1Yo0B8f1 +WW01e6sFNn3PJQxceoFLVmZARVkkyEK/o5UEywTfUYx6TgNAaJnYNCt4dXrFl9+84OR8xsV0y2Id +aYsdpcgrU9d2+iFSShzn9ab++q1/BK2U8+13z/ny6+/46ukrXry6okKxjhVJafPooWbJ3bSGvLic +8ts/fMHTZye8PJ1yNd+yTQXbVBH0R2xjbXHoOg5SSnqBh21KDZi2bPfvkT1urXtaixYpwDYlvmPq +XMQ2UUqQZhVxnDCdzXn+6pSr2ZoaE6TBYpMwXUTM1imzVcLBZMHZ5YKyAsu0sC0TyzSIy4IsL1it +Nrw6OeObb55RoagwcP2A+Srmcp6w3GRcznZMFxuuFhFFbVCLSksKt366b43GkqiqUKrGsSH0FMPQ +YTz0CX0HJQVRkvDqbMrvvnzOq9MZs2XEcpPgbis8NyerLVwvwHUdemFAP/Tohy7745Dj/QHV5Y51 +FFGWmVawuhGb7Zbvnr3gq2+e8fV3Jzx7ecEmypHKxbDhYh4zne/ohSFKGviehec69AMX37PYJCll +nnZs8rqqqNWHgJY1SgosU2JZ2l+7rqEoSuI4IU4zirKibnJEGh/tjun95t3ItXiRVugSosZUJpZS +DHs2w77LaOCjJFRlyXyx4qtvnvG7P37FbLljuoypkSx2gt5O4vorgnBBLVq1AI/xsMfxwR7rbUx2 +uWOx2jXS3oZunq6rW2e120Y8e3HKN98+49sXFzw/mTX5sQHC4HK+43K2od8PGaseQaDXc893CTyL +bVxRVTlJkrEUgvlK51x5XlAUWictLwo225jpfM0mhk0MWSmRskKoilVUkZe3r83l1Yzzi0tOz6ec +nF1xcnZFUUnyUrFLC5brgqQwUYZJYMB4GLI37rE3ClFSUVcVi+WGb5+d8rs/fsNsGTFbaTULb1vh +rytsZ04QBFqKP3Dphz7jQcjR/oDlJiK7jFms40aBTHUNEX9u1E0zj5A1linwXcUgtJmMAsZ9H8uU +FIXOJ4uioKzKRl3MxjANLFPXoouyJElSXMfGMCSuq33o90Yhq02Emms1jroqbz2u4iTl4mrG5dWc +08slpxcLlquYrKzJC8E6KlhFkBaKCnUtR/2OPEwIqGtBjaSuFXFWcX614Q9fn3Fvv8/BpE/Ptwl8 +D8/Vz1fHthj0QpabiNU6Zh2lbHcZmyjTuXqcE6U1UZqTZXnTKGEhRY5t1oSeYtz3ONzr0wtdlISo +yCmaL4GuWVVOg2cZBgJBXhSkWYbrOlimSeA5DPs+Umrp8ps4UpKknF9ccX4x5eT8itPzKfPllryU +5JVkuUlYx4KksChjSIuC5S4nTnOKqtIqcL6gF7iEvo3nKIQoKfMYQ0r6vs14oPN4wxAkacJsvuT8 +cspmm1JUUivTSYVQRqfY9+YjpsZQAs9U9AKLYc9hMvQJfRtTiq7Z6k9fPeXV2YyzyyWrTYLl+Jh2 +TVSu2WYmo36MkqBkzWqbYpgWg37I3rjP4f6Qfs9DSaHHV+QUeQ51jWka2I6NaeoGAyGgLEqyLMPp +JNkdhj2PUc8nKXLqXUFdarsP1Sr/iTu58/eJvwngG/TGpwWIW1lnzQqxGI1G3L9/nzAMb/kN34wW +dGyZ4nVdN96TqgPPnzx5wqNHjzrv7iTREiXL5ZKrqyt2ux2PHj3i/v37AJ1E+NXVVXdc7dGn5Q9b +lnHrn93Kdk+n0w64bZnNeZ6zXC65uLhgu92+AXzfu3ePX/7yl3z22Wc8fvyYBw8edMCv67pa/tz1 +Gp/NjGfPntHr9XAcp/N2bsHSFy9edCBrFO065mwrsW5ZFuPxmEePHrG/v4/jOLc2uO+KNE25urrq +ZOk3mw1pmnY+1Y8fP+a//bf/xs9+9jP29vYYjUadDHUcx3z++ees12tWqxXr9bprInjx4gWXl5cd +c/z75nk6nXbS6bPZjDRNu3mWUnv2rNdrzs/PO9D9fSNN0863uh1jex1byfn/8l/+Cz/96U/p9/sE +QdAB+P1+nzRNSZKEV69edf7Tl5eXfPXVVx1TvW1cKIqCXq+HYRjd2gU64Huz3txqWHh9HK2U+XK5 +7MDv2WzWsZXTVIPELcN8b2/vvZjvvu/zySef8N//+3/n0aNHPHnyhOFw2K2N4+Njjo+PWSwWnex6 +67s9nU67tXSzIUII8YaveJqmzGaz7loul8vOd/7mPdN6bD969Kj7vNfD8zw+/fRT/uEf/oGHDx/y +0UcfdU0yhmEQhiG+7+O67jsZ30EQsL+/z2QywXGc7rmy3W5ZrVZdM8Fd3MVd3MWHhmh8uGqlOh+3 +sixJ4oQoismyomlEFV2yKDrZai3RVFYV26iAoqTnSfaHDsPApN8LOs+qMPCxbYswCNjPR0ipfYU3 +my0vXr7i93/4I8+enbLZrDtJNhrfM8tUOLZqfJe0T5xuKNMJxenZJctNglAuwnBZbnPOZwknl1u+ +eTHDsR1my4hNVFFUCpShZZel6rpRf3iboRniUtZYSuB7GkwNfe1RvttlLFdrLi6uuJqtqYSFXwnW +6x2L1YZe6OP7DqZp0ms64bNCEBU1WRajrJS6lS+7KRV+I7k1lGSyN+azjx4xHo5YbRPiJCcIQvwg +ZDzsMdnT8oCB52KZJheXM7559oo/fPGM8+mKEhNl2kjTRijt1Uz9/oxvQwk8WzEITO5NXI4nHraJ +ZmBUDVuorlpzcaq6Yhdl7OKE86sNL8+WrJc7astBWQ7rTcK3L6YURc5PPz7AsU3GA2394zo2B5Mh +YeBynOSkWU6Sl2SZ9otfrLZcTpecXy15fjLjxemCKCkRykEaFrYtCVzBIDDohzZh4DXegrohUUqJ +5zkYlkm/16OuQUqBaRoEvkdVacC4tenxXJtRXzdkvg4Ez+dLfvv7L/jdH77gcr7lahaRlgJlBQjT +Jy8gr/R7Pk1rSmAbl8RZ3vi8SVzHJgxchn2PzS4lzkuKbIdr+uwNXMZDF8/W3prr9ZaXJ2ecXUyJ +4wxlaDufQc9jPAgYDgJ6gfbguh6vwPNcDNOkH4ZUzXgNQxEGHlQ1RZ53no6uYzIaBChl4NjWjbUJ +i+Wa3//xC37zuz9xOd9xtdiR5Hq80vTJK0VZWZTCAGmhpIUyLF3o+IBmk7u4i7u4i7u4i7to96AV +piHwPYvxMKAfaFubaJezWq65uJwyna2ppE1RSb0HXW4IAw/f1Z6wuokvICt3eg+aJxr0LnNsQ+Jb +2k902HcZ9n0C30Eg2Owivn12yv/3n//A5WxHWppkpdF4FBcYVszBfMv+3hbHselX7w9MLlcr/vTl +t/zzv/yOq0XEbJUgDIdUZNRmxjqqKKprkCovCl68OuVffvU7Xp5esYlr4lxQyIyMlMO1VjjS/qT6 +HB1b+7dKWSNadmYHfL9txq//JCWYpoFjG52HeZuLxEnCfLnm/HzK5XwD0kYol+W24GIac3K54emr +Of0wwLI0AWkiroGndl+63my5vJry6vQUpA3SxnQSZusc/yrm1cWawdML0qzgarYhrzSQJRVIZTfM +ZKjfktC0vtRK1gSOwahv0w8sPFfnMFmes4tinp1c8Zs/fsfZ5Zq8timx2KQlxiYlr1fYtoWhah7e +P8CxLS0xPe7x4HiPdVxyNt1QFhnVa8D3crXhT1895Z/+56+YLRPmq5gcC8OWuIbNKqq5XEQM+hGu +Y+N7NrZlaAasa6Nk0oDeecP4rt5kuH/fNWxUtKQQGEo3aajG8irLcnZRQpKk2iNXXTd2i+8FvW9G +Y0vVzLlrK0LHYBjahJ6FYxnkRU6SZlxczvjDl9/y//m/f00pHCocpOWzyXKWcYIbrHG9KQjJ/UMI +fZtBP+T+/X1W24j5KiHPdghlUJo2VZW/pcEg4ptvn/H//r9/xWJTsdwWpIVEGAplWCx3FZfzHcPB +BtdxGAG2aeL7Dr5rYxja7iAvK0hL4rQky8vG21nndlVVk2YlUVIQpQZRYZJXClmbCEyyUlF1Msd6 +Xl6dnPHPv/od33z3isvZhqv5FmkGKCuglhZZKShKC981CTyDyXhAP/RxbYuiLEnTjOlsyRdfv+Cf +/vkPFNgU0kUqm3VSYG4TbHeF7V5S1zUPjib0Ao8wdLl3tMdyE7PYaKU0IQykYTZy0u/wun7vR7Nu +LJFCX/9Bz2YYOvQDB8+1KQttN1ZVFaah6IUeju0wHAxRUiKVlg13HZu8yMmLkqrWVlSh7zAZ9bm4 +WiJE3eXaN895u4v48uvn/Pb3X3G5iJmuMqK0BmWDtChrRV4qKmFSCwMh3icP06oGtVDUQhFnNS/O +VhR5xOdPDlCGxDEVRuNLPxr2cRyb/cmYJM3J8pK0sSXb7BIur5ZczpY8P53z4mRBnCb6OSktXFvg +2zDwda7e6/mYhmwsCDKE0E1Uo1GfwA8oq1rXq6Sg3+8hhSDLsgaTqHFsowF0wXUsXedppmu3i/jy +66f86te/5/RqydUiYheXSNNHmgFFJchLk7JWZHlNnFdsoooo0c3jUgps22pAX5dBYBEXNXEeYUpH +K4IMfQLPQgrBZhtzcnbFy5MLNlGCUDbCqJDSQDYqaN93GSxLYtoa9B6EDoHvYJmKqq5YrTd88fUz +/q//8S8sNjlpaVBiY2Q1liPY5gkXyymuvcSxJI4F202CZbv0woC9kb6/LNOgau4vAdi2xWjYw/cD +ylLbv0klGQ76SCVJs4yyKKGqsUyTfs9n2PeYb3dQJ1o1prHVQN3xvd83/s2B79ZXuazKDvher9fN +5k536QwGAy1zbpody/qtx6mqDkir67oDc2Uj+dd2SrZgeOv3fXx8rDuDygrXcbBsi/l8znq95uTk +pPP6jeO486ZuZZ7fBsa1P3sziqJgt9t1x7opP2QYBvv7+/zyl7/kH/7hH3j06BEPHjx46zirSjNo +9vf3GQwGuK7byanHcaxZwi9fduB2C8o/e/aM5VJLLrae50+ePGEymbw32xvoPms+nzObzYjjuJMB +d12Xhw8f8p/+03/i5z//OY7jdOD1zetUFAXT6ZSrq6sOAG/nebfbdfMspewkv98GYMdx/MY8l2XJ +drtlNpux3W4/CPhuGcatLHySJB173/d9Hjx4wD/8wz/wd7/4O4TU66kN0zCZzWa3APo8z5nP57x4 +8YLpdNqB9P1+HyEkvV4Py7IaSZqsk63f7XYslton/sWLFyyXWhq0vR9M0+yuQ6ty0M7n5eVlByIL +IXBdtwO+fd//wTlwHIeHDx/yf/4f/yeTfe2R3srXVlXF/v5+t/bm8zlwDQ4vFgsMw6Cua6bTKVEU +EUURnudhGEZ3D3zfunr938qGadhdy/zt17Jdd//5P//nTkp9OBy+93W/eZzRaMRoNOoA/FZFYLfb +kaZp91y5i7u4i7v4UVHV1Dc2qUKAlBpsNU2DqtLSVq/7TrfPnaqqyYuKLCtJkowoSlBSIBqWsJbm +E7iOSy9UVFWlGbVFTrTbsViuiHYRVVlhNCA8aK+6ViKtPbeamqrWoKou5lxjxXUFuzgjSresNgln +lyuU0glfhU52paGQP9jNfzP0eJUEQ4BjSULPotdz8V1byxQWBUWuv6qqQpoCUxlUVU3cyIy7LZM2 +9JmMh+ziimyVEyUJZsM8uZZif5O5oaRiNOwhxAP2J9p/KS9K+r2QXhjieS6e52BbVlfEurha8PT5 +GV89PWG2qUFYKNNDGQ5SGVR1iSi7q/6OGaipESgpsC1JPzB5cBDwsycjQk9dSynWdSNlqX+vLDVA +PV9uoMy5vNJgLkIglMUuyTi5XBMnKZZp0A/dZs+rvayVshj0beSw2Svf2N/M5ktenl7y4pVFmScs +F3OKLCcvK/Iyw3BcAtehF5iEno3r2MRJTZLoZlHPcxgO+0jZKgBovzEpJYO+9m68Xufg2ha90Mey +zEbK83rhrTc7nj57ya9/+0cW24LVtqCSHrZfYfta3s0wHf0RZUElBJu4ZLNNidMMz3Ua1rfL3iBg +tYrI1wVpEmGbNZOBw97Aw3NM3ZS32nByesnF5ZwoKZDKwnEcBj2PYd+jH7i4rk2SQJwUICSe6zIc +9puC3vV4hRAMBwGmdaOBtykS90IfJWXDcL9RaNnu+O7Zq2a8JatdQSVcLL/G9kyU5WKYFtJwkIaJ +ocxOBlE30PxbPOTu4i7u4i7u4i7+/y2u96BKgGsJQt+kH/r4rnO9By21kk1V10h0U1tV1cRxQprm +DZNQAy+TcZ9dUpNvNGNZGjaVmSJtWzc3hhaDwKYXuJimQZKkbLY7Xp1N+eLrlyw2BcoZoOxed37B +Jme5TdlF+vOqbs/ww3vM9WbLq5Nzvv72GXGhSAsTadmUqsTYVUSJVpdqoyxLptM53373nLPLJZX0 +wQyozYJSlayjiixr5k1JVJPLKCVQQvswa+ZweQP4/n4ATKBtaaUU1+q0DdOzaryC9W5Q+3tXtdDA +YLJluU04n26wbIvDyYiD/RFF4+Hd5i9lqXOaqqo6+epa1KRZQV5F7JKa+SrCPJ1RC0VZQVXLxjdX +ds28b53jGj1eKpQCz1GMejahr5W0hGyUMzdbzi7nPH1xwdU8wfKGmK6n2eKUIBMCd4lngu+5HO2P +dbPqMOT+4YiXZ0skJVWZUpUtOKfndLeLGrudb0lLk6Q0EaaJoxSytIhSWG0z1ruUcVYihW6O9n0X +x7FRAqoy63yt39ubub7+g2iun7phrZgXBWmakqaZBr4bNbBWhvj9amsN4xuBbSn6ob5/PNfEsgyS +NGG3i5nOFjx/ccY33zzHCvZwggmW5xOXNbuyYjBL6YVrPNem39M1ziDwuHcw4Wq2wnl2QZUnVKaj +mb9vadpoWftfff2MDI8cl1p6qNrAkDa7BJabhNUmZpLqZmvTUHiOpZ8PKqKuK8qyphYVaV5TFDXl +jTVe1ZCXNUlWkxaSrFTklY3EQAqTsjaa++B67i6vZnzx5Tf86etnLDY5y22B7Y+wfTDtUPuWKxvT +tun3HcbDHmHgYVkm2TYnihOm8yUvTy/45rsTTG+EE5oYtouQIJOScBYT+At812DQ07Vl33M5mAyZ +zjd892pOlSegLBDiLyp1Xtc1StQa+A5NeoFu3nAsk6gqKNIcpSRBoJuqhVTI9n5t8q0w8G4d01BK +5239EM+1UEJQVeUbdmhxnHJyesnv/vQN803FOpFklYVh+Zi2QJkKaeiG/5ue9T+8tEWj0KWbNy7n +O7abAkMpQt/BUgrXtfS5SYNeGDLoaxusm4TQ7S7SUugnDqIu2KzXRFFOVlbkeYZv2XiWRd836Xk2 +nuuglUgK6rrGbpjkvTBs6jaSVlKuF/pvKItohTYPIcCxTXTZQD9o4yTl5aszfvv7P3Exj1jtSpLS +wPYKLM/AsByUYSGl0mSEKmObNA38UaqvqW3he5rtPOi5lOuCbRxhyh7Dns3BOCD0LITU6mznlzNO +z6/YRhlCWai6vf6Gft688czW52oZCt83GN54VksBSZqyWK558eqMP375LWlpYbhjTNfBqCUmik1a +US93KAmeBa4tsI0Ky3aZjHwG/RA/8LCUIC71O9CyTPq9xn5WqVvNEWHg4b4xzwaB7zZ1kfTaiqWu +dMXoL+gi8O89/s2Bb4C8yJuX4bWcs2EYnQ+1ZVka9Fbff7qO4zAejzk6OiIMww5oy7KM5XLJr3/9 +a+I4Zn9/nzAMCcOQXq9Hv99nOBx2X+1nl2XJZrPpwN2Wzd0C3fP5nCzL3spAb6XH2zGVZdkBvi2r +5eZGQgiBaZp4nofrut8r597+rJKqk2U+Pj7m6urqFsP4m2++odfr8ejRI3a7HZeXl7x69YrVatX5 +pu/v7/PJJ5+wv7//Bjj9zmvVAP7z+bybF8uy8H2f0WhEv9/H97WXtZJvZ+crpToQVm+op11DwOvz +3DKL39bw0M5zlmUkSULRPLhbz/D2vz9kbLvdrvPIbsfmeR7D4ZAwDDVQ/RaPbNPSgPb+/j6+76OU +6s5rtVoRRVHnva6Pee273ev1Os/s1qu+ZY6/fPmS1WrVNSz0+33CMOzkt/M8Zzab8fTpU169esXF +xcUtxrfjOOzt7TEej98L+P6hkFJimmYnfd5ew9ebVlrf+fbfWnZ6y6B/PVq59/aeaRsW2t/XCWb1 +vedk23Yn2X+zIeFDwrIsgiDA87zuHNvPb+/ju7iLu7iLHxNt0l+VFbat9za+53J0eEBZCWarlNkq +YrGtSKqqAS2l9j1rJJosW3K87/H4wOXRUcjDe33CwGY2m/PV19+QpAVFWaKUyWQy4nAywvdcbNuk +F/b4yU8+xbZt/vT1c/709XNens4ohaQsU9I0Yr1xWCx37OKUsqpwlInnuuzvjfmP/+EX+L7Hch2R +5JBkFXFSEKUFUZwSRRlxWlAhqBBaEvADN+N1w142TEHPMdkbWASegSEb0L/WjWgPH9zj//iPv2C9 +jZHKxvN7fPT4iND3ME2pGwekot8LOD6csI1yVtGcqki0VHiRXUvIN3uEW3syKXFdlxrwfY881wVO +x7Z1YUhKiqIkSbecXcw4v5zzzXdnPH01Z7oqSCsLYboYhneDfZs3SeT7haBpADAEnmsx6HsMAksn +OjeKh9fAt/67qqpwnQVK1FRlBrVm4pSVJMkFq13N1y+WbOOCcd+hH5j0fAvfs/E9i9DT69JzbSxL +qwi4rsv+3gilFGmuZQyfncw4uVgxX66QPYlne/iOgWHoMRrK0J32wz5SKfq9XtOMegPklkL7WDl6 +T2TbFiCQSmJaFoZhvPE+r6qKvChJ85IahTRNpOFj2iGGHaJMB2XopLGucoTUhdzzqxWBa7A/EfR8 +h17gcXQwYrWJ2MYLVqsdrgn7ez32x308T3tnrbcRJ+dzLmcrkqxEmTau4xD4HoHndIwkZSgcx2Y4 +6KGkoNcLAHlbElOA6zh4rqVtYWwbhN4Tt3uqN8Zb1+RlRdaMVxkG0vAw7QDTacZrOo0/omoKLaqZ +63Yl3cVd3MVd3MVd3MX7hGkIQttir28QOCaGcXMPqnhwfMh//A8/v96DegEfPz4kDDxMQ4O/AP3Q +597BmE1UsInXFHmEMmzKIkWicCyDwLOwbYUQkKUZq9VGe6FGGSUGQimUqd/51Lo1UhkOQlk3wIkm +xA+rKgmaRlShkLLxVzYshHIQygapOhhVQ2oCIRVKmSjD0p6pymmYz1bXZHf7M7ixB6mv963fU5Pr +PgdBUcI2zllsEqKk0LUz6eA6DuPRkF/+4qc4jst8sSXJtFd0lGREcaabceOMJCuI4oTFasdiFREn +GVA3LHyPjx4/5B//X/+Zo8NDkqwkyWqSrCBOK5KsYNuAL1mRU1WSgoa9Kd9DRaduwTmwbZPA1aCc +FJI80/7bs9mSOM61So8lUJaPaXmNdVENUjM/l5uEKC0oyhrH1uBcvxfohtC6oirfAkwLoXNGaSCx +9FezTzQsD5RFhaS8cTmUkpiNf7wQ3GDodkP6gNAN2FV13aigT0silEI2jd/teug+5kMAdlFjSG07 +5DkmlqFzuThOtBLmakWWFwilfc2V5WFYAabjY9gBJSZxWrNNKrJc506ObRGGPoHvYRiSqtSM96pq +6/bVa6MUzZ7dRAmLSjjUysWwXAzTRUiTqpYa2G7n2WhqqI13czc3HfB5w6i7u5yiabhQyNpGVSai +aXDVgO7te6+satK8JC9qkCaGbWPYAaYdauBbGQhlYjtOk/fZmE2NPU4SlssVy/WWLK80SG66KNPD +sPym6cOkxGQXl6x3OUmm58W2THqBTxBcz59AUiuzuf//fHSubhjfommM7/k2nmNgSIWQQktFI9if +7GFZFkmSdo33opkn01Sdf7LnOhjKIJPaxtaybQzD0j7Nsm3KuLn0BHklyEtBLSyUZWPiYlie9oOX +BlKZusm7Bb15n6b/dj1pWfisqqjTiudnO9LiJU9fTen5Fn3fIvA0GBx4Nr7vEHgOtmVqyWzLYjzs +o6TsPNwDz+bV+YqzqxWUNY5l43smlqkbo6VQjS1qqGsentsQKG5jOI5j4joOrmvjNSRQISWmYWoJ ++dcwn5qaoqxI85KylgjDwFQOphNiNbmrNDTxsi5zqtIgzQWXc90UNBn3cWyLwHM42BtwNRkQpzOu +ZjsMWWmp9smwkWqX7KKEi6sFZ5dLojhHKt3Eo++hBlx+G+7dSp3bJoFrYVla1j1JE+aLFdOZtjkr +a6mf16bb3E/6i6aOJEVFKUuSosQyBZ5n0++HuI6NEgJDGdh2TRiG3Duq8by2MUN2a1PPs9WsT71G +hZRIqWsEpmXrGoCUlO/dLHQXN+PfHPiu67oD/FrWawsEW5Z1C/wGvvci3wS+e70eSqmOQdt6CX/7 +7bcEQcBgMGA4HLK/v8/h4SEfffQRP/nJT+j3+4AuQhVFwXq9Zj6fd4BsC+S10tTL5fKt59OCr+3v +6M6Ma0C2fMtGpWUVu677vaz2dvxSyQ74fvjwYScJ3cqQf/fdd9y/f58kSbQ358UFJycnxHFMlmXY +ts3h4SGfffYZBwcH75R/fj3eBnybponv+4zHYwaDAb7nfy+Y3rLwgyDAdV22220H3r4+z2mavvc8 +t39uv7dr6UNetK1PfMvabj3EW1A/DMNbgO/NsCyL4XDIwcGB9tJu1lAcx6zXa+I47ljblmVR13UH +euuHX9Wxi1tv8ZOTk84TvigKbNvufKyfP39OFEVkWcZsNuPZs2dcXFx0EuMt8N16xLf+8H9utAXa +m/PQqhDEcdyx9Fvgu6qqDtBumwDedS1vKje8bxNDO6dBEOA4zvfaIfxQtM0nN4Hv9rpcS7vcxV3c +xV18eJRFQRzFZHkGhM0zS3fQOo7DbLllPl+hLrZcrkqSONJM4dJumLICSykeHPT5T3+/z71JyCB0 +qMqCL7/6hl/95o8s1xvSvEIZFp998piyKDk62MOy+gz7PbyffsbHHz1mMPwDeVERRSnrRLJJMtIk +ZrOJWK4jokTLQpuGTih1QSDg737+E5arDYvllvlqzXS64mqx5vR8zsn5nF0UUwGlUFBXKHhPpncb +eq9kGYJeYLA3sHXRUerEoq5qDCV5/OgYISFNtXeVZVkc7I8JAwdDSWTDXO/3Qo7vHTBd7nh1saTI +Y8oioSwzZGmANLvPvBlSClzXwbHta7Y7Wr5PStVILqas11u+e3HGH796xrNXM04utkzXBYbtomwf +ZXpIQ1+/6gNA73bapBSYhiTwbIY9n3G/AXRfmzPQwHdVlZRlge9o/8iqumYpVLUkKSR5VLN9teTZ +yZSeqxj1TMY9m/HQY2/oszfqMRn3GA97AJiGoaUQbYNB38dQgtAzMZVgtVxxEa9RePiOxHcMLEM2 +e3hdRDNN3dlcVvVbV0GryKRl6HRDgVJGs88w3g4EFyVZVlJhoEwXaQcYTojp9LSMqGFBXVOWOYKC +KCk5n64IfRPfc+kHLv2ez9HBmNl8zen5jDLb4FqC/XHI/l6foAW+NxFnl1OuZiuSVGKYNrajgfPQ +dxpGup4nQyks06IXhpRV9Xa3QimQQjPqdRFV2zIZhqnnQL5exEID/VlJhYk0HaQVYDoa+JaGfUva +vJOMvAO97+Iu7uIu7uIuPij0HlTS8y32BjahZ6IakKrdgz56cA8hatK0QEj93j/YH9ELXAxDdnZG +/Z7PvcMJ02XMycWGMk8ozaQBvm1sSxJ6JrZpIKUgy3NW6y1X0wW7OKPC1NLJlo9hhw2ApBvw5I13 +/u03/Q+892ULjCoNigq7AdI1mI009DHqWkt5C5BSaeBNWQhlUxsWUloIaYEweEM5qQHgRStP3clF +vSOEBv/KsmYXFyzXKdtY+70qQ+EpD8ex+Y+//Cmff/aE1XrHcrVlsdxwNV9zNVtxdrnk5GJOlKTE +cUqF9mqOE12/sS0LyzD5+KNHHB7s8//8f/wDi8WaxWrDdLbiar7iarbk5emUNNkR5wUFJmWtQUvB +DzUW1B14LaTAsRSBb+HYmgGvPZsj5osVcVo0ylAmhuVh2EEnB13LiiSrNPAda1sc07DxPId+P9TK +QFTUZaHlt2427jaNCrq510BJR+8bTRdpughpUdWaKd/+lpRKS8MbWgVM5w035ak/DLRs2flVVdOq +8LcMcNmxSOvr47/34a/n1zQEnm3g2tcNt3GcMF8sWa42pFkJUqshaUZugGGHGLavgdukZhcVpI2f +fQv+BYGWf66qXOdQZUFdl03TyY1TFaLzD5bSQkmX2vAaoFgD32WtGwzKbg4khqEwmnmmZUK397EQ +jeLb7XtCSN3IoJRJJe1O1Unv+28vyKIoyfJCA9/CwrScrlHWsEOdc0mF7did7LptaTZskqQsVmtW +qy1pUSMNRwN9lm4YEFIzmctaECUVm52W2gYt3RwEvm5ANwRVlSOFulZ6+AsA322+LqVmwfYCG7dp +TGprwaZpYtkWo+GgUUZ77YYVosnldT4vpUAksiFc2pim1QD88o2bva6bnKyEWlooFSBV0MyR0zQd +qxsqBh+Q9zcEhxrIypq0qkkutpxczHGtmlHPYq9nMx74jIc+k3HI3jCkHg+ofbfxMjcZj/oMBiGG +qfAcA9dSRFHEy5cbRGXgWH3dbGUpBGAYCmUoLNPE9z0O9ydvP71mzoTQjV2iqYfoXL1ANj7gN+eq +HOXMawAAgABJREFUKCuyoqKqJdJwMVXTgOH0kKaNUiYIockIhUGSw9V8x8n5Atu22Bv18D2H/cmQ +w7m2WyuzLZYqGA897h0O8D0PIQW7KOFyuuL8ckFUWEjlUIumUaJtYnj7ksJQAtdumtCa5p8kSVks +lsxmC6IooawkwrBRpt+MIUDZfnONa+qqoiwziiIlFAaeZzLohRq8brA7W9lYhonnuRzsT7rmsnfN +s5QCpSRGg4uqZm0KpNa++JAS21382wPfcA0u3QQqW8/m9vsPdTVYlkUYhuzv7/Pxxx/zs5/9jDAM +Wa/XnWd4C4JHUdQxZi8vL5lOpx0r+pNPPuHTTz8lSZIOsGtBN8uyOp/gli3+Lnb2zej3+xwdHXVe +0LPZ7Na/t1LsUsobEpxvDy0d6XF0dMSTJ0+Yz+ecnp52IOiLFy94+fJl50++XC6J45i61h0mvV6P +yWTCvXv36Pf77wTa33atWiZ9Oy8tC7htUFANA+Zd0Y61BYhvznMLQNq2TRiG9Pv992alh2HIvXv3 +uH//PoeHh51X8/tE24Tx+tgMw8C27c5/+vvGY5omjqO9RYUQbwC3rZpBe61bQL3X63Ue7W2jgpSS +s7MzZrNZx9Lv9/vcv3+fzz77rGPyF0XBYrHg+fPnndR6660tG9ba3t4ek8mkkyz/c6PdnLXX+PXG +jpYd3TZF3LyWrS/9+0Tb3HHv3j2Oj49xXZfFYvHO9fTndD+167H1qGrH1q75qqp+9LHv4i7u4n/v +WG92vDo5Y7laMegPGA4HOkkMfMIw4P69fXZRgrIuyKsZcRohqpQqjzEsW3dTewajvsPBOGAQag+i +dRxxcTXj66fPWK4TKgxM2wPlk5Q2823J47Tk+BBc26Tf73F4MOHB8SHz5ZpX51s20Y40jYkSl+Um +5uXZnD98fcq9/T4938b3nIah7hH4Hv1ewHAdMOz3mKw2jAc9xsM+51drLuYRV4uYqsyoy5SqtLUs +YF2DqKEW37NJb4tjFZZhMggsJkOP0DeRqrWq0eoek70RhlIUZdkkByZhqM/NtqzmPV0z6Pc4vnfA +6eUC2zyhLjPKItVFR2UggbrWBaSbUVYV69Wa1XpDFCfEjY9WjaSuBWlWEMUZ623M85NLXpysmS4L +4sIAw0c0xQ9p2AhldEWND4myqsnyil1ccDbd8u3LKZczo2FiNA2VNwpGVVmxWK2ZL9ZcLXZNA4Td +MAM0K6CuBUUlKArN6CiKiqysiNKSVVwyXef0pjG9YE0/dBj1fcYDn0E/YNDz8T2XvdEA6pqr+QrP +MaBKkRQYEpQSSEHHvIjihM12x3K1Y73d6QYOZSBFm7C1aX4bgqKouJjOubicM+j5DPp9rmlM7Xu5 +uWJCM5ylMvWXYXV/rusaCYhasEtKLuYRg96Ogz29N+oFuiA9m695eXrJdGYQeCah72JbBkWRs0sL +lusdi3XCNsopsJHKxjAtTNO4xdpIkna8EcvVjtVmpwvTjZ1AO97X1Z7quub8cs755ZwwcOkP+rfX +SntbNIwQzda4Hqcu0DReiXeA913cxV3cxV3cxY+K9vVsWwaDnsH+yCX0raYA/doe1Gj2oEiUMggD +jyBo96CaGdzv9bh3tM/p5RrHOrulOlRXBRLNXG2xrrIsSbOMOEkoigpomLvNO55aQKXZs/LmPuGD +X/kdMq0BnrYR73UG+c3fEPKaYX7zi7fv6dsdWwPlUr8LQG2ZeQjKWpCXNdu45OxyzZ++PmW7jemH +LoFvN7VYrbrTC30GvYDhIGRv1GM86jMe9bmYrlhuMlablPlixXcvLxj1XXqBSz90cWyDXi+g39dy +s4P+lkG/x954wP7ekMFA//nscsXFbMNsEVGhAbyqyDuJ9HesJIQAJQWmYXSy7VVZk+eFlvsuq0a0 +WzYMUXXtJUWtAba8oqzqxtpa+2abhkIpgUCzkL+PFCIa4E3WqlMDklJ110wLbDVWV4IbjZOvgdLv +u6JEdyDKCtJcN6i2alSGoXRDsWNhKIGghLqgrnIthV8bjZ3grWlsvrXKXC2AWkJtosS1LQFAWen7 +J88KzaZswWmhtKRwMw81kqLQjNTWultKzXo3DaMB/1ulgrqz+nrttJomA3mDUSq7poNb83zj56W4 +zn3apup33KXNrSE6MFUrSd3877eRiZrfkQZCtLmC1SjI6XNWUmEo1Y1Xz19FluVked7Mn+zmT2qD +e4RU1NTkZUlelBRV0xje1MuNG80TnYLBXzRanEg3hhtK31tFUTYWqAmL1Zblak2SZs3z09TPuO9h +yUZRzNnlnIvLBfNNQlFJLGVpBa035lfbHiCUzjXbeZXWrWvy4eXodrHrtVbVgqSAutQ+71WVkWY1 +67hitk45n0f0ghW9YMao7+uvgcegHzLoB4wGPcqiZBclfP30JaLOEHWOIWtMJbprnqb6fbPdRqzW +OxbrrT5/de1P/mbuqtfy5WzJxeUCqAnDXmO5cS070sr16/eYQnS5uv4uGkXDtqkkyWtmq4jzmclk +PKCua3zP4WB/zGqz49XZFaFrEHoGPd/Bc2yoS3bbiPUmZrmJ2ewySqVVTOpKNwcJ8WYTw815l0Jg +KNE0UDT3QvcuzijKSltdtA1jytTsfmneOG5BXUmqWoLU6mymaXa2cWmWkcQJ212sCSSrjS6LSfON +OsHrMVusuZwuOb3asotLEGbzPH8PBZK7uBV/E8B3Cy7dlAC/CQS/T5iGSRAE7O/v83d/93fkec4X +X3zBN998w8nJCVEUdazUtki12Ww6NvTXX3/NwcEB//iP/8hgMCCJkw6Mb8NxHHq9Hnt7e3zyySf8 +9Kc/JQzD9zo/3/fp9XpIKVmtVnz33Xd/1pwFQcDx8TGr1Ypnz55hGAabzaZjRz9//pxnz55xcnLS +MYAdRxev+/0+e3t7HBwc4DjOBwHfr3up37xWb5Ol/MHjVW96s7uuSxiGjEYjPv30Uz7//HMGg8F7 +Ha8F9kejEQ8fPvxgsPf1dQi8VwPG63PQviTa+Xp9Y9qy3ieTCcPhkO12S13XbLdbzs7OqKqKi4sL +lstlJyW/v7/PkydP+Pzzz3n16hWWZRHHMYvFghcvXrBer9lsNh3wDdq3en9/n8lk8heROm/P/V1r +4yZI3ALeg8GATz/9lJ/85Cfs7e291+fYts1gMKDf7/Pw4cP3vtf+nHG9DqC36+Ft1/Au7uIu7uJ9 +Y7la8+3TF7w6OWc8HrG3N+be4YT7x4caoD2aaF8fabJYRyyWGwpSinyHtAS+bTLsmfQCLf1kmY0y +zXbHxeWM5y9O2UQlygmxXUVaL7hYC2YbDdoKavZGPUzTIAg87t87YL3ZsYlecXKxIM8kSZKy3iZ8 +9/IKqHh8POLR0ZDjw0EjraXlmZXSBYzhoE+SZjy8H7NcbTm7XPIvv/+OzfYZUZZQl3YnKy5Viagl +9RvslOvQPn4lpmEz6NkcjD1C39KFQSkwDJ20D4fDa5moJpG1TBPTMjGU7qava+j3NWv52csLHFtR +lyl1kVIVCaUyAdl0o99+thd5wfnFFd89e8HldMF8FbHaxtS1QYWW+46ykiQt2cUFUVKQlQaVtLXv +sumiTKcBedWNxOj9E5SiqonTksU649mrBUm0wzIqyiKnLPLO67srK9Y1UZQTJSlXi4goBcPyUYaN +lGZX1BRCdoyevK7ZppAWNctdijnNMFSNISssA472Qu7tBzy+P+GTJ/cIA5deL8AwTSYnl/iuQlAg +qZCi7jDXqq5Zb7ZMpzOevzrnm2cnvDy50pKDTTNAV0Rtrru4AWpv45RdFPPw3oTDg/Gbk9NKOdKO +STWd+9dFGkSNrCtEDVFSMZ1HjHouUZojGyny4/qAxXLDs5cnTC9t+qFN4GuGfhzHLNdbVptI+4Nn +NdJSGIalJT9v7Anrmma8c168Oueb56c8f3V5Pd6GldF1htftktBj3kYp2yjh+GDI4f64G2O3ZLpi +lwKaIl7H6JE3CpYftsbu4i7u4i7u4i7u4naYpmIYOhzu+YSB/cYedDQc4Hlus3e83oNalqnZWYbR +7EFDoOb5yQzbMqjLTPsnF2nnody6VYOWhy6LkjwvNYjQgc3i2prnL0jxamV4u++v78/rmz/Z8bev +PZlFWxN61/l8CONbaFCyFMRZzcvTBZKSq9mSx/fH3D8cEQYuQWONI6XEdWwG/ZDD/TEPdwmLdcRs +sebXf3zG7798yXy+4ItvTkjzkodHI54cjxgPAwLfxXUsAt/FavKiyd6IB2nKo4dbVqstf/rmBf/r +118ym15R14VWVSoS6kozyMXbxlTfHJJAE0fb/a0Gq8uupnQDvOwMzfXf13XdyJE3AKK4rlMJRPM5 +3zen4vqr3S+3LNR3SS/Xr//hQ8Dv9vMkRVmSpAVRVlCUDaO6ITD5Tf4qKBrgW39VVYmkoq7fBuRW +ndpBXVWIuoKqAKoGWtbnWLU15a4xoVm34np9t1Nc1TpXuYU7NA0uNBL9dTfHb5uHG/N5w6f8Nijd +zsv3TXR7X/zAHN/8sdae63svQ/v5OkeS4mbOIG/MhWaTyhvMZF3HLW/UPK/XUI2g0Txo1C9emz9o +GP3NM6S55/+ildP61kprGkr0f+V5znyxZDqd8fXTl3z99CXz1Q7T1PLzN3Om1yMvSrZRzC5KmU53 +ZKUkbHK3W5evu87XOVjbNNTaIFyXyD/kGd3m8VXXIgQ0TesWFRW7XJBtYR1nXCwyTLlFNfn6/sjj +aC/g4b0Rnzw5phd4eJ7DwWTEarMj8GykKJCibBTxrpfnLoqYTmecnl3x9XenPH1xpm3EDFs3Vne5 ++g1lhmaNRbHOXcPAY288pKrq14Z9fX/o91jbFNLmro0tlzSQdU2a1cxXMYGreBxlVDUNO3qPOEl5 +/uKUUd+h3/i6m6ZBFCVaon+zY73NiNIKZUuUaWn58eadKd5xPdrGn+umlOZdXFWUVanB+3bSuvex +eO24Ok9viQ6yUbFr36lxFHM5nXF2fslX377k22cnlJXEaOzZurX5FowlyUqiJGcXl6x2FUizsRpp +7u33lNO/i78R4PsvEaqRapBS8tmnn3Xgacu+XS6XOI5Dnucd0LrdbknTFCkljuPw8uVLJpMJP//5 +z5FSkuf57ckyDDzPYzQa8fnnn/Pf//t/5+Dg4L3Or5Xg2G63/OY3v/kgsPlt4Xkeh4eHJEnCeDzG +NE2KomCz2VCWJS9fvuTrr79mOp12QGjrbT4ejxmPx52n+YfG64xf4EZH3Ie/4l4HUlum8nA45Cc/ ++Qn/+I//yP379z9onk3T7KSv/5Jje9f43gZyv+14AFLIDvgej8dcXl5SVRW73Y6TkxOqqmI2m7Fc +Ljk4OKDX63Hv3j0ePXrEZ599xm9/+9tuPa1Wq07Kvv0qiqJjW4/HY/b29nBd90dd79fn923X7PXx +tmEYBq7rdsD3f/2v/5UnT56892e1suqt9/a/dvzYNXwXd3EXd/Gu2EUxZ5dXPH3+ivlqy2y5pShr +wjBg0NeM6fGwx2aX8Or0gulswSapWScxsjJxTZeeJwldA9cxUQrStCSKYmbLFZdXc3YZOL6JVXrs +8h2Xa6iFQejZDEIH27YYDUI812F/MmK53vL05RVUOUWlZRZ3UcLp5YokzdjtWguaomHf5liNJKOS +ikHPwTINirIky3KOZ0uu5iu++fYZWV5QV7rAp4zG44u6KVK85f3RJvR1jW0JBoHDwdin51lIKcjz +giRNyTJdcJJKSwe2SUJZQ5nmGKrCqmtMpf3wLNPURUrHQhPBC826UVbnp/U6s6GsSuaLJd89f8nz +V5ecT9fMljtqYVMLi7yUpHlNVgmU0jKChmlqH0ZLA81SmU1yd9M//PXvb05B+6ey1D5tmyinLDKW +KxB1SpGnumhalQ1L4vqYRamlxeK0Ji2l9qGyHO0Jpa79yqpSUAtJUVUUGUR1pY9VFpRlRpknyDpn +NvdYLAOoK/bGAx4Lgdf4og36IbZlICmhY55UXUVJW5tseHVywR/+9C1ffP0Sw/YwLE/LZbYA9Q2f +8nbrUDb+gJZls4uy1woVrzEfui7sJqm+mYxKA2pIsozZKme2StjFmsngug6WaXJ0uMfx4YjFfMh4 +GOK5NlCxiyKupgtW64goLckLsCztFyjbLvVuHdckScpqtebV2SV//OIpf/jyGYblacn7xovzerxt +k4AukpSVZvgrZbDZpW+si1uFu7bI0ibI72B93MVd3MVd3MVd3MX7RbsHcWyTQc9jfxwSvmUPKqTQ +Mq03fqmsIU5zDCWxajBu7EGHgx6ea6JEjaiLBvxuPYTrW/u46la9qeV76n+7iYDd2kX+iLJF23sn +atHtRa5VdcQ7fusGhPA239RbUb/jv95+AapGmSjJKs6nG+Jox3a71SzUsmTYDxgOAu0rq7QFUa/n +Mzb61HVNmmWsNxHz5Zpvnr5itdnx8uSS9S4l2sXUVUGcZAz6Pr3AaySPBY7j0gtDTFORZhlZlmEo +wcuXr/hG5hQ15HlNkSdU1Q9Y39XX3+r6Rg5wg0HZ/cTN690xjFti0Y15r9vj1bfAse+Zxua7QNRc +g083mP7fe9otvqWlht4Ll73+XH18nYeUxElBVlSNmqdWzfQ9B88xcCyJlFWjxJUhlEFVmrfBn+4c +Wrlx3WRrqBopdPNvVRUN05QOuJI3FaI60LqC+k359uuPqqkq3lIDvNFEcONvuj+1dV5E08LSfv+B +JfJaM8jbyqrXP1J366JdK6J++8W5ltm+kSPJtkH2NR/x+uZMtH228npObiix0Xy+6Fj39VvhttdX +5188M7k53BvzVxQFURQzX6x4+uwVv/rtn7i4WmLaAZbTQxotw1i9cbC6hrKR509iLWst1Zu+1QK9 +xpASUev5rG81H/+Y0baNAzVC6HUqZN2wpQWVUJSVoMxrkqymqorGEzunzFPKIuFg5DCfe2RZzKAf +8Mnjezi2jWPbjIZ9PNfGkDWiuweusYosy9lsdpxfTvnim+f89g/fUGLo/NWwX8vVr+8Z0Sg7VHXN +ZNxnuUm6+/DWhHWNN3qurmX925pE07BOTVZmLDc5nh2z2aXkeYlta9/yLMs4Ohhx73DIZBwS+g5K +asLGbLZkudqxi3PSAhxbYRoWlAVVVbzWiPJ6iLdWZoS80chy6x+vyQb6OS1v//et5/l1k0qW52y3 +Oy4up3z5zTN+/dsvyEuF6QQYlqfZ8LcUBm5UhISiRlELkxILhNM0JWiLgh8hL/C/bfxNAN8tOGY1 +EpV1XVMUBVLKN7yaf0guUklFr9/juD5GStmB1Ov1mtVqxWazYbvdsl6vOT095eLigiRJOk/p6XTK +s2fP8H2fPM/xfb8Dz8tSy2hsNhvquiYIgg48fhcL9uZLNEmSP1uSGTQbdjgckqYp4/GYMAzZbred +v/XZ2Rn/8i//0nlWSykJw5Dj42Pu379Pr9f7UZ+rlMJxHDzPw7K0rEfLpF+tVtrLuijf/3iGPp7v ++9i23TUctIz8sizxfZ/BYPC9EiVvm+e22+ZDx+a6LkEQdOeSZVnn092uk7eFfuFGLJfL7udaP+yb +a/tmtCzu0WiE6+rO4fV6zatXrzr2N+gmh/39fe7fv99JuLeMt8ViQZIkTKfTTqK9LEuEEPi+38mL ++77/3nLxPzaUUh073bJ0J27r8d36lDuO88HXEviL3DM/FEVekCRJ1zjQjqmV8f9zm1Xu4i7u4n/f +0O8k7Y01Xey4XGSUwqI/6NPr9XAdE8e22Rv1+NlnD1FK8dXzOdGLGaKMUfgocc2srWstxVSVJXXV +Jpo6qdDsV4UyTKoa4rRgvU1J0tbCQ2AYqgGxG0ZBVUKtO73jJGOOVrxYb7Z8++wc3zUJXBvXNXFt +g9B3uH+0x/HRHo6tLU5cx2bYc5mMQ0oSdnlNUqSo0mkS5vr780IBUtQgKxxL0Q8dxoMAz7WRAs4v +pnzz3XNOTs5Amp28s5TGLWnGybjH/aMJh5NB46Vm4Hsug0HIaBBQCIOiyjTrWxqNf1t1K+Opa8jy +gl2UsYsztnHJLlUoy0aaHrUyUYbCRjUScvYtqW3dCd6+L66LWVRlU0Co2r++/sDuZ+uusFVUkJc1 +UQpFWUMtKQuDuqyoa0Vd6WS0DZ2EQiEAAyypfddsy8RQEg1QlyRJRpyn2ndbXHfAC6lQmNqfsJYk +OSzWCfNNQpLeLvSpZg1pdn1NmhVkmfYiRAocx6bXD3E9l1ookrzElIJSGShhIoWBEMb13LRedzdL +MI0k+o+637rfV+QlVEXFZlew3qVstlFja6IYDwd8+vFjbMvi8YN7eI5DkqRcXM159vwVy3VELQyU +5WrQ2zCpaklaVGRFqWUoETiNBZLn+SAUSVZhCoHRjlcqBKorHr2VRdKB6Xfs7bu4i7u4i7u4i79m +tAxI1zYY9D32hiGBZyOE4OJyyrdPn3NyetFYyDSF57YRrQGD9kZawelof4CSRrMHdej3fIZ9nwKD +rM7Ji1TLzKYp+Q17PcsysG0tkVpXOVVZU5Y5Rpl1DY9VaXVqRX+rzfo3CaodEPZDp1rfgCFqSLKC +uqyoqoLNLubZi3N8zybwbFzHwrUVvmdxfDjh/r0Jgae9bl3HZhB67I/8rr57fjEnz3KuZit6oYvv +WgSehWubeI7JeNjj+GjE4b6u65qmSeC7jAch++Meq23Oaqvtkury++qcogN7dB5REqc5RaEZ/KaS +eK5DL/CxTAVVTllWlEVClSfXYImqsZTAcwwsUyGlzsfSNCeKEvK8agAj4wdAnR9z1V4X6H7PuMGI +zIqabZmziXKSZvya2SnwXJv9cZ+H98ZsUpNtlpInGzqG9k25/Qakq0rNCrdNQc+zCH0Lz6opCs32 +LHKdn1iWSRj4eL6LUtpnuioyyiKhzJJGVthAYmObFrYpdQM1kOe6lh0nCWVZdUpS18pKfwle5WtA +962GE/1V1bd/vq6rG/d9RlUL7Vus9H3xYwlnRVWR5hVJWnS1bdO0CHwP33UxlGjmTtuDqSJBSIO6 +KhCuwjYlrqUwlZ6XoiiJk6SR8UfbMwgFUvzl6rfNnFWNf3SSaT/zqq4bcqLLcNjHsmzKsiZOC0oF +VSFRQudiOjevr5s7uOE/LtAN010z9b92DnZtKWDKGkNqiy4tR1ASpxlxU9tvWebas1o1Vm2azZyV +gtU2ZbaIiJLilriGtlyQGEqDpLqBKycvKmqa51zgEwQhSpkkWUUlBJUyUPK1XL2qqKluNGK18ePn +Si9/bfNblIKoqNhEBetdxnYXQW1jGIow8Pno8X2KPOPB/Xv0eyFZnjOdr3j28pTpfEVeis5mTkqr +aTL/4fOqqpqsKEizgqK5FyzTIAwCemGAZSr9Lm6eJUUWI4Sikg3wTE1dlZRFSpUn5KlJktTEsU3e +4AmWaRL4HmEQIqVBklXklaBUElOZSEyUMK+P1zW1VI06g9E09zf3Vdf8/q+8RP+dxb85kiOEwDTN +zkO7BQdbtnUL5Gnfj3df3bqukUoyGAzo9XocHh7yi5//gjRLWa/XbLdbptMp5+fnvHz5kv/1v/5X +J4GepilpmjKfz3n+/Dnj8bjzxG6B7xaQ3W63VFXVAbLvAuVameSiKCiKAqV+2P/6faIFvoUQHfA9 +m820z2AUcX5+3o1ttVqhlKLX6/Hw4UPu379Pv9//UZ+rlNJen01DwE2ger1eE0URZfUBwLe6Br5f +n+fNZkNVVXiex3A4/KB51i8J8YN+6TdDSnlrbEKIZqOZstlsNKj/DuB7t9t14H9VVR0Q3DYJ3ATi +hRSdJ/3e3h62bXfAd5qmnRw/XLP7Hz58yPHxMUdHRwyHQ1zXRUpJHMdst9vuXmm9xlsQv9frEQTB +vzpwfHP+2qaIm9eyKIofdS3zPP/gJoYfE3mhz7X1m4dr4PttjQt3cRd3cRfvG0IopNLeYleLHbNl +gjBs7h1OOD7cQxDiOg6TcR8lH9MLPKL4jzx/cQZlhCRDSQ18AxpAa6SYbvqEdZ5mDeu4qiVRUrDe +avBSM0vVtSyjFE1iXVBXOsGO4owkLViutryocyQFlimxDEHgmQxCh/1xSPKzTwh8BzXs4dg2nmsz +7HkcjAPirCZdVZRpqv2+q4pa1d9TUNEbfCn0l2tJBqHDZBg0bAzJ5dWMX/369/z2918glIUwbN2R +3fpoN/HkwSF1VWn/J8/Bck1832HUDxkPeqwTySbJKUiRytaShdVtj++6qkizgl2csIlytknNLhNY +ysKyAqTpYUhDF1EakFszcbV/dcusaD2465vd+lV9O1/sMO8bvnp1RVVXmvVd1ORAnAOVpKoUdW3d +AFBvHuaGtKEJJuA4isA1UFI3HlRlRZGklNmOogTZSHd342iTqUqRFgXzVcxyHROntxWQhBAopZWW +6hqyLCPJCopCJ8WO7dALQ1zXp0YneaUSmIXCkBZSmEhh6uJtVUGtmePUJVLSeF5qb3K6sX3IDaf/ +TyDJCkGZVWyinM02ZbNLCDwb23IYjwZ8/ulHHO3vMR6PcF2Hy+mci8sZ3704ZbHeUWFgWBbScJDK +oqohyyuyvKCotCeh7dj0wgDf86mFHm+lBFWpdPEA3c3dSjVej7fqxlv/GcWDu7iLu7iLu7iLu/hx +IYT2ClZK4Domo57P/ijUdndScnU141e//SO//f0Xei9gOE2jY+u/rfdkTx4eUpTXe1DbdvE9h2Ev +YDwIWMU1WVxQ5ClplhLHOVmh65xt3chxbJSCuiooy6opuGfNfqlV+2ka6OofxpP/6vE6Hf2WZPQP +/WLLkqtJ0oK4ylitc16e5khKLFNiGwLfM+gFNuO+x9//XOcitmlg2zaua2vG/ihks0vYTCOmyxWz ++YpvnyssU2EZAseS9EOXQejy0cMDoGTQ87FtC8s0CQOXvVHI4aRHUa5ZbWLKPKWq8neMoZGGrmry +oiJOcrJce0krW+G6Nr3QxzZ1Y0OZXzM3W8agqBSmYeI7BpbRgkIVWZYRxUlTo5I3vKT/NkIzECuy +oiYrSrZRTpRoMEkpiYGB51rs74U8vr/Hq8uY3VVMkRfXdkAtwAcN2FZSlSlVkeNIk9A1ONozkaKm +yFPiOKUoW+DbIgy1jL1h3ABu84QiTxDKQCkDUUss08KxBKaiI9zFcUKapJRl3YDeRtcc/Jerob5+ +w75Ffv5WI3YDOFYFVZlTVQJUpT3eG7n3D7hAHctWq4oVpHlJXur7zrIMDXz7DfBdtvOXUhhJNycS +C8t0cGzZNFVDXhTEcUqcZLqpus2Lb6pT/UXWmP5eFFUH4FaVxn+0wmgfy7bIK0jSksqoqU2pm5Ck +hRDmNVO+bURH52FKcg0wt2zaf+1o8n5p1DimBsCpKkRVUWYZUbGjSMvOF1vKBvxU2kZNSEVeZqw2 +GbNVRBRntxn3QqAMiWEoBLqpP03zrs5tW1pVNQgChDJJi5pK6GYBQ1lIYTS5etW9f0Az/4UEJWud +u/7YuWpzdWnod2Jes4lKNlHKehtjGgrTNOiFAR89fsggDOn1Agb9kKIomS9XPHtxynS+IS8Fhukh +DVs3p1XFD687oRXXsrwiTUvKUjcEWKZFGPj0ewGWpYHv62dJ3NR8Giu7ugG+c/1+yNKCJK6JEps8 +zxvFC5MwDAjDAKlM0qwiqySVIagKhRImqnKuVQLrpk5QV0il9yBKaPCbtjnorl7wwfFvDnwDGMoA +i45V2YKOLSAdRRGbzQbLsjom7uvRMptbZm6apriuy2g0otfrYVs2o9GIQV97Bvu+z+npKd99910H +EJdl2X2mEBqYbB8GSimqSm86ttstJycn/PGPf6SqKvb29hgOhyipkOoaCI6iiKurK2azGUVR3PJk +Lsv3B4ffFm3DgOM4jEYjjo6OWK/XzGazDoBuAcRWzr3X6/Ho0SMePHjwo/2SW8C9rmvCMEQpRVmW +Hbh5dnbG119/jWVZXQNC3bSv7SINDK/X61uy3FmWcXh4SBiGGIbRsdZ3ux2np6d88cUXKKWYTCaM +RqNunovimqF7dXXFdDolz3Msy8KyLA4ODjg8PMQ13k/i27btbi57vR5KKfI879jsFxcXPH/+nDDs +NQ+vkCzLyPOc6XTKixcv+Oabb7rrbRhGJ2fe7/cxDfPW57We9DdlyG+CrmVZIqWWRL937x7Hx8eM +RiMcx+l82q+urthsNl2TQF3Xzab/eu07jvNXAY5N06Tf73Pv3j16vR6maVKWJXmes9vtODs744sv +vsB1Xfb2Jkwme29cy5a9Pp1OSZKkk60/2D/g4PD9bAV+bCRJwnK5ZLlckmVZl4Tato3jOJim+ed/ +yF3cxV38bxpCd80Kg6JSpIVgtkp4+mKG75/y5OEBjmNjGgaDfkBRljy6v8/55ZysqJGU7DZr0iyh +rkukNLBsE99z6YcBo0GP1a4AWVFkEUoZ1IWJEgGeY9ALHBzb6CQbd1HMdhuRZSUtY3XY9+kP+xim +gWWaxNGO9XpFtI1JIr0J39iKzdoiSRL2J2NW6x2e53SNcEpJLNPAMCSS11i8jWzd2/IjCdiWxDVM +Qt/Ec01MQ5JkGdtdxuV0wcn5jJenU5QdYJg+hgXSlCjjWhrKDzMu5jGHyx0IidtIc++NBxwejCmv +IjZRRFklVKbT+SzeTBRblkpVNVJjKM08lhbCcLuCp1RNZ3iTiLwhdVbXTbGi6AqXVZl1kn03pc/b +rv6qKiiLkhKFKgvN6O88thS10HRu7fn3ds85qBE1SFljWwaeo+j7itA38CzJ6cWciyvBdpeQ5CVZ +kVMXkrIBmQU1Stb4tkE/CBj2fTxHv//Komz8+3TBoSgFUVKw3GRsdilpds288H2fvfGA48M97h/t +k1ZNUl2kVFWFLEsMBYYhcG2LwLfxXYsiL8iLjGHPxXNuMOdv303XY/7e+02DyRpQ1t7si3XM2eWC +/fEAx7GxbYvhoIfr6H2TUoo4yZnOVrw6u2K9qTXj27BQpo1QFlkB613GapuRprq73jJNPN9nNOxx +73CPh8f7JKVJWlQUjTS9kAaG1IU2x7YIfc1cyouCIs8Y9T1c5x3KPOIH/+Iu7uIu7uIu7uIuPjCE +ENi2wncUvcDCcw1MQ722B53z6myB4QQYFigTOoyk0uxUP8y4mu+YLrfsCc3wdV2bvVGfw4MhxeWO +dRSRpQm7Xcx6GxEnmbZ3MQ3CwGc06BN6NrYJSZZTFRF5IpFC7+sM4eKYet9kWaqRdm6jBY7ftT+4 +sX96n22EeNuvvmMPJt7nQK/9fdM4YBmC0LcIQh8papQoSeId6/Wa3WZHmtRs6pL1RrBeGWy3Lvv7 +I5arDb3A03Vky2Q07PH44SFCGbjeBs/bsosStruYKI3Y1dofere1Wa9sbEtxfDRmt0toPdulVB3w +oqS4AZS9ex0JoSgqQZTkLDcpUZJTViWGaRN4HqNhn9EgZNx3NShOQhYvNbghBY7pszfweHA8ZjwM +sUxFXhSs1jsuZyt2ca59eJXZ1Pf+FuxumvxDSmqhKGtFlFbMljtenk7phx6DfoDvejw8PiDLcizn +ipoZ83VGVqUUSXntxSyVzg+UwPMVrm0zHnjcPxxy76DP+eWcy6sFi6UkjrVFkOPY9Hsh49GQyXjI +3mhAbSgoEspsjalAWorQcTkY6iZt39V77iiOmc0XzJdr0rzSrNFOxct4LwDt/eep/YVru6IamuaG +nKLU9VwpBLal8ByDpCip8oiyzFG2jWmCEuUtj/O3nlP95l8KIciLmu2uYL3NSLOCGp2X9Hoh49GA +vWHIaOBRSUFVRhRJjWlaKMsicG0O93wOJ33CQFuKxnHKbLFivtyQ5iVSJ+gNY/4vw/pulbyqqiZK +SxbrlG2UkxclUmilsZqa/cmYB8cHpFlJWtlkVUZVSHIERl1hGPq+9l2XwHewDE3Uyouc5bIkz6JG +Be9fuaWolbGnwhC1bsRxFaHv0fNMpvMlF1eKxWpHklekeUxVSqpaNqWUGllXuJak77vsDUP8RqGk +LCuKsiBJMrKsoqwESVbq3HWTEKfa+tcwDXzfZTDocbg/4uHxPpu4JikERZFqQLcsMaRASbAdbZ8X ++g5VVVIUOcOBj+9ab/ind0Yd77z0OpuvBc2LVKu0rTYpZ5dLlBS6IcPVqimmoXRjkmURrbcslhte +nc1YrBKKCqTpaHux9jnyrmdjcx8WRcU2yljtUs2Yr2rMpglkPOxzsDfm6GDMNoGSjCJZQVVAVeC6 +Dq5j4VgGSvoY0kORkWcxs5lgu40oywrTMPBcl+Ggz9HBmIf3D1lsc7JSUpSZXg41GEqiDIlt6s8P +PRtowPmiZpcKoqxVDvyba3n7m49/c+BbCIFsusBaZqXruvoBlOdkWcZ6vWaxWNALexiG8VYQb7Va +8+233/L06dMOvBqPx/zkJz/h448/7gBswzCwbA2m7e/v0+v1OqDrpqe367pMJpMO6G09tFup8u++ ++47/8T/+B9vtll/84hcdM9ySuiA8n8+5uLjgD3/4A1988QVVVTEajbBtm/Pz8zf8w39sKKUYjUY8 +evSI1WpFFEWABlBbILQois5n+cmTJzx8+PBHM75bJnHLdjcM49a8vHz5kt/+9rcIIfj444/xPK+T +YlosFnzzzTd89913zOdz5vM5YRjy5MkTHj161M1zC9YDPHv2jH/6p38ijmN+9rOfdexpE7Nj6E+n +U37/+9/zxRdfkGUZo9GIyWTCT3/6U/q9/nt7WzuOw97eHnEcd+eS5zlRFFHXNScnJ3z55Zd4nsfD +hw/xPb9jW5+dnfH06VO+/PJLrq6uyLIMz/MYDAbcv3+f0WiEZV8XNFsp8oODAyaTCa7rIoSgruuu +KaJVOfB9vwO+wzBECEGv1+Po6IirqyvyPL/1Oy3gvr+/T7/f/6tJdFuWxXg85vHjx4xGo07JoQXz +W5WFPM/5+c9/ThgG3bXM85zlcslsNuMPf/gDf/rTn9jtdp0f/U9/+lPC3o9r1njfiKKI2WzGfD7v +2PatDcNNaf+7uIu7uIsPDSkbTyOpdDJtwCau+ObFjLwSSKk42OvTC1xsy2LQ7/HkwQFZmnB2ueRq +sWOxXBDtoi5ZcXHohQGTvSH3DieIqxXbuCRNNigpUEpiyh7DwGR/LyT0bZQUpGnGarVhtlgSpxkI +he+5HE76HB/v4/s2oedwNZ3x4lXGeb4jSXKSNCMvIEtT6rpkvliz3uwY9H1cx+YNDzRx67++JzTT +WUjwbYNBAIPQwrUUVV2z20UsliuuZgum8w3zdYLtuViegSktDOmihEOb3W9Tg8U653K+w7ZtBoNK +M+n3Rtw7OmAdn3N+taIq0bJUZU71tgS3k6FTSFmjDJDK0l+G1fh4N35MNwDvpvm3YXk3TPpSe52X +eUJZOFRFwc2u/7rpFq7KnKrQDJAKSWWm1KajpeJaj7a3zN31OcvmeA2TWFRYpiJwFUeTgCfHIw73 +fJ4+d3n+UgPgV/MN89WOvKgpCs08UEqgLJNhz+HR0YCHR0N6gd7LZXlOkqZEcUKSlaR5zTYuUeuE +5SbtmOGGYRD4kv29IZ9+dEySpry6WHN2tSVKUmqhqISB5Zq4hsX+MODR8YTjwyHL9ZbVesfhJMT3 +/hyLFtHce5JaGuQlzFcRL88WGKbJcOAT+g6O42KaFoahUEqSZhmzxZaTszkpPggXZTZS58oiyTOW +m6wZr76W3XgnQz55co84jnl1seH0ckMe76iEro5bjolrWkyGPo+O97h/b4/VestqvWVvFBK6N8d7 +Q37vDvS+i7u4i7u4i7v4VwkpwHcMxn2LUWjj2MZre9AV8+WOxSbDLsEWJoawUcK9Vq+pKzaJYr7O +uZxtdA1xUOM6Nnt7A+4d7rPanXM+XZOmGoRdbDRLrygrAt9m0A/Ji5J+38W3BVGUU+RbqjLHMhXK +VFiqh+coeqGLY5sI+WMUca7/Q7z5l+/+PfFDjYevHV/8wM823aZ1XWNbkr2hy9FBH8818G2T5XLJ +i1cVJ0VEkmSkaU6eV2QpZHnKbL5mtd4SDXs4jo3jOkzGA8qPH9Dr9zg83HIxXfHi1SUvTy9Zr9LG +rz2nKjPiOKHf95kvN2x2EaZl4Hka4L62URY/jC+37EUhKSvYRQWzVcwmysiLGkMpgtBHSMHBpM/R +fo80y1jtMrZRjLRMlG3iOx6Hkx6fPTlmfzzAtkyW64TFesvZ5ZxtlFAL2QA8f/ka34+xGGoli2ka +vJEmSQYX0w1fPT3l0fE+rmMTBC6PHhxpuXfLRQrBi7M500XMbLXuGLemaWI5Jr5tcjQZcXw44Phg +xNHBiINJn39KYp4+e0VdFeyipAFubeRAcrA/5v7hhAfH+yx3JettTBlX1AYYlaLv7/HgMOT+gc65 +ATabiIvLGVezJWlWNHt+C2WYjZ+ufPugb+S5H1oj1ErFer3UtSAvKg1U5jl1pW3JbMsk8Cw2kZZr +L0uQZoAjJaYsdIN5p6jwnh+KJM0KlnXBYpsRpSV1rZWr+lJyMBlydDDk+HDIaluyinYURYKBi2F6 +DPw9Hh4NeXS8xyD0AMEujrmcLpjOlqRpgTJskMa15PlfQpmgUQUoa8EuLpitEta7jCyvkFJhWxam +YXD/3j4//8kTDMPg5GLD2XRLVZdQFcjawjQdfMvm/oHPw+M9Atdu8s4tsozZrmrqpin+XzO6R0td +YiiJZwnGA4eP7o/46MGIk7NLvnthcHIumM42XM0j0qIiL7S1mVI6X+/5PR4cDfjowR6jvt+RR5Mk +ZRclJKn2vq7iCrlJma9joqRomq0UruMwHvb56NEhm82aF2crTi83LDc73fgvFIZl4FgWo8Di4f0h +Tx7sE8UZq/UG27EYhA7ytffK7XfLO+4NIRB14y+vTEpq1ruMl2cLbMugF3j0Aq8jyMpGur0oShar +HScXc5Y7QV57GE3DSmtR8O6GFX1eaV5S7TLmq4Qo1iorjm0QBD7j8ZAHxxM+eXLM6eWG2TJhFy2g +TBFViuUOmIQmeyOHMHDoeTYXV5e8fLXi4ipivdlRlAWO7eC4DqNhn48e3WO1WvP8dMHJ5YbFekdZ +aWKE4VjYymYYWjy8N+Dx8R55UbLaxizWCefzjCjNqMoCaXyA2sNdAH8DwDe0MgyqA70dR3cPZVlG +mqYd21YIged7b2Ve7naahf3VV19xenrK+fk5+/v7nd/yeDxmsjfBMA2K/NpLRzOUVPe9ZVH3+332 +9/c1Y2VvjyAI2Gw2ndT56ekpv/nNbzov8tYH2LZt1us1FxcXvHr1in/5l3/hN7/5DVJKHjx4wGg0 +YrlcdqzePzeklIzHYx49etTJuNd13TUOSCk12N8wsO/fv8+9e/d+NONby8iEVFXFeDzG931c173F ++P79738P0M1NXddUVcWrV6/43e9+x1dffcXV1RXz+byTXj8+PmZvb48wDPE8r5vns7Mzfvvb33bz +5ThON8/b7ZaLi4vOz/w3v/kNZVly//59Hj9+zN7eHlmefdDYRqMRdV0zHo8JgqAbW+ub/sUXX2Ca +ZtdYsFwuWa1WfP311zx9+pSXL192XtuO4zCZTHjy5AmTyeSWx3YLaEuhfejbxomqqjoQu2X1h2HI +0dFRx4qXQtLv9zk6OuLs7IzlcnnL48UwjE7qfzAY/Kt7e7dhmiaDwYDj42Mmk0nnLZ6maXetfv/7 +33cy9K2vu23bxHHM+fk5l5eX/OpXv+LXv/41SZJwfHzMw4cPGY/HfPTRR/+q598qNEynU+I47sbU +eqXbtv1Xmce7uIu7+PcXNS3wbSBVjTQlcQpnVxvysuZgPODJgzVKKgLfIQxcHtwbYxo1hiHZbGNW +24g4iUmSDN+1MU2DMPQ5Otzjo8f3MUyby/mO9S7D9QwcXzHqWeyPb3a210Rxwmyx5vJqQRTnCGni +eh6jYciDwyHjkc9k6HPaNxBVQl0krLcGW0NRVSVSaDmwuq4pqpKirCjLChC6Yz0vtH9Uk8joYghv +9yNqsFspalzHYK9vMgwtHFsffxfFXE3nTGcrVpuEXVxSGpLasagql6p2UbjdwXaZyXxXMl3GDAea +ReO6DvuTMffvbTi9XCPqkrKsqYq8kXh/PbnVDAoptc+XbDrWNcu7lRszr+X43oimE7cuEXWJpMQ2 +wUTiOwrbNhqQVc+HoSS2ZeK5Np4jCJyapAIhKqoiR5mymT75HonctWe7FBWmoXBsxbjv8vHDMZ8+ +GhHYFb5V4lo1liqRdUKS1SRpjRAS01R4rsnxfsjHjyY8vDemFzrN9UhYrjYsVjuitKLEIMlBRCWr +bcZ6m7LZxtiWbmTdG/f56PE9qCss64yyKFltU80uF5Keb9ELXR4dhfzskwkfPTzk5HzG2YWiFzq4 +tqnnk5tz3SbUrdPY98+HaLrIhTLIC1isEl6d1wz7AUVRNRYteo9UFiV5oYtoi9WO2XKHtC2k08Mw +tbSpkCZ5lVPG7XgTNttY+3Ja2ify40dH1FXZjLdgsRYNm14SBhb9wOXBYY/PP9rns4/ucXo+4/RC +4XsOrnPbt/16vNfjfjv96i7u4i7u4i7u4i5+TEipPZX3+q7eg5rG9R50tmC6WLHaJuzSmsqS1KWN +WbmoykVJi6qRJt3lBvNtztV8x3DQo6hqXMdhfzxkdTTh5HKFoCLPMnZxwnoTs90lxElGP3Rxm+L4 +ZNTnYNKjqiEpFEUFrqP3ZuO+zbDnEvoujm3dYnzX9TXTTrzVA/RaWrn1Lf5hydRrZqoQrWHRu3+n +k4cWQisVfY/acbunbXnqplL0A5PjScBo4DEZ+kznDhINDKw3Edud0rWkukRKBbX2GM4LbbdTVzWe +6zAZD3Fdh71Rn3v7AxwT6jLlUlVstoLdDpRhoJTe81eVJp+UZUlV6npcXpTa+7WqAcktD+q3zFO7 +Ty8rQZyWrDYZm23GdpeQDwNMw6DfDzmcDHl0f09bFE63CGI8R+F5BocjlweHAx42YLGQgihOmc43 +nF0s2EQ5FUYH8Lw2oc23H1ZFevNaiM6q6YNldLXE0g0JYJM0r7mcbfnGLDU5ZhSyNwzZH4/YGw3J +8pKy0DZeSlaUedysE4FtC8LAoB9afHTc47OPDnhwtMdkPGA07PHHL54RRdpTerWJ2MUZliHxPY/J +eMiDB4d8Mn3I+eWK86s1RSUJQ4NB3+Bw5HG832N/HOLYBnlesN5sObuYcjVbkWQlhumiTG1vJFs7 +g7dO3+25ug1+vz6H1/eRXvNtfqzXX5YXbHcJYeBTNAoQvdBjf29IWq6I0oKsgF7PpNez6Xkmpvr+ +z7iNQwq6/wlJXtaUhWZ8b3YZuyjFshS+5zEeDXhwtMfHj444n25Q0x1JURN4iiBQHI418/5wMsR1 +tLrnehNzMV1xNV8TZxXSdJt7oVFE+761+K4GeXHz9EU3V1Wt2curbc5qm7LeJkRxhmUZuK7FvcMJ +URTpe9s4Jc1ySgxqYWKZJv3Qot/z+Oh4wE8/PSTwHU7OppxeVCzmJkrWVJX2Dn+/O+fPiKbhR0mw +DMEgtHh4r88vf3KPcajwzBLPqnGMkrqKiZOCJK0oa4FpSAxTcbQX8NH9PZ482Gc08FFSso1iFqsN +s8WaXVJQVoqqFIikYrXLWW0TNttI2y5YJoN+wKPjA4o8w7LOqSptTVciqCpB4Fn0AofDvZCfPJnw +88/us1htOTk3qOoKz7Pe0oD1+vOkefO8/UXQyOOX1JSsNyknFyWDnsfxgcYNWjxD26GWREnKchMx +XWxICgdpByjb7iThq+YdJMT3O3236gdZlWNvU9a7hG2cYpoulqUV4R4cH/L5p4+w7QuknKNWEYYt +sGzBXl9xf9/j/lGfYc9j2Peo8g3fPU3YbHYsVmvWm0jXREyTfj/k4f1DPc/2GVV9hqZKGNTCIPBM +eqHD/l7IZ4/3+Nmnx2x3CacXC6SsWe9yqAvAuFZR/Isvyn+/8TcBfLfhui7j8Zi9vT0Wi0UnfXxx +ccG3337bsZtbYPxtUVUVm82G8/NzptMps9mMP/7xjx3jW0pJWZZEUdSB5C1LWksw73Ws6P3JPqZl +8ujRI376059iGEYnXb5er3nx4gVJknB6esr//J//E8MwMAyjA/pWqxVnZ2ecn58ThiGTyeQvPmdK +qQ4UPD8/58svv7z1761/9mg0Ym9vj8FggOd5P5oF3DJgLcviwYMHfPbZZwAdU3Y2m/GnP/2J2WzG +F198wWQy6YDv9XrN5eUls9msY4G359jv93n06BGff/45QgguLy+5vLxks9nw8uVLkiTpAG7TNDEM +gyzLOpnvdp49z+vY4j9mLl3Xpd/vc//+fT755BPKsmQ2mzGdTrm8vOTXv/41p6en/PM//zOj0Yg4 +jkmShPPzc7777rvus1up8cePH/P3f//3HB8fv8E8NwwDx3U6YLXf72v/nhvrsZ2bwWBAv9fHtvUG +uNfrcXx8zOnpKScnJ28c9ybw/ddifLd+7b2wx4MHD/j8888pioKrqyviOGaz2XByckKaplxdXfGb +3/ymszbI85ztdstms+kaV0zTZDwe/1XOHWC5XPLy5UtevXrFZrPprsFkMuHg4IAgCP5q53IXd3EX +/76iKygIiZQGUmkJ5lIYxLni5eWaX/3xOR8/jPno4QH7ox79nn7mrLcJs8UWKaEsck7Op5RVRT8M +sG2Ljx4/AODj6ZLZYscuznFcH8fzuH804aOHB/RDH8OQJEnK1WzJ81eXfPfyktU2QRgOynSR0kQp +SRh4HO4P6Icuw57Hpx8ds4tS4iRt5NfAdWw+enjAg3sHBL7bgaLz5Y7z6YblOiEvrYYl2/ikvaXr +V0uMVwgBvmsyGfmMeh6ubeh9w2bH+cWU+VI3CBimi2F5GJaPaXsYloc0mz1hXVNhEiU183XGLikp +S3BdzfjeRQlfPj3VgHNWdp5prWfVm9erLUrUOqF7X5+yRrJRyQpDga0MxuGYUbjPo3sDPn44YdAP +cV0HJRVh4PHo4T0QgnvLlOkq4XKRc7HIWWxTDb4rs3ENrxEfkuE0yWbdnLeSivFoQA30+z0ePThi +tY7Ii5q8rBvGt8SyDCajkMO9PuNhgO865EXB+eWMr797xbfPztnFJYbdQ5outXTYJjUvzxb84asX +HE6GHB2M8FyHewdjLMNgMBzx5PEDoiSnqqGuBZ5r4jkWo2HI4WTAsOezXG+153WjWvT6ddH3kEKg +ELwuMf9m4t0qLeRVzXqbca5KHh5rBs71JavZRhHr9ZbpbMU2zsgrhYmJUpaWPVQmQplQm5SYbKKS +l6dzfvflC472Bxztj/A8l4P9MYZSDAdDHj96wC7Obo3XtU1GAz3evVFPswwE156d33P+7Xhp1uad +As1d3MVd3MVd3MWfH1KA7xpMhg7DnoNra4vDzTbi4mLGcrklLyWm7WNYfrcPlaaNUiay8eKtMNjF +ld6DxgVVWeG6NpO9Mbs4pf/0HEMK0qoizwqiKGe23HJyPsNQgl6gwezPPn5IXdXMllvirCLLwfdt +AtfhcH/A4/sTer2g8QNvbWpEB8xe74vEa+Ns9g/SQKCgks0+6npvcTtEJ98thGb/XfuLvp3JJ27s +W1oteCFu+sCKGz/X/qyEsvn3ZhsUeA4HkwHjQcAg9Pjo0RFRlBIlKVVZIkSNZUo+enikWcS9ECkF +213EdL7iar6iriH0XXr3AyzL5HB/zHq9Y5ckJEmOUlrpZ3884KNHh4xH2powy3JWmx1XszUXV2t2 +UQ7SQhmOBttvDfga8BZSIWvtxVpikBaSq2XEty8ugYqDvT6Dvs/x0YSy/Jzjo3u6gTJKsS0L2zaY +jAY8ebCP5znUVU0UxRr0vlrx8nzFKqqp0BY88m0NuO2esW58it8As8Wtb3Cd77TXQnSgEe8NqmjF +K71WlGGSVwXTZUqRJ3ieR7/nATWB7xF4Nvt7Q6qqZDQa8PFHEct1TLvnNUyFY5m4jslk1GMy7jPs ++ziOra1Hi5IsLymSnJOLFV9+d85kGDIZBfR6AZ99/BjHtlmu9XGrWuB6Pp7n8/j+PgeTIZapKApt +aXl+ueD5yZSzyxVxJjGdAMN0m71/w1p+relUyJtzJZs863pNvzbJ14oAyEZNTOcyUhmUlWAbpZxP +1/iBx35R4HsuTx4eYxqKj9YJi1VMXoLragvGR/f26AV6TjuBKCGQzb16nSPJ689vzq2qJXUt2SYl +Z1crvvj2jMk4ZH8cEvguH3/0CNM0WW4SlpuEotTKFbbj8Oh4j6P9IbZlUBQlq41WVHh1ttCqXqnE +MN0um9HN4+q1ZXTzudKstbeoKnTPB6H0XNVaSr9GkVeK+Srh6csppiE4mvQ5nPTphSEPH9zDdT0O +9vf5yacfUda60cBQBp5r4zoWB5M+h3sDqqpkNltee1i3udgtRTXRfe+k24X4sHz8++6bN2cGhKDf +D3n04JDA97l/b5/PVzuyvCQvaqpaaGU/Q7A3DDkY99gb9eiFAQiYLtZ8/e1L/vTtK+brBGkHWh1C +OSS55Oxqw++/fMH9wz0OD4Y4tsn+ZIhSgn5/wIP791htYqoaqpruXuyHHkf7Aw72RuSFZqqnaaF9 +yd94Hujri9RrXrMmpG5Wf9u0CX1PVXXNJs45n2Yc7Sek+W1yQpykbDZbLi/nbLYJWSEpa9Uo8dmN +NcGN91lXu7mxuMT12qrqmrwU7JKKi9mOr55fcX+/z2QYYlsW948PEUJwfHyfzxdbdnGGMiwMw2bQ +8xmPQnqBo3P4qqSqSvK8YBMlnJ7P+MOXz3n84KCpEdjs742QUhD2Bjy4f8xiHTW2bELPs2vTC1wO +JwMOJ30uWGKoJXWpj01VUsu2PvIOq4O7eCP+poBvx3EYj8ccHh6S53nn131xccHTp08ZjUZ8/PHH +7zzGTYA1jmOePXvWgWstANjKSadpShzHDePDxvM89vf3+fjjj3ny5Am+7wN04GXrZX15ecl8Pmez +2XBxcdF5UEt5XYhq/cKLouj8gn8sIPuuaJsBPvnkE168eNGdcxu2bXeg+3g87vzNf2zBrAWHwzDk +4cOH/PSnPyXLMrIs4+rqiqurK9brNc+fP8c0TZRSt+a8KIrOH3wwGACaad0C33/3d3/XSaefn5+z +WCzYbrdcXl7y5ZdfdhIXN+e59ZHO85zJZNLJVH9oGIbRbIBVN7Y0TcnzvAPil8slX331Vdfk0DK0 +2+ucpilBEHRr+cmTJ/zyl/8B3/feCnwrpfB9n16vRxiGbLdbsizr5sX3ffr9fnft2m6qfr/P8fEx +JycnfPHFF28ct5VC/2syvtv7KOyFPHr0iF/84hekaUqWZZyennZS/FdXV3z99dffey3buWwbC/5a +0QLfZ2dn7HY7QAPf+/v72vc97P3VzuUu7uIu/v2FaDb1QkmkqinRcs9xbnByuSHPYqqqYtjXoFgv +DPA9l+0u4Wq6oMgyijzn9GKGEBLLNNkb9/j4ySPu3zvSwN1mR5zkOI6Na9u4roPvOViWQZIkxEnK +bLHmxekFz15eUAoXoRyd4CsLaSh6gcu9/RGObfDZk6Mm0crJMi1jrZTCUEr7GjkWVVmSpBnb7Y7Z +YsP51ZrlrgbTRloWQpq6qHWr8HUdda0B18A12B/5jAcOtmVQVhXrzZbzyymL1YaiFBh2A3jbPsoK +UJbuyqduIHQh2MQ1803OLi6pKi11frC/R1nV9HshSkqoNOBdl0XjlXTjOgEts0Mn5fUHJLhNElKX +KAmuKeh7Bj95vMfPPt7jaBIy7Pv0AqfZI0nCMODJw3uMBn3Wu4T1NuGL72ZkX1wwXSwQytA+5LXx +ASSMt1SrhMAwFHujPoN+wOMHev9SVhVVdbtxWEqBYUhMw0ApgRSSLC84u5zzhy++4+nLGdu4wnJ6 +CMOmljbbuOLF2RLHFlQ1jAYhvVA3Su5Pxnyaa3/woqioap20KWVgKIlhKCxTUVU155ezdwDB14wS +Ud8s2H6/jF57HYsyZ7XLqIqa1TajKK+T6bqRNL2azpnOl2zjnKKSGEIn0vre0JKHZWlQYrKNCy3F +prRk3LAfMuh53DuYsL834pOP3jJeqTAM1c2tEEKPVzVM/deA/psWCTfH+wHipHdxF3dxF3dxF3fx +jhBSELgm+0OXUd/GaYDv9WbL+dWskSMVHeCt96DXFij6/V1QItjGNbN1xq6Rk/Uch/3JqNmDfouh +JHWVk+UluyRnvtxxcj7FtU1MQzEZ9/nJx495cHxAFCVEcUqal4S+i++7uLaFbZtYhtGoEt0AYm41 +bb59hyBbVm59Y//U+fC+jQncALpSUrcAKTea78Sbe02d7yiQEuRNtSLx2s/JG1+tio/W8Ql8h8O9 +IWFg88mTQ4qiJMv0vNV11UjeyiYXsZtaa8YuipnOV7w6vcL3XPZGfR7d3+f4YKxrTHlBXhTkeYGU +CkNp5U7XsbAtgyRNG8XRLZezFRfTFQUWtbBRpo2U2uaobSjVUy+RN5oUW5/rrFTMFjuevrjAUDW2 +ZTAahhzf22eyNyQvNEhSllWjPmpgmgrb0mthu43Y7RJmiw1nl2tena+olEstNSj7JhNZdtdXVO1+ +8cbci9dBbzrAu1W54pZM8IdAe6LBmgwENVlZMV1FzBcJvcBjb+DiWgolJb3AZW9vwGAQ8kmhlcOK +orrFWFdSfzeUgWnq85NCUNYVRVmT5hVRnHF2ueKrp5dUD2t6vkMvCPj8s4958vgBeV5QFCU1Queu +hoFlmjrHLAvW2x2r9ZaL6YJXpzMupxuwehh2D2Xaurmg81K/OdKbIPON++dGc8GbrHvRgXHX+aVC +SoOyFmyjnIvZhsnegDwvGYx7fPT4PvePDygKDfbXNV3t1FACKcRrtvOik4unfu3cuG4yqQpJVSl2 +ccX51ZYvn55SlgeEvs2w7/Ppxw959EDn/3r+QLUqsqaBZRlNY9CO7Tbicrri1fmCi+kWzABlhZ36 +WOvz/foEXj97boKT7UPqzeePXtPX91ZRKebrhKcvrzBkhWVoy7hhPyQIXI6PjijKgjxvcrBKy2Eo +ZWi2dLOuFssNtqWoGwW4urrpn3y7biFuPrMQ2pv6LxA3WdHtfTTsBQS+y4NjfQ2K8kauXrdLSejn +V2PVpaQ+r9l8zZdPX/LlNycs1hnKCkEY1MokzhVnV2v+8NXLBpNx6Ycek70ho2Gfxw+L7n6sKj13 ++tprjMQydd6+2mwxlCCpqyZff+MGaRpDRDNX7Z+7H3hjDoQ0KOuKbVSQJQmLdUqS3VZJjqKY2XzJ +5XTOZpeRF4JKmYguT1c3rpN44xpeL0B9L9RUFKVglxRczLZ88+wKavAci4NJjwfHRxzsa8nxrH1W +N2vANHU9rKxK5vMls8VKN+bkWj3u5HzOH79+AULgex6DfsDe3pDhoMfjh2WjVFJqkmijRm0Y+piG +YWAZkihOMJSgKpv1WZddfaSGuzrAB8TfFPDteR6Hh4fcv3+/A69bAPTp06c8ePCAJEmoquq626aJ +lpl5fHzMxcUFV1dXLBaLDrhswefqBoOkle3u9Xrs7e1xfHzMT37yk47dadt6E3V8fMzf/d3fAbrg +a9s2URQRRVEHnrdgpRCiA087Bmyvx7179zqZ8VevXnWA8J8bQojOR3symTAcDhkOh2RZRhzHne/y +w4cPOy/pt3mkf8jntQDn8fExv/zlLzs5ddM02e12xHFMnuckSXJL0t2yLCzLwnEcRqMRx8fHPHjw +oPPTvnfvHr/4xS8A/VK3LIvdbkcURWRZ1kmO35zn9rNt2+5Yzvfv3+9k5S3z/UHfdk1ZpsW9e/f4 ++7//e72xaADaKIq6scVx3El2w7X3+XA4ZH9/n8PDQ37xi1/wySefMBwOOpD7bZ/XsrrH4zF1XZMk +CUKIrmFhOBziui7KuG4iCEO9plo5ftPUUjOtx3cLjP81ge92bRiGwcHBAT/72c+awrZ+eLfXMm0S +ivV6fWse2maC9r48PDzk+Pi480j/15Aab/3ksyxjNpt1TS1pmiKEIAgCjo6OOD4+pte/A77v4i7u +4seFEBKpDM0eVTXSqDWQqSwqIdnGJXWVMrzYcHQ+px86BJ5DELiMBiEPjw9QSpEVNRdXa/IC0rxk +vUvwXRvfswmDANt2qMpCv3OkoqwqdlHEdJZyOV1wOVvyp69PuFokZJWFMB2UEVBLhyitma1SruY7 +zq7WDHsOjmXg2CaObSLD6+c1Ndqbbr1lvtxwNV3x6mzGq4s1cS6pUChla3aEMhDCuGYP3JwXdPJm +GALftZgMPQahh2OZVFXBLkqZLrasdzkFCmW5SMNtjmve8HFCyx1WNbukYrHJ2SUlRaXfQY6t56ff +7zEc9imIqKV5oxh0DZ4KCVIpXewwTKQSzT+/zlp5e7T9t0Lomp9lKvo9l6P9PvvjHo5tYlnXewLT +MAjDAMMwcdwU30u4XKbY5lR7jNUf2tl7LS+ZZhXLbcX5dMe3L2ZIKlzbwHUMHMvEskw8Q3XWP9A0 +KlYlWVoQJwnbRmZ/Nl/zx69PeHG+ZrEpKGoTw3E020mZpKXJdJWhXiyRyqFGsj8O9fp0LSxLFxZV +09ENNWVZUVU1aZaziGI225izqxWXiy29wCPNCm5WQIRQKGXp4mNlIyvtcagLSN9zXZqu87IuSAuB +qEs2cc5qkzDopdo3U0rW25izywVX8y1pXqNMD2k4Wk5StV51BkKZyKokLSqulilQavAfycFeSODZ ++K6tZfdeH29VUZU1aZ6zWm/ZRSlnl0suZ1s81yFJyxsgvl6X8v/H3n9+x5Fkd+L3NyLSlS94RwPa +brLNTPf0WGkkraT9Y5/fG52zRtqzWmlnR6MxPTPt6UmAAOFR3qaNeF5EZlQVCNB0k02Acz/noMEm +gar0FRE37g1hQ1guhHKRKDstB8dH1y0hhBBCXhpLM79sIVDI2ZipuqgWHDi2QCJD9Ach6s1+2ga1 +YDl5HRAT6dq/2eexApiyEasEPV+i3Y0w8CUiqQezPc9FuVxEuVJCpVpGpHwo7iGRHI22jwcbDSSS +I4wlEqnguTaKhbxeFzmI0iCtDiQkMoHvS0QigmPrZWWyQDMXDriIwU2b5UjghgsIYcOyPSTSAZeO +zpRLs+SeaqczgAtLL/diR1DMgRJOuu8ibYscPaZCt1nsHBRzIZkuGa2zZsd/LmvfOOCwwaUFCY7+ +MEGtPcRho4e9WgtxUoDn2rBtAce2wTkmxoHDMNLLQXX6OGx0cFjv4LDZRa3ZQ7UUIVcowsvl4TkW +XEcgn/OQbTZPXyeOJcIoRKfbw/5hEwe1Ju483Eat5SNSDpiVh2V5Ostf6CU3+fh+WDa47YInDkTi +QDJHT4hgNlr9BBu7XZ0VDK7Xc0/biZ7rIJ/zIDgzsbYojtHrDzEY+tg/bGHvsIUHGwc4bAWIlAPO +HHChg7L6+hudMB08d9Pz6+k2cnauTFZtWj0o7StwzsFEeu1YbjqZwx5lTr5kaCVb1kvKBImyoKSF +g2aAu4/rGPgxlltDNNoD5D0H+ZzOcs/n3HQyqD63MpGIEj05Yej7aLZDDHx9P3YHATZ3mwhihlBa +OGyHePCkla5HzLAwXUQ+56CQc+C6LkRWsStNjB4OdRW1ZquL3YMGdvfrWN9qoOsrJMyDZeVhu0V9 +D6XHYSJrOj2GJsOU2eDMgjLZyxzjmcqmwgEbZXibZbMsB9xykCiOVi/C1kEP5XIblVJDT9zOOch7 +LuDp99VBtQRxHI2qpo8HibnQa5Pbaf9IOaPnwPjEE6YrMQQxcNj0wRDr4wdgcbaMQs5BPueiWHBG +E2vS63PoB2i2Bmi2e+n12cT6Vh3dgUQCF4J7sJwcZJJAyWj0rBh/DoGZZxEXFriyTPb8xDWXPSNs +D1xyCDhAem8xbqM3lNg5HILxFsAtJFKhWNDnPuc68BwHpcIo8VH3b3VAdzAM0GoH2D9sYr/WRq3R +R9+PIJkFYXn6+h/PTs4yz4UNZrb3WywLMHq0mhMYRArdfoz9xhDr203kXI5i3tV9dVeXyS66Tnp/ +6/fVcS2FMIoQBBEGfoB2p492d4jb95/g8XYbh+0QfixguyV933MLktlodhOsb3fAxSEUBNqdAQp5 +D8WCC8e2UcjZsIQwfVcp9XHTywL04Qchdg9b2K93EYQRhkEMKc1egUGM+q42QyIdwKz3zsxtdPTD +mEFXIogShjgCusMYnbScvevooPtgGOCg3sbuYRs9PwasHLjw9CR1bqXXNjeVL0zFgac+C3n6OanA +lIVYctQ7IdZ3ulAQkBLoDQMUPDt9TjnpZ8eoMkkUJ2kF4j72D1vY2jtErdlHGDNI5qLVi/F4pwvL +qYMLB/2hXqqwWHDg2DbyeRv2Cce53++j5ofY2W9iv95Fo+NjGCbps0eMTYKn0PeLOlWB7yzItLq6 +ip2dHQgh4Ps+Dg8P4boubty4YdbsFmJykKtYLOLcuXOQUsL3fURRhO3tbTSbTXQ6HRNsywb2hBAo +l8uoVqu4cOEC3n33Xdy4cQPXr1/HzMwMXNc1GeLLy8so5EflqBcWFrC5uYnt7W20Wi2THSrTB6md +NkKzctdZKfIbN27g3LlzePTo0SstP+15HizLwtTUFGZmZjA1NYVer4c4juG6Lubm5nDlypWn1pn+ +LoQQpnx3uVxGuVzG1NQUtra2sLOzg06ng+FwaM4V59xkilerVVy6dAnXr1/HO++8g5mZGQghsLS4 +ZH5m/DhvbW0de5yziQVZIH16ehqXLl0y53FxcRGO+/L7ywXHysqKmbRQKBRQLBaxvb2N3d1dcz0l +SWKy2guFAqampjA7O4sbN27g5s2buH79Oi5fvmx+5qRBWdd1Ua1WsbCwAN/30Wq1IIQwa7LPzs4+ +dd5KpSKA0brftm2nDffYBL6Xl5cxVZ363kqdZxhjWFhYMJMRKpUK5ufnzblsNBro9/tmIkp2z3ie +Z9ZZz8rE37x5Ezdu3MDy8vJTGfOvQlZivdfroVarmYoFUaTLXxWLRRN8r1QqVFqUEPKtZAM8elBB +ggsd+NZBNYEgCRH1A+zVfaw/qcO1BM4vTcPzHFQqRVxeXUGxVMT9tV082TtAozNEvT3AzGEHy/MV +LM9PIZ9z9Kxf24KUClLpbJVsPe+HG3tY39zD3mEbjW4C5pQg7CKEW0LCPbQHCruHA7h2C5xJzE0X +MFXKoVz04LkOPE+vm6SUbpQ3Wh00mh1sbtfwaHMPG1s17DdGAXVu6zLkXNhjM2+PzPBlOqvVsQRK +RRezUwVMlT24joDvRxj4IRrtAXqDCFLZsF1HD8CZoPf4oIJCnCj0/QR2Dxj4sSlnnbUXqpUy5ufn +EKODfpCW4co6SWOBRM4t3WGzXHBLZ+jrQaCTs2ImqfR1ACEYPNdGqZhHseBBCJ2lkg3cCSHgQJls +CyE4cq4DyxJjy8KkM3tfYHpv1plWYOgHeuBISYkwDNBoNDE7lcP8dAGVUg7lUgHFgpfO4NZtBT2w +oss8ttq6BOfGk31sbtewV+9jv+7ryQ3Mg+N5ZmAohkK9m2Dg9xHEe2i2u1iZL+sy4LNllEoFFAs5 +OI4Fnh7nIIwwHPpodfqoNbo4bHTwaHMfT3ZbmJ8F+n6E8ZnhTAhwy4Vl55BID3GSDeplWQ382COi +Bwk4YskRgKM3SFBv91EtuWlpUWsy8B1zWG4xnWjhTAxwc24DlkKkIjS6ep3OMAZanR6W58pYnq9i +ab6CUjGPUjEHx7FH+xtEGPoB2t0Bao0Oao0OHm3sY2u3helqCYOj+8s4hO3o/VUuhHQAYaWlQ5+3 +hiMhhBBCTqIzS/V6qcWcjemyh3LBhWtzDIfAwI/QaA/RG8RIlJVmgbpgIiunOnodBSBJGAZSotWP +MQhixLEy7TzPc3UbdG4OCXoYRBZiZaPWCpCs1zH0EwRphaW56TLmpsuwLQHb0m3Pfn+AXl9XwrME +h+PYqJSL6RiTbptwy4GwJISQut1iki6y9VU5hO3CcnOIYx2kZdZoAuFTGY5gOpDm5GDZCRg8JHws +8M3EU2u7ciEgbBe2m0cCF5zpvg/nApyzNJ87DTxYNiwnB8EsWEoglhztXoTdgy5cxwJnEq2ZEqYq +6aRYz4Xn6mpJSinEcYJmq4tGq4etvTrWNg/wZLeJIFIIY4WpagxuuUgShmoph6mKh0LOgZdmzUul +Sxr3BwM0Wh0c1tt4uL6NRxu72NprotYOwNMlfbiVg+0W0qxvZkr2Zv07y85BMBecOQBzwS0PzHLR +HUpE+334YYxBEKHT6WJproqlhSoqJT25wXOdtOphgk5vgMNGG4f1FtY2a1jfqmGvPkCjE4E7pfT6 +y9afFun1l12HHNxyYDk5xIkLkThAGiQclTM/si415xDpBATLdtM+ljMK+H6L9b514MsCbA+ScdQ7 +MYK1BvbrPSwftLA8V8TCbBnzM2VMVXRmayHvgTMGwTmiOMZg6KPX91Fr6r7mQb2Dg0YfB/Uedg4H +8GNdTr7eSRA+0RPCh36IcwtFzM+UsDhTRM7TAVDOkVZgSlCrt3BQb+HJ9gEePd7D2uYuai0fw9iC +5Xmw3DIsr2iC2FxYujM3FjhjnKXXrodEuUhgA8wyE6SzSSFM6Cz17NwIy4awnbH+pQ7ixkqg1Y3h ++xFyuRY8RyCKAszPVDA3W4aVjqVnkyJ6fR85z0U+506M83Jh6eUX7ByE1BMxsmfVxFrLZgmoBLV2 +hG5/iDCSGAx91Od11bml+TLyngsvncCbJBJRHKPWaKNWb2Frr4a1zUOsbR6g0Q4xiASEW9bVMOw8 +GI8gE55eS2KyjzY+JpIIcDU2kX0iwzq9nm0PlhIQzIKCayYMDCOG/WYIP+rAD2I0Gm0szpawOFfG +7JS+rsrFnMmST6TEcBhg6PuoNTqoNzrY3m9gffMAW/ttdHoJJPT9Y9nORH/fTHCxXL29Zv3ybxl8 +ZACUnkThRzEanRhRHOlKBJ025qYKmJsqYLqaQ6mYR7mUh+tYUMqC4ByxTCATvSRcq93FQa2F9SeH +2Ng6wF6th/1GH90hQ8IcWG7eTGSRDGgPEvj+EGFyiE5/iJ39Gpbnp7C8UEW5mEeplEfO1et2c8Z1 +QqUfoNcfotbsot7oYH2rhie7LSgwdPvRZGV4ziCEA8vxIBIGIV2A22YCDp7VV2cWYsWhEpEmMgzR +bPVQKeVgWxb6wxD7hx3sHbYx9CUsO6+D3+kkdVPlgIt0acH0c5BlpfRZmqyXLhdhKXDmIEnvwTDq +ww8S9Ac+6s02FmZLWJotoVjQ95tr2yZDu9vro9Xq4KDewvrmAR4/2cPuYQehtMCsAnoBx9aBjwR1 +9IcR9g/bWJovY2mugnIpj3IxB3huepwZwijB0B/q49zooN5o4/F2A5s7DRw0ehj4FsBdsPFxNRoC +eGGnKvCdz+extLSEdruNe/fuwXVds1Y25xz7+/uo1+smA3Y8AzQrSWzbNnzfh1IK1WoVu7u7ODw8 +RL/fR7/fN9nitm1jbm4O8/PzuHHjBn70ox/hhz/8oQm62rZtXnt2dhazs7MolooTZcNzuRz29/fR +6XTQ7XZNEM9xHOTzeZRKJSwtLeHcuXO4dOkS3n33XczPz+PcuXOYn5+HlBJRFMFxHExNTcHzPN14 +5S9+BTPGTCb17Owszp07h4sXL6LRaMDzPCwuLuLSpUu4du2aCUY+7/WyNcG73a7JfK1Wq8jlcrBt +22QGzc3NYW5uDuVy2WQt3717F57n4fDwEN1uF/1+3wR9S6USZmZmMDc3h3feeQc3b97EpUuXMD09 +A845pmemMT0zjVKphFKphPn5edy+fRv5fN4EnDudzkSwNJ/Po1gsYnl5GSsrK+Y4X7lyBaVS6VsF ++hljo3NeLMGyLOTzeTx48ACu66JWq5nM9ixLuVqtmqz+jz76CJ988gkuXLhgjtmzeJ6H+fl5XLp0 +CVEUIQgCWJaFCxcu4Nq1a5ifn4frehPbV8gX4LoulpZ08HthYQFxHCNJEpNxvrS4hHKlfOL7u66L +SqWC6elpxHGMKIowPa2Pv2VbTwXrs/smK78+PT2NpaUlzMzMoFqtmioJAEzlgUq5gmKxiLm5OXMu +s7Ln2blMksQc40KhgOXlZSwvL+PSpUt45513cOXyFZQrZbiuC8dxUC6XzTbHcWy22XEcM8niRYVh +iEajYdaIb7Va6PV6cF0XxWIRU1NTWFxcxOLi4lPLCBBCyHHiRKE/TNDqhIijBEEQodXzMQwUEiWg +wMEEwKEHGsAEwkhCxjHq7RAbOy1YHJBKwbJteK7uJOfyRQQxw0GjhyhWOGz5qDYHGPgREglUSznk +cy4cx9ZLkEQxarUW9g5r2No5xL1HO3j4eAfDAEiYA25XINwCbK8IyWz0AwCtEJboIElCtLsDzFbz +mKkWUMh7KBQ8cMahZIIgjLF/2MD+YQOPNvZxf10PEsVwkTAPwi7ojndaJu6krIEs+Ju1e2IFDIME +fpCg2+uh1hzqtRIDpTtvjgthezrrNu1Ij70a4gRIYoXeIEazG6LWHABKt/sGgwCK2SgUK8j3gQh6 +IDOMFTr9EM3uEIli4AzoDiKECYOCzqYQlhqb9f/iHQ2loDOaowTDQM9cTv/lmJ+V8IMYQRgjjBUS +OZnV8sLMzHyGIJLwkxhRFGI4HKDZbGNproBOt4TZ6SKq5QCVtDPtODaUApI4Tgc3Oqg12lh/so8H +a9t4vHkIPxHwIwGIHGxPl/zMBniSJEbf99HrB4ijGN1uF622zmoOggjVso9KpQjPccAtPVigyzgO +UGt2sZdmtmwftLHfGIBbOdRbPmqtgS5LZwm0ugHChIFZeTBpQ3AHsNJr4YR1vvXh0Ou7ScWBhKPd +C7F72IVjCUxXQxRyDnYPO9g+6KDe9hEmArZbhOXkJjI+GOM6+xvpdRaE6MQR4qiNTqeLVquHft9H +EMV6f8sF5FwXXHAIztDtD9HrDdFodbB70MJ+rYXtfb2/kjk4bPmoZ/srBFq9AEHEwawcuHLBEkd3 +eM3+vooV5gghhJC/PAx66VGeZtzGicIwjDAMFLrdHmqtIZqdEINApQERF9z20olw41mMOlM6kUAi +JXpDoNkJUWv2waDAGDAYhlDMQqFQQn7IkQwZkgDo+QkGYQ+J1K+hpMRgqCdu5j0HnOlMz1qjg3qz +BSkVbFtX7ZkahJieStDo+BgEEpEUkLDAhItECfSHCertIWzOYAmBdj9CmAhwkYNgaRtKZO10gSDS +r+XYOqDa7w8wDCQgXAg7AZgHMFeXlWUCUSLR7Uc4aA6QrYvb6UeIlQ1hF8CYDQmd+TwIFeqtIRwn +guvYaHVDBDEHhAcuLXBLIIGNQaBQawWwrA6kjNHpDTE3VcTsdBHFvIdCIQeRTcKNE5Ohvf7kEA8e +H2Bzp5FW6/EwjC1w0UaUALPVPGZ7OZRLORRyHvI5B0olkFKh1e5j/7COvYM67jzcxv1H22h1A70f +TgXc9mA5HpTIqmMN4dgRXEeg2QsQxAyKu2DCBYMDwd10fWgHwzBAbxDDDxPEUYxBv4/eQJdsn54K +USx4yOc8vWxjnKDZ6mB3v4Gd/QYebtawtllDZygBkYdwSumkAxtKCQz80fkVQqDVDXUbWeT1hIPs +/KZt2EQCfiDRHcRodn3kGxZa6fUdJQKKuxBOXmevC71M1cu2Mk2wkNvQcXmOrj9Eq9NHo91HpztA +s9VBqzNAfxCgOwhQKeZRKhbScs1MZ/H3B+j0Btg/aGLvsIXdgxa2D7rYq3WRwEUMB+Auuj5DPwoA +dBAGAfoDH/2h7n8X8g4KaZZmHEUIwhA7ezXs7NfweHMP99d3sbaxB3APyspDeAXYXgmWU0DWT5OK +YzCUqLUGcLK2eceHH0FPQFAOeHqNm/6QBIII6A9jNDs+9us9tLoBesMEYcQgmQVue2nw24VUAv0g +RG8QorjfhWsDcRxjGMSIEgXb0mXwozhCq91Hu9vXAcpiQWfL27q6ZncQI4Gtt4s7eiLGWOB5bKaO +LrGeKHTCCEkUmOTBXl+PK8RJjGI+h0LehRACcaSXCdjdr2Fnv47NrUM82DjA2pNDgHuAVYDl5sHt +dAkyE5izMAwl6m0fwyCB61podIYYBgkUs3VpfW4B4BiGEo2ODz+I4dq6lLkfARAemBBgyoJgo6oE +um+dYBj48H09ubvZqaI/DNAfBqiUCpiqBBDcAhccSZyg2++j1x9i/7CJ3f0G9g5a2Dpoo94O4IdK +v5eVgx8C9fYAwzCGbQk02vpZqLfZ0muoP2epred++qTZz1EsESQx/DBGGAZoNVtYnC2iNVfCfK+A +aqWI6UEA19HZ30JwxLFeIrTe6qDe7GB7t4Z7j3bwYH0Hg5AhiAViuLBdb9RXZxxKSQzDIXphjDDu +odfX92OnF8API0xXS6j2fRTyHgTXJdQHwwDd/hDtdg+7h00c1FrY3u9gr9aDZduotYeodYYIYglL +CDTaQwwjANxLq+Y5ECytknLieBCgFAcgIZVAoiz0BzH2a31sFpuYHkYYhjH2ah1sH+p+8yAEuJ3X +feQsISIrY854WnWDww/0daXA4TkRWp0hBoGEhA1wBaZsSGmh5yv0/BBRlKST1Pvo9gOEYYRqOY9y +IacnTCUSUibpZKkW9g+bWN/cx8Z2Ha2u/vzjtgM/thF2EySqj+EwRKvTR6c3hO+HmJoqYaqc1xN+ +0iU3BsMA3d4A7Y4+zvuHDewc9LBb66PZiZFwW49/jY0D0CjAizt1ge/FxUWEYYiVlRVMT0+bdX/7 +/T6ePHmCL7/8EnEc4+LFi1haWjK/K4QwJb8vX76MfD6PK1euoNVqTWR8Z4HvLOO7UqlgaWkJ58+f +R6VcgZfzTixDngXmbdtGpVLB6uoqOp0OBoOBKcGelZp2XVdnF1WrmJqawtzcXFpCvYSf/vSnyOfz +aLVaJuiXBYGr1So8z3vRQzZhfn4eP/3pT1GtVk2gv1Kp4PLly7h06RLm5uaeG/h2HAcffPABwjBE +rVYzpcrHs+GPZt4WCgWcP38ejuOYoHa73Ybv+ybLPivpXSwWTUnyxcVFzMzMoFicDCjmcjksLi7C +tm2Uy2Wsrq6i1WphOBxiOBya45yVnR8/zrOzs1hYWEClXIHjOhD8u5WULxYLuHjxIlzXxfLyMt59 +910zISAMQ1PSPp/Po1qtYnp6GufPn8fsrD7WL1LSvlqp4saNG8jlcrh+/ToODw8hhDAB7dXV1aeO +EeMMFiwsLy/jb//2b1GtVs2EgPn5efzgBz9ApVrR1QDE8bf5/Pw8/vqv/xrFYtHcZ7Ozs/jggw8w +NzeHYrH41PYvLy/jo48+wtzcHN577z0cHByYa3tlZQXLy8sTP+/lPCwsLMCyLJRKeu3vZrNpysYf +PZdZ9nu1Wh2dy7H9mJubw1//9V8jn8+bbZ6ZmcH777+P+fl5FIvFl8pw73S6uHv3Lj7//HM8fPgQ +g8FgolLDhQsXMDMzg3w+/9wJDIQQAgDNToA7603Umj1wBBAqQL3Zxf5hG/VWhGFk6QEBNl4m0QZj +wDAC9hoh/LCB7doQ3zw4gCX0gJwfxtjeb+GgGUGCoRvE6AyH6PTreLI/gOsI2Jzpme1xjDiOMBgG +6PUGulR1RyLhJXBX6PJsIp0VbntglgXJGPyEo95NEMY+DpsBPLeDnKPXe3YsAUClWRa6DHl/4KPZ +GaAz5FCiBM5dMO5B2Nn6c1n5xOOPlVKAVMAwiHF/o4kgCOHZCkom8IdDrG3U0RlwJNCzeW2ut5mP +z2CfoAOcfqzwaKuFJImQcxiUjBH4Pja3D7DfjDCILMj0+O/Wffzx1g7ubzTg2BxMSezsN7B3MECn +LxFJC8K2jmSun0yvZaXXlO77EjJJ8NW9PdQbPbgug0oSKJmkxzL9nXRAIlF64HS/3ketNRyVHGRi +bLb+8+kOpB5wAFeIVaInD0gJPxrgsBki57XhOgKuLdIsC5hZzEmSYBiEGA4DtDpD1NsSIStAWTYs +oQdV9PqWTlpObGwmvxAIlURnKCHrEXpBE5sHA7i2gOtYEOnMZgAI40h3MIch+sMI/WGITl8hSDzU +uwpfPqij3lVgkGCQOKy3sLE3QKAcSKYHjJiVMxMhdOmvY84IS0vVcwtKAfuNIb68f4j1rSY8h8MW +wPZeHXsHQ7T6DLFyYLlcZ9akA9wYy9oAt8CFBGwPjHOEkOj6EqoRoRe0sHU4hOvqdRqz/WUMCKMY +YZTA90P00v3t9iX82EWjB3z9oI724IEeKEeCWqONx7s9hMpFwtIBd+GNZVtRh5cQQgj5NpQCEqkw +8GM83GojThI4QiKJBhj2u1jfPER7wBDBBYQHi+uSqnqAnT81+VK3QQWCWOHRkybiOELeZVAqQRgE +unJOM8Qg0Nmqwk4zQznHIOLYrocYhA083uuhmNuHLfQ6Pro0r4/BUCf3ZOvt5nIu8l4O3UGAg3oH +9WaAnq8A7qDdl/j6QQ2tXgyoBAwJ9g7qeHLgI1AeJHfAuZsGiV1IxbG+3cH//eMmCh6HTGIE/hAP +1hvo+RakyINxD5bQQWXFLTTaIT67s4fDRi9dgzTG9l4Nu/UAzC6Cc12au+sDtx7V0fPvQ3BAcOCw +0cbGbg/D2EHCbHDb1gFdbiFUFpo9hVj6qLcj5NwOcp5em1f3RfRxSaTCYBigPwjQ7g7R7ANSFAHh +AsLFMLKw34rhx31sHw6Rcxkci8EWXK+bKnUAww8C9Ps66HfYCOArD3BcWNzVwUPLBhcOekOGO+sN +9IYROFMQTKHW7GBzu4thbEMyB0x4emJAWsYZSi/IK5lCewgk9Rhdv4EnB33kXKGz+gVHkiaD+GGM +ft9Hd+Cj2Y4RIg/u8PR1PRNc7Qxj3HpUQ6fnA0jAVILDegOPdwcIlQvJXQjuglkeuKUnILf7ETb2 +e2j3fDzeqqOY46jVG6jXm6g3BxiEAo5XhuXkR8EV9rJjqQyMKQBpWxkMSkndLxESvQBQzQT9oIud +mo98rgbHFnBtyyx/lUiJMIwRRGlfcxiiN4jQ8wWkKAHMgWBp+eS0nPEwFDjoSPhRH7V2iEdPWumE +DwCQaYJQjG53gG5/iFZngM5QQLhV3Y+wdFa/lVZ6QlptqzdMcHvtEIPhAAwJGCQa7R42tjrwpe6L +gLvmfDMu0OnH2NzrodMPsLHTRuXOFurNDprNjq5iNlSwnXzaV9YlrBVnUFyg6wObe0O0uhHWdjoo +5XdNHyJOZFraOobrOvBcXSKeC933X9vYw2E7QQwXirv6XhVPLwll1lzO+jIA/BiodxXCeIh6+xCP +ttqwLQ5bMHCmECf6+swyzltdH+0hB7PL6TWml4diQk/QFQAkAwaBxL31OgI/gOAMQgDdvo+t3SZ6 +PkcMAcVs9AOFB5tNRLGEYAqcSbQ7XTzebKMf2YjT5eGEyOn3yLLqlYBiCn4k0ewrJHtDNLsxHj5p +w3E4PDstoZ8uNxVGEcJQX1e9QYD+IEKnz5HwAmBLWFwhkA7ub7Ygf30PlqWPbbcf4slOA/1AIGEW +5Asuf/acOwVggOIWmFJQjGMYSci+Qih9NLox1ne68FwO19bB0exaSNIxdN1XD9Ht+6g3IwQyD5ku +1+Ywz0ywAE/X2lYKsJXO/uYJ+oHEfkvCj9vYb/jwPBuurZ9L2XtFcXrdhQn6gxB9P0S3n6Af2hCJ +hXuPOwjiR7AtDgZ9jaxtNPV5A0/7rbmxMaHjsuTT5wZLl15QNpq9CHfW6jhsdOHaAp7LcVjvYHe/ +g3pHwo91hQN93WVZ+Pp1ubAghY0wUVjbakHKR3BtAWEBvh/jyW4Lrb5EEOvxIM5H8YNAAu2+QpwE +6A6a2DrQE+UdKx0rkRKJ1MHx4SBAbzBEu+ujPeAIlAdmc7jC0hPlhQPJOXohoNoKQdzDfjOE59bg +2PrzLEvc1uuIR/CDaOy5l6AfWJBCZ3tz7o5NyKeM75fBlFIvsmDg90JK/aF0eFjD//f//f/w3/7b +f8PGxgY6nQ7iOMYHH3yAH/3oR/j444/xox/9CO+99575XbNuQ5KY7NXsz9nax1KqdDF4PXBlpesa +ZhnTjp2WdDghYzTbvuz1wzA0wcYkSSZ+NsuWzdY3ztbA5pybAO74+tf5XB65fO652/As2breWRA+ +C6pnAcVsG541UKaUMmsxR1Fsjle2fdla1ePbl2WuR1GEOI7NcUkSaX4/OyZ6bQoB27Zh27ZZhzRb +v/q445ydy5c5zrZlpwOx3yJb6sg5D8PQ7NtoWyavJSG42Qbbtk3Q+0XeP4kTBGFg1qPP1nEafy3H +diaOUXauwjDEYDAwZeCVUqbMfiFfeOYxGP3uACpdQ9S2beRyOb2mOBfgYnKQPwv4R2GEMNLHxbZt +OLYD29El/scnV2TnMkkScxy/y7mMouilt/lZ1/rdu3fx3//7f8e//uu/Ynt7G3t7exBCYHV1FVev +XsUvf/lL/MM//AOuXbv2na8lQsjrEwQBhsMh6q0ebq81cHe9jp16hIMO0B58v9tiCd3R5ogRDZoI +hy39zJN67WvTwRa2KcukZKyDoTICUyEgQzAZQiURgPRzVAEJBBLFdadS2HotakhwyPQ19JeUOrgq +lYSU2erQFqQSaYmwrISznWZ6ZCFkBaZiMBUBMoJKQigZjV5bybE1p3V4TjELEpYu82bl0iwHezTo +g5NKUANQ+rOUIYHFYnBEUEmAJAoQRyGiKEQYRnrwxPL0+t6ODqwzbpt11EavJ802WiwCZzFUEkBG +AeIo0G2UKNJZ91x3HGzbMkF9KL2fSaIQxQnALUDk9NpRtl6z3AwenPB5oGSiP59kCJWEQBKCqRBc +RZBJCCVDJHFotne0riJPO24WFLeRKBuK2Wb2/LH7+0wq/ZyUgIzTayvWf1aROa9IIkglgWw98ey8 +MAaAQzExOr/ZOlqWDc5tMOGkCfAsfa80qC9DQOprCCo235WMx64fTc+Yt9IgvZWu0ycguIBjc1ic +IUlCyCRCkp6/KJZ6zTcrB255aflR75ml6PU5SdecUxE4IiA9HzKJkEg9sKOYBSZyemKIlVYY4OMT +D5S5bqWMoZJ0/2Ro9pWp7H6Z3F8zSMKF2Wd9fCf3Vyb6Wsmu1ziW+vq3X3x/XxulEPWeIO5tw0oa +KAgfJcfHzMzMxNf09DQqlUq6ZFAROc8zk4stYb1wW40QQsjbLQgC+L6PZruHW2kbfrsW4qADtPqv +972zdYUdS5c8T8IewkET4bCLKE4QxRLgjm53WnrCn14DNluTdvQ5NmoDRRAsglBx2uYLIOMIUZIg +jiXAbNNeBtOVFBkkOBIw6HYZsrY3EigpIZWEGVYz5VrTbYDuX0gIgDsA18vlOHYa4I0DJHGIOArT +8aZYL0Vk5dLS2Q4sy4YtFGzBoGTWDg8QRiHCINBri1reqP9gObAFgyUUBBK9j1GAKArhhwGiMAG3 +bPParqMDN0rGUEmkSycnCZIEOlPa0pP6OMsCWrE+FlK3V/X3BEolmKyrq4OziglIZutjK2wI4eiA +IAM4U0Ay6tPoIH1kzpc+vrqvpCtzCT2hMj1HuqS7gOAKFksgWGLaaUmcjPYja5Nmk1aFA6X0ckOj +9mE0ajPKON2fxLQTFbI2sQ4IKmangfS0LH3aFhVMB+kESyAjH0ns6/Mbhvr8Wu5kG9n2dECRKzAV +QcUBkASIogBhOESSKChYUMyC5RZhuYW0nHo6+fOlpUs1KQWZnrvx/gBkBJb1Q1Siv2e/A6RrojNz +LMD0pAhdMtlOqx9Zab9I94UZ4rT9rY+vSvsOKonS7P4EUikoxSAVh4KAQpp9befSdc71JIesP8kQ +w0IEjhgyDvU5TxJEiUScAMLWWc5ZgJlzAYYYXOm+FhLdp88SdhIFKOZAMcecF8aF6TsyGabXSDi6 +9tX4mDpLnxliLJitAMh0zeEECnyiz2DZOvMbRyZv636RhFJx2j8cXZ9KRvqeye6TNNaSVdBSTB87 +MDt9loyyyxm39bGXMZiMIJDtUwKlYr2WMQQSJcCE/j0hBGwBWFzp4xwHiJMoHT+O9c9Zo0CusNx0 +WS/oiT0qBpPZ8yLtjyX6+pLj91h6nysmxu4zR/957DPB4vr+Qta3VUACC4kUYJaXVp/7LvcHkFUV +0Ocge0bEo/tEjs6FPg9ZX12ZvjoYh2LptZyeE32s0vMh0us5TRVQUFBJOuak9LFiJ/TVswn6LB0P +AOMAt03fVUE/F3UlBJ1koJ+JEaI4Rhgmejvssedi9tl5wmR+01dPIjBE4CpMnxP6K+urS5VWQ8v2 +NaswmPaHZRJCxvp5r+/fKH3O6D65ZDqr3FxX6f0BAEwlALJ7Ir0PzedHnD5LZHo/6Ex1cz1xJ93H +dIyPCz1apmL9uub5n30GZZ8P6fhLNhpnxkX05znEaG17JmzzOfm6qXiIqLeNuLcFD10U7CHKnsLs +7OxEf396etokFefzeXieZ+JhlmWdijjKqcr4zoLQpVIRFy5cwHvvvQelFNbX1836u7dv30Yul8Ps +7Czm5+eRy+WQz+dNVjHn3ATCXtf2fdd1sh3HQaVSeeXbl23bd3ltxhiKxSKKxeJLHZcsuP4qvKrj +/Kq2JVtH/HURlkDeyiOfz7/U7zHGzHGfmpp66ffNjnG1Wn3h33nZ85ydSwCv5J78Ntt8nEa9gVq9 +hlu3buHhw4d48uQJer0eAKBYLOL8+fP44Q9/iEuXLqFcLn+riSiEkL9McSIRxbpjEfkKoQ8Aep0w +3Wg9sj4eYzrAyjgUFJJYTxyTcYIkjqFk1qllEJaVNs4tJLAgJUs7y4BMOGTCoCSDkjwdF0onYHE9 ++1ekDWfdsc/WW7Ky5AlkGRQykZCJhEosyCRJXzt9TaV/WHduRRrgdiCsUQeLjZUue247lwFSMvix +hIwlklhBJoBMAMABbCddu9BNS6c7Jtv16Ua07kgpxRDE0PsRKyQxIBMGKNusM5XtfwJgGOuAtUx0 +pjQDB2OWLs8l0g7V+OzaZ+0T07+fdUAlJJI4QRJHkAmDjDmUtNLBnbSTla55xQXTZbGtbIKCm5YG +e7GJdE8dC/B0zSxAMYVEMUgJyERBJdl55npyhMwWENffdRbS5Pnl5vxaZrb9aLcFoHRgXgLpoBJD +kjDIRJkvpRgwNnEwKx8uhA3GnPQ9LUgwDCPdGdTXgwIUB5gHYbN0AFYPvpr14555cpjJ1o7jBDJm ++iu91vT55mDc0VkF5tgfLW+f/ZmDcwsKDJLpnHSVcCSSQcZ6su1of/XUE72u3mh/uXDS9xVH9leN +9hce+LfaX0IIIYQ8i47NKfihxDBIEIcxYl8hDphuk1h8FEiwsvY7H60ZOiFtAzGBMI7T9mfWpmVg +TJeP5kIHvZhwTCUhpSQSGUMmQBLr5Y+k5FAyCz6MLeliMjfT/+U8Xc/U0VnWlg0JBj9SQJi2q2OZ +DtB7EA5Mm1oHENNS57FeZ1y/f9oOgQ1mWSaQKyzbTByNpUIU64F7/R4KSnKA5WB5LJ0gqdvOQawQ +REq3NxMA0BmnzLJM0GJ8XWkpR+0zmbZXVaKXDxorl6TXDk/7VzrQk50jCwpALJUOXCUcScxHfaWE +6eMhFbK+kll7WGTrpTvpOWLpNkn4sYJKklE/RR3Zj2yt2awKEbOgmIKSDFJBt8MTnaylsnaxHJsM +avbHTjPyR9njjNvZZYZEAnEY6WBuLJHESI+9C+644MJL98E2FasSJRFHMr3G0jmxCYdKHH0s07a+ +Xsd8strRy2Nm5SU9cVfoPqvSxz2RMNeYlAoq0X1gqLE+iDkfFjjXQR9hOab/yZg1dm4AJYVOSjPn +met2fnYfSZjX1dnONixLr5krxq9BrvszUBwyUfDjGEms9PWXwEye5pZlAoymKhgXkIlEnF5nMmZI +4rS/p7L+uAvL3HtpH0/oMtRSZf1wPR4gkwRKjl3y2faPBbFH80BsE4zNnlf6frXGJriPnSHGAM51 +qedEQUJfi/peTvv+6bHTE34BLkR67Kw06Jhen+mE6KzMtM7218u2BVGSPs+y7h83Ewz0c8WGYhxB +nGAo02dPrNLxDxfMcif3R1i6RHoWzGVcnxuwtGqZHk+QMh27kKNJGFm1OGbuMdvc89k1K6VElEQI +olAHN2W2xrdlgsn4zvdH9gRXo3EDWFAMkEo/K2QC01dPEpleB2PPP7CxsRgLwvRfx0pis8nMdAYF +JiwoLtJxFoZYcijJR9d4eq9MTNrmfPJZy9NnLOMIEgk/klAyu585GFxwG+l5c9O+axoAfcaScVmp +cggLMpaIYj7RV9fXl23GI7Jki2zMaXRtC32dQGesJ1EMJfX1qMfTsnGw0ZiGGUNKx0SkVJCx0ONg +MRsbC9PHJ1u3nnORXkdOej+Mj/PxNJjPkSi9DbpNIM3rmccesuX/xj4D2NjzhY+qD3636+4v06kK +fGds28bFixfx4x//GFJKtFotHB4eotlsIgxDU3J8ZWUFs7NzJiuWEEJexO7eLm7fvo3PP/8ca2tr +qNfrUEqBc45KpYJLly7hxz/+MVZWVmhtb0LIS9IdJ3AL3M4hm8KVlYzjwjIlmUbBzHTgCw6ywSBl +ORBJfqJyiung8NHaUqMsbDmRDTHqsGTrHY39rsgGDMYaz2n/i4t0EE1YUNKBkt7Ya0/0vvWkoGyf +JgYieNohyF74xEMFpnTHmysbsNJBA+mk5cBH+2062xMl29jJr4c0k53r8mgTr5d2LMazp0cz36XZ +bn28rHTg7lklujDx2nqgx0rnM6QDLJY7yng226LGNpyl54ibQQUmLDP49G0ye7OS52aiAJfgY8c3 +qzSgO9JjmdhsLCidbk82mYGPDUxOdF71TuvrIj12SljgVmIqEOgZ5kdmkWfZTun1g/FZzGlmNZcO +lMxNHK+Ja45bz+xIm2OBNDMk7Tjqa9zV22XWoxNm0NQEmI+ZYKFfT5d55+lglxK23t9kdM9kQW+l +VDpgxdJ9FOk69fyE/fWygzq2v2k2yQvsLyGEEEJelC61yoUN4RTSYPDYQDS3Tdv5pGoruv2ng9Cw +ssF3J21ryVG7OG2HTLblEyhlpZnGus0zygRM2wInDHaP2mrCtFuBbD6rNH0KlbZHxgMmo0m4urSt +UhxirH2E7JXYqB3O0j4FACjOoaQYe49RZqpu84xKkyulwJWJfJt9n+zXjF5Xt52TUTtN6Uqbo7Zg +tu9jbUGzT3xsTq+E4jqLW421R7O2b/pCR14r3SYTbGYAl2Cc67btWDvNtImzNrI5Rvp6YGkNb844 +mLJ0kER6Zp8m+nkTx8Q6cp7GAjtc1wPjYwHip8+vpc8Zs03AD2m5ZB3otkbbYPo8OmCUVST7bll6 +Y/1Appf9UUxAKR2Yk5aXZmGO9y9HfUzdHxZpH8GabDeb/msa0DX9nPR3pG2um8lzzdMJE2lwduza +Ma+dlQJnChx2ulySPlYyXaqKZcs88WztYjF5HZht0RMisvsoC7qP96nMfiiunxtcmMkxUsZH+vTZ +dTXqmyil87DN32UZ4WLyWjy2vDQAxViaRcpH94k9fuye7vs/fb+lE4LMGIgw5xxpRTMxdq1PXt/c +9IWY1Odh8lmC0fkfH7+YCObqigB6crENZXtHtn88Y368vz26v9J/BVcKKrt+xo591u+cuMe/c0ds +8h5h5jkiIIWdjsMksMb2I6u6p48Ln3hu8bFrazQWM7kkR/ZM4nD1+eY2lNRVQPTzQGdGm2srvZaz +6yrrp5vS4uk2KelAKC89b2nJcZ6NZYxvE4CTjpsZWxCmIiEXNqT09EydrPIdGxsrYdk1PvY6XOhl +FsbuSTU2zqG3azTGA7NUGht7lggzZjD5rEr79On9kJ2z0WdGtq/Z80ml5eytdJzAMdemNNne2etl ++3fktbLP6fF7jLyUUxn4tiwLKysrsG0b7XYbjx49wtbWFoIgQLfbRalUwoMHD7C0tIQoipDzPDCm +S0NTZiYh5DjjJfS3trbwzTff4JtvvsHW1hbq9TqmpqZQKpWwsLCAK1eu4P3330c+n6dJNYSQb0E3 +XIXl6k6QnmE4AACAAElEQVQCkGaHjGZqj3d2TKkyziDSQQGlZNpJzAZ41KiBPRH4zdaKlmONenVk +WwAdXOdHOkLHBI7BoJQAxGiG9GRnK9uWLDOapZ2fLKj3klnJ6Xsi7RhBWeBPlcOefI+Ty32PvR5n +4EwAXIKLZPL1zADi+OtkAVllOj+j92V48Rm2Y8fbdMQtCEuOjifGjuOoq2W2SS/v8SL7+6Lbknao +IcxaYqPyben1cnTlo6zD9tSxPyEAnx13AEyk1xDPBjoT8x6j63MU1DWDMkfLyGfXQdopnJwnwMcm +IrzIuRkdC56eFyhHTxQZPw/j1zKODhgcfb3seuOT+2sdt79j58GcU37s/up9HR9oyu758QkH1Osl +hBBCXp00S9kGpOUc0xZ8Xjt3LLDNGBRP27NSjrX7sjbtWPlPlraZ0jZoVtI2K8073l6abH5lxVHZ +qI8xtrxQVjx79HrZ+wNZn2C8TaHb/wC4hFLWM9rhgNmQNOP5aD9h4j2yrVVZ0HqsTWTagNm2pL+s +hC56Azk6JpCjMr+jI27aSMe13fRUVKEDPKZPc7TtO8qkn8imPRr4VTrYo7gcbc+J+zE6pvql03K4 +UFBCjoKAE32C8f6JSPsCk/1Fc94ZTycqiLQ9e+TYMzHaD9PW1PubZWArZafHIA1UmfcdBce/+wzL +8b7u2DFQTtovkmNt/MkzO2ovi2POBzv6Lqafw7gYBSzNfXS03zEWAB27D0d9sqy6WbYetkz75enr +mOtMn6eJ/gIXul8wNtli/D6Cec/xgLTSMTfG0/63Pp/8uInSR/6L9Aoyf8eyvszR9zju9KS1o9Ll +Asb7LpMT6Uf7PRrH4GPHcfI99KEQYCIN5IpRv2h0bo8uFyGguALSMZDx68EEx9noXnvqWgGguDBj +CFn/bjJLGmP3Z/b+Y9dVtuRA2kcez65++vnwqqpujY1dvNB+jD2zzOcTM/f8iWM849cgxvrq6fXG +0/vl2L76RKIGN8ciO2b65+WR8zaaOPZUf/eZhyM9P8KGSO9lbp63o+cpJsbTjk5Cw6gqABNQ4ugS +GfzEc/nUs0Q6EEfH2MYnHxwd3zvufoACkAbf5ei5lB23k55N46/1vS9t9pY5lYFvIQQKhQJmZ+ew +urqK9957D4PBAHt7eybz+8svv4Tv+/j444+hpMLyygoqlfJLlegmhPzlCIIAtVoN9Xodt2/fxjff +fIO1tTW0220wxjA3N4fr16+bEue5XA62bUOIb7tuCyHkL1oWnEszLyYDVsdlQ4+VhcsCeUqkjeXx +Hzsa+EpfS6UN76MBzPR30p98bodI/7guR8fYWAf/uG0Zb6B/68Z4lpPB0/c7Zj8mtvn5AU49qz4b +yErLyx0zoDKJ69+beNvnBT9P3gadbZ0OFnJmZuSP1nF6eqt1ABdPXSuvxvOPi5mJPz6g8NS2vHjw +X2XHMC2PDzOpYHIAZ/w6mvh7xvUxS79PDtqOfv7FT096DMD1AKAZvJrY8rHr7cVfd5RRro7Z36M/ +/vQ9PL6/+nbmo8SXickL7NVdEoQQQggZyTLujgR9X7yde7SdwXQ5YRxt005mC+p2d9pmzDJOj2vP +P2u7xwK42ZboVs2R1zvSJxhtd9qGOa7dfEK7xQQ1n/Ue5jV0O3vyaLEjrz/+fmllHdNWE8e0q44E +jZ5qux3f7jv+2B4NQB2dHJztkw5wPBWwPm4/jvy6UqNgNtJ23rP7ecf3F827MnXCsR8L/rOxrTRt +c562VTE5ydJMong9QZbRMQDMusFZ/0hvzNg+nnQ+Tp54Mt6fZFA6m3ksmDnq3xzpd5yQEW1Oa3bO +xypjTR7j4+7ltF/Jx++Bk/pUoz7aib/71L6O/994IC7b9pftt6XLM6XHXG/L6NyY43fkWcBw/LHT +r5eeC84mJjCzE457lgiQHYeJvz/mWB899+ZeGOvfsaMTkM25HZtoM/HsVMgm9j+1DdnPvLYg5OjZ +fdJ+TE5IOjIxxtwnL3fOWdbPTgcvju2rP/UZY26OUV9/rO86ft5eru960jEQYz9x9Pl40r7x9HS9 +7LnMniV6DOeZY2zj40QnjrEd83onPvdG408nfwaQl3UqA9+ccxSLReRyOVy5cgWNRgNRFOGLL75A +o9FAo9GA7/vY39+HUgpTU1OwHRu2bVHgmxByLN/3cXBwgLW1Ndy9exd37tzBkydPEEUROOdYWFjA +j3/8Y/z4xz/G5cuXUSgURjPoCCHkpaVlzhTP/hdH/nD876Sz8hmAZ0RIn36dF/qd573/6N9Hnahn +vK758e/6nGTPfr8XOnbH7UPWgT5m+096qafe99sH9EcdocmA+oufolf9+fPs43J0WOC7bcv4gJA6 +eccnXv64QaexDv5TG/htjs/4QNrYtfZS98hJ+5t9f3p/J97ixO0f7e/EPfDargdCCCGEZEz21ndq +Cx5pZxzb/nz69SbajMCz20w47jWP/Z+nX++Z7Z0T2s3Parc89z1O/Ktn/+vRttpJx+S5baQT2n14 ++dcymdMn/8Qz9/GF+lYv3F884bWe+ftssg/wwr/3qrzgMXipY/Fir398U//5feKJc/5C99Az+p/P +OTfP/N0XP8QvecyOP3YAe6rvOnnIXuTYjV7v+dfo6FkycazNP73A+MWLPjOeuR/sGdvwbY7rt2CC +xU/vx7M36bud8xd7Nh79n2d8bn2XcZTjjsHJG3TyvqWTXV7+XLIX/0z+Nq/3Sp975HlOZeAb0MFv +zjmmp6dx9epVAEA+n8fMzAwGgwEsy0KhUMCVK1cwOzuLfD4P27bf9GYTQk6p7JmRPVN6vR5WVlbM +B9CPfvQjfPDBB1hdXcXU1BQtm0AIeQVeZobrkd8b+/b6fudNvu739X4v+XqvZT8nr4PT0Y15+ri8 +vu36Luf0dW7g93fPvPhbnLoLhRBCCPkL8m3b7q/qtd5wO/hU9UGOvOh3fv1XsZ2vYidf5fH6Dufr +lW3Dmz4Gz3/97/Y23/c98X33vZ///t/f8XvF9+iZPP4n78fp7Ku/zg387skV3/1lXsNn8hu/tv6y +nNrAd6ZSqeDKlSsmWPWLX/zCZGjato3FhUUsLC6gXC4jl8u96c0lhJxSrutiYWEBhUIB1WoV77// +Pnq9ngl8Ly4uYnl5GVNTU/QsIYQQQgghhBBCCCGEEEIIOWNOfeA7n88jn89jaWnpTW8KIeQMcxwH +juNgamoK58+ff9ObQwghhBBCCCGEEEIIIYQQQl4hquVLCCGEEEIIIYQQQgghhBBCCCHkTKPANyGE +EEIIIYQQQgghhBBCCCGEkDONAt+EEEIIIYQQQgghhBBCCCGEEELONAp8E0IIIYQQQgghhBBCCCGE +EEIIOdMo8E0IIYQQQgghhBBCCCGEEEIIIeRMo8A3IYQQQgghhBBCCCGEEEIIIYSQM40C34QQQggh +hBBCCCGEEEIIIYQQQs40CnwTQgghhBBCCHkpjB39fzbxnRBCCCGEEEIIIWcTw9P9e8aY+TrNKPBN +CCGEEEIIIeRbGe/0UvCbEEIIIYQQQgg545j5z5lEgW9CCCGEEEIIIS+NAtyEEEIIIYQQQsjb66xk +eY+jwDchhBBCCCGEEEIIIYQQQgghhJAzjQLfhBBCCCGEEEIIIYQQQgghhBBCzjQKfBNCCCGEEEII +eS525PvEv52hsmeEEEIIIYQQQggZw8b7+vp/zmo/nwLfhBBCCCGEEEJeCEv/k3WAT1rr66x2kAkh +hBBCCCGEkL9ULA1/Mzy7T3+a+/wU+CaEEEIIIYQQ8gImc76P6+hyzsf+PPnvjJ/ejjEhhBBCCCGE +EPIXjU18MxPdj36ddhT4JoQQQsiZxxgbm4nIAMaA098OI4SQM0YBAFj6HcCxHeDxjjDjZ6dzTAgh +5E1g4Gzs84Qa8YQQQgghb4Yy/3lmi+y09+8p8E0IIYSQt4MZMKO4NyGEvE5K6Wfsae/sEkIIOb3M +ZKn0v9lSGtSIJ4QQQgh5A9SR/x9rk53U9z+tWeAU+CaEEELIW4EBEBwQnIGfsgYXIYS8HRSg5MQ6 +3yeVPeOMupqEEEJeAAM4G7XhqRVPCCGEEPKmSEBJAAoMDJxzcM6fW+ntuP9/k2g0ghBCCCFvDcZ0 +8Jvz0zfbkBBCzjylACQAEjDgxE4wY4xKnBNCCHlhjANC6OA3fWwQQgghhLwhSkGpBAwyrajJnvt1 +GlHgmxBCCCFvBcYYBGewbQZLgLK+CSHklVLQs79jQCXgTJmO7nEBcOB0zfgmhBBy+mSfGYIz2BaH +JU7vACohhBBCyNtt1OdnSge+x/v6R/v9pxkFvgkhhBBypo0CLwyOxZF3OFyLgVMrhxBCXimmEkCG +4CoCZwpCCAghTAf4uOA3QAFwQgghJ+OMwREcOYfBtQFBHxmEEEIIIW+AAlMRuArAEYFzZfr74/3+ +LAh+mie805AwIYQQQt4KnDG4NkPB43BtToNmhBDyqskYPA18s7HA99HOMGV9E0IIeRFm8qoN5F0O +1+IQNFJJCCGEEPIGKJ3tLQMwFYMzTPT5jwa/AZzafr/1pjeAEEIIIeS7YoxBCAbX5ih4HHmXwbUV +bCEhJUOiTlcDjBBCzgwloWQMJUNA+uAIIVgCWwC2bcOyLNi2/VQAnLK+CSGEvAjOGVxboJjjKHgM +ngPdhlcMUjKoN72BhBBCCCFvMaUSQEZQ8RAsCSAQweKALTgsyzJfJwXAT2Nfn+ZREkIIIeStwDmD +53CU8haKOY68w+AICcEVTmEbjBBCzgSlEsh4CBl2wZIhBELYXMK2hAl8jwe/Lcs6tvQ5IYQQMm58 +uSLP4SjlLBRyHDkHpg0P+vgghBBCCHm9ZAQZDaGiHpjUfX5LKNi27vOP9/sn+/ocnPFT2d+nwDch +hBBCzrRs0ExwDtexUPB04LvoKRQcCdeS4EyC8kUIIeQlKAWlEqgkgIp7kEETLOnDYjFsC3BsAcdx +4DiO6QhPrvctTm0nmBBCyOnBGYPrCOQ9C6WcQMEDCq6EKyQEteEJIYQQQl4PJXWfP/ahoi5k2AaT +Q9hCwrEYHNsy/f1sorup9sYFhOBgfFTq/DT1/6nUOSGEEELOPM71LEMhOCzLRiknsFBhCCOGRlch +6caIwJEoDqlo3h8hhDyPkgFUEkKFbSj/EPAPIVgfrs3heQ48z4Pruib4nX1ZlgVLWBCCg4tRxnfW +CT4tHWFCCCFvXrY8BucCtm2h6HEsVBiikKHekZA9iZDa8IQQQgghr5aSkEmgJ7oHDSj/ACxswGYD +eI7u72d9/vF+vyl7bomJ7O9xp6HPT4FvQgghhJx5jDFwoRtbti1QygssTnGoREEmEr2BhFQCSjHI +N72xhBBy6imoJIKM+lBBE2q4Bwx3YeVsuK6NXC5nOsJZZ3i8/FnWCdYlbE/PrG9CCCGnx/iEKJGu +IVnMW1iscrC0Dd/3JRJJbXhCCCGEkFdJKQWVBGmfvwEM98CjBuycDc9zTJ9/PPA9nvk9vrSZWeKM +n56J7hT4JmeeUgq+72M4GGIwHCDwA/iBj1wuZ27Q7DshhJC3z/igmWVZsC0bOddGMWdjumRhGIYI +wwR9P8IgZAgiIJEKiVSQSkGptMGnAKUAQCH9NvrzOKq2SL4FBZWWjh5dc2AAAzPXr+4fnI5OAnnL +MWD0oEsfakoBkGAqAVQCFQ/A4j543IZgAwgHyHkWCoUcCoUC8vn8sZ3h8ZLn4xneWQk0QgghJJMN +lmbVQnKujVLeQhBa6Ae6Dd8dRhimbfj4pDY8oNtZALXhCSHfk8m+ndKDCRN9O90OpjYwIeQNONrn +V/o7gwSUBGQExH3weAAWN2FxH7bDkMvZKOTzKBQKyOVyT/X7bduGbdkQwpoobc7Z6arMQ4FvcuYp +pdDpdFCr1VCv11Gr1dBqtTAzM4PZ2VnMzs5iZmaGAt+EEPIWywbNhBCwbAuu6yCfcxDFEaI4hlAx +Wt0AnX6MrooQhAp+kiCOlQmCKwnINPqtTCfWRMMNRaNm5FtQUkFKab6UlADTJT7ZWHmo0zI7lrzd +GJiZjIGs46skoGIwFYLJEFyF4MqHxRK4IoHjeigW8ygWCygWiyb4nc/nJzvBR9b7zsqcE0IIIUeZ +5TA41214x0bOc1EuRIiiGEIlaRs+Qneg2/DDRCJJJOKE2vCEkDdHKTXRt5NS16Wg/h0h5DQY9fkl +kH1XElCR6e8zFUCoADZP4NgSnpNL+/sFFAoF0+c3k92dtNKbbcGyBLjQ47Cn8TlHgW/yVhgOh2g0 +GtjZ2cH29jYODw+xtLSEOI4hhEChUHjTm0gIIeQ1YZwByWTw27Et5HMuZBIBSQAHDC6TsJUPkfgY +qAQ8ThAgQSwTRImElApJonSnVSkoJcGyDN0j76kUDZyRlyOlRBLH+ksmSOIYjDMIYUEICxACzLKA +U9hhIG8XxvQ871HnVwEqAUMCyBhMBWAyhOAxbB7DsQRybg75vO74ZkHv7Gt89vfRjG8a7COEEHIS +xhmYZHriqrBgWTYcx0bOcyATF0hCOGDwWAJbBRDJEAOZgMUJwiQBV9SGJ4S8OVJKyCRBHMdIkhhJ +kgCADgZxvfwP0jYxIYR8nxhjaRtoPOCdgKkETEWADMARQTDd53ctB/mczuwen+h+tNKb4zrpRHd7 +or9vqrydor4/Bb4JIYQQcubptQGFLnVu23AcB3Ecw/M8xHEMKSXiRMd3BGfwnACeDQRhgjCUiKIY +cZIgjpOJrFwp5US2iDrhz4Q8T4wYgQoQxiFkFEKGARjnsB0XtuPAFg5c7tLACHmtnu6IZuXOAAYF +xhQ4BDhzYAkHtmBwHdssHZR1fo9mfOdyuYl1vyxhTXaEOZvoEBNCCCEZxhi44LAsAdu24bouEhNM +ShAngFQMggGeE8JzQvhBgihSCKMYcZwgiRMk45V1lA6EZ6gNTwh51RIkCOIAiEOoKEQcBgAA5ri6 +jydsuNyFEOJNbyoh5C/IyX1+3e/nDODQZcoty4MtAM914Xme6d8XCgWUSiXT7396srsefx0PfJ+2 +pc0o8E0IIYSQM2+8TGIWAHcdPWiWJFkwW4FzwLEU8o7CwE7gBwxBoBAEMaIoQhRF+ndkAplIU74s +Wzdw7NsEGkAjzxPKAFBDyGSIKBxCDn0ILsCRgyU8eMghJ/I0MEJeueOCzeN/ZZ6f6TOUCwuCO7Bt +C7ZtwXUceJ73VPA7+3qq7JllQYx1hAU/naXPCCGEvHlH2/C27YwmoKZtcUAP0tpCIT9Q6NsJAp/D +DySCULfh4yg2k12TJKE2PCHktYtUBKgBVOIjDoaQ/hBgDEx5sHgOruMhx/OwLftNbyoh5C33wn1+ +zsE5g+C2qZZp2xbcNPDteZ6Z4D6e8T1e7txxnHSN71Gff3yt79OCAt+EEEIIOdOy9WMVV3p9QMuC +4zh6wEuNDZhxDiE4LMFhWwKuY8H3bQSBjTB0JgPfSTKZMWIGzJ6dLUKDZ+Qkvi9gcQWOBJARZMRg +WRyeK1DMuygUcigU8rokHiGvyPEd4KfLkGUztbMS5Tr4YJsKGllHOJfLIZfLmSzv8Wxv27FNxne2 +ztf4zO/T1hEmhBDyZo234S3LGqu4ZENJpdvjSpqfFYLBsQVc18ZwaCEILGrDE0LemDAMYXEJwfSa +uTLRlbtynoVC3kGhoEsG2zYFvgkhr8ez+vvjf54MfOv+/nF9/qy/n/X18/k88rn8iRPdT3Ofn0bW +CCGEEHLmMc7AFINlWVBKwbbticGubGAtC4zbti7dGwQBwjBEGIYTg2byaKlEGiwj39FgMIBt26aU +uZQStm2jVCqZr3K5TAMj5JV7kc7w0Yy7o0tHjAe/TQDcy8H1Rn/nuq4Jeps1vhk/cRsIIYSQrA2f +tdHH291SSdOGzz6XnLEqJONt+Kws+kmBb4Da8ISQVysIAtP2BUbPk2KxiGKxiHK5jHK5DNd13/Sm +EkLeci/a5z8a+D7a58/69eMBcM/z4Ll6fe/xie7Za2XZ3qetz0+Bb0IIIYSceUczRpRScBxn4t+E +EBBjawf6vm8GzOI4nhg0OzpgZgbgTlgr8Fl/RwgAeJ43EfSO4xiO46BQKKBcLqNaraJSqVDgm7xS +z+oAj68nn2V8m87r2CSh8VngnudNdIg914Pt6GfqeLmz8Znfp60DTAgh5PQYb8MLISYmrQIYBb1t +Sy9l5LoIggCe55lJq9SGJ4S8Cb7vT/TvkiQBgIn+XbVapcA3IeS1eV7AO3tGHZ3sPr5M5Hif33Vd +82WWNEvLm48HvSdKnJ+ytb0zFPgmhBBCyJnGGINS6qmMEQAm+A3ojJLxUuiOo0sjZoHvJEkmBs3G +B8yOGzQbR4Nl5Hlc1zXrTWbXneM4JhugUqlgenp64pol5FU4KfB8XCf4aEc4K2U2Hvw++mXbNmzL +NkEJE0Rn/GU2kxBCyF+Y49rw48FvpZT+PBGjz6XxjKTxak3Z+t4vEvgeR214Qsi3NRwOAcA8g8Iw +BGPMBL4rlQqmpqaQy+Xe9KYSQt5iz5po/qzAdzY+Ol7y/GifX1d1c2DblhkbGO/zn+aJ7hT4JoQQ +QshbgzMOpLGW8VI7WYMuC3pnmSHZ96NZIkmSHBv4PgkNmpHncRwHSZIgiiIEQQDf9+G6LorFIkql +EqrVKqanp+F53pveVPKWeV5H9FmBb875xKzurCOc/Tn7Pj7rW3Bd7jHrBJ/WjjAhhJDTwUyWOtKG +H1+CI8v2PlqpidrwhJA3pd/vm0peYRjC930wxp7q3xUKhTe9qYSQt9yz+txH1/lmjE2UKs/aWeN9 +/Bfp848HvU9jn58C34QQQgh5KzDGAA5wORo4yzK/s4aZEAJJksCyrIkM75PWBFRSQarJDBEaICPf +1nA4xHA4xGAwMKWjcrkcCoWCGRyhUnjk+3C0Y5qVKDsp+D3eGRbCgmUJXdrcEpNrevNRqbPT2Pkl +hBByOjHOnmrDm8FZPlqHMmvLH9eGzyrrZG14ABPteGrDE0JeJSGE6dtl6+ACQC6XQz6fR7FYRKVS +ocA3IeSNmCh5nlZjy/r840udZf358bZWFugeX8/7aKb3ce9zmlDgmxBCCCFnnimVeCT4nQ2YSSl1 +UCbhkIk0ge8kkZByFPCWiYRUcqLE4vggGWWNkG9LSol+v49+vz+xVlIW+M4yAyjjm7xKLzLze/zP +JvDNOLgY7wxbEOLpkmgTAe+x8uaneeY3IYSQ02Pic+KYNrwpx0lteELIKcMYQ7/fx2AwQK/XMxOY +Pc8z/btyqYxCkQLfhJDX40Wqu43/ebyfnvX5R6XPBYRIA+FcQFijgPd4sFwIcSb6+xT4JoQQQshb +YaJBJ/RagUqqp7IXkySBpSyTEZKVRBzP9AYAKbOBM8r4Jt9dFEXI5/Mm6D2+ZtJ48JvWgCPfh6Md +VJYGrDk/EgA/0rkd//+jAe+jmd6nuRNMCCHkdDGfHS/Qhs/a7dSGJ4S8ad1ud6J/xxgzVb3y+TyK +pSKKxeKb3kxCyF+gF+nzmwzusT7+ceXQx/v8Z6W/T4FvQgghhLxVjs3+ZqN/45ybwTEpJYQQT60D +eFx5RIAGzMi353keHMeBbdtmnaTxNZMdx4HneZTxTb43x5U7BzARwD66DthTf3dkPe+TXpsQQgh5 +HmrDE0LOkiiKTMA7698BMOvjZhW+qH9HCHlTjit3Dhzf5x8Pch/t85/FSe4U+CaEEELIW2e8MaaY +HuhinE2s2f3UQNmR71LKp16XBs3ItzWeCZAFvbNBEdu2TdCbBkbI9+W4zirn/Kl/H+/0Zv9/dLb3 +Sa9HCCGEvAxqwxNCzoogCCaC3lngO/s7Hfym/h0h5M15Vp//aBD7aL//rGV4H0WBb0IIIYS81Y5m +jwjoDMXxjJCjg2FKKQghnnotGjQj35Zl2WZN5KxkVPY9Wy85ywYn5PtwUsf16ZJo7MTZ4c96HUII +IeS7oDY8IeQ0Gw94W5Y1Kgec9vN0386i/h0h5I35Nn3+rL8//nNnsc9PgW9CCCGEvPUmBs6yvxP6 +7wSEGUAj5HWxLGEC3ONrJWUDIyb4bdHACDldjga6J/7tDHaACSGEnB3HfvZQG54QcgocndScZVGO +/70lLOrfEUJOvZP6/Ge5v0+Bb0IIIYT8RTgpaKOUAhNntzFHzobxQPf4uknjf2dZFrjg3/3NCHnF +znKHlxBCyNl33EAsteEJIW9SluV93Jq4ZoKzJah/Rwg5E962Pj89eQkhhBDyF+1ta9yRs+Esl4wi +f1noGiWEEHIa0ecTIeRNO7o27tE/E0LIWfA2PrMo45sQQgghf/HexkYeOV2OGwCZKL1PgXBCCCGE +kJdC7SZCyJtyXP9t/M9Z6XN6ThFCyPePMr4JIYQQQgj5ntDAByGEEEIIIYQQQgghrwcFvgkhhBBC +CCGEEEIIIYQQQgghhJxpFPgmhBBCCCGEEEIIIYQQQgghhBByplHgmxBCCCGEEEIIIYQQQgghhBBC +yJlGgW9CCCGEEEIIIYQQQgghhBBCCCFnGgW+CSGEEEIIIYQQQgghhBBCCCGEnGkU+CaEEEIIIYQQ +QgghhBBCCCGEEHKmUeCbEEIIIYQQQgghhBBCCCGEEELImUaBb0IIIYQQQgghhBBCCCGEEEIIIWca +Bb4JIYQQQgghhBBCCCGEEEIIIYScaRT4JoQQQgghhBBCCCGEEEIIIYQQcqZR4JsQQgghhBBCCCGE +EEIIIYQQQsiZRoFvQgghhBBCCCGEEEIIIYQQQgghZxoFvgkhhBBCCCGEEEIIIYQQQgghhJxpFPgm +hBBCCCGEEEIIIYQQQgghhBByplHgmxBCCCGEEEIIIYQQQgghhBBCyJlGgW9CCCGEEEIIIYQQQggh +hBBCCCFnGgW+CSGEEEIIIYQQQgghhBBCCCGEnGkU+CaEEEIIIYQQQgghhBBCCCGEEHKmUeCbEEII +IYQQQgghhBBCCCGEEELImWa96Q0ghJDjRFEE3/fh+z62t7exu7sLzjkuXLiAixcvYn9/H3t7ewiC +AK7rwnVdVKtVVCtVBGGAjY0NPHnyBOfOncPq6iqmpqZgWzaEJb6X7U/iBFEcodVqYWNjA5ubmzh3 +7hwuXLiImZlpCCFg2/abPsyvdH/jJEa73cbG4w1sbG5gcXERFy5cwOzsLCzLguM4b3ozySsUBAF8 +30e73cbOzg52dnYwNzeHCxcuoFQsYf9gH/v7++Ccw3VdFItFzMzMYG527nu7D8nbJQxD7O7uYmdn +B0op5HI55PN5TE1N6Wf8W/RMJYQQQgghhBBCCCGEvDwKfBNCTqUo0kHjer2O3/zmN/j0009hWRb+ +7u/+DlNTU/jmm2/w2WefodVqoVKpoFKp4MqVK7h27RqazSb+7d/+Db/+9a/x13/91/jHf/xH3Hj3 +BnL5HHJW7nvZ/jAK4fs+Hj9+jP/zf/4Pfv3rX+MXv/gF/uEf/gE5z0Mun3urgjRxEsP3fWxtbeHf +/++/41e/+hU+/vhj/N3f/R0cx0GhUIBt22CMvelNJa/IcDhEvVbH+uN1/O53v8Onn36Kmzdv4u// +/u9x4cIF/PGPf8Tnn38O27ZRqVSwuLiI999/H5VK5Xu7D8nbJQgC3LlzB3/4wx+QJAmmp6cxPz+P +d955B/l8/q16phJCCCGEEEIIIYQQQl4eBb4JOcWUUuj1euj3+wjD8NifYYyBMQYhBIQQcBzHZEBn +/3YWxXGMfr+PRqOBu3fv4re//S1s28a5c+fwN3/zN3jy5Am+/PJLHB4e6izSuTlYloWZmRkcHBzg +888/x7/927+hWCzi/fffx+rqJdjO9xcUiWMdCN7b28MXX3yBX/3qV8jlcrh58yYuX74Cy7aQy52N +4J9SClEUod/vo9frwfM85HI5uK4LIQQ450iSBGEY4vDwEF9++SX+3//7f5BS4urVq7hy5Qocm7K9 +3zZhGKLT7WBnZwdffPEFfvOb32AwGOD69esol8t4+PAh/vSnP8G2bczOzqLT6WB+fh5xHL/pTSdn +VBRGePLkCT7//HPEcYylpSV0u11MT0/j8uXLb3rzCCGEEEIIIYQQQgghbxgFvgk5pZRSiOMYn3/+ +OT799FMcHBw89TPjAe9yuYxyuYzFxUVcvnwZV65cgWVZsCwLnPM3vTsvjXMO27LheZ7ZB8YYLMsy +gX3HcSa+bNs+k/t6mimlIBOJ/f19fPrpp/jTn/6Eq1evppMJVlEoFFAoFN70ZpI3ICthnk2AAGBK ++HueZ/7Ntm1zv1qWdWYn4xBCCCGEEEIIIYQQQgg53SjwTcgppJQyge8vvvgC//RP/4T19fWJn8my +ubPg9tzcHObn5/Huu++CMYaV5RV4OQ+CC+AMxoI557CdUeA72+ejQbXxDHcKqr16MpGIkxgHBwf4 +9a9/jX/6p3/CT3/6UzPZgjFGge+/UFmQ23EcCCEmnklHJ6dkf84C5IQQQgghhBBCCCGEEELIq0aB +b0JOMaWUKffd7XZRLpdRqVROzG6u1Wp4+PAhisUihBA4f/48Ll68iEql8qZ35aVxzk2Q27ZtU1J7 +PKh29MuyLB3oJ69cGIZot9uo1WpotVoYDAaIoghSSiil3vTmkTcgW1ohm5ySVaBwHAeu456Y8U1V +GQghhBBCCCGEEEIIIYS8DhT4JuSMKBQKWF1dxfXr11GpVFCpVCCEgO/78H0fGxsb2N7exs7ODgCg +Xq/jZz/7Gaamps5k4DvLHM3lcibIn5V1z7JMj2Z827YNxinjm5DvA+f8qWUGhBCwLAuO6zyV8Z1N +TiGEEEIIIYQQQgghhBBCXgcagSbkjPA8D+fOncPHH3+MhYUFLC4uwrIsdLtddDodMMZQq9XQaDSw +vr6O3d1dzM7O4gc/+IFZpzmRCYbDIQaDgSlRXSgUTInicVlAPY5jcMbBhc62dhwHSZKg3+9jMBjA +8zx4ngcAiKMYYRQiSSSUkmCMw7Ytk/HpOi6E9WIZ2Ywxk006nilqWZYJfI8H1LLM8BcpdR6GIeI4 +RhiG5iufyyOXz00E5gaDAYbDIQAgn8+jWCy+8LEKggBxHE/8XJIkCMMQg0EfcRyh3+9DSgWlJACY +zPZsf8ZLt2fnMIojBEGAIAggpUQcx1BKpdvF4Ti2CURmX5mshL6SCkGoX0MfiwRSJuY1hOBmgkH2 +XrVaDcPhEFJK+L6PZrOJvb09xHFstiWKIpMJ/l1l+xsn+vV933/u/mbXZ7bNYRhCSokkSeB5HvL5 +PBzHMcdUSgklFcIoxHAwxGA4MMfedV0ILiAs8dS2BEGAJEmeuy3j1+NJxz7bp+w1OGcT1/jRc3ia +WJYF13EnqjJk183RySnmmKYl0V/2OkhkAt/3zbFPEjlxzVqWgG3ZsGzLvK9SKv3ZZOK8JUly7HnL +SrFnxztbbqLf76Pf75tqE0IIhGGYVjxQ5lnnus5TWe3ZNmfXYfZMHJ8wYNu22c+hr5/PYRiaZ5uU +Mn2/ON0uOXF9ZNUwoiiC7/sT22VZFhzbgWVb5lyMX//ZMy47ptmzKDs+2WuPr+We7d9J2+x5ntnm +OI7NtmS/az4PXHci+z87RlEUjZ2v0XkWQqDVaqHT6SAMQ7OdhBBCCCGEEEIIIYQQkqHANyFnBOcc +nuehUqlgenoac3Nz8DwP09PTGA6HaLfbCIIAa2trODw8xP7+PprNpgmwDYdD9Ho9fPXVV/j666/h +OA4++eQTfPLJJxPBDEAHEx4/fox79+6h0+mYoNXS4hIWlxbR6XTw2Wef4euvv8bly5dx9epVWJaF +7e1t7O7umvd0HAczMzOYmZnBpUuXcPnSZUxNT71Q4EtwMbGet+M4JvANTJZZzn7mRcsoHx7WsLe3 +a7Z3d3cXN27cwM2bN7GwsGDKd3/11Vf46quvwBjDJ598gp/85CdmTfXx99nc3MS9e/fQarVMQGd6 +ehozMzMT7zscDnFwcIDbt2+jXq+jXq+bAI9lWahWq5iamjLHdG52DowzcK4D6cPhELVaDWtra3j8 ++DG63S56vR7iODYBtMXFRSwtLWFpaclMkDCBrjRI5fs+Hj9+bCZI9Ho99Ho9s2/5fB4zMzOYnp5G +q9Uy77m+vg4pJXZ2dvDrX/8ajx49Qi6XQz6fx/T0NGZnZ1Gv19FsNpEkyXe63rNgd71Wx9q63t92 +u41+v2+uLcuysLCwgOXlZSwtLWF+fgFLS4vY3d3F2toadnZ2zMSQ69ev44c//CEunL8ALjgYYwiC +AP1+H/v7+/j6669x69YtXLx4ETdu3MDFixdRLpdRKpXMtjQaDXPs2+02er0ewjA0xz7blsXFRSws +LGJxYcFM9JCJRBAG8Ic+Hm/oY7+zs2OOfVbaP5fLmWvn/PnzuHDhAubm5r7HJ82L45zDdkb3aBbw +55xPrPU9HgB/2YzvMAwRBAGazSbW19extraGVquFXq9n7hvbts1zcWFhARcuXMDq6iriOEav10O9 +Xsfjx4+xsbGBdrttJmdkgeOFhQWcO3cuvYbmMTs7a67Ber2OP//5z/jyyy9RqVRw6dIlTE1NYXt7 +G9vb2yaY7jgOzp07h3PnzqFaraJQKEBKiYcPH2J9fd1ct5xzLC4uYnFxESsrKzh//jxmZ2fNc+DO +nTv44osvsLOzg9XVVVy+fBmdTge7u7s4PDxEkiSQUmJ2dtZsczaBaW9vD48ePcLu7q6Z0FOtVrG4 +uIjl5WWsrKzgwoULcBwHSin0ej3cunULd+7cQavVmgjSSynN87dYLOLixYu4ePEiZmZmUamUUSgU +zDbfv3cfn3/xOZ48eWKeXcPhEDs7Ozg8PDSfB4VCATMzM5ibm8OlS5dw9epV5HI5MMZMEN73fWxt +beHhw4d48uSJmaglhEAul4OUErdv30atVkM+n38lk2wIIYQQQgghhBBCCCFvDwp8E3JGZAP/44Hv +UqlkshJ93wegg9a1Wg31eh2tVssEHTqdDur1Ov7whz/gX/7lX5DP52HbNj744AOT3Yo0lquUwsbG +Bn7729/i4OAAhUIBxWIRN27cgLAEtre38W//9m/413/9V3z88cf4q7/6K7iui2+++QZ3797FYDDA +YDBALpcz64wHQYDZ2VlMTU+ZbMJnYWnmK2NsIvAthDDH49tmfNfrNdy9exfffPMNbt++jfv37+Nv +//ZvTXZ5lhX66aef4l/+5V/M+374wYfwct7EsQJ04Pv3v/89tre3zbF65513nsrUHQwGODw8RBzH +uH37Nu7du2cy5x3HwdLSElZWVvDzn/8cU1NTmJqaggUL4DoI1+l08OTJE/zxj3/En/70JxweHqLR +aCAIAhN8vHbtGt577z3cvHkTQggsLCyYY5Jl/He7Xdy9exe///3vcf/+fdRqNTSbTZMxXa1Wcf78 +eayurmJnZwdbW1vY29tDo9FAkiTY29tDs9nEF198AcZ0YD4rw885R7PZ/M6ZmNk1u7W9hT/96U/4 +4x//iIODAzQaDfi+b/b3ypUrZn8555ifn8Pe3h4+//xz3Lp1C/v7+zg8PMTf/M3fYH5+HisrK2Cc +mcB3p9PB2toa/u///b/493//d3z44YdIkgT5fB5CCBSLxdG2bG3hz3/+Mz799FPs7++bbcmuwaPb +Mjc3CwFhjv1gMEC328WDBw/wu9/9zkyAaDab5noul8vm2P/gBz9AuVw+tYFvwcVENnc2AeBo4Dv7 +ygLfL5PxHUW6MsLe3h4+++wzfPrpp9je3kaj0TD3jeu6WFxcxIULF3Dt2jUwxrC8vIwwCNFoNPD4 +8WP84Q9/wJ/+9CccHByYSULZpJkrV67gww8/xAcffADGGKam9DNqOBxib28P//mf/4n/8T/+BxYW +FvCTn/wEKysr+Oabb3Dr1i30+33EcQzP8/D+++/jo48+wvLyMqanp5EkCX7/+9/j008/RaPRwHA4 +hGVZuHr1Km7cuIEPP/wQxWIR01PTCIIAvV4fd+7cwf/8n/8T9+7dw0cffYSf/exnODw8xDfffION +jQ1EUYQ4jrG6uooPP/wQN2/eNBMl7ty5g9/+9re4c+cOfN/HcDjE4uIi3nnnHdy8eRMAsLCwoAPf +UmEwGODWrVv413/9VzMJYzAYmOef67ooFouYnZ3Fxx9/bCocOI6NfD6PMAzR6/Vx+85t/PM//zNu +3bqFTz75BMPhEM1mE7du3cL6+rrJKs+eK9euXYNSCsvLy8jlcqYSgu/7aLfaePDgAX7zm9/gyy+/ +RLvdRqfTgW3bKJVK8DzPTPhhjD1VVYMQQgghhBBCCCGEEPKXjQLfhJwRWYAxy8rNAj6ALhGbz+eR +z+dN+VhdYnZUtlZKiTiK0e120Ww24fu+LrWdyGODlMPhEJ1OB61WKy2dKzEcDhGGIXzfR6vVwu7u +LjY2Nsya4/1+H8Vi0WRhAkCz2cRgMMDMzAzOnTsHz/NQKBRQKpWeu7+C67LIq6ur+MlPfgLGGC5c +uAAAmJ2dxdWrV7GwsIBKpYKpqSlUq1VzTI6K4ygtk6sDj1999RUeP34MKSWWl5fNRALHcTAcDhFF +EXq9HhqNBjjn6PV6SGRijum4LIDaarXMcc8CSOM6nQ7W19dN1mmxWDQBTyklut0uHj16hJmZGczP +z8OyLExPT2N6ahrb29u4e/cuHj58iL29PSilTCBISmmCjWEY4uHDhyYTeW5uDrlcDo7joNPp4O7d +u7h//z4ePHhgMkMLhQLy+bwJ8DuOgyiKsLu7iyRJMDc3B8uyEEUR6vU6CoUCpqenUS6XzfV47tw5 +LC8vIwgCuK77UsHNcVlJ8J2dHdy5cwcPHz7Ezs4OpJQoFovwPA9Jkpj9jeMYjx49QhiGsCwLs7Nz +SJIE1WoV09PT2NnZQa1Ww5MnT/Do0SPMzs6aQOHh4SHu37+PW7dumfOclUQvFAqwLAtJkmB3d9ds +y/b29onbkiQJ1tbWTCby7OysuaZ6vR7u3buHe/fumdcJwxD5fN5kvWb3TTa54Pz582ZCy2nEBYcF +C+VyGe+++y5+8Ytf4L333sPc3Bxc18Xy8jJu3rwJ27ZRrVYxNzeHaqX63KoMWQltqSS2t7dx7949 +3L9/H48ePUKz2QTnHNPT05iamjLl1R3HQbfbxd7ennlm7e3v4auvvsKdO3ewublprs35+fmJyTe+ +7+Phw4em5HelUoHneSZrut1uo9FoII5j3L9/31TPKJfL4JxjMBhAKYX9/X18/vnn2NzcxPT0NCzL +Qr1eh+M4KBQK4JwjSRLU63V8/fXXpkLDwsICfN9Hkuiy6o1GAwcHB3j8+DHK5bIphV+tVk0GdK/X +w6NHj9Dtds2xaDQaCMPQPFey50EWMM8y1rPPEVN+XylTMaNcLpu/y4RhiCdPngAA+v0+HMfB9NR0 +mhmuJ3Q0Gg0cHh5ibW0N5XLZ/F6pVDKfV4wxHBwcII5jzMzM4MKFC0iSBIVCAUopPHr0CHfv3sW9 +e/ews7ODMAwnJlVk90m27EH2RQghhBBCCCGEEEIIIRkKfBPylsiC2wBM2WTLssward82EPk8zWYT +9+/fNyWCP/zwQ0RRZIKkW1tbePz4MRYXF3H+/HkU8gUsr6y8UOCbCw6mGN67+R7y+TwYY7h8+TIA +YGlpyaxBPb7O+En7qTMqe+j3dVblV199hX6/j6tXr+L69eu6vPWFiyiVSpCJfC0Bx263azIgq9Uq +bt68ada1bbVa2Nraws7ODtbW1kxQ78qVKyiXy3j8+DF+85vfYGNjA7lcDvPz8yaAa9s2hsMhhsMh +NjY28OTJE9TrdZM9PDU1hVKphP39ffzpT3/Cr371K0RRBKUUpqenTSnkbL3pLJDVarVw7tw5XLp0 +yWS2bm5uYmFhAT/84Q9x9epVFItFlEolk+meBay/zfWWBdySJDEVB9bW1uB5Hubm5jA1NYWZmRm4 +rmsCgE+ePMHm5iZqtRpKpRIunL8Ax3Zw/fp1WJaFvb093L17F41GAw8ePECpVML169dRrVaxv7+P +L774Anfu3EGSJLh48SIuX75syotnay9vbm7it7/9LR49egTP83TlgqkpTE9Pw/M8c+yzbTk4OEC5 +XMa5c+cwNzuHUrmERqOBzz77DL/61a9MoLRSqWBpaQnLy8sA9Brww+EQjUYDzWbTHPPTjAuOarWK +H//4x5ibmzNZ9Z7n4dKlSyiVSmZCQS6XM5MlnidOYkRRhMePH+M//uM/8M0335jrY2VlBcvLy6hW +q2at9Sw4nZXDD4IAm5ub+MMf/oBvvvkGjuOgVCrh4sWLmJ+fR7FYRK/XQ7fbxf7+PjY3N7G1tQXP +87CysoKZmRlYYnI7+/0+tra2kCQJZmZm8MEHH5jz1Wq10O/38fXXX8PzPFMmv1wu48qVK4jj2FRb +2N/fx4MHD5DP53H16lWsrq6a6z+TPT/v3buHubk5LC0t4erVq+ba6Ha72N7eNsHx7P1KpRLm5ubM +NVmr1XB4eIhms4lz587hww8/NOdCCIFyuWyeJeVy2QTNLctCq9UyVRZqtZop7X7+/Hlcv3b9qXMW +hqHJTs8+D1ZXV83nwf7+Pp48eYL19XUsLS3h4sWLAICFhUUIwXH79m38+7//O2q1GqSUqFarmJ+f +x8LCgrkXB4MB7t69i3a7/aYvfUIIIYQQQgghhBBCyClEgW9CzqgsCCQTiTjRQZV+vw/f9yGEMJmk +L7ru9bfdhqwMdLVaNcGgbNvW1tZMGe2DgwNsbm7qLNhyCcDyc1+fMV2S+tLlS7hwUWd6Z6XO5+bm +MDM9WkObcZYGsnoTr5EFc7NA0c7ODtbX13FwcADXdbGysoK//du/xeLiIubm5iCEwGAweC3HKwxD +s77w7OysKUfOGDOlhtfX13F4eIiHDx/CdV3kcjksLS3hyZMnuHXrFg4ODvDOO+/gwoULuHLlCq5c +uYJSqTQRLL1z5w7q9To2NzexubkJ3/dN4Onu3bv47LPPMDs7i5WVFaysrOD999/HBx98YILeBwcH +uH//Pvr9PpaXl0354tu3b4MxhunpaXzwwQf42c9+htnZWczOziKKYkRRCMaYCXZ+m+spWzd4a2sL +t2/fxvb2Nq5fv45z587hypUruHr1KiqVChqNhik1fvv2bQwGA2xubmJjcwPnzp3D+fPn4bkevv76 +a+RyOQyHQ6yvr5uy+AsLC9jZ2cG9e/ewsbGBhYUFvPvuu7h69SqWlpZQqVQQBAEGgwG2trZM1vD1 +69dx/fp1s5ZxtVpFs9k0VRTu3LmDTqdjjn0cxwijEAcHB7h37x4+++wzlEqliWP/4YcfmooKh4eH +ePDgAQaDwWu9d1+FbHJDsVjEezffw7vvvmsy1xljOkC9pO9zxpn5nedNisgmnvR6fWxsbOCrr77C +3bt3sbS0ZK6DDz74AOfPnzcB3rW1NTx48AC+7yOOY1Oa/s6dO3jw4IGZ0HDt2jW88847mJ+bx97+ +Hg4PD/GnP/0Jjx49QqPRwMWLF7G5uQkpJaampia2K0kSsw71zMwMPvzwQ4RhaAK633zzDba2tmBZ +FiqVCubn5zE3N4fr16+bySlZEHp/fx9bW1vY399Hq9Uy5eDH7wVdSryH5eVls8Z2Vrr/zp075vfL +5TKmpqZw+fJlXLlyBaurqxgOhwiCAF999RU2NjbQaDSwvb2NWq2GqeoUbMuGJSxMTU3h/PnzJkN/ +fn4etm3DdV08efLEVIe4f/8+1tbWUCwW9XINSQyZyKeOTxAEaLVamJ2dxeLioilrLqXE119/bcrU +7+3tYWNjA57nQQgB27bx6NEjfPnll5BSYmVlBefOnTPny3EcDAYDs4TH1tbWC11LhBBCCCGEEEII +IYSQvywU+CbkDIuiyJS93djYwJ07d7C7uwvP83D58mUsLi6aTOjXgTGGxcVFs9bs5cuXcfmSzsiW +SkIIYbJgbds2pcNfNrDMGDMB7/FAx3gw7SS+76PZbGJjY8OsywsAV69exfLyMm7cuIGlpSWUS2XY +tv1aS+cuLCzgo48+ws2bN3Hx4kVcuHDBBDYrlQra7TZ6vR4sy0Kz2cTDhw+Rz+dRLBZRr9cBAOVy +GZcuXcInn3xiAmtZgHxqagpra2t4/Pgx6vU6giDAw4cP0Wq1UK/XzfrGSinMzc3ho48+wocffmgy +aJM4QRRHmJmeMYG0c+fOmSBxVj0gC1RlQeRcLgfOAyglX3oN53FBEGB/fx/7+/uo1WqmnPvq6io+ +/vhjLC8vY3Z21mT4V6tVbGxs4Pz58zg8PEQYhnj06BFs28bMzAyqU1VcvXoVH330EbrdLg4ODjAc +Ds018+DBA5Nduri4iB/96Ee4du0aqtUqgiDAwcEBDvYPzM9kGcPZOs6zs7PI5XJmWzY3N7G+vo69 +vT1EUYS1tTV0u13MzMyg3W6j1WpBSomZmRn84Ac/wA9+8ANTIh7QEyOyUuyrq6u4cOECZmdP5/re +4xhjYJxBQJh7Mvt78LE/Ay90bURxhNphDds729jd3cVwOITjOLh48SJ+/vOfm+tyenoaYRgijmOU +SiUsLS2ZEuQbGxvY29tDGIbI5XK4ePEifvKTn+DixYtYXFxEsVhEvpDH3NycycIWQiAMQ9y9exdR +FD016WBqagoffPABfvCDH+DSpUu4evUqoigy2ef7+/u4f/8+isUirl69ips3b+LatWsm8B3HMXZ3 +d/H48WN4ngelFHq9Hur1OiqVCmzLNu/lui4uXLhgrskrV66Y7OdWq4UgCLCxsQHOORYXF3Hz5k0z +KWNlZQVxHJv10R8+fIh2u40wDFGv13XGe0lXasieQbZto1wum7LjSimzPEUYhma5gWyJjG63a5bA +yORyOZw/fx4//OEPzeSQleUVAPrzIKtIMRgMIIRAq9XCzs4OkiQBYwz1eh1xHGN6ehrvvfcePv74 +Y1MKPltqoVqtYn19HZVKZWJJDUIIIYQQQgghhBBCCAEo8E3ImZVlBHa7XTQaDWxsbODevXtoNpuY +nZ3F+fPnsbi4iFwu99q2gTGG5eVl/PKXv8Qnn3yCmZkZzM3NjUquM45Hjx6Z7OUs8J0FH1/mfY7+ ++bhsP70O+uTfZRmIvu/jq6++wh//+Ee8//77+OSTT/Dhhx/i2rVrWFpagiUscMFf65rKi4uL+MUv +foGf/vSnmJ6eRrVaNfswNTWFTqeDwWCAnZ0dE6TN1kNvNBpQSqFareLy5cv46U9+Ci/nmUBztVqF +lBKXLl3C+vo6GGMIgsCsi5yVge50OpBSYn5+Hp988gl+/vOfm+A1oK+rOI5x7fo1vW62sGDZFg4P +D00gMAuUZb+XTa5IksQEx7+NIAhweKCz3ev1uikHfunSJfzkJz9BqVQyr5/tbxZsVkohiiKsr69j +ZmbGlKO+evUqfN/H559/jq+++gqbm5vpdaKvzVqtZoKHP/3pT8263L7v42D/AA8ePkC9XjeB72y9 ++XK5fOy2rK6uIkkShGGIx48fmxLcw+HQBL5nZ2fx8ccf42/+5m+Qy+Umjr2UEleuXEEURWZiwVnA +OYdi6ql79dtMggjDEIe1Qzx69Aj7+/vwfR+2bePy5cv4L//lv2B5eRme58FxHBOkPX/+PKIwQqfb +wdraGtbX17G/v28C36urq/jFL36BxcVFOI4DIQTm5vR68N1u16x3HwQBHjx4AMuyMDc3h0KhYLZr +ZmYGP/rRj/CP//iPmJmZwczMjAkGV6tVfPXVV2CMoVwu48aNG/irv/ornal94SJsRwdonzx5gs8+ ++8wEvvv9Pur1Omzbnlj+wbZtXLp0CX//93+Pq1evYmZmBsViEb7vw/d97O/v49NPP4UQAsvLy/jp +T3+Kd955B6urq2YNc6UU9vb28Ic//MEE9ZvNJjqdDubm5uB5HlZXV7G8vDwxucgf+hgMByYgnSQJ +vv76a1OKv9/vo9/vT5RmH9/mf/zHf8T169cxMzOLqamq+TzodDq4f/8+6vU6OOdotVrgnKPf7wOA +CXxXq1V8+OGH+K//9b+aTHjGGJRSmJqawq1bt0yZewp8E0IIIYQQQgghhBBCxlHgm5AzYjAYYH19 +Hf/5n/9pgi6cc7NO7draGpIkwfT0NK5cuYJr167h8uXLyOfzr3W7PM/D9PQ0ZmdnUalUJt6vWCqi +UCggn89DCIEoikwp4pf1bbOIG40Gbt26Bdd1UavVAOjs6tXVVZNFOV5i+HVyXRfVahWzs3MopRmX +mXK5jLm5OSwuLpoS3oPBALVaDfv7+2g0Guj1emaN21/9v1+ZQNW4r776Ctvb22i1WoiiyBzrrJS2 +UgrFYjHdDp1dfFJJ7fFS+uPrxGeB4yywKYT4TgHvTBzHaHfaZl3hbrcLKSXu3buH//iP/zg2yHXr +1i1sbW2h3W6btYT7/T6SJEEul8PKygqUUqjVaiYDe3d3F5999hniOMbU1JSZKDI/pzNhLctCv9dH +p9vBwcEB6vU6er0eoijCgwcP8Otf//rYa+b27dvY2tqaOPZZMDtbUz2fz5tlAWZmZiC4gLAmz2M+ +n4dS6syVcX5V25sFow8ODkzJ96yc9/z8PKrV6rHXm1IKjDNIKVGv183vlkolVCoVs4710e2sVqpm +3fZ+v2/W7A6CYCLwbds2KpWKedYV8vrfsmoLnueBcw7LslAoFDA9Pa1/rlgw10upVEIul4PjOOCc +m/LgSZJMBJI558jn8+YeLZfL5vcKhQKKxaJZK911XXN8SqUSisWieZ1CoQDP88zPRlGEMAwhpUSS +JBNro/d6PfOM8X0fQRDA9300Gg3U63VEUWQmdQRBAMEFuBidA8YYCoUCZmdnMTMzi0qlPPF5kG13 +Pp83E3M6nQ6SJDHnPZ/Pm3M1OzMLxtnEeS7kJ/fnrN0jhBBCCCGEEEIIIYSQ14sC34ScEb1eD/fu +3cPh4SFc14XjOGCMmQBboVBAoVDAhQsX8OGHH+Ljjz82WYJHM/NepSwIOh4YHf+3LDM4C/JkwZPv +y+HhIb788kvkcjkkSYKpqSksLi7iwoULWFlZmQhsvW7PCtIIIVAoFDAzMwPXdc0kgU6nY9aQbrVa +GA6H+M///E+T1X1UrVYzGcozMzMQQpis8CiKYFkWqtUqyuUyPNd7Zmny7zuolK2hPL6/vV4Pv/vd +70xZ56Pq9brJFM32N7vGXNfF4uIiCoUCNjY2cPfuXbRaLTSbTRweHpr1om/cuIHV1dWJDPo40eud +t1ots356t9s123LcpINGo4FarWbWcB8/9lLq0v/VahWVSgW5XO5UHfvTJLsOWq2Wydi2bdsEe4UQ +xx4fxhiUVBgOh+h0Oojj2GSG5/N5CH7877mei0qlgmq1apaO6Pf7iKLo2PcYf9ZxwWFBl9weX45h +PNv96Htm10X281kQ+rj3Ovo62TUzfhyelVmfBeJt2wbnHFJKSCmhlEIQBNjc2MSdu3r9+qzSRBAE +CILAbGsURdja2kIYhmYiR7a944HvbBv1RBj+1P0qhIDjOHBdF5xzRFGE4XA4EfTPJinkcrmnXpsQ +QgghhBBCCCGEEEKehwLfhJwRg8EASil0Oh0T6BjPvnvnnXfwzjvvYGlpCdeuXcOHH3wIxvXPnVRa +/FUExLNA0HHBqKNBlyRJkCTJa11H+7jjtre3Z9bKLhZ1Fnq5VDals08Dy7LgeR5KpRJs2zaZoIPB +AN1uF/1+H77vo9frIUkStFqtY18nO8au60IIYUplD4dDxHEMIQRKpRIKhQIc1/nOWdqvUlY2enx/ ++/0+dnd30Wq1jg3uZfubla/OAp3ZdTc1NYWpqSmzJvfjx49xcHCA3d1dTE9PY2FhAe+++y4WFhbM +dXp0WwaDgTkXe3t7aLfbz9yW7HWyEs2+7yNJEnDOzfWXBf/I07KgbK/XQxzHcBwHxWLRBMCfddyk +kgjDEP1+H3Ecw3Vdc7wZPz44bNs28vk8CoUCOOcm0/loMPqpQHT6fBWWeCogf1JAOgsMjwe+s7Lk +xzk6qSj7/ywAnn0dN/EImAyyc87NkgDD4RDNZhNr62v48ssvsb6+jt3dXdTrdfOMtm3bXKfZ8VRK +IUkSxHF87Hl4ViCeM73dWbZ29ozLqiFkGePFYtFM7CKEEEIIIYQQQgghhJCXQYFvQs6IUqmEpaUl +LC8vI5fLwXVddDodbG5uYnd31wSWwzDUQYk0W+5FggevOxD9JgMY2XHL1hg/PDzEkydPcP/BfRRL +RczOzmJubu6Nbd+4LJsSwEQpZymlKadcrVaxurqKS5cuHRt4yrIxHccxQd9arWZKdn/bdZdfdj++ +DSmVCQJ6nodqtYpisYjV1VVcvnz52EkK4/tbrVYxNTWFa9euoVKpTvzceGWC8d8dL/180vnItqVQ +KODixYu4fPmyCd4dty3jAfdGo/G9Hvu3SXYdvexxO+n6e5nX+C7n6TSe4/Ft6na72NjYgFIKt27d +wp07dyClxLlz53Djxg0zCSDLYu/3+/j0008xGAy+2zbwyfXfx/98NIB/Go8hIYQQQgghhBBCCCHk +9KPANyFnRD6fx+XLl/HJJ5+YAN/W1hb+4z/+A41Gw5SyzoJ4Lxo4kOr7y75+EyqVCi5dugTLsnDv +3j3U63VsbW1hbW0N09PTEEJgdnb2ucdrPCtTKgmBV5sprsaCvkez6JVScF0XU1NTKJfL+MUvfoFf +/vKXxwZfs23MMo4dx8GXX36J4XCIer0O4OUDid+nLNic7W+hUMDPf/5z/PKXvzx2Xe3j9rdcLqNc +Lk/83HEl+bPs1/G10I97/WxbPM/Dz372M/zN3/yNyaR/3rZ88803CMPwTBz70+JVBK5f5/IOz/Ki +2/gqroGTSqk/T6/XQ7fbRafTwZ07d/Dw4UMsLi7i/fffx0cffYSVlRWsrKzoyhNxgp3dHXQ6Hdy/ +f/+1HKPnZckTQgghhBBCCCGEEELIi6LANyFnhG3bKJfLWFlZwdzcHBYWFlCpVNBut82axt1uF5ub +m1hbW8P9+/fNurWmvK5tmYzXrGRtFEWwbfvYYO54BvJZVS6Xsbq6ikKhgCAIMBwOoZTCgwcPTBng +xcVFuK4L27YBAEKMyrNngWcpdQll13WfCqpla+Y+q2Tx82RrSjebTURRZNYlzsqzd7td8z6lUgkX +Llx45hrRnHHwdJ3dqakplEoluK6LVqtlyndHUWQmSTwr2HRcQCrblldJCB0wLhaLaDQa5j0KhQLO +nTuHXC73Qvtr2zYc20EURej1euj1etjb2zPHtlQq6TW9PQ/7+/u4e/cuyuWyzqIfex3HcUzma7YO +cbYt+Xz+hbZld3cXpVIJnueh3W6bMu7jE1Qo0DcpO4eFQgFCCLPetO/7ptw2gBPLaWfnrdFomN8N +ggAyOf5ZlpVG7/V6ZtKF53nHTix5GwyHQ/OsCYIAlmWhWq3i3LlzePfddzE/P4/5+XmzDncUR8jn +86+lNL8QwiwJ0Ol0MBwOzbIMr4qU0lw3lrDABad7jhBCCCGEEEIIIYSQt9TbOapLyFuIcw7P81Au +lzEzM4OFhQWUSiVEUYRyuYyvv/4ad+7cQRiGmJ2dxdTUFC5fvjwRxHFd1wRLlVKI4xhBEMC27aeC +mFmw93UEOL9PWZB4YWEBAOA4DtrtNtbW1tDv9zE9PY133nnHrHut188VE8cqW9M2DENEUfRUtu+r +OFbZxIVarWYC7K7rmskLBwcHGAwGEEIgiiK4jgvHdY5dW31ctm5uuVxGLpdDFEVotVrodDoIggBK +KoAfH0RUSul1jNVk4Hs8yP8qrw0hBPL5PKrVKnZ2dswkhWyt5lwu90L7yzkH4wzBIMDBwQG2traw +sbGBg4MD+L6P6elpzMzMIAgCbG1twfd9LC4u4oc//CEsy4IDvfb5+LZkQddszekX3ZZ8Po9yuYx8 +Po84jk3wO1tD+qTXyDL//xIJIZDL5VAul2FZlgmGZmt+y0SapRzGZddrLpdDpVLB3t4efN83gd5E +Jsce1zAI0W630Wq1zOSGfD7/1ga+oyhCt9tFu92GZVmYmZnB0tISLly4gPPnzyOXy8ESlqkGkt3j +r+NzwLIseJ4HzjmklGaiShiGz/y9l9mW8SVAXNeFy92/2HuLEEIIIYQQQgghhJC33ds5qkvIWyjL +QM3n8yiVSiZ4Z9s25ufn0Wq1cPfuXTSbTayvryOfzwMApqenMTs7awLn2bqtUkoMBgPUajVIKSGE +QBzHplx6u91Gv99HEASvNPvu+1YoFLC0tGTWiC4Wi/jjH/+Ix48fo9fr4cqVK9jc3MTCwoKZHGBb +tjlWnHPEsc7GPjw8hJIKnHGTLZ/ECTqdjjlWuVzuxKBMHMdpEK4P29bvla0L3Wq1cHh4iO3tbQRB +YLKEFxYWMD8/j83NzYkM5k63g4IsIJfLmUx1YLRu9XhQWnAxUf671+uh1WqhVquh3qjD8zxTRjwL +NGclwB3Hged5Zs1x27YhpUQQBOj3+/B9H2EYpl+RyYz+NizLQqlUwtzcHHK5HOI4Hu1vp2OuYcdx +JvY3juOJSQeWZcFxHAyHQ2xvb+P27dvY3t7GcDiE4zhYXl7GzZs3cf/+fezu7qLT6WBzcxP7+/sA +YAKu2bbk83lEUYQgCEyJ6Cxg97xt4ZyjXC6jUqkAAPr9PprNJhqNBhqNBjzPMxMpsmOfTbLIgv3j +7/EsSZxgMBzA9/2JteLz+fxEtnwSJxj6Opic/RznHIVC4YV+LpfLoVAovLbgIeccxWIRMzMz5jrI +MpQPDw/Nsc8m7GQl6+MoRrvTBufc/G4URRgMBmi1Wmg0GnAcB46jJ4wkSYIkSdBs6dfNzsf09DQq +lcqxpfXfBkmSIAgC+L4PxvREgVKphHK5jFKpBACI4shMNqjX6xgMBq8l8G3bNorFImzbBmMM/X4f +rVbLfAkhzOdVkiRoNptm4ki2L88zGAxQO6yh0+2gXCqjWCqae9exHTDOXks2OyGEEEIIIYQQQggh +5PtHgW9CzjDHcVCtVsEZx/Xr103gtN/v49NPPwXnHJVKJV2n2EOxWEQ+n4fneRgMBtja2sLvfvc7 +rKysYHl5GZZlodFooFar4bPPPsPa2hp834fjOJiZmXnTu/ud5PN5XLhwAZVKBc1mE7u7u+j3+9jY +2MD//t//Gz/84Q/hOA6WlpbguA5KvGQCgUEQYHt7G7///e9x7tw5LC8vw3EcE7z885//jLW1NXS7 +XViWhenp6WO3odPpYGNjA9PT06Zc/WAwwKA/wNr6Gm7duoV79+6ZDMyVlRWsrq7i0qVL2NzchOd5 +6PV6uHv3Lv75n/8ZKysrOHfuHKqVqinfmwWMhsOhydJWSmFmZgbnz5/H/fv3IaXE3t4e/vznPyOK +IiwvL2N5eRlJnCCMQnQ6Hezt7eHg4ACXL1/Gu+/egBACnueZYOLOzg4ePHiAKIrAOUcYhvB9H51O +57nZmidxXRcL8wuQUmJjY8OUZr937x7+1//6Xzh//jzOnTuHqeqU2d8sU3cwGJj9nZ+fx+LiIrrd +Lu7fv///Z++9u+M6rjXvp0/qnHNANzIIgEnB9mv5jmVfz5076RPOh5hZa26yfcfp2kqURDEAIIBG +6gY655xPv380a6vBTIoSRWn/1uKiDDb6VNWpqnNcz97PxqefforhcIhwOAyn04n19XVsbm5iPB7j +4uICjUYD2WwWn332GTY2NrC+vg6n04lgcNYWMfaVSgXJZBK/+93vEI/HEY1E4fF4ntmW6XQKp9OJ +eDxONZKLxSJu374NXdcRiUQQjUYxnU4xGAzQbrWRL+RRKBSQSCSwubmJWCz2QuPXarewu7uLBw8e +kPjtcDhw9epVXLt2jYTcdqeN3d1d7O3tPfa5q1evkhD/pM/Z7XZc3b6Ka9evPbHO+etA0zT4fD6s +rKzg4uKC1trx8TH+8pe/YHl5GeFwGG63m1wYyuUy8vk8Op0OHA4H4vE4crkcNE1DpVLB2dkZPv74 +YyQSCYRCISof0Gw2sbe3h+PjYxSLRaysrGBtbQ2JRAIOh+OFhNW3DSE2D4dDCgjIZDI4OztDMBik +Eg/FYhHpdBrJZBLHx8evvK6fhcViQSgUgtVqxdHREYbDIQqFAu7du0eZ+06nE/1+H41GA5lMBru7 +u8jlcjCZTC/0XLq4uMBHH32Eg4MDBAIBBINBRCIRxGIxhIIhmMwmyjpnGIZhGIZhGIZhGIZhGObt +hoVvhnmLEcK3zWbD5uYmhsMhNE3DvXv3cHR0BLvdjlgsBrvdjkgkAo/HTWJuu91GNpvFF198gWq1 +im63CwBIpVJIp9N48OAB0uk0JEmCy+V60139xlgsFrjdbmiahvPzcySTSZyenuL8/By5XA6qqpKg +JrJCxVhVq1USiqvVKjqdDmRZRjqdprE6OzuDruuU2fskWq0WUqkUrFbrzHr5YaZ3pVLB0dERHjx4 +gOPjY1itVkQiEVy/fh0LsQXEE3Hcu3cPRqMRxWIRyWQSrVYLV65cQafTQSwWo4zsTCaD8/NzNJtN ++lkikcDKygoJl8BMfP3qq6/QbrextbUFg8GA0WiETqdDda+TySQ++OADhEIhyrI1m80YDofIZrNU +99dsNlOmeavVovrVL4tRMyIQDMDhdGBnZwdmsxm5XA7Hx8fodDq4cuUK2u024vE4JEmCLMvIZrM4 +Pz9Ho9Gg/q6urkJRFHQ6HRwdHeH27duIxWK4evUqtre3sb6+jo2NDZTLZezs7JDw/fnnn8NgMCAY +DMLv88Pv98Nut2N3d5cCIERbNjY2sL29jXg8TlmpuVwO5+fnlKkqSRIWFhawsrICWZ5l3U+nU5RK +Jdy9exedTofGfjKZoNvtolgs4uDgAIeHh3jvvffg8/leWPhut9vY2dnB73//e6pXHQgEIMsyNjY2 +SPhutVrY2dnB7373O/qc3++HJM0CaEj4bnfo+0SNeZ/PN/u+KxvfmvCtKAoC/gCMRiP29/dhNBrR +7/eRSqVgMBhQqVSwvb2NaDQ6CxzpdnF8fIzDw0P0ej188MEHuH79OlKpFGX+p9Np3Lp1C7VaDRsb +Gwj4A8gX8jTep6enqFar2NzcxPr6OuLxOOx2O+r1+rfSxzeJEL5HoxHG4zGq1SpyuRzS6TTNA1mW +cXR0hPv372N/fx+5XI7qpb9OzGYzQqEQfD4frFYrBoNZeYLd3V3ouk4idavVQiaTof02n8/D7Xa/ +UJsuLi7wH//xH/j4448Ri8UooGQ0GsFsNtOYsPDNMAzDMAzDMAzDMAzDMG8/LHwzzFuMwWCAqqpQ +FAXhcJjsa0ulEvL5PFqtFu7du0f2yQ6HAx6PB2traySCnZ+fo9/vo1KpQFVVdDodylgVtcHnrbTf +ViRplrFss9mwuLiIa9euQZIkZLNZ5HI5nJ6eYm9vj7JNfT4fgsEg1tfXIcsyAFA96Gq1CqPRiE6n +g06nAwAkAj7LlrrX66FUKlF2aTKZRKfTQavVQrVaxWQyQSAQQCwWw/LyMhYXF+Hz+SiDVWTjjsdj +1Ot1nJycoNfrIZlMQpIkSJJEtXsBwO12w+PxwGq1wuVyQdM0bGxsIJvNApjV+hUW6sVikSy22+02 +qtUqBoMBJpMJdF2HxWxBNBrF+vo6ptMpGo0GDg4O0Gw2kclkYLfbYbfbUa1WyYb4pe+R/LUdv+iv +oihUG1v09+joiIRlkbWr6zo8Hg/cbjfa7TaOj4/JPn4ymcBmsyGRSGBjYwPRaBRutxvRaBSrq6uU +NXx4eAiv14vl5WX4fX4oqkI14q9evUrlAFqtFk5PT9Hv93F8fExtabfbaDQa0HUdbrcbbvcs0MTp +dMJms2FjYwO5XA7D4RCTyQQXFxcYj8czC/2HGd/dbheVSoXG/mVrGYt71263MZlMoCgKut3upUAE +UUtZ2EaLWuOP2llPJmP6XLPZfOrnXjeyLMNsMUOSZ0EbV69epSzker2Oo6MjdDoduFwustkXASRG +o5Fs6mOxGDY3NzGZTKBpGvL5PGWHi4zvVqtF/9tut2NpaQmJRAI+nw+qov4ghW+bzUZrpVAoIJPJ +YDQa4ejoCL1eD6qqQlVVshQX9c6/DWt7VVWppEAikcDa2hoFMxwcHKBQKCCdTlNpDjGvVVWFpmkv +JFb3+33aC8T/Fvb36XSayoZ4PB54PB64XK5ZLXCjkcVwhmEYhmEYhmEYhmEYhnnLYOGbYX4AGAwG +qvc9Go1QKpVQr9cxGo1w//59NJtNsvr2+Xx45513YLfbKeO5Xq/j7OwMVquVBBGXy0U21jab7U13 +8bWO1UJsAT/96U+hqiq63S6Ojo5wcXGBe/fuQZIkbG1twe/3IxaN4f3334fFYsHp6SlSqdRjYyWE +ElFX+1ljJURzITALAXQ8HkPTNDidTly/fh1Xr17F2toaYrHYrMazqmF5eRnD4RDBYBAnJydIpVKU +7TydTslaWwhCHo+HLL9nmcsO2O0OvP/++7BardSfSqVCGcjArM60EKOEYKuqKqw2K5aXl/Gzn/2M +fjefzyOdTkPTNMTjcWpjv99/ZWFUlmVIBgmLi4v4xS9+AZ/Ph9PTU5ydnSGXy+Hi4uKx/qqqSvbx +oVAI3W4Xh4eHKBQKNPeDwSCWlpawvLwCm80KRVEQCoVw7do1AMDOzg7S6TQCgQA2NjYQiUTgdrth +MpmQSCTwwQcfXGpLPp9HNpuFruuPtcXtdsPn89HYOxwOqKqKd955BxaLBcfHxzT2lUoF9+/fp7EX +oq3NZoPVOmvnyyBqXuu6fqnW+NM+M//3075v/nPP+uzrXKOaqlH2/nA4RCAQwNnZGVKpFM7Pz5FO +py/VsRd1qoPBIGw2G4yaEQsLs3XucDiQTqdxfn6OcrmM/f39S9ey2+1YWFhANBrF9vY2FhYWYDKZ +vhVr7+8DYpwkSUKhUEA+n6dAmuPjY5hMJhiNRjidTni9XqysrFD5g9eNLMswm83weDzk6CDWejqd +RiaTofrzQpAW//2iddjF3B0MBqjX6xgMBmg0Gkin07h9+zZ8Ph8CgQAWFxdx5coVrKyswO12Q5bl +ZwYyMQzDMAzDMAzDMAzDMAzz/YOFb4b5HmMwGEhgnUwmcDgcMBqNUFX1sew7UQt1PB6jUCig1Woh +nU7j4uKChPDBYAC/3w+TyQSLxYJut4tsNjurLfwwk05kO2uaRrbYDocDDodjJsJqGkwmE1wuF0Kh +EFwuF9VHfbRNiqKQSCHsoG022wuJFa86XpIkUftETedZ5rpMbQwEA7DarFS/+cGDB9B1HdVqlWor +A0AoHMKNGzegqir6/f6lsRqPx/B6vfD5fDAajbDb7VQr2eFwwGq1QlM1mM1muFwuBAIB2O12Ck4Q +tbhFm7xeLwKBANlxLy0tIRAIUN8WFhZgs9ngdrupnnSxWEStVkOv17s0DzweDwnpsVgMPp8PdrsN +mqbh6tWriEQi+Oyzz9Dv99FqtVCr1agtkiSRM8DMHt8Do9EIi8WCpaUlGAwG6LqOUqmESqWCTqeD +8XgMs9mMYDBIFvGBQAButxsWi2UmZsvPz5wUAjIkIBqNkj29LMtoNBooFApUQ1sg2qqqKpxOJyKR +CFm912o1qKqKaDSKeDyORCKBaDRCvxsIBLC9vQ1d15HL5TCdTjEcDtHr9dDtdmG3O6AoCiKRCMxm +M9l8z7dFZPw/qS3hcPjh2NthNpuxvb2NcDgMm82GwWBAbRRZxWJ9uN1uhMNheL1esmJ+EeYFQkVR +yHrfbDZfylyVZRlWqxVut5s+53K5YDabL61hWVYufW4ymTzxc9/GOpYVGbIi0zwIBAJQ1VkG9vw8 +ELbcwWAQdrsdoVBoVtLAqCEcCl8q1VAqldBsNtFoNDAYDMi6PxAIYHV1FdeuXcPa2hqCwSCm0yna +7TatIxHkIrKfH80ElmUZdrudAodEwIhwi5j/nMi4drlctK+aTCYoD8fb4/GgVqvB6XRC07Sv18Uc +Yo/zer20B6uqClm6fD2j0QiXy0XXs9lscLlciMVisFgsyGQyKBaLOD4+Ri6XQ61Wo3ERmdg+n4+C +NOb7psgKZFmGxWKhrGmHwwGjZnzq80BcXzwPxF7pdDqxtrYGRVFgNpvR6/VQLpfR7/cxGo0AgMZW +9F+0z2azPbNGtwiQEPvufFkEk8kEt9sNr9eLYrFITgsiaMhqtVLpC1mWv7XMd4ZhGIZhGIZhGIZh +GIZhXg8sfDPM9xBxsK4oCm7evEk1k69cuYJ4PE6CyJNwOpzYvLIJs9mMSqWCarUKh8OBq1evwu12 +w2w2w+FwQNM0WCwWrK6ukhW6EIH8fj/6/T4JqkKYCYfC8Pv9MBqN+M1vfoNAIIAbN24gkUjAYrE8 +ZonucDiwtbUFo9EIWZZhMBjIYvrbQFVVWCwWLC4u4u///u/h8/lw7do1LC0tXWqfpmmYTqdYXFzE +hx9+CK/XC6/Xi2AwiHA4jHA4DGCWGSn+22w2Y2lpierimkwmBAIBBAKBS2NlMplgMpkQCoUQDAVh +d9jx61//Gl6vF1arFQ6HA5IkXbKUNxgMsNlsCIfDiMViJLjMYzQa4XQ4EY/HqXZzs9lEu92+VOfW +arWSeCayWMV8EYKnwWDA1tYWrFYrNjc3H2uLEHl9Pt9svjlmv+/1+kiwCoVClL0+mUwQCoUQjUYh +yzOx8p133sHGxgbW1tYoYONlMBqNcNgdWFhYoBrUjUYDrVbrUn8tFgv1NxqNUoCA3+9HrVbDYDDA +cDjEysoK/P7ApWsIQXVzcxMGgwHxeByLi4uU8W+zzcZKZL/GojGM3xvD5/Oh0Wig3W5fsnUXbXG7 +3YjFYojFYpSlqsgKLBYLDAYDrc+NjQ2yyxdjbzKZ4PP54PV6EY/HSbR9Eex2O65duwZZltHv9ykQ +Y/PK5qX9QnxOkqRLn9va3Lr0OZvN+sTPbW5ufmeZsGIeRCNRvPvuu/B6vWg0Guh0OhgOhxTs4nQ6 +qTzBwsIC2WB7PB76rnA4jEajgV6vh9FoRO4IgUAA0WgUkUgEPp+fxGpRf/oXv/gFbDYbotEoNjc3 +SQydF0AdDgd+9rOfwWAwwOfz4cqVKwiFQrDb7ZfEaKvVips3b9J/R6NRhEIhOJ1OGtv/8T/+BwqF +Am7evAm/3w+z2fyYgL6ysoJ//Md/xPb2NmUpe71emMyX664vLCzgww8/xPLyMmKxGKLRKAKBAJwO +J1RNxdraGmRZxubmJmq1GjqdDo2Lz+dDJBIhG/iNjQ1ySfB4PBR0sLW5hcFggJs3b+LGjRsIhoIw +m82PuRX4/QHcvHmTAkgkSSJXBFmW6V6pqopgMIh8Pk+OGDabDV6vF3a7nazQTSYTZaVHo9EX3mOm +0ymVERClHYSbQavVwvHxMYnsYk+ORqP0M84CZxiGYRiGYRiGYRiGYZjvL4bpt+1ZyjDfMiJr9/z8 +HJlMBplMBqVS6dKBtRCh3jZE1mGn08F0OiUB+lmZZ5PxBIPhAIPBAKPRCKPRGIoys5M1m82Xst/6 +/T4Gg+HDa+n0vaqqQp/o0KezusCSQYIkS1AUBZqmYTKZkFhqsVhgs9memJk4GU/Q6/cwHHxtGayo +CtVP/TbGa17MEO2zWq0wGo3UvvnPdTtddHvdmU21okLVZqKP0WjEZDzBeDLGaDTCcDDEYDiArk9f +aqx0XUe300W704YsK5BlibKmJ5MJtV2WZRKcNE2DqqiQla/FLl3XMdWnGI1HJOZOJhNMJjp0/evv +kSQZiiKTTe98piIAautgMMBwMMRoPHpiW0TfNE2bZXDKEkajESaTCQaD2fwajyeYTmfW1+JaBoOB +2mcymWZjrxlhkAwvVS93vr/D4ZAyMV+kvyKQYzwe0702Go3kWDB/DWE7L4IXxOeMRiNkaZZ1PJ1O +oU90jCdj6vvT2iLP3Xsx9iLoQwhuYm2KgJZHx17MBeHu8GhAydOYjCfo9rro9XrUb2ElLQIevo3P +fZs8a+yn06/rliuKQnNW7C9ivOfnrPjfX9vlS9AernlVVek7xLXH4zEFJ2iaRjbqkiw9tte1O220 +2x3ab00mE2RJvvRZXddp75Qkia4ryzIUWUGvP3MbGI3GsFotl+prz1+v1+tR0IvFbIHZMhOahaAs +6Pf76Ha6GI6G0FQNqqbSXmeQDA+fAQNa2+PxGJIkQ5IMNBYikGIwGFBwkclkgmSYXUe0eTgcUuDN +k9os1vFoOKKfqdrX81ysWVEGYvjwc/P7rSzJmOhf3z/RZ7FnP2lO/p//83/wv/7X/8Jvf/vbSz+f +DzBTlK+fS0ajkcT+lZUV3LhxA9evX6fAmG/ieFAul3F+fo6LiwtkMhlks1mYzWZEIhEK3FlYWIDF +YvnW1xbDMAzDMAzDMAzz6ogSWplMBhcXF8hmszAYDJf+/10sFoPT6XzTTWUYhvnRwRnfDPM9RtSf +FZbjL4KsyLAolucenIvD/VflRSyYZWVm64vvqET4fNasyWR64c954Hni54TdstFo/EZ9MJlM8Hg9 +r/4FmNlYQ5q16Vl9ex6yIkPGTFh92T4J0fhl7LffdH+fdw0hUD9rLcxbb3+TgA2RnfyiQvbLIivy +C+0Xr/tz3ybfZOznx/tV5pDBYKD5ISy2nzVWotzEs5Ak6ZljarPZXmhfFoFMz+N5e6HF8vxnBYBn +zoEXbbMYy6chgjy+i/0FANWHFy4eo9EI3W4Xmqah2+2i2WxSMEC9XkcoFHroCuCD2+2Gy+WC3WaH +zW6jNc026AzDMAzDMAzDMAzDMAzzZmHhm2EYhmEYhvnRItwBdH3mYKHrOkajEWXon5+fw2azweGY +lV9YW1vD6uoqYrEYFrQFyuxnGIZhGIZhGIZhGIZhGObNwsI3wzAMwzAM86NG2PkL0RsAOp0OarUa +WdKrqopYLIZ6vY5eb2bxPplM4Ha7ySZdlKoQQjhngTMMwzAMwzAMwzAMwzDMdwcL3wzDMAzDMAzz +CLquYzweA/i65nupVML+/j4ajQYePHgAr9eLQCCAcDiMSCSCUCiEUCh0qT47wzAMwzAMwzAMwzAM +wzDfDSx8MwzDMAzDMMwjCOvz8XiMwWAASZIwHo/RbrdxcXEBVVWhKArC4TA2NjawubmJzc3NS6K3 +JLEFOsMwDMMwDMMwDMMwDMN8V7DwzfxgEZlarVYL5+fnqNVqb7pJDMMwDMN8h5yfn6Pdbr/y70+n +U/pb13X0+31MJhP0+33IsgwAGI/HVA+8UCjg9PQUbrcbFosFVqsVJpMJJpMJnU4H5XIZlUoFrVYL +k8nkTQ8PwzAMwzAMwzAMwzAMw/ygYOGb+cGi6zqGwyGq1SpyuRyGw+GbbhLDMAzDMN8hyWQSrVbr +tX2fEKvH4zEMBgMMBgMajQbG4zEqlQoODw9htVpht9vhcDjgcrngdrvhdrsxnU7R7/dJPNd1/U0P +D8MwDMMwDMMwDMMwDMP8oGDhm/nBMplMMBqN0G63kcvlUCwW33STGIZhGIb5DkmlUmg2m6/t+4T1 ++TzD4RCtVutSPW8hfjudTvh8Pni9XhiNRgCAJEmUDc4wDMMwDMMwDMMwDMMwzOuDhW/mB4mwJB2N +RqjVajg5OcHR0dGbbhbDMAzDMN8hzWbztWZ8P43JZHKpnvdwOES32yVb9Gq1Ck3ToCgKTCYT/H4/ +NE1708PDMAzDMAzDMAzDMAzDMD8oWPhmfrBMJhMMh0PU63WkUins7++/6SYxDMMwDPMdMh6Pv/VS +J9PplP4Ier0ehsMhJEmCLMtQFAWqqsJkMsFms0FVVXi93jc9PAzDMAzDMAzDMAzDMAzzg4KFb+YH +i8FggCzLMJlMcDqdfMDMMAzDMD8y+v0+Op0OGo3Gt3odUe9bIDK7FUWBpmlQVRWapkHTNFitVphM +pksZ4gzDMAzDMAzDMAzDMAzDfHNY+GZ+kAjRW1VVuFwuLC0tQVF4ujMMwzDMj4lCoYB0Ov2tCt/i +nWNe+DabzVTH2+FwwOl0QtM0ejdxOp1QVfVNDw/DMAzDMAzDMAzDMAzD/KBgJZD5wSJJElRVhcfj +gSRJ8Pv9b7pJDMMwDMN8hxwcHKBeryObzb627xQCt/jbaDRSVrcsy5BlGVarFU6nEy6XCz6fD4FA +AKqqYjweYzKZQFEUDshjGIZhGIZhGIZhGIZhmNcMn7gxP1hEVpXNZoPf74csy2+6SQzDMAzDfIdM +JhMkk8nX9n2SJEFRFMrwliQJdrsdXq8Xfr+f/nY6nbDb7bDZbLDb7XA4HBgMBmi1Wmg2m+j1euj1 +em96eBiGYRiGYRiGYRiGYRjmBwUL38wPFlmWoWkavF4vgsEgAoHAm24SwzAMwzDfIZVKBZ988slr ++z4hfKuqCkmSYDAYqKTKlStXsLGxgc3NTXi9XphMJqrlrSgKarUastkscrkc8vk8RqPRmx4ehmEY +hmEYhmEYhmEYhvlBwcI384NGHDabzWY4nc433RyGYRiGYb5DLBbLKzu+GAwGqt8t3idsNhscDgdc +LhesViusVivC4TCWlpawvLyMRCKB5eVlOBwOaKoGWfn62pPJBI1GAyaTCaqqXqoJzjAMwzAMwzAM +wzAMwzDMN4eFb4ZhGIZhGIZ5BIPBAFVVoSgKNE2DLMvweDxYXl7GysoKIpEIotEofD4fXC4X3G43 +WZxrqgZJlt50FxiGYRiGYRiGYRiGYRjmRwUL3wzDMAzDMMyPHpGBLep3m81mGI1Gsiw3mUyIx+O4 +fv06rl+/jpXlFayurcJhd0CSJc7gZhiGYRiGYRiGYRiGYZg3DAvfDMMwDMMwzI8aSZIgyzIURYHR +aISqqnC73QiHwwiHw/B6vfB6vQiFQohGo4jFYvD5fDCZTDBILHgzDMMwDMMwDMMwDMMwzPcBFr4Z +hmEYhmGYHy0Gg4FqeGuaBpPJBIvFglgshuvXr+Pq1atIJBJIJBJwOBwwGo3QNA2KokBRFEgSW5oz +DMMwDMMwDMMwDMMwzPcBFr4ZhmEYhmGYHwXCjtxgMECWZVgsFqiqCpvNBqfTCZfLBafTCafTiUQi +gbW1NaytrSEUCiEcDsNoNLKlOcMwDMMwDMMwDMMwDMN8T2Hhm2EYhmEYhvnBYzAYKLvbYDBA0zRY +rVY4HA7EYjGsrq5ieXkZPp8PPp8PHo8HHo8Hbrd7JpArKoveDMMwDMMwDMMwDMMwDPM9hoVvhmEY +hmEY5geNJElQVRVGoxGKokBVVZhMJvj9fvj9fmxsbOCdd97BtWvX6GdGo/FNN5thGIZhGIZhGIZh +GIZhmJeAhW+GYRiGYRjmB4nI8DYajXC5XHA4HPD7/QgEAgiHw/D5fPB6vYhGo1iILcDv98Nms0GR ++RWZYRiGYRiGYRiGYRiGYd42+FSPYRiGYRiG+UEihG9N0+B0OrGwsICNjQ1cu3YNm5ub8Hg8cLnc +sFjMUFUVmqZBlmRIsvSmm84wDMMwDMMwDMMwDMMwzEvCwjfDMAzDMAzzg8Tv9+P69esYjUaIRCKI +xWJYXl7G2toaFheXYLNZYbPaICvym24qwzAMwzAMwzAMwzAMwzDfEBa+GYZhGIZhmB8kCwsL+NWv +foXV1VV4PB54vV54PB643W7Y7XZomsbZ3QzDMAzDMAzDMAzDMAzzA4GFb4ZhGIZhGOYHSSAQgMPh +wM2bN2E2m2GxWCBJLHQzDMMwDMMwDMMwDMMwzA8RFr4ZhmEYhmGYHySKrMBoNEKWZaiqCoPB8Kab +xDAMwzAMwzAMwzAMwzDMtwQL3wzDMAzDMMwPEkmWoEkaNE2DwWBg4ZthGIZhGIZhGIZhGIZhfsCw +8M0wDMMwDMP8IGGxm2EYhmEYhmEYhmEYhmF+PHCRQ4ZhGIZhGIZhGIZhGIZhGIZhGIZhGOathoVv +hmEYhmEYhmEYhmEYhmEYhmEYhmEY5q2GhW+GYRiGYRiGYRiGYRiGYRiGYRiGYRjmrYaFb4ZhGIZh +GIZhGIZhGIZhGIZhGIZhGOathoVvhmEYhmEYhmEYhmEYhmEYhmEYhmEY5q2GhW+GYRiGYRiGYRiG +YRiGYRiGYRiGYRjmrYaFb4ZhGIZhGIZhGIZhGIZhGIZhGIZhGOathoVvhmEYhmEYhmEYhmEYhmEY +hmEYhmEY5q2GhW+GYRiGYRiGYRiGYRiGYRiGYRiGYRjmrYaFb4ZhGIZhGIZhGIZhGIZhGIZhGIZh +GOathoVvhmEYhmEYhmEYhmEYhmEYhmEYhmEY5q2GhW+GYRiGYRiGYRiGYRiGYRiGYRiGYRjmrYaF +b4ZhGIZhGIZhGIZhGIZhGIZhGIZhGOathoVvhmEYhmEYhmEYhmEYhmEYhmEYhmEY5q2GhW+GYRiG +YRiGYRiGYRiGYRiGYRiGYRjmrYaFb4ZhGIZhGIZhGIZhGIZhGIZhGIZhGOathoVvhmEYhmEYhmEY +hmEYhmEYhmEYhmEY5q2GhW+GYRiGYRiGYRiGYRiGYRiGYRiGYRjmrYaFb4ZhGIZhGIZhGIZhGIZh +GIZhGIZhGOatRnnTDWAYhmEYhmEYhnlZ6vU6crkcisUiTCYTzGYzHA4H3G43nE7nm27ea2c0GqHf +76Pb7SKbzSKbzULTNMTjcUSjURTyBRSKBQyHQxiNRphMJvh8Pvh8PpjN5jfd/DeOGL9+v49MJoNc +LgdJkhCPx5FIJJDP51EoFDAYDGA0GmE0Gmn8LBbLm24+8x0znU4xGo0wmUxwfn6OdDqN4XCIhYUF +LC4uwmg0QlEUSBLnEsyj6zrG4zGazSZSqRTS6TS8Xi/i8Th8Ph8kSYIkSchcZJBKp9DpdBCPxxGP +x2E2m6EqKmRFftPd+E6ZTqcYj8cYj8fI5XJIp9NoNpuIx+NYXFycjYuq8lx7CmL8dF1HPp9HKpVC +tVrFwsIC4vE4bDYbNFV7bfOqWqkik82gVqvBbDLDaDLC6XTC7XbD4XC86eFgvmXEc6FUKiGdTiOf +zyO+EEdiMQGn0wlFVp4718Q+ORgMkE6nkUql6H1uYWEBsixDVVUYDIY33d2XHpdyuYx0Oo1sNouF +hQUkEgm4XK4f5d7OMAzDvFlY+GYYhmEYhmEY5q2jVCrh1q1buHfvHjweD4krG+sbP0jhezAYoFar +oVgs4qOPPsJnn30Gm82GX/3qV7Barbh3/x6++uortNttOJ1OuFwuXL16lYICfuyMRiPU63WUy2V8 +9NFHuHXrFlRVxYcffgi3242dnR189dVXqNfrcDqdcDqduHr1KoxGIwvfP0L0iY7BYIBer4c7d+7g +T3/6E1qtFn71q1/R/DCbzdA07U039XvFaDTCYDBALpfDn//8Z/zlL3/BlStX8Otf/xrXrl2DoihQ +FAX3d+7jj3/8IwqFAj788EMKNLHZbD86cWQ6ndJc29nZwR//+EdkMhl8+OGHsFgsCAaDAACj0fim +m/q9RAjf/X4f+/v7+H//7//h6OgIf/d3f4ff/OY3iEajsNlsMCuv5zmYL+Tx2Wef4eDggN49FhcX +sb6+zsL3DxwRENXtdpFMJvGHP/wBd+7cwS9/+Uv8wz/8A1RVhdlsfu4eNh6P0ev10Gg08Omnn+JP +f/oT7HY7fv3rX8Plcs2+Q5Lfqr1wPB6j1+3h5OQEf/jDH/Dpp5/iww8/xG9+85vZ3mXGW9UfhmEY +5u2HhW+GYRiGYRiGYV4b4hC/2+mi2+vSz41GI6xW62sTERuNBpLJJD7//HMEg0FEIhEAQCgU+s76 +qus6JpMJiRb9fh9msxkWiwWKokCWZBgkA9rtNjqdDgwGA8xmMx1qSrL0whk9k8kE7XYb5XIZe3t7 ++Pjjj+FwOBCPx/GTn/wEqVQKd+7cQa1Wg8/nQzAYhMfjwfr6+nc2Ht9nxuMxOp0OqtUq9vf38fHH +H0PTNESjUfzyl79EOp3G3bt3USqV4PV64ff74XK5sLq6+qabzrwE0+kUnU4HnU4Huq7TelNk5aXW +mz7VMRqO0Ov1cHx8jE8++QTNZhPhcBgffPABzGYzC5FPQNdnAQPVahX379/HX//6V7TbbaysrGB5 +eRmqqkJVVZyenuLTTz9FNpuFz+fDzZs3YbFYoKoqANDzQ9M0Guu3TQgSTKdT9Ho9tNttjEZjWCxm +ym6XZAlTfUpC2NnZGW7duoWzszP4/X785Cc/gcvl4gCL5yACLtLpNG7duoW9vT24XC7cuHEDbreb +Mua73S663R4URYbZbIbJZHrpZ7F4hnzxxRcIhUIIh8OQZfm1vXtMp1P0+320220MBgNYzBaYLebZ +O4UsQ5KkS+8e3W4Xw+HwiZ9jXj8iyCKTyeDLL7/ERx99BKfTiZs3byISidAe9ix0XcdwOES73cbB +wcGl97mf/exnUGQFE+MEMt6O/U4EBPT6fWSzWdy+fRt//vOfKYAwHo9DURSYTKY33VSGYRjmRwQL +3wzDMAzDMAzDvBam0yn0iY50Oo3bt29jd3eX/i2RSOC9997DjRs3YDAY3ioLx6f1tdvtzsSK0zPc +u38PR0dH2NzcxPb2NiKRCCwWCzRNwxdffIEvvvgCsizj2rVr2N6+CqvVAqvV+kKHpABgMBigqipM +JhMURYHBYIAkSXSYaDKZoGkajEYj/S3L8ls/zq8LSZKgKl+PnxAFFEUha3hN0y79YSvrt4fpdIrp +dIrJZIKdnR3cunULnU4H29vbuH79OpxOJywWC4vV32NGwxEajQY6nQ7u3buH+/fvIxaLYWtrC8vL +y7Db7bDb7W/VmhQC5cHBAW7duoVCoYDt7W1cu3YNPq8PFqsFqvJizwDm1RgOh2g2m6iUK7hz9w52 +dnbgdruxvb2NtbU1OBwO2G3270VQhZgvJycnuHXrFk5OTnD16lVcu3YNgUAAVqsVRqPxoYDfxdHR +Ee7fv4/z83NsbW1he3sboVAIdrudnUoYhmEYhvlRw8I3wzAMwzAMwzCvhel0iok+q4n7xz/+EX/8 +4x9JeL1+/TrsdjuuXr0KWZ4dML/tomy/30etVsOD/Qf47W9/i88++wwffvghFEWBqqrwen2wWqe4 +c+cO/vf//t9QVRWDwQCBQAC67oXJZHph4VuSJKiqSrWFxdgJ4Vb8mRe/ZzUi3x6R6NtEkiSo2tfC +N/3sYTDBo+NnNBrfuhqbP3Ymkwkmkwl2d3fxz//8z6hUKmg2m/D7/RQ4wsL395fBcIBur4uLiwv8 ++c9/xu9//3tsbm5iOBzCZrMBACwWy1sjfItAjMlkgmQyiX/913/F0dER/uEf/oGykBVVYeH7W2Y0 +GmE4HKJUKuFvf/sbfve73yEWi2EwGFBZlBexp/62mU6nmOpT6LqO4+Nj/O53v8OXX36J//yf/zPs +djuM2uzZr6kaut0uarUadnd38a//+q/Y2dnBr3/9a3qnkGWZhW+GYRiGYX7UsPDNMAzDMAzDMMw3 +Zjqdot1uo9Vqo1AooFwuo1arYTqdAgCKxSLy+TwKhQJsNttLZTt/X5FlGYqiwGKxwO12w+v1Un3G +mbWqAfpER7vdRrVahaqqaLVaGA1H0HWdxuZFr6WqGh1sS5JE158XvkX2shBuJYmFW+BrkVvcG2EF +O5/x/eg4svD9diGyvoWlfbVaRbPZxGg0wng0fqn1xnz3zAcnOBwO2k/Fs0JRFEhvaSBPt9tFtVpF +pVJBo9HAYDDAaDR7DjDfLsJhRtM02O12eL1eOJ1O2Gy2h88C5Xuzz+tTnazOa7UaKpUK6vU6er0e +RuMR9MlsvkiSBEVWYDab4XK5qE8isEsEFzIMwzAMw/xYYeGbYRiGYRiGYZhvjD7R0Ww2kc/nUSqV +0O/3qRblfN3X8/NzhEIhEhbfZlRVhdVqRSQSwbVr16BpGq5evYpwOAybzQaj9vqyS2eikPJE4Xs+ +S/nRjOW3JTvy28ZgMJAtvBgXMYai7vD8+H1tic4CAsN8F4i9zO/3Y3NzE4PBAAsLC4jH43A4HDCZ +TDBwIA/zkggHFkmSsLGxgV6vB6/Xi8XFRTidTphNpu+N8P0iGCQDjEYjrDYrFhYWqI751tYWgsEg +rFbbW/9uxTAMwzAM801h4ZthGIZhGIZhmG/MRJ+g2Wwik8mgUqlgMpnAZDJhPB5jMplA13XUajWc +n59D0zR4vV6YzebHvkfUCZ/oEwwGA/T7/Yd2sTp0fQJJkiFJBlQqFXS7Xfpuke2p6zpGoxHVwAQA +o2aEqqkYDUcYjoaYTHRMp7PMqXlbcEmSYDAY0O/3MRgMHn73FNOpTp9RVZX+WwipTqcT4XAY/X4f +wWAQTqcTkiSj1W6h0+mg3W5jMpnAYDCg1WohX8hjNB5hMBjAarXBaNRIqNY07YlitSRJ0DQNZrP5 +Uv1p0Q7xu/OZy8ISfb7+8WAwwHAwxGg8wng8ga5PHmbESVAU+VI/VVXFVJ+i1+9hOBhios9se2VZ +htVqhdlsJsFg/r71ej30ej0AMwtZk8kERVYgyRJ99lXaMj8u4npiHAeDAXRdx3g8y+wV36OqChRZ +od8xGk00fiJw4Emi98tkzIvxHQ6HaLfb6Ha7VHfdYDDQvBOZh8Je3Ww2z+auPn1uPzRNfWw8Hr2+ +GNN+v/9w3emXxlSSDND1Wdazqio0d8RYiPs2nU5htVphs9mozfP0+326hmSQIMmzuWnUjJBk6YXv +r6qoUDX1ifN+OBxiOBxiNBo9/DOmNQuAMvbFPdN1Hd1uF51OB61Wi/addnvmQCFJEoaj4cwyeC5I +RFXUF7Y4nk6nGI/HGAwGaLVa6Pf7UBQFk8kE0+mUxnE++GQwGKDXnWVrXgpU0YxPvO5kPEGn20G3 +26WgDDFfX8aBQKyP8WRM82oymTw2r8R9UNSvA2gGgwF6vR6GwyHVHBbfKX5PliVoqgbN+HAvlJVv +bBUt3BfsdjuCwSASiQSCwSA8Hg/MZjMkScJkMkGnMxuf0WhE62wymWA4HGI8HtOePb++xZqeHz+x +9gbDAfq9Pj0bnrZm5uespmnQNO2Z90O4fTSbTdTrdYxGI3IkKJVKsFqtGI1GsNlslAEu1pVYQ6PR +CO12G4PBAIqiPLFvYq7Nrx9x30aj2b4i7uV4PH7sHop7/zJrQaw38YxVZOXSHHrSuAwGA3S7XQwG +A7qu2INkWUa320Wv16PnvVjvYt6JNS/6K8bgeciyTM/DQCCAeDwOl8sFt9s9cwDRZg4g888w8Q4g +9pHpdApJkiHLEqrVKrXz0Yx9MS7ze/DL9KPT6dB8GQ6HmE6n6PV6qFQqyOfzs2dMp03uFZqmIRAI +wGAwUG1vo3G2l45GI3Q6HXQ6XciyRM80Yf0+v7aFC8r83vjoXjL/TjY/LqJPs/6Ldl1+V3qeED8c +DmnMxfeJ/WC+LWKNDAYDAIBkkKCoX7vePPq50XAEYBYsMP99Yu0PR0NaH6IsAQBa+4++7z1pvs0/ +71qtFobDIVqtFnR9Su+sYu/QjBrMJvNTx0E8X6bTKXr9Pu0/06lO159/vjx6j+b3+0fvkXgfEnuX ++L5H943xeEztmD03ZnN4/l6LcdE07YXX4fy9Fnv17Bk/hsVinq1FRYUkS29VIArDMAzz/YWFb4Zh +GIZhGIZhvjHT6RS1Wg3pdBr1eh1msxmRSGRmMzwew2q1otPp4OTkBE6nE4lE4onfMxqN0Ov10G53 +cHx8hOPjYxK5+/0+CanFYhHJZBKdTufSIa6wCL19+za++uorGAwGLC0tIRaLIZPJIJPJoN1u04F1 +LBbDwsICPB4PrFYrNFXD8ckxTk9PUavV6EA0FAohFAohEokgFltAJBKeHXiOJyiXy9jZ2cHt27cx +Ho/h9/vRbDaRTCaRTCZx9+5ddDodAMDt27fRbDZhsVhgNpvh9/uxsLCAxcVFhEIhyhZ/FFmSLwnb +4jBWHDg/eiA6L3yPx2P0ej10u12cnJzg9PQUpVLp4aF4hw6mnU4nIpEIwuEw9bfb7eL05BSnZ6do +NptoNpvweDy4fv06bly/AYM0E0ZFsEG73cbOzg52dnZgMBiwtbWFzc1NOOwO2OwzIVUIrPNtabfb +j7UlHA4jEokgFAohEAhcGhdxAF8sFnF6eorT01O0Wi0KMhCikMfjgdfrhc/nQzgcRjAQpPGbP7AV +guV84MCLZsyLgw9xOG0AAIAASURBVNyLiwvcvn0bd+/excrKClZXV2E0GmnePZrRKuoX93o9lEol +nJ6e4uzsDM1mE51OB+PxmMZDzD1xX4LBIB0Oj8dj9Pt99Ho9HB8f4/j4GIVCgYQpMS/mHRh8Ph8C +gQB8Ph/cbjc8Hi92d3dw//596LqO9957Dz/96U8vBVkI0uk0kskkqtUqjVU8HkcinoDT6US3NxOw +UmcpnJ6dIp/Po91uo91u032x2+2IRCKIRCIIBoMIBoNwOBx0jWKxiPPzc+RyOZRKJZRKJTrIlyQJ +ZvPsoFysHUmScHh4iKOjI3z++eeo1WrodDrY29sjcVGUJIjH44jH43R9Uef3eUwmEzQaDZyfn+Pw +8JBsq4fDmXjicDjgdrsRDAaxsrKCtbU1HB0dYW9vD6VSCU6nE06nEwsLC1hYWIDP53vsGo1GA1/d ++Qr37t2D0+lEMBik+x0OhV9YmBRCUq1Ww/HxMc7OzlCv19FutzEcDuk+uFwueDweBINBLCwsIJFI +4OzsDHt7ezg/P6cgByEKCcHOZDIhFovROIq+fRP0yUx0aTabODg4wN/+9jesra3B6XTC7/fTHr+3 +t4evvvoK5XKZ1lmj0aCgq/n74fF4EAqFsLS0hOXlZRJl5wXKVCqF4+NjXFxc0D45L5aKvot7GwwG +EYvFEIvFLolyj9LutOk5dOvWLVQqFQwGAxweHuKf/umf4HK5YLFYYLVa4fF44PF4kE6n0e12oes6 +BW2USiUqHSL6Jiy7A4EAlpeXsby8TIE28327uLjA6ekpPffa7TYF31gsFoTDYUSjUZpnbrf7he5V +t9vFvXv3cP/+fRgMBrLbFhn6JpPpsd9Jp9O4e/cuzs/PaRzF3m40GrG7u4v9/X3UajUSN8V+JQR+ +u92OeDyOxcVFeL0+OJ2O59aynupTegYeHR3h448/RjgcJrcWo9E4Cz6azMTSRqOBo6MjnJycoFar +odvtYjgc0rtHJpPByckJOp0OtVEEH3W7Xezu7uLBgweoVquP9UMEy9lstrl+zGzKZVnG/fv3cfv2 +bXz++ecoFAoYDoc4OjrCb3/7W9y6dYvmi9/vRyAQQKlUwsHBAUqlEiaTCQL+ACwWC60VMf9MJhNW +VlYQDoeRzWaRzWYpoELXdXpOLiwsYGlpCQsLCwBAz/Z+v49ms0nPF/FONhgM6HlqMBgowC8QCCAQ +CDx8X4ohFAo98x6VSmXs7z/A2dkZ7S/BYBCLi4vUFgCoVCo4PDzE2dkZBSoGg0Gsrq5icXGRPlet +VpFMJpFKpSiwwu/3U9/6/T76vT5y+RxOTk5wcXFB7w/T6ZTm26PP3Sft2SKYJZ/PYzweo1KpoFqt +0lo1Go20PlZWVrCysgKTyfRY6Y3hcEjPl3K5TO9FQiQW5XSi0ShWVlboHdpgMJDY3mq1cHJyglQq +dem9WbwvejweRKNRxGIx+P1++P1+CkAV5Yo6nQ5yuRzOzs5oPxIBcWJcotEoFhYWEAwG4ff74fV6 +X2jfmE6nKBaLyGVzyOZm87BYLGJra4vckkQwE8MwDMN8U1j4ZhiGYRiGYRjmG6PrOur1Os7Pz1Gr +1UiUEod/qqqi3W4jlUphYWEBo9Hoid8zHo/RbndQKhXx1Vdf4ZNPPsH5+Tnq9Tq63S5sNhvsdjsm +kwll14jD23nh+9atW/i///f/QpIkvP/++7h+/Tru3buHvb09VKtVjMdjAMD29jZu3ryJRCIBr9cL +q9WKL774Ap9++ilyuRy63S7G4zFWV1dx5coVXLt2DSaTCZFIeJZBORmjXC5jb28Pt27dgtvtxvb2 +NsbjMT755BN8/PHHqFQqaLfbGI1G2Nvbw9HRER3ahkIh3Lx5E++99x6JK08SvkXG0qPCtyRJkAzS +Y5mmwtJbCN+ixuzu7i4+++wznJycoFqtol6v0++EQiFsbW1he3sbk8kEDocDzWYTew/28Nlnn6FQ +KKBYLGJhYQEmkwlXr16FDBkG2fDwvrVRLpfx2Wef4fe//z0kSUKr1YLL5cJ0OoXJbIIkSeh2u6jV +atjb28Onn36Kk5MTVCoV1Go1ansoFMLm5iaNpd1uvzQu4qD3/Pwct27dwueff45yuUwHzqJPQpzb +2NiAqqpIJBI0fsLqHMAzx+95CNH/7OwMf/zjH/Hb3/4W77//Pv7Tf/pPsFqtuH//PnZ3d+m7hUgl +7JybzSbS6fQT+yHu9/r6Ora3t7G1tQVVVREMBi+tvW63i3q9jt3dXXz00Uc4OjpCvV5Hs9kkkVhV +VcrmSiQSWF1dxfLyMuLxOHRdx+eff45/+7d/I4Hz6tWrszGSZGBO/z8/P8enn36K8/Nzygzv9/tw +u92w2qzodrtoNBo0b5LJJN1fcXAuxP+rV69iOJxlYs8L36VSCXt7e9jZ2SHhVggAsizD4XDA4XDg ++vXrdLD/0Ucf4ZNPPkG5XEaj0UC320UymcTZ2RlkWYbBYIDb7cb169fx/vvvYzQawW63v7Twnclk +cHZ2hv39fWQyGQow8Pv9iEQiWF9fh8FgQDwex8nJCf70pz8hlUpRYEuv14PT6Xyy8N1s4Pbt2/iX +f/kXBINBrK+v48qVK5AkCYFAADJeXPhutzvIZrO4ffs2rd9KpULjJdZ8IpHA2toaACAcDiOVSuFv +f/sb7t69S2LQaDSieWGz2WCz2bC1tYX333+fXAGetG+91DNkqtO+nkwm8fnnn6Pf72NtbQ1ra2uU +1b2zs4N//ud/RiqVwk9/+lMMBgNkMhns7u5eEmp8Ph8WFhawsbEBg8GAWCw2EzkfZmj2+3006g0c +HBzgo48+ws7ODhqNBprNJhRFgcVigaZptGbC4TDW1tbo+0Kh0LOF73YHd+/exb/8y7/g/Pwc1WoV +rVaLhGix91gsFqyurmJ9ff2S8N3pdFAsFlEul7G7u4uzszMS5r1eL2KxGNbX1wEA0WiUxLT5viWT +SXz22We4f/8+1b0XwQsulwtXrlzB1tYWtra2KDDkReh2u9jZ2cG//du/QZIkhMNhEuJCodAThatM +JoO//vWvuHv3LlZXV7GxsYGNjQ2qvb23t4d///d/J5FeOLpMJhMYjUbYbDb4fD68++67lMmqaepz +hW99qtMeeXJygi+//BLxeBxLS0u4cf3GrNb6dOYU02q1UMgXcPv2bXzyySfIZrNoNBro9/v07iGy +8EWWrxAwxTVEP4SY+qR+eDwevPvuu5Tdr2kaFFmhMRXPZxGYkc/naQ8zmUz0PlKtVumdxufz4caN +G3C5XRRA8sknn+Cf/umfYLPZ8POf/xxbW1vY2dnB3t4eKpUKOp0OJpMJPSdv3LgBi9mCWCx2KbBK +uBR89dVX+Pjjj3FxcUHvZBaLBRaLhURySZKwtLSEtbU1XL16FWaz+bnCd7lcIsFf7C/r6+uwWCyP +Cd937tzBF198AVmWIcsy1tfXYbVaHxO+79+/jy+//JKytVdXV6lvg8EAjWYDJycn+PTTT3Hnzh0K +LtF1fWYlb7ViY2MD29vb2NzchNFofKrw3e126f1IvOOJtSoCTBKJBAaDAfx+P2WdzzMej+n5srOz +g93dXRSLRQq8EEGa165dg9lsRjwep3s0HA7RqDeQy+dw+/ZtfPHFF0in02g0Gmi32+QwE41Gce3a +Ndy4cQP6RIfD4bjkvNTtdlEqlZBMJmlcxJ6o6zoFbWxtbeH69evY2tqCpmovJXyXSiU82H+A3d1d +7O7u4uTkBH//938Pm80Gq9VKc5xhGIZhviksfDMMwzAMwzAM840QFsDVapUOrX0+HzweD2U8CQvi +dDqNcrlMB8bzVtnT6RTlchkPHjzA/v4+Dg8PUavVMJ1OSegS2SDiUFEcKM9bnY/HY7RaLVQqFYzH +YxwdHUFRFDQaDZhMJjgcDjpMrFQq2NnZQT6fh8fjgd1up0NmcZg7HA7RbDaxv78PSZIoc0hkgwth +WRxyj8djmM1mBINBLC0tUSaNoiiU2ScyLkOhEBYWFuD3+2Gz2Z5qGWkwGCjre3l5GT/96U9hsTw8 +oJYM8Pv9WF9fR7fbhdPphMfjgdvthqqqqNfr2Nvbw+HhIVKpFFlxBwIBuFwuqnetqiry+Txl9zgc +DhKAwuEwyuUyms0mstksTk9PcXBwAI/HC6/Xg3a7jdPTU+zv7yObzZKoaDabYbN9XXP00bZ0Oh0Y +jcantqXb7V5qi8iAFtdPJpOU7a0oCvx+P4CZNbzIVK7X6yiXy+j1elAUBYuLi/jpT39K4iQAeL1e +rK6uklW9yK6atxN91vzXdR39fh/1eh25XA6np6dwOBxwuVyoVCqUQSrEE1VVoU90XFxcYH9/H0dH +R8jn8wBAh9G6rtN4DIdDJJNJDIdDqKoKv99Poni9Xsf+/j6tmWq1SmMmxl7TNAyHQ9RqNdRqNTid +TsosFzbLrVYL1WqVLMJFlqI+1S8JrmI9CPtmYT8tfn5wcICDgwOcnZ2h1WpB0zT4fD44nU4YDAa6 +v6VSCV9++SUFWQgra7Fm9/b2kEqlMB6PEQgEqD2iXvu89bWuzw7xxXoTgo5YC8KW1ePxIJFIIBAI +wOFwPFO4fJThcDg7tH/wAO12GwBozQpBtVAowGCYrcdQKESZmNPpFNVqlfbGra2tx75b7Ef5fB6Z +TIbqzFN96+cEYcxbEl9cXODg4ACHh4c4Pj5Gs9mEJEkkUMzXt282mygWiw/teb/OXpVlGXa7nTJI +5zMUxV69s7ODXq+Hd999F4FA4LU8TyaTyUPXjzbt02JfB2Z20NVqFYVCAcfHx7Db7WTlLtaWEJZy +uRw5HCQSCdpnR6MRkskkzVWRrSmcOIQ7w2QyQa1WQ71eh8lkQr1ev5Tp+yxUVYHP58Pi4iL6/T5l +XzocDtpbROZ1PB5HIBBArVajLPNSqYTd3V0S3h0OB/VNzDVd1+H3+xGPx8ktZDwe4/j4GPv7+zg9 +PUW1WoWqqnC73bDb7fQ8URQF9Xod9+/fx2QygcVigd1mh8k8e8Y+y+1i3o662+0in89jMpkgkUhg +PBpfmidibpfLZWQyGeTzeUSjUVitVlit1llJjYdzblaOwkjzXvxsfp2k02la46JsyvPWhQgIEPNK +ZNJO9AntKyLYRuzHjUYDBoMBTqcTLpeL3j1EVuyjZVYeveaz+jEajXB+fk7zWVVVxGIxqj3e6XTQ +6/XQbDbJuUAELwkxOBAIoNfrUdDZfOmXyXhy6T2o0Whgf38fk8mE5oPdboeiKOSyc3p6CpPJNHN8 +CYdIiJx/J0smk6jX65Tl73K5aE6Ksi69Xg9utxvNZpNKEjyP+eCBZrOJRqMBp9NJ+6z4TLvdRj6f +x8nJCT0bzWYzms3mpe/r9XrkGiLeucT+ORqNcHZ2hoODAxwfH6NcLkOSJLhcLlit1oe29hLZ7z94 +8ADj8Rgmkwlut/thOYWv177YI46Pj2E0GimQS4wLMHPySCaT8Pl8CIVC6PV6j9m/C4eG+/fvo1Kp +0NxRVZUcblKpFDRNQygUQiwWIweASqWCnd0dCv5ptVpQVRVer/dS3yeTCdLpNJWysNpmQrPI2D87 +O8POzg6SySStabFOBeKdamdnB/1+H5Ikwe12PywVMcaTGA6HqNfrqNVqODw8xL1795BOpwHMgnbE +viys/xmGYRjmdcDCN8MwDMMwDMMwr8z8obIQ/abTKWVUCavcTCaDYrGIUqmEarV6KVPKYDCQYJPL +5fD555/jk08+IcEoHA6TLbMQGXO5HIl1z6Lf7yOXy8FgMMDn82FjYwO6rqNaraJWq6HdbpM4JMR1 +m82GaDSKaDSKXq9HmXepVAoGgwHLy8t0+Ps0Mcrn9eH9999HLBbD7373Ozpc3dzcxDvvvAO73Q67 +3Q6Px4NAIEC1bJ+W6WIwGCDJM8Fwe3t7ZsuuaVhcXITBYEA0GqWMXnFAL0QFkan1t7/9DSaTCRaL +Bevr6ySOC+tx0cfj42MS7mOxGCKRCFwuFxqNWYZUu91GOp3GV199hfX1dZhNJjpY/+ijj9DpdODz ++RCLxZBIJBAOhynDulKp4KuvvsJf//pXykJaW1uD1+u91JZSqYRUKoWjoyOYzWYEAgGYTCbY7Q7Y +7TYcHBzgD3/4A4kg4jA4EonAarWSWCSyHEXWqqIo2N7apqCG5eVlAEAkEoGmaRgMBjR+NpvtlTOP +KpUK9vb24PV6YbfbkUgkHlqKe7CwsACv14uJPsHJyQn++te/4vz8nDLjxEG9qqo0HqlUCul0mkTr +eDwOj8cDl8uFarWK27dv409/+hMJMcJCOBgMkmhYLpdxcHCARqPxre0DtVoNd+/exZ///GcKmlhZ +WaE+ifrRlUoFqVQKd+7codq7kiTRfnF4eIiDgwPU63UsLS1hdXWV7OlFZlyj0YDL5aK6xRsbG1hf +X8ef//xn1Ot1AMDa2hree+89uFwu2Gw2uN3uS+vtZbKUR6MRZfQJ2+vFxUUKvslkMri4uMDFxQWS +ySStGZfLRaL2xcUFZfHNM8v8n4lUwpJckiS6htvtfq4gMJ1OMZ7M6qaenJzgL3/5C/b392mfXVhY +QDgchtPpJNv4er2OarWKZrNJe7LFYoHH40EsFoPT6STxRVEUDIdDFItFFAoF9Ho9PHjwAIVCAS6X +Czdu3HipWq/flNFohEKhgLt379LzYXFxkerC53I5svqORqNIJpOUuT0YDLC7u4s//OEPaLVamEwm +8Hq9ZL8tnDIajQYODw/pOfMytWctFgtlZ1qtVjSbTfT7fSwuLuL999+Hz+cjJwuRNdvpdLC7u4vh +cHipb16vF4lEgvqWz+dxcXGBk5MTRCIRxONxelbquo69vT388Y9/xGAwgNlsRiKRoDUoBOBGo4Gz +szM8ePCA+u/z+eD1ep9b5kEEcS0uLuLi4oKyk6vVKsZz4pc+0dHr9dBqtVCr1dBsNtHr9cgRRgjg +0+kUdrsdgUAAbrebnsNCXG80GigUCqhWq6hUKshms5QFu762/o3mka7Psr0zmQw+/fRTfPnllxR0 +JEoxeDweevc4Pz8nIfJRJEmifjgcDjidzkv9aDab5LxQrVaRy+XQ7/cRiUSwsrKCK1eu0L7Z6XRQ +r9cRj8fx7rvvUmCDmDM2mw3T6RS7u7vP7WOn00EqlcJwOITX68XS0hJkWSbR+/z8HBcXF9A0jazJ +hUOFeCe7desWuT6Ikiher5eetWLf6/f7z3w3ehJCVPb7/fRuVq/XaYzFHjYYDFCtVlEsFukaoVCI +LMrnM6BbrRYajQbcbjc9ezXj7BmfTCbxpz/9CZVKBRaLBdFolOzeRTmWdruNs7MzsrV3Op0IhUKP +1R0XbkdHR0fw+/1wuVyIxWL0XCiXy7i4uEAul8PR0RHcbjf6/T7C4fAll5Nut4t0Oo3JZELvCZqm +YTQaYTAYIJ1Ok81+LBajEgWBQAD5fB63b9/G3/72N3LzWFpaQjAYhNfrJeeOcrmMQqGAZDIJRVEQ +DAZh1IywO+zQNA3JZBL/8R//gXw+TwGJ4o8sy2Snns/n8eDBA9TrddhsNsRisafWQAdm7+Gifv2D +Bw9w7949DIdDcvPY3NxELBajgAGGYRiGeR2w8M0wDMMwDMMwzCujT3SMxiN0OjN740qlQlm+y8vL +6PV6JHaNRiOykxRZc+KwbDgaotftIZfL0cFYOBymmpMbGxtYXl6+VOu7WCwil8uRbfiTELavnU4H +S0tL2NzchKZpKBaLyOfzuHfvHonoTqcTXq8Xm5ubWFxchMPhQL/fR6vVwieffELZeLlcDtVqlYTb +J+F0OREIBnDlyhUcHR3hyy+/hKIoWFtbw4cffgifzwefz0cZo6Je97OEI1GbdXV1FUtLSwBA9qd+ +vx8+78yG0yB9ffg7GAyoHvr9+/extraGSCRC47myskLi8L1793B2doaLiwuk02mk02mqM7y2toa9 +vT04HA7UajXk83ncvXuXsn3K5TKOj4+xt7eHQCCAaDSKjY0NxONx+Hw+DAYDDAYDFAoFHB0d4d69 +e1hbW0M4HKa2rK6uUlvu37+PVCqFTCaD8/NzpNPph5mSfUwmY5ycnODOnTuoVCqIRCJkaXzt2jV4 +PB5yBDg8PMRkMqHxlSQJS8tLiCfiNH4AZnUqPV9nDora5S9zeD/PYDBArVajDLq1tTW65yK7qdPp +4OLiAnt7eyiXy9jY2EAikZjVAV1egdVmpfFotVrY3d1FvV6nezMcDqlmZjKZxJ07d8hBYHl5Gevr +69jY2KBsU2Gfm06nn7lmXgXheiDqjN+7d48yq9fX17G6uoqVlRXKKt/f3ycxQPRHCP39fp8O+UW2 +7s9//nMKRBCZcUKoEcEckUgE0WiU7L0HgwESiQR+8YtfIBKJwOfzwWF3QFFfbL09aS8R1vTBYJDq +W4ss9i+++AKFQgH1eh0XFxdkA+9yuaAoCsrlMllXdzodjEYjmpOzYICv67qKABa/349YLPZCGd/z +Ga2np6e4d+8ejo+Pqc7u6uoqrl69ikgkQuN8dHSEo6MjEvqAWRZ7OByGLMtUB91sNkPTNPR6Pezv +78NoNOLg4ICE/Bs3bmAwGMBgMDyW/fptIfpbr9dpz1lcXCSB8ssvvySL8Ww2S/WDjUYjBoMBjo6O +cPfuXapZK0oibG5uUkZxNpulEh5ir31RzCYz1VUulWZWzpVKBbFYDP/f//f/YXFxET6vDza7jQI+ +kskkTCYT1aSu1+tkob+8vEx9++qrr5DJZEg8TaVSMJvN5DxwenqK+/fvw263Y3Nzk8oarK2tYTgc +0j4gaoibTCak02ksLCxAlmU4nc5nClCKolAWfavVorYIAXwynkCSJUz0CQlu9Xqd3ESsViui0SgC +/gBMZhP6vVmpBCH2RSIRBAIBKo1wcXGBw8ND+nN6egqbzYZSqYTxZPyYI8GLIjLSO50OMpkM9vb2 +sLu7S2tmZWUF6+vriMfjFISkKAoKhQJqtRrNB/G3oigkfCqKQsK56Ecmk8Hh4SGSySQODw9xcnIC +k8lEgXGJRALBYBCNRgN3796l7/jJT36CjY0NeL1eCqQRYv3T3kEeXSudTgeNRgMLCwtYXV2F0+mk +LOpWq4WjoyOUSiVkMhkK8rNYLCRy3r9/n/ZYYVW/vLxMmd6iH6IvL4PRaKSgpGazSaJ1v98nRyFh +JS/aK+6fcGEYDoeUqT0r9dCmTHAhfkuSRAEfu7u7GI1G2NzcxNLSElZWVrC6ugpZlukdRwRaiAz9 +TCZDIvo84noieEaUuzAYDDg+PqZnQrFYxOHhIWWqzwvf4hlaq9Vovft8Pui6Tq5JJycn5JyQSqWo +bnwul8Ph4SF2d3exsrKCtbU1ev4vLy9TsNLOzg7S6TRSqRROT09xdnYGq9UK72BW5ieVSmFvbw/t +dhtra2tYXFyk8g6KoqBYLJL9vyibsbS0hPPzc3LKmUcE3TUbTWSys8Cws7MzFItF2O12LCws4MMP +PyQB/3llCxiGYRjmZWDhm2EYhmEYhmGYV2YwHJDFebPZxGg0gsPhoEPswWBAWYKiXmqj0UA+n6cs +X6tlZqeZz+Vxfn6OVqsFSZIQjUbx05/+FGtrawiFQvD5fJTxJgS2i4sLGI3Gpx602mw2bGxs4N13 +3yWhVxwmiwx0YYW+uLiIra0tOtS12+1kI3p+fo6DgwMAX9dBdLvdT7VKliUZiqJCUVQSTIT9tqgh +Layqhfj1ohaPwmZW/Df9LX397+JAuFAoIJPJoN/vkwi7vb2NK1euzITAhxbiNpsNrVYLJycnyOVy +kGWZMjntdjui0SjVl0ylUuj1erh79y5dr9vtIpvNotPpwOFwUP3mYCBImcDFYvFSW0KhENXPnG+L +3W4n8e7i4gKyLNOBsZgzpVJpZtVptWJtbQ0ffPABZekLK+PRaASr1UrCoahB+6TxA74OGHgdgnAw +GMT29jaJ/4lEgqxjFUVBp9PBwcEBqtUqAMDpdGJpaQnvv//+LIjB54OmaTCbzXC73VTnWogByWSS +sp7z+TxlcQcCAbzzzjvY3t6m7NXRaER20UdHRyQgv05LUVGnXIjsmqYhEAhga2sLV69ehc/nI7tg +q9WK4XCI4+NjpNNpGI1GlEqlS2UPBoMB2cO73W7KDBUZ38IFQNh1CxFB2IML0UGsN1pzD+vMy7IM +yfBy4r/IXn///fexuLhIWffiO0RG28XFBQDg7OwMS0tLlDl9eHhIYqbI+LRYLLO112zh4uIC2WwW +AGgOiD69yL0SGem5XA6FQgH9fh9GoxGLi4v44IMPkEgkEIvFZra0D+ul22w2RCIRTKdTrK2tUf31 +7e1txONxOJ1OsqifTqdQFAWhUIgcBE5PT+meCzFqMpm8tnn1LEwmE5aWlnDz5s1ZsMjKCkLBEPTp +zIa4VqshnU6TPX6tVkMmkwHwdWDKeDxGKBTCtWvXcPPmTcpyFYKo2WzGyckJHA4HLBbLSwVKGCQD +CZ7Cmh8A2ajP5qP5Um1uEZBisViwuLiImzdvkmgdCUeob2K9DYdD6tvFxQWJv+12G6qqwufzYW1t +De+88w7ZHk8mOq2lRCKBVCoFq9WKer2Ok5MTyoB9FsJGeWVlhUo0CIvqarV6qVxDtVrF2dkZ2cWL +Pdxms8Fknq1Xk3m2P4syCA6HgzLBp9MpPB4PotEoBoMBstnspYC2TruD0XgEXX954VvM416vh4uL +i0u24z//+c+xvLxMWbNiH202myQSz1vCi9rpovayCCB4Uj+GwyFyuRz1o91uo92a2XqrikoZ9/Ol +IUSgnRC6RbDKi+xhDocDW1tbuHnzJom8VosV+lSn9xsRfCTcBiRJgq7rVL7GYDAgHA7jZz/7GVZX +VxEKheD3+ymord1u4/j4GLlc7pnvZE9C04yU8X12dkZ7ab/fhz7RqVROq9WCLMuw2WxUFkMEqZRK +JVjMFpgtZmqPcM7wer0wm81oNBoolUpkY+/xeLCysoL33nvvYVBaAJJkgMU8c2A4OjrC6ekpgNn+ +nkwmsbi4eMn6W5ZlRKNRvPvuu/S8j0ajdF80TaN3YxF0abfbqczK/D26cuUK3nnnHSwuLmJ5eRlO +hxP6VMdwOKR7JMTkQqFAz8qLiwv0ej1yvnnnnXewublJThhmsxket4dKKDSbTRgMBqRSKUwmEwQC +ATidTlQqFSo9srq6ig8++IBEafFu5vV6US6X6R1MBGksLy8/5pDT7/dRq9UwmUyws7ODBw8ekDvL +wsICrly5MnMysNq+U7cQhmEY5scBP1kYhmEYhmEYhnll+r0+WY82Gg2Mx2OqJ7qwsIDxaIzReIRU +KgWj0Uj25Pl8fmaz+FCQqlarODk9wcXFBdrtNmRZRiwWwy9+8QtsbGyQvaQ4QG61WggEAlQX8GmH +rA6HA9euXcN//+//naxchaVyrVbDzs4OZVyvrq7iP/2n/3RJqBSHqnfu3KHDTmHVrGnapYydeWai +h0Ji93xGmOiz+aHo8WjW2PN40ucfzU4WYnMqlUI2myXL20gkguvXruPK5pVZvWxlJjLoun4pmEBR +FOTzeVitViQSCbLLfeedd2AwGHDnzh0kk0k6mNd1nWpyOxwObG9v48aNGzRm9XqdMriF8B2JRHDt +2jVsbW09tS3pdJoy7ISVrsfjoUNXp9OJjY0N/OM//iNl/sqyTPNkYWEBw+EQsizToeyj4/ak8fum +RCIR/OIXv8BPfvIT+P1+EmllWUa9Xsfx8TFlYAOAy+XC8vIyfvazn0HTNJozov7s8vIyZeP3+32q +2yzEplarhel0imAwiPfffx/vv//+Y2tmMpkgGAxS7fXXKXy3Wi2cn5+j3++j1+vBaDRSYMPNmzcp +w1pkrE6nUywuLiKVSkFVVcqCFgEhonaosN2ORqMwm80kwAaDQbI4F3MQAGXAiZ8L8fxJ6+1l1hww +E1rX19fx3/7bf0MoFILb7b6UoSYELEmSkM/PgngCgQA8Hg+J7/1+H81mE5VKBZVKBQaDYWaD3WpS +/WNgJnz7/X44HA4YjcYXmp+iBvnR0RGtF5PJhNXVVfzmN78hAWS+pnIsFqOazCbzLBAnGAzC7XZj +MplQVrxw7ZBlGeFwGCaTCaenp5BlGZ1Oh4Tv+b3u20b07b/+1/+KeDxOTgqCYrGIo6MjdDodGAwG +VKtVygYV9e7H4zHcbjdu3LiBX//613SfgNkeajFbcP/+fTidThKwX5T5+aeqKo2L+JnRaIT54byc +jCcYyAOax5qmYXl5Gf/lv/wXyvwU9bkBoFqt4ujoiGq312o1yLJMWffiXvj9fly5cgXvvvsu2dXP +1qCHAgfOzs4wGAzQaDRwenqKcDhM9dSfhizL9CwVAWHCmrtSqcDj8dD+Mi98i8AKj8cDu91O61FR +FCqLIcbNYDCg1+2h25vVrjYYDCSgzbsbtNqtS3vAyyCE73w+j0wmg06nQ0FwH374IRKJBN0/sWYq +lQr8fj/Oz8/p3UOsTyF8h0IhAKD1MN8PMbd2d3eh6zoJ+O1Om/a/eav5+fki/l3XdQqUeJF+u1wu +vPPOO/Qe5Pf7KaNfZHsnk0ly5ikWi+Rwkc1maV+LxWL44IMPKEhm/vki3Ame9072JIxGDW63G6FQ +iALDhC3+eDKrVV4qlUj4ttvt6Pf7VJ6h3W5TMKIkSzSm3W6XhG+LxYJSqYRCoYBGowFJkuDxeLC2 +toaf/vSn9IwyGAzQPTo8Xg+Wl5dxenpKZXGOj48pmE6gKAoWFhbwy1/+Ejdu3IDL5YbT+fV7oagL +PxgMkM/nUSgUYDQaH6tLbrfbcf36dfzP//k/4fV64fV66Z1lOBgimUxSwNt4PEapVCIHEvHupSgK +YrEY3n//fWxvb9O7hM/nozWdy+Uo0CyVStHzyOPxoFqtYjqdwuVyYWNjA7/68FdQVIVqgIdCISof +lM1maSx3d3fJOUMgRPlarYZqtYq7d+/i7t27uH79Ot577z1cu3YN6+vrCIVCkCUZkvzy65dhGIZh +ngUL3wzDMAzDMAzDvDLdXpdqQw8GA9jtdrjdbjidzlnGl1GnDBJR73I8HiObzVJGo9vtpowdke0t +6vF6vV543B5I8uXMJmG9K7Klnya2iGzmQCAAu91OGbfiwF0IpbIsw2KxUNtFDU1gdvAsrjeZTOiw +ejQaPdVelQS5uazS+Z9R1ukrHNaL73ruvel2UalUUC6X0Wg0qM7n5198jmwu+9jns9ks2XKKjGlh +ySwOT9fW1ijbUJIkVKtV7O3tUabt6uoqFhcXEQqFZna5ivrstnz+OYl98+RyOZycnFBbhNWpqMc6 +GAxgNBqpTrrf74ciK0+cJ/O15F9m/L4JIrNRCKROp5P+rdlsUt14YWMuLKT//Oc/P7Ftd+/eRSaT +uTQe8zbZAGC1Wmd2/R4vZSLPf5fJZHqhNfMqiEASYafb7XZxfn6OL7/8ErVa7bHPC4G2VqvBaDTS +obxY9+K+iezKv/zlL3C5XLNMPKsFVqsVFouF7J0VRcFoNCKBbP6PJEm05l51vQGgPcLn810S7gQu +lwuBQACFQgGFQoGy/ETWvsvlgsViwXQ6RalUwvn5ORRFgdfjJbvofD5PQkowGITNZnvhNo/HM4Go +WCySJbPY0/x+P5xO52PC9Pz6EHS7XVSrVdTrdaoN2+12qcyEyPDMZDLodruUgTgczLLIv6vMPUmS +YLVaqYSAyMoWiFrIYq8XQQeifvdkMoHFYoHL5ZrV9/V4YZAuC6imufn1KmtGzL8nBSlJkgRJlkj0 +eXTeWq1Wyvx3Op2X+iZs9EUwyHzfdF2ntZjL5XDv3j3ouv5Y21qtFvb391Eul0lINZvNZC/9LMQz +0ePxwO12w26f1Qju9/vIZrMUKGA0GslWvdVqkeuFx+OBUftaHBVOMNVqFY1Gg7J15+ecCHwql8u0 +1kUW9qsGXOj6LOO5WCxSwJ3dbofL5YLP54Pb5X7smSKE6Sc9vyeTCRqNBmq12qV+iNIbYu3UarXH ++jEajkhgfNp8kSX50v79on0WQXrBYJDqpwvhW9dnDgBWq5XmgQgAAEDjIso2PGtcXvX5Mm91LvbI +4XBI1t/CLabRaFBwjrCet1qt6PV65DgyGc/aLwIDTCYTzcd+v49isUjro1gsYnd394l28aPRCDs7 +OygUCjMB/mGgVbfbvRQYIgIePB4PPRfmA3DEmAWDQVQqFap5/2iNeFVVYbfbEQwE4XA64HA4yFFo +oA7onbTf70PXZ1nwYu20WrPgD7vdDqfTCY/HQ+8c8/fB4XRQ4IOwgheuEbo+yywX71Uulwtuj/ux +71AUhe6VCC4Q7+7z4yKeczs7OzAYDBTo5XQ6yX1JONswDMMwzLcBC98MwzAMwzAMw7wyvV6PshvH +4zHVOLTb7WQnLIRsIVAAQD6fh8/nQywWw3Q6Rbc7qxEu7BqFFaqmat8oE+TRQ2PxM0VW6JD2UWH6 +SQe2wlJZfE6IsN9VPduXZapP0e/36RC+Xq+jVqvh/v37KJVKlw5mBe12myyYvV4vZeyIw0yPxwNN +09BoNJBMJnF0dIRer4ejoyM6zBR1Jd1uN1RFpXv3aFsajQZ2dnZQqVSe2JZOp4NyuXypLeJAXYy9 +1WolMVFkVD7p3n1XGajzCHvaeYtjgQgoqNVqNB6DwQAfffQR2ao+iqidOp1O4fV6KStOVVUSG4UA +pWrqS4kir4PBYAAAl0SfBw8eoF6vP9EVodfroVwuo1KpwOVyQdd1OBwO2O12JBIJpNNpTCYTVCoV +fPnll6hUKmTbGggEEA6HEQqFKDPuuxJbnzWuQmCx2+0kRo7HYxiNRjgcDng8HsqSLZfLSKVScLlc +VAdZZOKJsgKRSOSlap5OJhMSisbjMdmoC5HraRn+8xbzwCxTen9/H8fHx8hkMshkMlQTXIhJoq5s +q9WC0WicBaSMvw7I+K6YDyB69L5IkgRN00hYGY1G6Ha7tH8LK2phJ/99yzh8kmguEM8jIZSKmuAG +gwGj0Qj1eh31eh3D4RCtVgu3b99+7DtGoxFKpRIqlQosFgsmkwncbvcsoOs5tuFC7BM1rT0eDxwO +B4bDIS4uLkh8E8FR5+fn0HWdnvk+nw+yMpsn4jmTTqWxf7BPLiXFYpGEYtHn0WiEi4sLDIdDco8Y +j8eQDNIrnW6Kdw+xBwvx1mazQXn4/HqRfVRkPYv2PXjwgPpRKBQe68dkMkE6naaM5clkgtF4hMnk +23unmA8CevTnwpK/2+3S/eh0OphOZ+8R8+9kqqq+9rWiaRpcLhf6/T4FJwpBt1QqIZvN4uLiAs1m +ExaLBbFYDN1uF71eDzabjYRvEYwoHFBEaY35wIxqtXrpPaTf72Nvb++xNum6Ts9dRVEoA1vYqz9t +jB9FUVQKCBVW8uLZ8MR79JQ5J0okCNcGIViPx2MMBgNomkbW+vPvtY+OswgWFMEko9GIgjmm0yms +VivtiU/rkwgYstlsVHO91+tdKnMhbNXv3LkDo9FI+0s4HEYikUAkErlkGc8wDMMwrxsWvhmGYRiG +YRiGeWW63S4KhQLVCtQ0Daqqot1uU61bAJSNJjLKstksPB4PHVAOh0O0220Mh0MSpM1mMxT1m1vn +zotVZIGsyJey+J4legNfCyziwPP7LHoLxOG1OJQcDodkKf0koVCICNPpFAaDgWqQi2w6u90Ou92O +YrFIGaT5fB6lUgnj8RjXrl3DtWvXEI/HZ4EPD4UNkaXd6XQo+204HKJSqZA955PaIrLrhciiqipZ +5eu6DrPZTLbd3yST99tAzBdZflz4FlburVaLsgBFttSTsqOB2SHyZDKhzHpxKC0O0EWm4nwQwHeJ +yFycn2vVahW9Xo8CRh4dA9Eng8FAYpPf78fCwgI5NFQqFRQKBdTr9Yc1WP0Ih8MUFJFIJC7ZHL+J +IAeBqqqUiS4ESGG9LmqjBoNBaJqGSqWCVCqFWCxGdrXFYhH1eh3Ly8tYXl5GKBR6KeFbOFGIuWQ0 +Gi/Vc3/WGhHZfsPhENlsFru7u9jd3UU2m0U+n6f7C4DslhuNBtWtFcEo3/W++CxxWAj0Yv8SAtG8 +A4TIQBYC8veFRx0LniTqi8CaR/smbOnF/RQWy48iBFcRJCHqSKuqCoNkeG77RECBy+WC3+8ne/xM +JgOPx4N4PA6j0YRarYZ8Pk8Zw4uLi/B6vLSfj0Yj1Go1nJ6d4s6dOzg9PUUul6N6w7qukzArSRI6 +nQ49p8QzS4iBL8u8yCvmsqqqs3cPRX7hOSFqdddqdZyenuLu3bs4OTlBLpe7lFEv9m5RIkCsqa+f +d5MXut6rIBwvnnRvRSCFeMcZjUbo9Xq0L4h3MiFKv+61omkaWY/Pgg4UKjNTLBYpuHI4HMJsNsPp +dJILxXQ6Ra/Xw/n5OQXICWt2s9lM2eyiX/PP3eFwSAEgT0I4QwjBX5SKeJkAO0WR6V1FlmWMx+NL +++mT7tGjwQmP3iMAJPLPux4IF5KnBYIJ1xKHw4FisYjhcEhBhCLQQYzZ0/ZEySCRgG42myn7fDAY +PNandrt9KQhLCPMiK144Ar0KItjkuw7yYxiGYd4eWPhmGIZhGIZhGOaVENlSwrZXHGI3Gg2Uy2V8 +8cUX9NlCoYDj42PkcjmYzWZYLBZ4vV6qc/joIf93dZj1otd43dbQ3xVCNLDb7fD5fEgkElhcXITL +5Xrss0IIETUeRZ32SCTy2Jg9arU6nU4xHo+/toB/JGNQHFKKw1mfz4d4PI6lpaVntkXXdbKJnkwm +KBaLKBaL1I638Z6I/um6DpPJRGOdSCSwvLz8xD4JYcRoNFKGpagTXa/XvxdBGEJ0sNls8Hq9iEaj +WFpagtfrfWr/J5MJnE4nWaeK+rjXr1+Hrus4OztDp9OhOs2DwQDpdBqZTAa3b9/G5uYmbty4gZWV +FbKafZMI4VcIGEJw1rRZbePFxUXKxgWAxcVFlMtlqiE7Go1gs9mQSCQQDAafaMH7vHsgeJl9dL7+ +7P379/HgwQPkcjk4HA5Eo1ESj4RoJ8sy7t27h6+++gqj0eiNjvnzmO+/mHNinL6p/f33CbHHCgt4 +sR5EduWTPj/vnuF2uxEMBrG8vEx1zl8Em82GUChEtcFzuRzcbjdWVlZgNBrRaDRQr9fJdn9paQle +nxeyLKPVbCFfyCOVSmF3dxf7+/uYTCaIxWLY3NyExTIrayAEv06ng88//xzdbve1jdsl6/mXtBAX +tNttnJ+fo16vUz+GwyEikQiuXLkCi8UCi8VCAnK/38etW7deaz9e1zgIRFCBsMl/Whbx60KWZSiy +QtnEqqpiMBigUCggm80ik8lQCZFoNIput0vOKeVyGYVCAbIsQ9M09Ho9mEwm+Hw+2O12KIpCLgEA +yKbfZDIhkUggkUg8sU3iHcRsNsPlcsHr9SIWiz3RpeZZzIu0JG6/wr7zrPH/rt+FRJ+edm1hax4K +hSgYtlgs4vz8HMlkku6PcIF6GURAxnywyg9lH2cYhmFeHyx8MwzDMAzDMAzz0ohDr16vR1aUIsMs +k8kgmUxeyjoZj8f076JedigUooxv4PIBtDjA/654WwXUZzEvNgu76Rs3buDv/u7vsLCw8NTfAWYZ +WEajERaL5VJ96vnxEgfhBoOBhO/hcFbnV59erikrBEHRFl3XqS3xePyF2pLP5/Hll1+iWCy+1fdL +3BchfIt6nB988AE+/PDDJ/ZtXqgzGo3QNA337t3D3bt3Ua1Wv5XMp/mD7Rf9rLi/wgHg7/7u77Cy +svLU3wFA2Zwi00xkxkUiEVxcXOD09BSpVArFYpFEYiEeVyqVS3vFk4Iovium+mUhYN6CW9NU+P1+ +LC8vY39/H9lsFq1Wiyyd6/U6ZcfZbDYsLy+TMPMqzDtcvAjD4RD5fB77+/t48OABDg8P0Wq1EI1G +8fOf/5zEU5HVOx6PYTKZkEqlUCgU3tiYv+g40D2aE3vnAxTe1v3kSf0T9cF9Ph/W19fxi1/8Atev +X3/i74v5qigKrUGH3QGj9uLCt9VqJReGVCqFXC4Hp9OJcrkMm82Ger2OZrOJYDAIr9eLpaWlmRgp +K2i2mkilUtjZ2cGDBw9wdHSEYDCIq1ev4p133kE0GkU0GoWqqpiMJ8jmZuvm4ODgtY7h8zLsn0en +00E6ncZ0OqV++Hw+bG1t4b333kMkEkEsFoOmaZiMJyiWimi1Wjg8PHxt/XjdPOq6Isbq20CMvaIq +sNlstM8Mh0MUCgXk83nkcjl4vV6YzWYsLi5SQJSu6zg/P0c2m6U5LOy7hTONEL6FiG+xWODxeBCN +RvHBBx/gZz/72RPbJdaHLMuzmvWaEQbp5Z6z4tk4L3x/kyCC+d97VZeDR3/v0Z89713iRa4rSRJc +LhdWVlag6zoODg5QLpdxfn6Oo6MjOBwOqIr6SsK3mBv9fv/SPs4wDMMw87DwzTAMwzAMwzDMSyEE +7263i2q1imazicFgAIfDAafTSRaJj2YPiYPUZrOJZrOJRqOBarWKcrlMNQoVRSGbTXGo9by2fJ95 +9JB0Op1Cn343dsCapsFiscBkMpH1stFopMxa4OkH2UK0E3a60+kU7XabahFXKhV0Oh36PpvNhk6n +g52dHZhMJkSjUTidTrLl1DQNZrOZhDxhn/oybRmPx3C5XLBarWSpLezxxeH82yBgybIMk8lEopCw +B7Xb7VhYWHim8CIZJEiyRIfKwnp2Mpmg0+mg1+s9sXao4NF5Jw7i5y1UqebsQzvWpx2SCxEBmInX +wtLcYDBgOBxCVWeH2gsLC8+8N+L+imw9VVWpnrywDne5XFQTPJvN4ujoCJ1OB/1+H4VCAZlMhjLL +nyQGPqnvr5vBcGZZ3263YTAYYLFYYDabSTTx+/1YWVkh6/BWq4XT01Pcu3cPxWKR6rSL2qnPsqx9 +4tx4GBRhsVhQq9UwGAzQbrfR7/dJ6H3aPRBOHaJuN/C1oLm2toZ4PI5gMAi73U4BTA6H4zurrf66 +EHbBYo53u12qXf5t8Wjm/bwQ9rr7JvZXUeddlmVy7niRNSjqor9MDWer1YpIJIJ6vY5cLodms4ls +Novj42MaY5PJRO8H4h1BkiX0+31UKhWUSiUMBgOqGR6LxXDlyhUEAgEEAgFIkjRzlBmPYLFYXpvI +ZTAYLtmPC9vo+ezgF2EwGNDv9vt9qh//aD/ENfTpLMt+/n48KsB/2/PleYhgiPmAtvlSAd8Gwq3A +6/Wi0+mg2+1ScFCj0YDD4YDFYkEoFEKv10Ov10O1WsV4PKZa9WazeSZSP9xz7XY7Pd9EzW+xPkRW +8vPWh3juCtcRUa/9RRBlXsSz3mQyUfb/N0GUGzCZTJds3J9WPxyYBaB2u100m02Mx+NLFvZWq5We +F8K6/EnPDH06y7YW72DiWWc0GukdQrTP6XRiaWmJAg+GwyEmkwkODg7ovSMYCtL7w4uu61qtRgEm +Ho8HXq+XnHNELXXOAmcYhmHerv+XwjAMwzAMwzDMG2c6naLVaqFcLqNUKqHdbmM6nWJhYQHvvfce +fD7fU20zO50Obt++jZ2dHRLOc7kc+v0+iV6DwQCNRgO9Xg+T8eSZ7fg+8zRL8O/qINtknIkNIgOq +0WhgMBjQAfDz6v5SFtZDcatWqyGTyVDt0nq9DrfbjWg0CkVR0Gq18Pnnn8PpdOLKlSsIBoOYTqdU +O9bpdD7WFkmSXrgtok6m0+lEoVBAp9NBo9GgWqnzh9PzPMuO802gKApZC+dyOXS73VlG40Mrc0VR +npsRJg6brVYrjEYjRqMR6vU61b99Fo9mdgmBY752q7CtN5vNTxS+H63nLERqUW9YrF9RZ/VF55os +yZcs8VVVhcViQTQaJaEjlUrBZrNhPB7DbDaTYLu0tHTpu0RbJ5PJYw4E3wazGr81NBoNALNABhGY +oGka/H4/JEnC3t4eptMpGo0GTk9PoWka2u02uSu43W4KAnqZOStJEn1HPp9Ht9ulPXc8GkOf6E9d +HyJwQohIon5rNBrF0uLMmno++/xNCHGvA1mWqS6trusUzPNt2bVLBglTw/QxMVMElLzOMdQ0jSyY +xTNaWGmbTKYXyjQ1GAxQ5Jc7JrTb7YhGo+h0Orh37x46nQ4KhQKSyST6/T663S7cbjd8Ph8cDsdM +IJNmAtlgMAsWaTQakGUZHo8H4XAY8XgcCwsLs1rbskLr93XPOYPBAKPRSOt0MBhQmyeT5+8Zoj2P +it5ut/tSPywWC1RFvbQPPakvBoNE7w2PXkcEzX0XGAwG2nuF0CsCq77NIBFJkmCz2eDz+aDrOrrd +LlqtFqrVKtrtNiaTCcxmMwKBAAmp6XSaAiorlQqVD7FarXC5XHA4HBTwYrPZ4HA4YDAYLmWMv+j6 +AIDJeIIBXk74Fm0T9a5FDe1vitFohN1upwzo+fehJ733CHG8UqlQVrwYL6fTiWKxSOux3+8DwBPF +716vR+U5xD0T73Hzc8jhcGBxcRFOp5PWWqPRQDKZRLfbhdfrxfr6Omw220tlbRcKBXz00Uf44x// +SG4Ki4uLWF1dpe8TgQEMwzDMjxcWvhmGYRiGYRiGeSnEIWM2m0W5XKbszoWFBfziF7/A0tLSY8L3 +/MFcvV5HMpnEcDhEtVpFNpvFYDCAzWaD2Wym76/X66jVa3QoLssyxuMxJpMJ6vU6ut0uZZB830QY +g8EwE2Lnav0Cs4PHdrsNh8NBGW5inOatw7/x9SUDLNZZHXW73Y7pdIpOp4NWq4Vms0lZeI+KWWJ8 +5621RVZQuVxGMpnE6ekp1Sj2er24cuUKhsMh7t27R7bU2WwWwWBwZmf58ADd4/GQ5bo40G42m+h0 +OpeywZ/WlslkAovFAp/Ph7OzM/T7fTSbTXINMJlMlHUkhAJRB1IySDBbzC9dM/lleZF5KOzAA4EA +Tk5OKCNMOCGILH2RQS36Ph6PL4llkiTB4XDAZrNdXjMPrcBlWYaiKDSOjUaD1oyo3SoEDnE9kZnV +6/VQqVRIjBbXH4/GdLg+GAxIZDeZTHC73TSfxf0VnxVz7dGa8I+u3UfLHiiKAr/fj1AoRJ/x+/0o +FAo4PT2lDDzhOiFJEhRFoYATMbbdbheDwQDD4fCV15uYT91uF71ej+6PEI6r1SoKhQKq1SokSYLX +64XT6YTRaISqqPB6vbBarJSRJupqC3tzm82GhYUFynZ/2Ww1RVFINEqlUiTE1Wo1lMolAIBm1Cjj +WWQJDodDlMtl2lN1XScR1eFwwOGciZW6rpNY1GnP9pLnBVl839A0DXa7ndwK2u02Go0GBSyI54yY +n83m12vGYDBccjl4HgaDAQbJABnyJVFtPB6TY8pgMMDQNKQ1+U2eZZqmkcuGqCPdbrfRbDbRarVh +sZgfE/aEs8P8ujMajTDKL251bjabEQwE0e12YbFYaK85OztDr9fDcDiEz+dDIBCAw+G4JEZNJhP0 ++330+/1LwU0OhwN2ux0AMBqPyN2jUqnQHH0diGuKIKThcEj7aL1eg81mveQ4IvbZ+XcP0Q+x3oCZ +lbboh3jmDUdDEvrL5TK63e4T7/W8A4a4brfbJXeC4XCI0XD0VEeO1zkuTqeTMoiFS4+wrhd7/WQy +wWQyoQzgb9Iug8EAm80Gv9+PTqdDLh9ir5FlmWrFT8YTjCdjOBwOSJJEAZNi3NxuNwKBAJxO58MM +4Nn68Pl80DSNHDHEe4jFbIHJbLq07z763J1OpxiNR489s+bvkaIoUFWVHI7q9ToKhQJyuRzG41l7 +hWX767hHHo+HxrvVaqFer6NaraLRaFAAlQg8rdfrFLAqRHO32w2/3w+v14uzszN6ntbrddSqNSiq +Qi5Ouq5fcmlot9vQNI3W66MOIFarFaFQCNFolDLLP//8cxwfH2M4HOL09BTpdBqhUIjG7UVoNBrY +3d3FJ598QsJ3qVSiwBOR/W2z2qAZtdk7rCQ/MfCLYRiG+eHCwjfDMAzDMAzDMC+FruuoVqtIpVKo +VqtQVRWBQAChUAiBQOBhxrcESbp8wC6ES1Hr02AwoNls4uzsDA6HA+FwGK1WC5Ikod1u4/T0FJ9+ ++inK5TIcDgesVivV9t3Z2UEymaR/+z4KMJL8tX2zyWTCcDhEpVLB/v4+WWUKu2CRhW0xWyAr8je8 +8uxQ1OVyY3FxEeVyGQ8ePAAAZDIZfPTRR6hUKohEIohGojBIswwzkcXTaDRIELTb7QiHw/B6vUil +Urh9+zYymQxsNhtu3ryJjY0NbGxsoFarIZ/P4/z8HLVaDffv34ckSVhbW3uYpe3C4uIiKpXKpbZ8 +/PHHqFQqiEajiEVj1BYh2NXr9UvipCRJWF9fx9nZGVRVRbPZxMHBAX7/+98jEokgEonAYrFQNlg+ +n0ehUIDZbMbm5ia2trbe9LSAyWSibPiTkxMYjUZ0Oh3s7+/jX//1X+kg1+lw0kGtmPfCWlQcQnu9 +XoTDYezv76Pf7+P8/BxffPEFer0eZXE1m03UajUcHR1hf38fpVIJJpOJLIRFtrbVaoXJZEK73UYm +k8Fnn32GWCyGaDQKo9GISqWCarWKL774AkdHR2g2m1BVFW63m2za2+029vf3YTAYUCgU8Nlnn6HV +alGfhCg9GAzo/or5Cnyd2SiyHsXBvsvlBgDo+gSlUgm1Wg2tVgs2m42s0C0WC2X2iSzzer2Ow8ND +Ei+E44FYbyaT6YWzwobDIYrFIvb29lCv1xEIBGA2m0lYv3//PnZ3d1EsFhEIBHDlyhUkEolZNps8 +E+RN5ll2nc/nIyEkn88jHo/D5/MhHo/D4/G80uG8qqpkp55Op6FpGt33P/3pT1Sn2+l0UkBIqVRC +Pp9HqVRCs9kk+30hYqXTaRwfH8Nut1NAQyqVwvn5Oe7cuUP3721B2LdLkoRkMoler4dsNov79+9f +suPu9Xqo1+tIp9PY399HPp+Hy+WC3+9/qesZDAZAAj0DVFVFq9XC8fHxJSFJCOEiS/9V+xaJRGC1 +WpFMJiFJEsrlMr766ivouk5rUASg6JPZc7zemFkwGwwGmEwmxKIxxBZiMBpfTPyWZRkms4lqM3s8 +HhIte70eAoEAIpEIwuEwrFbrpd8V4pvD4aDSJ5lMBmdnZwgGgxTAUiwWkU6nkUwmSTR7HYgsc0VR +UCwWIUkSms0mjo+P8fHHHyMej8PhcMBkMlGAxN27d3F8fIxyuUzZtiLgSlEU+lwmk0EqlcLR0RH1 +o1Qq0Zo6Ojp6zDJbkr4O+BGBXN1uF2dnZ2Q7rus6lYL5ttwKJEmC0+lEIpGArus0LicnJ/j8889R +Lpdpz63X62g0Grh79y5OTk4ujcurXFcI34VCAa1WC7lcjrKSvV4vLBYLgFlwnwIFZrMZLpcLLpcL +0+kUlUoFZrMZKysrCIfDlPFttVoQiUQwGo1wfHwMVVVRr9dx//59aJqGaDSKSCQCm9VGVv+1Wg21 +Wg2j0YieSS6nC06Xk9o8nU5Rr9dxfHxMdcXFGuh2u3jw4AH29vZwdHREpV1WVlbg8Xi+0T2avePN +3qtUVcWDBw8wGo2QyWTwxRdfoNPp0Pt4u91Gq9nC7u4ujo6OcHFxgZWVFXIkCAQCcLlcODk5gcFg +QKPRwMHBAf78lz8jGAySTb8Q1g8ODihob21tDevr64jFYk8V861WKxLxBBwOB7k8DYdDnJyc4N// +/d9x8+ZNvP/+T2CxWF4oGG0+cKvVaiGTyaDf76NUKuHBgwfwer3w+XwIh8NYWFhALBYjBxbOAmcY +hvnxwMI3wzAMwzAMwzAvha7rqNfrOD8/R7VahaZpCAaDJHwHAgEAly0WRYaM3WZHKBT6+jCu1UI6 +ncbm5ibC4TBl7HQ6HaTTaXz55ZeoVCokvl5cXCCbzSKZTOLk5ASVSgV+v/97KXyLg1JRd3c0GqFa +rSKZTAKYHfqLbHlVVeFyuWaZKa9J+PZ43DCbTCgWi3A4HJhOp8hms/jss89QKpWwvb2N6XRK2WXN +ZhPn5+fIZDKUNSUEWovFglQqRVa2W1tbuHbtGjY2NrC+vk4CkThM3tvbo4zk9fV1eDxuWCxmlEol +yoDL5XLUls3NTUynUxIlm80mLi4ucHFxQW3xer1YXl7GysoKvvzyS/qcEDY3NzcxHA7hdrvp0Hl/ +fx8HBwdks/59EL6NRiNCoRCcTidu375NonIymUSz2cTm5ia63S4ikQj1/eLiAufn52i1WvSzRCKB +paUlxGIxaJpGIt5XX32FVquFcDiMWCyGXC6Hi4sLnJycIJlMolgskvgp5qc4FBbW5vl8Hl9++SWq +1So6nQ5UVUUqlUI6ncbBwQFSqRQmkwmcTiem0ymJJN1uF06nE5IkoVgs4osvvkC5XMbW1ham0ym5 +B3Q6HZyfn+Pi4gIAyBFBWFCLOelwOBCPx5FIJADMAmiKxSKq1SparRbMZjPZxovav6KevJgfR0dH +lMkqbNRVVaVMwBdF1JE9PDxEr9fDYDCA2WxGtVpFtVrF3t4e/dvCwgK2t7exuLhIorHIqHe5XCQ0 +VCoV1Go12hOXlpbg8XheqTapsFM3mUzY3d2lOXF6eorpdEpCZDgcpvVxdHSEw8ND1Go1BAIB+P1+ +yLKMdruNwWBA88Zms1Egwb1793D//n1cXFyg0Whccmr4vmOz2RCJRGiO9Pt95PN57OzsQNd1hMNh +hMNh1Ot1ZLNZEi5LpRIFbLwMQsQRwRiqqqLdbuPk5IRq3E71KfqDPllsfxPhe2FhAS6XCx6PB7Is +o1ar4d69e6hWq9ja2iIbe1EzO51OI5PJzFwxHu7XBoMBgWDghYVvEbjlcDjg8Xjg9/vR7/dRrVZR +q9Xg9XoRjUYRjUZht9kv/a6qqrDb7bDb7RiPx6jVasjlckin01QaQJZlHB0d4f79+9jf30cul3vp ++/Cstnu9Xsp2NRgMaLVaSKVS+Pzzz1EoFGivPj8/RzabxeHhIc7OzlCtVhEIBDCZTOgZrmkaJpMJ +arUaTCYTUqkUiYayLOPk5IT6kc1mKUN8fr7M72GappHwLfYP4coh3Cy+jXcfkTW7uLiITqcDSZLo +Xe327duo1WoIh8PweDzIZrPIZrPY399HKpVCrVZ75XcyEWwXCARgsVjQ6XRQKpUQCoUQDAbh9/sp +eEKSZmUERMa+x+Oh4CxRJiMUCsHpcFKNb7H2v/jiC2iahlarhb29PdRqNWxvb2N7exs+n49cHy4u +LpBOp9Hv9+m+rKysYNW4Sm0WWd2np6dQVRWdToeyoqvVKh48eIAHDx7g7OwMHo+HLLndbvc3ukeS +JMHlcmFpaYmCxEajEXK5HL766it0Oh1sbGxgOp2iUCigUChgf38fx8fHyOVyWF5exuLiIra3t+Hx +eGC323Hr1i16biaTSciyjLW1NWxsbEBRFMpcPzw8RDqdhtVqhcPhwPXr1+FyuZ76LLBarfB4PFiS +lpBKpZBMJnF+fo50Oo18Pg9N07C2toZAwP9CdvOi3IrIYheBNuIe+Hw+CgJ75513oGkadF2H0Whk +4ZthGOZHBAvfDMMwDMMwDMO8FOKg7/z8HPV6nbJHfT4fLJZnZ1SYzCaqfynq0KbTaaqDGYlEsLKy +gkKhAKPRSJ/JZrOw2+1U51ccRIosuu+rfaHBYEAgEMDKygqy2SxGoxFOTk7Q7XZRLBbpQDcYDEKW +ZKrR+jrQVA2STUIoFMKVK1dQLBbJujOTyZCIKOzYhcVlq9WCx+OB2+2mDKJ2u41sNoterwdN0xAO +h3H16lVEo1H4/X4Mh0MsLy9jY2MDw+EQ2WwWqqoiFothc3MTqqqS4LuxsYFCoUDWsaItlUqF2iJs +8VutFtxuN9xuN0wmE+x2OzweD5aXl3H9+nVcXFxAkiQUCgVyEDCbzWRrLSw/rVbrt1qb9GUQ2Xyq +qiKRSODq1auwWCx0iHt8fHxJQBbZds1mEwaDgcZDCL66rmNpaQmrq6swmUwol8sYjUYoFAq4uLig +bNJ5O+F5K20hLgUCAWxsbNBaEllU1WoVJpOJxNLpdAqTyURCtshQFLVL19fXkclkMBgMMJlMkMvl +MJ1OUa1WKeNb1D1tNptU11qUOdB1Ha1WC61WC9lsFvl8Hslkkg67RWaocI9YWlrC0tISvF4vJMPX +ARJCeBHCRblcpvUmrNMtFssLW84KC+eLiwu0220Kymi1Wmi32yiXyyTCJBIJrK6uIhQK0fcLEdTh +cNCaFy4Lom5tLBaD0+m6VCv1ZeaV1TqzZl5aWsK1a9dgNBohSRIF3MyXWRgOh2SLK0SU5eVl9Ho9 +5HI5ygb/7LPPSLQVlu4iU/5V2vkmEQEPovbs6uoqptMpms0mHjx4gHw+j7OzM4xGI8r6Ho1GJGzO +2+i/DG63G0tLSxTYJcp7VKtVEvMsFguazeYrZ/CKfcXr9WJlZQU3btxAvV6/tM83Gg2aE7quXyqv +4Ha7KXP5Zfo3L+57PB5Eo1GqnT4cDmGxWLCwsDDL+LZdzvi2WmeBCK1WC8VikUT4o6MjKicgsuQH +g8GsVraqvlJgyNPaLpwfxLuHsPG+uLhAs9lEJpOB1WqlEgeipILRaCS7b7PZDJ/PR5njoh8nJyeX +gtva7TZ6vR6tp6etHzE/8/k8ZFlGoVCg59np6SnsdjtsNhsajcZry35/dFzEXibGpVwuQ9M0ZDIZ +dDodZDIZ2O12cirodDr0TBB9e9m1It6BhEW5eC8RLinzwvf8/fN6vQiFQuj1eqhWqzAYDLBarfD7 +/bDZbZSNL0pyLC0t4fr16/TuUa1WcXh4iFarBavVSvNLrA/hbGKz2S6VpwFmImy320U+n4fBYEA+ +n4fFYqE1UC6XYTAYEAwGKdtbvO9+U0RJCvFeVSwWoWkaZWULxw/xPBX3MB6PY3FxkVxAHI5ZOYvF +xUVsbW2hUCgAAE5PT9HtdpHL5ciJSVj9izFfXFxEPB5/plW5oihUzmZxcRHXrl2Doij0fD89PaVg +SZ/PB5/P90L9v1TO5uFcE+8XohyLWDfie71e78PgIC+sVgssFguL4QzDMD9QWPhmGIZhGIZhGOal +EBnfImMpHo8jHo/D6/VC056dQSkElmg0il6vh0KhgHK5jFarBUVREI1GcfPmTQCzjGBhwytq9zqd +TsqMFnaaVqv1sdqC3xckSaI+mc1mnJ2dUZbL3t4eIpEItre36aDWN3qxA78XurYsQZVm9sfvvvsu +zGYzTk9PqUZ3pVLBvXv3SLiQZXlW39VohMPhICEkk8lgd3cX2WwWsizD6/ViYWEBGxsbsNvsMJvN +lNnd7/cpo20ymWBtbQ3ZbJbEWp/v8bbUajVUq1Xcv3//sbaI+pEulwvBYJAOn0WG+NHREY3pyckJ +jo+PL9XBFrWThV3t9wVZliEZJCwvL+OXv/wlQqEQTk9PcXZ2hmw2i/Pzc7L8FqKSpmnwer0IBAII +h8Pw+/1wOJxQFAVbW1skRuRyOZTLZVxcXNC9FNnNInDAbreTaC3q0cdiMbz33nswmUxUq71eryOV +SlHG1sx23IXRaARd12Gz2eiwWZZluN1u3Lx5EyaTCcfHx+TKIFwAAND91TSNMhsdDgd8Ph9lfJ+c +nMxqU5dKVCbh0brzPp8PiUQCW1tbuHLlyixTWp65FFy7dg2SJFGWeqFQwMHBAXw+H7a3t3Ht2jUS +TF6UyWSCdrtNWW+iLrMI4BBlAeLxODY3N7G8vAyXy/WYsG61WhEMBhGJRKjmqij1IKyqXyWQZz4A +YWVlBcPhEKFQiMYglUrh7Ozs0vqwWCyw2WwIBoM0lqJEgLCOFRl5JpOJgi2WlpYwGAxQKpVeW73l +7wIhfol50O12cX5+jmKxiJOTExLIrFYrZUaLZ43NZiPRWIh6L4rX68XW1hYmkwndDxHQYbfbsbi4 +iMXFRTQajVcWvsX9dzgc2NragiRJODo6wunpKfL5PI6Pj3F4eEifFX3TNA2xWAyxWAzBYBB2u/2V +AhpEpuXi4iI9X3Rdp0CQcDj8mNjncNiRSCTIISKXy6Fer9NeLuy+nU4nCfqtVgv5fP61zIfZ3qrB +YrEgHo/j3XffhaIoyOfzyOVyFDAg6qc7nU5yyJAkiURSkW3vdrtRLBaRz+dRqVRwenpK5SyElb7X +68Xq6ira7fZT++HxeLCxsYHBYIDT01Ocn58jn8/j5OQEVqsVi4uLWF5e/taEbwBUAmNhYQE3b94k +UVf0TwidwqpeiOWiLI0IUHipIApplvEtyuGYTCYa33A4jFAo9Jhdvgg6iEajqFarKBQKtGZ9Ph9M +JhNkSb5kIb+xsQFd15FMJml8U6kUuWOI56JYH6FQiPbnR51CptMper0e1Z8XZVbEc8toNFJm9tWr +V7G6ugq/3w9d19Fut7/RPRKuEaFQiDKbRSmK+fch0R8xVleuXMH169exuLgIv99PgXBra2vo9/tI +JpNIpVLIZrMoFov0vBDf43a7sbGxgbW1NVy5cgXBYBDj8fi5c1GWZSQSCUwmE2iahna7Te9vd+7c +ga7rlHX/ooi2iXIN4n5MJhMqmXN4eEj/3yEajV5qdygU+l4HzzIMwzCvzvfzdIhhGIZhGIZhmO8l +4pBJ/LfRaEQgEKCMy+dlTkiSBI97ZvfY6/UoK0bU3HU4HLh69SrMZjO+/PJL1Go1yhzTdZ0yNIX1 +rtPpRDgchsvlgtVqpSwsu90Or9cLp9NJnxV1G+fbYrVa4fV66XeMRuNjGWUGgwEWiwUejwej0Yhs +s00mE7Vb1O0VQpfImpNlGeFQGNevXwcANBoNZDIZtFotqhsZjUbR7/cxHo0vje83RRxSzkSXbQSD +QVgsFsoAFjbLApPJBI/HQ+MbCASovmoul6Oa0sJ6WthPA7MMtZWVFWiahmq1iqOjI8qabLVasFpm +h9Uejxubm1sIh8P49NNPL7WlWq1eaovI8rZarVRDXthpLi0uweV0kXAp+lKr1cjCW2QZeTweBAKB +15pN/+g4C/cBl8uFcDhMc0RRHs+epBqWEhCPx2G32+F2u6mWt6hh3ev16HecTic8Hg8JMLFYDD6f +D3a7DRaLGVeuXIGiKLh9+zYajQZljrbbbRJdVFVFv9/HYDB4LFjEIBkQCoVw/fp1+pwIbBF1ZD0e +D7xeL4xGI2w2G3Rdh8PhIJt0IVCJA2WLxYLBYIBut4tyufz/s/en3W1d6Zn4fZ0Z8wwQAOdB1Dy5 +XK64KlXpTuL0yhfoj9bfoVevXp286H6S/FOJnUqVXbIta7AGkhIpziCJecYZnxfQ2QI02JIli6J0 +/dbiokwSwME+A2Bc+743KpWKeDxd15FKpcQyAJlMBlNTU6Liu1KpQFVVDAYDUfnvf4AfiUQwPT2N +2dlZLC4uYnl5GXNzc6ISL5vNiskkfkv1TqeDer0O0zRRKBQwGAxEeP8isjIMOCORCFKpFJrNpgi6 +2u22WHPdr8rPZrNYWFjA+fPncfr0aUxOTj63As6fzJFMJhGLxRCJRMT1I5vJQpJ/fI3TFx2H/uPN +zs4iGo0im81CVVXUajUcHh6ObbOiKOJYnZycFOPZbrexv78vOm0cHR1BURTRBn96ehrLy8uwLAuV +SgX9fh+xWGxY1apqYlKDvyavX82vKqoIlv0xHQwGotrQb5ntH+P+bUOhEDRVE/vDn4QxGAwQjUZf +WIVt6IYYW7/VtH+8JpNJLC8vi4kCvV4P5XJZTGTwPE8EeJFIBJ1OB6FQSJxHr1px7IdNkiSh3++L +rgF+h41oNIpisTjWgj8Wi4lA6unn5ld6jj43//UrFAphaWkJ6XRaTA7zqz2r1arohOC33fev+X4F +Zzwe/0nBt6IoyKQzIpCt1WpoNBpibfFsNvvM+RAOh0W4vbu7i8PDQzx48AD7+/uiXXgwGBTHciaT +QbVaRblcFl0v/Ep8RXlyvfdf+/3Xc0VRxJj6E3j82xrGcPJNoTB8nfbfe4wGmbIsi0kifrW2v2yA +f5xPTk6iUChgZ2cHh4eHAIC9vT3RIcFfliGTySCfz6NareLo6Ei8d/GPYwBIJBJYXl4WFc+lUklU +7dZqNXG8+BMd/CpWv9raf/3z3wf5z1dRho8hS+Pvb/zA2nEccc0bXf7Cnyh0/fp1scxEu90W15xM +JgPP80RnFX+fPl0d/TLXMP/9k79sTjabRbFYxMzMzHPXiQ8Gg6Lzh7/euH+7ZCIprqf+l2EYmJ2d +RSKeEEvANJtNHB0doVKpiPDWD3j916lUKjVcA/zxBBh/bfFsNotIJAJN02CaJmq1GprNpnhd8Cc5 +nT9/HmfPnsXszCzCkTA67WFLdH8f+ePtv1cdHTe/sj0Wi6Hf74uJgZFIBLpuIBqN4OKFi0in0/jy +yy/Fe6Fms4l2uy26GkQiEczMzODSpUs4deoUisUiEokEgOH7+bm5OXGc+hOg/Kp3v9OLP/Hi4sWL +OHfuHBYWFpBOp0WI778PmpiYEO/X/GuYJEkoFAqIRqOwLAsbGxtYW1sTr/mHh4eYn5t/5WuP/77Z +7wrjd8dRFAWHh4fQNE101SgWi2g2m+K66L+38a8V/teb6ipBRETHh8E3ERERERG9NL8K5sKFC+ID +wqmpKUxNTr3UuqCKrGAiP4Hz7nBdwaWlJTQaDVy5ckV8SFYoFMSaqDMzM2i32yKM8D/UlmVZtG9O +JBKIx+PIZDLIZXOIxqL45JNPoGkaQqEQLl++LCpcRz/QV1UVV65cgWmaUBQFZ8+eRbFYFGHK6N/5 +60c7jiPWK03EE4jFY1hYWMBnn32G6elpXLx4Uazr61fUxhNxTLvTYt1Mv/LPtm0kk0lR7ZfNZl96 +XdVXMfwgOwxZlnD+/HlEo1GcP39eTDrw+RVe0WgUU1NTmJycHG5/PI65uTkRmqbTaczNzY09hv/B +sCzL+Mu//EuxVvC5c+fEmPrjEYmEoSgyzp8/j0gkgnPnzr1wWyKRCCYnJ0Vw4lcMBoIBxL242I58 +Po9OpyPWPPU/dPbXnPXX7v05+MfZ3Nwc/vqv/xqZTEYcB35r3hcxDAPxWBwzMzPwPA8TExPiw+rR +dWzD4bCYpOGfb9HYMOhyHAf5fF4E/TMzM6jX6+K2fjDjj6/fevVpfsUyMAwT5ubmYFmWqFrzwwTT +NNHr9eC6rpgAMjMzg1g0JgIiSZJw9sxZhEIhnDlzRjwn/wNq/4Nof83lqakpJOIJuJ4r1uHOZrNi +XeperyeCb79S25+EkUqlREWf/zwmJyfFxJalpSVx7oZCIczPz2Nubk6E8y86ZwzDgCRJuHz5MlzX +xeHhIaLRKOLxuGjva5qmmPgwMTEhjjN/Xd/n8asD6/U6gsEgJicnMTExMaz0/omh94uOq+npaTiO +g0wmI1rUjm6zH7jn83lMT0/DMAxMFifxy1/+EoVCAbVaDfV6XVS9+vurWCyKY9GyLFy4cEFUY6qq +ikKhgN/+9reIRqPDCvgzZxFPxEVIe/78eQwGA9TrdVy6dAm5XA7hUBiyIovb+iHNmTNnEE/EIUsy +ZEXGhQsXYFkW6vU6rl69Kq4LT4/3RH4CV69eFdcxv/uGP4Ekm82KIKxQKIjuAn4ng2QyCdu2ce3a +NdH690nYpL/SfvIDF0VRxLXCr0rUNA3FYlFUIi4vL2MwGODKlStizfann1sul8PVq1eHS2Q8rj4v +FApifWL/uD516hR0Xcf8/Lw4B/0lH/xJBOFwGPl8HlNTUygUCojFYlDkVw++dV3HRH5CXK9mZmbQ +6XTw0UcfIZVKiYkNo/xwKhQK4dSpU1AUBWfOnEGtVkOn0xGdLjKZjAgd/e4i/kSdVCo1nFShDl/D +//7v/x6XLl3CpUuXMDk5iXgsDlkZBsGffvqpCPkvXryIRDyBQDAgxsIPkyORCObn59HpdEQlqf/e +w29t7U+CSyaT4toYDAZx6tQpyLKM5eXlZ55HOp0Wk02i0ShOnTqFTCaDy5cviwkiijwcD/+abhiG +GEvHcSDLsrhmtttt0UL/0qVLmJ6eHu6/x9evTz/9VASRFy5cQCQynKAkydLYfpubm4PjOOh2u+Jc +n5+fF5NdisUiNFVDJBIR637745JIJJBMJrG1tSXa9/sTZX7s9e9po91A5ufn8dlnn2Fubg6Li4vD +oLYwbMs9Khoddg3wj5OzZ88im81ieXl5LPQefQz//FhcXISqqpiamhLnh99xwe/C408CnJycRLFY +hGEEYBg6lpaW8N/+238Tk4z8EL3T6aDX64nHTSQS4pqZz+cRCAagKqpY9ueXv/yleL975cqV575X +VeRhpbRt22i1WmJiw/z8vNin8UQckizho48+QjqdRqVSEa+dfuW6H97776ee7oIznISWxZkzZ8QE +Ab/Fv9/G3jAMcb3yJ374/1/geR4WFxfxt3/7t5icnMSlS5cwOzsr2on7fxcOh7GwsID/+l//q+he +k8vlUCwWxTXkdfkhuCRJsCwL/X4fjuOILguVSkWc0xMTE5iamsL01DQy2WFLdL/zCivBiYhOLsl7 +kyUFRMfAdV1sbW1he3sbu7u72N3dxdHREQqFgvjAzm/dRURERHQcyuUytre3sbOzg93dXezt7SEY +DIoPj6anp9/Ymn9vg+d5aLfb6Ha7ACCqjF6mUsLzPFiWJdbl84O1UCgkwgv/Z/76fLbtwPOGlZli +7UhJhuMOQwq/itBvvemH4v6HuKFQCKFgCJIsjW2b/zz8yiU/xFMVdazixv8wc/T5+h9MyrIM27bF +h53hUBih8JMKRUmS4NgOLHv4nIfPyYL3OODz1772q6NGq77e5P5yHReO6wzbcA5MWPaw2tWv/gOe +tModbYHrf2hoWzYc1xHb7I/V6GP4LZ/9NdiBYYDqV3v6Ffeu48L13OFYvMK2jI6P67qilehgMMBg +MHjmPvzjRVM1aLomPrT9Oc4Hf61J/7zw17v2w9MXfXjqui4814NlW+J4Hz4vF677ZE1yWVagqooI +BfyWy6qqirG3bVvcx2i7ZP+c2VjfwP/7//0//Ou//quYDHDq1Cn88pe/xKeffgrP9WA79vD8HJgY +mAO4rgfPcyFJMnR9eH77+w+ACCN1XYehG5AVGZ7nif3in+vP279+CDp6PvkfT/jtYofHlCvOl+FY +ytA0VVxvnq4sdGwHtmOL820wMMX1Q1EUGLoB3dB/9Hrl75tu70nI7Yd3w+PPFfcLYOw49a+Jz9vv +n3/+OX7/+99jZWVFTND45JNP8Nlnn+H8+fNv5JgcPa78cfCPq9Ft9o8Nf9t1XRchgX8s+vtNkmSo +qiKeo23b6Pf7cF1PrJXqt823HVus+2sYxpPWx48rTXv93uNJKo64rR+2jt5W14dtqA3DELf194d/ +W79K8unzzDRN0UlD7CNDF8eL/zrjj4/1+O88zxX7eX9/H//rf/0v/N//+39F2HzlyhUsLy/j7Nmz +L718gn9Mjp5bfgDtn1u6rsOxHXHe+ePiH9s/9tw0XRMV6f710T//Rlsvj7Yt9l/DRl/Dn379e1lP +v7YPjzkXkUh47Hk873ae54mJVf5+GU5gUiDLknhtVxRF/J0/4SgQCIhjY2AORIcYv8JTVYaT2BzX +eXxcdaFpw0lCo+8LXNcV70lGt3/0vYemamJyjud5InD3q879LhX9fn/seTx9vXvR8/CP46ev6YPB +YOz66b8GOLYD0xpeJ/3qbE0dBs2u5z6eDNaFqio/+D7I3xbXeXxdV2TxvgTAC8fFvyYoioIvv/wS +/+f//B/8+c9/xsWLF/HRRx/hwoULOHPmDBYWFl7pOAKAfr8v3lf5Fcu6rosxHz23/Ovc8Fw2oWnD +yQsvek/tnx9+e+7R9w+jH5H77zdG30NIkgRFVjAwB2L7FEWF+nibRt+v+veha7q4vf+64L9Ovux7 +Vf+48veRJA8n7fivu7Ztw3XdsePFP9/9101VfbKEjN8FYfRx/DEYPe5G34v4x7E/kWP0fvxzYjAY +PPN+eHRZFf/9qGmZ6HV76Pa6Y+/T/O37If/xH/+B//E//gf+5//8nz96PPnXHP/c8jvg+N8DgQCK +xSIuXLiAy5cvY2lpCUtLS8MJdS+xpEWj0cDW1hZ2d3exs7ODvb09SJI09v93U1NTiMfjL30OEBHR +m8GKbyIiIiIieiWSNFyHMRqN/qTb+h+YvYj/YevrTATw1+N8E8/Db6/8olbZfhD8IoqqQFGVY1tj +WpKk4TZg+KE7XrHj9w89t9HH8IOTH3ue/rZomvbK2+LzA8OXebyfm/+Brv8h6qs+D8h4reNj9Jz6 +oXOmXC6LIP55/OPUMIyfvF/87fH3zU/1Ovt09Hn8lGuUz983P/VaN2p08km9Xsf+/j6q1SqKxSKm +p6dRLBbf6MSn1zmuRgOvn8of/2Qy+dzf/9D19Mdu+7L744deZ/yWxLVaTSxXMdpWvN/vo9FooFQq +oV6vo9/viwr5iYkJ0d3gVcbjTZxbL/PcgPHr49uaUPcyr+0vup1fhfsy2/pD+z6kvvg+/Nc/v7Xz +88bsTRz7/mSv13keL3tN/yEKhh1mXvR8Rx/rh7a53W6LJWeG54kBTXsyicGfpHJ0dCSW2DAMA9ls +Ful0+qXG4unteZVx/Cnvr0bPj1fdPt8PHWsv+zxlWX7p96o/9v7Cf719neN3dKLhqz630XXEf2hM +/fejQXX4dymkfvL2vgx/IoM/ccSf7CQm+qiq6F41GAyws7OD1dVV0ap9uCRJHLFYVExOYRt0IqKT +gcE3EREREREREdHPxPOG1deWZaFWq2F/fx/lchlzc3NYWFh448E3/bDBYIDt7W2srq6KkD0Wi4lK +72q1ilKphPX1dWxvb6Pb7UJRFLGWczwR/0ntwIlOmkajibW1Nezs7IytCe8vB7O/v49SqYQ7d+7g +8PAQtm2LJWv85Q+I3hV+NwX/u99S3nVdVKtVsfxDLpfD/Py8WBplfn4emUwGABh8ExGdEAy+iYiI +iIiOib8GXb/fR6vVGmtPTET0JrXbbbHOpd/i1W8n22g0+GHuz8ixHXS6w/avBwcHojpSkiRRFWma +JhqNxnFv6gehVqthfX0dt27dgmEYSKVSSCQSooXv4eEhtre3sbW1hXK5DNd1RVv9QCAA13XRare4 +/iu990qlfaytrWFlZQXpdBqZTEas2QwAW1tb2NnZwebmJlqtlqhWH7a31zAYDHhdozeq0+n85P9f +8luyu+6TdvSmaaLb7aJcLouuH8lkEqVSCbVaDZ1OB647XDogEAggGAxClmUosoJ2py1au/uBOl8X +iIjeDQy+iYiIiIiOgb+WXrfbxdbWllgXjojo57Czs4Pt7W20220RcvvrUQ/X02Xw/XPxw59Go4G7 +d++i0+nA8zyxPmi73cb29vaxt+3/UDQaDdy+fRurq6uwbVu0vPXXYe73++h2u+j1epAkCfl8HoFA +AKVSCd9++y0URXnhmtVE75PNzU18//33WF9fF+2h/fNEkiTR6nwwGCAYDCISicB1XaytraFWq4nz +iuhNuXPnDqrV6hu7P8dxYFnWyHroElqtFra2tsT/o3333XdIJpOIRqOIRCKIRqOIxWJwXRfNZhON +RgO9Xk+cB0REdPwYfBMRERERHYPR4LvRaKBer7Pim4h+NpVKBTs7O2i32+L641d/+5V69PPo9/so +l8sol8vY29tDp9MBALRaLezu7qJcLosgiX5+7XYba2trWF9fR7vdhmmaYg1YSZKgKAo0TUMgEBDt +zQOBAA4ODtDtdsXfEb3v9vf38eDBA2xtbcE0TZimKaplZVkW50o0GkUmk0E6nYbrunj48CF2dnYA +8FyhN2trawu1Wu2N3Z/f9ty2bXGs2raNwWCAo6MjMdEjEAggHo8jkUggk8kgl8shEAiI26qqClVV +GXwTEb0jGHwTERERER0Dz/NgWRZ6vR52dnawsbEhPlAnInrTOp0ODg4ORJvQwWCAXq+HdruNw8ND +hhM/I9M00Ww20Wq10Gq10O12IcsySqWSqB6mt6ff7+Pg4ACVSgX9fh+DwQCDwUD83jAM6LqOwWAg +zot2u42DgwNomnbcm0/01tTrdRweHqLRaIjlMUYniQSDQWiaBtd1IUkSHMdBvV6Hruus9KafRbVa +Rb1ef6P36Xne2HfTNOE4ztj7IsMw0Ol0UK/XUa1WcXBw8KTtuaIgkUggmUwe9/AQEdFj/L8rIiIi +IqJjMFrxvbu7i3v37qHZbB73ZhHRe8q2bdHCeTAYoNPpoNlsolKpiPVa6efheR5M04RlWbAsC6Zp +QpIk7O/vo9FocNLBW+a6Lnq9Hvr9vtgnjuOI3w8GAziOI1rgNhqNsfbORB8Kf4Kmf644jjO2PnKv +1xM/7/V6qFQqkGVZrJVM9KaZpim6pvxcPM97Jvj2J0f5a4JXq1VomiYmSrmui1AodNzDQ0REjzH4 +JiIiIiI6Bp7nwXVdWJaFVquFarWKRqNx3JtFRO85RVHEh7qO46Df7x/3Jn1w/Arvfr/P8T9m/rrF +L+K3dyb6kPkTP17EdV1ez+itcBznrVyT/erv0cft9/swTRODwQCyLEPXdQQCAQQCAfR6vbEJVERE +dLwYfBMRERERHQNZlqGqKgzDQLFYxPLyMlqt1nFvFhERERER0Tun0+ng6OgIBwcHP9tjSJIESZIg +y7L4maZpCAaDCAQCCAaDCIVCCAQCYm3vZDIJwzCOe3iIiOgxBt9ERERERMdAlmXxIcrk5CQURWGl +DBERERER0XMcHBzAsqyfNfgGIFr2+3RdRzgcRjQaRTqdRjabRTAYFL8PhUJcNoaI6B3C4JuIiIiI +6BhIkgRVVREKhTA1NYVcLje2biIRERERERENPXjwAPv7+2/s/iRJgqIoospbkiQEg0ERcodCIfHv +WCyGeDyOZDKJZDIJVVXR6/XQ6/Xgui5bnRMRvUMYfBMRERERHQO/1Xk4HEY+n0ckEhHrvhIRERER +EdETmqbh1q1bb+z+/OBb0zTIsgxJkhCNRjE5OYnp6WlMT09jZmYGmUwG4XAY4XBYtDwfDAYol8so +l8uo1+toNBrHPTxERPQYP1kjIiIiIjoG/gctuq4jl8thenoaoVDouDeLiIiIiIjonVOr1RCLxX7y +7SVJEt9lWYZhGAgGgzAMA5qmQdM0ZDIZLC8v4+zZszhz5gzOnTuHfD4PwzDG2pk3Gg1sbW3BMAx4 +nodOp3Pcw0NERI8x+CYiIiIiIiIiIiIioveSv8yULMvQNA2qqiIajWJiYgKFQgHpdFqs310oFFAs +FjExMYFEIgFd16HIynE/BSIiekkMvomIiIiIiIiIiIiI6L3kB99+ZbemaUilUlhaWsLly5exuLiI +xcVFZLNZGIaBQCAgAnIuR0VEdLLwqk1ERERERERERERERCeeJEminbmu69A0DYFAAPF4HIlEArFY +DNFoFIVCAUtLSzh9+vRwXe+pacTjcciKLNqiExHRycPgm4iIiIiIiIiIiIiITjQ/9PYrtf01vLPZ +LBYWFnDq1ClMTEygWCwik8kglUohlUohEokgFA5BVuTjfgpERPSaGHwTEREREREREREREdGJ41dn ++2G3ruswDAOGYSASiSASiWB6ehqXL1/G1atXMTMzg9nZWcTjcVZ2ExG9hxh8ExERERERERERERHR +ieAH1n5Lc0mSEI1GEQqFkEwmUSgUMDk5iWQyiVQqhYmJCUxNTWF6ehqJRBKGYTD0JiJ6TzH4JiIi +IiIiIiIiIiKiE0OSJCiKIlqbh8NhpFIpzM/P4/Lly7hy5QpyuRwmchOIxWPQdR26rkNRFKgqYxEi +ovcVr/BERERERERERERERPTOGl2/OxKJIBgMIhwOIxqNIhaLIZvNIpfLYXp6GqdPn8bp06eRTCaR +SCQQCASOe/OJiOgtYfBNRERERERERERERETvNFmWoSgKotEostkspqamMD8/j8XFRaRSKWQyGaSS +KaRSKaTSKRiGAU3VjnuziYjoLWLwTURERERERERERERE7yxd1xGPx5HL5TA5OYnZ2VmcPn0aly5d +wpXLVxCLxxCLxWAYxnFvKhERHSMG30RERERERERERERE9M7K5XL49NNPoWkacrkc8vk8CoUCJicn +EU/EEQgEoCqMO4iIPnR8JSAiIiIiIiIiIiIiondWNpvFb3/7W5w7d06s6x0MBqHrOlRVhaIokCTp +uDeTiIiOGYNvIiIiIiIiIiIiIiJ6ZwUDQWQyGcTjcRiGAUM3oKjKcW8WERG9Yxh8ExERERERERER +ERHRO0tWZBiGIaq7ZUU+7k0iIqJ3EINvIiIiIiIiIiIiIiJ6Z0mSBE3ToGnacW8KERG9wzgtioiI +iIiIiIiIiIiIiIiITjQG30REREREREREREREREREdKIx+CYiIiIiIiIiIiIiIiIiohONwTcRERER +EREREREREREREZ1oDL6JiIiIiIiIiIiIiIiIiOhEY/BNREREREREREREREREREQnGoNvIiIiIiIi +IiIiIiIiIiI60dTj3gAiIiIiIiIiIiIiop/C87xn/u25w++u5z73737oZ0Q/ZjAYwLIsWJYFx3Hg +OA4kSYJt27AsC6ZpwjRNDAaD495UOiaSJP3gz2RpWJMqydJL/T0RvTwG30RERERERERERER04nie +N/waCbrFz0a+XPfJz0dvS/RT9Pt9DAYDEXDbtg0AY4F3v9+HruvHval0jEaDa0mSIEkSZFkW/5Yk +CZL7+OcjQbh/O/8axQCc6NUw+CYiIiIiIiIiIiKiE8UPsjudDrrdrgggbdseC7pd130m+GYATq+j +3W7j4OAAh4eHKJfLqNVqkCQJmqZBURR4ngfLshCJRI57U+kY+EG1/13TNAQCARhGAJqmQlVUKKoC +WZahKMp4IO4NQ/DRSnDP8xh+E70CBt9EREREREREREREdGL4IbZpmlhdXcXdu3dxcHCAZrOJdrv9 +TMjNwJveJNM00W63xaSLTqcDAAiHwwiHwwiFQohEIqz4/sDJsgxZlhGLxTA7O4u5uTkkk0nEY3GE +wiEoigJVUSErwwBcUYZhuCd7IgCHzPCb6FUx+CYiIiIiIiIiIiKiE8c0TaytreHzzz/H2tqaqL59 +GkNuetMcxxnrKAA8CTolSYKiKMe9iXTM/CA7nU7j6tWrsCwLxWIRbnF4vKiaCkVRRKcAVVXFd1ke +ht6yy/Cb6FUx+CYiIiIiIiIiIiKiE8VzPTiOg3K5jM3NTTSbTSSTSUxOTh73phERoVaroVarodvt +YmtrC4qiYH5+Ht1uF8ViEZFIBOFwGIZhQFVVaJoGTdXgeR5U9XF0x/Cb6JUx+CYiIiIiIiIiIiKi +E8PzPLieC9u20Wq1UC6XoWkaPvnkE3z66afHvXlERLhx4wa+/fZb7O/vo1KpoFqtotFowLIsmKaJ +XC6HXC4Hx3Fg6AZc133+sgySBwUKw2+il8Tgm4iIiIiIiIiIiIhOFL/NdLfbRbPZxMTEBC5cuID/ +/t//+3FvGhERgsEgKpUK6vU6Go0GGo0GNE2DYRgAAMuyoCgKIpEIQqEQgm5QBN6q9yS6UxQFruey +8pvoJTH4JiIiIiIiIiIiIqITxQ++iYjeRaqqQtd1RKNRhEIh5HI5KIqCUqmEXq+HbrcLx3GQyWSQ +Tqfhui4cx4HrutB1Q4TgrutC0zRWfhO9JAbfRERERERERERERHRieK73OPj2xloCExG9KxRFEcF3 +JBJBJBLB9vY2tra2sL+/D9d1oaoqbNuGLMtQVfWZ65kfbMuyzMpvopfE4JuIiIiIiIiIiIiIThTP +8+B5LoNvInonSZIESZKgaRpisRiy2SyAYSDebrdhmiZWV1fR6/VgWRZs20YikUAikXh8fRtf79sP +yqGA4TfRD2DwTUREREREREREREQnih8MMfgmoneZLMsIBAKIx+OIRCKYmZlBtVrFgwcP8ODBA/R6 +PXEts20bmqYBwEjw7Y4F25Ikse050Q9g8E1EREREREREREREJ4brDdf25hrfRPSuk2UZhmEgEokg +kUggHo/j6OgIlUoFd+7cQblcRiAQEAF2IBCA4zjw3CdhtiTJ8DwPuq6z7TnRj2DwTURERERERERE +REQniuu6rPgmohMlFAohk8kgGo2iVqtBURRUq1W0Wi2srq7Ctm1IkoRcLgc7YwMYr/z2ua4LTdNY ++U30HAy+iYiIiIiIiIiIiOhEGW11zvCbiE6CYDCITCaDYDAIVVGRz+fxxz/+EV9++SXq9ToAQNM0 +EWwbhgEAY5Xf/r9Z+U30fAy+iYiIiIiIiIiIiIiIiH5GiqJA13VEo1EUigUYAQOVSgWVSgU7Oztw +HAfr6+uwLAue58F1XSSTSQAYm+Dj/9t1XaiqCihg+E30GINvIiIiIiIiIiIiIjoxWOlNRCeZpmpI +JBIIBAK4cOECZFnG6uoq1tbWsLGxgcFgIP7Wtm3ouj52e89zx4JtSZLY9pzoMQbfRERERERERERE +RERERG+BoiqIRCKIRCJwXReRSAShUAiVSgV3795FtVrF7u4uFEWBJEkIBoNwHGfsPiRJhud50HUd +sixDlodt0Fn5TR86Bt9EREREREREREREdOJ4Lqu+iehkG677ncXi4iIajQYURUG1WkWr1cLq6ips +24YkScjlcs+E357nin/7ITkrv+lDx+CbiIiIiIiIiIiIiIiI6C0Lh8MwDAO6rkFVVeRyOfzxj3/E +V199hWq1CgDQNE1UdxuGAQAizJYkWfxblmUoigLXc1n5TR8sBt9EREREREREREREREREb5ksy9B1 +HdFoFIVCEbquo1KpoFqtYmtrC67rYn19HZZlwfM8uK6LZDIJAGMdL/x/u64LVVUBBQy/6YPE4JuI +iIiIiIiIiIiIiIjomGiqhng8BsPQceHCBSiKgvv37+PBgwfY2NiAaZoiuLZtG7quj93e89yxYJtt +z+lDxeCbiIiIiIiIiIiIiIiI6JgoqoJIJIJIJALXdRGJRBAMBlGtVnH37l1Uq1Xs7u5CloetzYPB +oFjze7Ttud8SnW3P6UPF4JuIiIiIiIiIiIiIiIjoHRAMBpHJZLG4uIhGowFFUVCtVtFqtbC6ugrH +cSBJEnK5HFzXhST/SKAtY1j5TfQBYPBNRERERERERERERERE9A4Ih8MwDAO6rkFVVeRyOfzxj3/E +V199hWq1KtYF9zwPhmHAMAwAw/XCh98lKLICSZJEGC5LT6q+id5nDL6JiIiIiIiIiIiIiIiI3gF+ +sB2NRlEoFKHrOiqVCqrVKra3t+E4DtbX12FZFgDAdV0kk8mRlufS2L9VVYUjOWPrfRO9rxh8ExER +EREREREREREREb1DNFVDPB6DYei4cOECFEXBysoK1tbWsLGxgcFgINb89jwPgUAAkiRBlmXxc1mR +RRAuSRJkV4YncZ1ven8x+CYiIiIiIiIiIiIiIiJ6wzzPg2VZ6PV66HQ6aDabCAaDr3QfmqaJ9bxd +18XW1hZqtRpkWYZhGJBlGaqqIh6PQ1EUKIryJPh+6jtkQIb8w2uCE51gDL6JiIiIiIiIiIiIiIiI +3jDTNFGpVOA4DgaDAer1OlKp1CvfT7vdRqvVQqVSgaZpyOfzcF0Xe3t7GAwGCAQCKBQKUFVVBOGK +okDTNDi2I6rAPc8ToTfDb3ofMfgmIiIiIiIiIiIiIiIiesNs20atVkO9Xkej0cDR0REikcgr34/r +unAcB/1+H7quo1gsYn9/H6VSCfV6HdlsFqdPn4ZhGNA0DZqmieBblmUorgLXdYeBt+txrW96bzH4 +JiIiIiIiIiIiIiIiInpDQqEQMpkM6vU6BoMBTNOE67ro9/vwPO8n369t29B1HYlEApVKBbZtw7Zt +dDoddLtdhEIhEXxrmgbDMKCqKhzbES3QXc+FAuW4h4joZ8Hgm4iIiIiIiIiIiIiIiOgNiUajKBaL +ME1TtClXVRWq+nqxnCzLCAQCUFUVhmFAkiQ4jgPLstDtdtHtdqHrOgKBACzLgmVZ4nFd132t0J3o +JGDwTURERERERERERERERPSGxONxTE1NQZZl0er8TYfOu7u7UFUVtm3Dsiz0ej30+30MBgMMBgMY +hgHLsqBpGlzXFcG353nwXA+Swlbn9P5h8E1ERERERERERERERET0hsRiMUxPTyMWi6HX66Hb7b7x +4LtWq2FlZQW9Xg+2baPf748F337F97AdugNd98aqvj3P4zrf9N5h8E1ERERERERERERERET0hsSi +MQSDQUxOTsJxHLiu+8YfY3V1FeFwGOVyGbZti8DbNE1YlgXTNGHb9uPHd+A4w3W+AYh1vhl+0/uG +wTcRERERERERERER0QfG8zxRIVqtVLG7t4vDw0NMT09jdnYWAHBwcICjoyMEAgEYuoFYPIZsJotU +OnXcm0/vsP39fezv76Pb7SIYDCIQCCAYDCIYDKLf72Nvbw97e3uYnJzEzMwMDMNAqVTC4eEhdF2H +ruuIxWJIp9PIZrMnMphVVAWKqvzo3x0cHGBvbw+dTkeMUzweRyqVQigU+sHbBoNByLIMz/PEOt/j +Vd7Dr2fanHOdb3qPMfgmIiIiIiIiIiIiIvoAtdttVCoV3L17F19++SVu376N3/3ud/ibv/kbAMD1 +69dx+/ZtxONxxONxzM3N4eLFiwy+6Qc9evQIX375JcrlMlKpFFKpFNLpNDKZDBqNBv70pz/h2rVr ++Iu/+Av87d/+LWKxGL755ht89913iEQiYn3sixcvIpPJnMjg+2VtbW7hT1/+CaVSSYzT4uIiVFX9 +0eB7lOu6IvQ2TROmaYpKc//7aNU5w296XzH4JiIiIiIiIiIiIqIPiud5aLVa6HQ6sCzrJ9+PJEmQ +ZQWqqiAYDCIcCr9Ulee7wK/4bjQaePToEb755ht8+eWXiMfjuHz5MgDg/v37uHbtmggtbdtGPp+H +aZrodrvo9XqQJElU9aqKClmRXzuo9EO5druNTqcDAKJiWJGVN/IYJ53neTBNE512B+1Oe3j8hcPQ +NA2KokCW5WPbtlKphNu3b2N3dxf5fB7FYhHFYhGO4+Dw8BDffvstPv/8cxiGgfPnz8OyLKyuruLr +r79GPB5HJpPBYDBAoVCA53rA8T2Vn3+sDkq4c+cONjY2kM/nUSgUEAqFMD09/dL3MVrx7Vd5O47z +wmpvsca360FSPuzziN4/DL6JiIiIiIiIiIiI6INimiauX7+OP//5z6hUKj/5fnRdRygUQjQaxcWL +F3Hp4qUTUw0tSRJUVYVhGNB1Xaz9K8syNE2DLMvid4ZhwDAMyLKMXq+Hw8ND3Lx5E7dv34aqqjh3 +7hzOnz+PaCSKSDQCXdd/8nb5wZxlWbh+/Tq+/vprAMCFCxdw4cIFRCIREfB+qFzXhed62N/fx7Vr +1/DNN9/g7NmzuHTpEqanpxEKhRCJRI57M8f80LEWCATE7/zjzQ/w6eV4njdW3T1a5e0fL2xzTh8C +Bt9ERERERERERERE9EHxg+///b//N3Z2dn7y/QQCAcTjceRyOdi2jdnZ2RMTfAOAoigicJRlGbIs +i5/530fDb0mS0O12cXh4iK+++gr/9E//BF3X0Wq1kEql4E64CAQDrx18O44D0zRx48YN/MM//AM8 +z0Ov10M+n4fneSIY/VB5rgfbsbG3t4fPP/8c//iP/4jf/e530DQN4XAYkiS9c8G3oigi2PYDbf8Y +Gw2+RwNwVWWE9TJGQ2//azT0dl0Xrsc25/Rh4FWDiIiIiIiIiIiIiD4oiqIgkUhgcnLyub/vdrto +Nptot9tQVRWqqiIQCCAUCo2tu2sYhmjNnEgkTlxQpyrjVbh+FbimadBUbawCdzSQ1DQNkUgEyWQS +uq4jEomM3cfr8itTO50OKpWKaE1vmRYcx2Fw95hlWajX69jf30e9Xkev14NlWWNrOb8rZFkWx5Kq +qmKihaqqzxxjgUBgLCCnl+O3NB9tbe44DgA/HPeGrePB8JveXyfrVZiIiIiIiIiIiIiI6DWpqorT +p0+j0+mgWq0+8/u9vT2srKxga2tLrJ2cyWQwPT2NYrEo/k7XdQSDQcRiMZw+ffqdq7L9MYqqjFXX ++hXffrjth5H+VygUQjgc97X7BQAAgABJREFURjwex9LSEtrtNhRFweLiIhKJBILB4LGuLU3vLv+4 +8rsJ+F+aNj7BYrT6m8H3y/OD7qfX8x79nc91XY4tvbcYfBMRERERERERERHRB0VVVczNzSMUCqHf +7z/z+3v37qHf76NeryMejyOZTGJubg4XL17EuXPnxN/5Vay6riOVSiEYDMI0TfR6PfT7/cdthj14 +niuCJ1lWIMtP1tf2185WVXWsWtpfl9e0TAwGAwwGA9HOGAAkSYYsS4//1oOiyNA1Haqmim36sRB6 +tNW0pmlQVfWFgaT/FQgEEAwGkc1mMT09DUVRkMlkEAqFRPvxXq+HTqeDbreLQCCAQCAAYNhi3rZt +OI4Lz3MhSTI0TX0SsusGLNtCq9VCrVZDq9USFd6tVgulgxIs28JgMEAkEoVhPKkgVhQFruuKsfJb +PnueB0mSoCiKqC72g35N00QoOBgMntlmSZJgmRZMy4QkSZAkSaxJHQgE0O/30ev2YFrmM9W1w7+X +oSiyGDtVVaHIChT1SegoQkrXQ6/fgzkwYdkWbNuB6zpj9+PvF/85lstlDAYDAEC/30e1WkWpVIJt +2xgMBmOP64/RYDB44WOojydC+OPkr/X+9DHp7wP/GB89JhVFRqPREMerf9yPts73jzV/H4y2OB+t +/H6ZDgqj49fpDvef47ji2LBtG5ZljR0P/liMhu3D7ZeG4bHjwnEdMc7+Mfu8sdI0TXyNjpV/Py8z +Vv1+X7Qnfx1Ph96jYzPcpvGW50TvIwbfRERERERERERERPRBGa6BHIbnZWHb9jO/Pzw8RCQSEeF0 +IBBALBZDOp1GoVAYux8/UA0Gg5AkCaVSCffu3cPq6ir6/T76/b5oP+15nqhozWazmJ2dxezMLGLx +GOLxuFgb2w9i+/0+9vf3sba2hkePHqHb7aLb7QKACHD9MC0cDiOXy2FiYgJTU1OYmpr6wQp0P8R1 +XVe0lh4N8Pww3n/+fjguSRJM08LGxgb+/Oc/i+ByZmYGnufBtm3s7+/j+vXruHv3LhYXF7G4uAhJ +krC3t4dSqTQMX00ThmEglUohk8lgfn4eCwsLaLVauH79Om7evIkbN26g3W7DcRzcuHEDnU4HwWAQ +oVBIVODPzc2JyQndbhcbGxvY3NxEq9USt/W3MZ/Po1gsIp/PI5/PY2JiAqZpwjRNbG1t4fr167h9 ++zaWlpawvLwMVVWxu7uL3d1dEQbncjmcPn0aZ06fwerqKu7du4f9/X2xr/01lRVFEfva388T+QnE +43HE43GxH1zHRa8/nCixsb6B9Y11HBwcoN1uj7XaD4fDSKfTSKVSqFarKJfL4rgAgN3dXXz++edY +WVlBKBRCMBjE1NQUZmdnkc/nkUwmkUwmsbGxgUePHqFUKqHT6YjH0DQNsVgMxWIRxWIRuVwO+Xwe +0WhUbKt/TB4cHGB9fR0PHz5Ep9NBr9eD67oIBoMIBAJYWVnB7u6uGFvP80Ro74+JH377EwpGJ4KM +td+Xf7h1vn+uDAYD3LhxA9999x3a7TZmZ2exsLCAo6MjlEol1Go1sT2xWAyJRAL5fB6Li4s4deoU +NFWDrMiwLAudTgetVgubm5vY2NjA4eEher0eut2uOEdisZg4z3K5HDKZzDNj1ev1cHR0hIcPH4qx +6na74pwLBoNYW1vDzs4OOp0OTNP8Sdez57UtZytz+lAx+CYiIiIiIiIiIiKiD4okSYiEIwgGg88N +iHZ3dxEOh0UlcigUQjQaRSaTeSb4Hv3ueR4ODw9x7do1/OEPfxDhq1+B7LouwuEwIpEI5ufn8Ytf +/AKKoqDgFhAMBkXwDQwreBuNBjY2NvCnP/0J33zzDer1OhqNBgAgFAohEAjAcRzYto1kMonFxUUs +Ly/Dtm2kUqkfbb2uKipg4JlKXD+M9NdbHv09AFiWic3NTXz77bcwDAPT09PD52g7UDUVDx8+xL/+ +67/i3//93/Hxxx/j17/+NRRFwZ07d7CysiLC0lAoJMLrwWCATCaDcrmMa9eu4Z/+6Z9QqVTQarVg +mibu3buHhw8fQpZlSJKEiYkJXL58GR999BGmp6cxPTWNaq2KP//5z/jmm29QqVRQrVZh2zYMw0Aw +GMTy8jLOnz+Pc+fOQdd1TExMwLIsdLtdPHr0CL///e/x//1//x9++ctfot1uIxAI4NatW7h7964Y +h6WlJWiahlOnTuHBgwf493//d6ysrIig2q8s9sPqSCSCq1ev4uOPPwYAKIo6Fnzbjo1er4dmo4m7 +9+7iq6++wtramth+PwROJpNirLa2trCzs4ODgwPUajV4nofd3V1UKhWx/2RZxrlz5/DRRx/hzJkz +mJ6ehud5uHv3Lq5duyYeo1arif2cy+Vw9uxZnD9/HmfOnEEsFnsmzG00Gtja2sJXX32Fr776CvV6 +HfV6HZ7nIRKJIBqNiuA4GAzCsiwAw+4Io5MoRivK/UkYT7fWf5l23I4zrMxutVr47rvv8I//+I+o +VCr46KOP0Ol08ODBA9y/fx/7+/vo9Xro9XrIZrMoFAo4c+YMJEnCzMzMcDtkCY7joNFo4OjwCDdv +3sTXX3+Nhw8fotFooNlsiueQy+Vw4cIFXL58GbZtIxwOPzNWzWYT29vbuHbtGr788ktUq1U0Gg04 +jiPGqtfrodVqQdd1MUngp3heW/PR+2IQTh8KBt9ERERERERERERE9MFRVAUKnh+s6dqw2tSvfPbX +vvbDuefxK68990nwFAqFYBjGc9fcbTabWFtbg2mauHr1KhKJBMLhMADAsixsbm5iZWUF9+/fx87O +Dvr9vqgU97dFVVURDnueh3Q6jXa7LSqPf4wkS1ChIpvN4uLFi+h2uzh79ixisRg0Tcf09DQ6nQ4S +iQQSiQRSqRTC4bCosu12u6Kttuu6cCQHsiuLNvH7+/t49OgRYrEYIpEIut0uIpGIaDMNANVqFb1e +D+l0WoSz0WgU8/Pz8DxPrCOeTCaRTqdFdXIul8PU1BRSqRQajQYqlQrK5TIODg4gSRLi8ThCoRBc +1xX7sN/vY3V1FZZlwTAMZDIZ0Qq71+uNbXM8Hkc0GkW1WoWqqggEAiLIfnqf+hXA/tg8HTiWSiXc +vHkTvV4PkixhcrIojpdyuYKVlftYXV3FgwcPcHBwANu2EYlEEAqFROt5wzAwGAywt7cHz/OQy+Wg +qqqoEo5Go2Kygz9Gc3NzmJiYgKIo2Nrawvr6uqiG94+lRCIhgnJd10V1s+u6iEajT9qP6wa2t7dx +//59rKysiA4EiqIglUqJMQoEAiLsHm3f7XdGSCaTOHfuHA4PD3H+/Hkkk0kEg0FMTk7iwoULiEaj +SCQSyOVyiMViP3oM+y3FXddFp9NBtVoVFemRSAStVguSJI09F8/zcHBwAEVRkM1mkc/nkcvlkEql +0Ov1cP/+fXz//ffY3NxEvV6HLMtIJBKIx+Njk1329vbQ7/cxGAxEmG8YBnRNx+7uLu7du4eVlRVs +bGyg0+mI49jvBuBPXPHH6nXDaYbbRAy+iYiIiIiIiIiIiIjeGN3QkUgkUCgUEIvFEIvFEAgERCXu +4eEhDg4O0Gq1sLGxgY2NDQQCASyfWkYmkxmu622aWFtbw7//+79jZ2cHjuMgFothYmIC+XxerF08 +GAywsrIyDFQl6ZW3VZIkQAYKhQL+6nd/hcXFRczPzyOZTEKWZZw5cwbZbHZsHXBZlkVY92M8z0O1 +WsX9+/cxMTEh2pNblgXLslAul7G7u4tHjx4hn89jenoahUIBy8vLKBQK+Jd/+RdUKhW4roszZ87g +F7/4BSKRCGKxYWv4bDaLdDqNr7/+Gt9++y2q1SpCoRAKhYJoC66qqmgRv7m5ic3NTTSbTcTjcUxN +TQ1bxmv62HaXy2XcuXMHqVQK0WgUs7OzSKVSSCaTmJmZQSaTAQCEw+HhPvM8sa/99aN7vR4ODg5w +dHSETqeDmzdvolarIZvN4he/+AVs24Zt2zg8PMDXX3+NL774QrTD91vqFwoFUdHvh7q1Wg1zc3OY +n58X7bs3NjaQz+dx5coVLCwsiGriTCaDbDaLRqOBb775Bt99953oYLC0tIRUKoVUKoV+v49ut4tK +pYLNzU2sra3BMAzkcjmEw2Ex8WF9fR3/8R//gYcPH8JxHNFyfmJiApFIRGz/ysqKmBQxeqxJkoRs +Notf//rXKBaLmJmZwURuAsFQEKdOnUIikRDV56FQCLFY7Ccd1/6xtbKygmQyicnJybHuCDs7O+Jr +ZWUFsVgMS0tLkGUZ7XYbt27dwu9//3tIkgTDMDAzM4NcLodcLodut4t2uy3WU19bWwMApNNpcQzE +43E8evQIf/jDH7C6ugrHcRAMBpFOpzExMYFoNCrGam1tDd1u97WqvZ8+515U9U30IWDwTURERERE +RERERET0hoRCIUxMTGB+fh6FQgHFYhHxeFyExisrK1hdXcXq6iq2t7dxeHiIM2fOoNUetvS2LAvt +dhsbGxu4desWGo0GJicnMTMzg+XlZZw9exbhcBjtdhuNRgPtdht7e3uisvlV+GFkJpNBKpnC5SuX +ocgKZGV4P+FwGFNTU5ClYRto0zTR7XTRaDZe+jFM00Sz2UQymUQmk8HFixeHVbqui/X1dRwdHaFe +r+Pg4ABbW1uIxWIiAH/48CG+/fZbeJ6HpaUl/O53v0Mmk0Emk0EgEIDnDScJdDodrK2todPp4PTp +02K98KWlJQQCAVSrVVQqFTSbTXz//fdot9vY3t7G1taWCH99fjV7rVZDIBBAsVjE8vKyeNyJiQmk +UilRVV4sFsVEh8nJyWHVr26gWqtiZWVFtNr2K6Q//vhj2LYtwub9/X3cu3cP169fRz6fx9TUFKan +p3H+/HlcunQJ/X4fvV4PpVIJq6uraLfbmJycxCeffIJyuYzvv/8eAJDJZHD58mV8/PHHYlv9cPX7 +77/HwcEBbt26haWlJeTzeZw+fRqLi4tYWlxCvVFHtVrF3bt3sbW1hb29PWxtbWFrawvJZBKWZUGW +ZWxubuL27dvY29tDsVjE1NQUlpeXcfr0abHGer/fR6fTwd7eHjqdzlhw7VdOX750GRcuXBg71qan +p1EsFsWx9vQyAq/CdV3R/rxQKGBubm5siYJr167h4OAAzWYTu7u7CIVCkGUZkUhEtEf//vvvUSwW +cerUKfF15swZUVG+traGvb097OzsIJvNYnFxEdlsVlTgb25u4vvvv8ejR49QLBYxPT0t1o7PZrPo +9XqiWnx/f19Ulr9Jfujtej/e/YHofcHgm4iIiIiIiIiIiIjoNUmSBEVRkEgkcOrUKVH9mUgkoGka +PM+DbdvI5XJi/d+trS2Ypoler4d2u41KpYpGo45yuYxyuYzBYIBwOIzl5WX86le/Qj6fR7FYFOsB +t1otPHr0CGtraxgMBmNrhL/y9ssSFCiQ5PHA0V9n2Q/J/d+/7Jjk83lcvnwZZ8+excLCAubn5gEM +wzhZlrG1tYXd3V3ouo52u416vY5CoQDDMESVPICxVtr+2uyVSgVHR0doNBqQZRnxeBwLCwtj4a+m +aQgGg4jH43jw4AE2NjbQ6/XQ7XaxtraGubk5hEKhsW2emJjAhQsXsLy8jJmZGczOzoo2535FryIr +KBQKuHz5MgaDARKJBGKxmKi2dVwHxeKwpXmpVAIwXLe93W6j1WyhXBnu462tLdGqPpfL4aOPPsKF +CxcwOTmJfD4P27JhWiaSySRSqRQWFxeHFcjZHJrNptg/sixD0zQEjCdj5Ld/39vbE+24c7kczp07 +h/PnzyOTySCeiEM3dEQiEfR6Payvr2Nrawu6ruPo6Airq6uo1Woi8B0MBqJi/NNPP0WhUEA+n0co +FBKV/Nvb21hbWxNt4H/KsTb6/VUFg0EsLS3h448/xuzsLGZnZ5FMJsXvW60Wms2maBv/6NEjsYa7 +4zhotVqiDfrFixdx5coV5HI5ZLNZRCIRJJNJaJqGw8NDHB0dQdd17O/v4+bNm8jn85iYmECpVEK/ +30cwGMTCwgJ+/etfi30aiUTEJJfd3V2kUimYpglN037ycyaiIQbfRERERERERERERESvyQ+G/bW6 +FxcXxfrMtm2LsNVfm9kPzEzTFO2TK5UySqUSdnZ2UC6XYZomstkszpw5g88++0y0HJckCZ7nod/v +4+7du0gmk2g2m88NGV9l2z3JGwve/J//VIqiYGpqCr/73e/w0UcfIZ1Oi0pkYFiR+uDBA6yvr0PX +dbRaLdTrddi2PdYeHhgG334b7EAggG63i2q1ivX1dTSbTTH2CwsL+OSTT8Qa6JIkiarl+fl5bGxs +oFQqodfr4cGDBwiFQigWi2PPuVgs4i//8i/xi1/8ArlsDtlcVqzzLssyVEWFrMgoFoui1bmqqpBl +WVTyAsPK42AwiHv37gGAqPJutVuiwn17exvNZhOe56FQKOBXv/oVfvGLXyAYDIqA3580cfr0aRGQ +KoqCza3NseBb13UYAQPBYBChUAi7u7vY29sTIaxhGCgWi7hw4QIuXboETdOgqcP1yV3XhW3bePjw +ITY3N0WwOxgMUK1Wnwm+l5eX8Xd/93cIh8LQDR2qqoptXV1dFWGuYRjPPabe9LE2KhQK4fTp0/j7 +v/97sXZ3MBgUv+90OmLt+FKphO3tbWiaBsMwoCiK+N3ExASuXr2KX/3qV9A0Dbquw3VdsV/39vaw +v78v1vv2W6DXarWx4PvUqVP47LPPEI/Fnxmr9fV1pNNptNttGIbxRp4/0YeMwTcRERERERERERER +0Rvgt8n2w69Wq4V2u41OpyPaQPf7fZimKdaa9tcd9lt212o1HB0dwbIsGIaBeDyOVCqFbCYrWkCP +VsTqui6C0NcNDn/s9j/l/gOBAFKp1LC6OB4fq66ORqOIRCIIh8OQZRmmaWIwGMC27WeCUEmShq2x +HwfQtj2szD08PEStVkO73YbjOLh//z4+//zz5+6b27dvo1QqodFowLZtOI6Dbrf7zJrl/trV+Xxe +VHI/7/5arRZqtZpoOd9sNtHr9cbaWJumKQJjVVXF2uytVgvlchmNRgOSJCESiSCRSCCdTiOdTouQ +/XmP62+7H3r74zP6Jcsy+v0+arUayuUyms0mOp0Otra28M0336BcLj9z34eHh3jw4AFqtRqCwSAs +yxKTFBzHQb/fh67riMViSCaTyGaz0HX9mX1lGIZo7f+iY+bnrGxWFEWsv+6v0x4IBMTvU6kUJiYm +cHh4iFKphGazKcbJD8jD4TDi8bhY3/zpbY5Go0in08jlcqjX6+h0Ouj3++I593o96LqOUCgkjv9w +OPzCsXrR/iaiV8Pgm4iIiIiIiIiIiIjoNfnr6dbrdbGG9+7uLnZ3d9FsNkXg7VeB++2jAYhqW78F +er1eh+d5iEajiMfjCAaDYi3kk9YK2Q9hZfnZYF6SJFFpK8syHMcZC1t/iOPYYqJArVZDvT5sEW9Z +FtbW1p57m6OjI5TLZUiShHQ6DVVVn/t4sixDVVVRxf00v1p3f38fKysrWF9fx97eHnZ3d2GaJvr9 +PjzPEyH91tYWut0uQqGQmOjQ6XTQaDTQ7XahqiqSySRisRgCgcAPTmKQJAmyJMOVfnyM+v0+Go2G +GKNms4k7d+6gWq0iGo0+8/fdbhflchmVSgWpVEpUsiuKAtd14TgOwuEwEomEWBf7TVZqv2kvapnu +TyiJRqNigoA/ccF/zvF4HJFIRCwf8PR9aJqGSCSCdDqNXq+HWq0mJqvoug7bthEKhcRkD7/7wLs6 +VkTvCwbfRERERERERERERESvwfM8WJYF0zRxeHiI1dVVfPvtt9jd3UWpVEK324Vt2/A8T7Qr96uC +/RDVdV0MBgN0Oh2x5nMoFEIkEhHtzU+iJxXIz26/HzD71cF+KPxywfewWrvVaqHT6WAwGIhxrlQq +L7yNH0iqqopgMCge++ltflHw7dgOLNvCYDDA9vY2bt++jZWVFezt7eHg4ACO48B1h+uX+/u61WrB +tm0ATyY59Pt9sd2qqiIajSIUCkHX9TdW+WtZlmij3+v1YJomKpUKut3uc9vi++G24zhj3QRs20ar +1YLruggEAqKCWpFfv8vAz+lF26ZrOsLhMILBICRJEpMV/HGRZVn8/kWTEGRZRjAYRCwWg6ZpsCxL +tFAPhUJirPzJDD9U/U5Ebw6DbyIiIiIiIiIiIiKi1+B5Ho6Oyjg4KOH27du4e/cuHjx4gFgshnPn +ziESiSAUCiEUComK70ePHuH69et4+PDh2P344aPneaJa+n0PzH7K8/MnDHieh2AwiEQigWw2i5mZ +GSwsLDz3Nv7YBgIBJBIJJJNJTE9PIxqNYn9//6Uet9vrolQq4eDgAHfu3MG9e/dQq9WQyWSwsLAg +9rO/XrSiKPjzn/+Mmzdv/ugYvOn97I+PoiiIRqPIZDKYmprC/Pw8UqnUc//eD+79Ft+e56FcLuPo +6GjseDzpx6TfocGffPG8yQ+vcj/+bV70RURvB4NvIiIiIiIiIiIiIqLX4DgOyuUjrK2t4d69e1hZ +WcHm5iauXr2KS5cu4fTp0ygWiygUCqLq+I9//CNKpdIzwbf/+w8p+P6pRoPvdDqNTCaDTz/9FL/9 +7W9f+PfAcA3o0bWVX2V8e70e9vb2cP/+fdy7dw9ra2uQZRkLCwv4i7/4C0xOTmJychKRSGRs/fb1 +9fWxkPTpbfo59rE/kcKvKM9ms7h06RJ+85vfvHBygL89fgv6crmMGzduoFKpjIXeJ/mYdD0XruuO +taN/3nHwY8/xRaG3P07vyyQBopOEwTcRERERERERERER0WvwPA/tdhsHBweo1WqwbRuBQEBUAS8v +LyOXyyGTycA0TZimiWQyCcMwxu5HlmXoug7DMNBqtTAYDNDr9USb7Pfdi6plRTt4zx0Lr4PBIMLh +MOr1uhijWCyG6enpHwwcZUmGrMiQZRmDwQD9fv+lt9E0TdRqNezt7aHT6UCWZUQiERSLRZw9exa5 +XA65XA7BQBCmNdzXkUgEiqKM7Udd1xEMBmEYBur1OlqtFrrdLizLeqkw/OkAerQC3qdpGkKhkGiV +b5omdF1HLpfDzMzMDz6GoiiiCtpfp7rdbouW4JZlHe/B8hpM00Sj0UC73QYAhEIhhMNhRCIRGIYh +ftftdse6L4xyHAe9Xg+NRgOmaYrW+aFQCMFgEP1+X7TfHwwGz530QERvHoNvIiIiIiIiIiIiIqLX +4Hkeer0e6vU6+v0+gsEgstksJicnMTMzg0K+gGAo+MxtnqYoCgKBAMLhMI6OjsR6368SzL4PJEl6 +pgLXr14eDb5DoRASiQT29/fR6/XQbrfF2sp+Je+Phcf+etYvy1/vulKpwPM8xONxZLNZTE1NYW5u +DsFgELquj93mefta13WxfrtlWajVami1WjAHJlzHhSQ/P7j3PG/4O0/60fDbMAzEYjGEQiHYto1G +o4F+vw9JkoZrdI+E2y8an0AgINaaPzo6Qq/XQ6vVQq/XO45D443odrqoVqtoNBoAhpMl4vE4EomE +mKBQr9fRbrdhmuaTcR/ZH7Zto91uo1KpYDAYiIkM0WgU0WgUjUYD3W4XiqJgMBj84Pb4+4vhONHr +Y/BNRERERERERERERPSaLMtCv9+HZVlQVRWRSATRaBSJREKE3v1+H+12G+12G/V6/ZlATNM0hMNh +JJNJbG9vo9vtotFooNFooF6vQ1EUqKoKz/Ng2zZ6vZ6oKLVtG67rHvcwvBF+m2g/vPY8D5Zlod1u +IxaLiXELBoNIpVLQNA2DwQDtdhvNZhPNZhOBQACBQACapon7HW0j74eMpmm+UuDoui5M00S324Xn +eTAMA9FoVISn/uMMg/gOOp02Op0OHMcZu59gMIhkMoloNCo6BtTrdZQrZVSqFbH9/rbatg3LsmCa +JgKBAAzDEOOjqipc10Wv10On0xHVxqqqIhaLIRKJwPM8MZGi1Wqh0+mIxxgNzm3bFlXOAODYwzXR +U6kUtre3MRgMxPFYrVVhGAZUdRg1+bf1g3HLsp553m+D67qwLAudTkdMRPAnTjiOg2qtioODA9G+ +PZ1Oi0p9vyq/0+mgXq+jVquhXq9D0zToug7XHbZJbzabqFQqODw8hCRJomI8k8kglUrh4OBAVHrX +ajXUajWEw2FxXD89VqZpHstYEb1vGHwTEREREREREREREb2mYDCIeDyOQCCAcrmMSqWCnZ0drK+v +w7IsyLIM13WxtbWF7e1t3LhxA6VSaew+AoEA8vk8dF3HxsYGTNPE0dER7t27hz/+8Y+IxWKIxWJw +XReNRgPlchl37tzBzs4OXNdFMpk87mF4baNVzH4VbbvdRrVaxf3799Hv9+G6rggis9ksIpEIVFVF +u93G3bt38f/+3/9DoVDA1NQUYtEYZGW41rIfQJqmKR4jHo+LwPplKIqCSCSCVCqFVquFRqMBWZax +ubmJtbU1UWVer9exvb2Nra0t3LlzB51OB6FQSNxPPB6Hoihot9tYWVmB4zjY29vDtWvX0O12USwW +MTk5Ccu0YFrD1tylUgmHh4dYXl7G2bNnoaoqAoEAIpEITNPE7u4u1tbWYNu2CFfF5ItgEJIkYW9v +D19++SXq9bpYj9yv/O73+yLoHW0zDwALCwvY3NwcBsfVKlZWVvDFF18gkUggHotDkofj22g08N13 +32FrawvtdhuJROKtVzJbloXDw0Pcu3cPhUIB2WwWgUAAnU4HnU4Ht27dwt27d1Eul5HP5zE3N4fp +6WnMzs6i1WqJ/XFwcIAbN24AgAjGu90u2u02Hjx4gNXVVWxubmJychJzc3OYmZlBMplEMplEqVSC +53moVqtYXV3Ff/zHfyCVSiERT0CSh8dHvV7Hd999h83NTdRqtWMZK6L3DYNvIiIiIiIiIiIiIqLX +4LeEjsfjMAwDvV4PR0dH2NnZwaNHj8aC75s3b+L27dvY2NjA0dHR2P2EQiGEQiEkk0lcu3YNlmWh +XC5jZWUFhmEgn8+jWCzCtm3s7u5id3cX9+7dQ6lUgq7r701LdL/i2zAMEVhWq1Wsra2JKutkIglN +05DL5cQa2o1GAysrK2g0Gjh79iz6/T4mJiZEsLu9vY3t7W20220oigJFUbC4uIilpaWX3jZV1cQ+ +2traEms8b29vY319XVRhl0ol3L59G3fu3MHu7u4zwXcikUA6nUa73RYV2QcHB7h+/TparRbOnTsH +VVHR7XXR7Xaxt7eHlZUVrK2todfroVAoiOA7FAphMBhgd3cXwWBQtIF3HAfRaBSpVEpUdh8cHODr +r79GuVzG2bNnxSQCP4Tf3t7Gzs6OqEyOx+OYn5/HqVOncP36dRF8r62twTAMFAoF5PN5SJI0dkzu +7OyIKvO3zbIsHB0dYXV1Fb1eD4PBAIZhoFKpoFKp4O7du1hbW4NlWZibm8OFCxcwMzODqakplEol +RKNRuK6Lo6Mj3L59G6Zp4tSpU5AkCdXqsFp8dXUVDx8+xO7uLnK5HIrFIi5evChanX///fdwXRf1 +eh0PHjxAMBhEoVBAoVCAoijY3d3F3t4e7t27h+3tbViWJboIENFPx+CbiIiIiIiIiIiIiOg1KLKC +ZDKJ2dlZ1Go1lEolHB0dodFo4MaNG9jc3BThYrlcHltjeZSmaQgGh23RZ2ZmcOrUKVQqFfT7fdy/ +fx8HBwfY2tqCJEliTWvTNKEoCnRdh6qqkGV5bN3nk0qWZGSzWSwtLcEwDDiOg/X1dXS7XRwdHSGX +yyGbzSKZTCKdTuPs2bM4OjqCbduo1Wp4+PAhut0uotGoCHGbzSYajQYURRGVuaqqvtJYBQMB5Cfy +6J0aTm7Y29tDv9/Hzs4O/vM//xOapkHTNPT7fTSbTYRCIei6/sxa2oZhIBwOo1gs4uzZszg4OIAs +yxgMBtjc3IRpmjg8PIRpmjBNE61WC9VqVbTE9jwP4XAYk5OTOH36NGRZRq1Ww/3799FoNLCzs4NM +JoNMJgNVVTE9PY3Lly+LFtt7e3twXRfValVMDDBNU7SKTyQSSKVSCIVCIjyfmZnB0tISdF2H4zh4 +8OAByuUytre3oaqqaL3vt/uXZXms1fzbYts2ms0m9vb20G63xTnTbrfFOIZCIYTDYczNzWFpaQkT +ExNIp9PwPA9LS0s4f/48JElCp9PB6uoq6vU6tra20O12xX0AENXeftV4IBBAMBjE9PQ0FhcXIcsy +PM/D+vq66AKh6zq63a5oB++P1Y+tSU9EP47BNxERERERERERERHRa5BkCdlMFqoyDP8ODg5QLpfR +brdx48YNUZnrr5XsV8hWq1VUKhVxP7Isi79bXl5Go9HAw4cPUSqVsLm5iZ2dHaiqimAwiFgshmAw +KNYTD4VCIkx/OmQ9iSRZQrFYxOXLl2EYBjY3N7G9vY2DgwPcu3cP+Xwe586dw/nz55HNZvHrX/8a +BwcH2NjYEH+7ubkJz/PGWqf7VeITExOYnJxEIpGArusvvV3BUBBT01MIhUM4ODhAqVTCzs4Odnd3 +sb29LarUo9EokskkTp8+jU6ng4ODg7H7URQFgUAAhUIBv/zlLxEOh/Ho0SNsbm6iXC7j8PBQtNn2 +PA+6riMSiSCZTCIcDkNTNRgBA0tLS2g0Gnj06BG2trbEsaLrOs6cOYPz588jk8lgeXkZ09PT2NjY +wKNHj1CpVFCr1XDnzp3heEsSFEWBYRjQNA2RSASxWAz5fB7pdBqxWAyLi4v45JNPEI/Hsb+/j729 +PZRKJXHcJhIJ0e4/Ho+Ln79tjuOg0+ng6OgIu7u76Pf7YsKA4ziIxWKYnJzE7Owszp49i4WFBUSj +UQQCATiOg0uXLkGSJDx69Ei0q9/a2oLruuJYMgwDiUQCc3NzuHLlChYWFkR3AVVVsbCwgF/+8peI +RqMolUrY398XY+Xf1h+rWCwGz/MQCoUYfBO9JgbfREREREREREREREQjNF0TwaUfUEUikRcGpH5L +6Eg0gnanjb29PVSrVezu7mJ/fx+2bSMQCCAcDmNiYgKzs7OIxWKoVqvodDpIJBIwDEOEppFwBAsL +CwCGa4ebpomDgwP0+32xbnM4HEYkEkE4HEY4HEYwGEQwGISmaa9cxfwq/MpUP4Q1DEOssa2qqgh0 +E4kEJiYmkEgkEAgERCX62DirGsLhMBKJhGg9HolEYBgGJElCPp/HxYsXAQDNZhPb29totVqwLAue +52FychK2bSOfzyObzYrW8Y1GA0dHR6hWq2Pt3/0qZl3XkUgkUCwWh4GnMazSTSQSyOfzP7jNgUAA +mUwGyWRSBPGWZWF3dxdHR0di4oKqqpifn8fZs2dRq9VQLpcRDAYRjUbF+uSGYSCbzUJRVBSLRXz1 +1Veikr/ZbKJer4uJDPF4HOl0GpOTk0ilUtD0YTi9sLAg2ugfHR2hUqmg0+nAtm2k02l0u11RgZzN +ZnHt2jUMBgN0Oh3R+ttnGAaSyaSo9PYnCCQSSQSDQczOzsJxHAQCAbiui4ODA1G17K9xn0gkRDCs +KAomJiZEqPx0h4Ofk+d5cF1XrA/f7/fFsTsxMSGqupeXlzFZnISiPtm2M2fOIJlMIhAIoNlsolKp +oNFooNlsinPUb29++fJlMakgmUyK+5iZmYFlWQiFQrh+/ToODw/R6/VgmqaYuOKv8+5fV45rrIje +Jwy+iYiIiIiIiIiIiIhGZDIZfPzxx6K6NhqNYmJiAtls9oW3kWQJChRkMllcvnwZsVgMtVoN9Xod +juNA0zRR4VssFtHr9ZDL5VAqlXDhwgUsLCwgFosNQ19ZQiKRwOzsLDRNQyaTwYULF0TFql85HgqF +8PXXX6NarYq1mv3g+Oeo+lbkYUgXjUbx8ccfi+rWS5cuDcNrWYEkS1hcXMTf/u3fYnJyEpcuXcLs +7Kxo+T0qFo/h/PnzYl1qSZKQTqdRLBaHv4/FMD09DUVRkEgkcObMGTiOA9u2xdrTc3NzYgKAv4Z2 +oVBAs9kUreB9/mSBdDqN6elpTE1NQdd1BAIBzM3N4W/+5m+Qy+Vw+fJlsc3Pa9Xth5J+EOy3uG80 +GiLUTiQSYk1nwzAwNzcHXddx6dIlpFIp8ZwBIBIJQ5YlnD9/HuFwGOfPn0e320Wn0xFt2kOhENLp +NDKZjJg4MQzOcyIcLxQKqNVq4jiZnZ3F3NwcJiYmEAlHEI6EcfbsWYTDYZw5c0a0/vb5x08kEkGx +WMTU1BSy2SwikTAkSUI8Fsf09DQ0TUMqlcLy8jJs24Zt29B1HZlMRoTtvV4PkiSJiSNTU1OIRCJv +5fwNBAKYnZ3Fr3/9a7iuK6q9/YkM+Xx+eB4WisjlcpDkJ5MbFEVBLBqDJEm4fPky4vE4Dg8PxXNS +VRW6rouq8cnJSUxMTIyt3z567Oq6jng8jqWlJViWBdu2xTmdTqfR7/fR7XYBDCdm+MdNLBZ7K2NF +9L5h8E1ERERERERERERENCKXyyEajeLKlSuiStQPNF9ElmV4kodsNoNYLIqzZ8/CcRxYlgUAkCQZ +siyJ+3FdFx9//DFM00Q4FEYwFISu6ZCVYTiXTCZFuHbp0iWY5vB+PM8V1dGWZePo6Ai3b9+GZVki +lP65KkZlRYYhG9A1Hb/61a9w4cIFSJKEUCiEYDAIWRqG7UtLSygUCvjrv/5rhENhhMLD0Pvp6ulE +IoELFy5g+dSy+JmqqaLiO5lIIhKJIJ/P4/z58xgMTHieO/w7VRVV8n44nM1mUSwW8dFHH8F1Xdi2 +Dc/znty3v+60qkHTNbFNkiSJavy/+qu/QigUEhX+T2+z//eSJGF+fh6FQkEEmrZtP/6dDFVVRGv1 +5VPL6PX7kOXhWIVCobH1nCPhiFhL+8zpM7BsS2y//zf+JANRLa4PJ0hksxmkUklMT0/jV7/6FWzb +gee58DxPjI+/xrgsy4hEIlheXoZpmuIxRp+bP0a6rot14xV5eCzF43GEwiEUCgWcO3cOlmXBdT14 +ngtJkqFpw+1zHAeu+3g/KSqUx2Ohay/fUv51BAIBLCws4L/8l/8i2omPTmAYfW5Pr7+uaZro3pBK +pXDu3LnH66K7cF3nufvX/xoViw6XIsjn8zh39hwG5uCZsVIVFa7nHutYEb1vGHwTERERERERERER +0YnjPg5Afw5+MPaq/HWk/QD6p/I8D61ma1i96zqivbIkSZBlGZZlodVsoVqtolarodvtimrdQqGA +RCIBVX3zH//7gS/kYUXri6pSg+qw7fqPEYHhCwqBFVWBoipizeyX8VPXlNY07aW2eXQs/LbmP+pH +Cp0VVYEC5QfH4kX84/Rlt/15Fewv66fsj+OgKAqCwaBo2+5Xx78MSZLE/nid83h0rIjo7WHwTURE +RERERERERET0DnEdF/ulfayursI0TSSTSSQTSajacB3tVquFUqmEnZ0drK+vo9lsIp1OIx6PY3Z2 +FolE4icF90RERCcZg28iIiIiIiIiIiIioneI4zo4OjrCysoKOp0OUqkUUqmUaHPdaDSwu7uLnZ0d +lMtlyLKMWCyGTCaDQqEgWjATERF9SBh8ExERERERERERERG9Y1qtFsrlMvb29vDw4UMAEGt7O44D +0zRh2zZSqRSKxSIWFxexuLgIwzDG1o8mIiL6UDD4JiIiIiIiIiIiIiJ6h3ieh263i3K5jEePHqFe +r6PVaokw2zAMRCIRxONxXLx4EVevXsXy8jJmZmbEWsYMvomI6EPD4JuIiIiIiIiIiIiI6B0iSRIm +JiZw9uxZhEIh1Ot1NJtN8TvDMBCNRpFMJnH27FmcOXMGU1NTiMfjDLzpg6LICjRdQzAYxNLSEj79 +9FP0+32cPn0a4XAYhmFAluXj3kwieksYfBMRERERERERERERvUNUVcXS0hLCoTAazQYGgwEGg8HY +7w3DgGEYSKfTyGQyiEQiotqb6EMhKzIMw4Cqqrh69SrS6TRs28bMzAwSiQR0XYeiKMe9mUT0ljD4 +JiIiIiIiIiIiIiJ6h8iyjKmpKUxNTR33phC90yRJgq7rAIDTp0/j9OnTx71JRHSM2N+BiIiIiIiI +iIiIiIiIiIhONAbfRERERERERERERERERER0ojH4JiIiIiIiIiIiIiIiIiKiE43BNxERERERERER +ERERERERnWgMvomIiIiIiIiIiIiIiIiI6ERj8E1ERERERERERERERERERCcag28iIiIiIiIiIiIi +IiIiIjrRGHwTEREREREREREREREREdGJxuCbiIiIiIiIiIiIiIiIiIhONAbfRERERERERERERERE +RER0ojH4JiIiIiIiIiIiIiIiIjrhJEmCJEni3wAgS7L4N9H7jsE3ERERERERERERERER0Qnkh93P +C70ZeNOHhsE3ERERERERERERERER0Qkmy8PKbv+7+JL9EFyGLDEWpPcbj3AiIiIiIiIiIiIiOnEY +4BARDT1d9f109fdo5bcs89pJ7y8e3URERERERERERER0IrGNLxERRKW3oiiQZVl8jYbfsszrJb3/ +GHwTERERERERERER0YnxdPUiEdGHzA+9NU2Doijiyw/CR6+XEsNves8x+CYiIiIiIiIiIiKiE4cB +DhHRk+BbVVXxNVr1Lctc25s+HOpxbwARERERERERERER0asarfyu1+v49ttvEY1Gj3uziIjeiuvX +r6NerwMAFEWBruvQdR2apokv0fpckZ+7/jfR+4bBNxERERERERERERGdKKPBjSRJKJfL+MMf/oC1 +tbXj3jQiordif38f1WoVkiRBVVUYhiG+dF2HqqpQFGWsApzofcfgm4iIiIiIiIiIiIhODFl60r43 +HA4jHo+j0+ng4OAApVIJnueJLwBj//Y9/d9EL8t13bFjzHVdAIAsj1fUMmSkn8vYmt2ShHg8jlAo +hGAwKEJvTdOeaXvuH5di0hCXi6D3EINvIiIiIiIiIiIiIjpRJEmCYRg4f/482u02Dg4OYFmW+DJN +E5ZlwbZtWJYFx3Fg2zZc1xVfo+E3g3B6Wf6xNPoFYGx9Zb/Sluh1PR1y++G1LMuimjsajWJmZgaR +SAThcBjBYBCBQACGYQxbnqsaZFmBIitjnTKevn+i9wGDbyIiIiIiIiIiIiI6Ufzg+8KFC8jn82i1 +Wuh0Ouh2u+h0OuLfvV4PvV4Pg8EAg8FAhOGO4zxTucvwm16G4zjodrvo9/viCwACgYD4CoVCDL7p +tY2G06MV2/7kCr+teSgUQjweRywWE6G3ruvQtWHlt6IqUFVlbJ1vWZIZetN7icE3ERERERERERER +EZ0okiRBURTE43FomoZ4PD4Weo+G391uF4PBAKZpwjTNsepvBt/0qkzTRLvdFsebqg5jlnA4LL4i +kQg0TTvuTaUT7nnBt1/lrSiKCL79yRahUGis4ls3nqzzPVolzsCb3mcMvomIiIiIiIiIiIjoxJBk +CZL7OABSHlc+6oao5PY8b6wyUtd1mKaJwWAg/sav+PbXZ2boTS+r3+8Pq2gfV3S7rgtJkhCJRBCJ +RBCLxRCLxWAYxnFvKr0nXtTiXNM0EX77YbcfgPutzkfX+x5bh57re9N7isE3EREREREREREREZ0o +IghSZGiaBtd1EXADAABZehIOaZqGQCAwtua367oiIPe/Awy/6eX0+/2x0NtxHADDiu9YLIZEIoFE +IsHgm17baGW2X6ntX9v88FvXdei6LoJuv91+MBiErutQ1Seht7gPST7up0b0s2HwTURERERERERE +REQnhh8GiXBb1cZalfuBuL8Grl/pbVnWWKW3H3wDDL3p5fV6PQAQLfNN04QkSSL4jsfjSCaTCAaD +x72p9B4Yvd6NVn37HS00TROV36MV4H61t65rot25aJkuS2x3Tu8tBt9EREREREREREREdKLIkgxP +8qCq6lhoPRoG6boOy7IQCATG1vX2q7y5tjf9FJ1OB67rwrIsmKaJfr8PAIhEIohGo0gkEkilUgiH +w8e9qfQeES3Kn6r69jtbPPs1DL39Nud+1bcsyWNrhxO9bxh8ExEREREREREREdGJIskSFCjwvGH4 +LX4uSZDlYRjkB99+6O0H3q7rwnOHYbfruQy+6ZUoioJer4dut4tut4tAYNhiPxgMIhQKIRKJIB6P +M/imN0ZUfUtPqrUlSRLtzkdbn/sTf/xAfLTae3Rtb4be9L5i8E1EREREREREREREJ4YIbGRA9sbX +qpUVGYpiQ1UVuK4r1v92HGes0hvAC78T/RBJktDpdNDtdtFut8Va3oFAAOFwGJFIBLFoDOEIg296 +PaPh9GiVtv9vRVagqMpY63NFUcTP/TBctEh/XO3N0JveZwy+iYiIiIiIiIiIiOjEkSQJsiQD8pP/ +dhwHiqKIdbz9Nb39Sm/XfRJue577+DsDb3o1rVYLgUBArKMsSRICgcCTqu9oBJFI5Lg3k94T4wH4 +8IIny0+1PpcVyIo8tga4/+VfK/1qb6L3GYNvIiIiIiIiIiIiIjpRJEmC53mQZAmyOwy//RDIr+oe +DcE9zxPtzYFhi3Mfg296FZZlicDbbyUNAKqqQtd1GIYBwzBEC3SiN2G03bn42Uj4Laq6R8JuSZKg +yMpYe3NWe9P7jsE3EREREREREREREZ04ovWvIkHyJHiuNxZ8y7L8TBtz13XH7oOhN72qwWAwFnr7 +wbf/s2H4HWDwTW/U8wJrWZbF757+Gq3wHm2TTvS+Y/BNRERERERERERERCeSqPyWJEAGFCiistv1 +3GeCb0VRxm7P4Jte1Wjgrarqk1bSsgxFUaBpGjRNhaZpx72p9B55UWj9dKjNwJs+dAy+iYiIiIiI +iIiIiOjEejrUkZTHYY83/D7a4pzodSmKIr78ttJP/1xVVGgqg296e54Ou8XPGXrTB4bBNxERERER +ERERERGdeM8LfDzPE0E40ZvgV3mPrqP89BrLiqpAVuTXfzCil8SAm2iIV14iIiIiIiIiIiIiei8x +DKKfw2gL6ef9m+ht4nFH9AQrvomIiIiIiIiIiIjovcVQiN6k562bPPpvv/U5jzsiorePFd9ERERE +RERERERERERERHSiMfgmIiIiIiIiIiIiIiIiIqITjcE3ERERERERERERERERERGdaAy+iYiIiIiI +iIiIiIiIiIjoRGPwTUREREREREREREREREREJxqDbyIiIiIiIiIiIiIiIiIiOtEYfBMRERERERER +ERERERER0YnG4JuIiIiIiIiIiIiIiIiIiE40Bt9ERERERERERERERERERHSiMfgmIiIiIiIiIiIi +IiIiIqITjcE3ERERERERERERERERERGdaAy+iYiIiIiIiIiIiIiIiIjoRGPwTURERERERERERERE +REREJxqDbyIiIiIiIiIiIiIiIiIiOtEYfBMRERERERERERERERER0YnG4JuIiIiIiIiIiIiIiIiI +iE40Bt9ERERERERERERERERERHSiMfgmIiIiIiIiIiIiIiIiIqITjcE3ERERERERERERERERERGd +aAy+iYiIiIiIiIiIiIiIiIjoRGPwTUREREREREREREREREREJxqDbyIiIiIiIiIiIiIiIiIiOtHU +494AIiIiIiIiolfheR48z4Npmmi32+h2uwiFQgiHw9A1Xfxdp9tBt9sFAIRCIQSDQSiKAkmSIEnS +cT+NYxkzz/XEuHieh3A4jHAoDEmW3rlx8TwPruPCdmz0+330ej14nodQKIRQKATLstDv9WE7NmRZ +hqIoMAwDhm5AUZXj3vxj97zxcxwXkUh4bPws24KiKFAUBZqmwTAMaJp23Jv/XIPBAL1eD5ZlQZEV +KKoCXdehazr3ORERERERMfgmIiIiIiKik8WyLJimie3tbXz77be4desWLl68iCtXrqBQKEBVVciy +jK+//hrfffcdAODy5cu4cuUKAoEAAoHAOxvs/Vxcx4VpmRgMBrhx4wauX78O27Zx9epV/OKjXyAQ +DLxzgbFlWWg2m2i1Wrh37x7u3r0rtvnq1avY2trC6uoqGo0GQqEQIpEIZmdnMTc3h0wmc9ybf+xs +20ar2UKz1cT9+/dx9+5d9Pt9XLlyBR999BH29vawtraGSqWCcDiMSCSCqakpzM/PI5fLHffmP9fO +9g7u3L2Dw8NDxGIxxGIx5PN5TE1NcZ8TERERERGDbyIiIiIiIjpZbHtYwfro0SP827/9G/75n/8Z +n332GaLRKMLhMFRVhaZpuH79Ov7hH/4BwDBEnZubQzweh67rr7cBJ5DjOuj3+2i322JcTNOE4zhY +WloCgGHV7zsUfNu2jUa9gdJBCV999RV+//vfw7IsWJaF+fl53Lt3D1988QX29vaQTqeRTqfR7XaR +TCQZgvrj12xgf38f165dw7/+67+i1Wqh3+9jdnYWq6ur+Pzzz7G5uSnG7+rVq0gkEu9u8L27g6++ ++gorKyvI5/OYmJjAuXPnEIvFuM+JiIiIiIjBNxEREREREb19nucBACqVCjY3N7G1tSV+l0wmMTs7 +i9nZ2ee233ZdF67rot/vo16vo1QqoV6vo9/vw7IscZt2u41qtQrP89But2HbNhzHEY/9vvDbmG+s +b2B9Yx22bWNychLT09Oi9TcAeK4Hx3HEuJimiVarBdu24XrucT+NH3x+3W4XtVoNpmmi2+1CkiTY +to1ut4tWq4VAIIBgMDgM813nuDf5neEfG71eD/V6HY1GQ4yfZVnodDpot9tj4+e67+6xMBgM0Gw2 +Ua/XxTb3ej3Ytv1WHt91XZimCdM0sb+/j52dHXQ6HRSLRUxOTiISicAwjLc+ucafENJqtbCzs4Pt +7W3EYjFMTk4il8shEAjAMAzIsvxWt4tOttFlRdbX17GxsQHDMDA1NYV8Pj98feFxRURERO8YBt9E +RERERET01vnrTe/v7+Pzzz/HF198AQCQJAnz8/P47LPPMDM9Azz+PP1dWnv6XTK6jvOt27fwz//8 +z+h2u/j1r3+Nv/qrv0IymUQikTiRwYQkSVA1VbSm9yc0qKoqAhfDMBAIBKDrOgzDgKqqUOR3p2r9 +OPljFQgERPt/MX66Idr+67o+Nn48117McRx0O100W03cuHEDf/jDH1AqlfCrX/0Kv/nNbzA5OYlU +KvXWg+/BYDAMvbd38PkXn+NPf/oTpqen8Rd/8Rf46KOPkEqlxDFA9LI878lkqWvXruFf/uVfEI/H +8Zvf/Aaf/sWniMVjUBTlg+yiQkRERO8uBt9ERERERET01tm2jcFggFKphJs3b+I///M/xe8ODw8x +NzeH3/72tyKUYxj3fJ7nwXEduK6L9fV1fPXVV2i320gmk7h06RICgQAikciJDCaGIa0mAtnR4Nav +YPWra/1/a5oGSeaxAgzHT9O0sYkDsiwPgyrjSdj99PgxHH0xz/NgWiY6nQ52dnbw/fffY2dnB+l0 +GmfOnEEymUQsFnvr2+U4w6UMqrUqHj58iO+++w6NRgNTU1M4tXQKoVDovet0QW+H53no9/tYXV3F +n/70JySTSeRyOZw9exaariEajR73JhIRERGNYfBNREREREREb1271Ua5Usbh4SF6vR5kWR5rq1qr +1bC5uYlUKo10OoVAIHDcm0xv2TC4VUVVsqIokGUZqqqKCRGjoS2D23FikoDxZPz8n42G3qOTCDRN +Y8X8D5AkCYZhIBKJYGZmBleuXEE+n8fp06eRzWYRDoehqm//ozZN0xAMBpHJZLC8vIxyuYzp6WnM +zs4iFo8hEAhw8hARERERfRAYfBMREREREdFb5Xkemq0mSqUSDg8P0e/3IcuyWLvbD763t7cBAJFI +mMH3B0iW5bGKZVmWx4Lvpyu+R/+OnlR8G4HHgfZzJg48t+Jb4fi9iKIoMAwD4XAYc7Nz6Pf7qNfr +OHXqFLKZLCLhCDRNO5btCgaDSKfTOHfuHBRFQTqdxuzsLOLxOAKBABSFExqIiIiI6P3H4JuIiIiI +iIjeukajgZ2dHZTLZSiKgkwmA9u2Yds2gsEgOp0O1tfXEQgEMDEx8cYf37Ed9Po9mAMTR+UjlMtl +tNttDAYDmKYJRVGgqipCoRBSqRSSySQikQii0ShM00SpVEKpVIKqqtA0DZFIBOl0GtlsVjxGu93G +4eEhDg4OAAyDyHA4jFwuN/acut0uDg4OcHh4CFmSIcnDv8tkMshkMj9YqdlqtbCxsYFHjx5hZWUF +7XYbvV4PDx8+xL/9278hkUggFoshHA6LgPPo6AimaQ7XK+52US6XxQQE0zRhWRZs20Y4HEYkEkEi +kUAmk0E2mxWhsj9BwbIslMtllI/KaDQbGAwG6Pf7UBRFhHGJRALxeBzxeByxWOylJzHIkjzW1lzT +NNi2/dzK5bFW55IE13VhWRZM00SlUkG5XEaj0UC/3xcTLcT2xRNIJIfjFI/H4Xkejo6Gx0S320Wv +14Ou65iensb09LRYB9sfg8FggP39fezs7MA0TRSLRRSLRQSDQRE4+ttSrVZRLpdRr9fFWEmSNLYt +8cSTsQoGg2I8/P3S6XTGtq/f78NxHPGcwuGw2HehUAihUEgE3M8LwEfXSn/ZiQN+2+xOp4P9/X3s +7++j1+uJ48rzPLiuKx7DP48ymQxCoRCCweBY+33LsmBZFjqdDg4PD3F0dDR2f5qmQdd13Lx5E/v7 +++j3+7Asa3guP27zXS6XsbW1hc3NTUQiEWQyGQSDQdRqNdTrdViWBc/zoKoqUqkUUqmU2A7P88Tj +DgYD2LYNWZbF+ZNMJpFMJhEOh8Wx2el2UCqVcHR0hFQqhfn5ebieCwBi8k61WkWz2USz2US3233u +uKTTaaTTaYRCIVGZXa1WUavV0Gg0Xuq2fpW5YzuoVqvY2tqCZVkoFApjLc7949U/hqrVKrrdLgaD +ARzHgaIo0LRh++pkMolEIoFoNIpoZNjO2nZsdLtdPHr0CFtbWwgGg8jlcojFYmKcB4OB2DeRSATh +cBipVAq5XA7xePylKs89zxOvBwcHB9jc3ES1WsXExATy+bwY33a7LY7lVCqFQqGAdDotrk31eh3l +chnValWcbwDEutSJREJcn6LRKCKRyNhrhGmZ6HV7ODwaHhudTkccH/61399eSZIQi8WG9xOOIBwJ +Q1EUbG9vY2dnB7FYDHNzc5icnIQsyZAVWYyFZVliIphlWeJ4z2QySKfTUGQFpmWi3++Lc7/T6Yhr +tn+9jUajiMfjw+cUiyMai4ptdGwH1Vp1eEw2mmi2muh0OmLCmT+pIxAIIJVKIZ3OoN/rYb+0j/X1 +dayvr2MwGKDZbOLevXsIhUKIxWKIxWLifEqlUmI8T+LyGkRERPR+YPBNREREREREb5XneWg2m9jZ +2UGlUoGu65iamhJhRTAYRK/Xw8bGBnK5HAaDwRvfBtMy0Wg0Ua/XcPv2bdy5cwe7u7toNptot9si +bM1ms1haWsKZM2cwOTkJTdPQbDZx8+ZNfPPNNwgGgwgGgygWi7h48eJY8N1oNHH79m189913olo5 +n8/j6tWryOVyIvRoNBq4e/cubty4IUKciYkJXLx4EZlM5gefR7VaxbVr1/DFF19gbW0NzWYTpmni +7t27ODw8FGFwPB7H1NQUpqensbe3J8LfVquFUqmE7e1tbG5uisCx3+8jl8uhUChgYWEBly5dQiad +gScNAx7bttHr9dBqtXDv3j3cvn0bm5ubqNfrqNfroqI4lUphYWEBi4uLmJubw/zc/EsH35IsidDG +3x+jwbcf0vlhzZNW54oI9Z/ePj9M9IOlZDI5tn2KosDzPKyuruLWrVsiHIxGo/jtb3+LXC4nQivb +ttFut9FqtfDNN9/gT3/6E5rNJj755BP85je/QS6XQzqdBjCc3NBut8W2PHr0SIyVH2SObcvsHObm +58aCb//x9vf3cfPmTdy5cwflchm1Wg2DwUAE2/5+m56extLSEk4vnxbt4v3gG8DY+P2UVvGu46LV +auHmzZv485//jGq1KoJBx3HEJJZQKIRsNoszZ87g/PnzKBQKyGQyzwTfrVYL+/v7+O6773Dr1i1U +q1W0Wi1YliWC/KOjI+zv78M0TZimKQLSXq+Hvb09fP755/jiiy8wMTGBCxcuIJ1O4+HDh3j48CG6 +3S4cx0EgEMDy8jLOnDmDdDqNWCwG27Zx8+ZN3L59G43GcAKHqqrDY3Z+XlwDQsEQHMeBZVvY29vD +jRs3sL29jUgkgtOnTyMej4tJF1ubW7i/ch+bm5vY2dnB0dERHMcR2xCJRMbGJZ/PI5VKQZIkbG1u +YWV1BZubm9je3sbR0RFc14Vt2+K2mUwGZ8+exfnz55FIJBAMBtFqt/DgwQN89dVXWFhYQD6fx+Li +IgKBADx3uFZzrVbD/v4+vv/+e9y9e1dM+un3++I8mp6exqlTp7C4uIjp6WlxHPZ6PZRKJfzhD3/A +F198gXQ6jUuXLmFqagoPHjzAxsYG6vW6CJjz+TwKhQJOnz6Ny5cvi4klPxZ+e56HwWCAXq+Hu3fv +4ve//z0ePHiAK1eu4Be/+AWazSYePHiAvb09ManjzJkz+NWvfoVUKoV+v49WazgWt27dwoMHD8S5 +DwC6riMajWJ+fh6Li4uYn5/H7OzcWPBtWiba7TbK5TKuX7+OW7du4eDgAK1WC71eT1z7AYhJA9PT +05iZmUGhUEA+n0cgEMAXX3yBP/3pT5iamsLf/d3fIZPJDK8/yvjx//DhQ3zzzTfivuPxOM6fPy8m +LbXbbVSrVdy6dQvff/899vf3Ua/X0el0xBjk8/mx61kgGBDXUNuxsbu7i5WVFWxsbGBnZwelUkmc +q/6YJBIJnD17FhcuXEC5XMZ3332H27dv4+HDh2Iyyo0bN7C1tSWuo8ViEWfOnMHZs2exsLAgrjdE +REREx4HBNxEREREREb119Xodu7u7KJfLCAaDmJ2dxWAwENV0g8EAjx49wvz8vAhR3hTP89Dr9nBw +UMLm5iZu376NGzduYHd3F51OB91uV3ygn81mRSW44ziIx+NoNBrY2NjArVu3RFDUaDQwMTExFuq0 +R0Iovxp3fn4eExMT+Oijj8T2tNttPHz4EF9//TUMw0AoFEK328Xk5OSPhkTdbhd7e3u4f/8+KpUK +TNMU1aGtVguSJEGWZcTjcQwGAxHcO44D13VRrVaxsbGB9fV1rK6u4uDgAN1uF91uF5lMBhMTE2g2 +m4jFYjh16tQwHFU1DAYDHB0doVQq4d69e7h165YIc1ut1liY608m8DwPiXgCkWgEiqxAUX+49bIk +SdDUYRAbi8WQzWYRCoUQjQ6rTwOBABKJBAaDgajcDIVCUFUFg8EA5XIZ+/v7ImxeX18XFbR+tWYy +mUSj0UC73YbruojH49A0TVSZbmxsYGtrC5FIBLlcDh999BG8hIdAIADbtlGtVrG3t4c7d+7gm2++ +QbvdRiaTwYULFxCNRkXla7lcRqlUwv3798W21Ov1sW2Jx+NirFzXFZXffsjfbA6XB3jw4AG+//57 +fPvtt6hUKmKyw+gx6+/HVCoFRVUQj8eRzWZFVaosywgEAojH40in06K692XXqPY8D67notvpYnNz +Ezdv3kS5XEav1xPniuu6Yg3xTCYD0zQhyzIcx0EwGEQsFhPVyK1WC3t7eyKo/Oabb0Q1suM4Ivg2 +TROdTgeqqooKbsdxYJomer0ebt++jT/84Q/I5/MYDAaYnJzEysoKHj58iHa7DdM0EQgE0Gw20ev1 +MDExgUwmA8dx8P333+PGjRtoNBro9XqQZVlU13qeh2w2i2w2K55bpVLBxsYGNjc3ce7cOfHc+70+ +ur0uNh5t4ObNm9jY2MDu7i6q1Sps24bjOKJdei6XE9XKuq5DVYch5aPNR7hx48YP3jabzYrb+uPY +7/ext7eHtbU1yLKMSqUiOgU4roNWq42dnR0xzjdv3hQTFkzTFBNW/IriwWAgOnIoiiKC8zt37uA/ +//M/kU6nYVkWGo0GVldX8fDhQ9RqNfR6PbiuK7pb2LaNXC6Hqakpcbz/0HXNc4cTGvr9PjY3N3Ht +2jXcuXMHvV4Pmqah1WphZWUFe3t7CAQCCAaDMAwDZ86cgWVZqNVq2Nvbw8rKCr7//nvcv39fnPvA +cE30cDgszje/w0Umk4aqqJAVGZ3OsKJ/Y2NDnG8HBwdiXPzg2+84AABLS0tiEoHf4ePOnTv44x// +iPn5ebF9/gSb0YrvnZ0d3Lp1C51OR0xsSKVSOHXqFGzLFhOU7t69KzofNBqNsderUqkkOgTIsoxM +JgNZluF5HtrtNjY3N3Hr1i2srq5ib28PR0dHY8dVKBRCMpmEoihIJpM4PDzEgwcPsLa2hmq1Kjow +OI6DWq0mXl8ajQYMwxBdHRzHeaOv2URERESvgsE3ERERERERvTV+UNVoNLC3t4dGo4FkMikC7sFg +gGq1iv39fRweHqJSqaDf78N1XUiS9FJtcn+I4ziwLAuHR4e4fv06vvrqK3Q6HSj/f/buszmOK0sT +8JumvPe+gIKnp1xL6umZ7o3pmf0N+x/2x23sp52Ynu6enmkjegIgAcIXCuW9zcqsNPuheK8AEjSS +KJFUnyeCIYlEATezMhMU3nvOkSTk83leocpaU+u6jm63iz/96U+YzWbwer08ZIjH41AUBePxmIc9 +549TVVX0ej3U63UeELjdbgyHwwuhB2tf3W63kUgk4Pf7EYlE4PF43ni8wWAIt2/fhq7ruH//Pra2 +tiDLMvL5PJaWlnhbafY5Y7EYms0mDg8PMRwOUS6Xce/ePUiSxENA1u59PB6j3+/j8PAQ6XQai4uL +PIzp9Xp4/Pgx7t69y4PuQqHAW2ezMIUFkvfv3wcA+P1+OJwO+Hw++P3+Nx6fIAqQIGF9fR3/83/+ +T0ynUz7DOJFI4ObNmxgOh/B6vfOwKBKF0+m8sL7BYABRFPn67HY7r55l63vw4AEAIBAIIJ1OIx6P +4/PPP4dpmmg0GhiPxzg7O8PDhw+xuLiIXC6H2WyGg4MDPHjwAPV6nYfL+Xwe6XQawWAQNpsN3W4P +W1tbuHPnDgaDAQRBwMLCAtbW1i5dy8OHD2FZFm8L73K54PF4cHBwgDt37mBvbw+dTgeiKCKbzcLj +8cDhcFy4xg3DwGAwgKqqEEURKysr+O1vf4vhcIjr16/DZrMhFovh5s2byOVy/PzFojHezvttyDaZ +b0pgxyvLMkRx3saZ3R+6rqNarfJQLhgMIplMwjRMmJaJk5MTfPPNN9jZ2UGr1eKt5dmxybIMWZZR +rVZRLpff2AViPB7zdt9OpxPXrl3DZDLh52Q8HuPhw4e8pbfT6YSmaVheXoaiKDzEns1m2N3dhcfj +wcLCwnzNpvnKYG80GqHf76PVamF3dxc7OzsAgEKhgGvXrvFnGHu/7XY7H+1Qr9d5t4Xd3V3s7u7C +siwsLi5e+lqbzcZfO5lM+IaQVz1zdV1HqXSKv/3tb9je3uYBaygUgtvtht1u55X0hmGgVCqh2+3O +N6w831hyvm06MN+wcHh4CE3TIAgCCoUC794xnU4xGo34Zo1sNsuDUVb1/F1omoZqtYr79+/zayKf +z/MxCoVCAX6/H4qiYGdnB3fu3OGtw7PZLJaXl3lbe9ZGfTKZYGdnB7qu87EObJNFuVzGN998g62t +LbTbbciyjFwux1vSs2u81WrxbiHvGmtv32g0cOfOHWxvb2MymcDr9WJtbY2fB/a8ZRsT7t69yzdF +hEIh3vadXVeKoiCbzWJjY+PCdTWbzXhgrygKvF4vbt26hXg8jvv372M6ncJms2FhYQELCwt8nEMy +meTdEVKp1IXnESGEEELIT42Cb0IIIYQQQgghPwnLsvgP8ln16nQ6hcPhwNLSEp+/LAgCisUiKpUK +Wq0WpsoUlmlBkH5Y6A3MZ1PrMx2tVgubm5v4wx/+gEQigUQigVwuh+XlZaysrPBKz6OjI2xtbWFv +bw8ejwe5XA7JZBIOhwPJZJJXrZ8Pvlk4pGkaer0ems0mAPBq29FoBMu0eNvw2WyG0WiEXq+HWCzG +A9S3CSDD4RA+/fRTLCwsYDab4eTkBKIo4sqVK/iXf/kXRCIRPsvYsiyYhom9vT04HA40Gg1UKhVM +JhMsLy9jfX2dV7QahoHHjx+jVqthOBzy4Ju1Wu52u9jc3MTvfvc7RKNRZLPZ5+2CF1AoFDAYDNDr +9XB8fIx79+5ha2sLTqcT2Wz2+dzyeRX367BAxhIsrK+vI5PJwDAM+P1+yLKMZDKJYDDI5+2yObei +KOL09BRbW1v4/e9/j1AohEwmg8XFRSwsLGBpaQnD4RC9Xg8nJye4e/cutre34XA4kMlkEAgEEI/H +sbi4iGazic3NTXS7XZTLZWxtbQEAgsEgDMPgFf3APNTP5XIoFApIp9NwOBwQBAGDQR/b29t85nom +k+GtiBcXFzEej9Hr9XB6esorW202GzKZDKKRKIKhICRRwuHhIf77v/8bz54947OE8/k8NjY2kEql +eFjLKtWHwyEPJFdXV5FMJqHrOvx+P+x2O99kwdrHs7nHNvntA0lJmleTJxIJeDweJBIJhMNh3nr5 +7OwMh4eHOD09RbVaRb1eh8PhwPXr1+eBrDm/1k5PT/HXv/4VT548mc+U9vmwsLCA5eVlxONxHqTO +NxJ0MZ1OXwphz2MbFXRdx9raGq5cuYLJZMJbpZ+enqJSqUCWZT6DulAoYG1tDZZl8ZnhOzs7ODw8 +hM/nw/Xr17GyssLD/cuw66pYLGJvbw8HBwf8uXLjxg2+MYR1IxgOh3A6nbyynXUBYK/NZrP45JNP +Xvlal8uFyWSC8XjMuwtchoXlZ2dnuHv3Lh4/fox4PM4Dy+XlZcRiMbRaLX7cT548wXA4RCAQwPLy +MizLutB6n53n09NTKIqClZUVXLlyBU6nk2+8ePjwIU5OTvhGhkQiwWenf5/gm3UySCaTyOfzyOVy +iEajiMViyOVy8Pv9GI8n2N3dxR//+EcAQCaT4UFtoVCAaZrodruo1Wr45ptvsLe3B1VVkUwmkclk +EIvGYLfbUa1WcffuXdy5cwd+vx/BYBC5XA5LS0vIZrN8k8KzZ88wHA4xHA6/0/G8Dfb9kgX+33zz +DdLpNLLZLPL5PBYWFpBOp/l89adPn+Lu3bsol8sIhUJYXFzEaDTCZDJBu93G/v4+9vf3EQwGce3a +NXz22Wf8uur1eqhWq+h0OvB4PFBVFaFQCLlcDoIgYDwe4/DwEH6/H7dv38Y//uM/8mDd7/fD4/HA +4/HM25zbqM05IYQQQt4fCr4JIYQQQgghhPwkZrMZr7rs9XpQFIVXtubzeWiaBlVVMZlMIMsyFEVB +v99Hu9NGpzv/Ybzb7f5Ba2DB1NnZGW/n63K5+AxfFpBEo1F0u11IksTbXk+nU5yenkLXdViWhWQy +iWaziclkwlvOskrv6XTKW4rLssyrRGez2bxCvNflc5XZ7w0GA1iW9Z2Cb1mWecXq+Ypbv9/PK5ej +0SgcDgem0ykUReHVig6HA9FolAd+GxsbvF27aZo8jJ1MJlAUBdVqdT4jWpTQard4K2q3243FxUVc +uXIF2WwW2WwW4/EYw8EQoiiiWCzC6XRiNpvx0NHpdCKVSr5VBT+rQHS73DAtk8/4ZuGqZVoQROF5 +yDzAYDBApVLhLZxTqRQWFhZw7do1ZDIZ5HK5+XXYH/D1uVwuzGYz1Go1lMtlFAoFRKNRHnSxMHRz +cxOWZfFZ2aenp6jX60ilUigUCvNzkMnC6XRCVVX0+30+y348HiORSCCfz+PatWv8XLFZ6TabDScn +J7zjQL1ex+HRIcLhMG8F3mw2MZ1Okc/neRC7srKCWCzGRwWw4FgQBCSTSciyDIfDAZfTxc+fKIp8 +5vf588fO99u8J6Igwu1yI5PJ4MaNG89bRUcRCAR4a39JknjoWq/X0e/30el0MBgMeFg4Go1QLpd5 +dwd2ftjGgHA4zJ8Nw+GQB612u/2Va3W73Uin01hdXeUztBVFQafTQSAQQLfb5ZWzgUCAh9NXr16F +KIqYzWYol8uoVqt8Xnyv1+Pz3l9VXc02u7Bny2w2g8PhQDqd5hX+NpsNsWgMkUgE/X6fV82yZyOb +ua1pGt9g86rXso4AgUAATqfz0rEQrMMGa8vPZsKHQiFcu3YNy8vLWFhYQCQSQafTQafTwXQ6RbVa +haIoGAwG2N/fh67ryGQyFz63y+VCKpXC0tIS1tfXcfXqVbjdbliWhU6nw5+dADAYDFCv1793K2yb +zcbD58XFRSwtLSGfzyMYDPKqdUVRUK/X+f3GOjhcv34d2WwWuVwOhmFgOBzC5/Ph9PQUBwcHME0T +rVYLBwcHGA6HGE/GfFPTaDRCNpvF1atX+blKJBJ8oxZrx95ut9/Ywv27YBvEyuUyarUaf9/Y2ImV +lRVks1nEYjH+3BuNRjg6OuLfl05PT9Hv96HrOv8Y1mkgHo/zqnHWPj4cDqPT6fBrMhAIIBqNwrIs ++Hw+iKLI3we2iSkajcLlcvFnoiRKECXxnZwDQgghhJDvg4JvQgghhBBCCCE/ifNtzHu9Hm9DzIIn +FpC1Wi0+Q5m1ym00GnzG8w/R6/XQbrdxenrKW5wnk0ncvn0bN2/eRCAQ4NW0rDqxXq+jXq9DkiSc +nZ1B0zTEYjGk02ns7e1BURQ+/9YwDN76vN/vQxAE+Hw+PrscmM/lbjQa8Pv98Pv9vPXyaDQCMK8m +ftvgW5IkHmzKssyrpCVJgsPhgNPphNvlhs1ug2mamM1mEMV5KOF2u7G2tobf/va3yGazvNqZVdKW +y2UcHR2h0WjANE00m01eoc5mJDscDiQSCVy9ehU3b97klX+shbI207CwsMCrPlutFk5OThCLxWCZ +FvCW+YggCBAkAYL1ckB7vhPAYDBAqVRCqVTCZDLhlc1Xr17FrVu34PP5eFvvYDAI3dBxdHR0YX3F +YpG3YmahLnvvd3d3YRgGBEGAy+XC2dkZRqMRP5effPIJEol52NxoNHB6eso3D7Cw6cqVK7h9+zZf +i8/n4xXkCwsLfKNAu93G3t4eD6Pq9TrvkLC8vIzf/OY3SKVSiEQicLvdz9tZG4hGo1heXgYAJBIJ +XsEtSuIbz993Ce1ESYTb48bS0hLfdMECMDYmIBwOYzabYTqd4uDgALquQ1EUjEYjXnVbr9d5yGq3 +27G0tIRf//rXSKVSfG67rhswDB2VSoWHvq8LviORCD755BN89dVXyGQyyGQyfIOJ3+/H/v4+78Cw +vr6OTz/9FEtLS1heXoZNtkE35mvf2dmBw+HgM5I7nQ4kUXrlvcnC+eFwCF3XYbPZ4PP5EIvFkMlk +IIoiZFmeB4qx+dxzWZIhyRJKpRKfQ81ey2bLv+m1rO12r9d7aU2z2YwH2rVaDaqqwul0Ip/P46uv +vnpeKR2A6/nmmXQ6zedbs/bXBwcHEAQBXq8XTqeTf+5wOIzbt2/j66+/RiaTQTab5eeLzUA/OjqC +3W6Hqqo8SDZN8zs+uQGHw4HFxUX84z/+I1ZXV5FKpRCPx+cVxnYHRqMhms0mzs7OMBgMIEkSIpEI +1tbW8Nlnn/F7jW0ucjqdODg4wMnJCSzLwmAwwMHBAQ+Iq9Uqnyefy+Xw9ddfo1AoIBAIwOPx8Pbx +7XYb0WiUzxtnz9cfis3RPj4+Rq1Ww2w2g8fjQTabxe3bt7G4uAiv1zsfheD28I0Lx8fH/Dool8vo +9Xp8A4phGJAkiW9SyWaz/LoKBoOIRCKYTqeQJIlvonI4HBiNRhe+v8iyDLvdDqfDyUNy9n3nXYwk +IYQQQgj5ISj4JoQQQgghhBDyk1BVFc1mE6VSic86drlcCAaDSCQSPEiIx+Nwu90QRZGHxNVqFS6X +C/F4/AetgVUD1+t1qKrKg9uNjQ1cv34dwMXwz9ANHozquo56vQ7DmIeLqVSKzwYej8cYj8dQVRWD +wQCNRgO9Xg+CICAYDPJqa5vNxqspdV2HIIh8prCiKBBFEYFAAOHwPMx8mwCBtahm4RcLIGw227wq +2jGfaz3TZhc+xuVyoVAo4Fe/+hXC4TCCweCFUCuTyfA55pZl8TBF0zRMJhOMRiMIggC73Q6v1/s8 +oNTR7/cvrI1VpLIwzOPxYDgczquPIb3V+/aqauQX/3s0GqFSqaBSqVy6Plb9ysiSzNcHgLf5HY/H +sNlsSKVSuH79On/v2QYIWZbh8/n4rO1wOIzl5WVsbGzwGdej0QjVavWltbDOBS+uhZ0rVk3c6XSg +qipGoxGGwyE6nQ4Mw4Db7UYul8MXX3yBQCDAv955bPPCm87X9w2o2Ovsdjuvtp3NZnzzCps7raoq +D8/YHGFVVXk3h1qthpOTE7TbbcxmM35sn3/+OSLhyDysP7dG1t6eBeyvEgwGceXKFXz11Vd8BjTr +uMDeL9ZJoFAo4NatW7wC326382OLxWJ8XjGrfmbh6WXOz4+2LIvfb6zjgyAIvGW43++/UCE7mUxw +fHzMN8iw17IZ3a97LbtOLqPrOjqdDk5PT9Fut2EYBjweD9/UkUwmX7oWGo0GD1B1Xcfp6Smfc37+ +GREIBLCxsYFf/epX8Pv9CAVDkOT5Pe3z+ebt+qNRzGYzXg0/mUy+V/AtSRLS6TS+/PJLrK2tIRwO +XxiXcHJygn6/j9PTU949w2azwe12w+v1AsCFduTsGejz+fi4AU3T+LO42WxC07Tn3SlSuH37NvL5 +/EvB7snJCQKBAL8m32XFd7/fR6lUQqPRgKqqvFMHG1fAZrIzsizz42VdIzweD3w+H2y2+eanF68r +dt06nU54vd75802SIUoiv57Z1z7//cVut8Nmt8Fut/N7hhBCCCHkQ0DBNyGEEEIIIYSQn8R0OkWj +0cDh4SE0TeOzlH0+HyRxXikmiiI8nnn1WjQahSiKaDQaODs7QyQS+cFrYBWZiqLw6kmPx8Nn9r4Y +Wsg2GV6vl1fTjUYjTKdTOJ1OJBIJHiiwdsjtdhvVahWlUgntdht2ux2ZTIaHKV6vF5qm4fT0lFfF +9vt9mKbJ2xbPK4E9P0mYwM75ZVWKrKqPrYMF3qIo8pbx/f58frWu6/jmm29e+hz9fh/Hx8c4OTnh +Vdaapr12HvEPfX8HgwE6nQ6vut/e3oZhGLh79+5LHz8YDPj6PB4PAoEAVFXl6wv4A8jn8xiNRjg+ +Psbh4SFmsxmOj495Beknn3yCjY0NRKPR+axxUbqwlm63i06ng16vhydPnsA0TTx48OCltQyHQxwd +HaFYLMLtdiMQCPCQCZgHmC6XCy6XC16vl7edvyxo+ykqLi3LwmQy4RX29XodrVYL/X6fb/QwTZO3 +zS+Xy5jNZryVvqZqvPJ7Npvx4NDj8UCWZAji9z8GFtCdDyklSYJlWbDb7RcqU1/8uPPYBhJ2fxiG +8drZ4m63G4lEgofYqqri+PgYv//971EulxEMBhEMBvls5FAoBJ/PB7/Pz1+bSqX4a09OTl567fnX ++/1++H1+vI5pmnz0Auuy4XA44HK5+HG9eOwOh4N3ImBtw8fjMQ/lz2PPD1F4+RnCOk9YlgXLsjCb +zd54Dl/3nrJNJ6z6+DxN0zAcDtHtdvnM6/39fUiShP39/Zc+n6qqODw8RLFY5BuOBEHAaDTioTLr +SOLxeH7yamY2XqHf7/NjarVauHfvHiaTyaXt9uv1Ou/SEQgEEAwG4fP5EA6H4fP5UCwWoWkaSqUS +/uu//gudTodfU6FQ6NvryueHz+/jzx5CCCGEkI8JBd+EEEIIIYQQQn4S6nRe8X1ycgJVVREOh5FK +peYzgSUR4vO+1x6PB+FwmLcaZ8F3oVD4wWtg4Qirvg4Gg/B4PK9smyxJErxeL6LRKA+9WUvmZDI5 +D+0libcrb7VaqFarODs7Q7fbhd1u5zOlWWg8m834rHAAfBa4w+Hg1Xlej/cnmZPKZjFfFuiw0Ipt +CmDBt2EY6PV6PFh+9uwZSqXSpUG9ruv8nEWjUX6uvm/49Sbnw2YWzCuKgnK5/Mb1RSIRiKJ4YX2B +QABuz7yaeWdnB5FIBK1WC6VSCS6XC1988QW+/PJLXL16FdFo7EIVsqZqPITv9Xp8VnK5XOZVxJet +ZTqdIhQK8QpxALxamQXibLPGixXRPxX23imKgpOTEzx48AD7+/s4OTnhlbKapsFmm1eEWpbFA25W +aapq6kvBt9frnYeMz9t3/xAskGXnh80nlmWZB3ovht7nz+X5zgnsHjBN87XXrcfjQSKR4KGpqqoo +lUoYjUbY3d1FMpnkLcEXFhaQzWb56zweD5LJJLrd7htfm8/nkcvleNX6a98r03op+Lbb7XC73fze +f5HD4eCV8t1ulwffL25YYedOFMWXnleiMG+h7XQ6MZvN5psdNO17zfc+/57a7fYL7yEzm814Z4RO +p4N+v49er4dGo4E7d+689LlM04SiKFBVFV6vF4IgwOl0YjQaQRRF/h6ySujLwvYfEwu+Wat6NqZD +URQcHBzwa/I8VrHOuqewTRvhcBjJZBKPHz/GbDZDpVLhnyeRSCCdTvNrMp/Pw7IsuD1uCr4JIYQQ +8lGi4JsQQgghhBBCyI/OsixM1SmazSaKxSJviwwAf/vb3zCZTPjHlkol7O3t8Tm5k8kEXq8X/X6f +h07fNzRlLYdZy1ebzfbK8AcABEHk4RdrWQzMQ2Gn08nnWQPz0IHNKy6Xy1BVFdFoFNFolAffiqJg +Op2iVCoBmFdVD4dDPmPV5/PB4XDwdsE/lVdVDbNQi51zFsCebyPMZpKz83DZ+TYMA+FwGPF4HEtL +S/MQXHz3x2iaJnRdh2mafH2BQADRaPTSCskX1xeLxbC0tIRYLDbfFCBLkOT5TFw2yxaYB+ysXbDH +44HT6YQsX7yODNPga2FBo8/nQzQavdCi+cW1mKaJYDDIW/6z64a1zmbX7PlQ96c2Go14W+mdnR1s +b29jOBzyDSGSJH3bDvl5e/+joyP0ej1e8c3+yTYZsA4Dr7sfv6vLWru/qcr7xY8/f55Z5fKrsK4N +AHD16lX0ej0+05pVE5fLZQyHQ1QqFcRiMaytrWFjY4NXYedyOVy7du2Nrz04OMD6+jo2Njb4aIjL +mJbJg1D23LssOH7xuFl1tSAI/Lq8rEX5667D85sP2Ln7IRteXve+sWM0DIPPm7fb7YhGowiFQi99 +PGvjbZomn8N+viOFYRh8/e/ymmRf+20+hp13YL4ZgXUfOd+C/8VzwNrix2IxxGIxLCwsYHFxEbFY +DFeuXEGn00G73ebX1WQy4fPMa7Ua9vf3sb6+jvX1dcRisQut7QkhhBBCPgYUfBNCCCGEEEII+VGd +rw5ttVo4OzvjM19rtRrOzs4utMlmLcO73S4cDgdvN8vmtv7QSuEXX/82gcaLM5NFUYQkSjz4ZSFK +vV5HrVZDuVzmM2kXFhagKArG4zEajQZOTk74n7M2wCygCQQCH9S81BfPDTv/kiTxEHd9fR03b95E +Op2+9HOwkJPNno1Go8hkMj+olfXr3ie2Pva1VlZWcPPmTV5d+6b1RSIRZLPZCxWV5zcBnA/eDMPg +Lesvuy5ZWM3a9y8vL+PmzZvI5XJvtRZN03BwcICjo6NL34/3pdfr4eTkBLu7u9jZ2cH+/j7C4TCW +lpaQy+UQDod5Bb0gCCiXy/i3f/s3lMvllz7Xi+f1QzlGtrbL/v1VWPDt8Xjw1VdfIRaLoVqtotls +otPp8Nnm9XodqqrC5XKh1WpB13XkcjkkEgnk83l89dVXiEajqNVqaDQal77W6XTy16bTaYTD4deu +7cfosPChYcfodrsRjUaRSCRw48YNrK6uvvLj2ZgJr9cLURT56IM3Vff/lMdkt9v5XPdr167h1q1b +CAQCl36sZVkQBAFerxderxehUAjxWBwerwe/+MUvEAqFUKlU0Gw20Wq1oCgKRqMRWq0Wn+XNZoqv +rKwglUpR5TchhBBCPioUfBNCCCGEEPJ37nwF7YcWOpCfB9MwMdNnGI1G6HQ6aLVa/FpjM7dZBTTD +foDPZmO3Wi0MBgOoqvrKkPFtsEpGURR5dfDr2m5bz6slWateVv3HWvt6PB4Eg0HMZjPMZjPUajUe +VrEW6vl8ngffhmHg4OAAtVoNDocDbrebzzaOx+PfOfh+VZvm8+fwXTj/NVgFPJtHXigU8NVXX+Hq +1auv/RySJEGWZMg2GQ6H4we3sn7VOllFKwuP2fquX7/+ndYnCAIM3cBMn0FRFH4NsA0Ldrsduq6j +3W6j3+9DVdULGyTYWs6fq8XFRXz55Ze4efPmW62l2WpiNBqhVCrxyltWpXpZ9e1Phd2zh4eHKJVK +aDQaiEajWFpawldffYVsNss3Gui6jp2dHTx58uSlAO18VfGLFa4/th/je50sy/B6vfOxBW4PlpaW +UCwWcXBwgGKxiFqthnq9jna7jWq1CtM04XK5EAwG4XQ6kUgkkEwm4XF7UCgUXnpto9FAu91GrVbj +re+DwSAAXFoBDMxbjrN7gnWtYNfQq7CNPOxZyyr4f8g9+2P/3eL8/ca6cWSzWXz22Wf45S9/+drX +sue6oijQNA2VSgUAeFX0d70mX7WJ41XX+Pnn9PnnNjsm1jHCbrdjfX0dv/71r5FMJl/79c/PQ2cj +KxwOB/L5PEqlEg4ODnBycsKvyW63yyu/z1+TPp/v0pCdEEIIIeRDRcE3IYQQQgghf+dYKMnmQbKZ +kIS8C5ZlYTSet0VuNBoYDocwDAPZbBaLi4sIBoMvBQSs/bGu6ygWiyiVShiPx2i32zg7O3tjm97X +cTgc8Pl8cLlcaDab6Ha7GI1G0DSNb/44T9d1DIdDtFotaJrG282ykIlVFrIW5mdnZ2i1WhiNRrDb +7XA4HIjFYtA0DdPpFP1+HwDQ7/fRarXgdDoRDAYhSdL3Cr6ZF9sN87bC5rsLvwHAbrfD6/ViOp3C +MAwMBgMoisLn476u7TF7n2VZ/tGeMez99Xg8/Ho7v77zrdtftb7zbcT7/T6arSafzd1ut2Gz2ZDN +ZuFwODCdTnH//n3YbDak02kkEgm+MeDFtbx4rt5mLawSNRAI8KpfAHzWOgvmfuoNS2yW+nA4hM1m +QyQS4XOCM+nMvDpVlGCYBq+cffE6ZIGex+NBt9vFdDqFaZqYTqev2Yjybq/nH4NlWRAFEXaHHQF/ +AJlMBg6HA6lUCv1+H8PhEDs7O9ja2kKz2YQgCKjVakilUlAU5a1f22g0IAgC6vU6n+F8fsY8I4jz +68jv9/PzfH4jzmXPPVVV0e/30e12YZomPB4P3G73pZ//Q2Gz2eDxeODz+dBoNDAajTAej2FZ1lvf +b6Zp8g0zsixDURQ+bsMw3hx+s2uTBfBsHjkAPuNcVTXYbDZYpgWIF197vspcFEU+Y1xVVWiahtFo +BFVV+ez0Nz1vXxyJ4HA44Pf7kU6nYbPZkEgkMBgM0O/3cXBwgMePH6NUKkGSJDSbTTQaDSzkFxAM +hl76/mIaH0ZFPCGEEELIiyj4JoQQQggh5O+cYRpQVZW3XZUl+cIPYwn5oYbDIRqNxoXgO51O4ze/ ++Q1WVlb4HFVG13XeCv3//b//h3q9jslkgk6ng0qlAr/f/70r0NgP/p1OJ2azGfr9PkajEZ83/mII +NJvNeBtYwzB48M3CadbCut/vYzKZoNfrod1uYzgcwu/38+CbVY3XajVYloXhcMhDVMuy+DzWQCDw +ncMlURABEZduIDCtd1s9a7fb4fP5IIoidF1Hv9/HdDrlIY0kSZBE6dI25pb5bUgiSj9e8O33++Hx +eC4N5t92fezPB8MBSqUSjo6OUC6X0ev1+Nxcl8uFYrGIw8NDhEIh3LhxA6qq8upK1p74xbUAeOu1 +nA++JUnCZDKBpmkYj8d8DvH5luz89S+05n/X2CYOFnzHYjHe1j+by0KWZIiSCMM0Xjq3/BoQRX4/ +iaIIRVGgqiqfZ/46H3rgJogCXE4XXE4XPF4PUqnUvMraMKEbOhKJBEzTxNOnT3nw3Ww2MZlM3vja +ZDJ5YcNDrVZDNBrFZDK59LkoiiJcLhd/tkynU0ynU0wmE155/OJzj21s6PV6PAx2u92XXmsfCvZs +8vl8AOZz6CeTCQ++RVGELMmvvd8Mw7gQfKuqyj+P+fxafpXz16QgCDz4ZvPBWaW9pqkwdCdMy4SE +bzdwsS4OLPxmzyy/34/BYABN09Dv96FpGmRZhsvl4pX8rzoevp7nx+ywO+CwzzuNxOPx+Uxw3YBu +6Hj48CGA+fdfWZZ58D1RJpAk8aVZ96ZFwTchhBBCPkwf7t9YCSGEEEIIIe8ca9urqiq63S56vR46 +nQ7a7TYmkwnW19dx5coVamtJ3qnBYB4eNptNAIDf70cqlcLS0hLW19fnlXjCt0GoYRrPAwIN29vb +vJX4cDjE0dERMpkMnE7n91oLq3YbDAY4PDyEqqpoNBrY2dmB3W5HIBCA3+/n4dDR0RFOTk5wdnaG +WCyGeDyOXC6HUCgEAHC75hXfbIbv2dkZD8gDgQBvd25ZFq9I9Hq9cLvdMAwD7XYbsiwjHo8jmUwi +FAp954pvQRQgmvPqWbvdDlEUMRwOUSwWAYC3/Z5MJrxi8Pu2k/Z6vUilUhiPx9jf34dpmmg2m3j6 +9ClsNhvC4TDC4TAPSVi15Hg85p/D6XQiEo4gFA6982DW6/UinU6j2ZxXabNz/PTpUzidzpfWZxgG +JpPJhRCQvXeBQACVSgVbW1t49uwZLMtCJpPB4uIiNjY2IIoiBoMBTk5O0G63sb+/j1AohHQ6jUw6 +w89Vu93GwcEBTNNEu93Gzs4O3G43XwsLxi5bi6ZpEAQB2WwWZ2dnEEURo9EIp6enuH//PpLJJMLh +MFwu1/MuCQaGwwHvLJBOp5FOpSHJ73ZGryzJ/HozDAPj8RiDwQCdTud5FbMIURT4f+/u7vLW3ozN +ZkMoFIKu66jVanxzwOnpKR4/foxYLMbDPUVRMJlMsLu7i0qlgn6/j2Aw+MEFb8PhEL1ej7cHFwQB +LpcLbrd7Hr5KImzivKsKq15nGwCm0ylKpdKFsPR1r2WjF1hV8auqmWVZ5vO/G40GJEnCeDzm13av +14PfH4DL6YQynUJVpzg8PESxWESj0UAsFkMul0M6nYbX633fp/iV3G43EokEer0ejo6OAMxn0e/v +7+Pu3bsIhUIIhyOw2228tf54PObdEwDwDUqJRALVapW/p2dnZ9jc3ESv14PH44HNZuP36s7ODs7O +ztDv9xEIBGCaJm8rzt47u92O2WyGarWKra0t/ry3LAuj0QiNRgP7+/v8HrHb7QiFQvD7/chms3xD +zWw2Q6VSwcOHDzEYDBAOhxEIBM6NDtEwmYwxnU4vnJvzG2EkSYLT6eRjNth1JUkSv64EQeD3N7u2 +WJBvWRZ6vR5OT0/5sXq93gtt2alzECGEEELeJwq+CSGEEEII+TuiGzom4wl6/R729vawv7+P09NT +nJ2dYTQa4V//9V+RyWQo+CbvjGVZ6Pf7KJVKaLVaF0LeRCKBeDwOAJe2UWWVkdFoFL1eD6PRCEdH +R7Db7YjFYt9rPcFgED6fD5PJBA8fPoRhGKjVanj06BFUVcXCwgIWFhbQ6XTQ6XSws7ODg4MDnJ2d +IRwOI5vNYnV1FdFoFIIgwOP1IBqNwufzQVEUVKtVBAIBhMNhxGIxHhSdDxwCgQAikQjG4zF6vR6f +vZtMJr9zxTefvS0JfJZ0v99Hrze/xw3DgCiKCAQCvJpWURQe9HxXfr8f+XyeV5eyVst37txBp9PB +ysoKVlZWeFiiKArq9TpvyywIAsLhMNbW1hAMBd958B0IBJDP59Hr9RAMBiGKIur1Ou7evYter4fl +5WWsrqzCZp8HM+pURa0+n53MzmcgEMDi4iLsdjvK5TK2trZQLBYRi8Xw6aefYnl5GRsbG5hOpzg9 +PYXT6cRwOMSzZ8/4e8fey3w+j9FohMePH0MURTQaDdy7dw+DwQBLS0tYWVnh8841TUO1WkWz2eQV +nw6HA6FQCEtLS/zaZ0H6f/7nf2J5eRnLy8uIRCL8/S2VSnzTw5dffolYLPbOg2+749tq9kqlgna7 +jXK5jJOTEwQCAd7F4ejoCPv7+3j27BlKpRI0TeOfQ5ZlRCNR+P1+HB8fA5gHlQcHB/jzn/+MdDqN +SCQCh8PBO0Zsbm7i5OQEvV4PkUjkvc45v0y73Ua9Xkev1+PnIBKJIJVKIRQM8dbvrVYL3W4X4/EY +gUAAwWAQhmHwTRRvem2v1+P3YDAY5N0lLsMq8sPhMA/Wp9MpisUi/va3v6HZbGJhYQGRSATtdhud +Tge7u7s4OjpCo9FAOp3G6uoqFhcXEQgELryHHxI201vXdWxvb0OSJHS7XTx+/BiKomBpaQmrq6u8 +w4BhGKjX66jX65jNZrx6OhCYt5gvl8uQJIlvuPrb3/6GarXKn+vNZhP1eh1Pnz7F0dER2u02wuEw +DMOYP+sdTj7qwOl0QtM0HB8fw+VyIZ/PYzgcYjqdolaroVQqYXNzE6enp3C5XPD5fJAkCeFwGEtL +S5hOp7xzxOnpKf7rv/4LlUoFKysryOfzvKV5v99HvV5Hp9Ph1xAfe/F8k4UoigiFQkgmk4jFYnze ++/lr0ufzwe/38+4oLNB2uVwAgFarhWfPnsEwDNhsNvj9fthsNsiyzDuCUPBNCCGEkPeFgm9CCCGE +EEJ+pizL4j/QnM+VnLfsZBV4jx8/xvb2No6Pj3F2dobJZIJCoXChMpOQH8oy58F3uVxGq9W6MAs5 +EokgGAy++rWWxcPx2WyGyWSC4+NjHvJ9Hz6fD16PF/1+H9FolFdCHx4eQtM0DIdDqKqKZrOJZrPJ +w5/pdAqn04l8Po+lpaVvK77dbt6iXNd1NBoN3po6kUhcCL5Z69pgMIhEIoGzszN0Oh2MRiPIsoxk +MolgMPidW52z8JhVEQ+HQ4zHYxweHvLWuZFIhFf9sdm+3yeYYFWVuj7flBAMBqGqKg4PDzEcDnkr +bla1OplMUKlUUKvVYLPZ4HQ6MZ1OkUgkXppx+y74vD44nU70ej2+PhY4serO2WwGh8MBSZKgKAoq +lQqq1SpfXzgchs1mg9vtRqlUwvHxMVqtFhYXF3H79m2srq5ieXkZ/X4fW1tbiEQimM1mOD4+5m3r +r1y5AofDwWfosrXMZjOcnJxgPB7zduWs7TnbOFGtVvkM30gkgmg0imw2i1QqhXA4jHa7jW63i83N +TYzHY2iahlgshslkAkVRcHJygqOjI9hsNuTzed7G/11yuVwIh8OIRCI4Pj7GdDpFq9XC8fExZFmG +LMuQJAlHR0fY3d1FsVhEp9O5sBbWAcHhcCAej8Pj8QCYVyU/fvwY9XodiUQCTqcTjUYD9Xodx8fH +6HQ6UFUVs9nsg6v4Ho/HqFarqFQqME0ThmEgGo3y5828Kl9HpVLBaDSCZVlwuVx8Nnyv10Oz2Zy3 +oH7Na4fDIW9BnkgkEA6H4XA4Lj0fkiTNn3teLzKZDMLhMOx2O3q9Hp48ecKvxVgshmazyd9H9n4F +AgGsrq4inU7D7Xaj3W6/79N8KbfLDbvdDtM0kUgkEAqFMB6PUS6XMRqNMBqNMJvN5vPnJYlXYFer +VViWxcck+Hw+JJNJJJNJHuK2221sbW2h3W4jlUohGAzy0JxtVmDPPl6J75jPr2cbnUajEZrNJra3 +tzEcDjEajTCdTlGtVnF2doazszMMBgMA4EE8q/geDAaIxWLweDx8zvtwOOSz2tn91u12+UYUNh+c +dZNg16NhGAiFQuj1euh2u/z3SqUS+v0+DMOA0+lEPB5HLBbjLe69Xi8ikQjvasC6WLDr1Ov1Pt9U +IPGAnBBCCCHkfaDgmxBCCCGEkJ8pFnSzNtNnZ2eo1WpotVpotVqoVqtoNBrodDoYj8e8xSUh71q/ +30elUkG320U4HL5QyfkmrDW5oigYjUYoFotYXFz83sG3KIqQbTJisRhu3brF23CPx2McHx+jVqvh +8ePH0DQNmqZB13UEg0F89tlnuHnzJgqFAg8DgHmVYSwWQzQa5e3XWZCVSCTg8/kuVDU7HA5EIhFk +s1m+EcVms8Hn8yEej8Pr9X7n4JtJpVK4fv06bDYbhsMhSqUSb/sbiUQQj8cRj8f5nPXvE3xLksSr +Im/dugVVVdFqtdDpdHggUywWL1Qb6roOXdeRSqV4mPJjtUwWxHllPZu5PZ1O0Wg00O12MRwOsbu7 +i9PT00vXxyogfT4fGo0GWq0WTk9PYVkWgsEgMpkM1tbWkEqlePXl8vIyPv/8c5ydnaFer+Pg4ACF +QgHlchmRSARut5uvRVEU/swdjUZ8Laxa88W1RKPReagZCsPr9WJlZQW/+tWvEI/H+Xztk5MTNJtN +OBwO6LrOQyzTNBEIBHir8HctGAyiUChgNpuh3W6j0WjANE3s7u6iVCrB4XDw+5tVvrNwnOEzvj1e +LC4u4tNPP4XT6eTjB7rdLorFIt+kwDYD+P1+aJr2ox3bD8E2BKiqimq1ilqthmq1ir29PciyzCtv +x+MxvyeWl5dx/fp13kK60+nw173Na2/cuME7DPR6vUvXxWbOZ7NZfP7553A6nRiPx7xTwenpKex2 +O3/uGYbBW5tfvXoVmUzmtZuUPgSiJEKGDI/Hg42NDd6inP0d5+joiG9wYRuR2P0WDocRCoWQSqUQ +jc67EGQyGdy4cQOz2Yw/54bDISqVCtxuNyRJgiRJPDAXBIH/Pvv8drsdhUIBX3zxBU5PT9Hv91Gr +1fi4AofDwTfZhEIhTKdTPgoDAK9CT6VS+Pzzz3nHBzai5tGjR3j27BlvM87miIuiyDc5OJ1OmKbJ +r8lGo4FKpYKDgwPYbDZ+XU0mE74paWlpCdevX8fy8jJCoRBkWUYul8Pt27dRLBZ5FXyn08H+/j7S +6TSWlpawtLT0vEvFh32tEEIIIeTnjYJvQgghhBBCfqY0TUO/10elWsGjR4+wubnJZ3aySq7ZbAZd +13kbTfYDUELeBcuyYFom+v0+qtUq+v0+kskkFhYWEIvF3jr4zmQy6Ha76HQ6vHJcVdXvda1KkgSb +zYZ4LI5PPvkE4XAY9+7dw8OHD1EulzEej6EoCmw2GxwOB6LRKFZWVrCxsYHr16+jUCggGo1Ckuat +o1mVKptHzH4vkUgglUrB7/df+Po2m523TG80GiiXy7Db7fB6vYhGo3A4HJDl7/e/6qlUCrdu3YIo +ivx4WMV3MBjE+vo6rl+/jtFoBF3Xv1fAzlq2h8Nh3L59G/F4HPfv38fDhw9RLBZRLpd51SIA2O3z +ltiBQADRaPRCJbwgvts25wD47G62vlgshgcPHuDRo0c4Pj5GuVzGcDjk1w5r08uqMln77mKxiGKx +yNuOs80KVzauwOP1QJbmQeTKygoEQcAf//hH7O3todFooFgsolKp8Nb2oVAIN2/e5Gt5+PAhjo6O +UKlUMBgMLl0Lm++bTqcRDofhdruxsrICl8uFbDaL+/fv4/HjxyiVSnxuOzv2eDzOR1a4XK533k4e +mIfZLJCuVCo4OzvjYZqiKHC5XLxDwurqKiKRCBqNxoWgmrVy9/rmwfcXX3wBt9uNzc1N7O7uYjqd +wjAM2O12pFIpZLNZPipgNpvB7Xb/KMf2QzidTkSjUWiaxjdDsOB1Op3y9YZCISQSCeTzeaysrOD2 +7dtQVRW1Wg2iKL7Va3O5HJaXl3Hr1i14vV4YhnFp8M3uWVmWkc1m8fXXXyMajeL+/ft49OgROp3O +hU4NLCBfW1vjz718Lg/ZJmMymbzvU/xKgiBAlET4fD5cu3YNoVAIW1tbePDgAd+cwSqagfn3Ana/ +sVnqLPj2+XzI5XL45JNPIMsytra2sLOzg8lkwtt+J5NJZDIZfk3abDZ4vV7+vQGYb3RaWlqCYRhw +u924d+8e6vU6qtUqDMNAJBJBPp/nFeoALrQUZ8F3Op3Gl19+iWQyyZ9nlUoFx8fHGI1G/NpwuVz8 +WZZOp3mIb5om70bAupmMRiMoisJfy57N7Lq6ceMG0uk039CVy+Xw6aefQpIkPHr0iI9UsCwL+Xwe +qqrC6/UiGAj+KF0mCCGEEELeFgWFcde/AACAAElEQVTfhBBCCCGE/AxomobpdIrJZMJnE7Mfblar +Vezv7+Po6AjNZhODwQD9fv/C3MfvO++XkNcRBAGiICKbzfLq6pWVFaytrSGZTPIfqL9ONBrF6uoq +JElCIBBAPB7H+vo6IpEIvF4vbt++jdFohE8++eR5haIPsjyvxFtdXcXXX38NAHy2K2sr7XK7kEym +eOttURSRSCR4C1qn0wmHw4FYLIaVlRWsr68jm82+NIObtWtOpVL49NNP0e/3cePGDVy9ehX5XP6l +4NvtdiGTyfCgkrXWXVlZ4Wv7vmEemwXN5kM7nU6+ucXr9WJ5eRmLi4vQdR0ulwsulwtra2vwer2w +2+0vVc/GYjGsr6/D7/dDFEVIkoSFhQV+HlmVOwvlotEout0uut0uD3MdDgefQ7y0tITFxUVe2f5j +hZYsUI1F55srVFWFIAiIRCLodDro9Xq8uwWr1mTrKxQKfBOQKIqIx+OYTCYIBoNYXl6G1zc/V8A8 +oIrH4xAEAd1uF5qm8Ws8GAzC5XLxitBYNMbPFQCEw2F+rtha7HY7P1eFQgGFQmHeNcDvg81mez5T +fR6EzWYzyLKMTqeDfr/PrydJkhCPx5FOp7G4uMhbaL9rNpuNn5+NjQ2oqopKpYJ6vY7xeAyHwwGn +08lDNLbJxe/34+rVqxeuI1mWEQ6Hsby8DEmSYLfb4fP5MJlMoOs6r3hNp9NQVRWDwQCmaSIajfKQ +n81WvnHjBjqdDjY2NpDL5eB0OnmFL+P1enHlyhX86le/Qi6Xw8rKCt9cIArf3gNutxvLy8v46quv +EAqFsLq6yr+Wy+XCwsICfvGLXyCTyeDq1au8hTh7dqiqyttPs3nO7PpkweTCwgKWlpYQDocvzM5W +VZW/v5e9NpVKYWFhgc93Z9XaoVAIV69eRaPRwOLiIhYXF+H1evmGGtY6m7X6lmUZrVaLt91ma2fn +ZWVlBdlsFm7PfJOBaZoIhUK4du0aWq0WFhYWsLCwwN/H85tZbHYbMpkMrl27BkVRIIrzUDqVSr1x +c48gCrzCn53neDyOa9eu8bbul13XgiBAlubXkyzL0HUdlmXB5/Px+202mwEAPx/BYBD5fB7Ly8u8 +jbndbuf3IQu6vV4vRqMRf20qlUIymUSn0+Etv198fkuShGg0yjvqsE05rJtIOBxGLpdDLBbDeDzm +Yy9YBXgqleJV4alUir/PoiginU6j2+1e2GjkdrsRDAYRiUSwsrKChYUF/jxTFIV/r2s2m7xVOjtv +wWAQ6XQamUwGy8vL884cHi9ESYSu63wTmGma/Pvx+W4e559X33fzFiGEEELIu0B/EyGEEEIIIeRn +QFVVdDod1Ot17O7uYm9vD5VKhbcg7vV6GAwGUBQFqqryVrgAqMqb/KhEScSVK1fg8Xgwm80QCoUQ +Dod56PU6rHrVbrcjnU7j5s2b6Pf7yGQyyGazMAwD//qv/4orV64gm80il8shEPDzytfPP/8csVgM +wLxajQUasixDEiUEAn44HHZ89tlnWFhY4PO9Z7MZr5BkAUQoFILP53tpdikLX7LZLH77z7/FysoK +bykeDod5RTjjdruRy+V4ReHnn38Om82GwmLhQovc74MF1GyW76effsorvlmQEwqFsLKygl/84hd8 +BrTP5+Mzuc/LZDKw2WwYjUYQBRGCOA+3g8Egn/lqt9tx9epVJBIJ9Pt9TKdTHtKx88NCUPb1/X7/ +W216+CFkWYbb44Zsk3HlyhXeHlxVVR72APOKSjYLNxAIPA/MbDz0ZwEVuwbPBzp2mx2hUIiHq4VC +AbquI51O83nILKB7cS29Xu+NawmFQvD7/HC5XTzMDwQE2GwyXC4XlpaW+Pk2DINfOx6Ph8+Zj0Qi +P0oIxaqIfT4fNjY2EAlHMBwN+fcYNuebVdTKsoyFhQV8+eWXiMVi/H5kobTP50Mmk4Hf50c2k8Un +n3xyYV6y1+uFz+uDbuh8trfT6YTb7YbP54Pf54dpmfjNb37DuzLk8/mXKnCBeWXr119/zduDp1Kp ++cYElxOi9G3w7fP58NlnnyEej/PZ736/n1cH37hxA36/H4PBAPl8HqlkCqIkYjabIRwOIxwO48qV +K5hOp1BVFbquX6jM9Xq98Pv9iETmG0jYs8nj8SAcDmNjY+OtXytLMt+I8I//+I9YWFhAMBjkzxq7 +3c43w4RCITgcDng8Hqyurl74ewFr3802YASDQfj9fthkG//ayWTywtdg51mW5ZfGOiwvL8Pv90Of +6RBEATabDdHom7t9sOtdkiRcvXoVHo8HvV5vXh0dT8DlckGWLr+uRUnkm06Wl5cRDAZ5+D6dTvnf +fdj5YJtf2Pt7fqZ1NptFwB9AOp3m1yS7V30+HzxuD57uPEW320Wr1Xpp85AkSfz7jtvtRjabRa/X +g67rME0TLpeLf82ZNoM2m4fa5zuOOJ1OSOL8XrPZbLxyn3V6OD/2g20WcLlcCAaDCIcjsNnk52MU +5rO919bWeIU/2wTAzoXX64XP50MkEoHH4+H3AxsfwTp4LC4u8k07pmnC5/MhGo0iEonwdRJCCCGE +vC+CRT/hIh850zRxenqKUqmEcrmMcrmMZrPJW6GxH4pls9n3vVRCCCGE/J1qtVp8xna5XEalUoHL +5eKVNblcDrlcjs90fBM2w1HXdWiaBlVV0W63Ua1WUSqV8ODBA2xubqJWq/E5sOdnvwK4NOgOhUL4 +X//rf+F//+//jStXrrzv00YIIYQQ8kH77//+b/yf//N/cO/ePSSTSaRSKVy/fh1ffPEFbt68+b6X +Rwj5kfT7fZyenqJcLvNxH4IgXPj/O9YpiBBCyE+LKr4JIYQQQgj5iFiWhfF4jG63i3a7jdPTU5ye +nqLZbPKKo3K5jFqtxtujzmYzXpXDPgchhBBCCCGEEEIIIT8nFHwTQgghhBDykWAtySeTCeq1Oo5P +jnH//n08evQI9Xodk8mEt/JkleBvqvImhBBCCCGEEEIIIeTngIJvQgghhBBCPkCmacIyLUyUCUaj +EQaDAbrdLrrdLhqNBmq1GiqVCvb29nB8fIzBYABN0/jMRl3XaXY3IYQQQgghhBBCCPm7QcE3IYQQ +QgghHxjLsnildq/XQ6lUQqlUwv7+Pg4ODtButzEajTAajdDtdtHv9zGdTmEYBp/lTaE3IYQQQggh +hBBCCPl7QsE3IYQQQggh74llWTBNE9PpFIPBAFNlCsOcB96qqkJVVVQqFRwcHODg4ABbW1vY3d3F +YDCAYRiYzWaYzWbQNO2dhNyWZUHTNIyGI/T7/fd9egghhBBCPmiKokAQBNjtdkiSBEEQoOs6JpMJ +/V2KkJ+xwWCA8XgMRVF4py1BEN73sgghhICCb0IIIYQQQt4L0zT5D0ZPTk5QLBahqipGoxGGwyH/ +1e120W630e12UavVMBqNoGkar+xm87vf1Zrq9ToePX6Edqf9vk8RIYQQQsgH7ezsDAAQjUbh8Xgg +CAK63S729vbQ6/Xe9/IIIT8SRVHQ6XTQ7XahKApUVYXL5XrfyyKEEAIKvgkhhBBCCHkvWPA9Ho/R +brf5r1arhU6nw3+QwlqYswpvVlUAgFd5v6uW5rquo9lsYmtrC9Vq9X2fIkIIIYSQD9psNgMAxGIx +SJIEURTR7/f5xkZCyM/TbDbDdDqFoiiQZRmyLFPwTQghHwgKvgkhhBBCCHkPLMviQXa5XMbh4SGa +zSaGwyGv+mbV3SzY/rFnd+u6jna7jadPn8Lr9b7vU0QIIYQQQgghHxz2/2SWZSEYDCIUCr3vJRFC +CHmOgm9CCCGEEELeg/OtzqvVKvb399Htdnkb89lsBlVVL7Qy/zFDb+Db4FtVVcgy/a8CIYQQQggh +hLxIkiTIsgy73Q7TNOF2u9/3kgghhDxHP80ihBBCCCHkPbAsC6ZpwrIsjMdjdLtddLtdmKYJwzB4 +MP7ia37sNY3HY+i6DkEQ3vcpIoQQQgghhJAPjizLcDgccDqdUBTlwmZlQggh7xcF34QQQgghhLwH +oijyH5gkk0ksLy+j3W5DURQoioLpdIrJZMJnRwLzKnHTNH+0NcmyjGg0ilgsBofD8b5PESGEEEII +IYR8cERR5L9CoRD9vxMhhHxAKPgmhBBCCCHkPRBFETabDS6XC6lUCoZhoNlsot1uo9vtotfrQdd1 +HnSzau8fs925LMuIRCJYX19HMBh836eIEEIIIYQQQj44pmliNpthNpvB7XbDbre/7yURQgh5joJv +QgghhBBC3gNBECDLMtxuN3K5HKLRKAaDAfr9Pnq9HrrdLvr9PkajESaTCcbjMYbDIYbDITRN4+3Q +32UVuCRJiMViuH79OpLJ5Ps+RYQQQgghhBDywdE0jf+/GRtVRQgh5MNAwTchhBBCCCHvAWt17na7 +EY/H4fV6YVkWptMpb3M+mUzQaDRQqVRwdnaG4+NjnJ6eYjweX6gyeFdV4Cz4vnnzJgqFwvs+RYQQ +QgghhBDywRmPx2g0Gmg2m+j1euj3++97SYQQQp6j4JsQQgghhJD3QBAESJLEZ3zncjm43W4A83bm +s9kMuq6jXC7j4OAAe3t7cDqdME0T3W4XmqZBVVUelGuaBtM0L7RE/65EUUQ4HMbq6iquXLnyvk8R +IYQQQgghhHxw+v0+PB4P7HY7LMvCeDx+30sihBDyHAXfhBBCCCGEfGAEQYAkSoAMBINBLC4uwu12 +IxqNYm1tDa1WC51OB+12G+VyGbVaDePxGLquQ1VVmKYJXdd/tFnghBBCCCGEEEIIIYR8aCj4JoQQ +Qggh5AMkSiJESUQwGITP50M2m8W1a9egqhoajTqKxSKOj4/x4MEDaJrG26QDwGw243PmKPwmhBBC +CCGEEEIIIX8PKPgmhBBCCCHkAyQIAgDAZrPBZrPB6XTC6/UCANxuF/9vWZYRCARQr9cxGAwwHA7R +7XbR7/d5C3RWBf6uZoETQgghhBBCCCGEEPKhoeCbEEIIIYSQj4QgCLAsCy6XC4l4Ah6PB9FoFFev +XkWr1UKj0UC9XsezZ89wdHSEdruN8XgM0zRhGAZ0XQdAVeCEEEIIIYQQQggh5OeHgm9CCCGEEEI+ +IoIgwOl0wul0IhwJI5/PAwB6vR6q1SrOzs7g9XohSRLcbjc6nQ76/T40TeO/dF3nIThAQTghhBBC +CCGEEEII+fhR8E0IIYQQQsjPgMPhQDgchiiKAIBEIoFarYZms4lms4lKpYJarYZ+v4/JZILRaARg +HnqzNuiEEEIIIYQQQgghhHysKPgmhBBCCCHkZ4AF34FAAKlUCrdv30a320W9Xke5XMajR4+wubmJ +s7MzWJYFVVVhGAYPvE3TfN+HQAghhBBCCCGEEELI90bBNyGEEEIIIT8DoihCFEXYbDY4nU4AgMvl +gtvths/ngyAI8Hg8KJfLaDQaaLVaGAwGGA6HmEwmUBQFgiBAEIT3fSiEEEIIIYQQQgghhHxnFHwT +QgghhBDyM+VwOBAMBuFwOOByubCwsIBOp4NWq4Vms4nDw0McHx+jWq2i2WxC0zQKvwkhhBBCCCGE +EELIR4mCb0IIIYQQQn6mbDYbbDYbvF4vYrEYAEBRFPT7fbTbbTx8+BA+nw9utxuyLGM4HMLv90OS +pPe9dEIIIYQQQgghhBBCvhMKvgkhhBBCCPk7IkkS3G43AGBjYwM+nw9LS0uo1WoYj8f47LPPEAgE +3vcyCSGEEEIIIYQQQgj5Tij4JoQQQggh5O+IzWaDLMtwuVzw+/1YWVnBaDTCaDTCdDpFNBql4JsQ +QgghhBBCCCGEfHQo+CaEEEIIIeTvCJvhLYoibDYb3G43PB4PgsEgdF2Hw+GAzWZ738skhBBCCCGE +EEIIIeQ7oeCbEEIIIYSQv3OSJPHAWxRFCILwvpdECCGEEEIIIYQQQsh3QsE3IYQQQgghf+dEUYQo +iu97GYQQQgghhBBCCCGEfG/00y1CCCGEEEIIIYQQQgghhBBCCCEfNQq+CSGEEEIIIYQQQgghhBBC +CCGEfNQo+CaEEEIIIYQQQgghhBBCCCGEEPJRo+CbEEIIIYQQQgghhBBCCCGEEELIR42Cb0IIIYQQ +QgghhBBCCCGEEEIIIR81Cr4JIYQQQgghhBBCCCGEEEIIIYR81Cj4JoQQQgghhBBCCCGEEEIIIYQQ +8lGj4JsQQgghhBBCCCGEEEIIIYQQQshHjYJvQgghhBBCCCGEEEIIIYQQQgghHzUKvgkhhBBCCCGE +EEIIIYQQQgghhHzUKPgmhBBCCCGEEEIIIYQQQgghhBDyUaPgmxBCCCGEEEIIIYQQQgghhBBCyEeN +gm9CCCGEEEIIIYQQQgghhBBCCCEfNQq+CSGEEEIIIYQQQgghhBBCCCGEfNQo+CaEEEIIIYQQQggh +hBBCCCGEEPJRo+CbEEIIIYQQQgghhBBCCCGEEELIR42Cb0IIIYQQQgghhBBCCCGEEEIIIR81Cr4J +IYQQQgghhBBCCCGEEEIIIYR81Cj4JoQQQgghhBBCCCGEEEIIIYQQ8lGT3/cCCCGEkHfBNE1YpoWJ +MsFkMoFpmnC73XC73ZAkCYIgQBCE971MzrIsmIYJ3dAxnU6hKAoMw4TX64Hb7cZsNsNUmWKmzyBJ +EiRJgs1mg8PhgM1me9/Lf+/Onz9VVaEoCnTdgMczf891ff77mqbR+SPvjKZpUBQFqqpClmRI8vy6 +Yr8IIYQQQgghhBBCCCHvDwXfhBBCPnqmafKQ8/Hjx3j48CEURcHt27fx6aefwuv1wmF3QJKl971U +Ttd1DAdDDIYD7O7u4unTp5hOp3zNlUoF+/v7aLfb8Hg88Hg8yOVyKBQKiMfj73v5793587e/v4+n +T59iMBjg1q1b+PTTT9FqtbC/v496vQ6v1wuv14tMJoNCoYBkMvm+l08+UrVaDU+fPsXp6Sn8fj/8 +fj8SiQSy2SwSicT7Xh4hhBBCCCGEEEIIIX/XKPgmhBDy0TMMA6qqYjQa4fHjx/i///f/YjAYQNM0 +FAqFeWWmJH1wwXd/0Ee1WsWdO3fwu9/9DsPhENPpFAsLC9jb28Mf//hHFItFRCIRRCIRfPLJJwgG +gxR8Y/6eD4YDVKtV3Lt3D//+7/+OZrOJyWSCXC6Hg4MD/Od//icODg74+bt16xYCgQAF3+R7q9fq +uHv3Lh48eIB4PI5EIoH19XW43W4KvgkhhBBCCCGEEEIIec8o+CaEfLQsy0K308XR8RGOjo6+8+uD +wSAWFxdRKBQ+yFbYPzXLsmBZFvr9Po6OjnB8fMwrGSORCBwOBxwOx3tbV7FYxNHREabTKTKZDLLZ +LNxuNxz2+ZpMw4RhGBiNRuh0Ouj3+xgOh9B1HYZpwLKs932KX3lsiqKg1+uh3+9jMplAEATMZjOM +x2OMRiM4nU64XC5omgbTNN/3sj8Y7D1l56/T6WA8HgOYB+OTyQSj0QgOhwMulwuqqkLX9fe9bPIR +02YahsMhut0u7HY73G43FEXBbDZ730sjhBBCCCGEEEIIIeTvHgXfhJCPEgsMq7Uq/uM//gO/+93v +vvPnWFhYwL/+678inU7DbrdDluW/++DbMAzU63X84Q9/wO9+9zvcvn0b//RP/4SrV68iGAzCbrf/ +pOeIzXE2TAPb29v4t3/7N3Q6Hfzyl7/Er3/9a0SjUQSDQcjyx/ftTBAEyLIMp9MJWZYhiiL/PYfd +AafTCafTCbvdDrvdDofD8Xd/jZ7Hz9Xzmd3s/EmSxDdpOBwOfv7sdjtsNhsk6cOp+ieEEEIIIYQQ +QgghhBDy7nx8SQEhhOB58G1a6HQ62NzcxO9+9zs4HPOwUBRFHoK9zmg0wrVr12AYBlXRArBMC6Zp +otvtYmtrC3/6059gGAYWFhaQzWbhdDoRCAR+8nUZpgHDMHBycoJvvvkGjUYDwWAQN27cgNvlhtfr +/WiDb5vNBqfTCZvNBkEQIIoiJEmC3fFt2M3+eT7gJS+fvxc3DpwPvun8EUIIIYQQQgghhBBCyM/f +x5cUEELIK0SjUSSTSYTDYbjdbrhcrtd+fC6Xw9raGux2O291TshPhVd8O5wXrkFZli+E3k6n80Jw +K4lUsQxcrJi32+0XKr5tdtuFwJtVz7PKekIIIYQQQgghhBBCCCE/PxR8E0J+NiKRCK5du4aVlRVE +o1FEo9HXfnwgEMDi4iJkWab2x+QnxyqWHU4Hb8EtiiIPvl9Z8S1RcAsAovDtubLZbPw+fnHjwItt +zyn4JoQQQgghhBBCCCGEkJ8nCr4JIT8bLpcLsVgM+XweqVQKqVTqtR9vmiZGoxEePXwEZapgOp3C +5XIhl8shn89DEAQIggBDN6DNNGiahnK5jLOzM5imiVQqhXQ6DZfLBYfDAUmSoGkaZrMZWq0WWs0W ++oM+VFXFdDqFJEmQJAkulwvBYBCBQACBQAB+vx9Op5OvazabQdd1DIfD+edptaAo8/UZhsEDUo/H +A5/XB6/PC6/XC5/Ph0qlglKpBMuysLi4iMXFRdjkeVh6vqK90WigVqthPB7z0HA0GmE4HGJnZwfV +ahWGYaDZbOLhw4cYj8d8reFwGJFIBMFgEG63G263G51OB61WC71ejx8va9vtcrkQDAQRDAXh9/sR +CAQuHO/rDIdDnJyc4OTkBLu7uxgOh1BVFUdHR/jjH/+IcDgMv98Pr9fLQ85mswlVVWGaJiaTCTqd +Dnq9HqbTKX9/DMOA2+2Gx+NBIBBALBZDLBqDJEv82pjNZtA07dJjY5XF7NgCwW/fyzd1GmBYcHs+ +lL0sAD8f3LJW3W+7PqfTiVAwdGF9oiii0Wig1WphPB5DURTYbDZkMhnk83k+x900TWja/LqvVqs4 +OzvDdDrl173H4+Fzx9laut0uWq0Wut3upWsJBAIXrv3z58rQDcz0GSbjCRrN+fomkwmm0yl0XefX +E3vffD4fv/5Yu3MWfvOW8Zecv++yycWyLH4/ttttfmzT6ZQfmyiKsNvt8Hq98Hg8CIfDCIfD8Hq9 +/Bx0u100m030+/PngaZpfI3sPQqHw/D5ffD5fHA6ndB1Hbquo1aroVgsotvtzq/TWAzT6RS9Xg+D +wQCmacKyLPh8PoTDYQQCAX7cg/4AjWYDvV4Puq7z6569D6FQCKFQCKIgQjd0TCYTFItFFItFOJ1O +xGIxeL1edLtdfn1ZlgVRFBGJRBAOh/nzjz1X2P03m80gCAKCwfl9z76W3++HZVkAgFarhUqlgmaz +idlsxu9N9uc2mw02mw2BQACRSAShUAgulwtut5s/KxVFwcnJCU5PT/maA4EAer0evw5nsxksy4LX +O39WhsNhxGIxBINB/lxkz3lloqDRnB/HeDyGqqowDAM227yLwO7uLsrlMhRFga7rfK2EEEIIIYQQ +QgghhJD3j4JvQsjPht1u5+FPPB5/Y/DdarWwu7uLvb09tNttdDodRCIR/PrXv0Y6nYYkShCleSA0 +Go0w6A/wzTff4C9/+Qt0XccvfvEL/PKXv0QkEkEkEoFlWZiMJxiNR3j69Cm2trZwenqKbreLfr/P +q3gjkQiWlpawvLyMxcVFFAqFl4Lv0WiEcrmMx48fY3t7m4e3mqbxgDSZTPIQMpvNIpfL4ZtvvsGf +//xnGIaBf/7nf0YikQBcgA02HuoCQKlUwt27d1Gv1+FyueB0OtFsNlGv11EqlXBycgLTNFGr1fDn +P/8Z29vbPIhdWVnBxsYGlpaWEI/HkUwm8fTpU2xvb/OArt/v86AoHA6jUCjw42Vh39vo9Xq4d+8e +/vM//xPPnj1Dr9fDeDzGs2fP0Ol0+Lnw+/38HJRKJb5JYDgcolqtotFooFgsotFo8E0E0WgUqVQK +hUIBN27cmAeAzzcIGIaByWSC0WiE3d1dbG5u4uTkBL1eD71ej5+LUCj07Xu5sIjFwuJbB9+COK/4 +tiyLt+s+H8xKkjSvCL8k+DYMA4qiYDgc4tmzZ9ja2sLx8TE/9yz0DQaDfH0LCwsoFAqw2WzY39/H +1tYWGo0G2u02nE4n/umf/gmxWAyyJEOU5l9jMp6gP+jj4cOH+POf/4xWq8Wv+1QqhVAoBEEQXlrL +0dERP1dsLYFA4MJ1v7h48VxpMw3j8RiNRgOPHj3C9vY2ms0mut0uFEXh908sFkMymUQul8PKygrW +1tYubBxg895FUXzp/H3Xim/TMDGdTjEej/k5Ozg44MfGqva9Xi+/H9fW1nD16lW4XC4Mh0N0u108 +e/YM29vbOD4+xmAwwGg0giRJcDgcCAQC/DgWFxeRzWbhcDigqioURcH29jZ+//vf4+joCNeuXcPN +mzfR6XRwcHCAcrkMXddhmibS6TTW1tZQKBR42HxycoLNzU0cHh5iOp1CVVXEYjH+3NnY2IDX64Uk +SVAUBfV6Hf/1X/+FP/7xjwgGg7h+/TrS6TT29/dxdHSE4XAI0zQhyzLW1tawvr6OeDwOv98PAHj8 ++DG2trb45gBBEPh7vbq6io2NDR58W6aFcrmMv/71r9ja2sJ4PMZ4PObhNwC+yWFxcRFXr17F6uoq +YrE47HY7LMvCdDpFo9HAX/7yF/zhD39AIBDAzZs3sbi4iIODA34dKooCy7KQTCaRTCaxsbGBmzdv +IhgMwrIsCIIAbaZhOByi3W7jwYMH2NzcRKPRwGAwgKqqcLvd8Hq96Pf7qFQqGI1GmE6nME3zR/ue +RgghhBBCCCGEEEII+W4o+CaE/GzIsnyhmjESibx2bvd4PMZwOESxWMTJyQlKpRIikQhSqRQ+++wz +HghrmoZ2u41yuYwnT57g7t27sCwLsVgMN27cgNfj5dWZzVYTtVoNOzs7PCzt9/sYDAYXgmAWfpmm +iWAwCI/HA0mUIIgC+v0+arUa9vb2sLm5iQcPHvDqTl3X+eep1WpoNBro9/uwLAt+vx97e3v4y1/+ +AsMwsLCwgOl0ysNUCd8G341GAzs7OygWi7x6ttls8krgwWAAwzAwGAygaRoajQafoazrOj83qqpi +MpnwcPj4+JgfLwuHg8Eg+v0+RqMRLMvilceSKF0I4y8znU5RqVTw7Nkz1Ot1Xmnb6XQwGo14xa3P +54OqqrDZbBgMBrxKt9froVgsolQqYW9vD9VqFYqiYDKZIBwOI5FIoNvtwufzYXl5GV6vFzbZxqv2 +2Xt5Psxlx8YqUfv9Pg8E/YF5RfvbHJsgCJAlGYJD4FXngiDwqmxWIc0qXdl1wiqs2602qrUqdnd3 +eSDb7/fR7/dfub5AIACv14tms4lisYijoyOcnp7CZrMhFovh9u3b/GsbhoF2p41qtYqnT5/i3r17 +aLfbCIfDuHr1Kg9XdV1Hp9NBpVK5sBZ2rliA7/f7+XWg6zqvAmYV4cPhELVaDUdHR9je3sa9e/f4 +tcjeW7vdzjcsjMdjBINBXNm4Ar/fzyuh/X4/ZEl+7fl7E1bFq83m936tVsPu7i4ePXqEZ8+e8XMq +CALsdjs8Hg9qtRqazSZkWUY6nUY8Hufn+cmTJ3j48CEODw8xHo8xmUx4pbjf78dwOOSbNdxuN3w+ +H6bTKRRFQbFYxJ07d7Czs4PBYADLstDtdrG/v4+zszNebZ9Op3nXhmg0ikgkgoODA2xubmJ/fx+K +okBRFEQiETSbTQwGA7jdbuRyOTidTl6Z/uTJE/z5z39GIBDAdDrF0tISdnd3cXh4yDsuyLKMwWAA +RVGQSqUQjUYhiiKePHmCx48fo91uYzqdwrIsNJtNtFotGIaBSCSCdDrNz3Gn08H+/j4ePnyIyWQC +RVEuBN9OpxMulwudTgeGYfDneSg0r9RWVZWv+S9/+Qt8Ph80TeObYw4PD9HtdjGZTGAYBuLxOBKJ +BGazGSKRCLLZLK8qH4/HqNVqODk5wfb2Nu7fv496vY7xeAxN0+DxeOD1evmmGMMwMJvN3knwbVkW +TNOEqqr8/iDkXWCdawDwbhis28T5PyOEEEIIIYQQQgj5uaDgmxDys2JZ1ku/LiMIAjweD5aXl2FZ +FgzD4CFHqVTCgwcPkMvlkM1mMRrNQ5QHDx6g1WrxdsK5XA7pdBo+nw+yLPNK1bt372I8HkOWZSwt +LfG2w6zVMGt/fPfuXd4KmIX2Ho8He3t7uHPnDg4PD9Fut2Gz2ZDP5+HxeGC32/lxmqbJq5pVVf3B +545V0LbbbTx8+BCdTgeJRAKFQgGJRAIulwsulwvZbBYLCwuw2+04OzvDnTt3eAVroVDglbmGYcAw +DGiaBkVR8ODBAwDgrd39fj/8fv9rf/DOKjh1Xce9e/ewubkJURSRz+exvLwMt9sNl8sFn8+HaDSK +WCyGfr+Po6Mjvlnh/v37PNiNRCK85flkMsFwOMTh4SHS6TTy+TySySTC4TBGozG2trZw9+5d9Pt9 +CIKAhYUFrK2twW63wzRN6LrOj+3Ro0f82NxuN/x+P3w+32uriwVBgCAKkCBhZWUFv/3tbzEcDnH9 ++nW+3ps3byKXy/EWzZFIBC6XC4PBAJtbm7h79y56vR4AYGFhgVc1v7i+x48f83bYhUIB0WgUn376 +KQ8mh8MhyuUyHj16hKWlJWSzWYiCiKOjI9y/fx+VSgU+nw9+vx8LCwtIp9P8uh0MBvxcdTodWJaF +fD6PlZUVOBwOvhbWlnpzc3O+SeD5uWLv4dHREe7cuYPd3V10Oh0AQDqdxurq6oUOAey6YoG4IApY +WlrCP//zP6PdbuPWrVtwupyIRqO4fv06kskkP3/RaBQ+n++N94JpmNANHZ1OF48ePcK9e/dQqVTQ +7XZ5dbfb7ebV+ewZwq4pRVHQ7/fx5MkT/PWvf0Wz2YSu60ilUvyYWSt5Xdehqiru3bsHRVHgcDh4 +Jfz555dhGGi1Wnj27BmcTieSySRisRgGgwEP4cvlMlqtFgKBAEKhEEzThM/nw5UrV3jwPZvN0G63 +sbm5iVgshvX1dQSDwZfC1slkwscm2O12/jkGgwHfgMI2uwQCAXg8HkynUywuLiIej/OP0XUd+/v7 +cDqdyOfzF65Tp9OJUCiEdDrNA2gWyFmWhdFohMlkAsuyeJcH0zSRzWbh8Xheet9GoxGOj495SL64 +uIhMJgNN03jlfr1ex+HhIbLZLGKxGEKhECKRCCqVCv72t79hc3MTrVYLoigilUrB4/HA5XLxNvrt +dhulUgn9fv8HP3MB8PtDURScnZ2hXq9TGEl+MLahiI3NsMk2OJwO+Hw+eL1eOJ1OuF3uN27QIoQQ +QgghhBBCCPnYUPBNCPnZYYE3qxp8kSiIgAh4vV6srKwgEU+gXC5jc3MTvV4PpVIJjx49gmmaz0PQ +Ifb39/HXv/6VV2wvLS1hcXER6XSaV011Oh1sbm7id7/7HRKJBLLZLA96CoUC+v0+er0ejo6OcPfu +XWxtbcHj8SCbzc7n7QZDkCQJ+/v7+NOf/oSTkxMeDrO2xNFolAdKJycnODo64kGUZf6wWbO5XA5X +rlxBr9dDu93GkydPkEwm8fXXX+PWrVsIheZziNlMZdYu/fe//z2vnmRtjQuFAobDIXq9Hk5OTvjx +Op1OZDIZxGIxiKIEn8/3xuD7k08+QT6fh67rKBaLMAwD6+vr+Jd/+RfE43G+Jva+n5yc8Gr0Wq0G +RVGwvLyM9fV1JBIJGIYB0zSxtbWFra0t9Ho9ZDIZLCwsPA/5HBgM+tje3sZ//Md/IBAIIJvNXmgX +Ph6P0e/3USwWcffuXTx9+hQ2mw2pVArxeByCIMDr8QJv6KotiiIswcLq6iqSySSvhLbb7UgkEryi +ms2Hl2WZz+h+8uQJfv/738Pr9SKbzWJpaYmvj1Xvnp6e4u7du9jZ2YEkSUilUgiHw4hGo8jn8+h2 +u7zlebVaxfb2NkRRRCAQgM1m42H0bDaDz+dDOp1GoVBAJp2Bx+uBIAioVqt48uQJ/vCHP8DlciGT +yaBQKPBrgVXlnp2d8bUAQDKZRDwe51XfJycnvK0+2ziQy+WwsbGBTCbDr/tSqcRbhquqCkEQeNt9 +tk4265lVALPQ0mazwW6zv/FeMMx5NW+v18Xm5ib+/d//Hbquw+fzIRaLYWNjA1euXIEsy1AUhV/n +JycnGA6HmEwmGAwGePr0Kf70pz9BEAQkk0n+Pq2srEBRFLRaLf7s2d3dhaZpSCaTSKfTfKMJo6oq +r5AuFApYX19HKpXinRpKpRLOzs7Q6/V4dXImk8HS0hJWV1d5i//Dw0Ps7u6iVCphaWmJV6m/OPt8 +PB7zVupra2u4cuUKZrMZGo0GarUaSqUSDg8P+SaiUCiExcVFrKys8NbpvV4PT548wfHxMRwOB65d +u4Zerwe/3w+bzcaD72w2i0QigUQiAa/XyzdvHBwc4Pj4GOVyGUdHRzg4OEAqlcLXX3/N53y/uObT +01NMJhPeWt3tdsMwDIxGIzx8+BDFYhGyLCObzSKZTMIwDHjcHlQqFdy5cwd/+9vf4PPNZ62z+z6b +zfLwfH9/H71e750F3+x7laIoaDQaGA6HVPFNfjBW1c1Gi7CxCvF4HLFYDIFAAHa7nYJvQgghhBBC +CCGE/OxQ8E0I+dlot9vY2tqCoigIh8OIRCKXfhyrrGbtkWPxGHK5HJaWllAqlTAej3kFr91uh6qq +KJVKaDQafFb11atX+SzewWCAwWCAarWKTqeDyWQCt9uNxcVFXLlyBdls9nnl+AjDwRAAcHJyApfL +BU3TUK1W+WzywXCASqWCVqsFTdMQiURw7do1rKysYGVlBaFQiAcwrEW5ZVlIJBIQxB9WJeh2uxGL +xmC323ng5nQ6EQ6Hkc1meUX1ZKJgMhnzds+TyYRXgbPzks1m5+FffwBRFC8cb61WQ7FYnFetJhKv +DYdtsg3BYBBOpxPBYJBXhPp8Pj7fPBqNwuVy8YpWp9PJ5ydHIhEUCgWsra1hY2MD6XQapmnCNE0M +h0OUSiU+Q7per8PpdEIURT7rdzweIx6PI5/P49q1a8hkMsjlcnymtSzL/NhYKFgsFnlwfb69/Kuw +4NDldMG0TF6lx2ZWW6YFQZxvrhgOhxgMBjg7O0O73cZkMkEkEkEul8P169f5uZ9OpxgOh7Db7SgW +i3C73fNW/M0mzs7OUCgUeOBYKBSgqipUVcWTJ09gWRZsNhtcLheKxSJqtRoPyq9cuYJcLge3xw1N +0/i8406ng/F4zDshnF+LpmkYDAZwOp0vreXw8BCRSATD4RBnZ2doNptQFAW5XA7Xrl3D6urqfHNK +IsGv+0AgAJ/PB13XkU6nIcvyW50/dq7fpppWURS0222cnp6iVquh0+nwDRBXr17FysoKVldXIYoi +P9derxeBQADRaBSapuH09JRX04fDYeRyOdy8eRP5fB75fB6z2QydToePGSiVStB1nY85SKfTvC04 +ANhsNj5Tfn19nW8IaLfbaLfbsCwLlUoFiqIgGAwiFothYWEBq6urWFhYgKZpUFUVs9kMJycn6Ha7 +/Dr3eDwvVVC73W6k02msrKzwoN80TbTbbUQiEfT7fRweHsJut8Pn8yGTyfBno91uh6ZpaDabqNfr +2Nvbw2QyQb/fR6fTgSzL8Pl8CIVCWF5ehsvl4s8Xdg/OtPm4Alb5XSqV0Ov10O12MRqNeNvx81wu +F5LJJJaWlviavV4vLMtCr9fj9ycADIdDVCoVAPPwmV1/4/EY6XQa165d4xs4UskUVE3l3SJKpRJa +rRZsNtt3mhl/GdM0MZvNeIV9sVjEZDL5QZ+TkPMV3+xZ6PF4EAgE+DiYYDD47X8HgvD55109HA4H +dR0ghBBCCCGEEELIR4uCb0LIz0atVoOqqnj27BlvL34ZVvG6vLyML774goeZN2/ehM1m47O8gecV +uZaFcrmM8XgMr9eLjY0N3Lp1C8lkEgDQ7/dxenqKUqnEg9dEIoFr167hxo0bvLUoq4pUNRWHh4c8 +IG02mwCASCSCTqeDRqMBVVXhcrmwsrKCf/7nf0YsFkM4HIbL5YJhGNB1A9FoFKsrq9BmGq8E/iEk +SYLNboPD4bgwC9Rmm/8em7fb6/VQrVZRqVQwnU7hcDiQSCRw9epV3Lp1ix+v1+udt1A2dBwdHaFY +LMLhcKDVaqFYLCIWi82DyteEw4IozMNLy7pwjGx+OFsTm0mt6zr/GKfTidXVVfyP//E/UCgUkE6n +EQqFePto1vKYabfbkCSJt+SeTCZwOByIx+O4cuUKbt++zY/N5/MhGAzCMAwsLi7yIL/dbuP4+BjR +aPQ7z/4VJRGC9W0wy/8pfRtADIfzAJBVtbKW6BsbG/jkk0/g8Xjh83n57HgA/Fpjs5JPTk4QDodh +t9uRTqdx/fp1AEC5XMazZ88wm80gCPO546VSCcPhENlsFqurq/jss8+QSCRgk228ivvk5ASj0Yiv +ZX19HZ988gm8Xh9fSyAQgCiKODw85BsF2JxqFtzW63UoigK73Y5CoYB/+qd/Qi6XQyQSgcfj4ZX6 +sVgMhUIBlmkhkUzw+/xtzt/bhjksiD85OeHzpcPhMG7duoV/+Id/4BtrBEHg7fyTySSuXr3KZ3if +np7yquBIJIKrV6/i66+/5sG9aZq89Xq9Xke9XockSWg2m9jZ2YEoiojFYnxNDocDi4uL+M1vfoOV +lRVks1mEQiGMx2MMh0PeWp216P7iiy+wvr6OpaUlpNNp3iK+0+lga2sLzWYTmqah0+nM56K/MPs8 +FArh5s2b+OUvfznfxJDJwjANjMdjRKNRHB4eQhRF+Hw+rK2t4bPPPsPS0hKWl5fhdDihGzoajQZ2 +d3fhcrlgWRbG4zEP+9mz+MqVK8jn87zC3TQt6Pr8HkwkEgCARqMBWZb5iILRcISJX4FhXKyMDgaD +uH37Nn75y18ik8kgm83C6XTy4Jt1yRBFEaqqol6v8zbj7BkvyzLy+Tz+4R/+AUtLSwj4A/D6vNB1 +A4aho9/v49mzZyiXyzyk/yFY8M3GbGxvb2M4HP6gz0kIwwJwQRD4OAG2WcXn8yGZTGJxcZF3DFlc +WOQjVSj8JoQQQgghhBBCyMeIgm9CyM9Gv9+HqqpoNBo8tL1MNBpFPB6HLMt8ZnM6ncaNGzcwm81Q +rVZRq9XgdDohyzIcDgf6/T5kWUYkEsHq6irW1tZgs9l4FW6lUkG1WsV4PIYgCHA4HPB6vXC5XNB1 +nc9hBuZhusvlQiAQgGmaaLVamE6n84rw4RDdbhemacLr9WJhYQG/+MUv4HK5eBj9rTQsy+Itk9vt +9g86f+wH4+cDZhZ8sx+WOxwOTKdT3u54PB4DmFfGe71e3lb4fBtgURT5TG/LstDpdFCtVjEYDF45 +g/38mtga2A/v2e/Z7XbYbfM1ORwOaKoGURT5xzgcDiwsLOAf/uEfkEqlEAgELrSOZu2Vh8MhD8YA +QNM0aJqG0WgEYB44ejyeS49NkiS4XC5+bN1uF9VqFf1+/zsF3y+GtS/+PjMej1Cr1VAul3k4dv7c +W5aJwWBw6blnoafT6cRoNIIsy0gmk7h27RqfH802YciyjFAohHa7DVEUeaUxa+8tSiJGo/lazs7O +XlqLx+N57Vp0XUe324Wqqvy6b7fb0HUdTqcT2WwWn332Gb9Pz1/3qVSKXzdvOl/fN7gZj8f82Ng9 +HQqFsL6+js8//xyyND8HL35+y7Kwt7eHx48f4+zsDJPJBLIsIxwOY3V1FTdv3nyp6tztdvPNCZ1O +B91uF5qmIZFIXGh5Lcsy0uk0vvjiCxQKSwgE/PB4PPx6ZQGzKIqIRqO4evUq7zgRj8f559nd3YXf +74ckSdB1HYPBgFfrn+f3+7G2toZf/vKXCPgD/M+1mQa3241IJMKfZQsLC7h9+zbfRMRmsgcCAV7F +LQgCn30+nU5hmibcbjdsNhs0TcNsNns+73w+i3s8HkPXdTgcDthsNh4QT6dTTJQJVHX60vlna/7V +P/wKgWCAt+wHgG63y8csTCYT6LqOdrvN73W24Yh9P/jk9idYXFx86X0ulUoIBoNwu92QZfkHh4OW +ZfFjGwwGaDabF+4bQn4Idn2xf7INRD6fDy6XC/F4HLVaDf1+H4qizJ8Lhg632w232/3WXTIIIYQQ +QgghhBBCPhQUfBNCfjZisRiSySTC4fC89fG5kPM8Nqt6YWEBmUwGwLxScHFxEYPBAEdHRzg6OoKq +qjg4OOAVvl988QU2NjYQDocvzMSdTqcYDAY8tOr1etjc3MRsNsNf//rXl75+r9fD8fExTk5OeOWw +KIq84tIwDLjdbgQCAXi93guh74vexw+l2fF2u110u130+/N52Lqu486dOy99PDunxWIRHo8HwWAQ +qqryNsY/lvMh+WXnSJKkeXhut0MQBF5NKooiFEVBp9PhM4pN08SDBw9e+hzD4ZC/l+w9i0QiP9qM +XlVVX7rWdnZ2YFkWb89/3mg04ufe6XQiEAggHA5D13WYpgm/349cLofJZILj42McHh7CMAycnJyg +1WrB5/Ph1q1buHr1KmKx2Py6FyV+vs5fB71eD7u7u7AsC5ubmy+tZTweX6j8DwQCfOOIIAiYzWZw +Op287bbdbudf67L39sekqiqGwyGGwyFEUYTf7+cbJ0RR5K3nL1sXC1JHoxEEQeCVlWyjzIskSeLd +ICaTCW9BzsLhFz//vILz2409siQDdvANAuxjXvVsYM8a1qabBWKXHQv/xVrFiwKflX5+Iwq/1wTx +0uM7//EseGObYNhs8larhVarhfF4jOl0ClVV+ccdHx/zjSmWZfGQ/MUqdXa/S7L00loEQeAbmTRN +g2maUFUVoijCMAxMp1M+c9zj8UC2yT94fMTbYO+H3W5HMpnEysoKVXyTd4LdP7PZDLPZDKqqQlEU +TKdTAPPv5WxMQrfbRblcxtHREVZWVrC2tsY3+L2Llv6EEEIIIYQQQgghPxUKvgkhPxuRSATXr1/H +6uoqotEootHopR/ndrt5xXUymYQgCAgGg/D5fFAUBU+fPr3QdjwQCOCLL77Ar371K6yvryMSifAq +QmAekvX7fR7+DQYDPHv2DKVSibcMPU/XdUynUyiKgmg0ykMP9meGYfCA2OPx8KrC14V9P2X4fT58 +7fV6vPXv2dnZG4+XVYmy4PvHxsKuy8JvFnyz95LN2WXzv9l7ub+/j3K5fGnrfHZs0+kUoVAIgiBA +VVUYhvGjhPovhs2sWrdSqVy6PsMweNARDAZfWp/f5+cbRHZ3dxGNRnlFuc1mw+eff47PP/8c165d +Qzwef+m6H41G/LpnM99ZC+i3WQu7Xli7cJfLxWfX22y2S6uqfwqs+pYF32wOrsvlemOV72w2uxB8 ++/1++Hy+V87NFUURHo8H4XCYP3MmkwlUVX3pGjq/kYOHzZIIu2Tn4fJlH3eeJEmQZZnfF2zmvWW+ +fL2+uHHkfAeI81/r/Nc5//VYm+XzH8+qTlnwvbOzg62tLRwdHeH09BTD4ZBfo6ybw3g8Rq/XgyRJ +/PWGYUASpUvXyzYnnA+/WUcLh8PB73P2dRRFgaqqfKSDz+eDLP80YR87Ry6Xi3czUBTlR/+65OeN +3dfs+h6Px/z7NgvD2e/3+32USiWUSiXs7++jUqlAFEVks9n5M0+SAcq9CSGEEEIIIYQQ8pGg4JsQ +8rPhcrkQi8WQy+WQSqWQTqcvDZpYtSOrLgXAwxy32w2n08lDRFVVMZ3OW+q63W4+//r852WzpU3T +5J+DzRH3er0vff3zwU0kEkE8HofD4eBzpS3LutDeG/iwZm2ytrzseD0eDwKBAJ9X/LrjDYfDiMfj +KBQKiMViP3qw9LrzxgIytgbLsmAYBj82u90Ot9sNn8+HaDQKv9//ymNjM7Xj8TgWFxcRi8X4e/dO +z71h8s0R7FrzeDyIxWJvtb5YLIaFhQXeQlySJUiy9NJ1r2kaDyfZn71YscyuA8MwIMsyb40bjUZf +apv94lpYC2yv13vhumeh7Plq4vfBNL69Zl8cAfCmNbHriL2WHdOrrnUW1rKPYa9/Vav8F8/LSzPN +37C+14Xib/pa53/vssD7sgrpFz+OjX4oFos4PDzEzs4O9vb2oGkar6w/XwVts9lQq9Wg6zq/Ttgv +0zJfu/ZXHfuL9zz79/PXnyT9NEkfG9vgdruRy+UQj8f5mgj5vs4/R9iGoxc3Kg0GA/7P0WiEfr/P +Oyn4/X7Y7XYsLCygUCjwDh2EEEIIIYQQQgghHzoKvgkhPxs2mw1erxfhcHje9jyRvPTjxOdtglnV +40t/zioGn4c1lmU9nz376vbcpmlCkiT4fD7EYjGsr6/j1q1bSKVSl66BtSBlwepkMsHBwQGOjo4A +vJ8W5m+LhU6SJMHr9SIajfL5xax1/OuO1+v1IhKJIJvN/ijh8Hdx2TlmGw88Hg+i0SgKhQJu3bqF +XC73VscWDoeRzWbnVXLvmGmZfH3sPBYKBdy8eRP5fP6163O5XBfX98K1f/66P/9aVVWfB9wvh4zn +z1UkEsHCwgJu3LiBxcXFS9fC2mqztRiG8crr/n1f/+w6Z77ven7K4/hQzt2Lazr/vhqGgUajgeFw +iL29Pezu7qJUKmFxcZF3FohEIvD7/fw1d+7cwWg0wunp6Y+6PhaI/5Tnj30v8ng8SCQS8Hq9l35f +IuS7Ys8wfaZDN3RomgZFUaAoCsrlMqrVKo6Pj7G/v4+TkxMYhoHBYIBSqQRZljEcDvHVV18hFArx +jVUf0rOFEEIIIYQQQggh5DL0kzVCyM+GLMtwuVzw+/0IBoMIR8Jv/VpWuaooCq92lSQJDocDdrsd +mqah3W7zNrwsEPt25q7EK3B9Ph+Wlpbwy1/+Euvr66/9upIkQZZkVGtVDIdDnJ6eQhAEmKbJq8jf +1vlwif3A+3WVo+zjzr/+bbCAlB2v1+tFoVDAV199hWvXrr3V8co2+ZWtn191XK/62HfZUpyFXzab +jYezhUIBX375JW7evPnWx2a32yH+CBWjoiC+tL6FhQX84he/wO3bt99qfZI8v65lWYahG5jps5eu +e7vdDkmSoOs6b6muqvOZ0y8Ghaza2+v1Ip/P4xe/+AU+/fTTN65FFEX0e32Mx2OcnZ3x+c/nZ0C/ +L2yWtSRJFyq42breNHZAFMWXXvuq42GbAdgz6Hx7/g/d9wnBDMNAp9OBpmk4OTlBpVJBv9+Hz+fD +zZs3ceXKFWSzWcTjccxmM2iahul0ivv37/8owfeL1fbnf/1U55B9r0kmk8jlcnC73T/J1yZ/fyzL +gmmYODk5wfHJMR8JM5lMMBgMMBgM0G63YRgGer0egsEg1tbWEIlEYLfbLx2pQQghhBBCCCGEEPIh +oeCbEEIA9Pt9tFotHB4eolKpoNvtwuFwIJ/Pw+l0Yjwe45tvvoHNZkM6nUY0GuVht8PugN/vh9vt +Rr1eR6/X4+3RnU7na9sjswDR6XDC6/UiEAig1+vxmbbj8ZiHka8Kf8/Prj0fmOn6vMJLn+kwHRfD +bxa2vRgwvtgC+bIQyOGYH6/H40G9XsdgMICiKPx4z1dOvup4WfD5NsGZKIgwBfOljzcM47Wtjr8P +u90On88HTdPQaDQwGAwwmUy+87G92A7/na3PMV+fx+PhVbPfd30AMBgO0Gq1cHBwgHK5jE6nA0mS +kMlk4HA4oGkaHj16BFmWkUwmkU6neTtom80Gn88Hr9eLZrOJ4XDIZxO/7Vqmzim/7tvtNhRF4TOy +Wcv599H9gHWP8Hg8aLfb6Pf76Pf7mE6nMA3ztbPHz5+XTqfDWwlPp9MLG2YY0zQxGo3Q6XSgKAof +w/C2G0M+ZK/qjqEoyvPNFCrcbjfi8TgymQwWFhaQSCT4CIofO4Rmc+bdbjdmsxmm0ylmsxkmk8kb +242/z40ZhPwQgiggEAwgn8/zMS6ZTAZbW1t49uwZf5Y3m00Ui0Xs7OzAbrcjlUohmUz+8AUQQggh +hBBCCCGE/Igo+CaEEAD9Xh/FYhFHR0eoVqvo9XrIZDIoFAoQRRGnp6fY29tDIpHArVu3oGkar3xy +OB3w+Xxwu93QdZ2HXKIowul0zoNGUbp0/q1lWvxzsFnZgiBgPB5DkiSMx2Poug6bbIMgXd6W+3zF +IAtcWXv26XSKmWf2UkhzWfDNg1HxYsj+4sex4NvtdvPWqOeD77c5XgBvXREtiAJE62LQyNb/rsMn +Fniqqnrh2AB8p2O77M/fBYfDwUNV0zRfCpu/y/oEQeBtbc8H36FQCPl8Hj6fD8ViEdvb2/D5fLh+ +/TpUVeUbLC5by2Qy+U5rURwKPB4P/H4/JEnCdDqFYRj8umfzw196/SUB8rtkt9vh8Xjg8Xj4dcDu +a8OcB6KS/HKbfsuyYLfb4fV6L5yX4XAITdMurRZnx8uCb9b2mm2a+blhnTW63S40TYPH40EwGEQ2 +m+XBtyzLL93vP0bQfD74Ho1G0DSN31Nv2ymDkI8J+94eCATmo0oiUSwuLOLWrVuw2+1ot9uYTCaY +Tqe8C82zZ8/g9Xp5VwJCCCGEEEIIIYSQDxkF34SQn43ZbIbxeIxutwuv1/vGdrGSKMFmt8Fut+Os +fIbNzU0cHh5CEAQsLCxgeXkZGxsbmM1m6PV6vAp4b28PPq8PqXQKmUwGfr8fmUwGzWYT+/v7fIbt +kydPIEkSwuEwwuEw/4GzruuYTCYYj8d8LVNlClmWkc/nUSqVIAgChsMhisUi7t27h0gkglAoBKfT +CcMwoOsGBoN5FaplWQgGgwgGg3C5XHA4HFAUBa1WC9vb20in04jH43A4HBiPxxiPxtjZ2cHZ2Rm6 +3S5sNtuF+Z2SPG917XQ6MZvN0Gw2USqVAMyDYV3XeVhlt9thGAZarRaePHkCh8PBj5dVaBuGgclk +gslkwsMkp9OJUCiESCTyxrbRlmVBFEQetkqShNFoxOeQSqIEn9+H8XiMyWQCVVW/U4v48zweD5LJ +JHRdx8HBASzLQrvdxs7ODtxuNz82tsHgsmN78Ry8Sx6PB6lUCp1OBwcHBzBNk6+Pze9+0/rsdjsC +gQCCwSBqtRq2t7exu7uL2WyGTCaDbDaLjY0NOBwODAYDFItFtNttHBwcIBaLIZ1OI51O83PF/sw0 +TXQ6Hezu7sLv9/O1sBDTNE1+3bO1zGYzAEAmk0G5XIYoijyMf/DgAbLZLEKhEN9koesGRqMhBv0B +DNNAKpVCOp2GzWZ75+c5kUggnU7j6OgIhmGg3W7j2bNniEajCIfDCIVC/ByrqsoruyeTCex2O7LZ +LI6OjqDrOn+/UqkUAoEAfD4fTNPEdDpFpVLByckJTk9PIUkS4vE4stksotHoz3LWM2stzu5lTdOg +qip6vR46nQ5kef5eGoaOZrOJZrOJ3d1d9Pv9H2UtPp8P6XSab0zo9/solUrY3t7GcDiEy+WCzWbj +99HTp09xdnaGfr+PQCDwxspwQj5ENpsNNpttPtrCYYfH68HKygpOT09hmiZqtRpqtRra7Tb29/fh +9/uRSCTe97IJIYQQQgghhBBC3ujn9xNVQsjfLU3T+HxKm832xmpQ1tba5/Ph9PQUjx8/Rr1eRywW +w2effYa1tTWsr6+j1+uhWCzyIPDp06fzYFEUkMlkEAwGsbCwgG63C7/fD1EUUa/X8c0336DdbmN5 +eRmrq6u8vbeiKKjX62g0Gt+2Onc6EQ6Hsbq6ir29PdhsNvR6Pezv7+M//uM/sLy8jOXlZQQCAahT +FVN1ipOTExSLRdhsNnz22WfIZDLwer1wuVzodruoVqu4d+8e8vk8FhYW4HA4UK1WUavV8OjRIxwd +HUFRFN5q+HwrbKfTCZfLhdlshkqlAr/fD9M04XA4MJvN4PP5EA6H4XK5IAgC6vU67t27h36/j6Wl +JayurMJmt0EURahTFbV6DY1GA8A8zA6FQlhbW0MoGLq0cvY8QRAgSPPKTJfLBVmW0e/3eTAtiiIi +agSKokBRlLdqU/wqPp8PuVwOlmXh8ePHEAQBjUaDH9vy8jJWVlbgcDggiiI0TUO9Xke9XufVvIFA +AGtrawgEAu88+Pb7/cjn85hMJnj8+DEkSUKz2cT9+/cxGAz4+li18GXr83q9KCwWIMsyKpUKtre3 +sb+/j1gshps3b2J5eRnr6+sQBAFnZ2d49uwZRqMR9vf3YbfbYZomEokE/H4/crkcJpMJNjc3+Voe +PHiA4XDI1+JyuSCKImazGV+LYRgQRRF2ux2hUAiLi4soFouw2+2YTqc4Pj7Gn/70J6ysrGB5eRmx +WAyqqmI6neLs7AzFYhGz2QxffvklIpHIOw++fT4fstksxuMxtre3AQDtdhubm5swDANLS0tYWVmZ +t2ufTtHv93l4HYvFsLa2hqWlJTx+/Jhvnnj69Cnf3JLP56FpGjqdDu8mcXp6imQyiVgsho2NDcTj +8Z9l8M2ed8FgEP1+n88WPjs7w9HREUajEURRxHQ6xd7eHvb29rCzs4N2u/3O1yJJEoLBIBYXFzEe +jyGKIkajEY6Pj/HNN9+gWq0iGo3C6/WiXq+j2Wzi6dOnODo6QqvVQigUouCbfNRYtxgAWFpaQrfb +hWVZ0DQN1WoV/X4fx8fHCAaD2NjYuLRrBSGEEEIIIYQQQsiH5Of3E1VCyN8tRVHQaDRQLBYxHA7R +7XZf+/FutxvRaBSRSASnp6c4Pj7GeDzG8vIyPv/8cywtLWF5eRm1Wg2ZTAbhcBiqquLo6AiiKCIe +j+PatWtwOp1IJpPI5XJIJBLzcFpVcXh4yNs/z2YzXl01Ho9RrVZRrVZ5ZXU8Hkc8Hsfi4iJSqRRC +oRBGoxFarRYeP36M8XgMTdMQDocxmUygKAqOjo5weHgIn8+HQqEAp9PJK2273S4GgwGePHmC4XCI +0WgEh8OBSqWCarWKYrGITqcDAFBVlZ8T1vrX7/cjEolAFEU0Gg3Issxbi7vdbjidTvh8PgQCAYRC +IWiaxkOryWQCXddht9shSRImkwk/Xja/OJlMIpFIwLRMSHhz8M3er2AwCI/Hw4+ftWGPxWK8/TD7 ++t+H2+3mrZYTiQRCoRB0XcfJyQmvKJ/NZryVt6Io/NhYFWsikUAymfzeVeev4/V4eeV+IpFAMBiE +rusoFosYjUYYj8eYzWZwuVy8dThbHws4QqHQvL2zx42zszOcnJygXq8jn8/j1q1bPLTVNA2bm5uI +RCIwDAPFYhGmaSIajeLKxhW4PfMNE4qiIJlM8nN1enrKr4PZbAa32w1JkqCqKr/+2BgAVhWeyWSQ +TqcRjUZRr9fR6/Wwvb3NW+6m02l+3bORBJZlIZvNfu/3+k3XQTKZhKIo/J42DAOlUgmapvFjY9d3 +t9vF4eEhjo+Pcf36dVy/fh25XA7xeBw+n4+fF8Mw+KxwNkO3XC6jWq1iPB7DZrPh/7N3n89xpeeZ +8K+TT+fcjZwJgnGSNLIky7JXr/1lvf/rlsrrda0lq1yWPJrATIIZGQ00Ouc++f3QPM8AQ84MOUMO +CPL6VaEmEOg+qU+DfT33fU9MTOD86nmYEfO1B/pvA0VRxJz6RqOBIAjQ6/VwcHCAhw8folariUrw ++/fv4+HDhyiXyyc6ZLwusiwjlUphamoKtVoN8XgckiSh0Wjg7t27qNfrYpHH0dERKpUKtre3UavV +MBgMYNv2G3mdE/1UZFmGLMtQFAWzs7NwXRfD4RC7u7sn3r9zuRyazeb4vicrkBWZATgRERERERER +vZUYfBPROyNs+VypVBCJRBCJRL7z+xOJBCYnJzE5OSnai+dyOdHqOZ/PIxqNIpVKYWVlBZ9++inK +5TIqlQqePHmChYUFlPfLiCfiiMViyOVyuHr1KmzbRq1WQ6PRQLfbxfr6Ora2tsQHzOH8bdd1MTU1 +hVQqJYLDaDSKc+fO4e/+7u/w9OlTtNttdLtdbGxsoFKpQNM0eJ4nvgAglUrBNE1IkoSZmRl89NFH +ME0TrVYL9Xodo9EIh4eHMM1xkBa2ug7D4jDsCcViMSwtLeGTTz5Bq9VCp9PBo0ePUKlUsL6+fqKC +fHl5GZFIBPV6Xezv/fv3sb29/cL9nZycRKFQQLFYRDwehyy9fEV0qVTC5cuXIcsyut0u9vb20Gq1 +sLm5iVwuh0KhgFKphE6n84PD0LDiPZPJ4MqVKxiNRqhUKmg0Guj1enjw4IFoSR0es3DfJiYmkM/n +USqVnjumr4skj7cvlUrj8uXLontAo9HAYDDAo0ePsLe399z2eZ6HYrGIXC6HVColrs+trS14nod0 +Oo3p6Wmsrq5iemoaiUQCw+EQi4uL+NnPfobt7e0T1/3u3i5KpZKYS3/p0iURkISzqr9rWwqFgjhW +2WxWXHO/+tWvkM1mxXW/s7ODer0OwzCetTof/7zv+8hms4hGo2/kOKvKeBFD+Jq2LAuVSgWdTkeE +8ltbWwBw4vUYjUaRTCbFPy9cuIBWq4VqtYp+v4+9vT00Gg3cv38fnufBtm24rgvTNHH58mVcunQJ +KysrKBQLr32f3hbh+IdwMUG1WsVoNEKz2cQXX3wB0zRhGAY0TROLG8JwvNfrvdZtkSQJmqYhHotj +bm4OH374IYIggGVZ4n62u7sr5toriiI6hQRBgFgsBkVRfvyGEJ0ySZKQTCYxMzMjRkzEYjEA48Vx +vV4PrVYLrVYLkUhE/D5BRERERERERPS2YfBNRO+Mer2Ofr+Pzc1NMV/6u8TjcczMzGB2dlbMj81m +s5idncX58+eh6zo0TYNjO1hdXYUsy/jTn/4k5s3u7OygfFDGRDCBSCSCXC6Hjz/+GBMTE7h27Rpu +3LiB7e1t7O3todPpiOcNK6rD8DkMvrPZLGLRGNbW1hCPxzE9PY3r16/j1q1bqFar6PV6sG1bBMql +UknMGI9EIpAlGVNTU/j5z38OwzBw7do1bG5u4ujoCJ7nIRaLYXZ2FouLi0ilUqJa+pshbTwWx8rK +ClzXxfXr10V79LDi++LFiwiCAOfOncPKygo+/PBD3Lp1Czdu3MDW1hb29/fR7XZFBXY4QzydTovg +tVQqIZFIQJJfPrQslUq4evUqJEnC9evXcXh4CMdx4Ps+EokE1tbW4Ps+ut0uXNf9QW3Gw/nDmUwG +V69eRaFQwPXr13Hjxg1sbm6iXC6j0+k8t29h5Xs4BzWRSECRX38gFl7XmUxabN+NGzdw48YNPH36 +VLSm/eb2hduYSCSQTCaxubmJnZ0dHB0dwfd9ZDIZzMzMjNvPZzJQlfFc7uXlZQDjMQJPnjwRM7/L +5TIMw4BhGEilUrhy5QpyuRxu3rz5UtuSTCbFvPJsNot4PC7aos/OzuKrr77CrVu3sLe3h263C8uy +xL7n83kxYiBs6f7aj7Miw5AN5PN5fPTRRygUCuIaf/z4Mfb399Hr9USrfdM0MTk5ienp8aKBSCSC +VCqFy5cvIx6Pi+OytbUlZkWHrd6TySSWl5fxwQcf4PLly1hdXUWpVBIdBt41qqqiWCxicXERQRCg +XC6jXq+jWq1iY2MDkiQhEokgkUhgeXkZq6ur8H1fHPPXKexwEYuP742ffPIJTNPEnTt3cP/+fXQ6 +HdGWP5xtbxgGkskkVFVl8E3vjHBMRywWw97eHjKZjBhtMhwO0ev1xMIfYPx7DBERERERERHR24jB +NxGdSZIkAfI4qL569aqYH/0qwrbWExMTsG0btm2LQOZ4GGxGTJRKJSiKgkajAcdxEAQBlpaWkEgk +YJomVFWFqqooFAqIRCIYjUaQJAmFQgGNRgOtVksEgGFL8nQ6jeXlZSwsLHxdAa2MW++G87Y9zxPz +vtvttgi+FUXBxMQEpqamsLi4OJ7RLUtiXm3Yftc0TViWJdpfz83NYWZmBsPhEP1+H77vIxaLIRqN +YnZ2FvF4HLqhY2JiAsC4mlVVVRwcHIjK5tXVVczPz4vwMWy3DUC0Q221WmIbwnA0lUphaWlJbG9Y +Tfaywn0Lj2MkEhH7Fo1Gsby8jMXFRRGcybKM8+fPI5FIQNf15wLScKZ6NBoViwnm5+eRSCRgGAYK ++YI4fuH3N5tNNJtNsW9h9Xw6ncbi4iIWFxe/rvh+hVD/Va99TdOQz+VhGAYsy0IQBMhkMuJaC7sB +hMFquH1LS0tIpVIIggCKoqBQKGAwGIjgOZVKiXmvhmGgUBhXHrdaLViWhVarhXPnziGdTot565qm +IZ/PQ9d12LaNIAiQTqfFsQq3RdM0cawWFhawvLyMyclJpFIpaKqGRCKB6elp0RZdURTU63XRGjy8 +7vP5PKampjA/P4/Jick3Mgc7fP0dP86u6yIIAqRSKbTbbbEAIgy+w2B0ZWUFmUxGHBdFUcTrI5VK +idEDYev5sKPEuXPnsLCwgFwuJyrcgyDA/Pw8Pv30U0xMTODSpUuiVf03r+fJyUl88sknSCQSuHr1 +qugm8c2Aqlgs4urVq4hEIlhZWcHS0hImJiaQSo8Xw1y6dAnVahXz8/OYn58X97fj17Npmjh//jx+ +9atfoVAo4Ny5c6IC//h2maaJpaUl/OIXv0AkEsG5c+cwMzODYrGIdDqNmZkZXL58GaqqihnaQRDA +MAwkEgkxSz2c416r1cSCj3g8LqrCL126hKOjI0xPT2NhYUFcm8e3WdM0TE5O4tKlS2KOeDQaxczM +DCKRCHRNx8LCgrimY7EY2u02XNeFJEnifhsurrEsC4VCAYVCAbOzs0gmk2/k9U70Uwl/j4nH40in +0+J+ZVkWbNtGt9tFtVqFIiuIRqPivYKIiIiIiIiI6G3C4JuIzqQwlJ6cmMTvfvc7LC4uvvJjhOFG +PB4X7ZOjkSimpqdPVEBrmoZsNivClJWVlfFzP6vwDFt+KooyDo51HRcvXhRtt0ejEUajkXi8MPAy +TRPpdFpUCoftysMq1rCicGVlBaPRCJZlwfM8EcrF43HE43GkUinkcjlIkoREIgFVURGNjucvf/DB +B3BdF77vi4rbcOZwGFLqmg5NH++jCH0LBUSj4zbWq6ur6PV6okI8l8shn88jmUyKqt8LFy6gWCyi +3W7DsiwMh0Oxv2EwaJomUqkUstnsif19WYlEArIsIxEft6j/+OOPxb6FAXQmk8H58+fx6aefQpIk +zM7Oisq1bwakU1NT0DQNnU4HsiRDkset7jOZDBRFQTQWhaqpWFtbQ6FQEOHvd+1bJpNBMpFEJBp5 +o/NPj2/f+fPnkc/n0el0MBwOMRqNxOKAcPt0XUc6nUY2m4VhjCuZV1ZWRKttTdMwNTV14hipiop0 +Oi3aTi8sLGA4HGJ6ehrT09Mi+FAURbw2wm358MMPX7gt4fUSXvepZApmxBxXWBuGuO5N08Ti4iKG +wyEsyxLhoyRJiEajJ657XXtzlYcvOs4ffPABbNsWCw5Eu+xnr8dwbrmiKEgmk9B1Hbo+XkzSarXE +IhtJksRc+Gwmi3QmLV6fkiSJY3v58mUkEgm0223Mz8+jWCyOj7dy8npeWlrG//yf/xN/8zd/g6nJ +KUxOTYrq8+NmZ2fxu9/9Dh999BEy6QzSmTTi8TgikQiCIMBvfvMbzM3NIZ1OY25uDvF4fBwiH+8K +EY/j008/FeMgpqamUCqWYEbMExXQkUgEH3zwAbLZLDRNE+c9Hh/Pqp8oTeCTTz7BwsICBoOBmON9 +fFFAJpNBv9/H2toaBoMBZmdnMTs7i0gkIha5/OpXv8LMzAySyeS3brOmaVhaWkYikYBjO+P/p48X +J5imiSAIMDU1hUQiIbpLWJYlFrmE5zcIAvG6iUajYhxGJpN5Y9ch0U9BkiSx8CSVSiGfz8P3fQyH +Q7iui06nI0bJ5PK5095cIiIiIiIiIqIXkoLwE2miM8r3fezs7GB3dxf7+/vY399HtVrF5OQkZmZm +MD09LWYWEhEREZ2GWq2G3d1d7O3tYX9/H+VyGZFIBFNTU5ienhahfjQaPe1NpffY+vo6/u///b/4 +05/+hJ2dHezt7SGVSuGXv/wl/uEf/gFra2u4cOEC8vn8aW8qEREREdGpCUcg7u/vY29vD+VyGZIk +nfj73czMDFKp1GlvKhHRe+f1D8YkIiIiIqITuNaUzoKw40IsFoOmaZBlGb7vw7ZtDAYD2LYtOiEQ +EREREREREb1tGHwTERERERGRGJERi8Wg67poge66LobDIRzH4SIOIiIiIiIiInprMfgmIiIiIiIi +KIoCTdNgGAZUVT0RfNu2DcdxWPFNRERERERERG8tBt9EREREREQESZKgKAp0XYeiKM8F367rwvdZ +8U1EREREREREbycG30RERERERARJkqCqKjRNOxF8+74Px3HgeR6CgBXfRERERERERPR2YvBNRERE +REREAMbhNxERERERERHRWcTgm4iIiIiIiARJkk4E4EEQiC8iIiIiIiIiorcVg28iIiIiIiI64XjI +HYbgrAYnIiIiIiIiorcZg28iIiIiIiICgG+t7GboTURERERERERvO/W0N4CIiIiIiE5XGHQGQfBc +m2t6fwRBAN/34fv+c+G3LMvPrg2unSYiIiIiIiKitxODbyIiIiKi95zv+XBcB67rQtM0aJoGWWbA ++b4JggCu68K2bbiuiyAIIMsyVFWFrutQVRWyzEURRERERERERPR24qdZRERERETvOc/3YFkWBoMB +bNtG4Ac//kHpzAn8cfBtWRY8z4Pv+5AkCaqqwjAMaJrGbgBERERERERE9NZixTcRERER0XvEcz04 +rgPLstBsNtFqtdBoNFCv1zEYDHD+/HlcuHABqVTqtDeVfmKuN672HgwGcBwHwHi2t6ZpME2TnQCI +iIiIiIiI6K3G4JuIiIiI6D3iei4G/QFa7RYePXqEx48fY2dnB3t7e+j1evinf/onTE9PM/h+D/m+ +j9FoJCr/Pc+DLMvQNA2RSAS6rrPim4iIiIiIiIjeWgy+iYiIiIjeUeHMZt/3Yds2LMtCr9dDo9FA +tVrFrVu3cPfuXWxubmJvbw+DwQCLi4vo9/unvel0ChzHwWAwQKfTgWVZCIIAkiTBMAzE43EYhgFF +Vk57M4mIiIiIiIiIXojBNxERERHROyoMujudDnZ3d7G7u4tKpYJarYZarYaDgwMcHR2h0Wig3+8j +CAL4vn/am02nIAgC2LaNdruNWq2Gfr8P13WhqioSiQRKpRKSySRUjX+FJCIiIiIiIqK3Ez+1ICIi +IiJ6R9m2jXarjfJBGTdv3sStW7ews7Mjwm7XdeE4DlzXxWg0QiwWQxAECILgtDedfkLh+bYsC51O +B/V6HcPhELZti+C7WCwikUhA07TT3lwiIiIiIiIiohdi8E1ERERE9A6wbVvMZ240GqKdebVaxcHB +AR4/foyNjQ1Uq1V0Oh20220RcgdBAM/zTnsX6BQEQYBer4d+v49KpYJGo4HhcAjf9xGJRBCLxZBM +JpHNZhGPx6EobHVORERERERERG8nBt9ERERERO8Ay7LQaDRQqVTw4MEDPHr0COVyGbVaDY1GA61W +C51OB8PhEJZlwfM80dacVd7vt3a7jYPyAXZ3d9FsNjEajWAYBnRdRzweRzqdRjqdhq7rUFX+FZKI +iIiIiIiI3k781IKIiIiI6IzxfR+u68J1Xdi2DcuyUK/XcXAwDi+vX7+O27dv4/DwEO12G91uF67r +wvM8UdnNoPv9Fi52cF0XtVoNTzeeYnt7G61WC8PhENFoFIlEAtlsFqlUColEAgAgSdJpbzoRERER +ERER0Qsx+CYiIiIiOkOCIEC/30ez2US9XsfOzg52dnZQrVbRbDZRq9Wwv7+Pw8NDdLtdjEYjOI4D +3/dPVHjT+833fNiOjeFgiM3NTXz11Ve4e/cu6vU6JElCKpXCwsIClpeXkc/nATD0JiIiIiIiIqK3 +G4NvIiIiIqIzIqzSHQwGqBxWsLm1iWvXruHmzZuoVCoYDAYYDocYjUaiEpxV3vQinu/Bsix0e11s +bm7i1q1bePToEbrdLmRZRjqdxsrKCs6dO4dcLsfQm4iIiIiIiIjeegy+iYiIiIjeQr7vI/ADDIYD +9Ho9dDodNJtNNJtNHB0d4fDwEOVyGY8ePcLm5iY6nQ5s24Zt23AcB67rcnY3nRAEAXq9Hvr9PqrV +KsrlMnZ3d3H79m3s7+9jMBhAURTkcjlMTU3h3LlzWF5eRjabPe1NJyIiIiIiIiL6Xgy+iYiIiIje +MkEQiErtVquF3d1d7O7u4vHjx3jy5Anq9Tp6vR56vR6azSba7TZGoxE8zxOzvBl603FBEMD3fLRa +LZTLZTx+/Bh37tzB+vo6dnd30Wg04Ps+YrEYkskk5ufnceHCBSwvLyOdTp/25hMRERERERERfS8G +30REREREpyQIAvi+j9FohG63e6I1+Wg0gmVZKJfLePLkCZ48eYI7d+7gwYMH6HQ68DwPjuPAcRzY +tv1aQu4gCGDbNvr9PjqdzmkfHnpF4TUQLnoI/AB+MJ7t7nkebNvG5uYmNjc3ce/ePVy/fh3r6+vi +GorFYshms5idncXS0hKWl5cxMz0DWZFPe9eIiIiIiIiIiL4Xg28iIiIiolPg+z5c18VgMMDOzg72 +9/dFAN7tdtHpdER783q9jmazicPDQ/R6Pdi2LSq7w/ndr2ubqtUq7ty5g1arddqHiF5BuIgirPp3 +XVe0vh8MBuh2u+j1eqhWq6hWq6hUKjg8PIRlWdB1HZFIBJOTk7h48SI++OADXLp0CalUCpLM2d5E +REREREREdDYw+CYiIiIiOgVhO/PRaCRmd9dqNVSrVdRqNdTrddTrddHCPKzwHg6HcF1XPMbxf/5Y +nuehXq9jfX0d1Wr1tA8RvYLj14ht27AsC4PBAP1+H+12WyyeGAwGoptAOA8+EokglUphfn4eP/vZ +z/C73/0OmUwG6XQassxqbyIiIiIiIiI6Gxh8ExERERGdAt/34TgO+v0+dnd3sbm5iWq1ik6ng3a7 +jV6vh06nc6KN+Zue3e26Lo6OjnD37l3E4/HTPkT0Cnz/65bmYQt8y7JgWRb6/b6o+A6vH03TkEwm +EYvFMDMzg5mZGaytreHixYuYmpqCYRjQVO20d4uIiIiIiIiI6KUx+CYiIiIiOgVh8B0EAcrlMh48 +eIBGoyFCyzC4PN7K/E2G3gDgOA4ajQZs24amMfQ8S8Jrw/f9E23Pw+ssrAQ3TROGYSCdTmNmZgZz +c3NYWVnB6uoq5ufmUZooIRKJQFVUzvYmIiIiIiIiojOFwTcRERER0SkIw0nHcdDpdEQr6jCw/Ob8 +7jcZeB9/jm63i9FoBEVRTvsQ0Q8kSRIk6evZ3LIsQ9M0mKaJWCyGRCKByclJrK2t4dKlS1hdXcXa +2hry+fxzP0tEREREREREdFYw+CYiIiIiOgWyLENVVei6jlKphIWFBSQSCQyHQ1iWhdFohH6/L8Lv +sJL3TQbgiqIgl8shl8vBMIzTPkT0ksKgWpZlKIoiri1VVaEoCkzTRDQaRSwWQzKZRCqVQj6fx8TE +BCYnJ1EqlRCNRhl4ExEREREREdGZxuCbiIiIiOgUSJIkQslSqQTLspBKpdBsNtFqtdBut+F5HobD +ofj+MAR/U+G3oijIZDI4d+4cksnkaR8iegWKokBRFGiaBl3XYRgGDMOAaZoi6C4UCkilUshkMohE +ItB1HZqmiS8G30RERERERER0ljH4JiIiIiI6BbIsQ9d1xGIxzM3NIZvNotvtotPpoNPpiPC71+th +OByi3++j2+2i2+3Ctm0xvzn8eh1UVUWhUMDly5cxMTFx2oeIXkFY7X08/A4D8Hg8jnQ6jUwmg3g8 +jlgsxhnuRERERERERPTOYfBNRERERHQKwnbUsVgMxWIRsWgMfuDDtm2MRiMMBgP0+31Uq1WUy2Xs +7+9jc3MTOzs76Pf7Yj644zivrQW6qqoolUr46KOPsLi4eNqHiF5BOJs7bHUeXl/j+d46DEOHaZrQ +NI3z24mIiIiIiIjoncTgm4iIiIjoFIQhpWEYmJycxOzsLKLRKIBxK3PHceC6Lvb39/HkyRM8evQI +pmnC9300m03Yti1mgY9GI9i2LeaAh4/xqmRZRi6Xw/nz53HhwoXTPkREREREREREREQvjcE3ERER +EdFbRpIkKLICqEA6ncbCwgKi0Sjy+TxWV1dRq9XQaDRQr9exv7+Pw8ND9Pt9uK4Ly7Lg+z5c131j +s8CJiIiIiIiIiIjeNgy+iYiIiIjeQrIiQ1ZkpNNpJBIJzMzM4NKlS7AsG0dHFWxvb2NzcxPXr1+H +bdsIggCj0QgA4DgOPM8D8MMqv4mIiIiIiIiIiM4aBt9ERERERG8hSZIAAJqmQdM0mKaJeDwOAIhG +I+K/VVVFKpVCpVJBp9NBt9tFs9lEu90WLdDDKvDXNQuciIiIiIiIiIjobcPgm4iIiIjojJAkCUEQ +IBKJoFQsIRaLIZ/P4+LFi6jVajg6OkKlUsHDhw+xsbGBer2Ofr8P3/fheR5c1wXAKnAiIiIiIiIi +Inr3MPgmIiIiIjpDJEmCaZowTRPZXBZzc3MAgFarhYODA+zt7SEej0NRFESjUTQaDbTbbdi2Lb5c +1xUhOMAgnIiIiIiIiIiIzj4G30RERERE7wDDMJDNZiHLMgCgVCrh8PAQ1WoV1WoV5XIZBwcH6HQ6 +GAwG6PV6AMahd9gGnYiIiIiIiIiI6Kxi8E1ERERE9A4Ig+9UKoXJyUl8+OGHaDabqFQq2N/fx82b +N3H79m3s7e0hCAJYlgXP80Tg7fv+ae8CERERERERERHRD8bgm4iIiIjoHSDLMmRZhqZpME0TABCJ +RBCNRpFIJCBJEmKxGPb393F0dIRarYZOp4Nut4vBYIDhcAhJkiBJ0mnvChERERERERER0Stj8E1E +RERE9I4yDAPpdBqGYSASiWB+fh6NRgO1Wg3VahVPnz7F5uYmDg4OUK1WYds2w28iIiIiIiIiIjqT +GHwTEREREb2jNE2DpmmIx+MoFAoAgOFwiHa7jXq9jhs3biCRSCAajUJVVXS7XSSTSSiKctqbTkRE +RERERERE9EoYfBMRERERvUcURUE0GgUArK2tIZFIYGlpCYeHh+j3+/jkk0+QSqVOezOJiIiIiIiI +iIheCYNvIiIiIqL3iKZpUFUVkUgEyWQSKysr6PV66PV6GI1GyOfzDL6JiIiIiIiIiOjMYfBNRERE +RPQeCWd4y7IMTdMQjUYRi8WQTqfhui4Mw4Cmaae9mURERERERERERK+EwTcRERER0XtOURQReMuy +DEmSTnuTiIiIiIiIiIiIXgmDbyIiIiKi95wsy5Bl+bQ3g4iIiIiIiIiI6Afjp1tERERERERERERE +RERERHSmMfgmIiIiIiIiIiIiIiIiIqIzjcE3ERERERERERERERERERGdaQy+iYiIiIiIiIiIiIiI +iIjoTGPwTUREREREREREREREREREZxqDbyIiIiIiIiIiIiIiIiIiOtMYfBMRERERERERERERERER +0ZmmnvYGEBERERER0dspCILxlx9gOBrCtmy4nnvam0VEREREdGq6nS46nQ76/T4cx4Hv+5AkCbZt +YzAYoNVqIRKJwLKs095UoneKqqhQNRWapkFVVaiqCkmSTnuz6C3D4JuIiIiIiIheKAgCeJ4H13XR +bnfQajUxGo1Oe7OIiIiIiE7NcDhEvV5Hp9PBcDiE7/sAgNFohE6nA0VREAQBIpHIaW8q0TslEokg +Go2KL1VlxEnP41VBREREREREzwmCAK7rYWRZGA5HqNfrqFQqGAz6p71pRERERESnxrZtdLtd9Ho9 +WJYFz/MgSZIIvn3fh23b0HX9tDeV6J0Si8WQSqaQTCURBAF0XRdV36z8phCDbyIiIiIiIjrB9314 +nodef4CjWhPVehON6iHqtQpGw8Fpbx4RERER0alxXRe2bcOyLNi2faLiOwgC2LaNXq/HalSi1ywW +i6Hb7SIzyMB1XUiShEgkAtM0oWnaaW8evSV45yUiIiIiIqITPM8TwffhUR2b22V0mhV0mkewRgy+ +iYiIiOj95vs+giAQXwBEEC5JEmRZPu1NJHrnxGIxdDodDAbjv5OGYbeqqgy+SWDwTURERERERCf4 +vg/HcTAYWmi0+ziodjDsWxgMANfhXyPPEkUOoEgBVEWCpsrQNUV8GMuWgERERESvzvd9uK4L13Xh +eR5c1wUwDt8URYGqqlBVleE30WswHsH19ettMBhAkiTEYjHE43Houo5IJHLam0lvEX5iQURERERE +RCcEQQDP8zAa2Wh3R6g2h3AdwHUj8APOKjxLdCmApAKKLiEW1ZGMaSc+jGX4TURERPRqHMfBYDDA +aDTCcDgUrc4Nw0AkEkE0GkUkEmGrc6LXwPM8DIdDDIdD2LYN27YRBAE6nQ7a7Tai0SjS6fRpbya9 +RXjnJSIiIiIiohOCIIDv+bAdF72BjXZvhCBQ4CMKgJUrZ0kQuJADF1BkmFETyaR5Yg5eGH4TERER +0cuxLAvtdhvtdhu+78O2bQCAaZpIJBJIpVJIp9PQdS4YJfqxXNcVr7d2u43hcCgWn/T7fViWJbou +EAEMvomIiIiIiOg7BJAQQIYLHU5gwIdy2ptEr8B1Bs8+jPWQjptQVBWxWAzpdBrRaFRUfxMRERHR +y+n3+1BVFb7vw7Is9Pt9SJIEwzCQSCSQy+VQLBYRjUZPe1OJzrzRaCQWkYSBt+M4p71Z9Bbj326J +iIiIiIjoe0hwocOSEvAC7bQ3hl6BbTuQHR8yPNhuAEVWEI1Gkc/lkUqnoOs6q5GIiIiIXkGn00EQ +BLBtG71eD4oyXhgaiUSQSqWQz+cxNTWFRCJx2ptKdOYNh0PIsgzbttHv96EoCoNv+k4MvomIiIiI +iOg7SAAkBFAQQIH/kn+NlABAkiBLQMRUETUUKIoE1wvgegEs24PlePC84LR38J0WBDI814fjevB8 +D0EQQJZlqJoKXddhGAYMwzjtzSQiIiI6MyzLgqqqUBQFsjweAyRJEmRZhqIoYmGhaZqnvalEZ57v ++9A0DaqqckwTvRQG30RERERERPT6SYAsA4osIZ3QUcpGYGgyhpaPkeWi2bXQ7AbwPO+0t/SdFgTB ++Mv34T/7Cv8fERERERER0buEwTcRERERERG9dhIkyJIERZGRieuYn4gjaqro9h10Bg48P0Bv6MKy +GXy/WQECP4Af+PD9AL7vifCbiIiIiIiI6F3C4JuIiIiIiIheO+lZxbcqy0glDMyUYkjGNDTaNsy2 +hXbPhqqwTR0RERERERERvR4MvomIiIiIiOiNkCQJqiIhHdcxV4wjk9IRNYdQVQmHjQFURT7tTSQi +IiIiIiKidwQ/ZSAiIiIiIqLXTwKkZ/9UVRmGLiNiqDA1BYaqQJUlSCz4JiIiIiIiIqLXhME3ERER +ERERERERERERERGdaWx1TkRERERERG8VSRq3SZclQJYlSNK4OjwIxn8eBAF8P4AfjP89/P/fJEsS +JHn8T1kCJFn6+nuD8c/7fjB+vODF2yFLEhRFghz+bBAgAIAACADI0rNvHD8ogmeP6fnfvl1ERERE +RERE9Pox+CYiIiIiIqK3iiJLMHQVpqHA1BVEDBWqIomAejByMBh5GNkuXDeA4/ovfBxNk5/9vALT +UGHqCsbZdQDb8TEYuegPXbieB8cdh+khWZIgy4BpqEjFdWQSOmzHh+X4cD1fhO+GrsLQZEiSBMf1 +4Xg++kMH/aED2/EZfhMRERERERH9RBh8ExERERER0VtFVSRETQWJmI5MXEc6acDQFHi+D98PUGtZ +qLdHAIBh4MJxX/w4hqYgHh0H1+m4gXTCgB/48H2gP3JRb40QBAGGFuB5Lo7H55I8rjaPmiomshHM +T8bRHTjoD12MbG9c1e35iMd0JCIaZBkYWh6GIxdHLQm248N2AkgSK7+JiIiIiIiIfgoMvomIiIiI +iOitEDVVxCLjoLqQMVHMRpCI6EjENOiaBN8D/CBAMWOj2bXR6IxQbY5Qa41gOR4cx4ciS4g8e5x8 +OoJixkQmqSMZ0xGPas8qtYGh5aKVjaDZtcRjdAYOHMeD7fjjFueyhIihopg1sTyTxGDkYmCNnyds +Zx41FcRMDZIEWI6PkeMiVR0gEdVQb4/QHbjo9u3TPrRERERERERE7zwG30RERERERHTqJAlIxjRM +5qOYLsQwW4pjbjIGQ1WgaeMQOuxEPrI8jCwPh/UBHu12EADo9m10ejY0TUEhY2IiH8VcMY65iRhy +KROaKsHQ5PFccACu62Nkjx/n4U4Hj3fakOt9dPoSbMeGJAGKIiNiKChmI1iZTsLxfDjuuGI8eDbk +W1UkqKoMSQI8P4DnBiimI8inBtg96mH7sMfgm4iIiIiIiOgnwOCbiIiIiOgNkyTptDeB6K0lYfwa +kWUgnTAwW4xjZTaB+YkEFqbicBwftucjeNaHXJIBVZagyhJyKR2uH2AwcnEoSbBsH6Yuo5iJ4NxM +EguTCSxMxpCIanC9ALYbIHw1qso4TFdkGYAE23Hhej48b4BuH5Cl8ZxvU1OQTuiYLkZFy3LvWZtz +zx+3MfeDALIkQVUkaKqMaERFIqJBUSR0+g7KR30EwXi2OLueExEREREREb0ZDL6JiIiIiIjo1Mjy +uGLa0BXk0ibmJmKYzMdgaAq6fQeH9SEqjSEGlgtFGQfepWwEE7kodE3FVC4K3/OhqTIc14ehyZgq +RLE8nRwH3j5w1LRQqQ9RaQ7HYbciIRHVxo+TjSAd17A4lYQfACPbw1FzBEgSvrlmJQgCeAEwGLpo +dEdodmzYz1qjG/q40ryQNqGpMnJpA71hBLsVAxFTg+N6cN1xWE5ERERERERErx+DbyIiIiIiIjo1 +sizB0BXETBWFtIn5yQSmchFYro/uwMXWQQ8PtltodW3omgxdk3FhIYNYREM6bmAyH0UqrsFyA7Q6 +NjRNxlQuinOzCVi2B8sOcNQY4cF2Gw+3W9DU8WOUchFAAoppE+mEDl2TgQCo1IeQ5edDbwDwA8Dz +fPRHDg6qQ2wf9sZzv0cuElENjpNEzNSgqzIKaROW7SOT1BEzFQwsiLngRERERERERPT6MfgmIiIi +IiKiU6MoEmKmikzSQCquIR5RIEkS2l0bh40h9qp9lGsDtHs2VEWGpkpIxXTkUgYkCTB1BdmkiYls +BPViDJoqIZcyETFUtHsOKo0Rdipd7FX7OKgPxjO5FRmu5yOT0FFImzC1cfCeTuhIRDVETRXyCyq+ +LdtDb+TiqDHC7lEfT/c7GFkeLNtDIqbB0BVETBW5tIFCyoRpyIhHNCTjOgKMq8OJiIiIiIiI6M1g +8E1ERERERESnRlVkxKMaChkTiagOTVMwcjxUmkM82etg/6iPRnuE7sCFLEtQZOCwPkQ21Yemyihm +DKTi+rNq8RhkSUIqrkGSxvO1d4962DroodIYoN13xOxuSZaQrQ6RSQxQzJgoZk1ETRXxqIZEVIPn ++XC9k9XZQ8tFo23hoDbAbqWPrXIXnheM54xbHiKGCk2VAQDJqAZNkRGLqMgkdDiuj97QBcDwm4iI +iIiIiOhNYPBNREREREREp0ZVZMRMFbmkgXhEhaZIGFo+Gh0Lu5UejpojdPo2BqNxYCxJQL1jodIY +IRnTkY7rMHUFmZQOx49BApCIa5AkoDd0cFgb4qDWR7NjYTB0vn5eVUKtNcJhfYCoqaIIwDRkEX4P +LQ/+yD2xrSPbR2dgo96xUW0OcVgfij9zvQBRQ4GhyUgndEy7USiqhIihIBnX0ek7UGTpZQ4JERER +EREREf0ADL6JiIiIiIjo1MgyoKkyTEOFpsmQJAl+4MNyPAxHLhzXw4mx2M/mbFu2h5HliapsU1OR +immQnv17AMDxfIxsFyPbh/eN6m3fB2zXx8ByYTke/CCALMnjbdEVuK4PW375/fD9AI7nY2h7cBwf +gQ8o6rituq4qUBUJzL2JiIiIiIiI3hwG30RERERERHRqZEmCrsmIGgp0VYYsSfD8ALbjY2CFIfLJ +0NoVgbYLxxtXghu6jBQMAICpjxNr1w0wsn1Ytvtc23I/CGA7HgbD8XP4PiBLgK7KiBgKLNuDJL18 +Uh0EPhw3wGjkPQvrA0iQoCoSDF2Gosh4hYcjIiIiIiIiolfE4JuIiIiIiIhOVRCMg2icyKalE/94 +oeNJcvgY+MbDvMp2hF8/6AGkrzfp2Hb5fgDPCxAEwQ98XCIiIiIiIiJ6GQy+iYiIiIiI6NSEobfr +BvD88X9L0rgtuKJIkCXpuUppSZKhyDJkGVDkZ9Xd/rg9OgBoqgQDCiQJkBUJ8guqrSUAsiyPW5DL +z54jAHzfh+v68PxxWP2yJEmCIktQVRmKLEEC4AWB2C7X8xl8ExEREREREb1BrzCxjIiIiIiIiOj1 +CkNv2/Hheh6CIIAiAZomw9RVqKp0srL7WSCu6zIMTYXybHC27fjoDRz0By4cZ5wwK4oEQ5NhaAoU +5WTyLcsSNFWCaShQtXHQ7gc+HC+A7frwXjGolp9tc8QczyqXFQlBADhOgJHtwXEDUZH+k5PGQf/4 +MEqv1MKdiIiIiIiI6Kxg8E1ERERERESnxvcDjGwXnYGDke3DCwKoqoJUTEMxayIR1aEq47+6StK4 +sjpiKMglTWRTOiKGiiAA+kMH1dYI1fYQ/ZELCUDUUJFLmsgkdJi6Ip5TkgBNlZGI6ShmI0hGNaiK +As8HLNtHb+jAcrxXCqo1VUYqpmMyF0U6rkNXZXhegKHlotNzMLJceL5/Ksf4eMzN0JuIiIiIiIje +VQy+iYiIiIiI6NR4/rgiujtwMLRceH4AXZWQjOsoZSKIRzVoavhX13FL8qihIpvUkUsaiBgyAgD9 +oYdq00K1aaE/dAEAUVNFNqUjmzRgGuNJX2HVs6pISEU1TGRNpGIaNFWC541D+P7QheX48P2XC77D +ID0V1zBViCKTMKCpMjzfx9Dy0OnbGNkegpd8vNcrnJUuQ5IkSPK44psBOBEREREREb1rOOObiIiI +iIiITo3rjSusa60hWr0YegMHmYSOdEzH3EQMjhdAAtDu25AlCYoiYXE6jolcFDFThe0GOGoOUa4N +sFvpQ1Uk5NMGhlYUpi6jlI08mx8eQJHH87xlCShmo5gpxZBNGZBlGZ2Bg3pnhE7fwdDyIEuA/I1w +2DRkpGMGSjkHnX4CrueP55HLQC4VwcJkHKVsBIYuY2h5aHZstHsOemE1+ykUfEsIxvPSn81DVxUV +siwz+CYiIiIiIqJ3DoNvIiIiIiIiOjWu66Pbd+G4ARptG92+g2RMQyqhIxHVYGgq0nEdg5ELVZWg +KRKmCjFM5iLQNRmtnoN2z8buUR87lS4MTcF0IYr+0IWuK5jIRhE1NcQiGgrpCFRFgqrKSMc1TBVi +SCcMdHoO6u0RKo0ROj0blu1CVeRjleZjMVOFnJGgKICpKcinDCiqBFWWkIzpmMhFUcyYGI5cdPsO +au0RWj0LnYENzxuH7z81SQJkOYAqA5qqQFVVKIoiwm8G4ERERERERPSuYPBNREREREREp8b1Angj +FyPbRb09wlFzhERMQzquI5nQoT5rIe56AVRFgqbJiBgKYqaK3tBFozPC7lEfe0d9VOpDmIaCSnOI +SnOIVMxAMq4hFlURNVQUMgY0RYGmSTA0BVFDha7IGIxcHNSGOKgO0OrZsB0fsgQEwcng29QV6JoC +Q5OfPZ4JTZGgazI0TUFEV2DqCto9G0etIQ5qQzQ7FoYj76c9qEEABB6CwIUCF5riw9QAQ1dgGDo0 +TYMsK1AU5cc/FxEREREREdFbgsE3ERERERERnbIAQSCh2hphfauF3sjBdCGKmUIMgISIoY4rlyUJ +kgzYtoduz0G1PcLGfheb5S4qjSGGlgfPD7B90IMiS+PHKEaRiBqQJCAdNyBLgCRJ8IMArb6Fo6aP +p/tdbJQ72Kv00e7Z37qVrg94rg/XAxRVQtRUTlSG94YuGl0bm/sdbB30sHPYR7Nr/fRH03cQOB0E +dheq1kfS9JCO60jEDESjURiGAU1TWfFNRERERERE7xQG30RERERERHSqggAAAtRbI9iOh07Phm37 +0FUFqbiOdFyDoSvwA8D3AnR7Do6aQ+xU+ni828GTvTYs24Pj+HA9GbuVHlo9G92BgwDAdGEceqfj +OvwggO8DvaGDdtdBtTXE0/0OHu92UG0OYdvfXp3t+z5sz4fvB1BlGYohQXkWfFu2i95gXLW+We7h +wXYLlfoQA8v96Q+ob8O32vAGh9CSEhKmhEwiimTMQDQShWmYYta3LMk//vmIiIiIiIiI3gIMvomI +iIiIiOi1CwLA9wM4ro+D6gC3njSQjGpodG20OhaqTQuW45/4fst24QeALA+hqhJs10MiqiER1aCr +z4LvwEejM36Mo9YQR40h+gMHvg/4QYAAPnpDB7YbwNT7CAKg3rKQjI0fJ3yMwchDq2Oj0bVQrvaf +tSQfP/+3sRwf3b6DTt9Bu2+j07ehyBIUWYZle2j3bHT6DnYOe6i3LPSHDtyfcK534A0RuCPIbhcG +ulB1C5l4BPlMHPlsGqlUArF4DLqhQw0rvmVWfBMREREREdG7gcE3ERERERERvXZBEMD1gCDwsHXY +RWdgQ1MV2I4Hy/bQHTiwvlEN7XoBgsBFsxPAcTwcNUbQVBm6qkCWn42uBmA5Hmzbw9By0R+68Pzx +84mw3fHhew4O6wN0Bw62dAWaKkNTFQQYP4jr+bAcD5bjYzB0MBi58P0A3xVTjywPrZ6NvaM+tso9 +bB/0AAmQJMD3AdtxYTvj4L03cOB6AfzgJwy+nQH8UQOy30FUHyERBQppE4V8Bvl8HqlUCrFYDKZp +QtM0KLLCVudERERERET0zmDwTURERERERK9dEABeEMDzA1TqQ1Tqw+/9Gc8ff7/t+ugNHADf/zMv +el7XC+B6Aay2hUb79c3YDkPtatPCxn4H9zaaP+Uh/fYdDjwEgQu4PUhOA7rcR8JUUExpKOYSKOaz +yOVySKfTiEajME0TiqJAVtjmnIiIiIiIiN4dDL6JiIiIiIiIzqjAdxDYHfhOB0bQgWk4SEVVTOQT +mMgnUSwWkM/nkclkEIvFYBiGqPYGwIpvIiIiIiIiemcw+CYiIiIiIiI6q3wbvt2C2z9EIuohFfVQ +TEcwUUhhanIC+XxeVHtHIhEYhgFZliErMkNvIiIiIiIieqcw+CYiIiIiIiI6YwJviMC1ILldGOgh +atjIJjQUM1GU8ikUC3kUCgWk02kkEglEIhHoug5VUSHJDLyJiIiIiIjo3cPgm4iIiIiIiOiMCZwB +fKsBzesgqo+QiAQoZqOYLOVQKuRQKBSQy+UQi8VEi/Pjc71Z7U1ERERERETvGgbfRERERERE9B0C +AD4QuJDgQgrGgWkQHP/zd4/vKQhkD46jodcboNbsotkeoNMZYjAYwHEswHd+wv2XAASQnv2r7PUh +OS0Ycg/JiIJiUsdkIYWJYh6lUgmZTAbpdBqmacI0TWiqBkmWGHgTERERERHRO4vBNxEREREREb2Q +BB8KXKiwoAUyXM+D7ysIggDwAwQIxv8evHvht2vLgCKhCgO30EC7tofewEFn4KDZsdBojhCMRj/N +vksSJPGvAWQpgKF2YSYDJEwThUwMxWwcxWIBhUIBmUxGtDfXNA2KojD0JiIiIiIionceg28iIiIi +IiJ6IQnBOPiWbCiBD8UbwfeAwPfhez78wEfg++9i7g1PAnwA9YGETkPG40cyvADw/QCuF8BzfQTe +T7Pj0rMyb1kKIMs+VMlHPBogHQuQS0VRLGRRLOSRy+WQy+WQSqUQiURgmiZkWR63OJfl0z6kRERE +RERERG8Ug28iIiIiIiJ6IQk+FMmBCguGZENSZASqAk3VoGo6fN+H7/vvZMX3dx2V8ddPGCRLEiQJ +kCUJmhJAlYFkTEUmriKTiiGfzyObzSKVSiEejyMSiUDXdaiKykpvIiIiIiIiem8w+CYiIiIiIqIX +kqQAChxokgVZATRJgm6YSCWjSKZS46pv3x+3O6c3RpLG4bUsS9BUBaqqIBHVEY+ZSMSjSKVSSKVS +iEZjiMWiMAxjXOWtyOLniYiIiIiIiN51DL6JiIiIiIjohSQEkOFBhQNVBiRFQjxqYLIQRamUFRXf +DL7frK+DbxmapkFVVUQiEUQiEUSjUfFlGGNFavwAAIAASURBVAZ0XYemaqz0JiIiIiIiovcOg28i +IiIiIiL6XoqiQFEUxGIxZLNZTE9PIwiC8ZfP4PtNC4Ps8Dzoug7DMKBpOgwj/HcNiqIw9CYiIiIi +IqL3EoNvIiIiIiIi+l5h2BqPx5HL5UTwDYAV32/Y8RBblmVIkgRVVaEoClRVhSzLIhAPq8OJiIiI +iIiI3jcMvomIiIiIiOh7ha22VVWFaZqIxWLizxh8/zTCcxCG2+F/y5LMed5ERERERET03mPwTURE +RERERN/rm3OmDcMQf8bg+6dxovJbkiHJ0on/z9CbiIiIiIiI3mcMvomIiIiIiOilhW21NU077U0h +IiIiIiIiIhLk094AIiIiIiIiIiIiIiIiIiKiH4PBNxERERERERERERERERERnWkMvomIiIiIiIiI +iIiIiIiI6Exj8E1ERERERERERERERERERGcag28iIiIiIiIiIiIiIiIiIjrTGHwTERERERERERER +EREREdGZxuCbiIiIiIiIiIiIiIiIiIjONAbfRERERERERERERERERER0pjH4JiIiIiIiIiIiIiIi +IiKiM43BNxERERERERERERERERERnWkMvomIiIiIiIiIiIiIiIiI6Exj8E1ERERERERERERERERE +RGcag28iIiIiIiIiIiIiIiIiIjrTGHwTEREREREREREREREREdGZxuCbiIiIiIiIiIiIiIiIiIjO +NAbfRERERERERERERERERER0pjH4JiIiIiIiIiIiIiIiIiKiM43BNxERERERERERERERERERnWkM +vomIiIiIiIiIiIiIiIiI6Exj8E1ERERERERERERERERERGcag28iIiIiIiIiIiIiIiIiIjrTGHwT +EREREREREREREREREdGZxuCbiIiIiIiIiIiIiIiIiIjONAbfRERERERERERERERERER0pjH4JiIi +IiIiIiIiIiIiIiKiM43BNxERERERERERERERERERnWkMvomIiIiIiIiIiIiIiIiI6Exj8E1ERERE +RERERERERERERGcag28iIiIiIiIiIiIiIiIiIjrTGHwTEREREREREREREREREdGZxuCbiIiIiIiI +iIiIiIiIiIjONAbfRERERERERERERERERER0pqmnvQFERERERERERPS1SqWCcrmMXq+HSCSCSCSC +VCqFbDaLaDR62pv3xgyHQxwcHKBcLkNRFEQiEcRiMWQyGWSzWcjy2ajf8H0ftm3Dtm00Gg1UKhW0 +Wi3Ytg3HcWAYhjinxWIRExMT0DTttDebzrhms4lyuYx6vY6IGYFhGkilUshkMkgmk6/lORzHgeM4 +GA1HOKoe4ejoCP1+H7Ztw/M8RKNRxGIxZLNZlEol5PP50z4sr4XneigflFEul+G6rrgvZzIZZDIZ +GIZx2ptIr8BzPbiei0ajga2tLezt7WFmZgZzc/PIZNLQNI33ZCKiM4zBNxERERERERHRW2R3dxf/ +/d//jXK5jGw2i1wuh5WVFSiK8k4H3/1+H3fu3MHnn38OXdeRzWYxNTWFc+fOIZFInJlwyfM8DPoD +dLodrK+v4+bNm3j69Cl6vR56vR6SySSKxSLm5ubwySefIJPJMGShH+3w8BCff/457t+/L+4b8/Pz +WF1dfW3Bt+u66Ha7qNVquHHjBm7fvo1KpYJerwfLslAoFFAsFrG2toZPPvnknQm+Xc/Fo0eP8Pnn +n2M4HCKTyaBQKGB1dRWmaZ6ZexONOa6D0WiEzc1N/OEPf8Cf//xn/PrXv8b/+B//A6urq4jFYrwn +ExGdYQy+iYiIiIiI6L1jWRb6vT56/d4r/6yu64jFYojH4wAASZJ+0Db4vg/P8+A4DgaDAQaDgagi +03Udqqr+qArXIAjQ6/XQ7/cRBAEikQii0SgUWYGsyK+83UEQoN/vo9/vw/f9H/149O2Ojo5w7949 +bGxsoFQqYWpqCpFIBDPTM6e9aW/UcDjE5uYmvvrqK0QiEUxOTmI4HCKbzcL3/dPevJfmed64+vag +jAcPHuDGjRt4/PgxBoMBhsMhUqkUJicnMRqNRNV3Pp9HJBKBaZpQFOXMVLfT26PZbOLhw4f44osv +MDExgampKUiShImJidf2HMPBEEdHR9ja2sK9e/dw7do1VCoVDAYDuK4rOhgAQDqdRiaTEe8Vmqad +2Wvb8zzs7+/j5s2b6PV6mJiYwMzMDFKpFBYXF7/358P3e8uyMBwOMRqN3shxCYIAlmWh1+thOByK +rhmqqp7ZY/8muK4Ly7JweHiIGzdu4I9//CNisRjW1tYwMzPDhQxERGccg28iIiIiIiJ6bwRBgCAI +cHBwgM8//xzXrl175ceYmJjAz372M/ziF7+ALMtQVfUHhb6j0QiDwQB7e3u4desW7t27h/Pnz+PS +pUuYnZ1DMplAIpH4wfvoeR5u376NL7/8EqPRCFeuXMGVK1eQSCQQi8Wg6/oPfrzhcIhLly7hgw8+ +eOXHI3rXjUYjPN14iuvXr2NnZweu6yKfzyORGL+mk8kkkskkIpEItre38fjxYywsLODy5cs4d+4c +otEo4vE4Qyp66zSbTdy7dw+3b9/G4eEhNE3DzMyMuLYNw4BhGHBdF3/5y19w7do1rK2t4fLly5iY +mEA8HheLxt4XQRCIRS8bGxu4c+cOnj59igsXLuDy5cuYmpp6LcfF930EfoCdnR188cUXuHfvHi5e +vIirV69iamoKsVjsne4YQkREFGLwTURERERERO+VwA9weHiI//zP/8T//t//+5V/fnl5Gaqq4sMP +P4Su61AU5YcF38MR2q02Hj9+jD/84Q/4j//4D/z617+GLMuIRCJQFBnxePwHPbbnefA8D3fv3sXv +f/97UfldKpUQBAEMw3iloDp8vHv37uH3v/89Op0Out0uJkoTCIIAmqYx+CZ6ZjQaYWtrC19++SX6 +/T5UVUWhUMDa2houXrwI0zQBAPV6Hf/xH/+Bzz77DOfPn0cQBMhmswCASCTC4JveOo1GA/fv38dX +X30FRVGgaRqmp6dx/vx5LC0tifeGhw8f4saNG9jb28Pf//3fwzRN0cnkfQy+R6MRms0mHjx4gH/7 +t3/Dl19+id/+9rfQdR2GYUBRlB99XAI/gOu52N3dxR//+Ef88Y9/xO9+9ztEo9Hx7xTyuz0qg4iI +KMTgm4iIiIiIiN4bQRAAAGzbRqvVwsHBAXK5HBKJhGgx/n0B7tTUFNLpNGRZ/lHBlKzIUFQFpmki +lUohm80inU6L1qc/9LHDCu2w1Xmj0UC320Wn04Ft2/A875XbRn/z8drtNrrdLizbgud54rgS0biN +brPZRLlchq7rmJ2dxeLiIlZWVnDu3DlRESvLMoIgQLPZRKPRQK/Xe/YaPTtt3en9MhwNUavVUKlU +MD09jampKSwvL2N1dRVLS0totVpot9sYDAY4ODgQrf3DcFdRlNPehVMhS+PuMJFIBOl0GtlsFqlU +CqZpjtuQy6/nuIStztvtNo6OjtBqtUQbes/3TvswEBER/SQYfBMREREREdF7LZvNYnl5GVNTU0il +UkilUt/5/aVSCefOnRtXe8s/rNobGM8Kj0ajmJycxJUrV+B5Hi5evIipqSnRMpZzs4nOHs/zMBgM +0G63USgUMDExgbW1NSwuLmJiYgKqqsLzPPR6PUSjUb7O6cywbRvdbhfdbhemaWJxcRGrq6uYnZ1F +qVhCPB5HJpMRi6UmJydx6dIlTE5OIh5PQNO0096Fn5wkSdANHbFYDNPT07h69SpM08Tly5e/Pi76 ++3dciIiI3hQG30RERERERPReS6fTOHfuHC5evIhSqYSJiYnv/H5d1xGPx1Gt1qBpKjRNg6qq39k+ +fDAYYDAYwPM80e7V931omoZEIoFSqYSFhQVMTk6KKrAwIAiCAL7nw/VcjEYjWJYF3/fhui6CIIAk +SZAkGZqmQlVU+IEPy7Jh2xa63S4cxxEhW6VSge/7sG0biUQShqGLKndN1aCoz1edOY4j2td2u91x +5dizxzs6OgIAWJaFZDIFXddgGAY8zxPfF84HD/cl3F5FkaFrOvRn26DIyonnPz5b3LIs2JYN2xlX +w/q+Jx5HliX4/rjiPDwfYet1XdefO36j0Qi+74ttk2UFsjwOHn0/gKKMK/PEY2g6ZEUW2+I4znh7 +bFuch2/uk6aPt+GbxzTcFscdP4ZlWaKNPADxGM1mE6PR6AdV04fHzXEcDAYD9Ho9aJoG0zQhyzJs +2352bnwEwbiyWFVVKIoCwzDEtReGseGxcl1XbPPx4xfuu65/fcxVVYWqquIxjp9Lx3EwGo6eO5eK +oqBaraLf74uuBN/c9/DYu64L27bhOK7YBwBi7IBhGOL8ffO6+j7hNXJ8P8N9AABZVk6cZ13XoSoq +ZEWGZVkYDoc4qhyhXq+j2+0ilUqJSs9w1q8sy/DccThu27bYt3a7jUqlAtdxYVkjRKNR8TzhOQrP +w3ddf6qmnnhNh9fdcDQUz2maJkzTFPeDrx9HEteBYRgvdcx83xczjMfX89fXVniNKIoCWZbFDGhF +UaCq6it1tnjRfoSP5/s+HMeB67rw/QC+70GWFaiqAk3VYEZMGLohXsuu62IwGKDfH0BRZNGK27Zt +2LYttluWx38WLlDwPR+e74nXQvhaOn5P0p4df1V9dh6OvZ5+iHC/Pd8T1+bx+2t4TdbrdQwGA/i+ +/8LXTzgD2nbs517LAMT9NHzthO9tqqqi3++Pr+2jIzSbTfE6DY+b67ro9XuQZRmpVAqlUgntdhuK +oqBYLD5bzKVDluVj96Y+ZFkS9xzHccR17fsBgmD8HqkoCnRdF8/1ouPyXedDVRWxP+HX8evu+HEZ +jUZiG8LjoigKRqMROp2OeD991fuyqqowTRPpdBpTU1OwbRulUgnJ5Ph9WJIkcVwGgwEAiNdgeE2O +74vj4xKel7BVuq7r6PV76HQ6aDQaGI1GAIDhcIhGo4GDgwPY9nj/wp8JH0NRFPEe67gOXNf73uMn +SdL4vu4H6PV737rN4XEKvz/8d1VRoTx7XMMwXngfsG0bw+H4tX78ucPrcjgciveS7/o9I7xHhJX1 +r/J+QEREZxODbyIiIiIiInqv6bqORCKBXC6HYrGIycnJ7/z+ZrOJe/fuYWtrC6lUCul0GqVSCbOz +s5ifn3/hzzx69Ai3bt0az8WemECpVBLV5c1mE+vr6/jzn/+M4XCIbDaLXC4P0xx/iBt+oF+r1vB0 +4yk2NzfR7XbR6/Xguq748DqXyyGbzUKSJNTrddTrdVy7dg2dTgej0Qh37tyBZVmIxWKIRCLI5XKY +nZ3FwsICJiYmMDU1hWQy+cL9vXbtGm7cuIFr166h3W5jOBzi7t27sCwL8XgckUgE2WwWMzMzmJub +Q6fTQbPZRLfbFUFiGGaGQZJpmpidncXc3JwI/FOp1NeBq+djMBxgNBrh6dOnePr0KQ4PD098EB5+ +YB6GFPl8HsViERMTE5iZmcHs7Kz4sL9areLp06d48uQJ+v3+iYUIYXDm+z4Mw0ChUECpVML09DRm +ZmaQTCTFtmxvb2NzcxMHBwfo9/vo9Xpi1m08HsfU1BSmpqbEIorjxzQMRmu1Gp4+fYqNjQ10u10R +VpmmiUgkgidPnmB3dxe9Xg+WZb1SyBKGR5XDCq7fuI5bt24hn89jcXER8XgcBwcHODw8FAGaLMtI +p9NIp9NYXFzE8vIyJiYmRFA5Go3Q7XZRr9exubmJ7e1tdDodDIdDOI4jwu5SqYSZmRlMTk6iWCyi +kC+IgCEMK4fDIba3t/H06VPs7++LcxmGWqPRCOvr62g2m5AkCa7rnti3er2O3Z1d7Jf3Ua1WUa1W +RSAkSRIikQgikQimp6cxNzeH6elpJBIJJBKJ7w1Yw3B+Z2cH6+vr2N7efu7aBSCu3cnJSSwsLIw7 +RSTH1+7Ozg5u376Nmzdv4sGDBxgOhzg8PMRf//pX7O7uIh6PIx6PixCo1Wrh3r17sG0bR0dH+Oyz +z7C/vy9m8k5OTmJ+fh5TU1PIZDLIZDKoVCrY3NzE/v4+er0eer0eJGkcVoYdJKanp8X1l0qlYNvj +oPP+/fu4efMmDg4ORNv1Xq+H/f19NBoNESrNz8/jwoUL33o/++Yx6/V6uHfvHu7fv49ms3kiVA0X ++JimiXg8jvn5eczPzyOXyyOVSr70XOOwhbNlWXj08BFu3LyB3d1dzM/PY3FxEYPBAJVKBbVaTSwM +iEajSKVSKBaLWF5exvLyMkzTFIsMrl+/jps3byIej+PcuXOYnJzE3t4eyuWyuLZjsRgurF3A2oU1 +GIaB4XCITqeDzc1NbG5uihb1o9FIBJG5XA4TExOYnppGsTS+H73sIoJve02Hzxvew5rN5olFDJFI +BOVyGU+fPhUt84+PtAjnTI+GI+yX97G5uYm9vT3xXgJAnKfJyUlMTU2J96p4PI579+7hzp07+PLL +L7G3twfLsvD06VN4nieOYTKZFPeQw8NDfPXVV9ja2oLneSiVSojFYuKauH79Om7cuAFd17G8vIzp +6WmUy2UcHByg0+mIRUHj98OcGBcwNzcHACIoHt+fetjc3MDm5iZqtZq4v4fXczabxeTkJCYnJ8U+ +RSKRE8clbM3+9OlTbG1tiXtT+H6lKArW19dRqVSgKAocx3mlc+j7PjzXQ7VaxZ07d3Djxg04joNi +sYh4PA7f9zEcDnHt2jXcvHkTkiRheXkZ83Pz2C/v4+DgAO12WyxSyWQyyOVymJ6extLSEqanp7G+ +vo6bN2/iyy+/xP7+PjzPw/b2Nv7whz/g9u3biEajiMVimJmZwfz8PCZKE0ilU0gmk9jYGB+/arWK +Xq934vilUqnn3tcM3YBlWxgMBuJcAsC5c+ewtLSE/f197O/vi/u7qqpiwZJpmkgmk8hkMpibm8Ps +7OwLf/eoVmu4desm7t27J97Xw3/m83k8evRInJPji9p834eiKOJeHd5zisXiS3X1ISKis4/BNxER +EREREb3XwuA7m82+VPDdbrexvr6Of//3fxeB8erqqgiMXuTx48f4wx/+gGq1itXVVayurmJxcRGK +oqDVauHBgwf44osvEI/HcfHiRSwt2aJ6KWwtu7e/hy+//BJffvklqtUq6vX6iYrHqakpzM/PQ1EU +7O7uYn9/X1SdDgYDPHr0CFtbW5BlGZIkIZfL4YMPPsAnn3yCy5cvI5VKvfDD50ajga+++gr/8i// +gmq1ik6n88LHy2azuHLlCj766CNUKhXs7u6iVquJcC6sUFQUBbFYDIlEAlevXsXPfvYzUVmZTCZF +8O16LobDIdrtNu7fv4+//OUvePjwIdrtNjqdDgzDEPPQwyq82dlZnDt3DhcuXICqqqKyrtvtYn9/ +H59//jk+++wzNBoNtFotuK6LaDSKaDQqqmej0SiWl5dx7tw5XL16FclkErFYDMPhEK1WC/fv38cX +X3yBhw8folarodlsihA+l8vhwoULuHTpEtbW1pBIJE4c03Bbdnd38cUXX+Cvf/0rarUaOp0OXNdF +PB5HIpHAcDhEr9eDqqpwHOeVgu+wknW/vI///M//xL/+679iYWEBP/vZz1AoFLC+vo7Hjx+L8FqW +ZXEdf/rpp0gkEigUCgAAWZYxGAzQaDSwubmJv/71r/jqq69Qr9dFCBOGC+Hxunz5MhRFQTabhYJx +8O354+rm8Pj95S9/wfr6OjqdDjqdjghFNU0TQVw0Gn1h8L1+fx13797F06fjRSCj0UhUKyeTSSQS +CVy6dAk///nPRdeEaDT6ncH38Yr0nZ0d/OUvf8H169dPLDAJnyMMry9cuCCuXUmSEE/EsbOzg//6 +r//CZ599hkqlIqo3w3tG+FoJg2/f92FZFnq9HhzHQbfbxb1798T3LC0t4aOPPsKVK1cwMzMD13Xx ++PFj/PWvf8Xdu3fRaDTQaDREJXU6ncba2houXbqECxcuIBqNIplIPnuOPtbX1/F//s//wcOHD/HL +X/4SruuKIG53d1dUen/yySdIp9PfG3wDQOAH6Pf7uHfvHv793/8d5XJZBGdhNwPDMBCPx5HP5/Hx +xx+Lilld114p+LZte7wf99fxr//6r7h79y4+/PBDfPrpp2i323jw4AG2t7cxGIwXqSSTSUxMTGBp +aUlU2EqSJBbC/Pd//zf+5V/+BdlsFr/5zW9w4cIF3L59G3fv3oVt29B1Hfl8HkEQYH5hHpIkodfr +4+joCDdv3sQXX3yB3d1dEX6Hx292dlbcB/zARzab/VHBd3jfqFQquH79Ov77v/8b+/v7aLfbGI1G +4r7hOI64Xkej0Yn7RhAEGA1HaLVb2NjYwF//+lfcunVL3MOCIIBhGEgkEjh//ry4hkxjXGW9vr6O +f/u3f8PDhw9RrVbR7Xaxvb2Ng4MDsUgmGo3i008/heu6qFQquHHjBh4/foxcLoerV68im82K+9Pn +n3+O3//+9zBNE7/85S9x9epV3LlzB+vr6yK8dl1XLKa6evUqIpEIZmdnv36PcF30+31Uq0e4c+cO +Pv/8c2xvb6Ner4v3CNM0MTU1hYsXL47Ph+8jnU6fDL6H42rujY0N/OUvfxELvDqdDiRJQiKRQDQa +Fe9lqVTqBwXfrueiVqvh3r17+PLLL5HNZnH16lUUCgW4rgvHcfDll1/i97//PSRJwi9/+UsMPhng +7t27uHfvnuiIYdu2WNxz5coVmKaJUqmEBw8e4P/9v/8nFg95nofd3V0cHR2JDhyapuHSpUv4+OOP +cf78eczMzGBiYgJ3797F559/LhZztFotcfwmJiZw4cIFXL58GZ7nIZ1Kj6vgnx23cJuDIMDf/u3f +wvd93Lp1C3fv3kW73Ra/o4QLV6LRKKampjA9PQ3HcZDL5b4l+D7CX//6V/z7v/87VlZWsLq6irW1 +NSiKgkwmgydPnuBPf/oTHj9+LM5NeM9RVRWxWAzJZBIfffQRhsMhAEBRVAbfRETvAQbfRERERERE +9F4L2/Aeb5H9XcIKV13X0e/3sbu7C13XsbS0JFpsAhAthG3bRrVaxf7+PjqdDhYXF0U78+NtX5vN +pqiiDdv22raN7e1tPHz4EI8fP8bGxgY6nQ4URTkRToZV361WC6qqIhqNYmFhAQDQ6/UQiUSQTqeR +y+VEVWI+n8fc3JxoQ6uqL/6IIPxQfXFxEb7vixa3qVQKuVxOHLNsNou5uTkUCgUR2qqqKkKDMFwM +BUGASqWC27dvYzgcwvd9TE1OIZDG39PpdPDw4UM8fPgQDx48QLVahe/7iMfjiMViYj/Cashms4lE +IoF2u41+vy8qTsvlMu7fv4+HDx+KUExVVeTzeREE6LqOVqslfq7VaokFA67jotft4dGjRyJYa7fb +0DQN+XxeVKnLsgxN01Cv13Hjxg34vo9EIiFavxq6gf39fdy/fx8PHjzA5uamqBZPp9MnKtR830ev +1xPVa68i/JnRaIR2u41GowFVHX/Y3+l00O/3RdtmwzBEELW1tYV0Oo1isQjTNJHNZpHNZlEul3H7 +9m08evQIOzs7YrFFsVg8cb2HiyHCcxkupDAMQ/zZgwcP8OjRo3E7b9dFJBIR7YtN0wQA8fPH2+S3 +223s7e3hyZMnePDgATY2NmBZFvL5/Im2w4qiQFEUcZ22Wi2kkqmXWjgQ+IG4RoMggKIo4tr9Ztvo +IAhQr9dx7949DIdDfPzxx8jn80gkEpiamsLs7CxGoxHq9bpob5xMJk+0vQ4rmMNqxWg0imw2K1qj +a5omuiFEIhFUKhXs7++jXC6LxRaZTAaJRELcx8J7wO3bt+F5nqgcdxwXruug3++j0WigWq3iyZMn +J8LSsGI8FoshFot9730wPA7+sZbmYXiqqiqSyeRzr3nbtrG7uyvuS7qui/vYy5yf8bXticUY1WoV +29vb4rk8z0MikRCVy5IkodFowPd95PN5Ue0btmIO20IPBgPcu3cPo9EIR0dHohNEGJyF12atVsOD +Bw/w8OFD7OzsiBAvbPUd3gcAYG9vD6PRCJIkIZPJiI4OrxKAh8evVqthfX0dDx48wOPHj9FutyFJ +kqheDe8b/X5fBMbHj314rW1ubYrXT61WgyzLyGQyiMfjz1qmy1AUBb1eD+vr6+I1qmqqCJ3D+2v4 ++kin0+IYxOPxcYeMZFJU3ofV1+MW9OOq53B8Rr1ehyRJePDgAYDxwpZw/IemaaJbx+bmJgzDEAt0 +wveAer2O+/fvi3vTcDhEJBJBqVRCLpcT50NRFJTLZdH+O51OAwAMw4AkSdjc2hSPs7u7K9qBh+9v +YSv2sPvG8Q4Qr3oubdvGYDAQ7y/h+73vjRdehd01HMfBw4cPoaoq6vW6eC9VVVUEyNvb29B1HRMT +E5goTUCWZUxNTYmuC2Fon81mEYvFxAKthYUFlEolAMDW1pZYxDYYDGCaJorFItLpNGRZFr9fhF1C +giBAMplEqVQSowW63S4ajQaGwyHW19ehaRoqlYpYPBKNRhGPx0VngXDBTb/fx9TUlDgvx+8Ttm2L +Fu37+/uYmJgQ15hhGCfu1ZqmiQVq37zn+L6Pg4MD3Lx5E8PhEJIkYWpq8uVP3CsKFzQ5jiPO449Z +9EIEQCyGAyBGq4TjE8JRB982rojofcXgm4iIiIiIiAhffzDtey/+QFt6Ngc6DP0WFxdRr9fRaDRg +miY6nc6J73ddF4P+AN1eVwSpjuMgmUxiZWUF8XhcBCrPbYv/9WzrR48e4U9/+hM2NjYQBAFUVcXM +zIwIAMJq1GazKao/z58/j6WlJfzxj38UH0ivrq7ik08+EVWxmUwGxWIRpVJJfDD+Iul0Gh9++CHi +8Tj+8Ic/oNlsQpZl8XipVArxeBzpdBqFQgGFQkHMRA2Do2QyKYLwMFw6OjrCcDjEnTt3UKvVkMvl +8PHHH0OTNQT+uA30rVu38Kc//Um0mw63d2JiQlRo1+t1UQn+3Ln0fWxtbYlqcc/zEIlEUCgUMDEx +ceL4PX78WLRIPs7zPbTaLdy5cwf/8R//AVmWEYlEsLS0JNrwhi3MG40Gtre3cfv2bSiKgnw+j2Qy +KVqJb21t4c9//jMePHgg5uOG7WOTyaQIWJ88eSKqVl91luyLhAs0HMdBJpPB2tqauEY7nY5o77y5 +uYlsNgtN07C8vIxIJILt7W18/vnnePDgAXRdFwseisUiYrGYmP1eqVSwtbWFcrk8DslmZsUCiVZr +fPz++Mc/ioUTuVxOtJRXlPEc6na7LVqAh/vtui4ajQY2NjbEQoiDgwPRUjkajYqwu91uo91ui6Cx +2+1iZL3aMTzetj+8dsNWvY7j4OjoCJVKBcPhUGxLKpXClctXMD09jV//+tdIJpNwXRf7+/vIZrO4 +fPkyVldXxXUgyzI8b9z2+L/+67/Q6XSQzWbxwQcf4OLFi6JtdDabFcHwF198gRs3bohreH5+XixO +8DxPdEfY2trC/fv3RZvodDot2vmHbNvGwcEBgiBAOp1GIpHA4uIiMpkMstnseHFO8uWrIsNuDcVi +EdlsFsnkuIV5OMe71Wrh6OgIjUYDtVoN+/v7sG0bs7OzWFtb+8Hzrx3HQaPRwJMnT5DP55HL5bCw +sCC6S1QqFezt7WF/fx9PnjxBOp0et4Wemj7xON1uF0+ePEG320UikRDBXzabFfcbVVFRLpfFeYhE +Ikgmk5ibm0M2m0UikRBzzg8ODrCzsyPa209NTYlxFK8SgvneuEq4XC7j888/x5dffgnf9yFJEiYn +J0XAG97r9vb2xFzkUPhnw8H4ev3Tn/6EdrstguzwHhYEAYbDoajkfvz4MUajkfjzyclJ/MM//ANi +sRg6nQ6q1SpmZmZw5cr4uk+lUshkMsjn8ygUCqjX6y+1eGIwGGBnZ0dU/s7Pz4suHsPhEDs7O9jf +38fGxgZmZmYwPT2NiYkJTE5O4ujoCNeuXcMXX3wB0zQRi8UwNTUlKoiHw/F4hUqlgu3tbWxtbYlx +AKqqitdGeFzK5bJYsBS+15imKQLrBw8eiPD2TQv33XVd0eo9XOhlWdaJ4zI9PY3JyUmk02n85je/ +QSqVwmg0QrlcxvT0ND766CPMzc2J+0o+n0c+nxfH7+7du2KOfXhNpdNp0e49XGCysbEhFhaEiwW/ +eS63t7dhWZZYFJFOp5HJZJBOp9FqtcRirGaziVqthmq1CsuyTlyvYReMZrMpFlqoqorJyUlxX5Yk +CbFYTCxgO/57hqIoGAwGODo6Eq3bb968iVarhUKhgI8//viNnbcgCERL94ODA+zu7r7yIgmibzq+ +iCdcwBKJRMQIlLDrB4Nvoq8x+CYiIiIiIqL3WljhG87u/LaQLKzCHA6HSCaTWF5eFh9OHx0dodVq +wXEcKLICWZHHoWK3g+pRVbSkVRQFyWRSVGP7/oufK2zz7TgONjY2cOvWLfEh9uzsLFZXV3HlyhXk +83nx4f6jR4/g+z50Xcfa2hp+/etfY3d3F9evX0cQBFhcXMTf/d3foVgsjoOBRBKqNq6aDitlXyQe +j2NtbQ1zc3PY3t7G9evX4fs+FhYW8Jvf/AalUgn5fB7xeByqqoq50P1+H7lcTswGDavmwkruMHDY +2dlBu93GRx99dKLavVqt4vHjx7hx44aYZTo/P4+1tTVcvHhRPMfW1hYajQZ2dnZEK+njLaR3dnZw +584dbG1tiWrcsB16WJ3e7/fhOA4qlQparZao2nTd8XkI54zfvn0b09PTuHDhAlZXV7G8vIyVlRX0 +ej00Gg08fvwYe3t7ODg4QC6Xw+7uLorFImzbhizL2NnZwd27d7GxsYHJyUmxLaurqygUCuK5wg/N +w5/7scL9sG0b2WwWly5dEqHU0dGRqPhuNBp4+vSpqEzOZrPY29vD/fv3sbGxgeXlZSwuLmJlZQXn +z59HJpMR4cLnn38uWs5ubW1he2cbnj9uO1uv1fH48WPcunVLhFPhvoch/GAwQLlcRr1ex/b2tjiX +4aIO3/exu7sr5hKn02n8/Oc/Rz6fh2EYcBwHe3t72NvbE10PwtDvZUiyBCkYtzKfmJhAEARiXnZY +0ToajfDw4UOYpomHDx9ib28PAHDlyhUMRyPRASAej+Pu3bui0v78+fP427/923Fl5sQklGf3h62t +Lezu7uLmzZtIp9O4dOkS/v7v/x65XA75fF4E+uVyGa1WC+vr60gmk7hw4QJWVlawtLSEc+fOierI +7e1tVCoVHB0dIRKJYHd3V8wHz2azJ66HwWCAWq2GWCyGXC6HxcVF5PN5Me4hkUx8/zGTJMiSDE0d +V5/Pzs6KsQvFYlF8QL+7u4tHjx7h8ePHePToETY2NpBIJFCtVuG6rmiV/arCe0W/30epVBLzjkPr +6+uo1+uoVqs4PDzEgwcPRLv6sFIegKhADqvow0r7QqGAfD6PTCYDx3XEY9y/fx/nz5/H/Pw8zp07 +h+XlZUxOTopA78svv8TTp0+xt7eH7e1t7OzsIB6Pi7D8ZQRBAMu2xjO59/exvr6Oe/fuifBveXkZ +q6urmJ+fF/eNsNK2Xq+Lx/E8b9z9oTNeFHH37l1IkoQLFy5gaWlJzD6XJAnNZlOMyAgfY2dnZzwP +emICq6urGA6HuH79OlRVRalUwocffoiLFy9icnJSdGBwHAcPHjx4Lhj9tntTv99Hu93G7OwslpeX +kc1mxczrXq+Hp0+fiq4pW1tbAMYLVA4PD/Ho0SPcu3cP586dE/eU5eVlzM7Oipbd169fx9OnT7G/ +vy/ORzQaFYtINjY2cPv2bfT7ffEee+7cOZw/fx6xWEyMKwg7T4T3pjcpfI22Wi1xXefzefHeNhgM +ROV+uVzG7u6uWEgiyzJu374NWZZRKpXw8ccf4+rVq8jn8+LYBkEgKqrv3LmD1dVVTE9Pi/e1xYVF +NFvjxXS3bt3C1tYW9vf3sbOzg52dHcRisecWzIXjGsLfc6ampjAzMyMWpdRqNdTrdTFL/PDwUHRc +8FxvfA+WJHFvEn/2bJFYuOAhrNRPpVKYnp4WCzOmpqbGi3x0A/XGeEGcpml4+PAhNjc3YVkWfvaz +n4nuA29CGHxbliXeG/v9/hu9VujdF3amUhRFjA4Ix4cUi8VxAJ7OIJ6IQ1M1aPq4gxWrwOl9xuCb +iIiIiIiI3mvlchmfffaZCGOOByLHhW1lw0rtlZUVUanWbrdFxXXYXng0HKFSqeDp06cYDAaiPWsm +kxl/6O56sB37hc81Go3Eh75hRVQ0GsXKygp+9atficq3sFWx4ziIxWKYmZmBLMtYXFwct6h9Ntcz +bIMctpSORCIwI6aoIFEU5Vs/yJelcQtvQzde+Hhh5UnYPnvcSnRKVBCmU2mk0ilRhS1L43asvu/j +6OhIzNwN279aloV2u42dnR20Wi34vi/mkV++fBmTk5OYnJiE67minevGxgaSyaTY5+PBeVidaxgG +FhcX8atf/UpUyCXiCVj2uG3s3t4estksHMeBruuQJElUQ4eVb2G1/9raGj744AMRioWtXF3XxcLC +AnZ2dmCaJur1Oh49eiQCsYODAwyHQ5imiaWlJXEuS6USEokEbNuG67o4ODgQFejhtvwYuVwOH374 +Ia5cuYK5uTnMz8+L1va5XA7tdhu9Xg9BEKDb7eLp06eifXLYsjasMv7FL36B2dlZlEolRKNRMRO8 +1WqhWq3i4OAAo9FIzJktlUqilX9YhfzBBx/ggw8+ECErAPEcm5ubSKVSiEajUFVVBGDh9SRJkqh0 +Cqs7w+8Lq/nDtvPRaBT5fB6q8t0ff4XHdzxCoIiLFy+KysKwQjtcVFIqlUSnga2tLdHKudfrIh6P +QVEUsZgkfD2E7fTHVZWRccW3Ow5zwvMQthQ+/hrt9Xqo1+vY2dlBv9+HpmkoFApYXV3FRx99hFwu +96zi20csNn7up0+fisrWVms8zzkcrxCKRCJibvLi4iLm5uZEB4lwLn00Gn2pa0uSJZgRE3Nzc+K4 +f7PtcC6Xw8zMDCzLQrlcFhXqYWvu8IP8Vw2/TXP8vJ988gmWl5dFQBsKuwiEx/bg4ADpdBqLi4sn +7vPpdFrMgJ6dnRXV9OG93rIsbG1toVKpiHvt9PQ0PvjgAywsLKCQLyCZGrf1T6VSqFar2NjYENXB +u7u7ogK2WCjCD759hIEkSVBkBZ7viXBwb29PtBefmZnBL3/5SywtLWFiYgK5XE6Mxeh0Onjy5Akq +lYoYpTEYDHBwcIB+vy/GZGQyGaysrIgW/fl8AbIsIRaNwTAMcQ0piiKq4XVdx+TkJDRNE8FveK2H +12skEoFlWfA870Rr3u+STCZx8eJFfPjhh1hcXMTS0hIS8QT8YDyuIAxaFUURi5PCjgkHBwfivXF6 +ehqXL1/GuXPnkM/nkU6nxfloNpvY2NgQ3Ur29/dFG/xoNIparQbHccQir5///OfPtcXvdrtiHITr +ui8V6v8YiUQCa2tr+Oijj8RxSSVT8IPxCJVwUUXY+aRarSKbzYrXYLiQTZZlcY7CrzCAPn5vn5yc +xKVLl3D+/HkUCgWk0ilouoZ4PC7mnx8eHkJRFBweHor7zfHgO5lMYm1tDVeuXBGvo3BRXDQaRTKZ +RCaTwXA4FNXzrVYLjUZj3IUgGoFhGGi1Wtjd3UW1WoWiKMjlcsjlcmJ0SHgNTk1NiYUW4WiN8J7j +eq7488PDw/Es92eL5bqdLlzvzYTf4fmwbRvlchkPHz5Eo9F4o9cKvR/Cqu+wq0E4RiDsrBR23Zic +nMTk5KToWhGO43nTi3WI3jYMvomIiIiIiOi9Flb5fl8AXCgUUCwWce7cOXzyySe4evUqrl27JkKH +VquFer2OIAig6zos20KtVsPGxoYIvsMWotFoFKPRCI7rvPC5bMvG0dERDg8PcXR0JD7cX11dxT/9 +0z8hFouJFs/hB72zs7OiTXfYVjyspAxDCsMwYBqmmK18fF+/bb8lefyzZsQUwXf4HMc/TA/btofB +dz6fF3Njwxbnw+EQsizD9VyYpolHjx5BkiRRwdbtdtFsNnF4eChmaXueh0KhgE8++QR/8zd/M96H +Z+1nw+r8u3fvIpFIIBKJQFEU2LaNWq0Gz/NwdHSE0WgE0zSxvLyM3/3ud0glU9CNcVVzWP329OlT +5HI5MXsYGLdA3t3dxWAwQL/fF8HnpUuX8NFHH4mA4Xhl8cLCggiNwkCl0Wig0WigUqnAsiyxcOL/ ++93/h0wmA90Yt9IO92lzcxO5XA6dTkdUt/0YxWIRP//5z/Hb3/4W2WwWmUxGhIzFYlHMm93d3UWl +UhHPaxgGjo6ORNi3uLiIv/3bv0U2m4Wu6yIQ9jwPrVYLBwcHosL04cOHYjGI7/uifXk+n8dHH32E +3/72t+I5wpnXkUgEd+/ePbGIIXw8y7IwGo1Eq/l0Oo3p6Wlks1nx88ViEa7riuswbLUtK98fqIYf +DBcKeaRSSXieJ34+bGWvqRqmJqdgGIaoSj9+7YYtRw3dOBE8qap64rUXBnfhPOzjM+LD11IkEhEt +wnd3d08E3xcuXBiPBdA0cd34fhamaWJxcRGbm5twHAetVgubm5tIJBKYmZkR+6rrOpaXl/GP//iP +WFlZQaFQEAF/eB8MA/mXOWamaWJhYQFTU1PiuAPAaDjCYDiu2ATGVax37949EXyH4bCu6cArFn1r +moaFhQX8wz/8A1ZXV0/MPAfGi3a63a4IvyqVCtLp9HPtqrPZLD7++GP84z/+oxjXYJqmuJdsbGxg +d3cXh4eHYnb67OwsPvzwQ8zMzIggLgze6vU6NjY2xDW/v78v2qKHXRCOz6Y/TlEUQB0fq1qthidP +nmB/f1+0e56bm8Pf/ebvsLi0CF3Xoeu6uG80Gg0Ui0Xs7OyIhQRh6/V6vY5OpwNZlpHL5bC6uoqf +//znJ0LSbDaLZCqJJ0+eYHNzE51OB91uF5ubm2JBQbiv4baGVf2maSIaiYqK15e9Z4WjNP75n/95 +3Okgl4dujO+//X4fT58+xaNHj8Q87HAG+2g0wuHhoQhuw4UIy8vL4r0vlUqJEQgLCws4OjqCLMti +AVKj0UAsFkOtVoNt2ygUCrh06RL+6Z/+6cRijCAI0Ov1cP/+fWSz2RPvEW9KMpnEBx98gP/1v/6X +CH7D91jbsvHkyRM8evQI7XZbXCtzc3MiEPvmOQrDb8MwMBgMsLe3dyL4Do/f6uqqqBQNx28MBgM8 +ffoU5XJZBN+6riObzWJqakpsczwex+XLl/HP//zP4velMCAPZ92HbfJVVUW3Ox4D02w20e60ISsy +DMNAp93B7u4uarXas8VIBdG+PlxgB+DE7xlht5nxIrVxq/8gCGAYBu7fvz++H4UL7HrdNxYEHq/4 +Pjg4wIMHD3B0dPRGrxV6f3zzd3ZZliHLsgjAi8Uizp8/j7W1NaytrY0XnCSSgAwG3/TeYfBNRERE +RERE77Ww/WwkEhEtnl8knIs5MTGBYrEowqJ4PC5asm5vb4tgJJxlHbbYzGQymJ6eRjqdFh/Mf9sH +UY47br0bBqW6rouZv8ViEbqmQ1ZOtluNRCJfVzs9m3H7og/JJPnrD8teRvgB8Tc/KA7/OwzXw8cL +ggCDwQCNegOtdgu9Xg+dTkdUTYchZtjOO2wBH35YHFa5hvOeY7EY0un0+MP/bE5s//H9DgPE4+2x +w5nqw+FQBIrZbBb5fB6xWOy5/TEMQ4RA4eNbloVms4lut4t2uy0Cg+vXr6Pb7T53rOr1cUvvZrMJ +VVVFS9UgGM9sHwwGIijKZDLIF/JIJBLPbUtY1XZ8ocGPEc7mDmeOx+NxsY+jZy26JyYmUK/XYdv2 ++Pw1GqhWq+L4RaNRMU8ynLF6XDqdFnNj+/2+WAQSXiPh6yI8l/lc/oXnMqyWPr5gIwyCj46ORAi1 +sbGBP//5zygUCqLjQFhZGF4TrxpOhVXlYYV+r9cTrfC/ee3u7e1hMBicmOkaBrxhy97jrxNZliEr +49eJIo8XS3zzHvD19473P1xQU6vV0Gq10O/3US6XcevWLRHwH9ftdvHgwQPUajWxXYZhPDcrXpZl +xONxEfJ+MzB+Va7ritm9YfeAXq8nZl7bto3RaCRaaIevC9u2RZeDHyKs/g8r379ZqZ5Kp8S9ular +ie0KFwiFNE1DOp3GxMSEmEd8/H1gNBqJKtkwDN7Y2MBnn312ooV8aGNjA9vb22L8RRhM7u3tIRKJ +oFqtPjfbOJTP58X7RKVSQaVSQbfbhSzLSCQSyGQyyOVzyKQzL3wPCEPP8PVj2zba7TZ830ez2US/ +38fh4SHu3r37wvc6y7Kwvr6Oo6MjcU2HC5fCa+f4c4bPFb63HL/uX4aqqkgkEiiVSuN7UyJ+4nUb +i8VENw3f98UCgHCBQXg+tra28MUXX4hW6Mdtb29jc3NTnA/HccR75WAwgG3bME1TvMcW8gWxL+F+ +hAvavm+B3Ouiadr4uBRLSKbG86vD42JpFuLxOOLxOEajkQinw+v6+HZ/831akRVxjwvfZ8PxFF9+ ++aUY33BcOEu81WrBtm0xj/1Fr6OwxXn4OgrD+vB4q6p64rUaLk47ODgYXwvxBFrtFvb29lCv12EY +Bubm5sR9/vjvGd1uF41GA51OB51OB71eT/yeYVmWWCx1fMFZOCP9+IKP1+n4LOZoNIpUKgXHcX78 +A9N7LXz/DhdZep4H13XhOI547ff7fTEiqdPpoF6v4+joCEtLS5icnMTExMRzv+MSvcsYfBMRERER +EdF7LZvNYmVlBTMzM6Kd+YuEbdBLpRIWFhZE285sNotms4nBYCBawpZKJVjWuOJ7e3t7/GF6oSCe +45u++SF6OPe00WiIwDBs/6woyrdWsIaVr6clfO7Dw0MxF7pcLmN/fx+j0Uh8SB+GB2ElYyKREB/k +hfNUw4AjnU4jkUiMq+9eonIXgKgoHY1G8DwPsVgMiURCtM9+2WqvMOQMg9B2u40HDx6g3W7js88+ +e+77R6MRqtUqGo2GqP4M9zUMIsOWr7FYTIRIP0Ulzrc9j6qqYl6raZoioA/Dftd1EYlEEIvFRPvk +Fz2OaZpIp9NIpVKihbUsy6JyMqzATCaT48d5yXMZXgOlUgmVSkUEXjdv3kS73RYz5ovFopgBG86U +fZXgO7x2a9Ua1u+vi5nA5XIZ/X5fXEthBffh4aGooA3b277u2bHhzNxms4lWqyWCp263i2vXrr3w ++6vVKur1ugiVkskkXNd97r4QtkD+sQFQEASwLAs72zu4/+A+dnZ2UC6XRaeKMNwNW1Xv7e3Btm3x +AX54zPzAh4Ifth3fdm2HC4bS6TQURRFB2DeD9jCo0nX9ubb44eshrExtNpuoVqu4du0aDg4OYBjG +c88bVrUOBgPRkaDRaGBzcxO9Xg/37t3D/fv3n6s8B4CVlRX84he/wIULF8S8bsuyxGKZWCw2rrp+ +yddPGLyPRiNxHYWLEu7cufPc93ueJ64hXdeRy+UQi8VEWPy6Ha9alKXnQ/VwoVAYdocBj+/7YoZ3 +o9HAjRs3cHh4+MIW/d1uF7VaDZ1ORyxUCBdKhddi2Ir++L3ptCskJUl6bnFDKByLcPy4eJ6HwP/u +c+QH/njm+7PxLOF1fefOHVSr1ROty0O9Xk8svsnlcgAgWtp/c3sVRYGu6S9c1KNp49bp6XQa2WwW +6XQaQRCIdv6pVAoTExNot9sol8toNpvi3l4oFGAYX4fovueLiurNzU1xrw5f4+H1I8sydnZ2MBwO +RQeAsCPBmwq+wyr7UqmElZUV5PP5N3yl0LsufL8M32/DLjThgg/P82DbNprNJmx73DFqb28PT58+ +FV2qwt+DATD4pvcCg28iIiIiIiJ6r6XTaayuruLSpUsolUqYnJh84ffphi5m6oXtyjOZDAqFgghs +wxbVy8vLGA6HqNfrKJfLSKVSKBQKmJ2dRTqd/t5tCucad7tdMbs4bOX9uqqAX7fjFZy7u7u4d+8e +1tfXcXBwgEqlcqIKPWyN2+l0RDVUWMUSVn2Hba3DkFjTtJfe73BbHMeB541nKYfH79uC2xcJQ6Ow +kiwMwYfD4QtD1XAfwv0MW8yGjxG2t47H42/NuZRlWVQ76roO3/fFh6rHtzkWi33nHObj8yYBiLb2 +4TUcVhnHYjHo2svPLVcUBel0GjMzM9je3kYqlRKtm7vdruiCMDExgWq1imazibm5OVFdGH591/P5 +vi9GFuyX93H//n3cuXMHBwcHODg4OFHNHbYL7na7olI0XNQQfjD9uoQhX9iKPnx92baNw8PD574/ +7CwQLrAIOx286LVzfFboD70Gw+rZZrOJjc0N3Lp1C5ubm+L8eJ4H3/dFeClJEvr9vgjiv6vl98v6 +rm1XVVVck2FXCcdxnlucEIZV4aIiWfr6Gj8etoYVdWE3j16v98LXQxiuhYF6uO+9Xg+O4+DJkye4 +d+/eC7tGuK6LmZkZTE9Pi2pmx3FgGAai0eiJxTsvI5yBHIbdlmWh3+/Dtm1Uq9VvvYY8zxMLIyKR +yCvdf1/1/B3vRvJNx2dWB0Egunh4nicWpIQLjsI56N92PgCI86Gq6oluA2HHiG+OADlNxxcEfNdx +ASDeX/3g+xfehNdA+D4bXgu9Xu+FIw7C4xd28AgXM33ze8V4CU2FIj9/X9E0TVSF53I5FItFSJIk +RhBMT0/D8zy0220cHh6i3+9jbm4Oy8vLKBaLMAwdnuvBduzxYpudHdy5cwcPHz4Ui23Ce87xRVfh +uIPji23CBQOv2/GxMmGFbRjEE/1Q4X057O4wGAzQ6XRQq9XQaDTEe5NlWWi1WmKE0+HhIRqNBgzD +wOTkJEqlEjKZzLd2tiJ6lzD4JiIiIiIioveaYRiihWyhUEBpovTC71MUFao6rgoMq/ySySRmZmZE +ALS9vY2pqSnU63X0ej20Wi10Oh1IkoSJiQnMzs4ilUp95wfr3/Zh7NvyYfy3CWeuHh4eitC7Wq0i +k8lgfn5eBFBh9a+iKLh27Rpu3rx54nGOh8fhfr/qvh8Pb3zf/zpY+QHHMKxQj8fjyGazmJiYwOLi +IgqFwgufN9z+VCol5iZXq1XUarUT2/E2nc/weB0PAF91+8IwMQxHgOfb5P+QcxAe+3w+j8uXL8Nx +HMzNzYkgEhgHsPv7+zg4OMCtW7dw7tw5XL16FWtra+NZs4UiFPXbq/ts20alUsHBwYGoxg3nMn/8 +8ceIxWKIRqMwTVOExXfv3sWNGzcwGAze+LmRJElcf9lsFgsLC5icnHzh9/q+D9/3RWVlPp/HzMzM +CyuTf6xup4vDyiG2trbEcfN9HzMzM7hw4YJ4zYcBXb/fxxdffCHO208hXIzwY67t8LhqmiY6dszP +z2NpaemFFcbhOQjnGqfTaTFiIHwdZLPZFx6HxcVFXLhwAflcHuV4+USHih96Lwz3PxqNIpvNIhaL +YWFhAbOzsy/8mfC+GS7yKhQKWFlZQSQS+cnOW+hF+328bXYymUSpVMLs7CyWlpZe2LI/PB/AeKFb +JpMRHVm+eV8+S37M9obHL5FIIJ/PY35+HouLiy/sSBO+PwRBII5f2F3jh4hEIiiVSpibmxNjAKLR +KBYWFtBut9FqtUR7/kQigaWlpWcV3wYGwwEODw/FvXp9fR2dTgfFYhHLy8vinnN8ZMbnn3+OW7du +/WTnJFzoMz09jZmZmTN3XdHbJ7znhQuxws4G3W5XjAeqVqs4OjoSi0DCsTB7e3u4ceMGfN/HlSv/ +P3t3+tzWeaYJ/zor9n0jQHCnRIlabMeJEyfppDM9b9f8tVNT/WG26uqedKeTdBJv2hdSIsEV+76d +9f0AP49JW7JEWRZF6fpVsWRbxMHBwcEBjOu57/sGbt68iXA4/NZ9DiV63Rh8ExERERER0XtNtN9M +pVJy1vHzKIoya8f6dWVaIpFAuVzGYDBApVLB/v4+lpeXZVVwp9OR81kLhQKWlpae2Ur0eUQI/qqh +x5s0GU9wcHCABw8e4P79+3j8+DEcx8HS0hI+/fRTlMtllEolJBIJWZUKAE+fPj0VAJ0MrQGcmld7 +Ft+uUnuVbYgvGsV83Vwuh+vXr+NXv/oVNjY2nnsbAPLL706ngy+//BLtdlvuw9vUZtL3ffieL4Mh +UaX67X18lfPvecH3syo7n0fTNEQiEeTzeUSjURQKBRweHso5ytVqVVY9icDk6OgIruvKKtV0Ov3C +4Fu0zb137x4ePXqEdruNUqmETz75BKurqyiVSkin0/Lc/Kd/+idUKpUfNfg+ef5FIhFks1lsbGzg +l7/8JW7evPnc2wCzaudAIDCrrld/nHOu1+9hd3dXht5bW1soFou4fv06PvroI/maNwwDruPi4PAA +vV4Pjx49+tGO2bOOn7gGiEULZz0WIjw3TROJRALRaBQ//elP8Xd/93ey9fOz7ltRFLlQSrx2XNdF +Pp/HxsbGM2f/plIpFItFOdtaBPU/ZMGMOCcikQgymQwWFxfxq1/9Cj/96U+/9/dFO23REeJZIf95 +Ec9HPB6HYRj4+OOP8etf//q579/iMYnnY29vD1999RWazeapY/s2v8eedNZZ6t8+FmIhRzweBwB8 +9NFH+NWvfoVyufxSx0/XdKia+krzq4PBIAqFAlZXV7G7u4tKpSIXiPV6PXS7XXS7XQQCAcTjcVy6 +dEmOO2m32zg4mHXlENccwzCwurqKX/ziFyiVSiiXy4hEIrKDzGQywfb29ht7XnRdRzAYRD6fR6FQ +QDQafSP3Te8++XnN907N+t7b28POzg4ePnyIv/3tb7JL0WAwkGMJ2u02PM9DqVTCQnkBUN/+BbVE +PwSDbyIiIiIiInqvybmUpil/XlYymcTCwgLa7TYqlQqazSb29/dlsGPbNiKRiJwdHo1GX2r7ov10 +NBpFv9+Xbc/FnOEXfUH/vC/xxZfXZ23x+X1Vd2KmqGVb6HQ6ODo6ki2Ao9EoisUiNjY2UCqWkMvn +EAlHYNmzNuTRaPQ77VJPtqIV83XF4z7L/opWjqKdq2jze5bHLipVxTG3LAuGYSCbzWJhYeF7nwcR +sBm6gWQyiUgkgl6vd6rV8Iv25U3Ma/c8D6PxSM6GNE1TtiSPRqPodruy/fx0On3uDGvbtmX7XN/3 +EQqFEAqFEA6HEQgE0Ov1ZGvis4YlYmZqIpGQM8lDoRASiQQajQaazSaOjo6wtbUlW9LXajXs7e0h +m82+cO6267ro9/tybrfv+wiHwygUCrh8+TKWl5dRKBQQj8dlq/FEInGma8VJiqpA8ZTvhFffBLWz +/RWtusPhMNrtNmzbhqZpSKfTL33+nZwJ+rpNJhM0Gg05z1vMY19YWMCVK1eQz+eRz+ehqrOAzLIt +WWn2JogZ6aJ6NBgMyhEDL/1cfd3aWZzHoq19KBSSVa/Pezyqos5ap3/9PIjFOKFQCJlM5juzxgEg +HAojFp9VLYvXj2jL7bouJpMJvOfMcRbnz0mapsn23e12W86pF8/Ty5xDYh78jzET+VUYhoFwOCwX +NjmOg2AwiLm5OSwuLsrn7fse03g8RjweRygUku+t4/H4lYLct4VcmPf1Yxfn3bfPCzGWQnSBEK30 +RRX2yxw/z/Ne6VoOfFPxvbq6inq9jtFoVsX99OlT3L9/H91uF4ZhIBaLyc4pYp65mGN8dHSE4XAo +P2eUSiVcuXIFhUIBuVwOoWAIlm1hMpnIBSRv6jkQP2LEyrOq6Ileh5MLLUOhkLzWh0IhVCoVHB4e +YjgcotvtwnEcPHnyBNvb20in00gmkkimkm/VQkyi14nBNxEREREREdErSiaTWFxcRL1eh6Zp6PV6 +ODw8xIMHDxAMBuF5HvL5PDKZDOLxuGzz/SKapslgr1qtYjQaodvtYjQazYKLrwOV5wXbqqLCV/3v +BNWe58mg+ixObu9kSOd5npwp6jgOhsMhWq2WbFEqZjMvLy8jHo8jGAx+Z1+/Tdd1hEIhBAIBOI6D +Xq+H0WgEx3ZevKNif0/M92y323LhwFnDPzHjVlShi0UI4ovFF1WPKoqCQDCAUCiEaDQqAxdVVb8O +sF48j/XHJo5xvV7HeDyWVYDJZBLJZFLOjhTP78nW0SdNJhNZce37PiKRCGKxGGKxmFw8INpyPivw ++z7iXBPz0cWXvMViEePxGOPxGIeHh/jjH/8of28wGODw8BArKysvvD8xL1iEy+I+5ufnsbS0hHw+ +/502zz90UcKzuhB8O6QyTVMuQPB9X8769n0fwWBQVue/aH75WY/3y5pOp+j3++j1etB1HZlMBsVi +EQsLC1hYWJCzocWilTexkOPb+9fpdNBqteC6rjyWZ5lvKl7rYqGDZVno9XqwbVvOUH+Z64B4/L7v +z8ZrRGPPnMdsGIa8H9G2WQR+4rXoui//fOq6LrtPHBwcoN/vYzgcnukcUhQFuvbyc8V/TGIhQjwe +x3Q6hW3bckGRCPlf5vkIBoPy+Pq+j+Fw+NLvEW/6PD7LsVHUZy9SE9dtVZm9NyYSCYRCodPHT335 +42dZ1isHZiL49jwP9+7dky2bt7e35WKrSCSCXC6HRCIx26evZ4Y7joN+v49WqwXf95FIJOQYmeXl +ZfnZQT72V/i8Q3RRKIoC3/cRj8ehaZocT7G+vo4//elP+POf/4xKpSIX7O3t7eHhw4dIJBJYWVlB +LB5j8E3vLAbfRERERERE9F4TwZCYm/eiL74VRYGmatB0DbHY7Eujg4MDGIYh54ZubW0hHo/D930U +i0Vks1k56/ZlnJy9ubOzg+l0il6vh1arhXq9jkg4gmBo9gW1CFNs24ZlWQBmrURPztJWFAW2bWMw +GGA4GmI6nZ5qvyt+53nBhqIqUPzTM3JFEDoejzGZTDCdzCqZe70eHMeBaZoy+EwkEjIAm0wmsjL4 +WSGoaKsbjUbheZ6cld7pzkJVTdOg6zocx5Fh6mg0gmVZsgJQBCCBQADHx8eYTqdydmin00EkEjn1 +OMR2RAWbCOpEwCKqHkejkQz6BoPBqQobALL6UMxDBWbBciAQkFVrou2k2BfP8+T2xW17vZ6sXD5L +pfv3nePT6RSj0UiGYOK/97qz0Pvw8BCj0Uh+cZrP55HL5dBsNmFZFsbjMbrdLtrttmzhLCr/xONp +NBrodDoIBAJyHnUmk5EB/2AwkNvodDqnnkvP89DpdDAcDmX1sJjNKwJWx3Fkh4ZsNotCoSAf49HR +Eer1OnZ2duSXvL1eD5PJ5KUq6y3Lwmg0gud5MnAWCwDEIpbRaDR7DQ2GMvx8FScrM79dmS2q4qfT +qaxmjMfjUBQFk8kEg8FAhpcipDr5unVdF7Zty8fsud+0/H/dxOt5MpnIgDgWiyEej8tZy+LcEXNI +RXD/Oojr3mg0mi3a0GcdGsQ52Wq1UK1WUa1WT80nPsusatFqP5fLyWuSCEnF61RcawTxniLCRuCb +tuGmab70/YsRHCcDSnEdSyQS8twR9yUWCZ28Fop24KILhuje0Ov10O8PEAoGEQwFT4Ufon2ueN3K +lu3q658Tf1YnZ5WL17Z4TYjHHwwGTy2yetbzcXKGuaZpcnGZuC6rqiqvQScX7Uyn01PH5m0jFtSI +BQ0n33Mnkwks25LnRDgcPhX69/rPPn7Pe19znFd7bwoEAsjlcvIzgud56Pf72N/fl4s8RKCdSqVO +ddbwXE9eqwHIqup4PC4rq8XnucFgiH5/9l7tuu5b07GA6HUS18RwOIx0Ko1isSg7HB0eHqLdbsvF +IsfHx3j8+LHsQLW8vHzeu0/0o2HwTURERERERO81EZA1Gg3ZTvP7iC9rRYWFCHsymYwM+UR7wXw+ +j9XVVeTz+VNfJL9IIBCQVaY7OzvQdR39fh8PHz7EP//zP6NUKqFUKiEWi2E6ncKyLBnwmKaJK1eu +YPPqJgKBgGzl2m638fDhQ9nuezKZyCBbtAB+VjB/MhwX8141TZPbEy27raklw7rpdCrb6lYqFWxv +b8M0TSiKgl6vh729PVQqFdy6dUvOQBdisRgWFhYwHo/x6NEjjEYj7O/v48svv4TnebJtvAiBtre3 +8ejRI9TrdfmluWmayGazyOfzODo6gud5aLVaePz4Mf7jP/4D2WxWBtq9Xg+dTge3b99GpVJBu91G +MpmE7/tyXwaDAR4+fAhVVVGr1fDXv/4Vo9FIzhM1DAOqqso2rJ1OR1ZFi2rx5eVl7O/vw/d9tFot +PHr0CP/2b/+GXC6HeDwOXdfRbrfR7Xbx+eef4+nTp2i1Wkgmkz84ZOn3+9jb28ODBw+Qy+WQz+Ux +taYYDobY29/D3bt38fDhQ9mafn5+HouLi1hcXES73YZhGOj1enj69Cn+8Ic/YGFhAYVCAeFwWC4C +uHfvHra3t1Gr1bC+vo6NjQ1kMhmk02l5/CaTCQ4PD/Hll19CURQkEgnE43HZinN3dxf37t3D8fEx +stkscrmcrFISiylENX8qlUI6nYaiqPB9D7VqDa1WS1Yfixa54XD4hRVNmjp7HadSKfklsThm29vb +8rU+Ho+xu7uLvb09fPHFF2i1Wq/8nKiaKudwh0IhuK6LarUqX6OqMusKEA6Hkclk5ONoNBr4/PPP +4bquPP/E4gvHcdBpd9DuzGZ5inmzyWQSyUTyB51DzyLaEcdiMVlZfXBwgN3dXRQKBVnRXqvVsLu7 +i62tLWxvb8sFOj+UeF1vbW0BgKwQHQwGGA6HuH37Nu7du4fd3V3Mzc1hdXUVy8vLZ2o9LBZZ2LaN +g4MDRCIROI6Dp0+f4l//9V9RqVQwPz+PQr4gZ6mL7gH9fl8uKMpmsyiXy6cWa7zoftPpNJaXl3F0 +dCSvnU+ePMGf/vQnHB0dyVbdIqz96quvsLW1hUajgXg8DsdxEA6HUSwWEY1Gsb29DcMw5O9qmob5 ++XmUSiVEo1F5vWq322i323KhiWEYKJfLz53//CaJNu3Ly8tQFAX379+H53nY29vDv/3bv+H4+Bil +UgnFuaJ8Pkaj2SiHXq8nnw+xKGBlZQVPnjyBbduo1Wq4e/cuUqmUDFIdx0Gn00GtVsOtW7ewt7cH +RVGQSqXO+1B8hwi9xft0KBTCZDJBpVKRM6/F+5G4Por2/fv7+/jDH/6Aer0+OyeKJXn8xuOxfG8S +x090QTjLIhJBU2fjTCKRCJLJJDKZDCaTCbrdLh49eoRcLoe5uTnMz8/LGeTytvqsqjWVSqHf76PT +6UDXdXl9EdecTqeDSqWCSqWC+/fvYzgcfmdbRO8aRVXkZ+2FhQXcvHkTjuNga2vr68VOfezs7MjP +tm/rAh6i14HBNxEREREREb3XRPAt2pW/SCQSgaKoiMViMAxDtobOZDLI5XLodDqoVqsYj8coFotY +W1vD3Nzcmb4gFm1YS8USvvjiC+i6LgMe13Vx9epVWJaFTCaD0WiE4XCIhw8f4tGjR4hEIggEAti8 +uilbQmuahk6ng8ePHwOYBeue58kqRNHG93kV6eLLblGtqGkaut3uqe2J9t0i+O50OphMJjI8FDNi +G40Gbt26hTt37mB/fx/9fv9UECWqbIfDIUzTxGg0wuHhIW7duoXpdIpisYhSqYRqtYrDw0M5s7DR +aCASiZyaw722toY7d+7AdV20221sb28jFovJhQO6ruPg4ABHR0e4e/cuDg4OMJ1OUSgUZPvIxcVF +jMdjJJNJKIqCer2Ozz//HI1GA5ubm3J2sKga3NvbkwG3ps26AiwtLWFlZQV3796F7/vodrvY2tpC +MBjE/Pw85ubmYBgGDg4OcHBwgPv372N/f19Waf/QCtnhcIi9vT0kk8lZ2/iv28U2Gg05V3V7exuX +Ll1CoVDAzZs3MT8/j3K5jCdPnsA0TYzHY+zs7CAYDKLZbGJjYwOpVAq1Wg21Wg0PHz7E06dPMRgM +cP36dVy9ehWFQgGxWAzValWGMCefy0KhgFKphGaziYODA+zs7MhFDKI6dTqdYn9/H0dHR3KhRiQS +weLiIhzHkVWNtfo3wXcikZDhSCgUemHwrWqqDGE0TcNwOJTH7MmTJ7K6tt/v48svv8SdO3ewt7cn +OxCc1cnFJN8OvkV4IxaifDv4brVauHXrFlqtFjY3N+E4juw8Yds2KpUK9vf34TgOVHX2uNbW1hAO +h3/QOfQsonJUhKytVgtHR0fY3d1FNpuVVadbW1u4ffs2Hjx4gKOjo9c2b1xUdW9vbwMAxuMx0um0 +nPsuFnQcHh5ibm4Oa2trWF1dRTKZfOlOCrquI5vNIhwOY2dnR86W3tvbkwuOrl27Juevi2vc/v4+ +arWaPAbLy8uIRCIvH3yr2tcLOxRsb29DVVUMBgPs7Ozgr3/9K2q1GorFIhKJBPb393F4eIhHjx5h +Z2cHzWYT+XweruvO3kdKJeTzefztb3+Druvodru4e/cu2u02rl27hslkgnQ6LavlxTkkFjKJyuBc +Lvfaz6GzUlUVqVQKa2trcF0X0WgUrutif39fhtfXrl2TFb5ikdbe3p68hqiqilwuh6WlJaytreE/ +//M/Yds2Go0G7t+/D1VVUSgUUCwWMZ1OcXBwgL29Pdy7dw9HR0cIh8OYTCbnfSi+Q7xPi3EhwWBQ +Bt+BQAC6riMYDMJ1XcTjcaRSKdmp4PDwEH/+85/l8ROdSMR7/d7eHg4PD08dv7W1NSwsLJz9OdRU +BNSAfC5zuRwajQb6/T6Oj48RjUaRy+WwsLDwnUUquq7LTgiVSkXOLhafM8Q+Hx0d4datW7h37x4O +Dg4wGo0YfNM7TywMNE0Ti4uLcjxNr9fD7u4uBoMB9vf3EQqF8MEHH8jOIG/DGAui143BNxERERER +Eb3XRCA8HA5lBer3mZubw4cffohcLgtN1aBqs2Agm83KCoperyfnT66uriKXy51qhfsiopI8HApj +bW0NN2/exO7uLlRVRb1eh6qq6Pf7CIfDsq1zq9VCp9OBaZqyiiOTyWBtbU1ud2dnB+PxGPV6HXNz +cygUCpibm5OtEl9UlS62J4LY3d1dTCYTNBoNhMNhjEYjRKNRWQXqOA6Ojo7wxz/+EYFAAIZhwLIs +dLtdhEIhWcF7kggCi8UiVldXcenSJQQCAbRaLRk8PH36FJPJBOPxGL1eD67ryuBezLSNRqPIZDJY +WFjA2tqaDHVEReTe3p4MdEejkWzXLLYhKtzFXOBLly7hww8/lHPWj4+PAUBWRIuWw91uF71eTwYL +ojIuk5lVfJ58PkRV9/7+vgz5xTxtEYq+aP7uy5hOp2g2m9jZ2UGj0cDjx4/l3HMx11oEDSsrK1hZ +WUEmk5Ez2q9evSoXPhweHmI6naJeryMcDssWwO12Wy4AWVlZweLiIjKZjDynVlZWcOnSJQBAr9fD +gwcPcHx8jN3dXdmuWsweFi2hxWMXz4Voyes4Dmq1Gra2tqAoimyJX6/XEQwGkc1msby8jLW1NWSz +2ReG0yLcXF1dRbPZxNHRESzLQq1Ww1/+8hc5pkBUGIv58T90NmYgEECxWMTGxgYGgwFGoxEeP36M +fr+Pw8ND5HI5ZLNZKIqCYrGI69evYzgczoL+r0PVbrcrzxPRJr/X68ngXwReP8YX29HobBFJr9dD +tVrFwcEBLMvC1taWnBdvGIYcIxAKhV7rvog2zcfHx3BdF4eHhzBNU1bsiwUUxWIRS0tLWF9fx/z8 +PCKRCLrd7kvdh6IoCJiz41sqlbC5uSlfo+PxGJVKBbZtY39/X7aYFtelyWSCVCol2zWf5XwR7ysA +UCqVsLa2hna7LV+Dg8EABwcHiEaj8rohFh4Fg0F5nEUImk6nsbKygps3b6JarcJ1XTQaDTx8+FCe +L2L/RDcN0VkhmUz+aOfQWYnrUCQSQalUwpUrV9DpdOA4jlwkI953xPMxmUxkG3TxfIjrcjabxeLi +ItbX19FutzGdTvHo0SNUq1VUKhUoiiJHHHieJ0dFvKgzzHmKRqNYWFjA5cuXoaoqms2mrFyvVCrI +ZrPIZrMIBAJYXl7GZDKR7e0PDg5OXV/EQjbR2UMcP8MwXrl1uAzotVk3ilKphEajIccmBAIBzM3N +oVwuf+fzWDAYRHFu1spZjOiwLAuVSgV/+MMf5DVHjIUIh8OyApbofSBeX6lUCsvLy+h2u7h3754c +2TIYDNBqtdButzEYDGYLYA0Tms5RAPRueXvfpYmIiIiIiIjegFarBdu2sbe3B8MwTs2TfJaVlRVE +o1HcvHlz9oWqqiAYDKJQKGB1dRXD4RD1el22215dXUUmkz1Tq3MR9IUjYVy+fFm2KtzZ2UGlUsHT +p0/x5MkTOd8bmFWiixbsoVAIiqqgVCzh5s2b0HUdOzs72N/fl+2U8/k8Njc3AQCh4CwYeZFisYib +N29C0zTZ8rlareLBgwdIJBIol8uyTbmo+j4+PsbBwYEMC6PRKNLpNC5duiRD+JNM00QkHMHc3Jys +Rjw4OECtVpNV+aLqKx6Py4prMVdYtKsXsz/X1tbwySefIB6P4/j4GPv7+zg4ODj1O6LteSQSAQCE +w2F5P7qmI56I44MPPoBpmtje3saTJ0/QbDbx8OFD3L9/H8A3bWZFtY3YdqFQQDqdRjwew+rqKn72 +s58hGo3i6OhItqcXt0smk0gkEggEAkgkEnIO7esIvkXVngjIRNAh2oJvbm7ixo0b2NjYwMLCgqwY +XFpawi9+8QukUins7u5if38f9XpdPm7x2OPxOJaWllAul7G5uYlyuYxoNCrn0G9ubmI0GqFSqeD4 ++Bg7OzuyelDMgdd1HZFIBKlUSj6Xomo+lUphf38f+/v7OD4+lnN7gVkAqqoqDMNAMpnE0tISrl69 +imvXriGVSkHXXjy+YG5uDoFAAN1uV3ZsaDabqNVqs/nGX7fmTafTWFpakhWiP6TyMxgMYnl5GR9/ +/DG2trZQqVRwdHSE/f19GUpdu3YNCwsLWFxcRKFQkFXo1WpVtvk/ef6JRQOiNXWhUJDn9+sWj8+6 +GYgRAMfHx+h0OrILg5h9PVv4MVs0I6o6XwfXdeXz1O125Wx0cW6LavlCoYBr165hfX1dXjPOEnyr +mgodOubn5/Hzn/8ciUQCT548kdXVtVpNtpYXYbNoA59KpZDNZpFOp8/0HgBALuBZWFjAT37yE1nJ +enx8LINJUXUvZtGL2eqRSOTUwpFQKISrV64CALa2tvDkyRPZZUG8l4j9F4uURGeMYrEo2/2/DcRj +mpubw89+9jPEYjE8efJELiSq1+vy+QC+ma8eDoeRSCSQTqfldTmZTGJjYwO9Xg/b29s4OjpCpVKR +16ZIJIJEIoFIJIJQKCTHJ5xlIdublkgksLGxgeFwKD8zHB8f4+nTpwiFQrhy5QquXbuGRCKBq1ev +YmlpSR6/TqeDZrOJW7dunTqfxXuaWNCVzWYRCUd+8L4mk0ksLCygXq+j3W5D0zTE43EsLCw8M/iO +RCIozc8jGAqiVqvh6OgIh4eHsio/EAjI10E6ncbly5cxHA5RrVbP+2kheqNisRjm5+fRarWQyWQQ +CoUwHo9h2zYGg4Ec82MYBtSYyuCb3jkMvomIiIiIiOi9oSgKfMwqSpPJJIrFIgDIqumXEQwG0el0 +4HmerKwOBoOYm5vD5cuX0ev10Gq1EIvFUCwWsbi4OAtDje8G6qI6L5lMYm5uDslkUlZFii9wV1ZW +kEwmkcvloCgKWq2WrNYQrWhFVV8ymUShUEA0GgUA5At5XLt2DQAwGAywt7eH4XCITqcD27YxNzc3 ++yLMsV+qnXYul5PbE62gT26vXC5jbm5OBj+Hh4c4PDyU89ODwSBUVcXy8rKscm02mzI4nVVrmQiF +Q9B0DVevXkUwGMTnn38uq4pt25azZ0XYEw6HEQ6HEQqFZMAu/rtoiR0MBvHFF1+gVqvJik3xe6FQ +SFZ2+76PYDAoA3Td0BGPx3H58mXk83nZTn00GsmWyuLYmaYpZ0+Hw2HkcjmUSiWk02lEIhEsLCzA +cRyEQiF8/vnncl8sy5KhggiYxOxVEVyGQqEf9MWk7/uwLAudTgftdhsAZMiQzWZx/fp1XLt2DSsr +K6faMYuW8CKIr1arsqpY7LdY+HH58mVcu3ZNHiuxiERRFGxsbMjW+2LmrvgSVlRci3byokI1k8kg +Ho/P5pLn8xgMBrJrgNgHEXCFQiEZ9q6trWF9fR0rKyvQVO2Fx80wDGTSGSQSCTQaDRwdHcmK2kaj +IVuPe56HcrmM9fV1eJ4nA9d4PH6qOl+E9el0GqlUCtFoVFZInlzEEAgEZJcIMftdhO62bSMUCmFx +cRGGYWBubg7ZbBaPHj2C4zgYDAZoNBpotVpyAYCY5y2+5M5kMpibm5OLCERwL6rzRSXkqy6siEQi +8rUiFqdsb2/j8PAQnU5HvrZisRhyuRwymQxarRYajYY8Lrquv3IoLxb++L6P0WiEVquFfr8vW1wn +EgksLi7i+vXruHLlCsrlMkzTxGAwkIGxeI5mQfGzq5oVRYGma8jlcjAMA/lcXnYgEAt8er3eqeOS +SqWQSCTk4pdsNnumcRei+4SoWL9x44a8FrZaLQyHQ9i2DWC2UEcEs7quyzEbJ0PaUDCEldUVJFNJ +Gf6Lqvhmsym3JdpPi24B6XRabku8LyWTSRkai/sUs5XFAgzxfiR+LxAIQNd0aLp2ajyIOA/E7b99 +DMR1UbSSF4/VNE3EYjFomoZ8Po9gMCg7d3S73VMLG8T1JBaLIRqNolAooFAoIJlMIhKJYHV1VV5D +LMtCvV7HaDSCbdvwPE8eDwDyepfL5RCNRuV72ss8nyePi7jeBYNBeVx0T0csFkMmk0EsFpOLF1Tt +9LE5eVwmk4l8v5gt2NIRDs86xYguJNVqFZ1OB4PBAL7vI5PJYDweY35+Xs68F6HYeDxGo9FAp9OR +9xcMBuV7WjQaRT6fRz6fRzQ2e/2Ka10wGEQkEpk9l+qLryuKeroqtdvtYjAYoFQqYX5+/pmdcoLB +IPL5HFKppFzI5zgODg8PZcePYDAoF0FdvXoV7XYbjUZDPv8nu7qI50N8/hJjS1ghThddKBSS3ZPE +Z1DLsuQIHfH/KmKR49u8mIfoVTD4JiIiIiIioveKoiqYm5vDb3/7WxkQn4VodS5aHcvQbb4svwRe +W1tDMBiUv6dpGhT19BepouJ4cXERv/vd75BKpWRVYiwWk19CiS+4l5aW4HkeCoWCbIts27bch3Q6 +jWw2K6v0RJhcni/LIG59fR2O48jZqMvLy1hdWUUun3upUCYej2O+NA9VVRGNRrG+vg7btuG6rpy7 +vLy8fKqaRAStJ+ehi32MxWJYXFyEoii4efMmstkcotGI/CI/n8/L6vdyuYxWqwXXdeG6LhKJBFKp +FHq9Hv72t7/h6OhIBgIiGAGAeCyOcrkMwzCQSqVw6dIlGWiI/5ZKpfDll1/KFsmGYcggSdM06Jr+ +9Wx3BVeuXEEoFMLly5fl74vgW1SiR6NRFItFlMvlWUAQjcrnY35+XgbJ6+vrsCwLjuPMwtdMBplM +Rn4xCUBWgZdKJcRjrzajNJVKYXNzE+vr65hOp7AsS4ZU4XBYzk0Xz8lJompWPMZisYh+vy8XD4gW +83Nzc5ifn0exWJy1F1e/CZsNw5ALN0R770ajIc9FEYKJdvnD4RDRaBSJRGIWHkZjiMVjcl+uX78u +QxoRfJumKY/fwsICstmcHEXwwmuCokBRFejQUSqV8PHHHyOfz8vz99vtlUulEgqFAsrlMsbjMa5f +v45SqSRDlXg8jl/84hdyFu7m5iaKxaIM6gRN05DJZGQolsvlZBtq13VRKBSwvLyMcrmMSCSCSCQq +z9uVlRXZZl7MqxbzzyORCIrFIubn51EoFGZfapsBbG5uyrnUs3ENOXmOvwoR8ofDYVy6dAmapsnW +08PhUHbQENcl0aVhY2MDc3NzuHHjhgxPX2UfTNNEqVSSXRSm06m8JqqqinQ6fapqWVyzRXD5y1/+ +Ui5QuXbtGqLRCAKBwHeu1SfvLxaLwfd9fPDBB0gmk2i1WvJ6LIhrUCKRwPz8PBYWFpBOp88UfJ8k +zjtDn12XlpeXMRwO5YIHce0XCwBs20YikZCBXjqdhqp9s8hqdXUVmqZhYWFBtrAW2xJVzmIe+fz8 +vLz26LqOxcVF/MM//AMWFxdx5coVXL58WZ5H4rUuzof/9t/+Gy5fvoybN29icXERiWQCmqYhGAzi +5z//uVx8cOPGjVMdHgRd17G8vAzXdTEcDmVXh5WVFYTDYfl+C0Buo9lsYjAYYDgcyu2Ibg3x+Oy9 +YH5+HplMVp53qVRKvq6y2Sxu3LgBx3FOdQ1IJpMYjUYYjUYyVE2lUiiXyy9VyS/ej9bW1vCP//iP +WF5exvXr17G0tCSPi+M4+NnPfiYXCH344YeIRqPyM4S8bqgaFhcX8Ytf/AL9fl8ufpt1o5mNZRDv +nZqmYX5+Xo4E8X0fy8vLWF5eluF9KBTCtWvXEIvFcP36dfT7/VPHT1z7xPvX/Pw8UqmUDIl/9rOf +yarwjz76SH5+edFrWlEUZLNZXLp0CaFQCPPz8/jJT36Cjz76CKVSaXY+qN/dhugisry8DMdxsLy8 +jHa7jW63K685YmFjsViUCwiDwSBu3rwp59mL8+sf/uEfkMvlcOPGDayursqFTEQXmaIo8H1fdh/J +ZDKwbVt+/h0MBqhWq7PXdiwOnP1/h4jeagy+iYiIiIiI6L0hwr5isYj/77/+f/j000/PvA3xJfDJ +atZwOIzyQhn5Qh5Xr1zF1JpCURREIl8HKc+oqhRVR0tLS8hlc/j1r3/9TTWbYcrwRVSLizD1448/ +hud5cF0Pvu+d2p6hG9ANXd5nPBZHOBxGvpDHlStXMJ1a8jaapiFgBmAGTBlKv8jLbs91XdmK2nVd +2Lb99TFQoeuaDEs3Lm9gak0BQFZsnwwr8/k8MpkMlpaWYFkWbFu0tvZkxdb29jb29/fx2WefnQq+ +DcOAqqhIJBIIR2bh7tWrV7/+0s+H73tQFBWGMWtnPhqNcP/+fXQ6HRmoyGpGTZXVrWJhg2VZp9pt +i3NBhA0nZ8Fqqnbq+Zibm8Pm1U1Mrekz98Xzv+kmIKoBTdN8ZteAl5HJZHDz5k383d/9nazuEVWK +Yl/Fc/Lt+xDVcyL0+OSTT74+/9wT7ZFVmKYh22yLY3byNSPCo8XFRXz88cewbUeeO+K51FQNrjc7 +d0SYI1oO6/qs1fT1a9dh2RZ83//OPui6BkM3YAZm+/HtSsnvo6oqfMVHqVRCJpPBhx9+KFtmz55b +FZqmwjRMGKYhZwq7rodwODQL4rTZ49Y1Hb/85S9x48YN2ekgGAzK8+DkazadTsuFDR999JE8LmLW +uWidL8LcVCqJlZUVWJYlA3KxKEGcf6IFtnhdK4oCTdVw/fp1rK6uwrYdRCJhGR6+anWjuF00GsXG +xgaWl5flQhjHcaCqGlT1m9bfmqZhc3MT0+kUpmnKc1Hs81npuo5yuYxPP/0Uy8vLCIVC8vGK43vy +OIhrtlhM85vf/AYff/xTGMasSjYUDH3vOWOaJvSvK2rFYpLZY3Xhee6Jc0mDpqnyfsRr4lUXGIj3 +m2w2i/VL6/K8E68fce0HIF8/ItgTj1+8T4lxE4uLi6euYc86hwzdgPH161qc24uLi0in05hMJrKr +hq7r8jwS93f58mXMz89jMpkgEo4gHAnLffSiHn71q1/hgw8+hK7PFk6EQ2EoqnLqPNB1HaurqyiV +SvDc2WMVo0XEe5ymanJe99UrV2E79ks9H+L4iAp3cX27efMmLGtW/e773xxHXdfl+5po/31yey96 +nYj3+0uXLqFUKmE8Hn/3uPgePv30U3zwwQdQVfW5x0XVZl1T5ubmTh2XQCCAgDlbuFEoFOR753Q6 +PfU+JbqiGIYhF87FYjFc2bgC27Gf+b4mjpW4JomqaFVR8fOf/xw3btyQ51gkHHm5iu+vg+9EIoGV +lRW5KCsajcrZ3M+6jTieKysrKBaL8looOsGI91Lxup8t+LKgabOK+3A4DFWZHc/gymwRzG9+85tT +n79eZsEU0dtOvFZEZ5/BYIDBYCCD73q9jmw2C8t+uY5XRBcJg28iIiIiIiJ674gvftOZF8+1fhli +RvNZqoTEF7giFHvevqiqCqiApmtnnhGr6bM2z4FA4DuVvK/idW/v+ypMptMp2u022u02TMNEIPjN +F/WaqmE4GsoZ4p1OB9PpVFbv5fN5xGIxKKqCXq8n28J/0wZ1Fuy4riurAxuNBobDoZyrPTc3J9v9 +nmzhKyoaf+jxe5NEJbpooRuLxV46aBSBq6hCfBUiEAPwytsAcObz/1X286yv42fRdE1Wq7/o/kQ4 +87KPTcwqfhWiG8HrPmYi8HqZ/Xot140T9x0MBpFIJJDNZhGLxV5qH8T5KCpDz3J/mq5Bgyarv9+E +V3l/ed52fuhrWVxDv+8Yveh80PDyr48X3d/J5+NVnfU1+CrEcRHth3/s4yLO75e9Vpim+coVny+z +z88jjv1Zr2kvcwyE77vmvcrnKqKLRLT0F4s5NU2D53mwLOtU9x6idw2DbyIiIiIiIiJ664xGI+zs +7ODRo0eIxWKnZgIbhoFqtYqjoyM8fvwYBwcHmEwm0HUduVwO8/PziMfiUBQF9Xodjx8/RqfTkW3N +RQXmdDLF0fERjo+P8fjxY7RaLQCQFZHZbI4tT4mIiIjowjk5qkV0j/E8D7ZtYzKZYDqdyg5DRO8S +Bt9ERERERERE9NaZTCY4ODjA3bt3ZbVyNBqVrXQPDw9xcHCA/f19DAYDhEIhWe2dy+VgmiYURUGr +1cLjx49RrVaRTqeRyWRkNdx4PMb+/j4ODg5wfHwM3/flLMRisYhEIoGAyeCbiIiIiC4WTZt1GRLj +AxRFkeNqJpOJnPlN9K5h8E1EREREREREbx3LstDtdnF8fAzbns1cPTlD27IsWNZsLuHS0hI2Njbw +k5/8BMViUc6kBYDxeIxWq4WdnR3s7OwAmLX81TQNvu9/PTvcRjQaxYcffoj5+XlsbGx8M/9YfbX5 +x0RERERE50V8bhadjlRVhe/7cF0XlmXBcRwG3/ROYvBNRERERERERG8dx3bQ7XZRq9XQaDTQ6XQw +mUygKLMgOhKJIBaLoVgs4sqVK/jJT36CpaUlGXwLo9EIzWYTe3t76HQ66Ha78u8Mw0A0GkUsFsPV +q1fxwQcf4Pr161goL3wTfCsMvomIiIjoYlEURY4I0jRNfqZ1HEcu/PQ9/7x3k+i1Y/BNRERERERE +RG+dcCSMxcVFfPDBB2g2m2i325hMJgBmX+RFIhHE43EUi0Vsbm5iY2MDqVQKkUjkVFidyWRw+fJl +KIqCTqeDTqcD3/ehKAoMw0AsFkM8Hsfly5exubmJpaUlJJPJCxt667qOQCCAXC6Hmzdvot1u4+bN +myiXywgGgzAM40I+Lnq/Kersy/tgMIilpSV88sknyOVyuHbtGtLpNAKBADRNO+/dJCIiequIqu+T +n/1834fnebM/fVZ807uHwTcRERERERERvXVisRhu3ryJfD6P8XiMyWQCx3Hk3wcCAZimiUgkgkwm +g1w2h2AoCNMwT21neXkZpmni2rVrmE6nmE6n8P1ZdYto/2iaJjKZDDKZzGyud+DizvU2dANKWMH8 +/Dx+97vfYW1tDfl8HouLi4hGo9/58pPoIlAURYbbm5ubiEQi6HQ6WFxcxFxhDqFQSI43ICIiohcT +n4eJ3jX8REhEREREREREb51oNIpLly7h0qVLP2g7hUIBhULhvB/OG6PpGjRdQz6fRz6fx09/+tPz +3iWiH0x0aDAMA8vLy1heXj7vXSIiInrrKYoifwTf9xl60ztNPe8dICIiIiIiIiIiIiIiIqLXR4Tc +J4PubwfhRO8aBt9ERERERERERERERERE75hnVXcz/KZ3GVudExERERERERERERERvQN834fjOHBd +F5qmQdd1hpzvId/34boubNuG4zgyANc0DYZhQNM0aJp23rtJ9Nqx4puIiIiIiIiIiIiIiOgd4Lke +bNvGeDyGbdvwXO+8d4nOgVgAYVkWXNeF7/tQFAW6rsM0TRiGwQUR9E5ixTcREREREREREREREdEF +I+Y3u64rQ87RaIR+v49+v49sNotcLoewHj7vXaU3zPN82LaN6XQK27Zl8K1pmgy+VZW1sfTuYfBN +RERERERERERERER0wXiuh/FkjMlkgv39fRweHuLw8BDVahXVahWffPIJfv3rX2N5efm8d5XeMM9z +YVkWhsOhrPoWFd+hUAimaTL4pncSg28iIiIiIiIiIiIiIqILxnGdWYV3r4+trS3cunULDx8+RKVS +wcHBAXzfx8blDQbf7yHf9zGdTjEcDjGdTuF5npzvHQwGvw6+OeOb3j0MvomIiIiIiIiIiIiIiN5i +ruNiak0xGU/QbrfRarXQarfQaDTQarWwvb2NJ0+e4ODgAPV6HdVqFd1uF5Ztnfeu0zlwHAf9fh+t +VgvD4RCO48AwDIRCIaTTacRiMZimcd67SfTaMfgmIiIiIiIiIiIiIiJ6i9mOjW63i06ngwcPHuDR +o0eoVCqo1WpoNBro9Xro9/sYDocYj8dwHAee5533btMb5vs+AMCyLPT7fTSbTQyHQ9i2jWg0img0 +ikKhgGQyCdM0z3t3iV47Bt9ERERERERERERERERvCd/3ZXBtWRam0yl6vR6Oj49Rq9Xw5Zdf4tat +W9jZ2UGz2USz2YTv+3BdV/7peR5835dBKL0fHMeB4zgYDofodrtot9sYj8fwPA+6riMWiyGbzSIe +jzP4pncSg28iIiIiIiIiIiIiIqK3hGVZsoK7Uqlgb28PR0dHaDQaqNfrODw8xPHxMdrtNkajERzH +kSG3CLzp/TQYDNBqtrC7u4tms4nJZALf9xEMBhGNRhGPx5FKpRCJRKBpnPFN7x4G30RERERERERE +RERERG8Jy7LQ6XRwfHyML774Al999ZWs7u52u5hOp7AsC5ZlwbZtOI4jb8vQ+/3W7/dxcHiASqWC +ZrOJ6XQKRVFgmiZisRji8TiSySQi4QhUTT3v3SV67Rh8ExERERERERERERERnQPLsjAajTAcjtBs +NtBut2Vld61Ww6NHj7C1tYVarYbhcIjBYCDboLuuC4Bh9/tOtMZ3XReHh4e4c+cO7ty5g3q9jul0 +ilQqhUKhgIWFBWSzWQQCAWg6q73p3cTgm4iIiIiIiIiIiIiI6A0SYfVkMkGtVkO1WsX9+/fx8OFD +HBwcyFbnrVYL3W4Xo9EItm3DsizO76ZTPNfDdDrFZDLB3t4evvrqK9y6dQu1Wg2u6yIWi2FxcREb +GxsoFApQVVZ607uLwTcREREREREREREREdGPyPd9eK4H25m1JremFqbWFI1GA3t7e6hUKvj888/x +1Vdf4ejoCNPpVLY0t20brusy6KZTXMeF4zoYj8dot9tot9t4/Pgx7t+/j93dXdi2DcMwkEwmsbq6 +io2NDeTzec72pncag28iIiIiIiIiIiIiIqIfiWhLPhgMUK/XUa/XcXBwgP39fdTrdbRaLbRaLRwc +HKBer2M0Gs3CccuSbc0ZepMgzoVev4dGo4GjoyM8fvwY29vbuHv3Lo6Pj+E4DkzTRDQaxcLCAi5f +vozLly8jm81CVVjxTe8uBt9EREREREREREREREQ/Etd14TgOer0eKpUKtre3cevWLdy9exfValVW +dk+nU4zHY0yn01NhN0Nv+jbf99HtdnFwcID79+/jL3/5C7744gu0Wi30+334vo9YLIZ0Oi3bnF++ +fBmBQACqxuCb3l0MvomIiIiIiIiIiIiIiF6R7/vwPA+2baNer2MymUBVVUwmE0ynUwyHQwyHQzSb +TRwcHODw8BBPnjxBpVJBp9ORt3UcR1Z4vw69Xg+7u7vIZDLnfYjoDMT8ds/z5I9jO3BcB7ZtYzwe +YzKZ4OjoCIeHh9jZ2cHjx49xfHwM13Wh6zqi0SiWl5dx6dIlbG5uolQqIRqNQlEUKIpy3g+R6EfD +4JuIiIiIiIiIiIiIiOgV+b4vW5MfHBxgOp1iMBig3W6j0+mg1Wqh3W6j3+9jOBxiPB6j1+uh3+9j +MpnI+d2ve453p9PBw4cPX1uQTm+GDLudWdBtWRYmkwkmkwmGw6E8r3q9HrrdLnq9Hnq9HobDIUKh +EKLRKEqlEj744AN8+umnWFtbQzabZeBN7wUG30RERERERERERERERK9IBN/T6RRHR0c4OjpCtVo9 +Nb+70+lgOBzK3xdzv3/MNuatVgsPHz5Er9c770NEZyDODRF6W5aF0WiE0WiEwWAgzyfRHt91XRiG +AcMwkEgkUCgUcPnyZXz44Yf45S9/iUQigVAoxOCb3gsMvomIiIiIiIiIiIiIiF6RqM61LAvHx8e4 +f/8+jo+PMR6PMRqN5Nxu13VP3ebHJiq+Dw8Pz/sQ0RmcbHMu5sOfrP4ejUaYTCZQFAXBYBDBYBDp +dBq5XA4LCwtYWVnB2toaLl26hFgshkAgAE3TzvthEb0RDL6JiIiIiIiIiIiIiIhe0cmK7+PjY2xt +baFarcJ1XRmKf7u6+8es9BYajQYGg8EzK30VRfneffix/v687vei7hfwTRB+slOA4zgy1E6n07h0 +6RKuXr2KjY0NXLt2DYuLiwgGgwiHwlA1ldXe9N5g8E1ERERERERERERERPSKFEWBqqrQNA3hcBiJ +RALj8Ri2bcO2bTiOg8lkAsdxAMyCzBcFoa9DOBxGOBxGIBA470NEZ6QoijyvFEWBpmlQFAW6rsM0 +TRiGgWg0ing8jlwuh+XlZaytrWFpcQkLCwvIZDJyG0TvEwbfREREREREREREREREr0gEk4ZhyBAy +GAxiMBhgMBjI2d7j8RjANxW84p9/LPF4HOVyGYlE4rwPEZ2BqqpQVRW6rkPXdRiGIcNusbBC/CST +yVN/xuNxxGIxht703mLwTURERERERERERERE9IpE8B0IBJDP56EoCrLZLJrNJjqdDtrtNjqdDkaj +EVzXheu6sG0bk8kEtm0D+CYAf51BeDKZxPr6Osrl8nkfIjoD0T1ABN6maSIUCiEQCMgK71wuh2Qy +iXg8jlAodN67TPTWYPBNRERERERERERERET0ihRFgWEYCAQCWFxcxMLCAmzbxnA4lFXf/X4f3W4X +zWYT7XYb1WoV9Xodo9EIjuPAcRw5D/x1hd+ZTAbXrl3D1atXz/sQ0RmIim9R9W0YxuxP3UAgGEAs +FkMkEkEoFIKuM+YjOomvCCIiIiIiIiIiIiIiolckZi8Hg0Hk83kUCgVEIhFZ3W1ZFqyphWqtiq2t +LTx58gS3b9+W879Fhe90On2ts78zmQw++OAD/OIXv/jO34k548/zY/39ed3vRdwv8e/fnvctQnEx +95uIvsHgm4iIiIiIiIiIiIiI6BWJYFJRFASDQcRisVNztV3HheM6iEQjME0TsVgMoVAImUwG1WpV +VoW32210u13ZAt1xHLiuC9/3XykMN00TkUiEM76J6L3B4JuIiIiIiIiIiIiIiOhHomoqdOiIx+NY +WFhAKplCeb6Mjz76CLVaDdVqFdVqFQ8fPsT29jaazSYsy8J0OoXjON+ZA05ERM/G4JuIiIiIiIiI +iIiIiOhHoigKNF1DWA8jHA6jUChg1V8FAHQ6Hezu7qJSqSAUCgGYVWqPRiOMRiNMp1NMp1NMJhM5 +AxxgCE5E9CwMvomIiIiIiIiIiIiIiN4gMcs7GAwil8tD0zTouo5SqYTj42O02220Wi0cHBzg6OgI +nU4Hk8kEo9EInuf9oBboRETvKgbfREREREREREREREREb5iiKAgEAsjlskinU1hcXIRlWeh0Ojg6 +OsLR0RE+++wz3Lp1CwDQ7XZh27YMvV3XlQE6EREx+CYiIiIiIiIiIiIiIjoXqqrCNE0AkK3Oo9Eo +gsEg4vE4fN9HJBJBuVxGo9FAs9lEt9tFt9vFaDSCZVmyChz4pgU6w3Aieh8x+CYiIiIiIiIiIiIi +InpL6LqOZDKJYDCIUCiEpaUlNBoN1Go11Go1bG1tYWtrC8fHx+j1ephOp1BVVbY+d133vB8CEdG5 +YPBNRERERERERERERET0ljAMA4ZhIBaLIZfLAQBGoxFqtRrqtToymQwCgQCCwSDq9ToMw4BlWXAc +B47jYDKZwLKs834YRERvHINvIiIiIiIiIiIiIiKit5iu64jH4lBVFdevX0cymcTGxgYajQYajQaO +jo5QrVbRbDbR6/XQarWgqioURTnvXSciemMYfBMREREREREREREREb3FDMNAIpFANBZFOp3G1atX +MRgM0G630el08MUXX+DOnTt4/PgxPM9Dr9dj6E1E7x0G30RERERERERERERERG8xRVGg6Ro0aDBN +EwAQiUQQjc6CcMdxEAwGkc1msb+/j6OjI2xubiKVSp33rhMRvTEMvomIiIiIiIiIiIiIiC4YXdcR +Doeh6zouXbqEbDaLK1euoNlsotVqYWNjA9ls9rx3k4jojWHwTUREREREREREREREdMGoqopAIIBA +IIBYLIaFhQXYto3xeIzxeIxQKIRQKHTeu0lE9MYw+CYiIiIiIiIiIiIiInoHaNqsFbqiKDAMA5qm +nfcuERG9MQy+iYiIiIiIiIiIiIiI3gGiClyE34qinPcuERG9MQy+iYiIiIiIiIiIiIiI3hEMvIno +faWe9w4QERERERERERERERERERH9EAy+iYiIiIiIiIiIiIiIiIjoQmPwTURERERERERERERERERE +FxqDbyIiIiIiIiIiIiIiIiIiutAYfBMRERERERERERERERER0YXG4JuIiIiIiIiIiIiIiIiIiC40 +/bx3gIiIiIiIiC4Oz/Pgui5s2z7vXSEiIiIieuNs24bruvA8D77vAwB834fv+/A8D47jwHX4eZno +dbBtG47jwPO8U685oudh8E1EREREREQvJL7Is20bg8EAzWbzvHeJiIiIiOiN6/f7GAwGGI1GsG0b +nudBURTYto3xeIx+v492uw3Lts57V4kuvMlkgl6vd+r1RvR9GHwTERERERHRC4ngezqdot/vM/gm +IiIiovfSaDhCv9/HeDyGZVmy2tuyLIxGs78LmAFMppPz3lWiC0/8/+d4PMZ0OmXwTS/E4JuIiIiI +iIhOURQFiqpA13UEg0GEw2EAszbno9EItVqNXzgQERER0XvJsiz0+30ZxrmuCwAYj8fodDpwXRfj +8RiGYZz3rhJdeI7jYDAYYDgcYjKZQFEUBAIBBAIBBINB6LoOTdPOezfpLcLgm4iIiIiIiE5RFAWa +psE0TYTDYcTjcUwmE0wmEwyHQ7iui263e967SURERET0xrnubH63bduwLAuO40BRFNmKeTgcotls +Mowjeg3EuC3btgHM/l81GAwiFAohFAohEAhAVdXz3k16izD4JiIiIiIiolMURYGqqggEAkgkEshk +Muh0OrBtG9PpFLZto9/vn/duEhERERGdC9HeXPwAs4DOsqxZ9yRFOe9dJHrniMA7Ho8jFoshFosh +GAxykQmdwuCbiIiIiIiITtHUWbV3NBpFoVCAqqqoVqswDAPD4fC8d4+IiIiI6Nx4ngfHcWDbNhzH +geM4AABd12EYBnRdh67rrEIles2i0Sii0SjS6TQKhQIymQxisRjHCtApDL6JiIiIiIjoFFVToWqq +/BIhkUjAMAx4nodAIHDeu0dEREREdG4cx8F4PMZ0OsVkMoHneQAAwzAQDAblj64zfiF6neLxOFKp +FDKZDLLZLHK5HAzdgKpxkQl9g1deIiIiIiIiOkW0ZhQr5zVNQz6fh+/7GI1G5717RERERETnZjqd +otPpoNfrQVEUWfEdCoWQSCQQi8WQTCa5YJToNYtEIrLFeSwWg2ma7KxA38Hgm4iIiIiIiJ5JURQY +hgFVVZHJZBAKhWDb9nnvFhERERHRuRkOh6hWq9B1HY7jyIWh4XAY6XQa+Xwe+XwekUjkvHeV6J1i +miZMw4RhGjBNUy7YJjqJwTcRERERERE9k6IoUBQFqqoikUggkUic9y4REREREZ2rbrcLx3EwmUzQ +6/WgqioURUEoFEIsFkM2m0WpVOJnZyKic8AeAEREmlSQmQAAgABJREFUREREREREREREREREdKEx ++CYiIiIiIiIiIiIiIiIioguNwTcREREREREREREREREREV1oDL6JiIiIiIiIiIiIiIiIiOhCY/BN +REREREREREREREREREQXGoNvIiIiIiIiIiIiIiIiIiK60Bh8ExERERERERERERERERHRhcbgm4iI +iIiIiIiIiIiIiIiILjQG30REREREREREREREREREdKEx+CYiIiIiIiIiIiIiIiIioguNwTcRERER +EREREREREREREV1oDL6JiIiIiIiIiIiIiIiIiOhCY/BNREREREREREREREREREQXGoNvIiIiIiIi +IiIiIiIiIiK60Bh8ExERERERERERERERERHRhcbgm4iIiIiIiIiIiIiIiIiILjQG30RERERERERE +REREREREdKEx+CYiIiIiIiIiIiIiIiIioguNwTcREREREREREREREREREV1oDL7pwvM9/7x3gYiI +iIiIiIiIiIiIiIjOEYNvIiIiIiIiIiIiIiIiIiK60Bh8ExERERERERERERERERHRhcbgm4iIiIiI +iIiIiIiIiIiILjQG30REREREREREREREREREdKEx+CYiIiIiIiIiIiIiIiIioguNwTcRERERERER +EREREREREV1oDL6JiIiIiIiIiIiIiIiIiOhCY/BNREREREREREREREREREQXGoNvIiIiIiIiIiIi +IiIiIiK60Bh8ExERERERERERERERERHRhcbgm4iIiIiIiIiIiIiIiIiILjQG30RERERERERERERE +REREdKEx+CYiIiIiIiIiIiIiIiIioguNwTcREREREREREREREREREV1o+nnvABERERERERER0dtq +Op1iPB7Dtm1oqgZN12AYhvwRPM+D67pwHAfj0Rij8QiGYSAUCsE0TSiKAkVRTmzPQSgYRDAUhK7r +0HUdqsoaFSIiIiKiV8Xgm4iIiIiIiIiI6DkODg5w9+5dVKtVxONxxGIxFItFlMtlZLNZ+XuTyQT9 +fh+1Wg337t3D/fv3kc/ncfnyZSwuLsI0TQQCATx58gR3795Fo9HA2toa1tbWkM/nkUqlEIvFzvvh +EhERERFdWAy+iYiIiIiIiIiInmN/fx9/+tOf8PDhQ+TzeczNzWFzcxOxaOx08D2eoNVqYXt7G//6 +r/+K3//+91haWsLf//3fAwAikQgikQhu3bqF//N//g92d3fx85//HLZtw3VdBAIBBt9ERERERD8A +g28iIiIiIiIioh+B7/vo9Xp48uQJnj59CsdxznT7aDSKlZUVrKysnGqVfVaWZcGyLPT7fVQqFRwc +HCCTyaBUKiGXyyEQCCAYDL7Stn3fh+/7qFQq2NnZQb/fR7lcRrlcRigUQiAQgGEY8H0fAFCpVPD0 +6VPUarVT2xGPTVVVqKqKcDiMSCSCWCyGTCaLTCb9g47BDzGdTtHv99HpdBAIBBCJRGatyh371HGo +N+q4e/cu7ty5g2azCdM0EY1GkUgkkIgnEAwFEQ6FEYvFkEwm0Wg00Gq18OWXX8J1XUSjUWQzWSiq +wpbnRERERESvgME3EREREREREdFrJgLhZrOJf/u3f8P//b//F5ZlnWkbc3Nz+K//9b8il8shHo9D +1/VXCn2n0yl6vT52dp7i97//Pf74xz9iY2MDn376KW7cuIF0Oo1AIHDmbfu+D8/14HouHjx4gP/9 +v/83Dg8P8atf/Qq/+c1vMDc3B1VVZfDtez4ePnyI//W//hfu3Llzalsi0BZzs7PZLObm5rCwsICr +V68iHA7Jv3/TwffLHAff91GtVvHll1/i9u3bUFUVa2tr2NzcxPr6OhaXFqFps9ngKysr+OCDD2AY +BlqtFr744gsoioKFhQWsra1B07RzCfiJiIiIiC46Bt9ERERERERERD8C3/PR7XZx9+5d/OEPf8Bk +MkEwGISmaVBV9YXB5vz8PK5cuQLbtuF5HnzPB16hENh1XUzGYzQaDTx69AhffPEFHMfB4uIiVlZW +EIlEXnnbnu/BdV1UKhX85S9/we7uLpLJJDY3NxGNRhEOh2W1t+d72N/fx1//+lf86U9/QjAYhGma +8niIkFzXdaRSKeTzeTQaDWiahkwmg2w2i1gshlAodN5PreT7vqyoPz4+xqNHj/DkyRNcunQJV65c +webmJpaXl1EoFORtyuUyxuMxFEXBn/70J1QqFSSTSRwdHaHb7cpqdwbfRERERERnw+CbiIiIiIiI +iOgNyGQyKBQKyGazCIfDCIVC3xtuFgoFXLlyBcFgcBaUq68WhOq6jnAkjHw+j6tXr6LT6eDy5ctY +XFxELBabtTl/xW2/qnA4jGKxiGKxKMNs0zThui5c18V4PMZ4PMaTJ0+gaRpGoxGuXLmCK1euYGFh +4Y3u6/exLAv1eh31eh37+/vo9/tQFAXZbBbXrl3D2uoaEonEqdvE43EsLCxgOp3iyZMniEQisG0b +BwcHePDgAcrzZSwsLsA0zfN+eEREREREFwqDbyIiIiIiIiKiNyCZTMrwNpPJIJfLQdO05/5+NBrF +0tISQqHQK7c5BwDDMBAOh5HP5XH9+nUYhoFSqYTFxUUkEolXanP+QwWDQczPz+NnP/sZisUiMpkM +QqEQ+v0++v0+7t+/j7t37+Lo6Ajj8RjVahWWZSGXy71Vwbdt26jX69ja2sLh4SEGgwEAIJ/P48MP +P0SpVEI4HD51m3gsLhc9fPXVV4hEIrAsSwbfhmGgWCoy+CYiIiIiOiMG30REREREREREb0AwGEQ2 +m8XCwoKsdv6+4NtxHLRaLezt7SEYDCIcDiMWiyGdTiOTyUBVv9ub/OjoCJVKBf1+H4l4AvFEHMFg +EIFAAJZtodFoYHd3F6qqYn5+XrYhB2YhruM4GA6HaDQaqNfrsvLadV2oqgpN0xAOhxGNRqFpGnq9 +Hvr9Pu7evYtutwvLsrCzs4N///d/x/3795FIJJBIJJBKpZBKpXB8fIzJZAJVVRGLxTA3N4fFxUUU +i0VEo1EMh0MMh0N4nofRaARFUeA4DnZ3d1Eul9Hr9WRrcdu20Wq2UG/U0ev1MJ1OYVmW3M9QKDS7 +32QKsXjsO23SbduGbdsYDoeyans0GsGyLLiuC8MwYJombt26JQN427a/c8xqtRoeP36MVquFQCCA +QqGAXC6HZDKFSCQCXT/99Zuma1A1FfF4HIVCAeVyGbZto9ls4vHjxygUCrBt+7xPVyIiIiKiC4fB +NxERERERERHRG2AYBmKxGDKZDPL5/AuD73q9ji+//BJ/+9vfEI1GkcvlUC6XcfXqVSQSiWdWBD95 +8gT/7//9PxwcHGBxcRHLy8soFosoFAro9Xp49OgR/vjHP2IwGGB+fh6Li4sIBoPwfR+O42AwGOD4 ++BhfffUV7ty5g0ajgXa7jclkgkAgAMMwkM/nMTc3h2AwiOPjYxwfH2N7exvtdhuj0QiPHj1Cq9VC +MBiEYRhIJpNy5vXe3p4MtE3TRDwel9XvqVQKjuPAsiwZLodCITx69AgHBwc4Pj5Gv9/HZDKRgfvD +hw9x584d7OzsyGpxXdcRCASQSqWwvr6Oy5cvY2lpCeVy+TvBd7/fx/HxMb744gvcunULzWYTg8EA +lmUhEokgEomg2Wzi8PBQBu4ng2/LslCtVvH48WP0ej0kEgmk02nkcjmEgsHnVuorigLTMJHP53Hp +0iXs7++jXq9jOBziypUrcB33vE9XIiIiIqILh8E3EREREREREdEboOs6QqEQYrEYkokk0uk0DMN4 +7u/3ej3UajXcunULoVAIuVwO3W4XqVQKGxsb8vd834fv+3BdF/v7+/jss8/w9OlT9Ho96LouA+bx +eIz9/X08fvwY8Xgc7XYbju3AcRy4roterydD7Dt37uCzzz5Do9GQVdaiAjqXy6FYLCIcDqNaraLR +aKDZbGI8HsOyLBkeq6oKVZ1VNiuKgng8jk6nA9u2oaoqTNOUVezJZBLZbFY+JjHrezKZ4OnTp+h2 +u+h0OhgOhxiPx6jVajg4OMCdO3fwxRdf4MmTJxgOhxiNRtA0DYFAAIlEQgblrusiEokgkUhAU2cV +1/1+H4eHh9je3sbt27fx17/+VYb3juMgEokgGo3CsiwMh0NomiYrvsWPZc2q6Hd2dmCaJsrlMlZW +VpDP52EGzGdW5QuGOVtEsL6+jn6/jydPnmA8HqPRaGBqTeF5HhRFeeNt6ImIiIiILioG30RERERE +REREb4gITD3fm/3pec/8PUVRYBgGstkslpaWMBwO0e12sb+/j06nc+p2nuthNB5hPB6j3W7Ltt/R +aBQrKyuYm5tDOBxGt9v9zv14vgfHcTCdTrG9vY2//OUvePjwIVqtFhRFQblcRiQSQSAQkLcRobTj +OFhdXcWHH36Ie/fu4datW/B9H4uLi1hbW0M0GkUoFEIymcTCwgKWlpbQ7XZx9+5dDIfDFx4rRVFk +eC4CZN/3MegPcOfOHfztb39Ds9mE67qYn59HKBRCJBKB67qwLAuO42A8HuMvf/kLJpMJQqEQwuEw +IpEIwuEw9vb28Kc//Qn37t1Do9GAaZqYn59HJBJB8OtqbV3XcXx8jIODA4zHY1ntLSrkRfV5q9VC +NptFKpXC4uIi0un091bzA7MOAOl0GsvLy9jZ2YHv++j3++j1euh2u4hGowgEApz1TURERET0khh8 +ExERERERERG9YSL0dt3vtrRWFRVQIYPv1dVVbG9vo1KpwLIsdDqdU7dzPRfj8RidTgedTkdWaEci +EayuriKZTD43hBUBrjW1sL29jX//93/HgwcPEI/HEY/Hsbi4iCtXrqBYLGIymWA6nWJ3dxdPnz6F +4zhYW1vDL37xC4RCIVQqFYzHY1y6dAn/+I//iGKxiHQ6jXg8/nXoHMHTp09PtRt/HlHpLEJv8e+u +66LX7+H+/fv4/e9/D9M0MTc3h4WFBayurmJ9fV3OKN/b28Pt27dx//59uK6Lubk5FItFuK4LXddR +qVTw5z//Gbdu3UI8HkcsFsPS0hLW1tZQKBQwnU4xnU5x7949tNttjMfjU8dNBOy9Xg/tdhvJ5KyK +f2Vl5aWCb03TkM1mYZomvvrqq1moPxjI4DuVSn1nPjgRERERET0fPz0TEREREREREb0B7XYb9+7d +g+u6SKfTyGQyz2yFLSqXgdkM6aWlJTQaDViWhel0+nVV92yWtWmYsCwLrVYLe3t7sr15NBpFMplE +KpVCOBx+ZsAOAI7joN1uo9Fo4PDwEI1GA5PJBIuLi7h+/TrW19exvr6OfD4vg+BYLIZYLAbHcXDp +0iWUy2Ukk0mYpglN0xCLxVAoFFAul5HNZpFIJGSIHQqFnhsIe54n7+Pw8BBPnjxBpVKB4zhIJBII +BALo9/vY399Ho9FAv99HsVjE0tISrl+/Lmeaj8djtFotRCIR1Go17O7uwnEcHB0d4f79+8jn88jn +83Kutni8165dw8rKCpaWlpDL5jCZzoL+fr+P3d1dDIdDGIYBRVHgOI6cVT4ajWBZlmzrnsvlEIlE +oKkvDr6DwSAURUE0GpVt7y3LQrfbxXA4RDAYfKmFAkRERERExOCbiIiIiIiIiOiNqNfr+Pzzz7G9 +vY1AIHCqffhJYt713NwcFhcXsbS0hCdPnsDzPPT7fXQ6HTSbDRiGjng8Dsdx0Gg08PTpUwwGA9nO +O5lMIhKJwDRNTCaTZ96XCM1brRaq1SomkwkCgQBWV1fx93//9yiVSkin07KFuOd5sgpdBPixWEwG +wgDkjG0R2gaDQXl/onr720Tleb/fR7fbxdOnT3Hnzh08evQI0+kU+XweoVAI7XYbtm2j3+9DVVVk +s1lcv34dn376KeLxOBLxBBzXkeHz8fExqtUqAoEAarUa7ty5g0KhgGaziaOjI4zHY5imiZWVFfz2 +t79FqVRCMplEOByG47hw3Vlgns1m0el0EAgEoCgKbNvGcDiE7/tyhriu64jFYkin0wiHw1DU75/N +rSgKTGM2BzwYDCIQCMAwDNi2jW63i16vh3g8ft6nLRERERHRhcHgm4iIiIiIiIjoDRAtyBuNxqn2 +3d+WTqeRz+dh2zbK5TIWFhaQSqWgqqpsrV2v12WoLLYpgu9IJCIDWFFhbVnWM/fJcRy0Wi3s7u6i +1WrBcRyEw2EsLi7ipz/9qWy3fbIyvVgsylnljuPAdV1omiZ/R9M06LoOwzBgmiZM05StwcXj9jwP +4/EYzWYTx8fHUFUV3W4XjUYDrVYLW1tbePLkCWq1mlwEkEwmMRwO0e/3MR6Poes6MpkM1tfXcePG +je8cz0AggO3tbezu7qLT6aDVaqHX62EwGGA0GqHRaMC2bYTDYZTLZXz88cfIZXNQtdPh/NzcHBKJ +BEKhkAz4bdvGaDSC67qwbRsAZKV9IpE4dTyeR1EUaLoGTdcQCoXk9l3XRb/fx3A4hOM4533aEhER +ERFdGAy+iYiIiIiIiIjegEwmg7m5OWSzWYTD4VlV8DOCb9GivFQqYW1tTf57Op3GaDTCZDLBzs4O +TNOULcebzSZ2d3fheZ687cvMmRYtuzudjgyBA4EAIpEIDN2ApmrP3EdFUeD7/isfi8lkgu3tbbm/ +omp8PB5jNBqh2+1iNBohnU5jfX0dV65cged5GAwGaLVa0DQN8Xgc0WgUpmk+cxGBqqqIRCJIp9OY +TCZot9uyfXggEIBt2wiFQtB1HZFIBLquv7BKW/A8D7Ztw3GcWeW2aSIQCEDXdWjas4/Z99F1XVbJ +q6qK6XQqt09ERERERC+HwTcRERERERER0RuQSqVw9epVXL16FdlsFrlc7plVwcFgEOFwGLFYDLlc +DqlUCqlUCplMBt1uVwbfiUQC5XIZANBsNrG3t4dMJoNSqYSlpSUkk0moqgrP8567T57nYTQaod1u +w3EchEIhGSjrhv6d6ueTnlex/jKGwyF2d3dxdHR0qvrd9314nod4PI50Oo3FxUVsbGzgt7/9Lfb3 +93H79m0MBgMoioJ4PI5YLCaD728TFdjpdBqtVgu2baPX6yEQCMA0Z7PRxTz1aDT6ncr27+O6LqbT +qQy+xTYNw3jpbXx7X4PBoJz5LWadf99zR0REREREpzH4JiIiIiKiM3MdF447+7L/LEEBEdH7LBgM +IpvNYmFhAcViEcViEbr+3a9mNE2DYRgIBoOzymvDQDwex9zcHHq9HhzHwd7enpy1reu6bOWdTCaR +yWSwuLiIdDr9wuBbBM2O48D3fXnfomr5VYPtFzEMA5FIBIlE4juhsaZpyGQyKBQKWFxcxJUrV1Aq +ldDr9aBpGlzXle8/L6quVlVVvk+JduviB4BsyX7WKm1x3Hzfh6Iocr9f9XiJbYhjLrb9Q6rqiYiI +iIjeNwy+iYiIiIjoTHzfh+3YmE6nMvA2TfO8d4uI6K0nWmqnUilks1kUCgXo2ne/mlE1VQa2IhiP +xWIolUrodrtot9vY399HoVBAo9FAKBRCp9NBr9cDAORyOSwtLSGVSkHTtBe2yxYhq/Bjhd0nBYNB +lEolXLlyRVa1x2Ix2YY8FoshkUgglUwhl8/JVujfXmh11n0Vvy8CZhE4/1Ant/dDMewmIiIiIno1 +DL6JiIiIiOh7eZ4H35uF3ZZlYTqdYjAYoNfrIRgMIp/PM/gmInoJuq7LVuLJZBKZdAaarr3UbePx +OMrlMrrdLlqtFqrVKg4PD3F0dIR4PI5+vw/LsmAYhmx3Ltpmv4imaTJgF9XfZ2mx/e37eJng1jAM +zM3N4ebNm1hZWUGpVEIul0M4HEYkEkEgEIBhGDAMQ97GNE1Z5S2qt0Wl+vO4rgvbtuF5HhRFkRXt +uq7L27que+aw+WTQLVq0/5AK7ZMV5ABOVX8TEREREdHLYfBNRERERETP5fs+ptMpJpMJ6vU69vf3 +cXh4iOPjY9RqNZRKJfz617/GT3/60/PeVSKid1osFkO5XEan08HDhw/R7/dxeHiIhw8fIplMwrIs +pNNpZDIZxGIxBAIBaNqLQ3VN0xAMBhGPx9FutzEajeD7PobDoQzAvy+AVRUVnuLJoFb4dhX5tymK +AtM0EYvFEI/H5Rxz0zRhmiY0TfvO/huGgWg0ikgkglarhW63KwN/0XL8JMdxMBwO0Ww2MR6PYRgG +YrEYotEootEoLMvCZDKB67qYTCbPDfvFdk8+HlVVEQgE4LouPM+DZVmwLEuG6WcNrB3HwWQywWQy +ge/7CAQCCAQCHCVCRERERHQGDL6JiIiIiOi5RPDd6/VQqVTw1Vdf4e7du9jZ2cH+/j42NjZQLBYZ +fBMR/cji8TgWFhbQ7XZhGAb6/T6Ojo7w8OFDZLNZ2LaNbDaLbDaLeDyOQCDwUhXDmqohHA4jkUhA +0zSMx2NMp1MZfLuu+8w55N8OgU+2ED9Z/fy88FsE39FodNbSPJVCMpmEqqhQ1Ge3DRdBeTQaheu6 +6PV6p4JvsV3BdV0MBgO0Wi1MJhM5Nz0ejyMWi6Hb7WI8Hssf33v2vj7rcWiaBtM0ZbW46Ihysnr8 +LOG34zhyoZkIvkVlOhERERERvRx+eiYiIiIiIgDfhNzT6VTOkG2322g0Gmi1WqhUKtje3kalUkGt +VkOj0UChUMB4PD7vXSciuhAcx8FoNEKn00E0GkU4HP7eqmxd1+XMa9M0ZYv0RCKBSCQC27ZxcHCA +4XCIUCiE+fl5FAoFhMPhl64U1g0dqVQKjuPg8PAQqqpiOByiUqngs88+Q7FYRCqVQjgc/rq1uIt+ +v4dutwvf91EsFjE3Nwdd12WV+XA4xN7eHkzThKqqp0JjUWEtZmufbD1+sq35s0SjURSLRbRaLWxv +b8O2bTSbTWxtbSGXyyEejyMRT8B2bEwmE+zv72NnZ0fuSz6fRz6fl5Xx/X4fnueh3++jUqng1u1b +yOfzCIVCUFUV4/EYo9EIDx48wNHREbrdLhKJBDzPg2EYCIfD8H0fhmHA933Yto3RaITBYADTNOXi +g+fxfV8uLhiPx5hMJnAcB5qmIRaLIRKJMPgmIiIiIjoDfnomIiIiIiIAsy/gB4MBup0unu48xaNH +j7C9vY1arYZ6vY52u41+v4/BYCArAl/UypaIiL5h2zb6/T4ajQYMw5Azp58nEAggnU4jm8meakku +gtvhcIharYbBYID19XUsLS1hbm4OkXDkpffJNE3kcjkkEgk8ffoUpmmi0WjgyZMn+P3vf4+1tTWs +r68jk8nIiuRKpYJKpQLP8/DJJ58gl8vJampd19Hr9bC9vS3DbTlfW52F4q7rvtLxE1Xvo9EIX331 +FXzfR6PRwO3btwEAi4uLWF5exng8RrPZxM7ODh49eoTd3V0sLCygUChgc3MTiUQCyWQSR0dHAIBO +p4Pt7W38x3/8B+bn55HJZBAIBOT73+3bt/H06VO0Wi1kMhl4ngdd1xGJRKAoigy4RWv1TqeDeDwO +0zRfGHzbto3pdCqDb9u2YRgGksmk3AYREREREb0cBt9ERERERO8p13Hhei4sy/q6Sm2MWq2KarWK ++/fv49atW7h//z6azSZarRbG4zE8z4PnebKV6/e1sSUiotMmkwnq9ToqlQoGgwHa7fb3VmbHYjGs +rq4ikUjANE3ouo5oNIp0Oo1CoYC9vT20221Mp1Pouo6VlRUUi0WEI+GX3iexzUAggFKphEwmIzt9 +3L59G8PhEJZloVAoYDQaYTweY2dnB0+ePIGmaVhYWIDrugiHw0ilUojFYphOp3j69Cl834fnebIi +PRwOYzgcwrbtVzp+4XAYhmFgPB4jl8shFovBtm3s7u7K1ueTyQTD4RD1eh37+/uoVqtyvnexWMTm +5ibC4TAikQgePHiASGS2SKBWq+Grr75CrVZDoVBAKBRCtVpFrVbDkydP5PvgdDqF7/vQdR2hUEj+ +aRgGPM9Dr9dDo9GAqqqIRCLf+/y6rovRaIRer4fBYCCPi2maSCaSiEajL6yCJyIiIiKibzD4JiIi +IiJ6D3meh/6gj16vh+pxFZW9Cvb399FoNFCv13F8fIyjoyPU63WMRiNMp1M4jiNDDAbeRERn1263 +8fDhQzSbTYRCIYRCoe+tCC4UCvjNb36DUqkEVVWh6zpM00Q2m8XKygrG4zF6vR40TUM6ncby8rJs +1f2yVFWFaZoIh8NYW1vDr371K2SzWfR6PfR6Pezu7qLRaCAQCMi23K7rwvM8pNNp+Rjy+TyuXbsm +W4eL1uC7u7vI5XJYWlrCysoKer3eKwffmqbB930kEglsbm6i2+2i2WzK1uqNRgN3796F684WdTmO +g1AohBs3buDmzZtYX19HPp+XreOXlpbwk5/8BIFAANPpFPV6HZ1OB7u7u7Jtu2jhHovF5HFSVVU+ +H6IKP5VKwTAMtFotPH36FLquI5vNfm9w7TgOms0m9vb20Gq1oCgKotEo4vE44om4DNaJiIiIiOjl +8NMzEREREdF7yPd89Pt9VI+ruHf/Hr744gvcu3cPzWYT7XYbo9EIlmXJanARegNg4E1E9IparRZG +oxEqlYpsA/59xMzuX/3qVwiFQvC92TzpfD6PlZUVNBoNHB4eAgDS6TRWVlbOHnwrqqxaXl+/hHAo +jHK5jM8++wxfffUV9vb2MBgMMJlMoGkaFEVBLpfD/Pw8EomEnIedy+Vw/fp1KIqCv/3tb6hWq5hM +JvB9H7FYDB999BFM00S324XjOK90/FRVlW3Ab9y4gVgshi+//BJffPEFdnd3MRwOMRqNoGkaTNNE +IpHA+vo6rly5guvXr2N9fR1zc3OzY6+oWFlZwXA4RDgcxu3bt3H//n2MRiO4rgvTNFEsFlEulxEK +hZBIJGDbtgy+FUU5FXxnMhlomoZWq4WdnR1kMpkXPk4xo3xnZ0cG37FYbBZ8x+OylToREREREb0c +Bt9ERERERO840WZ2NBqh0+mg1Wqh1WqhWq2iXq9jZ2cHjx8/RqVSwXA4xHA4xGQyged58kt7ht1E +RGenqAoSiQSuXbuGX//617As60y3n5ubQ7lchmEYs7BVnc2Tnpubw5UrV+D7PqLRKHzfx+bmJlKp +1KxKWDv9dY+o6k6n07hx4wZarRY2NzexuLiIcCSMQCAAXdeRSMShqgpUTYVlWdB1Hc1mE91uF9Pp +FKqqQtM05HI5lEolLC8vY25uDpqmIZFIYGlpCcDsPSMUCmE8HsO2bUQiEVy+fBnz8/OwbRuu62I4 +HOKDDz7A/Pz8mWZZK4oC0zSRz+eh67p8n0qlUuj3++j3+zAMA6ZpIpVKYX19HZcuXcLS0hLS6fSp ++0mlUlhdXZVBeTQaxXA4hOM40DQNxWIRpVIJlmWh3+/DcRzkcjnkcjmUy2XEYjEYhiEr2ofDIdrt +Np4+fYrFxcVTi8a+HWD7vg/LslCtVvH48WN0Oh1Eo1Fks9nZjHEz8MKFEUREREREdBqDbyIiIiKi +d5jv+3AcB+12G7VaDVtbW3jw4AG2t7dlG9tOp4Nut4terwfHcTCZTGQbW7ENIiI6GxF0ZjIZ/OY3 +v8HCwsKZK52j0ShWVlYQjUah6zoURZGzuKPRKEqlEj744AMAwNLS0iyI1Q2o2unAVNdmFd2lUgm/ ++93vsLq6ilwuh4WFBSQSCdlOOxAIAAAWFhYQCoWwtraG8Xgs3xcURYGiKAiHw4jFYkgkEshkMtA1 +HbFYTFYsFwoFfPjhh7BtG57nydA9m81ieXkZ165dg2VZKM4VUSwVEY1Gz1Slruu6bD1uGrPKbBHO +27YtA/pgMIhUMoV0Oo1YPIZoNHpqO7FYDOVyGYlEAvOleXz44YewLAue50FVVUSjUcSiMbieK/+7 +mFWeTCaRSCTgOA4KhQIuXbqEp0+fYn9/H71eD5ubm5hOp/Bc7zvPhxgXMp1+E3w7joNUKoVSqYRc +Lgfd4Fd2RERERERnxU/RREREREQ/wPNC4ZP/XczEPjkb+9s/4u+e5WXbnIptOY4Dx3FgWRam0ylG +oxF2d3dRqVRw9+5dfPHFF3j8+DEmkwmm06lsZ27bNkNuIqLXSFFmFd8fffQRPvroo9eyTTHjO5vN +YnV19aVuo+kaNGjydh9//PEzf88wDBiGgWg0inw+f6b9CumzmeW5XO6l9+tVqaqKcDiMcDiMVCqF +tfW1V9qOmLP+Q/Z5MBggn89jY2MD3W4XW1tb6PV6cl54KBRCMBg8VWnuOA5s20a320G1WsXe3h6S +ySRWVlZw+fJlWc1ORERERERnw0/RRERERERndHLWte/58PzTgfbJ3wEgZ2WLQNp1XbiueyqgtqaW +/JJbVNSJGaSKqsjw+3kh+MnAu1arybmvBwcHODo6ku3Nj4+PcXR0hMFgIO/bcZzvDd6JiIjo2USr +cwDY29tDNBpFt9tFvV7HnTt3MB6PUS6Xkc1m5W263S6q1SoePnyIarWK8XiMQqGA+fl5bGxsoFAo +MPgmIiIiInoF/BRNRERERPQKfN+X7cBFaOy67ncCcACyovpk8H0q9LYsTK0pNF07FXprmgZVVaH6 +KlRF/U6r1JM814PruphMJjg+Psbjx49x79493LlzB1tbW9/cz3Qqq8C/XYVOREREZ2OaJnK5HBKJ +BLa2tuTM9Xq9jnv37kFVVcTj8VPBd6/Xw87ODh49eiSD70AggHK5jCtXriAcDjP4JiIiIiJ6BfwU +TURERET0kkQ4LMJr27ZhWzZsx5bBtuu48HxPzscGgHa7jWaziU6ng36/L6utu92ubIFqGAZCoRA0 +VYOqqdB1HYZhwDRMGKYBTdMwGU8wnkwwnc5alItW5dPpFOPxGMPhEP1+HwcHBzg8PMTu7i52dnZQ +q9Vk2G7btvzn12E8HuPw8BC3b98+76eHiIjoXIjPB91uV76fdzod3Lp1C5PJBJZlodloAgAUVcH2 +9jbu3r2Lra0tdDodhMNhaJqGfr+Pvb09aJoGTdPO+2EREdFzDIdD1Go11Ot1dLtdOI4DwzDOe7eI +iAgMvomIiIiIXor4UttzPRkgTyYTjEYjjMdjGUKfbGUubiNmfTYaDRl+W5YF0zRlRZfnebPgW9Og +6zoCgQCCwaCcP6rrOhqNBur1OlqtFrrdLjqdjvxT/HS7XYxGI4xGIwwGA/T7fYzHY1mdLirTX5fh +cIinT5/iP//zP8/7KSIiIjpX+/v7UFUVsVgM3W4X+/v78v356dOnsqPL7u6urPYOBAJIJpPQNA0H +BwenOr8QEdHbybIs9Pt99Pt9eN6s8xaDbyKitwODbyIiIiKil+T7PlzPlaH3YDhEr9tFv9/HcDTG +cDiGZduwbefrcNmH7wPDwQCtdnv25Xd3gP5wgonlQdV78BUNtqtgansIBoOy0jscDiMcCiMWiyIe +dxEMmKjVanj69CmOjo5Qq9VQq9XQbDbRarVOheBiX0+2Y/+xjEYj7Ozs8IseIiJ6700mE4zHYwCz +6u/Dw8PZZ4ThUIbiiqLg6OgIe3t7GAwGKBQKiEajmE6nsksLERG93TzPm3X/sm2Ew2FEIpHz3iUi +Ivoag28iIiIiopdwMkSeTqcYDodot9toNJpoNtvoDCbo9i1MLRu248J1RdjsYzKZYDgcYTh0MRoq +GNoB6L4Or69g4tnoTQdoDICAaULTdJiGgXBkjHB4jEzKQtHyEIuYODo6xs7ODnZ3d3F4eIh6vY7x +eCx/JpPJqWpuUeH9YxqNRtjb28NwODzvp4iIiOhciTEok8kE3W5Xfl44Pj5Gr9eTldz9fh+j0QiW +ZaHb7cL3fTQaDQQCAc72JiK6AFRVlT+5XI6LgImI3iL8NE1ERERE9JJ834fjOLNq78EA7XYb1WoN +x9UGGj0bzZ6D8XQ259v5Ovj2MbuNY1uwLBe2rcK2TaiOhokH9C0bxmAIw7Rh6AY0XYOum4hGJohE +phhOXKiqCs+eBd+7u7vY3t7GwcEB6vW6DOPF3O6TQfePHXoDQL/fx3Q6ZYUaERERIEediEpA13Ux +Ho9PzewWAbkYjdLv92WAwhbnRERvP03TEAwG5YKlVCp13rtERERfY/BNRERERPQSfO+b1uG2bWMy +nqDXH6HVGaPWmaI71tC3g7BcBa7vwYMP+D58AJ7qwdNd+IoDRXehmy6gKFA0DZ6mwdYNeNDgeCpU +V4MGDe40gKmiQ+8BkbAF33NhexrCkSiSySRGo5GcNS5+LMvCZDL57r7/iAF4MBhENBpFOBw+76eI +iIiIiIjoR6dpGnRdh2maCIVCpxY3ERHR+WLwTURERET0kjzPg+d5sCwL48kY/eEE7YGFZt/DxI/A +VuPwAgbg+1C+DpsVAMrX/676HnTPh+f7s/+uqlAVRf4JRYGvKHBVFRPosG0NxlhHqOfD922oRgSF +uRIMXUM0GkUqlUK/35c/w+EQjuMAOD3j+8cUiUQwPz+Pcrl83k8PERERERHRj078v5bv+0gmkwgE +Aue9S0RE9DUG30REREREL8HzPRkki8rq0dhCb2ijO/TgmQG4ZhK++t0vPRQAL1MD4H/9AwAuAHjA +wPLQGXgwVBeRYBSlaBnZdAK5XA6dTgeNRgPNZhOtVgutVguhUEi2ThVV4NPp9NS879dZAR4Oh7G0 +tIQPPvjgnJ8hIiIiIiKiH59t25hMJhiPx9B1HbrOmIWI6G3BKzIRERER0Rn4vv/NTG3XgeN6cD0f +vvfjtBP3fMD1fLi+glg8gflMDob6Tfg+GAwwHA5l1Xev15NBeL1eR61WQ7vdlmG44zhyLvjrEAqF +sLy8jE8++eRNPg1ERERERETnYjweo9Vqod1uYzweYzwen/cuERHR1xh8ExERERG9JFE17XneLEB2 +3Fnw7fqny7VfI9/z4bo+PA+IxRJYXMwhnQjLuXKe68H1XFiWDcuaYjAYYGtrC1tbW7h//z6A2Rcz +juPAcRwoiiIfy+sQjUaxtraGTz/99E09DUREREREROem1+vh8PAQh4eHqFarqFar571LRET0NQbf +RERERERnIGa5zeZ9fzPbDb4PBa8nTH7WfQIKNE1DIBBAOBxGIBCAaZryd0QYP5lMoGkawuEwotEo +kskkyuWynAHe7XbR6XQwGAxg2zYcx5Gzy18lDNc0DcFgEIlE4ryeEiIiIiIiojeq0+kgFApB13W5 +uJiIiM4fg28iIiIioh9AFHr/2F91vGgst6Io0HUdwWAQc4U5RCIRzM3N4erVq2g2m6hWqzg+PsaT +J0/w+PFj7O/vw7IsOQfccZxvQnwiIiIiIiIiIqILhsE3EREREdGr8vF1Iv3jhcU+Xq6LuqIoUBQF +pmkinUkjnUljaWkJvu/Dsizs7u6iUqkgnU5D0zT4vo/hcChn0k2nU0wmEzkDXATgDMKJiIiIiIiI +iOgiYPBNRERERPQKfN+H/yMG3q+TrulIpVIAAMMwkE6nceXKFbTbbbRaLRwfH+Pw8BD1eh2TyQTj +8RiWZcH3fbiuKx8vERERERERERHR24rBNxERERHRO0xRFKiaimQyiVgshmKxiBvXb2A0HuHo8AiH +R4e4f/8+vvjiC3ieh263Kyu+xfxvht5ERERERERERPS2Y/BNRERERHRGFy0IVhQFhmHAMAwEg0EA +QNJLwjRNRGNRGIYB0zSRzWbRaDTQbDbRbrfR7XbR7/dhWRbG4zFs2z71+C/acSAiIiIiIiIioncX +g28iIiIioveQoiiIxWLQNR2hUAhzc3O4ceMGqtUq6vU6dnZ2sLW1hUqlgm63K0NuUQHOSnAiIiIi +IiIiInqbMPgmIiIiInoPKYqCUCiEUCiEdCaN5eVluI6LeqOOWq2GBw8eIBKJwDAMVKtVmKaJ0WgE +27bhOA4sy4JlWXJbRERERERERERE54nBNxERERERAQAUVUE4HEYul4PrugiFQlhaWkK9Xkej0cDx +8TGq1SpqtRq63S663S5UVWXwTURERERERERE547BNxERERERAQBUVUU0GkUoFEIymcTq6iomkwna +7Tba7TYeP36MW7du4f79+9jb28NkMmHoTUREREREREREbwUG30REREREJKmqClVVYRgGQqEQ4vE4 +wuEwkskkdE2HpmlIJBIol8s4ODjA2toaSqXSee82ERERERERERG95xh8ExERERHR9woEAtA0DYtL +i4jFY1hdXUWr1UKz2UQmk8Ha6tp57yIREREREREREb3nGHwTEREREdFzKYoC0zRhmibC4TAKhQJ8 +38d4PMZoNIKmaQiFQue9m0RERERERERE9J5j8E1ERERERGem6zrC4TAURYGmaee9O0RERERERERE +9J5j8E1ERERERGeiKAoMw4Cu6/LfiYiIiIiIiIiIzhODbyIiIiIiOjNFURh4ExERERERERHRW0M9 +7x0gIiIiIiIiIiIiIiIiIiL6IRh8ExERERERERERERERERHRhcbgm4iIiIiIiIiIiIiIiIiILjQG +30REREREREREREREREREdKEx+CYiIiIiIiIiIiIiIiIioguNwTcREREREREREREREREREV1oDL6J +iIiIiIiIiIiIiIiIiOhCY/BNREREREREREREREREREQXGoNvIiIiIiIiIiIiIiIiIiK60Bh8ExER +ERERERERERERERHRhcbgm4iIiIiIiIiIiIiIiIiILjQG30REREREREREREREREREdKEx+CYiIiIi +IiIiIiIiIiIioguNwTcREREREREREREREREREV1oDL6JiIiIiIiIiIiIiIiIiOhCY/BNRERERERE +REREREREREQXGoNvIiIiIiIiIiIiIiIiIiK60Bh8ExERERERERERERERERHRhcbgm4iIiIiIiIiI +iIiIiIiILjT9vHeAiIjeHb7vw7ZtuK6LSqWCSqUCx3GwuLiIxcVFBAIB6LoOVeW6KyI6X7Zt4/Dw +EIeHh/B9H6FQCOFwGMlkEqlUCqZpnvcuEhERERERERER0Rkw+CYiotfGcz1Mp1OMx2N8+eWX+Jd/ ++ReMRiP87ne/QzweRyKRQDAYZKBEROduOp3iwYMH+POf/wzXdZFOp5HP57GxsYFwOMzrFBERERER +ERER0QXD4JuIXovpdIrhYIjBcHDm25qmiUgkgmg0CgBQFOW8H865E5XTw+EQg8EAwWAQkUgEhmFA +07Rzq5j2fR/D4RDD4RCe5yEUCiEUCkHXdKiaCtdzYVs2xuMxtre38ec//xmDwQClUgm//OUvGXq/ +gO/78H0frutiPB5jPB7DcVyEw7PjrGkadF1/5deI7/twHEeeV6ZpIhwOIxAIyPPK8zz4ng/bsTEa +jjAaj2CaJkKh0Oz3VA2arp33oSL6wWzbxt7eHr788kvYto25uTksLS0hlUphdXX1vHePiIiIiIiI +iIiIzojBNxH9ICKoOzo6wn/+53/is88+O/M25ubm8LOf/QyffPIJVFX9QcHeu0AEj0dHR/jLX/6C +zz77DJcvX8aN6zewuLR4apHAm3IykL19+zb++te/Yjgc4vr167h58ybi8TjC4fB7/by9Dp7rYTga +YjQa4d69e7h9+zb6/T42Nzdx7do1pNNpxGIxBIPBM2/b9314rodms4k///nP+Oyzz1Aul3Hjxg2s +rq4iEokgEonAsiyMhiNUa1XcunULt2/fRrlcxrVr17CysoJYLIZYLMZ29URERERERERERET0VmHw +TUQ/mO/5ODw8xL/8y7/gn/7pn858+/X1dRiGgQ8//BCmacL3/fc6QPU9H47r4Pj4GL///e/xP/7H +/8Avf/lL6LqOaCwKVVURiUTe+DFyXReu6+Lu3bv4p3/6J7TbbfR6PeRyOQCAYRis5v6BHHdWjd1q +tfD555/jf/7P/4lGo4H/8l/+C2KxGBRFgWmarxx8u56LZrOJP/7xj/jv//2/49q1a1AUBclkEoqi +IBwKYzqdotvr4unTp/iXf/kX/PM//zOuXr0K27YRiUQAANFIFGDuTURERERERERERERvEQbfRPSD ++L4PYNYyttvt4vj4/2fvPpvjyNIzYd/pq7K8QxW8I+iaZLvRjDQaSSFtKBRvxEbsD9hfuJ+1UuyG +QrvSrrStnulpQweSILxHeZ8+3w+FPARI0DabAMj7isGwu1lApS1k5n2e5xygUCgglUohHo9D1/XX +hqGTk5PIZrNQFOWTDryfF23TarWKVqt13PbaQxAE57I8UdV3v99Hs9lEs9lEt9uF67rwfV8cC/Tu +JEmCoijQNA3JZBK5XA5BECCdTsMwjJ/X5jwIRQv9TqeDRqOBZrOJfr8P13XFcRUtg2EYyGQyKBQK +yGazotW+qvLSgYiIiIiIiIiIiIguHj69JqL3SpIk5PN5XLlyBRMTE8hkMkin06/8nkqlgqWlJaiq +CkVm+E2fLkUeBc6pVApzc3P4+uuv0el0cO3aNRSLRRE+/5I0TYNpmiiVSrhx4wYcx8H09DRmZmaQ +TqcRi8UgyTxHiYiIiIiIiIiIiOhiYfBNRO9dNpvF0tISbty4gXK5jEql8srXG4aBbDaLTqeDIAgQ +BAEURUEikUA8HhdBeDRHcRAGGAwGGA6HAIBYLIZ4PA5VUSErsnit7/uwbRuO7cD1XHiejyDwIUkS +JEmGqirQdR2qqkLXdWiadmre4uj9PN+DZVmwbRtBEMDzPNGOXZJkaJoKVVGhaio0TYOiKBgOh2L5 +TNNEMpk8fv3pwNCyLFiWBc/zIEsyZEWGbduwbRu1Wg3D4RBhGMK2bTQaDRwcHMB1XViWhVgsJpZb +VVUoivLa9dVUDZquibbkbzpPc1Ql3O120e124XkePM9Dr9fD4eEhJEmCbduIx+OjZXAcWJY1mq88 +DOF5HmzbRq/Xg23bUBQFQRAiDAOx7LquwzAMGIZxap9H+9J1XfGzo/0QrZuiyNA1XaybpmpQVOWt +j92T+zzaD77vv7DPdV07te01TYNt2xj0B3A9F4qiiOPK0I0zl8X3fDGfd1TlLcsywjCEoRvI5XKi +G8LY2BgSiQQMwxCvCcMQYRDCduyXbhdZlsTyOY4jjqvBYCCOq1ardeq4UhQFYRhCURTkcjnMzMyg +Uqkgl8uNzrPjqvMgCET7+5PbKqr+f35bRedaVDEeVZ8PBgP0+31omgbDMKAoClzHheu54j2eP35j +sdg7DQCItpvnPTunR8scnHmuqJoqjsvoOHyb9Y32a/TenjdqZd/v96Gqqqjit237uOp+dE5IkgzD +eLbNovP05OdQtF2042WMjkdN08RxPLSGGAwGcBxHnFtBEMBxHLiud7xcgfg8iD6/ZFkWx8PJ5VJV +FbqmQ9VGy67rujhXg+DZ53K0TcMwEOsedRKQZVksS3SeRMf0Wcsci8XEMo86XoyWJfre6LiJzo1I +tI2iz43R/nq2nxVFQafTQafTEecOu1YQERERERERERFdbgy+iei9MwwD6XQahUIBY2NjGB8ff+Xr ++/0+NjY2UK1W0e120el0kM1m8fnnn+OLz7+AJI8CY8/zMBgMMBgMcPfuXdy/fx9hGOLGjRv47LPP +kEwmkUqlIMsyLMvCcDjE2toaNjY2cHh4iF6vJwInTdOQyWQwPj6OiYkJVMoVjJXHkEqlxHJFYcnR +0RHW19exvr6ObreLXq8Hz/NEiFcoFJDP51EoFFAqlVAslnD//j3cu3cPAPCrX/0Kv/71r8XrT4Yz +W1tbePz4MVqtlghvGo0GGo0Gnj59is3NTYThaA71f/3Xf8XKygri8ThM08TU1BRmZmYwPj6ObDaL +XC6Hrc0trG+s4+DgAL1eD71eT6xvKpXCxMQEJiYmxICEk+v7Ks1mE99//z1++OEHfPfdd2g2mxgM +Bnjw4AEcx0EymYRpmkilUsjlcigUCtjZ2REhWLfbxcHBAVZWVlCr1dBut+E4DhzHEe20x8fHMT8/ +j4WFhWfhqh9gMBzAsixsbm5ifX0d+/v7Yl9GwWIqlRL7Mlq313UaOItt27AsC41GA6urq9jc3ES7 +3Uav14PjOCJYLJfLo+OmUkG5XEGlXMb6+jru3buHo6MjZLNZZDIZTE1NYXp6GsVi8YX3arfb+OHH +H3Dv3j2k02mUy2XRVjybzWJvbw/ff/89Go0GJEnC5OQkTNNELBZDGIajQRNDC5tbm1hbW8Pe3p7Y +57IsQ9M0xONxcWx2Oh3U63Vsbm7i6dOncF0XR0dH+Pd//3dsbW2J4yo6bwHg0aNHePLkCW7cuIFM +JoNyuSxCYMuy0O12Ua/XsbGxgc3NTXQ6HQwGA7iuK8LYcrmMqakpjI+Po1QqYaw0Bj/w4TgOqtUq +vv/+e/z0008oFAq4cuWKWPfd3V0Mh0PYtg1ZlsVxNTs7i4WFBVQqlbfuDhGFua1WC2tra1hfX0ez +2RQDMqJzJfr8KpfLmJmZwezsLHzfR6/XQ6PRwPr6ujg2Tq5vdGxMTk5iYmICY2NjYt9Hg1f++Mc/ +4qeffkI6ncb8/Dzy+Tx2dnawt7cHy7Lg+z40TRPHTjabhWmaCMMQa2trWF1dRb/fh+M4kGX5+Bgc +beOpqSmUSiXx2fXo0SP8+OOP2NnZwdzcHBYXF9HtdrG3t4dqtQrf9xEEAQqFAqanpzE+Pi4G6hwc +HIjjKjpXs9ksKpUKJiYmMDk5ienpaXE89Pt9PHjwAMvLy2g2m6dC+iAIRECdSqUwMzODubk5FApF +ZDJpJBIJMXjj8aPH+OHHH7C9vY2FhQVcuXIFw+FQLHP0ukQigXw+j7GxMfG5kUgkxKAMy7IwGAyw +s7OD1dVVbG9vi98fiqIgHo8jCAIsLy+jVqshHo/D87z38SuQiIiIiIiIiIiIzgmDbyJ676IgMgrQ +JiYmXvn6zc1NbG9v49tvv8Xh4SGOjo4wPj4OXddx+/ZtKFAgKaPgu9vtotFo4A9/+AP+x//4HwiC +AH/zN3+DXC6HSqWCWCwGVVXR7/fRbrfx8OFDfPvtt3j69Cnq9ToajQZisRgMw0C5XMb169dx69Yt +eJ6HVDp1Kgh2HAfdbhe7u7v4wx/+gG+//Ra1Wg2NRgOO44hK0MnJSczOzmJubg5XrlwBAPzhD3/A +P/7jPwIYtX+/desWTNOEIivAiSLr7e1t/Md//Ad2dnaQTCaRSCSwu7uL3d1dHB4eotlswvd97O3t +oV6vi+BckiTcuHEDX331FW7cuIHp6WmEYYgHDx/g22+/xcrKiljfKFCPWlffunULjuMglUq9cfDd +arXw3Xff4e///u9RrVbR6XTQ7/exsrKCjY0NsUzpdBpXrlzBtWvXsLu7C8uyAADdbhf7+/tYXV3F +gwcPsLOzI6pDK5UKpqencePGDSiKgtnZWVEV7PkehsMhWq0WlpeX8e233+Lx48eo1WpoNpsiTCsW +i2IAxPXr15FIJN8p+HYcB51OBzs7O/jjH/+IP/zhDzg8PES9Xodt22JbLi4u4ubNm/jss88gyzJK +pSI2NjbwL//yL1hdXcX4+DgmJydx584dZDKZs4PvThvff/89/uEf/gGlUgnXrl3DlStXMD8/j3g8 +jt3dXfz00084PDxEpVLBn/zJn8BxnlVBDwYDdLtdPHnyBN988w0ePnwo9nlUQZ9OpzE9PY3Z2VnU +ajVsb2/j4OAAzWYTtm3j8PAQ7XYbd+/eFftwenoa165dQzKZxP3797GysgLP80QXB9/zAQDWcDRA +YGNjA//xH/+B7777DtVqVQxqiMViiMViuHLlCj7//HPcunVLTIUQBIEIM//v//2/+Id/+AdMT0/j +t7/9LaampnDv3j0sLy+Lue0lScLU1BRmZ2fx9ddfI5VKoVwui/PrbfZvv9/H4cEhfvjhB3z77bfY +3d1FvV7HYDAQ53SlUsHMzAyWlpYAAOPj4/A8T6zvt99+i++++w5HR0dot9uiA0M8Hsfi4iJu376N +27dvAwByuRzCMMRwOMT+/j7+/d//HX//93+PsbEx/Mmf/Ammp6dx7949PHz4UAyqicViuHXrFr78 +8kuMj4+jWCzC93188803+Pbbb1Gv12FZFmRZxtLSEq5fv47bt28jkUigkC8cd1fo4+HDh/jv//2/ +Y3l5GV9++SX+9E//FPV6Hffv38fm5iZc14XneZiZmcHnn3+OmzdvolAooFAoYHl5Gd988w0ePHgg +OlOUy2Vcu3YNN2/eBIBnAyGCZ8H3P/3TP2F3dxfdbheDwUBUxRuGgWQyiWKxiK+++uq4ejuAro9a +60f7ZvnRMv7xH/8R9+7dw5/8yZ+g3++j0+ng/v37WFtbw3A4qgjP5XKYnp7GlStXEAQBxsfHkUgk +RCeEwWCAdquNlZUV/Pu//zt+/PFHtNtttNtt8TsqFouJwSL5fB6u677T7zwiIiIiIiIiIiK6GBh8 +E9F7J8vyqTa0r2tJbJqjyr2JiQk0Gg10u10AwMbGBh49eiQqVnu9HjY2NvD48WPs7u7C8zxRpZpK +pUTr7na7jUePHuHx48fY3NxEv99HLBbD2NgYstksZFmGLMvQdR3VahXfffcdwjBEKpUSy6vrOjY3 +N/H48WM8efIEa2tr6Ha7UFUVpVJJrGdUxd1qtXB0dISxsbFTAb0kSej3+6Lq8Xm2baPb7aLdbouA +SNd1jI+PQ9M0uK6LarWKZDKJfD6PVCol3nN+fh6VSgWapmF7exvr6+vY2tpCr9cTQXc2m4UkSafW +949//CN830c6nRathGOx2CsDxGigwMLCAsIwFIFWVK0dbbOomrNUKiGZTEJVVQyHQxwcHODevXvo +dDoiII9CxijYlyQJ5XIZc3NzyGazSCaT6Pf7ePLkCR49eoSNjQ10Oh3ouv7Cummahnq9jh9++AFB +ECCVSsE042+0bsCzFtj7+/tYXl7G06dPsbe3hyAIkEwmEYvF4Pu+CId938f6+jpc14WqqigWixgM +BuJ9ms0mhsMhisUi+v3+qfdyXReu66LVauHw8BB7e3ui7XO0XYBRSBu1AY/a4QeBL9qDR9XYKysr +oirXNE0xPUB0/vm+j6OjIwRBgGKxCE3TRIibSCSQy+WQyWRE2/aocj5qzd/v90VVs+/78HwPruti +d28X9+7dw6NHj7C1tXUq7I5aWwOj9txPnjzBcDhEEATIZrPQdf243baLdrstzpXl5WVxPsTjcYRh +CF3X4fs++v0+nj59KjoXFApFJJMJ0eXhVfs2miJhb28Pjx49wsrKCp4+fYpmswlZlkXXBkVRoCij +edajNv7tdhuu4+Lg8AB3794V6xsNhBgbG3thfZ8+fXpqfWOxmFjfTqeDRqMBz/OwsrIC27bR7/eR +TqchyzIGgwEA4OjoCD/++CO2t7eRz+fFMR4FyIqiwPd9NBoNPHjwALquiy4Eo8rxZ4NGarUaNjc3 +kclkEAQBJElCNpsVIfJgMMDa2hp6vR5yuRzy+TwajQZs2xafOVH7862tLbiui0wmg7m5OfE5eHJ7 +Ry3cM5mMOLcijuNge3sbADAYDGAYBvK5vGjTPxgM0Gw2UavVsL6+jnQ6DUmS4Lou0um0+KyRJAlH +R0dwXRfFYhEzMzMIggCJRAIAsL62jofLD/HkyRPs7u6K49MwDNENQZblU1Xpl6HVeRgG6PV7qFar +sPqtt+56QPQy0bVR9Lsg+h0di8XEVDI83oiIiIiIiIjoomPwTUTvnZh/+Pgr8IMzXyfJoweoyWQC +S0tLME0TnU5HBDDb29v44YcfsLS0hFgshm63i8ePH+Pf/u3f0Ov1RHvs2dlZUSEuyzLq9Tp+/PFH +/Mu//Iuowrxy5Qry+Tzy+byYf7tWq2FrawsrKysiGI/H40inM0inU3jy5An+1//6X1hfX0cYhtA0 +DdPT05iYmEAymRTzXDebTRHY27b9s7dfNKdytVqF67rY3NxEuVzGF198gcXFRSSTSdGKuVQqodvt +4rvvvsOPP/4ITRtVTy4uLooBA1Hb96jN9Q8//ABN0zA2Nmrtns/nEYvFXrlM2WwWX37xJRKJBP75 +n/8ZrdYocFlaWsLXX38tguqoaj0ej2N/fx/3799Hq9USwXbUBWBubk4EwDs7O9jZ2cH6+rqono9a +Y7dbo2rk//2//zdkWUY8HsfCwoIIKm3bxnA4RKPRwObmJu7evQtFUVAsFkUof3LO8Fcdr77vY3Nz +E//v//0/rK6uIhaLoVgsiiAwFouJoHBnZwdbW1uoVqtIpVKYmpqC7/soFouiHfbu7i6Ojo5EkBlx +XVcMjIjaqEuShFKphNnZWSQSiZcGudFcx41GA99//z3+9V//Ff1+H2EYnmrdDwC+74tt02w2UalU +MD8/jzAM8U//9E84ODhAsVjEF198ISq8U6mU2Ie1Wg2PHj2Copyen9z3fRFefvvtt3j48KGoLo/2 +byKRENMCHB4eYnNzE7u7u4jFYpicnEQul3thQEyn0xFtvIvFIq5evXpqe21vb2N3dxfpdBpTU1Mo +FAqYnJxEPB4XgwVeJgrr19fX8W//9m+4d++e2O9RW/JcLifO6SiMj/bP0LLE+t67d0+07J6dncXY +2BiSySR6vR663S6Ojo6ws7OD7e1txGIxTExMoFgsQlVOX3L1+33s7OyI4+b27dtif7VaLfT7fdy7 +dw+xWAzpdBqpVArpdBoLCwvwvFGoHb1f9BkWdQyIjuuTx1yj0cCTJ09QKpUwPj6OK1euiGMj6myx +sbGBVCqFTCYjOkKUSqVTn5nVahX1eh1TU1O4c/sO4vG4CJFTqRTGxsaQy+WQTqdFQK+qKtrtNg4P +D9FoNFCv18VgjYmJCVxduvrCPrNtG9VqFQ8ePDieQqKI2dlZ8blxeHiI7e1tbGxsYGJiAjMzM8eD +ZypQVQUPHj7AP//zP6NWqyEIAmQyGYyNjaFcLkPXdVGF/+jRI7Tb7Z/9uf2hBL6PTruDnZ0eFLhw +HOe9/N4hiqZriMVi4vdA9PsvHo9DhQpFVX7+GxERERERERER/YIYfBPRe+c4jqhmVRTlzCo6SZLE +/MxRUDlWGsPy8jIymQyq1SoODw/x448/ivl92+02VldXcf/+fTF3+LVr1zA7O4tisSjmta1Wq3j6 +9Cnu3r2LK1eu4ObNm7h27RoWFhawtLQkguoHDx5ge3sbe3t72NrawtbWFjKZzHEFoI+1tTX89NNP +ODg4wMTEBKanp3H16lXcvn0bxWJRhEFPnjyB7/tQFOWVladvqlKp4KuvvhLLKMsyisUi7ty5g9/8 +5jcoFAooFosiuHv48CGq1Sru3r2Lubk5jI2N4erVq7hy5QoWFxfR6XTQbDaxvLyM7e1t7O/vi/WN +QqBCofDKcDiRSODqtauYmp7C9vY2vv/+eziOg7m5Ofzud78T7ZhjsZhoi/z999/DMAwR0jUaDYyN +jWF6ehpTU1OiwjIKnKMwbGNjQ1TdNptNrK6u4u7du5iamsKNGzdOrVsUMq+srGBnZwf7+/soFArY +3t7GxMQEdF1HPpc/1V7+eWEYimNnZ2cHy8vL2NrawtWrV3H16lUxz3A2m0Wz2USz2cS//uu/Ynl5 +GZ1OR2xLx3HEcdhoNHB0dIRarYZ+vw/XdcXxEVW01ut19Ho9uK4rKvQnJyfh+wHC8OzBIo7jIAxD +HB0d4cmTJ/j++++RSCQwOTmJyclJ3Lx5E59//rkI9Wq1mqi2Hh8fx1dffQVN0/Dw4UMoioJcLofP +PvsMf/EXf4F8Po9SqSS2x5MnT0QVciSaOzkIQrGtnj59isXFRczOzoptVigUcHh4iGq1ij/84Q9Y +WVlBu93GxsYGtra24Ps+stnsqXXzPE/M3b6wsIAbN26I+aOr1SparRZ6vZ4IlicmJmCaJiqVyivP +p8AfLXOv18fW1hbu3r2Lhw8fYmJiAlNTU1hYWMDt27cxMzMjzun19XWsrKxgMBgcL1cXOzs7olp8 +cXFRtEK/du0axkpjODg8EB0koqkVornPo4EJJ0UDE2zbRj6fx+3bt8W869vb27h37x52dnagKAqy +2ayYL3xpaQm6rsOyLFSrVTSbTRwdHYnpEVqtlmjJf/K9HMdBr9fDxMQEZmdnMTs7i6OjIxwdHeHR +o0fY2dnB4eEh0uk0crkcFhYWsLCwgPn5ebGc9+7dO3Wu1uo1ZHNZaOqoU0bUflzTNExOTmJsbExM +R7CzsyM6FDx+/Bjr6+tIJBL4zW9+A8/3XhggFc0j32q1xCCnpaWl0WCqIMC9e/fEFBD7+/vY3NxE +PB4Xbf6jz2/f9zE5OYmpqSmxvwzDEIOBms0mtre3RTeHiy4IAnTaPfi9HhyrKwZnEP1chmFA13XR +RSeTyaBUKmFsbAyZdAZGbFT9rev6KCRXNUiy9F6ue4iIiIiIiIiI3hcG30T03u3t7eGbb77B6urq +S+eRjtpdZzIZTExMYGlpCVevXsX09DRu376NtbU12LaNn376SXyP4zjY29tDr9fDwsKCmNc2mus3 +aje+vb2N4XAo5uq9efMmbt68iWKxKFrlJpNJ0d53a2sLmqaJysl6vY5CoYBqtQrbtkXFeDT/8OTk +JFKplKg8jILHqHLz54rakkeDAqL/FrX4jSoso3bAe3t7sCxLtCO/efMmbt26hWKxKNosJ5NJ2LYt +1tcwDFSrVayvr4/ak0/PvDIclqXR+xv6qIVx1AY7CqhPtkz3/QC+54uH4YlEAouLi/jyyy+xsLCA +xcVFFAtFBMfhblQRXK/XEYYharUaDMNAEARinuAoGL5+/Tru3LmDUqmEXC4H0zRF9f3c3By2trYQ +i8VQr9exvr6OZDKJ6elpKHh5lZpt26MA8PBIVIemUinMzc3hyy+/FBW78fiodXo2m8XW1hbW19dx +cHAA13WxtrYmKuMkSRKhaavVQrPZRLvdFm3Ie70ednZ2sLe3hzAMRSeCTCaDeDwOx3HgOGfPNRxt +j/X1dTH/e6FQwOeff47PP/9cHJ/R+RJVrM/Pz6NcLmNiYgKtVuvM48qMj5bP93wEQXjmQI5oUEvU +vt5xHMTjcczMzOA3v/mNqIBOJBJIJpMolUrodDoimLVtG48ePYLjOFhcXDz1s6Pw9/bt25ifn8eV +K1egazqCMEA+n8f29jY2NzdhmiYsy0KtVhMDKF7F9VzUqjXs7u1if38fw+EQuq5jdnYWf/Znf4b5 ++XlMT0+LOZ6jltqVSgW9Xg+pVApbW1un1nd6ehq//vWvMTs7i3K5PGqtnzBFB4ajoyMoigLHcfDk +yRN4nocrV66cWq5cLodbt27hiy++wPz8PBYXF+H7PiYmJpDP53F4eIgnT54gmUziypUruHHjhghu +o3b1BwcHYkqIMBzNs12r1ZDNjsLoSCwWw/T0NL766itcvXoVi4uLKJfLKJfLYk72jY0NyLIsPjOv +Xr2KpaUlTE1NwfM80WI/GsQQdR7odDqiGj2quo7anIs5t8NQVOjbto29vT34vg/LstDv99HtduG6 +7ql9GS3zF198IQagTE1OAQCCcDRHfDS1g6IoaLVaYgoMVVVRr9fheZ4Y3PH111+jXC6jUqlAVVW4 +rotcLof19XUxBcbJlu0XVeAHaHWbaAz20WqMBi40m83zXiz6CERTPUSDVQzDEJ/l2WxWdJmJumTk +83nR4YWIiIiIiIiI6KK4+E/4iOhSieZJbjabIjg7q4pOkiQUi0WUy2XcunULpmnizp07mJmZwRdf +fAFJkvDDDz/g4cOHImSVZRl7e3tiPtwoNIrmdG21Wtja2sLu7q4IgsfHx3Hr1i3cuXNnNOexqiGd +TouW0aurq9jc3BTBd1R9GVXs2rYN0zRx9epV/N3f/R1M04Rpmqcq2aenp+G5HoaWhX6/97Or76KK +RV3XTwWUuq6LcNmMm9jb28P+/j729/fF+lYqFbFdNE2DpmrwA1/MYTs3NyfWN5r3d2pqCkEYvDIc +luRRmBWLx8R8v1HAFYXeUTA8Cskc8Zp4PI7r16/jP//n/4zx8XEUCgUkk0nxs7e3t7GysiJCr1qt +Jqo9o9biuq6Ldfvyyy/FoICo+hMA5ubmsL6+DlVVUavVsLGxgcnJydcGo47j4OjwCE9WnqBeryMI +AqTTaczPz+PXv/410um0CPuz2SzCMMTW1hbm5uZEJe36+jo0TcPs7CySySQMw8BgMEC73RZV4sAo +zOt0OtjdHYWwUSBYKpWQTqfFvNavCr6j6tZ2uy3m7f7666/xV3/1V2I/ROdiEAS4cuUKHMeBqqpQ +FAXD4VAcV5Ikida2RswQwbvne2KdT3JdVwy2ODw8hOu6ME0T8/Pz+N2f/w6lsZLYN9GczZ1OB/v7 ++/A8D7Zt4/Hjx1AUBWNjY6dalI+NjeHXv/41/uav/wbFUhGFQkEEkfl8Hk+fPsXKygpUVYVlWajX +6+j3+68Pvl0X1VoVq6urODg4EMH3/Pw8/vqv/xoTExOiXXoU0k5PT4tlX11dFYMconmi5+bm8Nvf +/hbj4+NifcfGxkTV+sHBAWzbFsG3LMuiJXokn8/j66+/xt/+7d+iUCigUCiI4z6fz4tuF+l0Gjdu +3MBf/MVfYGZmBrMzs9CN0bLu7u7i+++/F8dN1AFB07RTA46i9f1P/+k/YWlpSZyDUXeGarWK3//+ +95BlGePj4/j1r3+NmzdvYmZmRsxhDozmHf/973+P7e1t0dmg3W5jbGwMsVhMTDsRfTYAwHAwxGA4 +gO/7AEYVy/fv3xcV71GV//OdQVRVxcLCAv72b//2uItAEblc9tS58PTpU9RqNSiKgna7DVmW0e/3 +IcsyarWaCL5v376Nv/3bvxVhniRJCMMQjUYT9+/fRyaTge/7L7Tfv4j8IEC73UT7YAf7uxujyvta +7bwXiz4C0bVW9M+yLIswPJ1OY3JyEjMzM7h16xZ83xe/Uxh8ExEREREREdFFwuCbiN470zRFiBdV +Lp+lWCyKNprJZBKyLKNQKODKlSvodDrY3NyEqqpoNpt49OjRaI5JVcWVK1cwNzeHcrmMTCYjKhuj +1rX1eh3tdhuDwQBbW1v47rvvUK1WX3j/w8NDrK6uotVqwXEcUXEYhqEI6QzDQCaTQT6fH7XtVTXI +yukwPxaLIQxDxIZD+L73s4Pvk0H/SbIsP/tSZBE8nVzf7e1tfPfdd2g0Gi/83Gq1ipWVFTQaDRFQ +5/N5OI7zRst0ctmeX86TgxxkST71OlmWkUgkUCqVkM/nkUqlTj0oj+YGj1ozD4dDdDodSJIkwuNo +Xu0//vGPZ87HW6/XsbKygmazKaphc7kcLMs6s9X+Sa7rotPtoFqtotFoiPbjjx8/xv/5P//nVMvo +yMOHD7Gzs4NWqwXXdcXc0FEFeiaTgWmaooJ9b29v1Fo8m0Ov18P+/r6YCmBiYgKlUgmJRAKKrLyy +3fLJObuDIIBpmshms6JqXJGVM+dgDcMQgR/A871X7kNFHoUcL1sG3/PR6/VQq9UwHA6hqip0XUcm +k0Eul0MqlXrhe6NKwWw2i36/L45X27ZPBd+apiGbzaI0VkImkxGfCSePkUQiIQJ1y7Lged5r96/v +j5a5Wq1iMBhAVVUkk0nkcjkUiyVks9kXqtujc1qSJARBgEajIQYMRC2AX7e++XwevV5PVPxblnUq ++I7C6WKxiHQqjYSZEO+dy+UQi8VEgJxIJJDNZkevSybEMZlMJhGPx6FpGmRZhu/7sG37he0iyzJM +0xTLdfLzOZFIIJFIiIEOhmEgnU6LObpPLrNpmjAMQ4Ta0bHv+z583xdzo3c6HTHH+2AwgGVZsG0b +lmWJzyzXdeH7vvisVWQFsiK/sMzFYhGFQhGZTBqmaYq/j5bbNE1IkgTbttHtdsVAiOh8TKfTyOfz +KBaKL7RlNk0LsVhM/I66DK3OZUmCoRtIJEfHxHA4fO3gD6I34fu+mH4k+p3muqN55C3Lgu/74nxu +NpvY2trC/Pw85ubmkE6lkUwlT32mExERERERERGdBwbfRPReSZKEfD6PpaUlTE5OIpPJvDC3bSRq +j1upVDAxMQFg1P5X0zR0u12srKwgn8/Dsiysrq4ik8lgfn4eX331Fa5fv458Pi+CaACwLEsEL81m +E51OB/fv30e9Xj+z3fpgMECtVkO9Xkc+nxcVprIsIwgC+L4P0zRFiKkop4OZ59f7Q7OtUdATrW+7 +3cbDhw/RbDaRTqdfeH0053O9Xkc2m0UQBLBtG67rvsO7v7mTwerJ0DVyssIdgGipDEBUsLbbbSwv +L6PVap15PEVVq/V6HZlMRsxTHVWZvkoQBGLe7WhbdrtdfPPNN9jc3BTV0Sc1Gg1RUVooFET1uWGM +qqZFEK0oaDQa2NzcRDKZxNTUlAi+j46OUCwWMT4+jkqlcirUe+k+t210Oh3R3jmbzYoQ86wK7ZP7 +4H3wA18MRvA8T1SYm6b5woCQiGEYyGazyGQyGAwG6Pf7Yt7zs46Ts7pERJXphmHAtm3RseFNgu/A +D9Dv98UghWhbJRIJ6Lr20qBfkiQEfiAGYriuK0JS0zRf2s0iGggQrW9U0ex53svXVz4eJKLIUKGK +bg/R+RK9Nnrds58hi9bE0eujwTtnrc/JZY5C9WjbRoHw8687SVEUqKp66nMyej/btrG9vY3l5WXR +eSPqmmHbtvh+z/Owvb0t5quPQvNo/c9aZkWRzxwIpOu6qOB2XRfD4fDUOZ9KpZDNZhGPx1/62X3Z +SLKEVDqNtDGOlKmLKQWIfq4o6I6mIOj3+xgMBuL8bjQa6Pf7aDabWF1dxezsLL744gtYloXp6WlM +a9MMvomIiIiIiIjo3DH4JqL3LpfL4dq1a7hx4wbK5TLGK+Nnvk43RmFnVLUKQMwJ3mg0MDY2hkwm +g8PDQ1SrVTiOg88++wy3b9/GzPQM0un0qepW13XR7/fR6/UwHA7hOA7q9ToGg8GZVedRuO37o/mo +DcMQ1cJR5WA8HhcVyq8KFoHTVdEfguM66PV6p9Y3qkx90/U9We34S3pVmBbNMx1VkUYVoFHrZsuy +Tq3bWQ/Wn1+3KCR9kzl7gyCAZVli/mzbtjEYDHBwcIB2u33mMkfvpeu62JZRAJxKpZDP51Eqjdp+ +1+t1bG9vY3x8HI7joNPp4PDwEM1mExMTE6I99Ju0i/U8D8PhUFQfR1XQhm68EAz+EkZt2B30+334 +vg/DMES1/svePzrHE4kEJEnCcDgU1YNnHSdnDY6I5pPXdV2E3Z7nvVGlqx+M2tFHlfxRlbNpmiLA +femxEQZifT3PE/s5FoudOSAiWt9oXtyoEjmqTn/d+kqSBEVVRHvh132WyLIkwuhoeV4WfJ/1flHw +frJjw1ndHZ7fD9F2i46H4XCIZrOJtbU1/PTTT1hbW8P+/j7q9Tp8fzTVQnRORu3Io/0YVfCftR9e +djwAp0N4SZJEtXs0oEKSJFEVHoXjHwNZlpHJZJHWDbjFLMbGxsRAIaJ3FZ3L0Wdlu91Gq9VCs9lE +q9XCYDAYdUfpdNBut0U3E9/3oSgKfN9HKpUa/d5V1DM7jxARERERERERfQgMvonovdP1URVaFP6V +K+UzX6coKlRVEVV7J51V/XmyBbnruS8EZ9H8vKqqIpVKoVAoYGZmBnNzc8jn8y+8f1RtGAQBstms +mL/56OgIR0dHp5blfXndXNpvKwxDEYAWCgVMTk5ifn4ehULhzNdGAXHUqnlychJTU1OQpYtVDRnt +S1mWxbE0Pj6Oubm5M6sbn1+3bDYr5iN9WUD5/HuFYYhYLIZsNotEIoHZ2VksLCycGZ5H76VpGnK5 +HHK5HJaWlpDPj7Z7sVjE7OysqDTe2trCzMwM2u1RaNButzEcDmGapmjbH81V/ybbJvKyUPAiO49l +fn6bnfzzl36/59/3XVyUfXzWvuv1etjc3AQAPHjwAMvLy/A8D1NTU7hx44YY9BBVpQ8GA/z+979/ +r2HtyWV6fo7iy3iOvIoiq8hm4pgeK0NXAlGlS/RzRdMWWJaF4XCIwWCAVquFVquFarWKg4MDHB4e +ot1ui+kMnj59CsuyYFkWdF0/dT1FRERERERERHQeGHwT0XunaZqYQ3esNIZKpfLS10ZzQp/Vhjaq +LoyCiyj4dhxHzEV5UhReKoqCdDoNz/Nw584d/O53v8P8/PwLP//k90fViNVqFX/84x9F8P0+QrJo +ud63l63vn//5n+PKlStvtL7xeByZTObCtQGOguwo1B8bG8OtW7fwu9/9DlevXn3tukVVxul0+o2q +voFRmG0YBnK5HOLxOP7sz/4Mf/mXf4lYLPbS94vaLeu6LuZF7vV6KJVKWFhYwNraGra3tzEYDI4D +g1GIEM1bHo/Hsbi4iHK5/EYV38+//3kFetGxdx7v/67veVb4/Ut8b/Tan/N+P/f73tW7vF+v10O3 +20Wn08Hy8jKePn0qztevvvoKU1NTmJiYgK7r8H0f+/v76Ha7ePz48XtZ3rOq0gGI0PtDdEL4kGRF +Ri6fx9xcCfnMqGvBh+jaQR8/MQgsCOEHo0GBUcC9vr6On376Cffu3cP6+rqY8sK2bezt7UHXdZRK +JSSTSTENBxERERERERHReWDwTUTvXdQ6V9M0qJr6VnM+drtd9Pt97O7uol6vo9frwTAMlEolJBIJ +DAYD3Lt3D4ZhYGJ8AtlsVrTp1TQN8XhcBIiO44iHsdPT0wBeHuxEPyMIAmQyGSQSCXS7XQyHQ9Fu +O/CDl85j/OznqKLtcBTWR/MRx2Kxl4b1z7cnPqvF78nKZACn1leSJDiOA03TUCwWMTU19cpKx2h9 +o/bRbxJ4vWyZgiBAGLzfYD+aR9lxHMiyDNu2oWkaCoXCW6/b60ThdTKZRKPREF0AEokEJicnYZrm +S98rGrQRHX+6NgrBS8US5ufnUa/XsbKyguFwiK2tLTx69AiNRgOyLCOdTou5oBOJBDT19eGVoiiI +xWKIxWJot9uiPbvjOgiC4LXVrc/vw5PHVPAG+1CSJNHKu9PpwLIssX9e1nY8ahXe6/UQBIFY/tdV +4r8vJ5dZURQxt3TUfvxVgwhkSRbfq6qqWM9XzR9/csqFaH0Nw/hg6/uhRfMBN5tN2LYNVVWRzWYx +NTWF69evj7p+lMuQZRmu68J1XTFH+vumKIpoax59fg+Hw9dWRP8SA5N+Mcdzs8diMdHGnfMq0y8l +nU5jOBiK88o0TSSTSei6joODA/R6PbRaLWxvb+PHH38U136Tk5MfXbcFIiIiIiIiIrocGHwT0YXS +arWwu7uLtbU17O3todlsIpfLYXx8HJqmodfr4fe//z0ymQyuXr2KcqUsKm5jsRjS6TTi8Tg8z0O7 +3RbBXBS0vSpskSQJ8XgcyWQSmUwGR0dHoqppMBjA8z2oUM8Mv6PqV1VVxNzSkiSJ+Wsd24HrumdW +DwdBcCr4PhlOPh9SngwXDcNAOp2GaZpifYfDIWRZRjwef6P1lSQJivxmgZwsyQjl8OzwOwzea3gU +BdFRwBiFWNE+ept1e92Dd0VRYJomMpkMVFUVgajnedB1Xbzf6wJlWZYhyaOQtTRWQhAGePr0KVzX +Rb1ex/r6OvL5PKrVKmKxGFKpFHK5HMz48VzTiozAf/Wc1dGc2tE+73Q6IoCO5lo9aznDMIQkS5DC +F+dafzbw4vXzZSuyIroEVKtVMf/6YDBA4AdnVoFbliWq3AGcml/7Q1CU0TJH1f/D4VAE057niQEt +Z20zWZHF+u7v74uQt9/vi3P2+fW1LVvMjxuGoWj1/aHW90NzHAfdbhftdhuKoiCfz2NiYgIzMzOY +np5GPB6HqqgIjo+vXzJkjgJhRVEQBIGoRn9V8H1WhT4RjeiaDikhYbwyDl3XMTk5KQZrPX78GJub +m2i32zg6OsKPP/4IABgbG8Pnn38uBvsw/CYiIiIiIiKiD+njfApLROcqmv/YcRw4jgPbtl/5ehFS +KgpqtRpWVlawtrYmgqN8Po/r16/D933RZnNjYwP7+/uYmJhAJpMRAWU+n0cmk4EkSej3+6IFb7/f +RywWO9VOOmqdfrJtuu/5ME0TpVIJm5ubsCwL7XYbjUYD1WoVZtxELD4KVqJKWdd14TiOCEtjsZiY +z9bzPAwGA1RrVQCj8Nj3/dFrXQ+tVgv9fl98fySa31yWZaiqijAMxZybw+EQjuOIFufJZBIAMBgM +RAAVrW8sFnthjvST6yvL8qgaVX19+B0Fp6qqitA5qm4dDAawbXtU1em58FzvzHb0byqaa9uyLEiS +JNat0+mg1+shHo+/8bq9rko/mhN+bGwMGxsbcF0Xtm2L94vCtJNVldGAhpMDFqK/1zUd+XxerEO0 +/Pv7+7h//z7CMEQ8HkepVBq9Lh5741bFhmGI+VOjY7zZbKLRaKDRaIh9fnKbRNMD6LouKo+jingA +sG0b/X4ftmXDdmy4riu+7/n9p6gKUqkUSqUSdnd3xbZqt9toNptQ1FGVvaIo8H0fvu+j1WqhVquh +2WzCMAzk83mk0+kzB4H8EhRFQSKRQKFQQDweF8dss9lErVaDpmninD15Tnuuh063A1mWUSgUxGCD +Xq8nPhOiattT69serW+0P3K5HDKZzAdb3w/N933Yti3O1Xg8jlQqhXQ6jVQqBQBwPVd0z6jVaqOB +EsHrB1q8rWiajaiLRb/fF/uq1WqJz9Pod1Sz2XzW0eN4ABIRPaOoChR11GkkX8iLKUg0TYMkSej1 +etjd3RUDsOLxOG7fvi269Rj6m11fEBERERERERG9Lwy+iei9cxwH7XYbtVpNhEKvoqqqCDI3Nzfx +ww8/YHt7G6Zp4ssvv8S1a9dw7do1dLtdHB4eilD8wYMHUBQFV69eRTqdRjabxdzcHBqNBpaXlyFJ +Evb29vDNN9+g2WyKuWaj4M+yLDSbTbRarVNVzJIkYWlpCRsbG1BVFb1eD48fP8Y//dM/YXJyEuPj +40gmkyLYPzw8xOHhIVRVxcL8AhYXF5FIJBCPxzEcDrG3t4dvv/0Wk5OTmJychK7rIqj84x//iNXV +VXS7Xaiqinw+L5YlCuRM04Rt29jd3cWTJ0/guq4I1aO51KMQ+ODgAN9++y263a5Y3yiotm1brG+0 +nslkElNTU5ienn5tBTUA8cA7qoJutVp4/PixaB+dyWREOD8cDl/aDvp1kskkpqen4TgOHj9+DFmW +cXR0hN///vfo9/tiW2qaJtat1Wqh2WyK5U0kEqN1m5p+5YN3XddRLpcRBAE2NzcRi8VQr9fx5MkT +/M//+T8xMzODyYlJ5PN5EaJHFb2DwUDsr0qlgumpaWRzWRGCZzIZFAoFHB4eot/vY3V1FcViEWNj +Y5ienhYB9ptKp9OYmpoS2yUMQ1SrVXz//fcIwxATExOYmJgAMAq0e90eDg4PcHR0hOnpaVy/fl20 +g45C4P39fTx58gSO40CSJYRBCMu20Ol0YNv2C/OnF4tFJBIJ7O/vQ9M0tFotrK+v49/+/d8wOzuL +sbExJBIJ9Ho9dDodPHz4EKurqzg6OsLi4iKuXr2K2dlZpFKp1w6KeR80TRNzru/s7MAwDNTrdayt +reFf/uVfsLCwgImJCeRyOTjOqDNDrVbD4eEhut0uMpkMpqenxTy2w+EQGxsb+OabbzA7O4tyuYxU +KoVut4tut4uHDx/i6dOnYn2XlpbE+n6MVcVR2Bx97jebTezu7mJzcxOVSkV0GKhWq9jc3MTTp0/x +9OlTOI7z3pclHo+jUqkglUqJbguHh4e4f/++6OSRTqfFYI1oMMre3h5isRiKxeJ5b06iC02SJIyN +jeGzzz7DcDjE/v4+tre3xYC+TqeD/f19PH78GOVyGeVyGaZqnvdiExEREREREdEnhME3Eb13juOg +0+mgWq2+0by2MSOGbC6LTCaDra0t3L17F51OBzdu3MCdO3dw9epVXLt2DQcHB3j06JGY3/jBgwcA +RmHgtWvXRABcr9dF1ffBwQF+//vf4+joCJ999hnCMISiKFAUBd1uF9vb29jZ2REVsLlcDouLi7iy +eAU//vgjNE1Ds9nEysoKfN/HjRs3cPPmTRSLRVHp/OjRIzx58gSmacIwDNy+c1sE32EYYn9/H999 +9x3q9Tr6/T5UVcXm5ia2trbw+PFjbGxsIAgCpNPpZ9XKkizmc47H43AcB3t7ezBNc9SaO24iCAOk +Uink83lRyX50dITvvvsOtVoNN2/ehO/7YvBBr9cT6xu15h4bG4Ou6ZianAJeM+VuFPCebP/dbrex +srICYFSNbNu2qP4cDAbvHHynUinMzMzA933kcjlIkoSjoyN8//33p9Ytams8GAywvb2N7e3t0faT +ZZRKJaiqOhrsgJcfh4ZuYGxsDKlUCg8ePEA8Hodt21hdXUW/38e1a9fw2WefYWZmRhw70cP+qIpU +lmXcvHkTmUwG+UJeVALncjkRFHc6HRweHkLTNFy9ehVzc3PI5/JvNddxKpVCJpMRc9EDQK1Ww08/ +/YR+v4+bN28CGFWkDwYDHB0d4fHjx3jy5Am++uorFItFZNIZcVz5vo+DgwM8ffpUVOtKkgTXdc8M +vqMW9Lpu4PHjxzAMA4PBAJubm/j222/RaDRw7do1EfZH77++vo5Op4ObN2/ixo0bmJycRDqdRrVa +ffcPmjekqipKxRIM49kyW5aFjY0NAEC9Xsdnn32GqakpDAYDDAYDrK6u4smTJxgMBvjtb3+L27dv +Y2trSwTfW1tbYn2vXr2KSrkiBhg8evQI6+vrqNfruH79ugj60+m0aPf+MdE0DalUSnQJaDQa2N/f +x+bmJorFoqiyfvr0Ke7du4dHjx5hf3//Fxn0YJomKpUKxsbGxFQJR0dHePDgAYIgQKVSwfj4uKhS +3drawvLyMg4ODkSHCSJ6uWiQVz6fh+M4WFlZwaNHj0SHlG63i729PaysrIwGw6UzME0G30RERERE +RET04TD4JqL3rtVqYWVlBf1+H+l0Gul0+pWvTyQSKJfLGBsbw/b2NgaDATRNQ6VSwa1btzAxMYFS +qQTf9zE/P4/r16+LSmtN0zA9PS1aFhvGKMS8evUq9vb2RPvr/f19hGGIer0uKr6jgL7T6SCXyyGX +y4k5vnO5HBYWFnDnzh1sbm5ClmVUq1XIsoxutwvTNEXFd71eR7vdhmEYAABN1VAul3H16lURau7u +7sK2bdH+OArNAYiA3DCMU5XnpmlicnIS165dE9v18ePH6HQ62NnZQbFYRLFYhKIomJ6exhdffAHH +cU6tb6PREBXfUZjZbreRzWaRy+VGofhbtiEtFApYXFwUofbW1hYsy0KtVkOhUBBzUHe73XcOvqOq +8ng8jitXruCLL74QQfrBwQEAoNlsiorvaN06nQ7S6TRyuRw0TRNzrb+KrMhiXvbp6WncunVLVNR3 +u12sr6/Dsiysrq6KYydqdx0EgTh2TraAj/ZjKpXC+Pg4KpUKbNsWXRBExXcu+1bBdzToYHx8HNeu +XcPe3h4cx0EQBNjd3YXv+6jVagjDELZtYzAYoF6vi7nSwzCEbuji+y3LQq/Xw5MnT9DtdrG/vy9a +VDebTQyHw9PbSpYRi8VERf7169cRBAF0XcfBwQEcx0GtVoNpmmJ+5UajIeYzn5+fx8zMzGj/qG/W +3v3nUhQFcTMOWZExOzuLW7duiXVpt9tYXV3FYDDAysqKOKdbrRbq9Tp0XT+uaE5hcnISN27cEHO/ +Hx4eiurwkxXf9XodyWQSyWQS8/PzmJubQ7FYhKZqH2XwnUgkkMvlkM1mcXh4iN3dXTiOg9XVVViW +BU3ToGkaut0ubNuGaZrivH3fomkLSqUSZmdnsbS0hH6/j+FwiCdPnuDw8BBbW1tiYEiv1xPH75t0 +JyEiiM/FfD6PyclJzMzMYGdnB8PhUFznbG5ujsJx9/13diAiIiIiIiIiehUG30T0XkXhsuu62Nra +gqZpp+ZGPksymcTk5CSmpqawvb0NSZJQKBQwNTWF69evI5FIwDRNpNMZXL16FcPhEPfu3cPy8jIc +x8GVK1ewu7srAshSqYQvv/wSpmlidXUV6+vraDQaaDabuH///qk5xQ3DEK16M5kMyuUy8vk8zISJ +q1evwvM8PH36FBsbG9ja2sL6+jrW1tbEXMDR8ieTSaTT6VHFrCxhamoKv/rVr2CaJtbW1rC5uYlW +q4WNjQ0kEgnk83nk83lks1nRJjyRSJzaLqlUCleuXEG73cb6+jo2NzdxcHCAzc1N6LouKpFLpRKu +XLmC6elprK2tiWrTVquFhw8fAoBY3yjgSSQSSKfToi21JL95u+1KpYLbt29DlmVsbGxge3sbh4eH +ePz4sWg3Pz8/j263C8/z3ingipY3n8/j888/h2EYWF1dxdraGur1Oh4/fozl5WXx2mg+b13XEYvF +kEqlUC6XkUwmIUvyG72XLMmYm5vDb3/7WxSLRbHfDg4OsLe3hyAITrWh1zRNVHRHFXDR4IdIJpPB +xMQEpqam0Gq1cHR0BNM0MT4+jpmZGWSz2bcK26IuAMViEV9++SXi8TjW1tawsbGBer2Oer2Oe/fu +iXNRVVUkk0mkUikkk0kxoGBhYQG/+tWvsLq6iq2tLVHBbhgGJicnsbCwAFmWz2xXHx1H09PT+M1v +fiM6Nezs7KBareLRo0entm1UvT81NYWbN29iamoKuq7/Iq2uX7Z/dU2HLMtYXFyEbdtiPvfNzU1s +b29ja2vr1DkdzVMdVQ4bho6ZmRn85je/QTqdxubmJnZ2dlCr1fD48eMX1ndqagqTk5Oikjzq2vAx +SqVSqFQqUBQFR0dHODg4QLPZxNraGlZXV2EYBmKxmGj7v7i4iG63KwawvE/R+ZHL5cT0GOvr6zg4 +OMDW1pYYuJJIJJDNZmEYBkzTRDY76jjy/PlLRGeLphOZmJjA1atXYVmWuPZrNpvY2dnBzMzMR/u5 +R0REREREREQXF4NvIvpZJElCiBC6riObzWJ8fBwARMvrN5FMJkXg5Hke8vk8ZmZmMDs7i9nZWfG6 +VCqJhYUFGIaBVquF1dVVOI6D4XCIXq+HZCIJAMjlcrh58ybGx8cRj8dF2+1arYZGoyF+XhSQ5PN5 +JBIJjI2NoVKpIJvNIhaLYX5+Hpl0BsViEZIkoV6vo9lsotlswrZt0cI3Ho8jl8uJoDVqBXrnzh2o +qirm+bZtG71eT6xjoVCAYRhIJpOi1XkUniuqAtM0MT8/D0mSEAQBjo6O0Gg00O/3xc/o9/uYmZnB +9PS0aKntOA76/b4IQiOGYSCXy4mwp1QqHS9z6q32ealUEi21+/0+tre30e/3xZzXmUwGU1NTMAwD +mUxGtMdWFAWKrLxQgW0YBlKplAiBFUU5DhsN0ca+XC6LKvtoX9brdXHcRG3F8/n8c+uWfKNQX5Zl +QAYmJiYQj8dFJX3UnrzZbIoKfWDUXj+fz0PTNGQyGYyPj58ZfKfTaczMzKDdbov5j8fHx8Vc8bqu +v7A9JEmCqioinHMcB+l0WgxaiPZfdIxHc2V3Op1T85wriiKqcSuVCgqFAmKxGGKxGGZnZ0XQW6/X +cXR0hMFgAM/zoGkaxsbGkE6nkUgkUCgURDeEKPDXNA2TE5NQVRXZbBbAqM1+1FHAcRzxXlEHhlu3 +bmFpaQljY2MIggC9Xg+6rotANHqPs6r0ZVmGaZoinJRlGel0Wsxt/7rPKEVVoKgKpqamYJomyuUy +VFVFq9XCwcGBOHajKQ+iY2d8fFwEpJVyBfIXspiXvVqtot1uo9PpwLIs0T6+VCphaWlJrG+5XAYA +9Ho9GPromC4UCuJYjToFnDxOFUUR0xjk8/nj9vKjiuST6xuFuIVCQVTqR1MuqMqzz6Z8Pi/OxbMG +opx8XTabFVXZinx6UEb0OV8oFJDNZkV3jMnJSSQSCezu7uLo6Airq6vY29tDs9kU2yWZTKJUKqFY +LKLZbKJWq4l10zQNqqKOpnEwTbHe0XEfzRN+kqZp4hyJPjfS6bQYxLS0tDSq9j/+HVCtVmFZFlzX +RRiG4j2i3wXR8iUSCdF9g4heLhrEtbS0hL29PaytrcGyLHQ6HXH+u4573otJRERERERERJ8YBt9E +9LNJsoTx8XH89V//9Wvbmp/lZADteR5c10U+n8fi4uKp1+m6jnx+NCfyb3/7WxQKBUiShBs3bmBi +YgKpVEqEJIlEArIs48aNG0gkErhx4wa63S56vZ74eVGldzKZFFW5UfARLVcmm8Hc3BzCMES5XEav +18NgMIDruiKMKRQKKBQKmJiYwMTEBACI0CyaN3l+fh6u68L3fdGOfWxsTLSjDsMQ8XgcsVgMMzMz +IuArFktiDulKpYJmsynat8/MzGBubg6VSgXJRBKJZAI3b9xEIpHA1atX0ev10Ov1RDgcBUUn13dU +0Zp4q5AnqmiNAurFxUWxbrFYDBMTE5icnMTk5CSuX78OVVXxxRdfjAYUxGNQldO/eiYnJ/GrX/1K +hPyyLIt22KqiwjRNsZ9Nc1SJH7WVjtYtqmyOtnu0bqlU6q3WLQrrpyan4H7lolgsot1uo9frnZr/ +1zRNUbkfdSuIAtKTstksFhYWEIvFUC6X8dlnn2FpaQnz8/PQdR2qcjrklWRJhOG3bt0S7fjv3LmD +sbExMce7IiuntkssFsPVq1dPtdCXJElUhxcKBbFNdV0X5040z3u1WoXv+/A8D6VSSQxcmJycxOef +f475+XncvHET6dSzsDkWjyGfz4t9MD4+jk6ncypA13Ud5XIZk5OTYsoCVVHhS75o2f4Xf/EXYoBA +1OHh+QEBUcv7aP7x6Byfm5t7bUeJ5z9DojnSv/76axQKBbTbbfT7fTiOI7ZJJjMa8FIulzEzMyNC +51wuJwYMlMtltNttDIdDuK4r1ndsbOzE+o5BVVWxDuVKGX/+538uzsHofH1+fVOpFH79619DkiQx +kKdSqSCdTp8Ko03TxJdffik+Z6amplCpVJDJZJBOp3Hz5k04joODgwN8/vnnGBsbE4MLTpqfn8ff +/d3f4caNG7hx4wYWFhZGAyXisVOvm56exl/91V9hfn5eVLWXSiWk02kYhoGlpSXIsozr16+j2Wyi +3++L7VIsFjExMSG6YywtLYnuEdHnuqIouHH9Bqz/z8KdO3fwxRdfoFKpnLnMxWIJn3/+OQqFgqjk +LpfL4hiLjs1on3zxxRfiGE8mk2KwQHTOnKxKn5ycZPU30Wvo2rMBj6lUCoqiwPd9MfCu1+vB9Rh8 +ExEREREREdGHJYXRE2uiS8r3fGzvbGN7exu7u7vY3d1FtVoV4VcUSk1NTZ33on7Uoirjk8Hym5Jl +BaqqiJbPUYvmKAiOhGEoQl/LskQQGb1OVVTIiixe6/s+bNuG67pwXRdBEMDzPPHzJEmGosiidXM0 +J3RUgRkEAcIghOu5sG1bzKXseadbP2uaClVRoemaqMj1PR9+4I/mDLYd2M6o+j0IQsiyBFVVoWma +WE4AoprTMIxRa2ZFFoGy4zjH6+Idr18AwzBEq/YoHHddV6zzy9ZXlmVRPayqKhRZeat5vn3Ph+eP +BihE6xYE4fG+lETQ5TgOXNeFJEmjoNgctVQ/OY858Kw7gOd6ouo1Wi9VVcU2+hDrFoYhAj+A53ti +n/u+jyAITrX8joI2TdWgaqoIRqP/HnFdVwzmiNbTjJswE6YI1p4P5qNgdTAYiHnNTdN8oRr6ZdvF +94MTyzk6vqN9Em3T6DyK1tHzPARBiDAMxGtlWRZ/bxjGqBJWNyDJkjg/fN8XP8e2bfH+YRhAkmRx +rJ/cJ1GAGYYhXNfFcDhEvz+Aro8GZkTB+sntEgSBeI/osiVqb/821bmv27/ScVt8VVVO7VvDMMT2 +fpv1jbb9yc+vwWDw2vX1PR+9/miQTVS1bBgGFFmBrDyrfg6CQMxhLUmSeF9ZlqEqKmzHPg71XSQS +pqgwf/79ornebduBaR5/nqrqC8dzNFDHcZxTx5OmapBkSWyX6LiPtunJ7RIdV7btQNNUUaEeTUnw +psvsuu6ogttxxeeGqqrisyN6/2je9tH5EYrzIlq/aP9HHTxOTglx1nEVnW9Rt4udnR2sbtXwZKuJ +7aMhpFgFUrwMKPE3/tx5EwnDR970MZaV8PnVCr64Po5SPiWmeCD60LrdLo4Oj7B/sI//9t/+G/7x +H/8Rh4eHSKfTyGQy+C//5b/gv/7X/4rbt2+f96ISEREREb137XYbW1tb2N3dxc7ODvb29iBJkigG +mZ6extTUFDKZzHkvKhHRJ4cV30T0XkRBQS6X+8Xe4+TcyicD8Ze9VpZlETq9i6j9taIqr32/5ymq +AgWjAAXJd1/nKNCIx98sRNE0DaZpvvsbvum6qaP50X/OukWi8PJlTu7LX3rdTrbFfh8Vn9Hx+qb7 +L1oGSZJEBfsvsV3e9Dx6lWiwxbv+nGiQh2EYol36q94rHo+/1XZ82Xu+6/49ub3fdX2jz8nXra+i +KshkMq+9QZZlGalUCqnU2dMVmKr5RsdG1Jb+dV53rr7pPnrZ8r7NMkfH8Ov+/ucc40T0cqqqwkyY +YpqKaEBUNKApGhxERERERERERPQhyT//RxAREREREdGnIupGkUwmR10bZPnMLiRERERERERERB8S +K76JiIiIiIjojSnyqHNGKpkS0xhEFd+KojD4JiIiIiIiIqJzwYpvIiIiIiIiemOSLEFRFGi6BkVR +IEkSgiCA53nwfR++7yMMQ4RheN6LSkRERERERESfEFZ8ExERERER0RuTJAmapiEMQ6iqCkmSAACe +58GyLBGAA0AYhuLviYiIiIiIiIh+Saz4JiIiIiIiojcmSdILXwBElffJLyIiIiIiIiKiD4XBNxER +EREREb21syq5GXYTERERERER0Xlh8E1ERERERETvhOE3EREREREREV0UDL6JiIiIiIjonZwVcnNO +byIiIiIiIiI6Dwy+iYiIiIiI6K2cnMP7ZPgdhd4Mv4mIiIiIiIjoQ1PPewGIiIiIiOjy8T0fnu9B +kiSoqgpZ5pjaT0UYhgj8AJ7nwfd9EXyrqgpN06AoCmRJZvhNRERERERERB8Ug28iIiIiInorYRjC +9VzYti0Cb13Xz3ux6AM5uf89z0MYhpBlGaqqQtf10UAIhcE3EREREREREX1YDL6JiIiIiOiVgiBA +GIzCTsdxYNs2er0eOp0OYrEYxsbGGHx/Qnzfh+u6sCxLBN+SJEHTNGiaxg4ARERERERERHQuGHwT +EREREdFLhWEI27ZhWRaq1Sp2dnawt7eHg4MDHB0dYWJiAr/73e/wq1/96rwXlT6QIAhgWRa63S5s +20YQBKLi2zAMaJrGam8iIiIiIiIi+uAYfBMRERER0UtFwXen08HW1hZ++uknPHjwABsbG9jZ2cG1 +a9cwPj7O4PsT4vv+C8G3oiii1bmu66z4JiIiIiIiIqIPjsE3EREREREBeBZy27aNdruNZrOJZrOJ +Wq2GRqOBra0trK6uYmtrC0dHR6jVaiiXyxgOh+e96PQBOY6DTqeDo6Mj9Ho9+L4PTdMQi8WQyWRg +miZUlbeaRERERERERPRh8WkEEREREREBGAXfvV4P7VYb6xvrePLkCVZXV3F0dIRqtYpms4lut4te +r4fhcCiqfcMwPO9Fpw/IcRw0m03s7e2h0+nA933IsoxEIoFCoYBUKsXgm4iIiIiIiIg+OD6NICIi +IiL6RPmeDz/w4TgOXNfFYDDE0dEhDg8Psby8jLt372J5eRn1eh2NRgPD4RBBECAIAvi+jzAMxRd9 +OmzbRq1Ww9bWFlqtFjzPg6qqSCaTKJVKSKfT0DTtvBeTiIiIiIiIiD4xDL6JiIiIiD5BQRCg2+ui +0+ng8OAQW9tb2NnZQa1WQ7VaxcHBAfb391GtVjEYDGDbNjzPQxiGosqbgfenqd/vY29vD48ePUKt +VoPjODAMA9lsFlNTU8jn8wy+iYiIiIiIiOiDY/BNRERERPQJCoMQ3W4XhweHeLj8ED/88AMePnyI +er2OZrOJwWAAx3FENXgUegNg4P0JC8MQg8EAu7u7ePr0Ker1OlzXRSKRQDabxfT0NPL5PAzDOO9F +JSIiIiIiIqJPDINvIiIiIqKPXBAE6Pf7GAwGaLVaaDQaaDQaODw8RLVaxcbGBlZWVrC1tYV+v49+ +vw/LshAEATzPA8Cw+1MXhiGajSaazSZWVlawvb2NWq0G27YS2RNUAACAAElEQVQhSRISiQQqlQoW +FxdRKpUYfBMRERERERHRB8fgm4iIiIjoIxaGITzPQ7PZxNHREZ4+fYpHjx5hdXUVnU4HnU4HrVYL +7XYbnU4HnufBsiz4vo8gCMTPoE9bGIY4PDrE48ePce/ePWxvb6Pb7UKSJGiahnQ6jfHxcVxduopU +OsXgm4iIiIiIiIg+OAbfREREREQfiWjebc/z4HkeHMeBbdsYDAbY3NzE1tYWHjx4gB9++AErKyuw +LAu2bYt25q7rMuQmAM+OJdd14TgOLMvC2toa7t69i+XlZRwcHKDVaqFQKCCVSqFcLmNychLTM9NQ +ZAWyIp/3KhARERERERHRJ4bBNxERERHRR+Bk4H10dIRarYa9vT3s7u5if39ftDc/ODjA/v4+er2e +CMc9z0MQBAy9SYgGTTQaDWxvb2N7ext3797FgwcPsLm5iXa7DUVRUCqVsLS0hC+//BKTk5NQZAWS +LJ334hMRERERERHRJ4jBNxERERHRRyDwA/i+D8uycHBwgJWVFTx8+BD379/H06dP4TiOCDOjKvAw +DEXgzdCbImEYwnEcDAYD7O3t4aeffsLdu3fx5MkTrK2tod1uw3VdyLKMcrmMr7/+Gr/61a8wPT0N +WZEhSQy+iYiIiIiIiOjDY/BNRERERHQJhGGAXr+HarWKYa8JAPB9H7Zlw7It9Pt99Pt9dLtd7O7u +Ym9vD5ubm9jY2MDR0RF834fneXBdV/zz+xAF7cvLy+e9iegthGEo5nEPgtGgiaitueM4aLfbaLfb +2N3dxdOnT7G+vo7Dw0P0ej3IsoxSqYRcLocbN27g+vXrmJ2dRS6XY+hNREREREREROeGwTcRERER +0SUQ+D467Q52dnqQAlsE3c1mU3w1Gg20220MBgMMBgP0ej10u10Mh0MRckaB5/sSzR/+3Xffnfcm +orcQBd2e54l53ofDIbrdLrrdLprNJlqtFtrtNjqdDrrdLmzbhu/7yGQyWFxcxI0bN/Dll19iaWkJ +45VxxM34ea8WEREREREREX3CGHwTEREREV0CQRCg0+7B7/Uw7LfQaDRQr9dRq9VQr9dRr9dFWAlA +tC+PAu9fymAwwMbGBuJxhp6Xie/7orrbsiw4joNeryeC7k6ng16vh+FwCEVRoCgKDMNAMpnE+Pg4 +bt68iT/7sz/DlStXMDc3h1wud96rRERERERERESfOAbfRERERESXQOAHaHWbaAz20agd4ODgALVa +TVR3D4dDWJZ1qpo7qvD+JfX7fWxtbWEwGJz3JqK3ELU3j9reR0F4NAe8bdvwPA+maSKVSiGVSqFc +LqNSqWBhYQHXrl3D9evXUSwWEYvFznt1iIiIiIiIiIgYfBMRERERXQZ+EKDdbqJ9sIP93Q3s7e2h +VquJ8DKq7D4ZfP/SoTcAdDod2LaN/f39895E9I7CMBSDJE4eQ5IkIZlMIp/PY3p6Gp999hnu3LmD +hYUFlMtllMtlqKoKVeVtJRERERERERGdPz6hICIiIiK6BGRJgqEbSCQTyGQy6Pf7cBwHnufBdV24 +rgvHcV5oa/5Lh99R+2vTNM97E9FbkmUZsixDkiTIsgxFUUSQbRgGDMNALpdDpVLBxMQErly5MprP +e3yc+5yIiIiIiIiILhwG30REREREl4AkS0il00gb40jEVMTjcWQyGfR6PXS7XfT7ffR6PXieJ74n +muP7lwy/TdPE+Pg4xsfHz3sT0VtQFAWapok/NU1DLBZDIpFAIjEaXJHL5ZDJZJDJZJBOp5HL5ZDN +ZpFMJlnlTUREREREREQXDp9WEBERERFdArIsI5PJIq0bsPNp5PN5tFotNBoNNBoNNJtNtFotdLtd +MWez53li3uaT832/zyDcNE1MT0/j9u3b572J6C0oigJd16HruqjuTiQSSKfTyGazKJVKKJVKSKVS +0FQNiqqc9yITEREREREREb0Sg28iIiIioktAkVVkM3FMj5WhyT4sy4Jt2+j3+xgMBuh2u+j1euh0 +OiIMPzo6QrVaRbPZFPOAR6H48y3R35VpmlhcXMSf/umfnvcmoregKMqp1uYn25ubcRPJVBLxeByK +okCSpfNeXCIiIiIiIiKi12LwTURERER0CciKjFw+j7m5EvIZE6qqQlEUEWK7rgfHsdHr9bCysoLV +1VUsLy/jwYMHGA6H8DwPnudBkkYh5vsKvpPJJJaWlvCXf/mX572J6C1Fx4IkSWKe7+jPaM7v6O+I +iIiIiIiIiC46Bt9ERERERJeBJEFVVTEPs2EY0HVd/HUQBPA8D5ZlQVEUJBIJJJNJZLNZTE1NiXnA +2+02Wq0Wer0eXNeF53kIgkB8va1omTKZzHlvISIiIiIiIiIi+oQx+CYiIiIi+ghIJ4LxSrmCRCKB +SqWCGzduoF6v4/DwEAcHB1hbW8PKygp2dnbgOA5c14XjOPA8D2EYvtf5v4mIiIiIiIiIiD4UBt9E +RERERB+BqCW1ruvIF/LIF/KYnZ1FGIZwHAebm5vY2tpCPp+HoigIwxD9fh/D4RDD4RC2bcOyLDEH +eBSAMwgnIiIiIiIiIqLLgME3EREREdFHTlVU5HI5AICmacjn87h+/TqazSYajQYODg6wt7eHarUK +y7IwHA7hOA7CMITv+wAYgBMRERERERER0cXG4JuIiIiI6CMmSRJkRUY2m0UqlcL4+Dhu37qNwXCA +/b197O3vYXl5GT/88AOCIEC73RYV39H83wy9iYiIiIiIiIjoomPwTURERET0kZMkCZqmQdM0xGIx +AEA2yELXdSRTSWiaBl3XUSwWUavVUK/X0Ww20W630e124TgOhsMhXNcFALZBJyIiIiIiIiKiC4fB +NxERERHRJ0iSJKRSKaiKing8jkqlgtu3b+Pw8BDVahUbGxt4+vQptra20G63RcgdVYCzEpyIiIiI +iIiIiC4SBt9ERERERJ8gSZIQj8cRj8eRL+QxNzcH3/NRrVVxdHSER48eIZFIQNM0HB4eQtd1DAYD +uK4Lz/PgOA4cxxE/i4iIiIiIiIiI6Dwx+CYiIiIiIgCAJEswTROlUgm+7yMej2N2dhbVahW1Wg0H +Bwc4PDzE0dER2u022u02ZFlm8E1EREREREREROeOwTcREREREQEAZFlGMplEPB5HNpvFwsICLMtC +s9lEs9nEysoK7t69i+XlZWxvb8OyLIbeRERERERERER0ITD4JiIiIiIiQZZlyLIMTdMQj8eRTqdh +miay2SxURYWiKMhkMpiamsLu7i4WFxcxMTFx3otNRERERERERESfOAbfRERERET0SoZhQFEUzMzO +IJVOYWFhAY1GA/V6HYVCAYsLi+e9iERERERERERE9Ilj8E1ERERERC8lSRJ0XYeu6zBNE+VyGWEY +YjgcYjAYQFEUxOPx815MIiIiIiIiIiL6xDH4JiIiIiKit6aqKkzThCRJUBTlvBeHiIiIiIiIiIg+ +cQy+iYiIiIjorUiSBE3ToKqq+HciIiIiIiIiIqLzxOCbiIiIiOgtMegdbQNuByIiIiIiIiIiuijk +814AIiIiIqLLSJIkMPYlIiIiIiIiIiK6GBh8ExERERG9K1Y9ExERERERERERXQgMvomIiIiIfqZf +svZbOvFFREREREREREREZ2PwTURERET0DkSr8w+RSEsAC8uJiIiIiIiIiIhejsE3EREREdFbkCQJ +sixDURTIqgJFViArCiRZBqRf4PJakkYV5ZI8CtuZgBMREREREREREb2AwTcRERER0RuSZflU8K2q +KlRVHYXgsvyLFX9LsgRZliDJEsNvIiIiIiIiIiKiMzD4JiIiIiJ6A/JxxbUijwJvTdOgafrxlwZF +kX+RQFoE7bIMRVYgSbII4ImIiIiIiIiIiGhEPe8FICIiIiK6LCRJgqzI0DQN8XgcCTOOhDlEwvQx +DDVYv0DNtyxL0DQZcUODrqlQVYbeREREREREREREz2PFNxERERHRG4jajCuKAsMwEI/HR+F3woRp +xqDrGmT5/QfSiixD11QYhgZNU0dzissy5F9iPnEiIiIiIiIiIqJLihXfRERERERvKJrb2zAMmKaJ +fNZFeejD9WU0Bhq0gQTHDxCEQBiG4vuCIITv+wiCAEHgIfD90VzdsjIKshXluFW6DEiABAmKIkOR +JRQyGioFA+PFODKpGHRdPf5eVn0TERERERERERFFGHwTEREREb0BSZIgS8+C70QigUI+hB8Auqoi +2Q4Q6/gYWAE8P4TvP/tez3PhOA4cx4Hr23AcB7IsQ9d1aIoBXdOh6wZkJYQsyZBlQNcl6LqCct7A +bDmJmfEkChkThj6q+iYiIiIiIiIiIqJnGHwTEREREb0hSZYghzIMwwAAhAAUWULcUBGLW9B1C33L +h+MFcL1nFd+O5WIw8DCQhhj6Q4RBD4qkwlQSiBuAaaowEzJUVR1VlcujOb3jMQ1j+TimK0lMFFOI +x3VoqiLarhMREREREREREdEIg28iIiIiojcQBc2KoiAIAhiGgSAEZEmBphsw4i4yGRuOG8D1Q/h+ +IL633++j0Wig1XTQUvpo+TY0LUAum0Q2F0Mum0Yun4eu65AkGbIiw9AU6LqKdMJAPm0iHjegH8/x +zdCbiIiIiIiIiIjoNAbfRERERERvKAqcNU2DLMuQZQWGriOZ9JHL+vA8H34QIgjCU3N8N5tN7JsO +DtQetCBAOLQQiwFjGWC8HEOlkkFlvAwzHock47jd+aituqaOAnBVUaDIo1CcwTcREREREREREdFp +DL6JiIiIiN7Cycrv0Zzf8qgCPAwRBKMq7yAITgXfoW+h1zHQjWuI6wp0VYKhyTBjKpKmjnQyhlw6 +DtM0AeA4VB8F3JIkjaq8j9ubM/QmIiIiIiIiIiJ6EYNvIiIiIqK3FIXPkiJBCiXIkgwACMLgzNcb +hgFd06GqKjRNg6qqUBQFqqpCVVXoug7DMMTc4QDEzzw5nzdDbyIiIiIiIiIiorMx+CYiIiIiegen +wuhRRg0FypmvVVUVqjYKuaNqbkVRxFcUiGuadt6rRUREREREREREdCnJ570ARERERERERERERERE +REREPweDbyIiIiIiIiIiIiIiIiIiutQYfBMRERERERERERERERER0aXG4JuIiIiIiIiIiIiIiIiI +iC41Bt9ERERERERERERERERERHSpMfgmIiIiIiIiIiIiIiIiIqJLjcE3ERERERERERERERERERFd +agy+iYiIiIiIiIiIiIiIiIjoUmPwTURERERERERERERERERElxqDbyIiIiIiIiIiIiIiIiIiutQY +fBMRERERERERERERERER0aXG4JuIiIiIiIiIiIiIiIiIiC41Bt9ERERERERERERERERERHSpMfgm +IiIiIiIiIiIiIiIiIqJLjcE3ERERERERERERERERERFdagy+iYiIiIiIiIiIiIiIiIjoUmPwTURE +RERERERERERERERElxqDbyIiIiIiIiIiIiIiIiIiutQYfBMRERERERERERERERER0aXG4JuIiIiI +iIiIiIiIiIiIiC41Bt9ERERERERERERERERERHSpMfgmIiIiIiIiIiIiIiIiIqJLjcE3ERERERER +ERERERERERFdagy+iYiIiIiIiIiIiIiIiIjoUlPPewGIiIiIPoQwDH/W3xP9HEEQiK8wDF/4Ovn3 +RBeFJEnv5TVERETvitfwRHQRRfd1J+/vAIh/9n1f/D0R0UX1uvv5y3q/z+CbiIiIPjonH4CJG9Ag +RBAGp//bczeoZ33/SbxppXdlWRZs24brunAcB57nwXVd8e+2bWM4HJ73YtInRJbPbv518sY2+mdJ +kk79MwDIkgxJlhCG4Qs3w5f15piIiM4Xr+GJ6LKwLAu2ZcNxHLiuC8/zAACu68K2R/99OBxCVRm/ +ENH5eN09//P3/i+75z/rtRcdP3mJiIjoo/L8Q7DoYdnJytrnR2a/MEI7OP3QLHrYRvSuLMsSAbfn +eeIrCr4ty4JlWZfiBoI+LrJ0+mZYkqUXAu/oS5Zl8c+BFEAO5dH3H/+I6PvOCsOJiIhe5XXX8Cc7 +5PAanojOm2VZsJ1RwB0NbAYggvDo/k7TtPNeVCL6xL3tPX903x9KIaRQeuGe/zLc7zP4pktPkiWo +qgrTNJFKpZDNZhEEAbLZLJLJJGKxGC8yiIg+Ac8/9IoelAXB6CGZ43pwXB+268N1PTheAN8PEQQ+ +giA8o/00ABw/PAujf2IrRXo33U4P1aaD9lDGwNPhyQm4MNCzVTS6AeSYDU/pwNDt815U+oRIkHD8 +P0CSoMiAqkhQFQm6KkNVZSiyAlUd3fwqsgJZkcXN8MkvSZLEiPDLcjNMREQXg7j+Pg6ufXF9PrqG +t50AtuvBdX04ns9reCI6d4N+H9WGjWYfGHg6XMmEBAkDV0ezF0Jp2vCVDsyOf96LSkSfLOn4Xv/Z +Pb+mAIoiQVXk43t+CaqiiPv65+/5leO/kwIJiqwgRHhmFfhFw+CbPgqGYSCRSCCXyyEIAsRiMeTz +eWSzWSQSCei6ft6LSEREv6CzQu+oIsTzfPi+j/7QRWfgoNd30Lc89IcuHHf0d88emgUIQyAIQ+D4 +nwEgfPbUjOidWLaFXs9Fvy9j6CcQqCFcWUPXNRD0gEFgodFvsRUefVjS6FZYkiXIEqCpEmKahLgu +w4ypMA0Zhq5CURRo2uhPVT3+U1GhqKObYFVVIcsyQimEAgWQL89IcCIiOj+vu4YPghPX8AMXvaGL +wdCF4wTwAh+B/1xFOK/hiegDcRwHvZ6Dfl9C34vDV3IAgL4XR9iXYMFB0+pA16zzXlQi+pSduOdX +ZEBXJcQ0GXFDRiKmwIyp0FRZ3N+P7v01KIoCRVbEf1MUBWEYjgLx4FkFOHAxA3A+WaNLT5Ik6LqO +ZDKJMAyhqRpSqZT4SiaTfIhMRPQJOPnAzPcD+L4P1/MwtFxYtotWz0GzbaPVs9Dpu+gOPDiud/za +4Dj0Hj0wO2vOQIR8akbvzvN9OI4P11XheSZcWYUHGZ6jY9iR0B460Lo9KC+Zg4noFyHm7gJkWYKu +SjANGaYhI20qSMYVJGIqDEOFoSnQdQ2apkPT1NHNsD+6KQ7DUITfkiSJG+GLeANMREQXz4vX8D4s +28XQctDpOWh0bDS7DroDB92BC9vhNTwRnS/P9+E6LlxXHt3fSaNnz4GrY9hV0LU9aL0BFJnBNxGd +o+N7cuX4/jyuyzAN6fieX0XKVBEzFMR0FbqmQtdVcc+vqirUQD11zw8AkHHh7/mZBtJHwTAMAICm +aUgkEnAdF7qhwzAMxGIxVnwTEX0ioioRx3Vh2z66Axv7tQH26xaaXRudvovewIPt+rBcH74fnGqR +CEC0WAzPeEgW8sEZvaMwlOD7Knwf8H0ZQaCNAkJPPh49q0JRJLAsiT6U0Q1qePzPgCSFUGQJuhZC +UwKYhoeEISNtysinZOSTMgxDQ9zQYRxfZ2uahiAIoGuja21VVcVcYFHl97P3IiIietEotB5dw7uu +B8vx0B842KsNsF8fotmx0RmMBq3ajg/bDUTHpoDX8ER0Tk7e3wWBAt8fxSyKP6qMlG0FijK6xiYi +Og9n3fNraghNkWFoAUwjQMJwkEmM7vmzCQUxQ0fM0GAYBnRdf3bPr+svhN8KFAC4kJ3eGHzTpSdJ +0ujBm6rBNE0RXkjSaH5BRVEu3IlHRETvz8mHXaM5vQO4no+B5aDRsbC218OjzS6aXQd9K8DQ8RGE +AMQcgKd/DtEvIQwlAArCUD7+9/DZ/MpSND/SeS8lfVpe/MyLHszJEhDTAhgqkE8EmCrICF0gYRrw +4wZ834PneYjFYi98dkqSBFmWEYSBuBEmIiJ63suu4YeWg1bXwvp+D483O6h1XPSHPq/hieiCkRCG +ChAqx9MqhAAkwMML93lEROfj7Ht+SQogS0BcC2AoIcaywHQe8LMKkqYOzzTged5ooOHxtDLPnltJ +4p9l6eJWfTP4po+CJEmj+Qb4cI2I6JN0slLE8zy0u6NK752jAbaPhjhq2ceV3iEcjw/H6LxIx18n +8HCkC8r3AVsZzY2qyi6CQEI+GcJLefB9F7FYTMzDejJ0kCTpWeV3EF7YG2EiIjp/L1zD9xzs1/rY +rQ6wfTjAYctBp+/BdgJewxPRBSQ99yd4f0dEl4IkAZ4PWHIIRQ6hSCFcz0c+FcD3XPimB9/z4cd9 +BGEA4PRgQ1VV4Us+FCiQlIt3v8/gm4iIiC69qFIkGpHY6NhY2+tjdbeHWts+ns87hBfwLpSI6E14 +gYQwlNCzfYRtYGh5sB0Xkh8i9DV4nidC72jUtyRJkJVRVwPRfQnKhWx9RkRE5+/ZNbwP3/ePK737 +WN3podp20en7sJ0APq/hiYiIiN6fEPADCWEoozMcDUIcWj5c14USAoHvjiq+w9E9vyydvs8/2elN +Cp9Vg18UDL6JiIjoUjtdKRLAsl00Ow52jgbYOhxgYPsYWAH4vIyI6M0FoYQgBAJHhusFGAwBRXIR +lx0oUBCc+FCVZfnUza8kSQiCALIsIwxCSIrE8JuIiE45dQ3v+7BtD82Ojd2jITYORtfwQzuAH5z3 +khIRERF9XEKMgm8fEvpBCNsFLDuAJjkwFRcIPFHhLUvyaDphWYIsK5BleRR6B4Ho9HbR7vkZfBMR +EdGlF1WKDCwX7a6NVs9Gd+hhYPtwvZDdxoiI3lEICX6owAEwtC20ujYUyPBDCdLx3IWqokJRFCiK +Ak3T4Hu+uBkGAEmWLswNMBERXRxhGMIPQgwtF+2ejVbPQXfoPruG50U8ERER0S8qiO75/QA9K0Cj +bSMMAwTAqc5u0T2+qirwff/UAHgpvFj3/Ay+iYiI6NIbPTQLMLBctLoWWl0H3YGPgR0gDMGHZkRE +7ygIR+3PAklC3w7RCl3IgQ8JgKYAiqJAVVUo6uhP13UhyzKUQDk1AhyjDPxC3QwTEdH58z0fA8tD +qzMaXNUdjCq9A17DExEREf3iglBGGEqwEaA/9NEKLEihCxmAKuM47FbFvb+qqqN7fmU0rVkYhqLq ++6Jg8E1ERESXWnjcbtf3QwxsD+2+i77lwfF8BGyNSET0s4WQgBBw3BD9wIMmudBVCTEthKqq0DRN +fOm6DlVVEQSBaHdORET0vGh+bz8IMLRddPoO+pYH2/XZ3pyIiIjoAwohIQgl2G6IXuBCU3wYmgRD +g7jn1zUdru5C13Vxvx8F3wDY6pyIiIjofQrDEJ4fYGj5aPcdDCwPvscnZkRE70sIwPUCDL0AOjzE +NaCv+SLsNgwDruvCdV2oqgrf98UI8CAMoEA571UgIqILRrQ6t/3R4NWhB89nmTcRERHRhxaGIRw3 +QN8NoCs+YlqIuA5omoZYLAbbsaG7OjzXg6f60NRAtDy/aBh8ExER0aUWhIGoFrFdD72BC8vx4Ad8 +aEZE9D55fgjL96GFHuK6j4TuwbKs0U2wbcMwDHieB9/3Xxz9fcFanxER0fkLwxC+78NxffSHx9fw +HLxKRERE9MGFIeB6IeD7iCk+hpqPgRbAMAxYlgVd1xGLxeB6LjRfE89jTw52vyhV3wy+iYiI6NIb +zScTwPMCuF4A3x/NC0hERO9PEAKeF8CRfDhOCMtyYVkWHMeB67rwPG8UfHvPgu/oT5z/vS8REV0g +Jx+Wuv7oGt7zA/ASnoiIiOjDCwEEQTi653c92A5g2wFs2xb3/K7rioHuJwe7X7R5vhl8ExER0aUX +BAH8YFQx4nne6KFZyMdmRETvUxCECP0QLjzYTgDbDuA4zqmv0WewNxqA5AcIlWcjwKVwdBN8EUaA +ExHR+Rs9NA3hewFcz4fvhwh4DU9ERET0wYUh4AchAi+A43ij0Fv3RfAt7vdPDHaPvk7/nPOv+mbw +TURERJfayZGFQRiOHpgFIYNvIqL3LOqu4YceXNeD4wSnqr1PjvwOAl9U8xEREb1MNDgqCsH5a4OI +iIjoPIQIw2DUUdP3j+/5/VOV3tGffuDD9wOEx+02Tw52P+/QG2DwTURERB+JECfa64Rgm0Qiovcs +al3uwYfveXBdX4z4jm6APc87NfL75DzfREREETEnZBB1Bol+X/B3BhEREdEHFz675/c9H57nwnXl +U9Oanb7XDy7sYHcG30RERHTpjR6cBc8emoHJNxHR+xZiNIrbD4PjsPt06B1VfPu+/0LgfRFvhomI +6Pyd7N7Ea3giIiKi8xEVFAVhcDyo3T812D265z9rju+LhsE3ERERfRSeVXuHYI9EIqL37Hj0N45H +gAdB+MKNb/TnyZvfi3gTTEREF8Pod0Rw6jqevzWIiIiIzkc0vVmAAIHvIwiklwbe0etP/nlRMPgm +IiKiS+1Z2A0Ex6HM6HIrOO9FIyL6uIQY3eQeh9zP3/yeDL1Z7U1ERG8iGlT17HcFf2cQERERnQdR +9X18f+/7ODXIPbrnPyv8vkgYfBMREdFHIYq7QwAIAz4yIyJ6z8Tn7MnKvBM3xc+H3hfxBpiIiC4O +8ftC/DsHrhIRERGdj1B0egsR3efjVNh9kdubn8Tgm4iIiD4uF/zii4joMntd4H1qrtZLcENMREQX +QMjBUkRERETnLar4Fn+GOPt+P7zY9/sMvomIiOijcdzxnNXeRES/hBDHg4tef+MbsGqPiIjewAV9 +XkpERET0yXo22D08c7B79Jrnv+eikM97AYiIiIjeiyiQYfJNRPSLu8iju4mI6OI7+Tvk2VQa4HU8 +ERER0YVw4lrtJff+F7XymxXfREREdOlFbXiO/+1ZAE5ERO/dyY/Xs250T/0zW54TEdErPWujiTDk +JTwRERHRBfCqW/iLfn/Pim8iIiIiIiJ6A+GpP89qbRYEz1qcB8Fzfx9c7JtjIiIiIiIiok/Wcfed +8MQ9/2WZ1/skBt9ERERERET0RsLj/zs5r9dluPElIiIiIiIiolcTofdr+vBc5OcADL6JiIiIiIjo +tcLn/jz1d2dUfxMRERERERHRJXBq1sjw1ID3y4bBNxERERERERERERERERERXWoMvomIiIiIiIiI +iIiIiIiI6FJj8E1ERERERERv7bK2PSMiIiIiIiKi1wvDUHxdFgy+iYiIiIiI6J2cvAF+/k8iIiIi +IiIiumRC8X+XEoNvIiIiIiIieivPZ9sMvYmIiIiIiIg+DiFevL+/LNXfDL6JiIiIiIiIiIiIiIiI +iOhSY/BNRERERERERERERERERESXGoNvIiIiIiIiIiIiIiIiIiK61Bh8ExERERERERERERERERHR +paae9wJ8KlzXheu6cBwHYRjC9/0zXydLo7EIkixBlmVIkgRVVaEoClRFhSRLkKTRF70Z13Vh2zYc +2wEABGEAVVWh6zo0TYOiKNymRET0XiiyBFWRIMsSIAGSJCMIAoRBCD8IEYRAEIRnfq8sS1BkCbIM +SJIEWZKA499NYRjA90P4fogw+nYJp14vjd4SIUb/F4RAGIYIXvO+F510/H+ydLxtFUlcL4XhaLuG +QQj/eF3DN1xN6Xj7KYo82oZ48VpA/KgwRIjRzw7DZ+/7svcb7ToJcrSPTizzWcLj9wBC+AHg+aP9 +/UtuS0WWjq8rAVkSf4sQIY7/hzAcrV+0nkH45tuXiIiI6DJ7m2vQMBhdJ/1Snt0njK4vJUk6cZ0f +IgiOr9fOe6OdoMgSVFWGqjy7Bg6CQFxDByEQXrBlfvl6SFAV5dR6BCfu7y7Depyn6BYrOoaje1dZ +3HpJ4l4LGG3P6P71Y77/iO4ZR1vg2T3h6D7svJfu3cnS6JxRFAnS6KHIqecZQTi62ZTe8HWXeFO8 +leh3jnR8r67IAMRznrPu1U/fr3/M58rbiI4/VZHF+RWEx5/Zfiiek3FbffwYfH8grVYLW1tb2N3d +heM4sCwLQRCIv4+CV1mWoaoqVFVFLBZDLBZDJpNBNpNFJptBLBZDPBaHoio/Y2k+La1WC2tra9ja +2oLv+3BdF5lMBjMzM5iamuI2JSKi98aMa8inDaQTGjRFhqrKGFguOgMP/YGLoe3BcvwzQ+iYriCV +0JCMa4gbCuKGClmSEAJwvQD1jo1m24LrBwjD0QOwlKkhaWqIaTI0TYEiS8cPYQDb8TCwPVi2j6Ht +Y2h7lzL8lo8HE+iaglwmhmLagKaOHqD5QYjuwEV34GBgjdbRdvw3+rmKLCGXNpBLG4jr6ujBnPzi +ILgQo4eJfhDCcX1Yrg/L9tEfehgMXbh+8MINuSJL0DQZMV1FNmUgl9TFMr9MFNx3+g5qLRutrv3+ +t6UyuomOGSqScRWJuAZDk6GrCiRZEjeAnh/C8304bgDL8WE7PoaOD9v24P0CgTwRERHRRSPLo0DG +OL4GLaQN6O/hGvRdxHQFmaSOVEKDoamI6TJsN0B34KI/PF4Gy4N/ga71E3ENY/k4CmlD/LeB7aPd +s9Hpu7BdD44TXKhlPosZ1zCWi6GYiYn/NrR9tHoOun0H1iVZj/MiBhvLEhJxDWZcRdxQR/cgmiLC +vuB48K/rBeLe1bJ92K4H2wl+7mJcOLIsQVOOQ19JhiSNBq4Hvg/XD0+FmpeNrssoZmMoZeNisIPt ++Gh0bTQ7NlzPh+eF0LTR64rZONRXvO5TObfk422gawpSpoqEGd2rq1CV43t1RPfqo3Mlule3bB+W +48FxP75z5W1pmoxSLoZy3hTPd1wvQKvvoNNzMLR9OK4P1+O2+tgx+P5AWq0WlpeXcf/+ffT7ffR6 +PXieB+DF0FvTNBiGgXQ6jXQ6jXK5jJmZGQBAmAmh6zoUMKR9U61WCw8fPsR3330nqu7Hx8dh2zZS +yRS3KRERvTfJuIKJUhyTpQRiuoq4oaDesbFXHeCwMQQAOK6Psy6xY4aCQiaGcj6OXMpALq1DkWWE +YYi+5WFttwPb8TC0fYQhoCkS8hkDlYKJdFxDPK5CV2V4fgjX89Huu2h1bDS7DgAbtuPhMl7ay7IE +VVVgxlRMluJYnMwgERtdwjp+gP3qAPv1IertITw/fOOHjrIsIZcyMD+eQi5lwIgpiD03CG40kjqA +64dw3QB9y0O776DddVFtDWG7HvxAQvBc8q0oEmK6ilRCx9RYArOVpFjms4QAXD+A5wXYrfZhu8F7 +D76jh06qqiAZVzGWj6OUjSFpakjGNMjKaNBEEISwXR+2E6A3dNHpu2j3HUg9G57rj6qbPo1nD0RE +RPQJk2UJ2hteg/pvcQ36LmKGglIujvFiHOmEjpSpoTdwcdAY4qghQYID5/g67aJImipmKkksTaXF +f6t3LGwf9BFiAGkAeJ57oZb5LImYgulKEtemM+K/NToWto/62EN4adbjvMjSKNzVVAXppI5SNoZs +anQMJ2LaqJpVHoV5luNjaPloHgef7Z6DIAw/zuBbAlRVQUyXRRX86D4MCMLRQPnLOt44pquoFExc +nclA12RoioRO38P6Xhe242NgA0Hgj16XN3F19tWv+1TOLfm4u4QZU5DPxFAuxJGOa0jENei6jOC4 +WtlxfdhugL7lotNz0Rk4aPcceL4P17vc3QLeB11TMF4wcXMxJ57v9C0PO0d97Eg9SJIDPwjgeue9 +pPRLY/D9gQwGAxwcHGB1dRWdTge9Xg+u64q/j4JvRVGgKAo0TYNpmjBNE7lcDk+ePEGpVEKlUkGl +UkGpVEKxWEShUBDfT2cbDAbY39/HysrKqOW542AwGGBiYgK9fg9GzDhVfU9ERPSudE1FLhXDeMFE +Mq7CjGtQ5RDdbg9NyYYU2PA9B74fQpJkyBKgaxI0VUIxrWC2bGJmPI10QkPa1DGwbLQ6fTjWAPAs +qHCQSyrIJOPIpOIo5UYjpJNxBTFNhabJ8PwAnheiN/TQ7TtodR0cNXqoNoFWZ4BOz0J/6EKWFUiK +CklSjr+etVa/OEKoCpCIycinNVTyccyUE0gmNACA4/iQ4SH0bXi2h06zA7s/gKxokBUNkqxCkqMW +V8frdty2HGEI05BRyuoo5WJImBrMmHbyrcUoez8Yjaoe2r7YrqWWjnpLR709QKvdR7tnjSrDQwmB +rEEOQxiyjGxCwVTJRDqlv3wtgxCuF8B2PFiWjU3Zg+cMIMkKZFkFjqsAxDq80aYbraciA6oC6JqM +fMZAIRNHIRtHKRNDPmPANFTEYgpUSUaA0Yh61w3gej4G1mh9e0MHR/U+aq0Bmh0L3b6N3sD9ectH +REREdCGNrv9UJUQiJiGfUlHJxTBdNpFKjK7nRtegPsLAhmf76LV7cIZDyIoKWdGPKzjln31tHYYB +EIZQpQAJAyimNXE912gPMRwO0JYdyKGNwLMQ+IAkKed6bRYts6ZKKGR0zE4kxd+pSohWqws1GEDy +HPiuB9/H6Jpdiqp/L8b15LP1AArp0+uhqyFanS7UcAB4LgLXO76/UyHJF2s9PuAWO565KQAQQlcl +6JqEZFxHNhNDLhVH/rhzQjqhw4ypiMeU4+m9RsG34wawXR+dvoNu30Wr66DVtUZfndGftutf4u0c +Ht9fBtB0FaWshkrBhKap0FQZlu3iqNFHtW7B9kZV8EH4rA34xXXcfjsMEAYeZMhImzKmSnEYugJd +VVDvDFFrdCD5A8DzEXgBFEhIJ5TnXjdAte4idFoI7ACeHcDzpeNnFxqAj2Wq0tH5Ikuj5x2qAuTS ++vG9uoli1kAxG0cipiJmjLZNGAJ+GBzfq4cY2h76lof+cDQov9boo9EeotO30e07CCGPzhNJvgTH +0M/ZlNH0AD7CwIMUhEibMqbHTPF8p9O10e8PUVU8KKELhB7CIDj+XfmRbhdi8P2hDIdD1Ot17O3t +od1uo9PpiIrvSHSinZzbO5qLOhaLwTRNzMzMYH5+HteuXcOdO3eQy+YgHbdt4Il6tuFwiGq1ip2d +Hdi2Ddu2oaoqGo0GhsMhPM9D+KkPhyIiovfC0GRkEhrKuRiSCQ2phAZ7OMCuYkPyugg9F77rjh7w +SDJkVYYhq0jHFUzkFFyZSmBpNgtDVaDrMnb3Lez1uqgeNTDsu1ADF+V0GldmU5idyI+qdU0duqaM +2oMpQOCPAlvHH90U9QYOdg8l7B54WN/pwep20Oz1oOgmFC0uQmIoGqQQF+iGaHQzqKujG+KxrIZy +zsB4KY6U+Sz49pwBfDtEu2kBbgPDTguakYRqJKFoBmTogKxCGq3caP64wANCCTEtRC6pYCyrI5U0 +kDT15xcBwKgNeRgCnhfC9X04ToBWb4h2x8L6Tg3/P3t/9ibHkZz5wj93jz1yz9pRAAFwa6q1zJHO +uZqbmev5o+dIn2ZGy6gldXPFjgJqzX2N3f278MisAgmSYEtnejRKe55iESQywt3DPdLMXrP3/fb5 +nOV0TlUJykpQCQ/j+TgIWqHhsOfTvUOP+IOZGsiygix3uB4KlEkpsyXS8cH1EdIF5C96NFYnT6ME +BA60QsnHJyGff9RjvxfTiDzi0MWVEuVsEgi1RlhVa43riqKwQfXFjcvlDby+KHiZ5UyzJdIJfu/x +7WxnO9vZzna2s53972g2PaRxlaQVSvbbisOex8le+A7wXRUJOoPFNOOinJOtZjhejOPFSMdFSteC +0L//SDDaAkkCl9DTdGLYbzsc9kMUOTeDEsesoEypioyqlEjHQyoPY/4QAJHZgl+u1HQaDid74fb/ +psmawMkQxYwqLymzkkorpOOjnAAjNkq2f3in8u482t+bR56uCVWOKGbooqLMSkqtUI6HEv97zeN/ +0WrV1NwaowvQFZ7n0A4Ux3uKx/dbPD7tE9fdq76nalkwcbc22ervak1WWAaq5TpjPF0znq345kXB +apGyzgu7X/7NrXO9RrpCVwWBI7i/5/Onn/WIQo8wcJjMEr5+nrOe51AJCiMwWkBdqP6/rZlaHqzK +qcocUQliz3DQcQkDD99zkFSETgH5DJNDVQBaEvuGg65PFLh4noOkIJAJVTKkTAxFLii0g+MFqE1B +Ef/yoqI/tNkCCIMQBl8ZQg8e7Pt89rDL6WGLRuzSCD08x8rZSXUnVtdW03tDd15UmqvBnIsbwdvL +ipfnCdPJEiNcpOPbXM8G/P4/0DZNDbrK0WWOKTWRbzjqhTTqvFHoCi6uDQ4pwuSgNUYbhHTgf+ez +tbN/ke2A7/9FVhQF8/mc0WjEbDZjMplQVRWu6+K67hbs3pgx9su+KAqqqkJrjVKKq6srrq+vSdOU +RqPBwcEBURQRR/FOo/pHrCgKZrMZo9GILMtYr9e0Wi0WiwVFUVAUxQ743tnOdraznf2rmKMEcejQ +bnm0Y492w2MYgidzRLW03QB5iTYSpRyUcGiGisOuw+lBwKPjiIdHMUVRUhQVRZ4wHo0YDIYgHGLf +5bjn8dmDJp8/6iIxCAFlWVKWlfUtlK3qbccOruujdUDLL2h6BdlqwvVlhs5nYCqMMSg3BCFQ0sFs +cw8bn+SWwvt935TivX/41wmo7Fez7RZpRw4HXY/9rsde26dZA9R5UZAlLmXmchmUyHJGthrYBKFQ +NiAWAiVVnfyzCRmtS4wWuErTCATthkO36dOIPYqipKxKqrKi0hpjDI6USCFwAgfHsX5bknokaYgn +UmYTxWhQMl9bLewSj9KrKHNBVaZQpTjCwXddHEehlA3adVVRlCVZnlOSQ5VjyoQqTyjzFQ4CWXfl +c7dr/edXjxrBxlGSVuRw1HN5dBzzx4+77PciG0ALQVlVlEWJ3vhCApRXsxBJW4SptabhF4QqJVnN +ubjMKdMlygchFUoo+8F3EhDv0r9/f/982N754eb7fTy2H97r59bxRzb9T+7x/7Xz/f/y7O1sZzvb +2c529u/bLCDhKmhFgoOOx17bY68T3PFBy60Pen2tUXpJvhqD1rYzUQiMUIhfJKn3Pb/bGIwu0FWB +0C6essn8VqTotT3SRBGoAlGtoMyo8pyqqpl4pIuFSfQW7Pg5H0r81H/4QNdoA+xZsL7ClRB6Ysu9 +64ociiVFNqPMKsrcUBkPhEIqjUDai4kf3vTDYpEf84V+/lp3P20bl+08oMKThsgT29yhKwtEtaRM +Z7YjNTdoYzv9fziP99/YvOe+713UD5jPT83ppy/9rxu3CaPBVEhR0ghsjPvwKORXH7X4o096tY6z +zXmXhaaqyu24hRBIT6CUg6MUjqNIs4LJzGU0dRiPJjwno8yzOkZyESgM4k4I8ktj1w9f4w+NQd5/ +D7PdO1oX6CrHEQ57HYfPHzRoNQKaccDVcM5weMMLmZEgoBLoSiEVYOS/6BmJd//x/vl+yPn4wd65 +ZVPTukSXGaaS+I4tfIlDjyDwSJMET+XoYkmVQ1Uo0B6hB92GSxR5BJ5LshK4JqFcDykSSVFISgIr +E6u8mm1MfA/E/QXn+wPjuQ8+T7/vWbIvS5QwxIGi31Z8dBTyRw/bfHSvjSMlSgoqXdVyY7dMtdKV +KCVRysVRDkop2oEmUhk6X3I9KKjyNUb6CCExQtqmyXeG+K/xTvrXidV/eIsPiNW3t7bFVugKXRZU +RUJVlEhTEDiG0LV/a6VKy4xSrNBliSkF2kikkbfv6x+9x0+M94PGbH52jX7Z/Hf2obYDvv+A1mg0 +aLfbdDodGo0GURQhpaQsS/I8Zz6fM5/PWa/XJEnCer1mPp9zdnaGrDU/F4sFn332Gb/61a9ot9v/ +8kHtbGc729nOdrazf6GZd34bbTtCymxNmReUeYVQLq5raIUuD44a/OrRHg/v9Wg3Qoqi5OJqwOXN +iO+eX/Dk+QWD8YKP7p/w6OE+D0/32GvHSAyT6ZzpdMZwNOFmOGaxXBOGAVEYcXTY58G9I/q9Ns1G +xOkRDMdzzs6vGASG3GTkOWBq2nXpYKNqeTt+swnS64DinflRU2bdJnY2FFriF4G0P76OxmhcR9Bp +uhx0A5qRiyNvr6ukohFHHO5XdFrXKArKZIaUDsrxEVIipcIoXY9JboNMo+suGqPtnzGUZcVwNOL6 +Zsh8sWS5SkjTHD/wCXyPTrvN/l6PXsdqFQa+y0G/yacf7WN0xbOzIYvliPVqjTQZwuR89U3OcnLD +6XGX05M9Tg669TMKmS+XXF7dcD0YMZ6lTGZrXl+tGIyW27Fti/PML11SDVQ0Qo+HJy0++6jDR8dt +mrGHrjTJOmOdrBmNpowmM7KswGC1xRpxRCMO6bRbdDst4jhEYJDCIEyJqVLKYo1QLsYNMKaqq+/N +tuL/LqU8d/9c7xsD271zS7N/d5K3n70tUDS/V7GieM+9xDYZ+W7SxJj33Xdzoe9TDW7GzP/y+d6e +sXev969z9na2s53tbGc7+/dqdzqWlUu74XHQC2jGLuodH1TSbIQcVh3azQGOKKjyGaVyUG5g2Yak +AlmnXYX4ifvZ3z/wu42mKjN0mVOVEl2ktpvN0jthTEVZpDbGKAqqIkdrD6FcpCprQFDWILre+iU/ +Zrc+Tv3zjov0Q99EAEbI7fy2QJTRGK1Zr1LOr0d8/dRgqgKtS87OB1xeXTOfTkgyRVkoG3+Yyn5u +4x/d9Yne8Y/M90d8S1Vr7o7/3TV+109733psmDRv4xpLm1vV8xjz1VO5ncfbywEXF9fMZhOS1KEs +FFpKZE3zfTcefK9v+b37vxtT3TIx/dCv+6nr/bS993n9qE/8C83ommmqwnMNgUMd4/b56KTD0V4T +z5EkSUqaZsznS0bTGZPJzIZmgOs6xFFIHEX0ui363Q6Oq4hDHxAEvrIxSJ4g5Oac2bXanKF316Z+ +Flt3+3a/IATC/Fjcevcc/kRc8P09drd8/Ac+P9vzbSoLDlelQtf7ycpvGnRVUeYJWTKjyBRl7lBp +F6M10vmXNWz9UNrs+wXLd87Y3f15hxFuI9/w7rvizhrV3exV5djCEXP7LjNGo8ucMltSFQpduOiq +RFfVO5rURlcUeUper0FRuGhp7PPWGoSGTUFRTXH9g3fE9jyYd+b6/vH/1Br83Jq+e70Pi8M2V9Vg +CnzX4d5BzK8etnl40qXbDlBCkCQJWZYxGs8YjacsV8n2ClEU0ohDWs0G3fqsUMfqkgpT5VTFCpTB +OB4YF4OxEekHxOrb03J3H8P31uxODLx9N9er/gvj9bvvJvHO2fnhPt2cybvflcZUtoO7ytFlRpaW +XA+GfPPUJ/QdMJrZMuXtxYDZbEaylhSlC9q1+2l7fH/i+/hHvnvePec/VkzxI2fr7vW+915iW9Cz +i+v/JbYDvv+AFgQBh4eHPHz4kMPDQw4PD1FKkSQJq9WKs7Mzzs/PGQwGVFXFYrFgMpmwWCxI05T1 +es14PEZrzenp/R3wvbOd7WxnO9vZ/zZ268waXVIVKUW+pMwryrzC8zx8Ce0o4sFRk//rixP2ujGB +75HnGZdXQ7789hlPXlzx7NUN66zis8cn/NHHB9w/7tNqRQhhmE5nvHz9lmcvXvPs5RmDwZh2u02r +3eaPPntM4Pv0+23azZBWM+RmOGa/7XERaWbrnCQv0Aak42GMd8fRvhNU6OoOAPtuwmkTEFlnX9YU +WvJOfPz7OuqbAEHjOdBtehz1QlqRi7yTdJRS0GxYTbBu08clo0ynKMeldCNL9ej4SFN3XohNUF4D +36ZCm8rqO2GoqoqbwYinz15weX3DcDRjvkxoNGIacZMHp8dorQkDnyDwCQKPo70mmJLQF8wXC16+ +WpGure54kaesZgNev6p4fH+f//Drx4SuLV70PY/5Ysmr12/59ukLroYrrkYr5mtYlz7G+LBN2Og6 +sPrQvWfAVAhT0YoUj++1+YsvjmhGHo3YJ01Tlsslw9GE56/e8vLVOcv12q6pUuz1O+z3e9w7PkA5 +iigKAY1EI0yBLjPKPLGUlLrEaBcjdP3s671SJzg2a/suELzZIHf3jroT4LLVVTSmem9A/YNZm/et +wyYBJbbd/1JYrTMjZL3f391zm323ve/d/V6P016x/uw2abApVNC31/kF86U+Y7fJoZ+Z7zug93uu +twuSd7azne1sZzv75WaFQjG6wnOg3XA56IU0I+8d4FtKQTMO8V1FtxXgypwynSNVQOXFSMdDK9f6 +oD/hw90CdroGtG/9AVs8a7W7q0JSlTm6LNG6YgsylRllbotrq6KkMgbp+PZabPyFH4Ia77O7/tK7 +hX7vAwPu+DdC3PpIQmz9oXWScHE9IRAJVZlR5SnXwwmXVzfMZhNyE1JpH+G46JplSRhj/fUNbXbN +ULX1jbb+0e04hZSW8lqounOPd/y7je60BbINFnDa+G/1vO4AGBsQw3Z8VyRJyuX1hG9VRlWmVEXK +zXDK5fWA2XRKZkJKHSBcd+sP2pjI3O6nu76l0T/0MTcg02YtN6wB5m7NhPje9W4Bxdvr/bj/t/Vd +7zwvgbA+8baY8/fzHzdSUsqpCFxNJ4IHRzF/+vkR9486uI7CdRSzPGcynXF+ecOr1+ecvb1EG9DG +EAY+/V6HvV6Hhw9OCAOfjtckimzMFXgCU2VUeYLjhmhdIe8AjpvY7tYnvwX03tkvQoDczF/WBSKb +ud/u9XcKpO8WI//oi+M27tj651LVayvqc12hq5JqU8xSFXWsXW3/f1Uk5OmcMnMpcpfK+Pbed4C1 +dzrcf9Y2oKXeFrnfFjuwjVu0qbbz/eH52MQdEnFn7bZx2+bdoCt0maNLidblbSwL9dwyynxFmbuW +3baeP0aD3jQNaMpiTb6ekxcOZeljHGmLfkyFMap+T9y5r7Gx7/Y5ff95bfbIhiq9pksXd9dg8059 +Xyz3I8fi9nqb9fj5c3T7Lq3AVASuw+lhg7/44phuK6QZB4BmvU6ZTKe8fH3Oy9dvGY6mG+iafrfN +Xr/D8dEeCEG71URgEEIjKDHanhMchXJLpKMtGwPqTtxq6vX84XztrpE/XLcNQFtP5Afr/i+N1ev3 +uaX1/94+vROr67rD+/Y9aPM5VVlQlTl5Bjc3Q757pvEUaF2xSnPOr5dMpwuS0qciwmzWY7OL734f +v3OP979Lbt+l4h22wXemWL8b39mbd8/Wnetsztbmu2jDWLiL639/2wHff0BzXZd2u829e/d48OAB +Dx8+xPf9Lah9fHzM9fU1FxcXXFxccHV1taVJXywWXFxcAHD//n3evDnD81wajQZhGL73flprqqpi +vV6zWCxYLpfkeU6e5xhtkMrSWvq+j+/7lkI9jgmCgJubAcPhgCRJag0KQafTodPpEEXR9jPfN2MM +s9mM2Wy27VzPsowwDOvrN2i1mjSbzXc+oytNWZXM53MWiwVJkmzHqpRCSonruvi+TxAENOIGcSPG +87x/FQ2jDcV8kiTM53OWy6WlRc8LAKSSSCmJ43i7TmEYEgTBT153sVgwGAwYDUdUdfWb7/vEcUwc +x+R5Tpqm5HlOVVWUZYnrujiOQxzFNFtNWq0WSikcx3nvXO+u32Kx2K5flmXvrJ/jOARBsL1/s9l8 +7/otl0sGgwHDwXA75iAIts9/tVqRJAlpmm6p49vtNu12mziOf3Rv7GxnO9vZvxfTxtj3Z1UgdIWi +ohWFPDhu8/GDfR4cd+m1IwSGyXTKcDTl2asLvnt+wcXNlHVWoRyPMArodWJaDZtkS5OU65sbvv72 +KS9ev+X8YsB0vmK+rmguK5qtLqfDKceHS6IoJI584tCjGbk0QsU6LdFlCTh1tXllw3JjcB2J64Cr +wFUKJRVSmC3QV2n7XZMXFVmZU5QGrSUa66xL5dwG+r9nB6oUBikg9BSthk+3FREGLmDI84KqKqmq +yvounkscBTQbPs3YBQdMrXWo3DqhIIXFvWvbJs+2sY4NolarNYPRmIurEdeDKdP5mjBOiKOUdQ5Z +Cauk5PRkn9OTfcLQp99tUpQlrdjDkRqjc6rSIRUlRZ6xWhW0mmuW65S8KKiq2w7z5SphMlsymq4Z +TdYkpYNWHtJxENK5Dc5/oX/jKoFSkkbo0GkG9FohrqtwlGQ6m/Ptkxc8efaKy8GE68GUNK8sbbly +mSxKbsYZg2nGYJpwsDdmtbaFmfNVQl4UdxaSLUCttQBTgrHUlr4n8RyJUgJH1TSb2qCNoazsT1Fq +8qqkKKmLKBSB79CIXKLAqZMvuparyUiyDM918L1b2nglJUmSs04zpBREgUfge9tgPS9KkrQkzXMq +o2xngVAIuaEgtcGsq8BzBb6jcJSLK0EbTVGUZEVFVhryIqeqNolRiU2e6vqzEt+181WyptZ/z3yL +qiTfzFcqAs8hDh3i0N3ON89zVuuUJLXz9TwXdzNfpUjTnHVaUlSaygi0lgjpINnsGX6vc7ezne1s +Zzvb2b9rE7UPKg2Br2g3fPrt2Pokd3zQsqpwlMLzXOLQpxn5NCMX45jaD8wtAG0s0LrtJAZuwdgN +CFuihMb3FYGrULLG5dAUmaHIK3otl0YocR3rU4ktkKa3AK3WJb4X0G17tNqNDcZJURSkaUaeF7iu +g+s4KHUL8haFzT2leUGWV+SlxnGc7d8NfQfPc7bgkGWpLMnygqIy5IUGJKKW59l0emptalkfXXd1 +WklHq0tbUVFtuxDFlq7WYIzAkQLXBVeCoySOUkhptiu4iUWK0pCXhY1FTB2LILcggucIXEdYPema +OthCR1WdI7U/aV6SFyVlVZd6GrEF5HQlKauSoqzsPLSdW1layaKKagvW33ZBavSmTNGUSKHxaz/R +UXYcUgqM1ls/MS80WVlSaUFZgq51nS21s92cvitphB5RIG1xqbESmUmSkWY5Xv3crL8oa1ZRK4eV +5RVpXpDmJYg6zpDWH5bK5ffuMDQW2NG6xFWCg27I/YOA08MW/XZkO73ThPE64fmrc16+vuD8YsD1 +aMpwNN+Ceb7vMZrlXA1XjOYZ18MFh/td+r02rWaDPC/qAolNEYFGUOEIieNIfFfgOx6iPr/CGLS2 +FNF5XpHnBUWlKSooC6zfrNztGmzYq4wxeI4gjh3iwGHj65dFQZoWJFmG67h4rkI5EikFSkrKSlOW +mqKoSIuSNKvusD7cnv3IF/iRz1E/otP08VwH11FIIfBdh24r5vSgwzJXrHKHwrgo5SMd2/m+7WGu +w+z1OmO1TtFG47senmefv6MUldYkaUGS5VRGUJWbuLJ+7tj3neOA5wh8x0FKtudEG43W2u7NoqKs +SspKUGpu949Q23W7WwhszPdKber3ldYWvL4tcr4Dlm6BwgqtC4sNm8075U5RkNkUedtn47vgO8K+ +2+rxa63RRlNVhqrS9oyVBXlpqO6Mf3NP15HEsSIOfJv3wFCUJeskJ0lSHEdt4zHHUThKkeUVSa4p +qsrmfCzF1wecI4Oq900jrL9nOhFx6OG5itl8zcvXb/j62+dcXI+4GU6ZL1ObX5EO40XJzSTjepIy +mNizkuc5y1XCbL62TG5ms1vM92J1e3aU0PiesrGrrL9XhM2RGG0otaGsoCgNRWXXzdTvVtd1aEQ2 +Xhc1kF6WJet1yirJUErib9ZKKZSjyLKcdZJRVZoo9AkDz8qC1O/zJC1YZwWVlmhToY1ECMey99Xz +UAI8D3xH4ioHpez3ZJblFHnBcmVYFRVVab/T8qK0he7Yd3ZVf/eUVYUWxhId1oC/MAaJfXd4jr2H +2nxniM2etGfcfoeV5KUhL21Bgd1PcpvPsGVn2o7ZF/iOxFECVad1Nt+Rm725yRFkpamvp2r9cbkF +13f2y20HfP8BzXEcms0m+/v73L9/n08//ZQ4ju0BrUpWqxWr1YqLiwtevHjBixcv+PLLL7e052DB +7FevXvH8+XPCMOT4+Pi9wLcxZkuhPh6POTs7482bN1tgVGu91RvfAJoHBwecnJzQ6/V4+fIFv/3t +bxmNRmitkVLy6NEjPvnkE46Ojuh2u+8FN3Wlubm54dWrV1xeXjIej5nP5/T7ffb397l37x4fffTR +O8C3TaRbAPj8/JyzszOGwyGLxYLValU74C5xHG/Henp6yj3nHq7rAvyLwO/NWiXrhJvBDWdnZ7x9 ++3b7PIwxOI6D53kcHR1xdHTEwcEB+/v7Pwt8T6dTvvzyS7788kuqyjqfzWaT09NTjo+PmUwmTCYT +lsslaZqSpilRFBFFEQcHBzx69GgL+Esh36vrbp3/gizLuLi44PXr1wyHwy2AvwXS76zfycnJdl2/ +v36z2Zyvv/6a3/72t5SlBRm63S6ffPIJH3/8MTc3N9zc3GzHvVqtePjwIY8fP+bk5IROp7MDvne2 +s539uzaBsbRcpkKJEldq9jo+Xzze408+v8/Rfpsw8JjOZpxfXPPi9QVPnr/h6asr1pkB6RM3rSxK +HAb4vtU9y4uSi4trvvnmW86vxsxXGWluKExKUvr0RyuuBnNuRlP2jMH3XRwliQKXOPRw5gW6TDG4 +6DJDVyXSGJAaRygavtWGjkOHKFB4jkTWFFBJXpBlOfNlxWResMhzjJaUWoG04LS82w3+CzsIBNik +nzCEvqIV+3TbIUrZbuk8z0mShKIoieOIKAqJQp9uO2av02RVKlZFTlWkmE0lvRAY1LZS+A7ivf13 +ow3rJGU6WzKazBmMV4zna7wVeIFhkcJsmTGcLCi1Zq/foRWHtBoxWV7aYFXZQBJAG4GuJIUWpIUg +LwxlpdHaJscqbSjKijQrSXNNWggK46BcD+WENhkjFOKXdHsbmz7yXEHoSpr18/N9ByXtdYbDCf/0 +u2/527//J9LSjs0IF+UGSEczmle47prW5ZwXZzd0WxGe5+J5ivE0Ic21TRTUAKtBW2Z1Y5O3wpR4 +jqIbu3RbLqFvf4TAaoqXFau0YJ2ULNYF02VJUgPvQjoEseK473PvoIGoO82nswXnVwnDPKEdRrRa +AY3YJ/RtguBqMON6WOA6ioN9j4NuEykFQknmy5SrwYLBOCcrJVkl0ThIx0UosS2CcDxJJ1L0Wg5x +oGiELmVVMV+mzJYpk3nBJM/RJduEqqBCUOG6km7Dp9t0CQOXyLcFina+mnUNVM9XJfNVQVKUdYDs +4IWS477P6WHDUsqjmc6WXF6vuU7XtMOYVtOn2bBJAt9zuB7MuR4lLFYFmZYUpUA6VndUyrvJy53t +bGc729nOdvbhZlDCIJQh9CTtZkivE+Hc8UHXtQ/auOuDtiL6nZh1pVgXee1bhxhTg4zfd4VrOnWo +kJQ4QtMOffY6PlHg4LsKR8I6USSJoNcO6LYCAt/BUbJOnLOlrd2ASJGvON2PuH+/a5P4AuaLFcNx +wWKhacSSVtPH92w8IYRguU5YLDXTeckoXbNOUtwwxHV92mHEXjeg04qs5I2g9pUrpvOS+SojyQq0 +kSg3QChvOzAhLJOQ6zoIJ0D4DsFyjVIKbcAIoNZW3YBmeqN76wli16EZKRp1caDnWpBRQA3SlyxX +OdNFxazIMUaiK8vqI6VCofAdl1asaEQOsS+JfLUtbCjLkjQrSJKcySxjMk+oKo2uBJW+7Rw3RqGk +wHUVwvERnkMQJijloDe08NtuT3HLWAQgNFJUOFS0I59eUxFHLr7r4Ls1WFpVrJKC2TxluixZZxbQ +r4xCOm69lrabMPQdjvcC7u1HCKMRaBbLFTfDnPG0oBk7NCJFGPpb4Gmd5iRpznS+YjhOWWUrkL7V +gVee9R9rcOW2i/dDnUhTd3tbeuXA9Tk5aPAnn/a5d9Am9F2yPGc0nnI9GPHPXz7ht18952owJa8E +eVV3SkoHpTQ34wzHnfH6ckK3+YZ7x3t8/vEDPnl0j+Uqpap0vcR2z0shULIi9gTdpkO35VkA15UI +NGVZkZcl09ma6axivsqoqooiKZFOgHJ9pPKQyrESTvV1fdfhuB9wehBuO2nXq5TBaMZwXNCIFY3Y +J6qBSs91SbKcJC2YLVKGk4x1noJyQXp3GAWg2XA46gY8PGlx0I2JQt8C31ISBC7HBx0+//iEJBek +paDUtvhBSBeDLaoX2MIJqQQXV2MurguKoqTdcGi3HcLAI/I9sqLkelhyM8pICygrQaVtkbqQFjhU +0hA6im7To9vy8D1J4CpkHbcVZclskTKd5yzWOatCU2YGoTyU49XgHHWH6u276H2dv2ZD927uUG6/ +5+/cZTUw36fkxtj3qpYICgQVsefSbXq0YpfAl4SeUwOKJVluwetVkjNdFpRZTpYbG2crb3NTQtfl +oBPw4LCBUjYPsVpnXN7kXGYJjSCg3bLsaWHgEQUeo1nKzSRhtizIK0VZUctLqB+JwzbjB8cVRI6g +GTk0Qgfft+dVCFgslnzz3Qv+3//f37BYl6SFoDQK5fhI12c817huQnQ549XbEf32G3zfw/cd1knJ +Oi22rAY2G2XzUcYUCFMhTIHjQCv06DV9otAl9C0rQ1lWlJUmyUrWScEyKZktC9KixAgFwiHwBPud +kAfHTVwlEGiSdcrFdcZFlhAFPu2mV8euLqHvMZouuR6syYqSg17E0V6z3veCNC+5Hi65GqUkWWlj +dS2RytTyF5YBRDrQDB16TYdmnadSAmaLNbP5moER5OsKo0ukANdxCALHsrUIheMkttDmDuubqdkd +hLA/noRO5NBtukSBIvAUjhJ2XcqCdZqzWuUs1xWzZU6a5/Z5K8++UzfFRBgQGuWYep09otAWkrlK +bnMiSVayWtv9aa9XoIVj383KbAuT7ggp7OwX2A74/gOa4ziEYUi32+Xg4IB79+7R6XR+8Peur685 +PT3l4OCA5XLJ+fn5tmu7KArOz895+fIl3W6XRtzg4OBg+9lN929RFsxmM6bTKa9fv+bJkye8fPmS +yWTCbDbbAt++79Pv9+n1ety/f5+isADqs2fP+O1vf8vV1RV5niOEYLVaIaVESdsl/j6q9UpX3Nzc +8O233/Ly5Uuurq6YTqfbDnfHcdjb2wPe7UifTqdMJhO+++47njx5sv3cYrHYAvStVoter8f+/j5J +kiCEYH9/f9t9/Uttc/+yLBkOhwyHQ87Oznj69CkvX77cAsebtQqCgNPTU05PT3nw4MG2m9r3fXzP +fy8ovVgsePbsGX/3d39HWZbb7uibmxs++ugj21ldg9Sr1Yo0TYnjmEajwb1790jTFCEEvV6PXq9H +FEXbKt3N+JMkYTqdMp1Ot+t3eXnJdDplNpvheR6u69JsNun1evT7fdbrNVJKiqLYdrHfjnnO06dP ++du//dvtmPf29litVpRlyfn5ORcXFwwGVitjU6DgeR5hGOL7/nv39c52trOd/fsxG+BIYQg8Raeh +ONlr8OmDfb54fIhybMXzZDLn1dklT5694eXbAVeDBSifRqtJGDcJgojA9/Bc675pXbJYLRkOx0ym +U5JcUGhFSUmmC2aLjOkisVTdcYQxpg4AbPeDZFNJnaMr26XrSIHnKlqRZL/j0W95dBou7YaP54qa +4tGwSmwCZTgVeKrCESWLdUWZluhan88IYUFwI355QZoAxxH4UhAHDs3YoxkHZHlOlmYsVysWiwVZ +liGk2CYd+90mB/striclq2yTdKy1EOWms/fOk9nq4tnf2hjyomS1TlgsM+arjNmyxM1LVJaTFss6 +MMk4PjogzwtUu0HoKFqNiChwcJWpq8SxnSdaorWiqASlFmh9q5duu0U0RakpK0GhJRUuSnoot07C +SPWubuLPbDWbVzU4CgJPEvoC33fwXGebaJgvlrw+e8s33z1Deg2E28TxGijPwfFcjMnApAzGKy4H +NtnYaTXotBusM01e1MkCeUt1JtAIAZ5j8JWk11Qc7gUc9UIakUcj8mwCpdRkecl8nbFYZQwmgKko +i4pKG8qqwlM+hx2Pzx+0UMICwYMxUK6gkOz1PPZ6IZ1WRCMO8FwHUa1ZzQs8V3PYVjw8DnE9F891 +GE4dTJmQrDVmXZFVUJlqS0tmaro5T7n0Wg73D0K6LZ9u0ydJMy5vSkxZsVoVUGVUJZZBRyg8BwJX +0Gk6HPVt90Yj9GjGt/PNy5L5MmO2yhnP7NkrCk2lbce2pzwOOh6fP2gjhUZhGE4EslpRZUv2ei57 +vYhOO6IZW7pHqTPWyynJOiOrBFWxoSB0MMLcoWzc2c52trOd7WxnH2a2Q1cp8KUkDhxaUe2DZjlZ +mrJY2lxclmcIYX3QMPDodmL2e00Gc2O71ooMxyswRiOMsfLT3PU768I5pfGVIAps0d/JfkQr9oh8 +B9cRrNYu68QnDl36Hcvg5Hm37H93ASFjNIEvOd6P+NVHHetDCZjMJLFbMPZKOp2IXjsiCr1tl+lk +rpjNBVeqoCoEVaFpRoZm02G/53NyEHHQb27ZmBZLh5ugInAKhM5I1xlpBbrQCK23tK1Sgu86NCLf ++rZCM1+scR1nS/+8mYOp18SR4EhDM1TstR36bY9uw6fdtFqtUoEU2C7MtGA8k/iOtrFIollWtpvU +cwS+79Fruux3PbpNj3bs1LJJBiWhKEpWScZqlRK6JUpkTOYli3VJnld3KJ19XEfaeQgPJQ2rdYLr +bfTbb39tQDtRd4x6yhC6gsiv/cRuSKfpEYUege9QFCVlafVnbwKD72omixyjS9JSYwzoUiCkRkiJ +7/gc9kI+f9Cq9Xw105kkUDm+TOm0bSFwHAa2WNJ3Wa5SlquUa69CF9YfLnRGXhaYqsKh1hKXDka5 +v4D2/K4GdgWmJHB9Dnsxnz7cp9mI8D2H2XzN9WDE85dnPHl+xtdPXjOepTh+A8ePkUrZMA2NrkqM +LhmMFvgujKYrjHBw/YDJfE1ZGVvUoCSOssxgjUjRa3kcdkMOev4WrJLCUBQFeVFwHRpCp8BVBWKW +k2e5XduilmUy3qYN0wLfjsN+x699c2P37tIldAocUjrtgG47ohEFhIGL77mskjq2GRtMlZImmkKX +5JVGG7ktCG+ETe4dNHlw3GKvGxH4NsctpcB3PQ722piqIq+70ysj6oIEaWNHY8+AkpY9wBcZOl+S +5YK9ns9+P6YR+TSigOU6o8pXzKY5Ra4xJejK7mlHCgJHEQWKbtPloBdw1A+JAgv4KSEoyoKirBiM +JDe+YTjRCDLKIrc1z5UFJDdnZaOF/JME7Jv31U/8LfPO3rql4d4UyAgErtx0qSv2Oy5H/ZB+26cR +usSRS1mUlGXFOs2ZL1PmK4krNbq0LBmlNpRFtbkhDoJ+y+HTBy08R6CkYb5MUCQUqaLTsvFnrx3T +iAOacciryxlFnpKmFZXGMlZsGNt+puPbEQbfFTZWdxW+a/c1QJJmvL244utvn1EaF+E2UV4D5Skc +z8OQY7TGkUtuhguiQNFuxXQ7TZAOSVYiRN2BjLjznQOe0vgKGpHiqOdx3A9pxj7N2MOrge+81CzX +GbNlzmSRIoWmLHLyylBojULRazp8ctokdCVSGJYrF2USitShGbvs9SxbYSOyxepnb3PSpWZFzn5L +8PAoIAx8PFexTgukyUnWum6GLKjKmvJbOlsmASkk7Uhxby+g3/bpNAOkNFxel0i9Zjm3LBAb9ok4 +CmjGPo4jUW6G7y9/kFORgJJWXi9wJI1QctjzOexZFpc4dPEcSVGWFEXJYpUym6+ZzAxKFJRFRVZ3 +gVdViXJcJB6uK/BdQSNUHPV8jvohrdiyL3quQ1Ha/blYZcyXislcICkpcsuGVxmBLkE6LmYjUbBj +dPvFtgO+/w1YFEWcnJygtebly5e8fv0aYJvwnU6nvHz5kn6/z9HR0TufzfN8CyT/9re/5euvv+bs +7Izr62sGgwFZlpGmKcYYlFIopYiiiDAM2d/f5+nTpxweHvLmzRum0ynL5ZLZbEaSJARBgOd5OI5D +o9ng+Pj4nXsbY7Uyr66u+Prrr3n+/DnT6ZQkSWi32yilCMMQ17FVdev1mtVqxatXr/jyyy/57rvv +uLq64vr6eqtrnmXZdpye521pxp8/f853333Hp59+yq9+9Ss+//zzW22eD7Qsy7ZU8r/73e/46quv +tt3mo9GIPM/Jsmy7Vo7j8PbtW9rtNsfHx3z88cd88skn227n9xUCpGnKYDDg7OyMsiwpy5Lr6+tt +ccB6vWa9XpNl2ZY23PM8PM/jzZs3vH79mq+++oo/+7M/4y/+/C84PjneFgJsPnt2dsaXX37JN998 +84P1S9N0S3Xu+/6Wcv7Fixd89913fPLJJ3zxxRd88cUX2/VL03RbBLAZ83A4ZDwe8/Tp0y2V/Wq1 +2tKpbwonjo6O6PV6f+gjtLOd7Wxnf1DbdC+HgUu/3yJqtHn80QmH+22UEiyXSxarNU9fvuWrJ2c8 +eXnJZF4gvQbSDRFuiBEeGklVUyIJIXAdl/1+n0ePHiAcn+vhktmqsFWmSuG4HlFogzLfcy2VYVmy +XK9ZLBekWYauNEiNMSWO1Ox3fE4Omhz2Yw57EXvtkDCQBL6DKwU1Axx5WZEXBat1znyVMZomPHl5 +w/PXA9aFxmhJVQqUI24Drg82m1SLfId25NBuegSeoqoq5vMFo/GUyXTKbDYnz3OkUnTabcLAZ3+v +x/HRAat8wmA6tTp8pQX2hVTITdD8g3vedlSLmp5KKhfHteCpE8S4XgPluRgpKLVrExh35yWw1HrC +JiNMre0mlQtSWq3xu9Tl9aeFlEjpIhUoR2KEj3Jsx4yljJT1qvA+cawfrB0bzbOaXlJXtxpnG79I +KYVXy8UYJwQnRLkhrh+j3NviN6VAS0FaSWZrQaYLigpy7aJchayr5KWwVN+BK7h30OD0oMl+L6LX +tuCx7zr4vqUy09p2vadZSZKXjGdrrgZLrkZLLm8WXI2WKDJaseJkv4lSluay0wpoNQIene7RaoQ0 +GzZp63sOAsN4PObcrRBCoyiQQtOMPHqdBkHgkiYpWbrG6ITFKkWXtsvJUtzpWk/e56Ab8emDLs3Y +oxX7DEYzrq+HLOZTknVJkRWgBY7yCF2Pe4ct7h+1OehF9NohvZalLAw8W7GvtaHUdr5pVjGer7ke +LrgarrgcLLgcLpHGoxkpTvYbKGUTOZ12QLsR8OBen1YjtIUV9XyVEkwmcy6vDI7IoTJUOdt9i3Lg +Hf3yne1sZzvb2c529tNmQRUpDbGvaIWKTtMj8K0POpsvGI8nTKYzphsfVCi6HeuD7vXaHB/2WZcL +BtM5lUm3xZdWQ1nVbpqlN3eExnU0/bbPyUHMvf2Yfiek3wmIAptolxLyoqTIczzPoRn5NOKAKAxQ +St0Z+W0HrOdKOs2Ae7UP5SjBXidgv9tglaREwW2XqpI2kb5Obdfz/ZMlH43mTOZrWo2YVrNBuxXR +bga0Yt92cAtI0oLT4x7T+YonLy9xVcVomrDMMpIkQyrX+tEyotMKuHfUw3UEnmNjkVdnVzgSNMZq +qOoKowswDnutsI5DQg76MYfdkCBw6+JSgRTWv8nLiiK3HYmLdc5knvL0bMTz1wMQhoO9iIO9Dgf9 +iMN+TLvhErqKwLcMVkJAVVXkeUVWFDxYpiyWGWcXQ757fsXrtwPKSlIZiRIRnYbHvcMurmtB9aqq +ePXmGlfacgZxh3LemBJPWerlfifg9LDB6WGDTiOg0/KIfNu97jqWilpXmiQreLDOWSwzzm/mXNzM +GUwTJvOS6WpVr6mDI0M6sbv1kR0Je92QfidmvtwjCjyi0K8pma3EYZYXZFnB8UGHo/0W926mvD4f +c3Y5YZ0mYBoASMev9cU3+rU/e2S2QLErDdKFRqisD920BdtKSVbrhNdvrvjnL59yNVhSiRAvCnGD +Jo7ftPGRsvCE0SVGVyhRoUXFPFE8ezNnnrzi/HpOVklcr5bvanicHrV5cNzioBvSafi0GxZwcpVE +CJuTLivN0X7KYp0xGM05v55wfjVhPMsYz1fkeYLxG9w9UUr4tBoeJwfNGmCGPG/S7zR5eLpHFNp1 +9l0X15U4jmVjy7KC6SLhaK/N6dGcs8sZZ5dzlut8y2zlO0267ZD9XrMu4HVRNfuC6ylazSZSSCoD +lbY/1Hrvm/2CEDjKduj6nsN+r0FZapqNkFYjxPNdAlcxGC8YDMd4MkfokiqvMMbB9yVx4HF6FPHg +pMNRP6bd8Og0fDzXSlUJUd9Pa44OclbrjMF4xcXNlIvrGeN5wWRpZaiU4yGUVxcRfIju+O/zira0 +5kpqPCVpNxSnhy3uHcT0WgG9VkAzcq3UlutYmvvKUBRVzVZXcDmYczWcczlYcjlYcD1aoHV9+Qoa +geR4r0EYuCglyPOCViPg3mGHOPRpNULiKMD3bIf2Oiu4Gc3wlSYTm8IXudUg//GDU2MmurIsE8Z2 +gW+65qWyLAKBH5AbD+OESC/E8WMcL95eSQqDdiA3gkWqKKYVQsI6V0g3RCq/LqC3Z9SRhuO9mNPD +Jod7Eb1WSK8dEHoOft3ZXFVW1i7LS5LMsnpcDZdcjxZcDhZcDFZQJcSB4KjfIA5cXAey3BZcHB+0 +CAPPrlXoE/gW6E3Way4uDVmSo8htp36g6HSaVNqyeKTZGmNWrJOcKrcPRii11Y93cOm1fB6fdtnr +WFA7z3NGwzHr5ZxkPafI1lbuLQ44OezRadkCnOki5WKwxHMleQVVTUXuORD4gsN+yOl+zNFeRLth +GQR818H3JI60Z6GsNFlWkGQli2XC1XDO9dC+r89vlsyXa5wwwvPguNfi9KjJyX6Tbiug2/aJ/Hqd +paivZ+VTkqyqr7fkajjnarjiapQwWxdAYHNK0uaU/jXkff892Q74/jdgcWy1q+M45tmzZzx//pw0 +TWv9hDWz2YzXr19zcHDAF1988c5n8zxnNptxcXHBP/zDP/Bf/+t/3XaMb0Dcqqq2f1/Wh0hKSavV +4tmzZxweHm5puReLBTc3N8xmM9z6yzmOY+7fv//esZdlyeXlJd988w0vXrwgTVOqquLBgwfb7mjb +6WaB7/F4zJMnT/jLv/xL/uZv/oYkSUiShLIst2PdjG8DzDqOw4sXLzg5OWEwGBBFEZ99+tndHPYH +WZZlWxr4v/u7v+Ov/uqvOD8/J89ziqLY6hHdXavNT7/f5+zsjKurK4qi4ODg8L3Ad5IkDAYDLi4u +KIqCsizR2uonSSm397j7e+sAuS5Pnz6l1WqRZRknJyd0u93tGiRJsl2/v/qrv+K//bf/tl2/zfjf +t35SSl69erXtrA/DkM8++2wbSG2A77tjFkLw5s0bXNfddoHfHffR0dGWnj7Lsj/0EdrZzna2sz+o +CWGZpkLf4eG9PR49+oijwz32+x0cR7FYrbm4uOLZy7d8++wtz14PkF4T6bVw/QjpWODboNCVDZKE +AMdRHOz3+fTjh+SVZJFcMl/PkdLSoTmORxgENKIQ33dRUlIUJevVmsViRZ5bCnC0rhMMmqOexx8/ +7vLRSYfj/TZ73djSqMl3YzdTU5NZzazKVuPrgqvrG7I8p9TS6jJJZQM/+MAO1E3QB1Gg2OtYmqnA +k2itmc2XnF9cMRiOmUzGFHlJp93mwf2SKAw42OtxenLAzThB6NxqRJX5Nplm3huIvzumDRgqHQ/l +guNrXL+JG7RQrkRLTakVlVGY7zkbArbAt64fvBACgaoTAmpLdWc/ILeFCtIRKFdj8O29N5pztYCb ++aAkwm01vNbaBtNaYyrzzvyUspItnh9gVIB2ApQXorwY14+34LzAaqNlWpMlINICsHT2juds9QCl +0LhSE/uKRycN/u8/vse9wzZR4BEGLlK8S7ttjO2u19owna+4vJnw9nLCb6qcwXCEIqcdK+4dNHCU +xHMVlTY8vNcjL0qrhee6W6rPPC84e/OWwLH+qqRAiopWw+P4oGM7tZIV6XrOfJFAmVDmAqEcpPbB +VBht9eEO9yI+e2gTWnHoYqocQclyMSNdVeRZBcJBeYbQUzw6bvD//Ok9Tg87dRLM+4n5wmyx4vJm +ysXVmN+YgsFwjCSz1esHzR/ON6/n67g4dRKq1Jrzi2tCT+OQQ6UpC20rwnVgOy8+mB5/Zzvb2c52 +trOdbTS3BVZmx/qgHr6n0Nowm89rH3TEeDKlLEq6nTZFec8WX/a7nBzvczPNQZdoU7MOVWXdrVWr +lBqNMCWuowlczVHf408/3eOPPzkgjjzimvZ440dsdHIl1D651W7efs9vtMI3wLcj6TZ961M4Ctex +VatlXRAphdzqe9/O3VAZw3KZMJkuWK1TWq0G7WbDSixJheOId9aqrDTrJMVXFWU6R5cpWZowT3KU +F9iCSmXoNkNOj7oEvovvOSxXCa1GYKmEtdlSvusqRxiXvbbHrx62eXyvy73DNod7bYQE9R7fahOL +aG27E6X4jsHNECHh0XHEZx/3OdlvcrzfJq6Zh6T4/nO3ury6tD7z7749Yzmfcf42qeMIiaKk3fA5 +Perh+y6B77JOMlqN0FLgarZ62xvw25WC2Bec7Hn8h8/3+bPPj/E9t6aYl7U+861/r42pQbqSp6+u +eP7a4dmbMXk2ZzRZYRwfY3wcaWg1vNpHvvN8S01Vy1JKKba+qBBi63OvVgmjSYfrwRRT5lxcXFAm +Nl8npAvIbZGuLdb46bjtlmK/xJEG34FG6NBqBLSb8TbnuFxZ4Pt3Xz8nrXy0DPDjGDew8dVWa7k+ +H5Y2PcNUObNUszqf8/pyQakFlRH4fkAc+nSbHh/fb/EXXxxz76BV6xWrLe32pgPfcAsYD0Yznjx/ +SzMwPHk5YDpdk6eWPn0zBpsnNbQbHvcOWriOsjT3WB+8Kn98nY0xrFYZo+MFw+mc//6bF1zfDJmk +C0up7rh4rqHXjtjrNmg1rJzU5r6e69JtN2k1o3qPU++PjSyn1ScGtjrjh3tNsuzQnn/PxXWdely2 +4/TszMGTOVLn6CJH4+JJj1YY8fheg7/49QkfnXTt+nmOnQ+3es9A/f7QjCYLnr26pB3Ck7MJ88WS +MivAREgkxlQfGLP+gnfzZrdtZBCExnc0/Zbii4dd/u9fH9OILPW4V49f3jlbmzNutOHyZszl9Zhn +rwdk6ZrzyyVVZSzXh3ZpRIp7h01bPFKz7N0/7pHlhZU8da2+92Z9x9Mlr98qXFVZuWhtanrxn5lQ +fSZ1ZZlbzR2sQQir/e15Pn4YQOmhnRDhRThejOs33mksMKYkMxV5JljmFQiDEa4tZJF2DsJoHCoC +pbm3H/AXf3TIpx/t14Uy3vaddNe0MegKVmnG1c2Eq8GUf/pWM5nOKKuiLhKIaTUCvJrx9v5RhzQr +cJTC85x31mo0GhD7MBcZigJhCuLQ4bDfxHFc0nRNul6wXCbcjHLKPLf0/o4PGHRVoqSi1wn49EGX +ficmCj1miyVPnmvWqznJakmRJajQpxn73Dvq0+82CQKP5mzN09cDW/SVG6sPj8ZThtgX3D+I+fMv +Dvj0QRffdwl95/Z7Z8MGgS1qN9qwTnOubkZc3kz4p2/eMB5PGWcLhA++VBzvefz5Fwd88eiQMPSI +AvfOu5/t+bZMzYYkzbm8GXNxM+F3T65YLNeMp0ts17uLEQJh3F1h+y+0HfD9b8A23bmNRoO9vT1O +T08ZjUZMp1OMMSRJsu1K3mh/b17wk8mEb775hi+//JKvvvqK8/Nzlsvltlva93183986XEVRsFgs +tlrNNzc3rNdr4jgmjuOt/vWmi/zt27ccHR0xm83e6SbaaG+ORiNGoxGTyYTVakVRFLiuu+0o39vb +IwxDyrLk7du3fPXVV/zDP/wDz58/Zzgc4vs+cRzj+z6e5+H7vv1SqOc9n89JkoTZbEZZlrRaLT7+ ++GPO3pzRbDZptVo/u76bOW3W6p/+6Z/4+uuvuby83GqKh2FIGIYEQYCUcgv2rmqq1fF4zOvXrynL +cqs57roOjUbjHdrwzRrneb7tIN8EHJu5NhoNPM/bgtOz2Yz5fE5Zltv5boDqKIo4Pj7mYP+A8/Nz +vvrqK37zm9/w9OlTBoPBlm682+2+d/02z3o6nVKWJXEc8/HHH/PmzZvt+mmtfzDmDVjvOA6+7291 +wzed8N1ud1uw4Ti718zOdrazf9/WaEScHh8Q+C6n9064f7JPGAaUZcHNYsGLl+d8++wVz19dM57n +lMbFU6Glf/NChHIptWS6yDi7HFNVJe1WiOs6HBzs8dmnj5FOQBg1uLiegfQxKuD+cYf9foso8qkq +zWy+YDiecjOaMhzPWGcOBgfPcQgDh1bssd+NeXDc5nivQafp4SrDZDplMp2T54XVFBOCwPfxA584 +DIjjkHYjIPBsYghdYkSJEdWG/ww+lHbZtjUjgTh02OvaSmTfd9BaM53ZpOPl9ZDpbI7Wmo9mS6qq +Iqg7vperhGevBha0rXJ0lVnKc+UineqOxvc7N32/iTv/IsD3XFqRZK8f0owDVK0PXZYVeZaTZQVZ +nlOUGstQrkBY2swf1+q29HW2QdyC5NTA86Yr6K7W2Wbs5n3DrE1jyEWJNKXV9ctzG7Arq6PV7TT5 +7OOHLFcJ68IhyRWF8TDSRwuPsjIUFTVNu0FvdA5NXRQgbNeJ/W8Vgetw3G9w7yDi4WmXk/0Wka9I +1gsmo4w0z0jTrO6kkjhK0em06HZaNEKXg14TKQQXN1PeXDg0QknoK7w60bRJPHiupKo0WZaRrBbk +RWllctKM8WTCer2kKCrG4wmB59BtNZASmg2f/V6L9Trh/GaBMDlloVGOj3EDHGlwPWiEklbs0WkG +uI5ESevTzuYzhoMB60JRFIpmo8lRP+LB6R6PTruc7DdpRA7Jesl0ktnxpbmlZKsT1N12m06nRRw6 +HPRiXMfO96ypaAYQ+gLXUbVOoNXK811FGVT2esmSPC/s/POC0XjCerUmzzPKymqVbZJ7m8B8Fxfv +bGc729nOdvahZv0sAcSRw173thOu0prZbMnbi2uurodM53OMgY9mK6pKWx+032G5Snh+NkaKEl2C +rgrbwVopjLTgltG2OK/d9DjqxTy61+H0qMV+N7Zj0CXz+ZLlKiFJ0i14GIYBzUZEoxEBd4HoLSy0 +Be6Vsp2jXu1XlKXtqM7yjOlszmQ6w2iD47oEgU+v06bXbeO1Y3xPkaY2TyVMwXy6YL5csV6nVh7R +82g0IrqdJt12zMlhj8XikCwvmcxW5OkM15R1h16BlAbfc/A8lyCwnX9SYOMF7aB1iVIVvguNwHan +Pzhqc++wRbcd4rkwndoxpzU9tTEQBD6Bb9mt4jii3QrpNj0bv7gOJwcNPjpq02kFNCKXskgZTmfM +ZostEOw4LoHvEQQ+jdiubSPy8JRBmAIqiS4tsCQleJ5D4NvP+J6DlPb/bTu9RYnSFsDvNCMeHDX4 +5EGf08MW/U5MnmekyYokyVitE5I0w3UdXMchigLarSatRsjxfhtjKopKM1tkjCZzKipKbYt5lTB1 +57nEdZwt6F5VBePRgsl0TpoXgGWy2ut16Pc7ln2s28RzFQe9iE5DsU4MxuSU2RIhJHpTePshHd92 +Q2O0xnUEsa+IA2VjQiVrOUZNkhYsVinTRYrwAoQbofwmjt9A1QW3G9B5cz1dufX5KSxdeFEhlYNU +Do045OSgxaP7fR6ddDnaa9CMHLIsZ7ZeslonLFdrtNZ4NUNmu92g3Wqy122Q3tur9ZtzhqMpeZ5j +dGbXoO7KRpcoAZ7rbH1zYwwUOZqKyWTCdLYkTXMMBm1gr9dhr9cmDBz6nYgwUBz0YjoNh8XcUOiC +otDMZnPevL0idG2xQhS421xqlmVbdrO8qChLTWnAyhkJuu0G3U6TZhzZuKHWcQ98K9uZphnz9XKr +03x1PWA6m5ImCUVRoquSIPA47Ed8+mifR6c9jvYatBoueZazmK9YrVNWa9uA5rkujuvSbjXotBr0 +uzFZ2kUJwyorGU+XZHlGRYEuU1twXndm/+t0fm9iYAt6a13RikJODho8utex3f49e7ZmswlJkpKm +GWleWP13JQkCn06rRa/botMIkHTI8oI3l0NaoWCd5KRZSVUEdVGSLQDwavp5z3UoK6+Wu0hYllab +uao0o9GU9XpNURQ1gO7WRf8/P/dKV2R5xTopSNKCLLO61MpRRGHAw4/u8Rf/4Y+Zrw1JqUgr17IA +Co+ygkJjC38qWRdGCKz6u0LIu7G6xlGWGeJ4L+Dj+z3uHbTpND2SJOXmZkKa5qRZRlFWKGmB91ar +SafdJPQ8+p0I1xFcjxa8uXBJ0pLoDkW75zpIKXAdSRjcrtUiL7asC8PRhMVywXq1ZjKd2bxKM+Ke +2aMRe+x1m6RJl+vREiUqqiJBOR66CnAUeK4hCgTN0KXTColDF8cRVGXJcrFkOBoxXyzJi4IgcFH1 +O9vfvrNznFqP3CLOFa6CfsfnwVGbR/fa3Dto0msFtvmzyJgubcNIkmZIKSxLcv2dI6XEd23uIHAF +QmcIndJrdnn4oMvHp11OD1p0Wn7dBDkhTTOSdLPOtnim1WzQaTcJfJ9eJ8ZVgtF0zfnVmPEUtKyo +qhxdS7QJI/kRAfmdvcd2iNS/IZNS0ul0uHfvHpeXl7x9+5aqqt6h6E6SZPv3jTEMBgP+8R//kb/+ +67/m7du3LBYLPM/j6OiIk5MTer0ee3t7KKW2tOjPnj3j5cuXrFYrVqsVSZKgtSYMw63TXVUVy+WS +q6srLi8vmc/n6Eoj6vLJTff0+fk54/GY9XpNnue2Qsrz6Ha73Lt3j6OjI6IoIs9znj9/zl//9V9v +QeeNlvTJyQkHBwf0+3263e4WgL28vOTFixe8ffuWoii2QPyLFy948uQJ9+/fx/f9n13XjTb2zc0N +v/3tb/nLv/xLrq+vtxrmGy3xw8NDjo8ttfhyuWS5XPLy5Uuqqtp2RSdJwuHhIQ8ePCAIAk5OTt4B +vt9nm27+VqvF/fv3+eijj+j1etuO+m+++YYnT54wGo3qxGfC9fU133zzDUFgKa56vR4vX77kv//3 +/87vfvc7Li4uyLKMXq/HyckJh4eH9Pt9+v3+dv2urq548eLFlsJ8Op1u9eJ/bv02GqHGGMIwpNPp +0Gw2t9Tpjx8/5vDwkGazied5f+ijs7Od7Wxnf1Drtlt8+slD7q0P6wCiRZblLJcrLq9HfPPsFf/w +z08YzTJWmcQLO7hBC9dvIh1LaVRqwXC65smrAVVV8lD22evGHB3sEYUBvV6f03sn3IymrNOSdVrx +4PSQ44M2UeizXK2ZTldc3Qy5uh4yGE4xThPjeHi+SzPy6bUC9nsNjvfbdNsBUsJiubJ6bM9eslyt +qUqDUIpOq0W32+b4cI/Te4c4jovRuq7ytj/Iylbt13qJm07un7JN54AQgjhQHHRDem2f0HMoy4rJ +dMbZ+SWXV0PmixVSKuaLFUVZ0WrE7PV7ZHlJs/EcJSrQBl3lVGWOdOoO9/pO75q+M4Z6JMbcmVOJ +qUp8N2C/F/PRvW4dACp0ZQvEkjqQWac5eaFxXXClrHW3TQ1mU3cUcJtMqikbN3qImE31tq5/m9vu +B623YxN35mCDXFFfQyAw5FVBScFylbFeW+pJ37cJy729Pn/xf/2aw8N9bkZLBpMVs2XBMq1YpZpV +UrBKKjJtbocp1ZZKe9OZYekIC+LA59FJmz/5bJ+Tgw7tZsB6veLN20venl8ynswZT2dUVYXnuoSh +zyePP+KzTx7RbtU0f57L0V6Dg45Pq+EQeht9QXNnmWx59Gy2YDAcsViuSNKc5Srh7M0lk+mCoqjQ +xuo27u91yPOCZiNir9eiqjTfvrhCmBxdFFRuiCpz3EBZWtNIEQcuYeDW9Jc5i+WS0WjCzc0NRkUY +FRL6LT661+Ev/vg+9w46dFoRWZrw5vyat+cXjCczxlNbtOg6DkHg8enHD/n044/odtrEUUDge9v5 +hoFD5Ml357tZd2A+tzSF84Wd7ypJOXtzzXi6IElzqkoB7vc+tbOd7WxnO9vZzj7YzB3gO3A57MX0 +WpZmuyorJrM5by+uuLgesViskUqxWCWUZUWzEbPX75IVmlbjDCUqTKVrH7QA6SB1he1aK3AdzX63 +xReP+zy+1+GgZzurF4sVq9Waq8GIi4trBqOJTY4rxcFelwenx5wc7xMG71Kdb6VufgC42EK4oiiY +zeeMxhO+e/KSp89fUVaGRjOm2+7wq88eE0U2mR8R4CjJbLZgvlhyfnHNqzcXXF0PabWaNBsN7t87 +4tNPPuLBvSMO9joYY1isUl68vqBI53YMQlIVGUZXVqbpDr2v1laDVVcFppJIryJ0odOwBQfH+236 +nRglIUlSXr5+y9NnL5ktlpSVwRhBu9Wk221xuL/H/XsHdDsdwsCh2/KJwoCjfpPjgxZKgq5KboYT +njx9zsvXZ5SFpiw1YRTS7XbodTvcPznk1Dmsqcor24WuBVpvChj0OzieMRpT2Tlo7WC0wmD/LE3B +fjfi158c8Mn9Pgf9ppW3WiUMBuNtPDYcTYnikEYccHSwx+OH94mjgE4rwvMc8sJYquqBZJlWLDNN +VWZoXW62LABFWbJYLpnOFjx7/opnL14zmS3RRiCVw68//5g/+fWn7O/ZTkbXadHvROy1A5bLhFVR +sM4snbpyfYx2QMmf9ybNRkBK4ziCKHQsJX3d/VlVFUVZkuUlaa7JcnA9B28jr+RFOI5fM2TZ4GgD +mlpWJs/GbqZCa20ZyISg2Yj56LTPn39xzEEvphn5lGXBeDLlZjDm8uqaq+sBRVkSRw0azZjHH53y ++KEFzQ/3ezTikOFkwcXlFUmSsC4zkkxvu7J1Vdh41mg2dKJlWbJcrlgsljx9fsbzV28ZTWa2A9TA +H33+MX/8R59wdNC3RRKBR78dstf2mM0U83VFmpXc3Az5nU5Yr5dorek2A8IwIBSCxXLFi5ev+Oa7 +Z6zWBcskp6yMjb2Uwx//6hF/8sVjmnG4fQybDtI8yxkORwyGI5I0J80KrodTLq+GLJZrssxgKkPk +Kx6cdPiLP7nP0V6LTjOgKivGkxmD4ZjL6wFX10PSPCeOIhpRxMMH9/j40SmtVoOD/Q5xHDKeJ1xf +j1mtVqzzknWub4t9PhD8/QUvaPs8dEG36fHZR11+9XCvptV2GY1GvD57w+XVwMadkxmOYyVSe70O +nz5+gFIPUcqh025ynBUc9WP2Oy5DnZIlKWW+roH7d+9sw3TBar1mMBgzmy9JspwsK3l1MWM0XpAk +BUUpwHwAzGZbiCkrg84LlmvFKs1ZpymOCqwOfCPij7/4hFYz5ma8YjBZMp5lLNOKdaJZJCWrtCSt +NKZuMBBSIaW7Ld64jdVLPOVy76DBf/j8gPsnXfa6EVobrq4HvD47r4st5iRpius4eK7Dw49O+fTj +hxwf7RN4LtF+wPHehINOyGKVEAfqnXwE2E57LQTr9ZqbwYjpbL7diy9enTMcTZktViAEWVbQaTd4 +dP+Y/X6HbjsGs8fLtyNcWVEVayo3QFU5jquIPKvv3YxcosB2qZdFyWq1ZjKZcXMzZL6yzIM04w3H +yvdeWXVOpaZOd5XhZC/izz7f5/SwTbcdIqVguUxZLJe8Ob/i9dkF4+msbv5z2d/rce/4gE67SZoV +9YVLdJmiyDnqR/yHL054dHrAXq8BGK5vBpy9uWAwtHIpq00hmetw//4Jn338EfeOD+06H3Q5Hs45 +6IWMJ3OWWcWyyKASGKkwUtU9Jbt4/0NsB3z/GzIpJc1mk6OjI/r9vtVlNIY0TQFYLpdb+vINJfXN +zQ1Pnjzhyy+/JEkSVqsVYRjS6/V49OgRx8fHWzA3TVPm8znr9ZrhcMh6vSZNU5Ik2XZ736X3TpKE +qqoYDofMZjPWyRrXdfE8jzRNGY1GvHnzhvF4TJqmaK1xXZdms0mv1+Po6Ihut1trds632tQbSnSt +9bajeTPWo6MjkiQhyzKiKGI2mzEYDFiv1ywWC0ajEW/fvuXly5eEYcjR4dFPrulmrTbd7U+fPuWb +b76hKArSNKXT6bC/v8/jx495/PgxH3/8MZ7nMZ1OGY/HGGNYLpc2kJjNmE6nvH79mhcvXtDpdIjj ++Ae66983z/PodDqcnJzwxRdf8Kd/+qccHx9vQW3P87Z07+PxmOl0ymg04vnz5zSbTfb29njw4AFv +3rzhyy+/5NmzZ9u/H0URR0dHfPzxx9tih43Wd6PRYD6fc319ve3+Ho1GnJ+f8+LFC4Ig4ODg4L1j +dl13yxhwcHDA6ekpe3t7NJtNms0mDx8+5OjoiHa7/UHFBzvb2c529n+yNeKYIPC38hWO4zDKJ6xW +a65vhrx5e8PzVxdkpYsb93FDC3orL9pWv1daMJ6mvHw7xHcFnVZIvxPRarXodDrs7fU5PTlgOlsy +ma2Yzhf0u232ug1cR7Farbi4uubycsBgOGU2X+M3QnzXFqRFoU+n5dFrB/S7EZHvkGYZy9Was7cX +/Parb5nOFpQlSOmwt9djr99nuc4wSJrNBqtVQlkWaF1hRHXbnWwsWPuhYJyoNb7j0GW/G9Fthfie +oqoKZrMFF5c3XF4NWSU5juczWyZkWYlqK9qtJkWpaTVjfFexSi0wWxUZjhvUwPeG+mwDrN6alJZm +OgwC4sgnTqHQFV7o4vkue52I06MOj0779LuNWs+tYL5YMhpN66RCSVGCw0Y/uu7AERvE++5kRT3n +DZlgTWEpBY6yenJKArqm1TIatPlBMsF2+opaP9vS+VW6xJiCxSpjNF0yHM9pNmPaTUG7FRN/9oiH +D+5xdjHg/HLAYLxkPEuYzFOmy4zZ0upN5QXklUAbq3NokDXAXtPXG00zVJwetfji40PimsprOskY +jSe8fP2WwXDMYDi2wLfnEsURYRBwuN+nEYc0Q59mI2SvE7LX9YkCh8CVNfhvz0BRFKzXNgh9e37J +2ZtLxrM5SZKzSjIubyZM52vKUpOXhqyoeDxbkucljlK0mxFCQCsOcJVG6BxT2R9P+TQjl1bkEIc2 +2F/U+386W1g9z+kcPwI/cmiEDqeHbX798TGN2GplLuYzxuPxdr7D4Zi8LPE9lyAMCYOA/b0ezUZM +sxkQ+L5NOnYCPFfhe7I+K5YOtSgq1knCapXw9vyKs7eXjCcz1mnBKsm5GS2YzVckOZSVqgsTat2v +TZHAzna2s53tbGc7+wWmkUJYH7QX02uHBJ6iqkrrg14Nubwask4LHC9gtkjI8pKOUrRaTYrK0GwG +eI6VgUGX6CpDVg5aeUhhkFS40tBvB3xyv8v94zatOEIKwXK54u3FNa/fnPPitQWbHUehlOL+vSNc +16XVjBFC1XmW74Pcm87Id/3EdZJyMxjx+uwtX37zhN999R1VZWh1uuzvH9BstXj88D6tpkT5VrLl +cn3Nm7fnPHn+mm+fvuLt+Q2dbodet0tRGfb3+zy8L+h1moSBx/nlDYErKLOFlV1Sbi03VL67xMYW +lWq9AZcVUmgCD1qRBa73ujHN2K9jkYS3F1d8+e1ThsMJRWm5ofr9Dvv9HotlgkFQaasfHgUunWZA +rxPSa0ekWV530U549uIV//y7byiKiqI0NOKYvf09Dg/3a7Ymh8lsSZpmFpyvRC0rfEvfrLfTuFuY +WqJ1iRAlUlS4smKv7fPJgz4P73XxXRdjDJPJjJdnb3l1dsmb8xuuboa1XxgzXyaEYchev4PveTQb +bdZJxtl5zHnHo5rkLJO8XtPKUirXtQ9ZljEaTXh7cck33z7jd988YTReoIWDUh6u63N4uGfv1YiJ +Q4duO6LX8hlNFNm8pMoLSjfEqUqko7dSVT9rdUOMoyDwFGFwy0Zg9XBzstwWBRda4uAh3RDHi5Bu +gHT8beFufTmgpiE2pi5C0CijUbVGcbsZc3rY5ovHfTzX6lxPJgsur254+vw1Z28veXt+RV6UtNot +2q0OUjp02i2CwKfdjOh3W9w7POewFzGbLSjmJYssQzoBDmENGG7En+0/sjxnPJ5ycXXDt09f8tW3 +z7kZTNEGKiO2oFir2aDVjGg2Y7sXWz6DWJFmltJ/PC1I1gtMVXDQb7F+eIiSEt+3HbhvL6756tun +zBYZs2VOXpoakLdNU48enGBua7bJspzVas1wNOH1m3PO3l6QpDnrpGAyX3M9nLNaZ5TGvksascfJ +QZs/enxYd8IqZvMFV9cDnr884/XbK95e3JCkeT2XBghJp9MkCgMacUi/2+LNxQ2H/ZDJzKGaVSyK +vC5q0fZs/CtSngs0Ulh9727L4/G9Dp886BIFHq5SLFdrLq8GvHj1hpvhmOFgYossPI/Dw32ajYjj +owO67SbNRkBRNG3c2fZIVoaJySnzBF0V1FXrAFSVZp0kJGnKxdWAtxglANwAAIAASURBVOfX3Ayn +JGlOkpYMZznjWU6SQYl7W4XwAXFYqQ26MKySkul8zWA0Q2tNu2nj5EcP73F675CLqxHnVyOuBlNG +s5TJrI7VVzmrpCAroCih0ooKieFWKs2GhRWB63G81+DXnxzSaUXEUbBlf31zfsHl1ZDBcMx6neB5 +Lp7noKSk3+vQ67ZoNZs0Ww36nZi9boArKyJ/Q49evxsrzSpJSNYJ55fXnL254GY4tuxzScn1cMJ4 +umK5TilLy7hw7+SAdZpZvKsR4bsOnVaI54iadSO3gLIIiEOHdtPS0fu+i9El6yRhOlsynS+Yzhak +uUa5wZ1vxdt/btkDdAWG+ntHctCP+OyjnmVp8F2qqmQynfLm7RXPXr7h6YszbgZjXM/D83xOjg5I +spKDvR5S2i77PC8wprCyaf2ILx4fcLjXJY58siytWR4uuLgaMBiOWS7X9TrbRsFep0W/17HnrdVg +rxuz3w64bioKbVikOdoopPLsHHax/gfbDvj+N2au6xIEgdXGVreJwE3H9obGej6fMx6Pubi42HZc +a6231NO2Um25BT039N1pmrJcLre60Xc1vx3H2VJcb8DMPM+3oPHZ2Rm9Xo9er0eSJFxcXPDixQtG +oxFlWRIEAd1udwuSBn5AlmVMp1OuLq+4ublhPp9vNaHv3mM6nVptz9lsC+oPBgOKothSbAshKIqC ++XzO5eUlJycnFGXxk+u5WavVasXFxQXTqaW5EUIQBAH9fp9f//rX/Mf/+B/Z399nf38f13W3QHFR +FGRZhud5jEYjxuMx8/mcV69e0e/3fxb0BoiiiEePHvHnf/7nfPzxx3zyySf0+30cx0EIwSeffMJq +tdp22g8GA5bLJRcXF3Q6HY6Ojjg4ONhqr2/Wb9Ohv+mG11qzWCy2NO2DwYA8z/E8jzzPt3tgPp9z +cXHB8fExef7+9dt0pz969IiPPvqIBw8e2Gda781+v8/e3h7tdpswDH92DXa2s53t7P9kk0ri4KCN +3n5flbqqqY5SilKDcBDKQyofx7Uaz1I6CCm39GmrpOBmVNBpLNjvxnQaHnFkg0/HUZZySSnCMKDb +aRL4Hq7jsFiuePP2gq+/+Y6Xr89YLNe1trStHFeOW2t6KVzHutAbmYv1es1oPOHi4prpfI3GjnOR +aAbTnJtJwuvzKWEU8uLNgPmytLTLjqxpvuUdSPcD1qoGU11lk469dkSz4eMoySotmM4WXF8PGI1n +5CWEkWK5zJjNlzSbEb5nKRSbcUiv26A0KYXQVEVCVQaoqqyLCd4PxCsp2et1+eTxAxqtNqfznGWq +iaIGYdzgoN/i3kGb44PWtqhgPJ7w7MVrvnt2xuXNmFIrlOvaBIX0MKaEbaf5z5mlpwx9SbPp0mv7 +7LV8XIdbGsVaj2qTSzAYsrwiywqm85TBZMV0nmyLgKeLNU9fXSNMzuP7+3z84IBmI6x9HZ+DfgfP +cdjvZ6ySgnVWsk4L1mnJfJkxXSRMZmuGk4ThNCErcktNKR0CV+P7glbs0Gr4NCO/1tzOtxRecRRg ++l2CIEAbg6M2Ej4RWmvyPEdrO54w8Og0QsLAJmHu2ngy5bsnz3n2/BUX10Murkes0xKNS2UUy3XO +KlMYoygSqETJYlWSZgWVtmfPUln6dJoBq6SgECVVscZ3Pfptn34nIPIVxhimM5tAu7gasFpnyPrz +rWZErx3RaYU0GwFSQJHnlGWJlJIo8un37HztfSWe69JsxhijKfKi1tgUhIFLpxniuorA29DR2V+T +6ZQnT1/w9NlLLq5HXN6MWK0LrEKbwzrVrHNBoR2MDHBkiHS8bWf+zna2s53tbGc7+3ATwoJqnhLE +kUuvE9FshChHkqwKZvMlg8GI0WRGUUnCyGGxzJjOljTiEN/zcF1FKw7odWIKk1NJTVVkSGn9QseT +toC14dBt+nTbEVHgISWsk4QnL874zT99w/n1kMk8ZbnKtlrCwo3Z259zsDfHcRyajYgPLSqdTGd8 +/e1T/vl3X/Pm/Iab0RyDQ6o9Krlitsy2usEAZVXy+s05f/c//5FXZ5dcjxZMFwmFcckqj/2DhCSz +gLZyVE077uIoC+xjCnSZUVWW5vZdM9vCWFMDZBLr+/uuwJUWrNHaUBYlSZIymcy5uhpyPZphjIOR +DsvUxiJX44TXlzP6Pastm2UlnVZ8O5eyJMty5vM5V9c3vDl7S4VCG4UfZExXFTeTjOuh1X9dLBLO +LidkpaQy4jZmek933S0rkwXAHdcQ+5Jey6Pd8OquREVRFmRZwYuzC/7hn7/jzcWQxSpjnVYss4Tx +oqTCIwwu8DyXg/0eR/t9As9hr9vg9KhNms8YTROq0oKLd206m9fP9yveXAy5uBqzziqECvACn+mi +4Ga0pNNe1PTuLp6jiCOf0HeQ5JRFilM26m7dmqr6A8wYA1ojhS3YdWrd602T0TpJSZKMyggb3zpu +HePWElDi3c5yq31bs2Cx6WStMEhCXxL7gl47oBlbunmjrQTS1c2IL79+yt//5p+ZL3NbnC0U81Qx +Xkri1piocUWpYb9v5cBajYh7R33my5RlOqHIljgYhJToqtyyb21sPl/y3bOX/ONvv+LiasLF9Yzl +2jI6IBXjRcn1cEanY7uNW80GnqeIQpfId3BUhtElZSFJjGadVeS5BYp1LfirjSGv6a/XGSSFIq8k +0nhI45JXCmPklm0V4OzNOU+eveD12VsLkF6PqHDQuOSlYZVUJKVr468wYK/XptOMiEKPDaA/GE74 +5slL/v43v2O2zFmucyojmCeCcF4RRAOiqEFVGfb3uhzu92jGAUf7HabzNat0Rlms6/hU3Cl8/5fa +poAcQl/RaTh0Gh7Nhk/ou4BhnaYYXeI4imYcWea4OLYNB0rR7bYIg8BiCZUtGlFK0oh8+p2Y4XiG +NLZIqdIltrzFjn25WvHk+SuePT/j8mbC1WDGfJlSGUVlFGkhSAtJrl2Mui1E/olvmnqfy1reTJHm +Fa/Px/ytLHh8f4/HDw446LcRAqv53mkilaLdatTMfjZOX6UlyyRjOkuYzhNG04TBJGGZ5BipQDv4 +jsZ3odOQtBoezdgWXFdlQZ4XSCEJfI9+t22xgcJKoyklabWtFFlerxvG4HuKdiNAGF2v/+1jXq7X +PHn6nKfPXvLmYsDlzYjZfF3vRYd1UrBKDEXhUhpIy5LZsmCd5BRliQB83yMOfTpNn1bsUgpNVaZ4 +0qPTcNnvhjRCFylgtlxzcXnDm/NL5ss1CKeWmqvZ6b73btloqxujbdG569Br+bQb9p6uozBGM5st ++Oa7F/zPf/ydLTaYrFinBa5ncDxBUs4ZL9/QaoztdXzFzdB2hO/1WvTaDVqNGK8umsuzHLBNj912 +C9dxyIqyXmdFu9PCcdTt/jTguYpmI6DTDJmtU0yVYoxCax9pNAK5I3j7QNsB3/+GTAjxTqetUmrr +TGwSh1VVbTuoLy4uuLy8ZDKZMJ/PCcNwCxKXZbmlMR8Oh1uN7w0gDmyBV6DWJ1Tbjm7P87ag72q1 +YjAY8ObNm5qSo8F6veb6+pqXL18ymUwoyxLf99nb2+Phw4ccHBzgBz5FXjAcDjl7c7YFvtM03Wp6 +g61e3MxBSrml2F4ul1vgNk3TLXA7m824urraguQ/ZRvge71ec3FxYTvX12uiKMLzPPb39/mzP/sz +/st/+S94rofnW+3tqqoo8mLbEb+hnB8Oh8znc16/fs3e3h6ff/75zz7XOI755JNP+E//6T9t6dSb +zeZ2fMvlcju3wWAAsF3fOI45Pj7m8PCQ6+trFosFSZL8YP2m0ymLxeKd9VutVmRZtqVU38xrPp9z +dXXFdDqlKPIfHfPnn3/Of/7P/5lHjx7x+NFjut0uQortXtnstXcpuHa2s53t7N+fCSGQSiKR2+/V +qtKkWcp6nVJUGlMD38oNUK4FsKRybIew0WijWSY5eZrTargcjmb02/Y9Hwa+1d9yHKIwRHcsRXdZ +asqqZDyec/bmnC+//o7Xb29YrtYgLAWWVC6Osh2uoefgKBus3QLfCaPRhMurG6bzBFSIcELGsxzl +rQgvJoTRFZ7rkZaCrDAIFSKVREl5p8v5A7u9hbF6ao6kEXn0OzGt2AdjyPOypnweMRwvEE6AkYHV +jJst6XVbdi6eS6sZWp2ozDBLDcVGc2xLDfh9nW9rSin29/toozk6TFinBUVp6urbJu1mRLMR1fre +EtdRllbwxRn//OW3XA3WVCiUE6AcH6EcqMAI/bNLYGpSeFdB5Cu6LZfHJ00+vd+iESpbPa+rbReP +2VRXa818mTBfpLy6MCyXcwY1XaFUDrNFwvPXGYv5FKMNvU5M4Lv4vofvuez1HXqdFtpoqgr7Wxsq +rWsKwBFvLkZ8/fyG+XxGUhYgXITycH1JM3ToNF3aDY9GHJKmudV6KwuUFERhQBiE9Pu3c3WUotmI +0UZTliWV1ggBge/SbUV4vov/DhBsGI+n/O7Lr/mbv/sNN+Mlw8mCQitcv4njtxDKBekhhEQWhkJX +LNYFSZpvNepC36cZB3RbPrNFwjKrSIuEwG3Tbwf0OyFh4KK1YTZfcPb2kqvrG9brBCEdQt+j0wrp +dWI6zYhWIyLNrH55UZR2vkE9X9OrN7WlfWs2YozW5LXfzgbob4UoJW3l+p0OrclkxpdffcN//5u/ +ZzBeMZysyCuBG2zm6yOdEOl6KM9HeSGqBr5B7OjPdrazne1sZzv7YDN1Nza4DjRDj363QasRgDFM +84L5fMlgOGY8WYIMMCpksU6ZzBZ02k2UUviuSyMO2evGpDnMM0Nephbsq3yUcIgDC4x2W34NPvno +qmK5Snn+8g3/4+//mcF4iZY+Rtx2w7renOPDOadHS+I4tr7E1jZFpu8HmiaTKd9894S//5//xGJd +sExKhApYFy6FWFvgW99+tihK3rw55ze/+S1vr4bkpUOJQ1a6rAuXk6n1kYUQuI5T+98OjgJBials +Z7KuMktP/M5Kc6eIs2YzwnYM+57EcSRSWv82LwqSJGU8mXN1M+JmOMeoAKSPmmYoZ4lfxyKNKGKv +32G/36lFluxqVJUmy3Pm8yXX10POLy4w0gcZoNwUd5LgBQtevh0ShSEGQZKWZHVnOcLZgijv2zcW +mLPaw0pqGqGk13JpN1zi0EUpSV7kLFYrXp1d8E+/e8rlzQThxkgnAFEhREpWCqvX7Npxd9stfN9l +rxPz4KjLYJQgKKnKrI5nbp/XdDrn2yfP+R9/+w8s1xWLpMQIHyfwCIRksswZjBb0u0uajQa9TgPX +c4ijgChQKCqqPEWX2S2tu/p50HKjK7+RqVJKopRgg8nmecl6nZCkOVpTFwXbGELUBd7vKyiw/0mA +2HTeSgSGKHToNhX9dkQr8gk8lyRNybKCm5shX3/3lL/5u39EOhHCjXDcCGcp8QMI4xFhGNhGLs9l +v9+i3bLA92S25PzaAt8IgZQuuqaIvrvOi+WSp09f8jd/+xtWmWCVCUrjolyF49p1vh7N6fdmtJtW +8tJ1FHFg2bAcKdC6ojCaspKs04qstDropqaN19qyVq2TknUmWBcOhVZI7SC1Q14ptJF3F4o3b8/5 +u7//B7757gWDOkZy/CbOlkHOA+UTNyI6nSb7ex3arQZB4JGmGVmWczMa883TV/zdP/wOo0KkG9s4 +Q2mUzPGCEUHgoRS4rsPhfpdmHHBy2GEyX3ExWFIVCQaFlKouavmX053bkN3gCIg8Safh0Wl6tBsB +YeBt4zCtNa6jaDQi4kbMQa2JLoSNRQPfNnwVhaVhdx1FI/bpdxrEgUJSWmmGqrjTIGxYLNd89/Ql +/+1//E9uRmvG84ykAOWEKC9CuvZHuXbeclOA/BNx2O3+liAVaZ7z5mLEbHJDkRc0GyHtZkjge3ie +S7/bpt1qoLWhMgZdafvv2jBfppxfDTm/GvPdywGr1Yrlco3RLkY4KEcSB5JOw6Hd8Gg2IySGLMso +iryOvz1U16HT7WzTIwLotFsIKciznKqyOZTAd2k3AqQwhIF7O01jWK/WfPfdM/7qv/0tF9djhpMl +60zjeE2coIlQrt2LwkWUFcIUzFYlq7QgL0o8155NW6Tu0264LFLDMktxVINu0+WgG9KIXIQQLFfr +ugv/isUyBeFa4PsnChA29OeeI4hi134PN3wasYfnSMqqZDZf8N3TF/z1//ifLJMSLWzOyfUkjqcY +zaeoyzme6xCHrh0PBsdx6XUa9Lot2k3LrJhlOVmWW1Dfc+h0mrQ7LTBi25Xf7TTt90SN6dkCA9tU +0GkFXI0zdGUbDzbSD3bhP5xR8d+z7YDvf4P2fbqQDfX45qW+AUuHw+FW93vT5StqrYXxePyeystb +oHUDim6c6Q1Y6rou7XbbJlnn862+9c3NDS9evNjSi2+A2bOzM6bTKWVZEscxBwcHfPzxxxweHuL7 +PnlhKy83oHdZlpRluQViF4sFV1dXTCaTH4x1A7pv9MOrqtoWAGz+27vBwPttsx7T6ZQsy6iqCtd1 +aTQadLtd9vb22N/bR8jbL06AqqzY39/n+PiYi4sLzs/Pt+OaTqc/0Fz/MVNKEUURnXaHVqtFHMcE +QbBd906ns9U439Dbb+jZp9Mpw+GQ6+tr5vM5RVFQFMV2PywWC66vr5nNZj+476Zbf7VakdedSnc7 +/PI8/1FqGtd16XQ6nJ6ecnR0xN6+7e7e2c52trOd/dCqsqKsyi1rig24PbqdNkeHexxeLzjYW7DK +HYzrbEHZjSsrhLDVys2YvVaD06Mm/U4D15FMJvZ74BY8tJp3rWYD13W3Xb1HB/t88vghSA/kANQK +6VvgvKwq0ixnuU7JCktVLmtNrFarwaOP7vNnf/prSyOtHUqtKDWUlaXoqirNIs+oUBhcpLLd3puO +7w83g6MEkavoNC2dtO9JHCkpqwolBb1em4cPTul010gVEjfbHOx38TxbcSzAVig3Y44OeqxSzXqY +sEgSnDJDVyXKthLw3o5vJWlEIXqvT7tVkOcllTHEYUgUhfiei+sqtK6YTKZMZwu+fvKKZ68uOb+a +Mc+ETcY5EdLxkdJ2+gv9YZRnmzk4CgJX0m1aWrJ27N3RmbsFvcHSio2mDr4jGE1mKArKPEG5BiEV +RWmYr0q0rnjy6gaAw36DRuzTiHxLxe27NRju4/suUiocR+EqgSMMriPJSqvTdzVcMJ3nrNIFKoqI +AodGoPBcxxYDuBKtLQPB/n4f5Ti2U2mbBLC0783YdlLZwkyFQOA6Ej9wcR17rbua11leMFusGI6n +zJcFSabRwgPPQ4gApQKUY3XRMBWVEKySisF4QbsZ0mk3CQKPViPgoN9htszJRimrZE3gGg56Efvd +BlHooo1mNltyfnnD9WBCkpVIx8cPLCV7s+ETBrYY0qkLIOI4ZH+/j6yDbSnlOw+1GYc044gosoWo +CHCUrbKXUqDUu/ujKEtmizXD8cxWw2c28Ebb+TrKJvWUF6LcEMcNbEJB1PR2u0B4Zzvb2c52trMP +MOtnuI4kUNYHjSOHwHOshmhpC9v6vTYPPzql3VkjnJC42eJwv4vvubX2sEQoSbsVc7DXY5lCMiqo +khSpPJSTI5EEnqIRWS1kz7WFjcvVivFkynS2tp3AmUb5HsqLtiOshEdlFGVlu0O3nuAWefjx733L +7pizSlLyUqCNQgiPSvhoPNuRp28/b4whL0rWaUaWl9bHV27NsuNSGmcLvomaTlcKgcRqeW80su8W +bP5g1c3t2tuC4JzVKiXLcqrKIKUFghtxxEcPTviTP/4Vw5Eteiy1IC81RaHrWMRqjHu+/Vmuc7Lc +3tv6Wj6H+31+9dnH5FlOoUUdz0jbuYmkrAxJmlFWgkqDQW1Bbyl/GsjadEgrYQg8q0Eb+LYJRFcV +q2XCeDJjvshYpSVZpXBdH+k2wRi00WSVw2xZMBjPOV5aRjDPVTTiwNLJ+wp0ga5y9DaXW6+frkiz +jNU6JSsUGgXSR6gQ4TTQwicrBXlp7yUAV0l818aloNE6x5g73d4fQlN95xluKM/fBTuNvVS9T7cd +3ls/9ed8VcEmbBPCntE4tAUFvmdplrMsZzafM5nOmc9XrJIULwzwPA+jIoyKqEREWkimC8uOleQb +EM+n223Rasa4jqTKE5Ty0G6OqQsA7lqlDUleskoycu2j8RHKRzi2MNzgkZeQ5eW2kERJiec5uK5C +SiwgXO8nW2qzKVe5XRODRG+K1B0HUbmgXFBOHcvLurjDWprljKcLxtMFi3VOWoCnZB0jhSjHR6oA +P4xoNmLaNbuWFII8z5kvFkynC5arhFVa4vgSzwuQTgxSoaUiKQSTecZ4lrBON+vn0W7aYmDPEVRF +arvf8X4Ra8DPbzJbmOR7kmbsEYeWscBxFG5lmWlbrRbHRwe0mo1tLLbZQ67j1JICEUHgb/PlrusQ +hl6tSV+/t76nC11VmuX/n73/bpIkya480Z8So849PFhG8iJdhWrsgApW5K3sF92PsZ/giezMYGYe +0ItGk2LJg4dzYtzU3h9qbhGRmUW60RgAu36rIpk4MVVTM9N7zz3nbFIm8w3LKCfOJZnRaHwqArTw +kdI24whZqxn8ZNP/Nh+2ahKlMWySnDzLeHE6RWnJzXhOu+XTaXlNnu57Lp7n4vseqiachb6DpMR3 +JEVhyPMCR9tzNV9vEJ6L7/i0Q43vKrSSdb3EWrvt7Q3qhgsQ8s5xVxVhGNBpB7RaYU2sEygpcF2N +l9vn412p87woWUcJk+mcxSqyjR2lonIcBDZXl9q358ZYO7o4r5jOrcz7oNfGcx1agc9o2GV/2KGY +JCzXMa4qGXZ9DkddOq0AKSWbKObyesLZ5Zh1lCK0hzTY6+YH61C1NYOWBL5DJ3Tts15K8qJgtVoz +mc6YzlcslhFpKdGeRjkh6BboNkYISxLIreR4YQytQNFpeQz6XVqtAMfRNSHQKqLsDQd1w7+xzRF3 +6gRh4Dfz7LkOQora+sDBdR1bI6gMVdMsZqd8l+n/vNgB3/+BoqoqjDENq9uYbVedwvd9HMdpGL1b +MHexWDQA5pYZDlYe/WNgMtCA5Hmek2XZve/fyli3223Ozs5IkoQ0Tbm5ueHFixfs7+8TJwlRFHF9 +fc3p6SnL5ZIsy/A8j8PDQz7//PMG+J7P5ywWi8ZTfDu2LQC9Hcu94mEdxph7x1oURTNHeZ437/+x +MMYQxzGLxYLVatXInHue1/imhmGIVLeb+rvRarUYjUYMh0M8z7sHSm8/718anufT6/Vot9v3WNxg +fd1nsxk3N7azazt/W9B/Pp8TRdFHWdfb+SuK4p5M/vb9ZVliyo9vVIQQuK71l/E876PnZxe72MUu +dmEjLwriOCLPC8IwqL21Qo4OD5HKYbrMmC8jruY5y1gTlxZ8VpVlhQrAdSSPjrr88nmPh4cdDoct +fBe+f/GaFy9fs1pvSNMcpSRPHj/k2dNH7A36dLtter0uX3zxKfujIaPR9zju1xhxQVK6JKUhS1OW +q5jpPGITpbU8s0MYBBwe7PO3f/2feHhyxGwZsYkzVpuM5SpmsU6YzjdMZmsW65SqshLMSH2ni1/e +Fop+dHde1bJOkl5HMer5dEIHVT9fqspKRj95dMLf/c3/wibOkdLBC0OePzmh07bgqVQCgZUCe3C0 +z3ydMZ4nDYuhKjNM88xS269uQgiBH/gopShLQ1k3IWitcbTGVBVFLdv34tU7Xrx6x3evznnx5oqr +aQSqhdQhymuhHB+hNKLpyv15IQRIiWV++45lGHe9+4WkO/sb6+dtyPMM35GIKqfIozrh9zGVICmg +3MC3r8dcXk/pt10rsdn1GfRbDHotBv0Og77tKA98DyV9At9jf69H4HtUlcBzJMFLybfpFfPZEjlQ +tNyQ0FfWi0tYfzshBAPRw/d99kd7t/unO8VZ11HW28pxcOrGBSktiKy1Qr6XrJqqIi8MaW6o0GjX +Bd3CCwe4wcCqJWjPquuVBUoaNimcX89ph65VTQo86/F2OGC+ipgtYsp8Q+AKDvc6HOx1aPkulalY +rDacXdxwdTMnSUuU9vD9gG4rsDJxrkYI6vFKhLQS7qO9YdPBfddH3nWsGoGjdbOfs1ZCCinFB+Mt +TUVRGrJ6vMp10bqFG/bxwqEFu93ANlhoF6mcGnDf+rr9T7/V7WIXu9jFLnbxHzYcLei2XPb7Dp1Q +o6RogDylJI8fPeBv/+rP2UQZUjn4QcCzJw/sHlQplLJ79l4n5ORoyGKTM1lMKfMYqT1MkSLQ+I5L +J9B4jrL2MGnGfLHiZjxjE2eUOEitcfwu2u81wLHjdawdktpKuN4+6H9ymylEvUewQJpCo9w22m2j +3DZK+VCPVwiBQCCVQmmn3lsFVCq0DEfXMkGF/Fg5uUJgGh/vW4uejxzSnV+LomQTZcwWG9ZRSlEW +KNUi8APkUPGX/8uXHB/uM1tuWG0yVpuU5SpisYqZLjZM5xsW64Q0M8yWKbNVQlzb9rmui1KK58+e +4Ac+X37xGatN0vwsVynLdcx4tmYyW1OmJVQCUDanEbUs9w/VvO7I5woh8FxJO3TwXdv4meUFq/Wa +yWROnFqmvXYlbtDDDfoN01dqRZoL5ouEKLb1TB24hIG12XFdBVWBKfIPWPQWIFZIpVGVgxIOwmnj +BF28sGdBG+lgkFTVraqn1lsgrMKUtkkW88dIVFeYholq2IoHSCmQyjZ3Njnhlkz0R2xVHUcS+pow +UGjH5nFRlNTNyEvSvEBIF+20cPwuXjhAuy0cr41RIZtcsk4rLO4t8DyHTrtlbcMklIVlVpois4zv +9+rJ1gbU+qYrFeA4FljXdSOqdAKM0JSVxNTzrJRVMbgFCWuLM6ksQCvuy5bba7X+DsdF4aEra4Em +lYN2vJrNevuWbb6QG9uA7fqezRdaQ7TXRikrMe+HHdqtFq3Q2iwBJEnKbL5gvliR5RVKBzheBzfo +o70OUmmk0lTSZZNWrKKcLC/r82EbU9qhj5YVZZEiZIUQqlFy+BdHc22B5yq6LRffre+3QuDWOMj+ +aEi7FVrV1/oetp0kq9Kmb/NOrUlEZlVttVNb0X1cEa4CSgNFKUEFOL6LwEe7LbQXorTf5GHbXGzb +sPHjIUAqZKWpMGSlJs8Ur88XTOdLuqFm0HEZdn0GvTaDXot+v8OwzteDwCPwfVxH28YY36NCoGvL +tG9eXTOZrBBhC9/xaPka17H3MK0VQlgVYV2T2qjMHZU+Ow/Oe7mrFfKzNriOYxuCxL25qsjLkjQv +KStr/eY6Pm7Yxw2HaMe3eauQdQNPRlZIbiYrzi7GSCHodVq0Qo+DUY/JtMdinVEWEY4sGfYDjvd7 +dNs+SsJqnXB5PeP8asYmNkjtgREgRa3SUaugfSS0FoSeph1qPNee/yRJmc3m9lkcJZRGWGUQr43X +GqDdNo7XrusatdIHJUlZECqPMPTp9azdoK2J2AYzpSRKa3q9zta84R4o72j53jwLRN3Y4NYEgi1g +/0ONZLv44dgB3//BIs+tvPZdNrPjOM3PVv48TdPGu7ooinufsQU3xY/cibefF4YhSim63S7tdptu +t4vjOI1U9tYnejwe8/r1a548ecJqtWS5XDIejxmPx43ceBiGHB4e8vz5c0ajEa7rUhQFURQ13tTb +Md1lr289Ez8Wsi4yWq/IdgMQb6XKfw4guwWq787VlvHdbrfxff/j8jvS+oD3+306nU5TxCyKgjiO +ieP4TwJ8u65Dq9VqPEq2x7wF7VerFYvFgiRJPjiv20aJH2oAkFI20vUAnU7n/vypH573rd/8NpHY +xS52sYtdfDzS1CaUcRTT7XYwpsLzXAb9Lq7rMJ6uWa4i5NmC8iYhWRZQWZaElLZTPPQ0x/sd/uyz +Q46GLULfYbOJuBlP+M3vvmUynZGkOVo7rKKUspLkhUEqzUHgc3J8xONHJyA1s9WGxTpmsjCk65I8 +S1lHCYvlhtkiYrrYoJXAdTTtVovPP33G558+Zb2JmS1WTGcrbiZLbqZLzq5mOO4E5JI4E8QZdQf7 +H8Y43bI+XK3otRz2Bx7twMpzbT/FcTQPHxziuZq8KBGiTtZ6HdqtsE7A7Ot73TYnxwdcTdY4+oay +iCnLjLLMLRgt5O2z8f2iBtwCh6Zqmsqsb2BGFNl5+Ob7d/z2m5ecXky5nqxZxQav5eA5LbR7h/Et +8jvJzc+Yk6aAZhN9Y6x84t1k/C7juygMpswxZW7HWKRW1t2xRQlTCUwpyAvDer3hLF8TeoJBx2HY +9RgNO/Znr8/+Xsxo2KfXbdHrtgkDj1bg02mFVMYQeIIsjjk7v6LM18gqxHfAdwR6uxWoO5HtXkHf +jrueg7szUBQVRZFBnFGUJcuVba4IfEFpDNxZR1UFRVlRlIB0UI6HdDs4Xhc37Fs2g/aAClPkCAri +tORqvKTX9hkOuhwKQbcTcnw4ZDJbcXoxQVYZoScY9kMG3QDX1WR5zmK54epmznS+Ii1dlOPhep6V +bvdcnO0eaVuwrPdGPz3enDix+7jFcsMmTvFcpx7v3WvCAt95WYF0baHL7eD4Pdygh3QCOwd1kUUI +XSfKP5dFs4td7GIXu9jFLrbhOtY/dtS/bb6UdW3FcRxOjg9wHU1RGrsH1Zper02n3cJxnWYP2u20 +OD4acTWNcM8s8K0dn7IMEJWPo8F3JY624EiW5azXG2bzBUmaW/UkR6HcNl7QbfZ/2mtZG50tk+3e +Y/6nnvm2GVUqjcJFCRflWKle5YTWLma736o/bgukSu2A9KiUj3SCWv7YQ4j75WRRMyVFzWBmC3pX +t+zf98PioJY9FycZ81XMfBkxna9pBS6uYxtxP3n2mE+ePSKKU2bzFdP5mvFsyc1kyfnVjHcXU8TN +gtJIojhnvoxr39uoBhMUhwcjDg9GmMowmy2ZzReMp0tupguux0venI8RUjFbJGySkjgziDtyyT9W +P7Vbd8tKdbQg8DSOlihpQf0oSlgs12S5QSgP7blor4MT9G6BPceQlyXrOCPJCky9l/Y9l1bo42gJ +VYkxNRO5ute5axuOpUZqFy19hBui3bYFL7VPJRwqbpnCUlomrJICgWkY+lX1B0hU35kSY+yetShL +yppAI2uQTCvnVqX0D74yb49FKduE67lWlQogy1KWqzXrdWRzX6mRjo/jtdF+F9ezc4B0yApFkhry +omr8k1thQBh4SFlhioyyyK2/d7lV2ro7XEs8k8pFCR8jWqBt3mebUX1AU1Xidp6FQEtZ309uFVxt +M4XiQ0swax8plEZqDy1CqsprfNGl8j5owrDNsiWlAaFctOfh+B2coIvjdex7lcbzAoLAx/e3LGdB +mmWs1hvWm4i8BKmtfZLj2fdvrbOQFWmB9SXPDRW2qTcMPCuBLsGUGaIS9vV1U/u/NKoaYBS1Alno +K3zXrluwFl1387DbBuAP87Ass57qkLBcb1itY5I0Jy9KTAXqY/lTJTBIykpaVTevjVQttNuyjUDb +8yLfl9f+6ZUuhARp8ZWyKijLgmi24vJqhSMLm6t3PPYGHfb3Ouzv9RgN++xvUnrdNr1ui1YY1g0c +IWBwHYGocq6ux5h8Y32+VUXgSFx1e/1VlaUMWyXCrb75h3NWllCWBUlaAIL5Ym2VJdLaj/rOO6pq +2yRQUYl6/ao2rt/FC3qNnR8I66depKSF5Ga+4eJqTq/bwRhDKwg4HPWZzVecXy9QVYbvVAy6IaNB +p34Ol6w3VhltPF2QVSFC+SglrHncDzWC1zUnJQWeqwg8B13n9Fl9LSyWq/pZrOwYvDaO37PXlNeq +rQgrKlNSFil5noD08IOQdivE97zmebG93rVWeJ53uzbem+v780w9zylJVlKWlW1Y2vEN/6jYAd// +gWLLfp5MJg2Te8tODsOwYd9uX7st1FaVfQh0Oh1arRZ7e3scHx9zdHT0s75XKWXlvvf3cV2XsiyJ +ooiLiwscxyFJEubzOe/evePt27e8evWq8cvOMttF1Wq16PV67O3tcXBwQLvdtiD9nYLyFvQOgoB2 +u934Vz948IBWq/WzjrXT6TAajRqAvRX+9PtsQdk037/dkG29qn9sg7uVmb/7um1XsGk8Tf405/7u +Z23P7/b3rUw5gO/79+bvrmf4T0Wr1WJ/f5+DgwM+/fRT2u32n+T4d7GLXezi/80xXyz4/sUrrq5u +2NsbMhgM2N/fY39viOf7PDiyntLKuSItrojiJUbklGWK57i0PI9Bx/oydUMPRyuKomS92XB1PebV +m1Pmyw0GjeMGmFc3zCPF1TThi01GVuT0Om16nRbtVsiDw32msxW5mTJZzsgykInLYhXz8s0VgQsP +j/oc7nUZDdv4nkfge3ieS6/bRmtbJBjt9TjYH/LowT6XNwtevJvw6nRKWhRQlbfeXj/LfsgWfVwt +6bU9DoYtOi0PLS2Qp5WViRoMejiOBQmlkEip8AN7fK7roOsGwH63w4PjQ95eTPE9XXfvp/ZHKqRy +ajaKuVfWKYqCy6trLi6vmc2XrFYRmyimrIRlTmcFUZyziRKuJwvGkyWrGIxq4bekTYq8lk3slIYf +8K77sSiMIU5KZouEF2/HlHlM4FZ1IcayPKqy5Nbju2K9jlluIk4vF6zWCVKpRuZNKI1gm1xVCCko +Zck6ExTLik264WqWEZ4tCLxzWoHL4X6fo/0eDw73OHlwwMFoSKcdYKohe8MJvgeUCcLkQMm2M9wY +w3K1YrFccnk95ez8iuvxrPaT17es5qZTevtXKys5W66Zz9ccHg7o9bo8frhdOzUvSMhGFk4KC0Yr +x7vtttcu2wUnKkmUVlxPI4a9iCjOEVLQ7bR4cHTAdL7h9MLaAnVbLi3fSo4nScp6vWG+3LBcp8Rp +SaWkZV0o21wh7zRortZr5osl1+Mpp2dXXN1Ma1lMfduA+ZHxVhVMl2tm8zX7oz7dbocnj+4WXEQ9 +XnsOlXAR2rUAv3OfXUA9L9v37UDvXexiF7vYxS7+sHC1ot/xOBiGdEIXreo9qFa1JHLP1sOMsSCy +lPj+7R50q1LU63Y4OT7k9HKB5+rG69oUW/nkEnEHWNw2uhWlwSAR2kVWum7o8+1e1ZS3Erq2Q/MP +Gtu2KU5smd/ohgFubVLuStxu37QFfO1ehG2TXQ30vH8Md8St35O9/tEjA+weOy8FUVLx+mzCf/vV +t1xd73E06rO/1yXwPXzf2vN0uy0cRxO2AoZ9a3l08mCfm8mKN2dT3l5MmM2XfP39O6gKDva6HI66 +dDuh9cx1NGHo17VUn16vy8F+zOHhiOePl7w+u+H7V5e8PRtTSUNlajb0R6wib0d++7M9Pbf1QStN +b4yhQtR7YlDaRSorCS2qCinLunkY4BYUsbXJmkFa5y7vg9Ji+1+zT1YNU/eWifqeEtedY6+4lSjf +5hc/p5J5R3OAojAkiW3uLEo7Ftd1LcO4FeC5GiWsFD6VgcZH/McSxVuWo/3d2FyG92qjxbaeXAPG +yjYAWKazZ+daKkA18uLNGpfK2lpupa6rGvzH3Gs03r4erMqTFPb6od7zC+nc2jr9wPVpMbcPz93t +Z9/5nrqZVRhp87h7MvF88BmizoslDkrafEFpq4a1BdhlbSmq5C0j2a5vy9i3r3GsF3ut9mDXq0LI +rd3WLagvhGVTy5rJXpkSIcxts/afrB5OLe1cIe9Mb5blLBYL5oslZxfXnJ1fslht7PHX9zW5PR/3 +jkWQZDmz+ZLZYs359YI0N/jBLWP77iLfsu0VLkZ4VMpr1pUQdwDvn2E78ZGTV9+XNVJ54Jb1tZ4R +F4LJGuIiZrIqeHuxIvCvCDzN/l6Po/0uRwdDHj444OHJIWEQsL/XZ77Y0A40ssoQVYbAnheEnYP1 +ZsNisWQymXN2ccPZ5Q2mEo1H+e16vJ+7gmCxjpjN13ieS9hq8+TR8e21dPcZI0HWuavUHqrJXd36 +2WIHn+UVk3nCZWvNg+MUU0Gr5XOwv8c6Sji7mDDouPTaLq3AxXEUWZ6ziSKbq28SoqQERyAdvwGk +hfiRGswdH/O7l+uWeFmWhqq+lwhVX0uO19yzLWhd3daRKns/V0qjlG4UHDabyM7zdMbp+RVnF9eU +hluVtrtrpjFXt39fbRLbBLZImMwzqO83Qm6VM/4kl9b/K2IHfP8Hii3LejKZMJ/PybKs6WpqtVp0 +Op2GaXIXTN4C32EYMhqN+Pzzz/mbv/kb/uIv/uJnfa8QgjAMCYKAIi/YRJbp/fXXX6O1Jo5jhBDk +ec6bN294/fo14/GY5XJJHMd0u12CIGj8sg8PD1FSNWziLSt5e6zWU7TL4eEhf/VXf8Xf/u3fcnh4 ++LOO1XVdwiAkbIX0+wPCVviz5nX73U33XQ18y58oWG87GLee5Hc/708FfH/sM+6C61vJ8i3wvZ2/ +g4MD/vIv/5K//du/5cGDBz/ruxzHIQgCwjBkONz72YD5Lnaxi13s4odjPl/y4uUbXr56zWi0z/7+ +iDTN8D2fTqfDyfGI0bBLaSTXkwXX4xlJmVOUKRJJyw8Ydlz6bc9KNmtBmuas1xGX1xPenJ6zXmco +r4PjV8zjG95exsxXKRXWD4uq9hhuh5wc77PexIxnMZhrssKA8llsYl69uyLaLBk/GvHF8yO0gn4P +W3DyPDzHodMKMaM+xlSNX+DVzQLE77i8npIXOVT5PU9q4EfqGrcFDdcR9Lseh8OATuiipGiexUpp +HO3Q63Zv3yqoAXDZeAxWFfR6XRzXZf/tBZ6rrMR5YTuLt6yWqnI/eMYWZcnV1Q2///03vD275Opm +xmS2pDCC0kjSrCROSyu3LRwqoUFZDzSvFVo2jtuuEyP9gwWKH4uirIjSHLmMMUXEeHyDrFKKLKLI +Y6rSMhHAbMnhZLkhy0tWUclqU9ZFSSvxte1E3ybkQjoYk7HJMjZJztikVCaHMqMyOUoYHj0Y8vRk +yBefPak94vfotFuEgc9o0CZ0BcIkUOWIqrSyltjiyXK14vziiq+/fcWvf/cd3714g1I24d12X29B +Y+pu863kV1ZUFGXFZ+kjnj85uTcvW9mvbVFHVk6dSHtIx0NpB6k8W0BEQCWJs4ybacqwv2ET24bR +bqeN42jmixVv3nS47nh0Wy5h4KKkZJ1EzBcrlsuI1SYhSQ3aF2hl5UWVlLYAid2PLVcrLi6v+Oa7 +1/z6t9/xzfev7Xi1W6sLfGS8dfEqLyzr5NnTE54+ftCsaTteWbN37HiFvE28pfbteLVbe5jtwO5d +7GIXu9jFLv7YsHUoRb/rczSyzZfy3h5U4Tjv70GF9bV+fw/a7eC6DqO3N/iOpDLJbfNlmUNVNwzW +sbVyyQtbbJfKRVEr22jXsnClqvd1soY4//DYsiC3wIRQumnUEx9R8LOgwC2QemtlpG8Bvo9OZjOp +jSrOxw+IBvSpKmmB77Tk7dmEaL1gMpnx5ScnVlGo38Xzrb+tlVcO2DMVxlSkac4mTpkt1vx///5r +Lq+nzGYLvnlZMVts+PzZIVVlwSQlJYHv0m6FBL5Pv2+VuIrCEMUxmzjl1797xXq15vT0HGMqjKis +BPiP+RVvN+RUCFGDc++tL1MbVYu6FiqVY9n0W8a3sm+ygOtdYFbWYJR9HeYH/LdrNnXT2NAA31up +dlW/5iO0wRpU3uZFTePyz1tYgM2hkrQgTnLywvqIe46DbEvarQDX0UhRIWuA9L6s9MeSxK2XbT0/ +db4oqGrcp2qun9KUlKZufBWq9lp2bD5W/wglqYS8B3zba7tm89ffYxu3b+vDH3DfG7BcI4WGmhEt +VS39vG0iER+OZ/sdd7pLmmvg3nXBVgW1zgWqOteV77PDb18razBeaYei2gL+FqDdXvvbplw7f7f1 +a1uPp1aFsKC3VC6iHpu97ksqUXzYOFA3Zlg/95JKmttGhT9F3Lu2bBPPFqzM8ozpbM75xQX/96+/ +5te/+Zqr8QylrR3UXQuo2zysZoqbirysyMqKzSYhzSo7Zqk+UNOw6hcOEhelfCoV1GvLa873Fsj9 +w6OWPFf69nOEpipTkjInzjKmq4TKrMDkVGVGVWY8OOjy+GSPT589QGvF8dGIVujheQ7L5YaW7yBM +ijA18H3HFT7axFxfj3nx+h3/9Jtv+d3vX1BW8tY6627uuuV017lrURryAkajAQ9PDq26xx28v2nA +kAIpHNDbBnW/zo1tQ4KpPe7TImO6iGl5glWcYqqKMPQ5PBhSFAWv356z13PptVzagYtWis0mYrXa +sFhuWK4ToqTAUxJX2zqAKbKfVOmw90qaRgqBVU4o6yY026Tk1NeAe6fJ3rsHfG/vYUpb0NsqO9hr +LIpjrm/GvHz9lv/7n7/mN7/7lrwEpe362eb5zX1w+4wQgsIICqMo0eSVR4XfNDDtmtz/sNgB3/8B +Ygtsrtdrrq+vef36NZeXl433davV4vDwsGFSgwUwPc+7J/e97V7RWjMajfj0008BfvRmsH1YbUHg +sizpJJ3G63srib2VLD87O+PXv/41q9WK6XRKVVWEYcjBwQFHR0cMBoNGVhtAKiu1vfUo395ct8B9 +u93m0aNHPH78+Gcd5y1TW9Nut36WBLfW+t5cbec7SRKSJPlAKv5u5HlOHMf3XrdtRvhTSYBnWcZ6 +tWa9Xt+TYt+eY9/3CYIArXUzF1sWeKvV4uTkhE8++eQPnr9WK9xJmO9iF7vYxZ8gSmNIs5zVJiUt +JtzMIgqjaHe6dLsdHEfTCn2O9vt88uSANCu5mKRczjJUpXBVieeAo0FJhcBuyrM8J80K0jQnKwye +pzB45KWmKiTrxLBYZ8yWCcN+jqmszJL1OQ5wtKwBYU1VFpR5yXKdYkyBqSqSNONqPKfbDui2g5ol +YQtG/V6X4aBLGLh4nqaqYH/QZtj1KE1GYgxFkdWSeyWiklQ/Im8nhO3B91xFv+0xGrRohQ5CwGy2 +4Or6hulsTs1ZrqXBbiUoATrtkOHAelQjBK3Qp9Nu0esEdFoewpENAG4LHB8yJipTEcUJk9mS6/GC +y+s549naypAJl8JI8tL+aMeziYsT1BJ3LaTjoxy3Bpv1R4p9P10EMLVEWJwbzLokigsoE/Ispsxj +K79nisYzsarfUxrISokREu1qXM8nCDy0tvKCCEGWKdK0xBhFJRxMzbIxRmBMRZkbRGWYLVJcd8Xe +3oootrYtSkmUkriuRkmQGExZkGUZeV7cUc5RuI5DaSqWq5jr8RLthijH3JHlvvWqqjlINcvFHmeS +15Lm4v3iXF20pWZC17/bIs7dbvsKTEVaCCgKFuuMVZQRJylgGzlGe32ePj4mLzJOjkYEvkea5Yyn +M96eXjBbrilKAdJFKFs0q5Dkpakl8apmvI7jYCrbnX0zXiLdAO0Et+OVoikC3Urp1+NFkuaVlSf8 +8Mqo52rL0lF3xnuXubNLfnexi13sYhe7+GNCCFDCylP3OwH7gzatwALf9/agW7/QumZyGxXtVsje +oEev1wYBYeDTaQV02z7twAUtMCanyFOyLCXJcoq8gKpCCoGjlZWd/iHFwT+hr6fdA9/xwBVbt+33 +v/fWOmX7WtFQ5D72+nsHfPvzo4fdoN+AwlSKdVxY2x5TkWU54+mcTjuk220ReA6eq/Fch0G/w7Df +JfDtv7lasjdo0e/6FEXOah2RJDllWbDexAz713TbPp2W9cV1HUm7FTLot+l12igVEgYBh6Meo35I +v+MSpRWbNMUUiQW/f2ww4raJIc0LirKgqqyCppVs93EcfQdkvc+Ml0LgOBLf07hODdYZQ1HkZFl6 +a2Eofngmmz1hdcvWb5iYP3im3hvTHwRY2k+uhCTLDWuTsY4y0rSwFj5C4Cibew77IQd7HXLpUoiy +9oOWFgi7HcHtMVWWga2V9U0PXEngQlnmZGlGUcupO1oR+D5+4KK0bRAQbEHrWya7lPZzPFejahJW +WZakaUaWZh9YDvHeMTXjFbfXRXP9NL7lW9b9/dHw3p9/1hTfVU248/tH1+D2mOrcQKKaZoe7TH+7 +PkuyvKQ0VsFAa23nz7Oez9z5nu08isrUMvMOnuOgdU1iK0uyLCPN8kbe/l8lxK0yRpYXFGWJMRVS +yLox3iXNSqbzNVfjBY5bor2qkWkXW2nqu2dUbLUDJGWloG6WkEJ98N3NeZXqVv2ibnIQf1QeZu+L +StofgQCjqAzkuSKtFIVRGFFRCW7z9RLKoqTMBbN1gTte02kvWG2SxmbMlTZX10qgKKlMQZ5npHmt +xMAtgU8IySbOuZmtrCe3DmzdZrtm7lw/t2O1zSPtzFhJ8w+W41Zd5E7uum3aqIFewbYBqKIwOZu4 +ZL7OWK1TNpuEVujiuS79bofHJwfMZ094/PCAbiegNIb5cs3Z+SWT6ZIkq25zdWUbxYwsP2IhcH/+ +jako8vpaKO3acLQiCHxaYWDt2rak96YBh9sGe+40C9XWdHlRkhdFfa+25AytbS1gE2fcTJdkuUB7 +Bu2YRplje73fu5xlDXIrQVUrs9i1XK+7f72r7f9xsQO+/wPEFoRdrVacn5/z4sULzs/P2Ww2KKXo +dDo8fPiQk5MTut0uQoiGBe77PkopjDFkWcZmsyFJErTW9Pv9H2U0N6zi0lCaWynyVqtFEATs7VlG +8Gw2I8sykiTh8vKSqqrIsoz5fI6UsgGvHz58SPduhyy3Ht1hGNZ+HLIZ73q9pixLWq0Wg8HgRzt2 +tuznu8xxIcStpOaPxNbL/C54vPX9Xq/XpGn68e80FUmSsFwuWa1WjZe51hrf9xsw/18acRwzm89Y +LpeNZ/i2WSAIAlqtFu12G8/z7s3farWiKArCMPzZ81eZqjnXP8cffRe72MUudvHTIWrZsLIS3ExW +rKIJlXA42B9ZCelagnxv2OHLTx7iey6/+t05k9klwgiUyNHS+sYhts/mElPeed4py3bVrmUda7cN +KiTNBKt1RpLdKsC4joOjFUIYTJFjjC0smKoiyQpKU5EkM66up/iuwHclvitphx6dls+w3+bzT5/w +xWdPrK+h1gS+S6/jsz8ISHNDuTakeVIzVQyV3MrSffw5JIUdX+BKeh2f0aCF59qE7PL6hv/fP/4T +3373AoSiqiUXVb3538bDB4d8+cVzPn16YtnpvksrCOh3Wwz7bZJSkZiMMregtNl6D95l3RhjmSNR +wjpKWSeGdQrK9dBOCI6LdjVa1BJ6dSe9le/y7iTYNaui+egfKFTUjIoP7EyMlXwsy4rYGExZUeaS +stA1KUM0wLd9Ty0ViQKtcV2HMGzRDgMcx3ZdV1XFykCWmrrL2c6D0lZm0ZQ5pZOByckQzFc582VK +nN2XdtzKzElhpR/TNCFOUuu7LgW+79Htdmi12gjlWjC+1GjHQ+JaFopQNlkUt93n9pi2MmsuQn6Y +pjS1heoWPGbr49YUYmW9TgxFIciLitUmZ7lOWK0jfM8yhvYGPT775DG9bovHj44JAo/V2toHvHx9 +ynyxxqBRjkQ5Pko5mEqS5iVZnpOXBiEEvufR7XRptToI5ZIaiWM0FR4KD4FCoKCRTLRwN5VoxiuU +d28t353tpuC8Lbbd02Xbgd672MUudrGLXfyxIQAlBUpC4Dn0uyGjQRvXdRDYPeg//urXfPPdSwt8 +y20DWu3NW+8jHz445MvPn/Pp80f1PsOjFXr0OiHDXkiUK6I8Jy8ykiRt9k3Wx1nheU7drFjvBe6A +XtXWmqeWzP7TjVy896cfepW4t4P96V3HFgz5+ccqhEAoiVCKtCjJ84IkXXAzmfP7797ge5rAlbRC +j07o0e+FfP7pU7787BmDfhftOPi+R78bsD8IieOE+SpluohYbSLend/gexrfFQSuph26dFoeD4/3 ++fyTR3jPHLS2gHM79Bj2W+wPQ66nEesoo8gSKmNJKOKjbGv7iwWuSuIkJcstOOc6FkzpdNp4roMQ +23Na1kzyCjBIYX3mW4Fj2dE18ShNMzZRTJbldb7wQ2fszjkVt1C3QPz4Sb6bq/wx66tmJedlRZEW +LDcZcVZQ5EVNqpG0Ao/DUY+nD/cYrwSTdUGWxw1z+q4scbWVEDYlmBKlFL3AY6/vEzgVRZERx2lD +CnJdl07beh072gK3pp7fytz+aKnsGvAUjraWQ3leEMcJSZJSFOV9T/eaffuxqfuYV3l1718+1ryy +/ak++GdjqvdeVzW+8VsGOsIgKsEPM6nvA/LiHvho/16aijQvSbOibhyocF2HVjuk1bbzV1VFM2fG +lM31b+9TisDXto4AZHlBFKfEcWLn705O9ocqnv3w+qqnpYKiKEnilCwrKKsKpRRhGDAY9PH8gKKS +pDkYJakqt8473Ttg9q3CwJaprqRClhKla6D8vTr4dh3cNhfczb/+yDyssutcS/BdgSNtswqVYBNB +kVcUcCulXyt/mLJAFT6lk2FExSoqmS4TNnF+b10IQMoKKaEyhrTGa/I8t97srkun06HT6aC0R1ZK +ytpeQ+M3jeV2HZp6ddu6htgq4yrPMqI/Nv5tTi4E1Z35EnfXp9BICUUhyfOKZVSy3KSs1hFaCXzP +pdMJefb4AVop9kcDup0WeZ4zHs949fqUm+mcouS2DqMdTAmilD+pi1Ia20Sx9SoHcF3Hkia6bXzX +QWwVGkyJMQWyvi4QdVONsY0FlSkockOaQhxn5Lm1pHNdh06nRbfTQWqPrBCkpaQyDgYPhWNZ8R+s +TXvuda1WYYRHJb33pM53NYCfGzvg+99Z3JWWyfOcsixZr9fM53MuLi54+/Ytp6enTKdT8jzHdV0G +gwFPnz7l0aNHdDodW4TzrXxqu91uwNc8z4miiM1mQxzH5HlOEAT4vt8AvkII20FkStI0bVjPWzDZ +87x7wPdgMGA8HlOWZcPyjqKo8SMXQtDr9Xj8+DGPHj36APjW2gL0/X6fIAjubO5S1us1URSRJEnz +3Z7nNRuRLdBdlmVznHmeNwC9Ugrf9390vq0vk0+3270nFb9lsK9WK1arFev1umFZCyEwpSHLM1ar +FTc3N8xmM5IkaZoOwjCk3W7fY7f/2Dkvy5K8yO9JpJvSUJQFy+WSy8tLxuNxIyvvui5hGNLv9xkM +Bs38KaUoiuKD+TOlwQ8sGH9XtmQrkR7HMWmakmVZA4L/nPnbxS52sYtd/HRs5bEqNIv1movrOUHY +4c3ZNYcHI46rijDw6XVC3MeSwNfcTNa8Ob2ySQsFlclsce1OvUNIgeNoXNcjr0A7rvUX0gGOF6K0 +hxGKwgirigeN5GBV+x+ZskAgcZTE83TtgyVZRRtuohVFFiMpUKKkFbj0OgGH+z081+X4cA/f83Ac +jedq2qFDr+0yXSYsowJT2M9vZKB+hKGgFWgBoa/ptjx6bb9+xudc30z4/dff8Q+/+meEdKzUonKs +b5u63cou1zGdTpuD0QAhbJEpDD2Ggw6jYZfpqiLe5FRliind2sPtPhOlAvLSkGQ5cVqSZBVpqXEr +D6nalt2t3Vqqq5YDuyNxZ70P6w7jqmq+o7rTEfxh3JG3qn9KA6IEYwRVKTCloiwUpnSojLwnrWXn +VTbSYkI5aMc2yLVbrmXCaIGUoGVJVeSkEts5jmi+VioHrV0rX64KSpPXa+d2b2r3LLcF2CzL2UQp +8R3lG8/z6NbJdBC2cNwA4fhUok7cxHaeLIiulEAridaywXQ930cr9R4D5K4M4C0QfL/TfstKsuB3 +bgRFXrFOShbrhNl8w6BHDc63eSIE+3t9Ou0WvudyM55yfTPj9bsL5suISmiUo+oGB5fSCOI0J0oK +isLU43Xpdtt0u3a8rheACqiERyk8O1ahELWvoVYCrSVaq6agdncv/v610RQu7zJKdrGLXexiF7vY +xb88BGglGsCx17H78bt70N998z3/8KvfWGaqdC0rsJaP3u6HFquYdrvNwf4QIYS1jwt8hv0Wo2GX +m3luwcAsIU0Sok1CmlqwYush3m4FeK5jpb1zA6awnuCVgapE1kCJkuKe4pGNnwdY3ttRfUyN+Sfe +u2UV/6z33ZEo/qnYNua6vo8pBKas2MQbptMlWbxByRIljM0TOgGjoa3dHR+OCMMA7Wg8z2HQDTg+ +6FkwLiuZrRIm05jLIqcyBYoSJSt6bY9eJ2C9juh0Qh4cjQhDge+6BL5Lt+XQb3sslxGUWc1At+DI +xxiO2/+20uvrdUqa5pTGoLVDGAb0um3C0MdzFEqWiKqgKrNmsrRStEKPQS+kFXpoZWWFozhhsVzX +62XL4v7hk2sZ5e+f4J84YX9sP8W2tigUeQllWrKJctabhOU6JgxcfOkRBq5VNnt6hDzfkOQRJskR +VUZVasu+3crwY5uhpTJIZegFDodDj5ODgChKiKKYTRSTZneA7067nl8Px1UoUUFVYErb0EuV4zke +3dCh23JxXcsM39ZN15uIvChrEP+nmLx3ZL6bqfupC2Lrn/7h51Tm1sKSWonbArICgaGqCoyRWJK6 +aPKJ+6dB3Dndd9QcuP/3oqiIk5wozsnzO1aZnTbdToswcHG0RIqKyuRUZV5Ly1d42ubo3ZaP71og +Oc8L1puITT1/Nt+5k6v8yXIWy1bPsoJNnJFkOaawZK0wCDClodVu4/sh2vURyqOsNBUuYHNPKW3D +sVYCR0uUvGXmVwiKsmy80D/89tscbAvmip97H/zIWtgqGmgFviMJPYWrFK4jmC4qMAVUClPVrPTq +tgnCaAdd5kiVYSjIS8sGh/uWqJUxgCEvcqI4ZROlZHlR23o4tNstur0eYauN64dkpbbzJr36POpm +rUkJjpY1cYIacwpr1vjHL4/3GzHuzORt87rUFJWkKCBOS1brlOl8jesq28QfBDw4PqDbDvEDq+S3 +WsdM5wvenF4wmS3JjUC5QU1AcK0dwvYe+YMnSFCUhjjN2SQZWWbrNY5j56XX69BuhfieQ2FF0DFF +hlEZRjlopdBKIBVUUlApiZaGLEtYR5o0y6hMheNavKvX79FutXC9kAKopIfBBeFS4Vr5fuyzXWur +ACNlTfKQmgKH3Ojaf35HUPxDYwd8/zuLLdiZpinT6ZTpdMrp6SmvX7/m1atX/PM//zPL5bIBgsMw +5Pj4mM8//5xnz57R7/cbsPnhw4e8evWKMAyRUjaM5Ovra37zm98wGAx4/Pgxz549o9vt4mgHIQWr +9aphl799+5bz83Pa7TadToejoyOePn3K/v4+w+GQR48esVwuKYqi8R3fRp7nKKXodrs8efKEJ0+e +0Ov17o038H329/d59OgR/X4frbUFgWvG9atXr/iv//W/stlsePr0KY8fP0YphdaaLMtYLpcsFgte +v37NmzdvyPOcdrtNt9vl6dOnPwk8CyEaD/Lr62uCIACsvPhqteL6+ppXr17x29/+loODA/b393Ec +p2F6v3nzhm+++YY3b96wXq8bL/WDgwMODw8b6fkfi7IsiaKI2WxGq9ViOBw2TQfr9ZqXL1/y61// +mm+++Yb5fA7QyNs/fPiQx48f8+TJEwaDAY7jNIx7IQSvX7/mv//3/06apjx9+pSnT5/em78tsP/q +1SvevHlDkiTNuX769CnPnj37t74kdrGLXeziP35s/dmUtuxgt8UqLvnu1RVSufzyi6d0O20C38Vx +NP1emycneyzXEfNlTFoUzGZzoni/lnxzCPyAXrfD/mjA8eGI8SIjryRlkaHdDFHl+K5kr9/ieL9H +p+OjpCTLc+bzJdPpnChKqKgIfJeDUYeDw/26E95lsVxyfa2ZzqDIUrI8I0pKDAlKaaaLNcvVhk6r +ZaWgsN3qTYd47RW29TGs3uuDvw2bTAWeouNr+m2HwLO2I1Ecs1pHXN9MuZksmM43KDdEuQ7a0VZS +vLp9zi8SyWSRcjVeoqSm07H+fft7Qx4c7ZOZJdPVksKAVF4tGf6hbKFNZG2zgnJ8tOPgeG0cv2Pl +q7VXFz9r37A7P02CV92yOSpTYMqcqmkCeM/6vAbIjTGY0mBQqMp2CgshEdq13d7KaZghH/jDNWxg +WxBTSuF5Lq3A4WAYcjTq0O94nF9OObucMl1sWKxtEpxXUJSmKWw4WjLqtTkYeDw5GdHrtuzeIs/J +spwoTojjlCTNWW5i1GzFbB4RxZntInccCAMOD/b47Pkj1puU2bpgsTFkBU2XsucqK23fCRgN2wy7 +LeKaBXV00Kfd8po18oddb/UvQlqpfeWRl4LpfMPb8zGGirAd4juaIAzQWuF6HlJI4iRjPJ1zdnHD +MpZUaLTjo7Rn2dy5Yb5MmC1ioiRvkuQwgIPRkE+fPWS5jpmvSuabkjSnSbw9V+E5il7bZ7TXYdRv +EyVWfv34oE+ntWs23MUudrGLXezif2ZIIQh9h17bod/x8F1t96BRzGq94Xo8YzxbMV0kOJ5CuxLl +OIjKNl9umamLWDCeJ1zeLJBK0+m08X2X0bDP8eEeSTFjukxJk4TlOmK23LCJU4rS0Alder0O+2lO +rxsSepI0L8AkFNkaJcFRAt8xhIHTAOTy32MjXLUF96qft3urgRrPlYz6AXvDHq4GR1WsVkuurx0m +E0GRZ+R5SpJXVKuEqhJMZ2vmixX9nvVUD3zN3rDLp08f4AcBe3srriZrxpM5N5M5URRRZBVZkbOO +c0xVMZlvmC8jNlGMUhrfM7cAkykwprR7eFOAKX98SyoEeWlYRRnj+YbVJiUvDFopOu0QYwyHox7H ++11MtSI1BVmyRDsWUOr4bY73O3zyZMT+XhfX0URxxHS25OJyzGoTU1XiVrr538v5b1jFGqRDnFVc +XC/49uU7jvaHHB8OCcOAxycHOFrhBzdoZ8L1NGYdF2yyJVUt36u0xNUS31UMugGDXsDBoMXxQYfD +UYdvXpxyfX2DFAWbKAEgCHyqqsvB/h4PDvd5eLRPWvlkJqXM1ghX4QqHfrvLyWGbk4MuvTrH2Gwi +Lq/GXI+nJGlufZvrpmbLRP9TzrG40zhyy8gvypI4yWiFJaYyKCnwPYdO6JKWQGzVBrQO8ByJq4xV +gfv5X9uoYyVZyWKVM1tZX+SqAt9z6XY7HIwGHO8PeHDYJykdsiqlTJco38fRHv12m0dHPR49GNLr +hIBgHcVc30y5Hs+Jkxyh3FpRTNbr4k80b8Ky1TdxzngWsVxnZEWBEALPcxFC8OjkkD/74jnaDVhE +JfONaZp0lLIWAp7rMuq3GA3bBJ6u87CEq6uKIo1qksGfSlXjB+KOb72WVr5/r+dyuNfmaK/F9WTB +2eWM8WTJYpPW95GK3FRgDFKUSF0x6Ibs912ePdxjr99BCkFeFOR5wWoTE8UpSVpgqgTtbJjM16yj +tCG5Bb7PXr/H86fHjGfPGc9S5uuSKKVmaYPjKHxX0woc9vc6HAw7ZHlBnKR0WgH9blBLov8xp1XU +zHSFVC6mqpivE95djJFSEPgege/ie54lSWhrUVAUObP5mrOLCbNFTlEq9NY/XGqEyH/63iisMsdy +nRLMIzZJRmkqXEfTboWMhn1Ojkc8f3rMbG2Ii5IiXSGFwQiD1/bpd8O6CSTEdwWb1YLlfML1dcJy +/YCiKPF9q+QyHPR5/vSE/2U842oSMd8YovRWjd11FJ7r0Apc9odtRoM2poIkK4mSgtmqYLYxHygV +7uLnxQ74/ncWpbEeGVEUcXFxwZs3b/jd737HP//zP/Ptt98yn89ZLBYoZf0Eu90uJycnfPnllzx/ +/ryRNu/3+lSm4vDwkDAMEUKQZRlZljEej/ntb38LwN/8zd/Q7/cbJrCsZOMl/vXXX/MP//APfP31 +1xwdHXF0dMQXX3xBr9fj8PCwYZrPZjPm8zlVVTWsa7ByoY7j0O/3efr06UeBbz/w2d8/oCzLBri1 +iUZEnue8ffuW//E//gd5niOl5ODgAM/zUFKRpimz2YyzszP+8R//kX/4h3+gKAoODw85OTmxHaDH +xz8630IIOp0Ovu9zeXlJGIYARFFEHMfc3Nzw+vVrfve731EUBe1220qFrlZMJhPevn3Lt99+y+np +Kev1GrgFpf9Q4Hs6nTIYDMiyDCllw6B/+fIlv/nNb3j58iWLxQIhBGEYcnh42IDZT58+ZTgcNvOX +JAlZlvHu3btm/oQQHB0dNaz5LMuYzWZcXl7yq1/9in/4h38gjmOOjo44Pj5GKcXR0dG/9SWxi13s +Yhf/4cN2GGukdqwHtFexig3fv71mE+d0WiGfPDm20oiuS7+rePZwhBQlL9+Oefnuhvk8Jo4ijKnq +jb+i221zMNrj5MEBRi6YrUqiNAOTISgIPMleP+T4oEevHSClIE1TFssV4+mMKLaKKqHvcjjq8vzJ +fl1kaHFxOUaJnDxLWFUVSVoQFwVZnkAF0/mG5Spi0E9otwOkVLeA99aD2giqqrzHZL6PflspRCEg +9DR7fc2g4+K5GqiIopjJdM7NZMZ4umS6iHBDB9cInEqjcNGV35TWVrFgvEi5GS/ptFocmoogcNnf +63NyvM9kkYGZUOYlyvUxZWYLlu+dr62Hl1QO0hFor0J7bRy/i3ICex6lrhN6dQs4c1ekagtm2/kw +ZY4xBVX5XtJSGSv5bUwNkJcYtO3SrmhYB1IJKipUw5z5WNKzlb+rUNIykVuhw9GozS8/PeDpSZ/v +X7v0W/DmvOLsMiNPUkxRNbLdSltwdn+vw6ePBjx9dEC/Y/dGWZY3LIsoyYnTArlOqNgwXW6IktR2 +N2uN4zgcHoz4xaePqYDv30zJT2eUm9wm00Lguw6dlsvJYZ9fPD/i2aMR15Mlk+mSfq9FpxUABviY +BPhPhWgY8EY5FIVgOo94ezGm1Qo4PijQgVcXQrymez1JMsbTJeeXE0rVxagu0gmshJrySIuIYpky +WyXEaYa5M96D/aEdbwXfv52Sv5tSmsyy6gV4rkM3dHlwOOAXnxzx6eMDbqYLbiYrurUP/S52sYtd +7GIXu/ifF1IKQl+z1/MZdDwCr96DxjHj6YybyZzxdM1iFeOWHi4ajYusPGTl2IbPytg96DziejKn +3Q45LA2B57I/6vHgaI+beYKopqRZyXrjMF9uiJMMU5a4jkO/28EYGHRD2oFiHWcUZUJhQHsKrTSe +Ay1f02kHeJ5GyK0G8L8+VvOz4qfsvD/y+u2e1nM1+8OQZ4+GdNsu3ZbLzXjGd6qiyDNW64isMMRp +RpJa5uJkvmax3BBFMe1WiFKK0bCHoxV7wx4PjzdcT5Z8++IMU2aYMmeT52SZtYtKkrwBvlebBN/3 +MFXYMCbv5jS39kg/EHXjaVlUrKMUNbPAd1GUOI5G1z7fR/s9Hhx0SNKc8SJnE63QwkErl3bQ4cF+ +h8+fP6DbaeG6mtm8YLZYcXE1YbVOKCt5Cyz+u4gtA31rM+SQZIarmznfvbRy0oN+h16nxZNHRxwe +DNGOh8CgpeH82tpLIrVVhMIy97u+y6NDl08eDXh0NOBg1GM0aHN1dc1qvSLLYqI4psKqOLmOw+H+ +HifH+zw62edmnjNZZpTZCllqXOHTb0seHrZ5eNhpmk1Xm4jL6xvGkxlJmiHVfUWHPxrU+5F1ctcb +vKogK0riNCUvitqWzIKNnZbPMk6pypiqkChfETiuBb75Q695e56yImWxSpktU+JaccLzPJvHjAYc +Hw15eDSw87fKyLMC6RhcUTHoaB4f93lyPKTXDRDCNg7YBvU5SZrfNg1s8+M/gRTzVj7dVLCJM8Qs +YrFOSLMCKQSea60lHj88ZrWKcN2A799N2SRTK9ItLHs+8Kxy3dNHI37x7JBu2+dmsuRmtqDIEuaz +Wc2S/te/mW7l/LUUBJ5g1PP4xZMhv/zskNOLMXttwWu/5Owqp0gyqsJQ1NZnulZKG/ZaPH004NMn +B4yGHYRUFHlGFCd1rp4RpwVZAZWImMw2bKKUoiytT3uoGY36fPL0hCIv+e7NhBfvpqRFRFUJSgSe +69AOHQ72Ovzi+TFffnrEch1zM1kihaDfaaHUH3OORfObkMpKlFOyXCW8PZ/QCjwORj32BtYa13Xd +JlfPcntPPL+ask4URtv6jFKurUfVHuI/9tUCQZqXJGWK68dEcUZZloS+j1aaveGAxyeHfPbsAW8u +llyMI5LNEiFKhCjxleSw1+LBQUC/6zPohnz/MmNyHXO9TFiuNhSmREsf7QfsDQyfPH1ImuZ8/+aG +799NyCYbyxYX1lu8FXgc7LX57NkRX3xySJrkjOdrriYbymrDPNpg6kaxH3EP3MVHYgd8/xtGFEWc +np7yq1/9ipubG168eIHruuR53vhlX19f8+7dO16/fs3NzQ1FUaC1ptvt8vDhQz755BM+//xzRqMR +YWg3e0II/MCnV1mAessIXq1WzOe20/H8/JyqstIu4/GYvb09HMdBKdUwzd++fct3333H+fk5AGEY +ktRSlkII9vb2ePr0KdfX17x+/Rqgkc92Xbfxnh4Oh+zv79Pr9j6QztZa025blvODBw948uQJeZ6z +Xq9ZrVbMZjNevnzZSKm/e/cOx3FwXZc4jplOp0wmE7799ltevHjRgAH9fr+RaP+pCIKgAfMfPnzI +kydPWK/XjXf3ixcvcByHt2/f8vXXX+M4DpvNhsViwT/90z9xeXlJFEUAdDod9vf3+fTTT3n27BmD +weAnvz+OY16/fs3f//3f8/r1a37729/iOE4jtf6rX/2K09NTVqsVZVnSbrfZ39/n888/54svvuDx +48cMh0OOj4958uRJ4+89n8+Zz+e8evUKYwzr9Zrz8/Nm/pIkYTqdMpvN+Oabb3jx4gVAw9JP05Sy +LH/y+Hexi13sYhc/FaIBU5X2UBpMVRGlguky4835lN9++4anDw84Otij3QoYDtqAIUkLZos1s8WG +NEu5mcyhMgSBR+DbJO8v/vzPODxaMFmkbBJDEHYIWl2ePdrn8YM9+r02jlbkecF0tuTd2SVv3p2z +WEVUQqId1yY2gcv+sMPJ4YBuy8V3FQfDDqt1wnoT1T5fhlbg8vzJAwb9Dr7rYExFkiYsVhsmswXL +VUReKIRwgTs+RB/doFdIIWgFiv2+z6DrEbgaYyqWqzWXV9eMpwvizFAJB2SAdNpIt4NyQ6Tj159S +UYqAdVwxWWYcJjlFWVnG92jIchXx6nSKoKQqS+ttXuaW8f1B5aDuxJfKSs2rrYe6U4Pe9s8Ns5g7 +Umdb4FlUKA1KS4J2G9/xeHLc4/iwT6cd4nlWZScIAg72hzx9ckyrV9Ab5CwiwSZTxHmOFE5deLmb +xP1wQr6V3JSyQgqJEuBqSStwGHYDTg56SEpaocuo32Y835Dmhsz2LKK1lbx/dNjj8VGPo4Me7VZA +URSMJzNOz6948+6STZQhapmurFQsNwUX13Nevj2n12kz6LXptkMePjhASkm702c4nLOJC0okIGmH +Du3Q42i/y5OTEccHXSpjmwEdZaXZPxxpXWDbSrr9gAShEFZjwDYwaPLSsFinnF8bjg4S8tzcs36x +fpsJ0/mS5TpmkxToAJTrWm9y7SOVQ1loyhKW65Tzyxnfvz6j32sz6HXotEMeHu8jhaTV6TEcDlhF +RS0pL2kHDu3QNpk8Pdnj4fGAqiqJohhXiw885bZjubvOmn/bZbu72MUudrGLXfyLQwpoBQ6jeg/q +ubLZg15dj5nOVqRZRSU90C27B3VadUOcY323TUmBzyqGyTzm8CCnLA2+7zEaDnlwFPHqdIYQFWVW +kKYZ603KdLHiejLHcRSB79IKfZ48OuQ/ffUpD+YxaSEpjCbwXULf4fHJkIcPhnQ7Ib7vomTdGHhn +S1A1frQfjrXxV/3Ap/YH4j2Q7tZXlB/6goYhK+68/mMghLjzWVV1awVj1apCHux3GfV8Ql+xN2iz +3sSsopgiz6Aq8RzBp88fsbc3wPd9jIEkSWqmfoQxhmGvRbcd4mpFp+0zma1Y10xIKUAJwcPjPR4+ +2KfdClC1SuZ6s2E2WzCezNjECWUjnfvDnk1bL2WDJCsgTismi4h352NcR9Bttwh8lwcHQ778zBKC +JouMxSbH9xwCz+HkeMiTh/v0e220kmRpVoPeM96ej1mscgzS2u8off/83Z33O3//OSHEnb3mH0jR +tarqAmqvbqlc0qLkZrpBVimu59PvtpGitgZqt3hwOCDPM8Iw4GC04WYWg7AS066rafmaduBwctjn +5LDP/l6HfjckDDzAkMQpSZxweT3jzemYbtu37NNel0+ePSLLMybzlOkyxaBptbt0ul0+f7LP8ahL +GFi7q9V6w/XNlLenl1xcTYiTwiqLOVYyWSr9Udnre+v/Z8/X7RwLcVsXKI1gE9mm2zAM2ctLgsDj ++HifL+Pn7B3EHM9S8lLSCn1a7ZAnDwa0AsfKnVd3j+dOTlq9/2/2OwsjKEtYRzlXkxWv313T7YR0 +2wHddpunj45I4oTJMmO6yilKauuwgF88O+R4v0+77QMV6/WG8XTB6cUNF9czoqRAOX69Dm5Z3z82 +fT+Vz9xVM6sqSVEKktwwnUe8u5jSCV363Ra9bovhoMfzpyc4nke3P2R/tKC0dxa049Jp+bRDjycP +hjw52UNrQZGnLFcCLa2dhLUoM3cPgJ93pD8/qnt/qu99UhD6in7Hoyw6mLLAdxXDfpvDUY84M6R5 +hTE0wPeD/TYPj3qcHPbpd1tIAbPFktPzK168esdsuQHpYoQmN4pNUnI1XvL9qwsO9/sMem0C3+P4 +cERlKvywQ38wYDqPKSuBqSShr2mHLnuDkKcnI56cjLi6mTee7h9x6WpsH+4+iG7lzd+7d1Tba0Fj +qFhtMi6ucw72eiSptTKQ0n5OmlpVuMlsyWIVs4kysipAOY5tplHurVT9ezZo98+ePbbSVBSlYRXl +3EzXvD2fcLDXpdsOCEOfkwcHxMmn7I2WXE0illGBdgO047O/1+F4v8to0MJzJL4r0aoiz1LmiyUX +lze8eHXGg6MRg14H3/c4OhhRloaw1aU/3ONmFlGhqLDP/3bLY6/X4vHJkGcne0zma5I0ZSpBCusz +XlE2taYfsxHcxf3YAd//hrGVsR6Px4Rh2ADX7/tWbzabxpd7C1oeHR3xF3/xF/zd3/0dn3/+Ob1e +rwG9tz7T7Xab46NjPvvsMyaTCS9fviRNU6IoYjwes9lsuLq64p//+Z8ti1oppJT3vne1WpGmKXt7 +e413I4CSir29PZ4/f87Z2dkHzGbf92m324xGI0ajEb1ej7AV4mjn3uu01vi+T7/f58mTJ3z11VcY +Y3j9+jXz+Zzlckme5w2A+1/+y39ppLq3XtZb2fHVakWv1yNN0z/oPLiuW/sxHfD8+fNG4n3rif7q +1SsmkwntdptWq4WUsmHPb4HjoihwHIcgCDg+PubLL7/kF7/4BXt7ez/5/ZvNhpcvXzKZTBrPdSkl +RVGQ53nDqM+yzEqSBAFHR0d89dVX/Pmf/zn7+/uNnPxXX31FURS8fv2a2WzGer2mKIpGDv7v//7v +UUo16yxNU9I0beav3W6TJMm/9aWxi13sYhf/z4r62Swb+WxBacBIQVwo3l0t0L9+QZJmuK5Lr2t9 +tlzHYblJGE/mVCanyDPOLm4oS8PRfp9Wy+f5s0f0um1miw2LVUyUFARBiB8EDAcd9gdt+p2AsiyJ +44Tr8ZRXb8958fqUdQIVDlK7NUAo6LR8Dvd7HI46PDjss948JkkyktSCxNaDT3Ew6rG/18N1FFUF +603EZLrk/GrGdJmT00bUvkSNt9h7u/OqlvoSddHxYBgy7Ab4nsIYY4Hvy2tmixVFCdpr4fod3LCH +43fRbgvlBDS+acohyiSTpW0AMKYiDAP2RyPSrKT79RsktXRhzfyoygLLKr57vmq5c1kzvwXNn4VQ +TVLV4JHbN93x6laqwtPQ8hRPjro8Pgw5OezwYL/LoNcl8D20UnQ7IU8fPyAMfGbrnPkq483lmu/f +rdjcxCBEzS4X94qEPxiVsMMRt77c2/FprdgbdPA929Vv5d0KSmMoDLXcmEBJRafl0gldWqFH6LsU +ZcnZxTW//u23fP/qlFWUob02Qlsf61VU8OZsTO/3L3n+5ITA9wgDjweHIzrtFg+OD/nqFxlZZjB1 +ncZzFa6ja+94H9fR3NzMEFVp2fEfaV7cSgNuWQQ/PBvbAqBESk1hMmarFEzKfJWQ32nsqyq7J59M +Z9yMp5ahYwSK2prAqeXTlIMpNaXRLDc5b05vaPsVnzw9IfBcgsCrG1daHB8d8NUvUtLteKt6vFoR +hh7dtk8rcBlP5ghR2fFW5YdjuFMkuzuuXexiF7vYxS528S8PIQWtQHMwCBh2XXxP13vQDReXN3YP +Wgmr/ON1cII+jteugW+3trQxoCRRCuNFZpv8KrsHPdgfkOUF3W/eomRFWeakWcY6TplMV7w7v8HR +koPRgE4r4LNnD+m2ApablCgtyYuKdugThh79TsBev0WvE6K1qgGBehy1nPGP+hLfA6Lff6342Dvu ++9reBc5/bE6FbNR9Ptb4uoVCbnMDu1GqTAGUtAKXg1EX52jA8UGf1eePSNLMMlTL8taWZ6/P4aiP +71m7v9Um4no85+JqjOu6HB+OOBn2GfbbPHt8RJSkpElGmmZIJVFS0G2H7O912Rt0asvAnMVizeV4 +yvnlDUmhqCpZy4t/TIFo2xxg8wQjFGWlSAu4ma74/vU5kpInjw5ptUYcH43wfY9PNylRWpBmZc0G +V3RaHsN+m1bokyQpmyhmMl1wdjnlzdmEtPQwlWMtl4T+4WP52XvF9xog/si95hbkkkpTaZe8TLie +xayWKa1WyF6/hVaC0bCH77mMhj1c1+H46IAoyUnSHOtbK5BS4DgSR0vaoUc79Ak8jetaW4EsK0hS +a7d0ej7h9y/OeHy8h/PAKqF9/skT9oYDkrSwlkRIXM/D83z2+gHDfgtXS+I4JYojLq4mvH53yfnl +lKzy0G5o/YIbye4Pz/mWeXqba/zUfH3YcHILfMNqk3B1Paff7ZDlJd22z/PHDxj0OmySkiguKCtw +HW0t0Tp+rYoF91uEP3Iuxe11bFnTkspIVnHJ+fWc330veXyyj6P3abcDPn32iOGgS5yU1uKsAsfV +uI7DsGfPpasVSZqyzlIur2e8Pbvh/HpGZgKUG9br4Y4q2g/6pIsfXmrb+5SQCGR9T5GUlSQ3iuki +5sXrK7SseP74kG6nRa/bRknJcNjn2ZOU5Tq1HtmVldP2POsbbX3KA6I45lILTJlSljnG5BgjMVXV +rMd/taiZu1tbhYqyWUu9TgslFaNBhyjJiOKCwhhbO6oqlBAICe3AgvmtwCUIrXrZeDLn99++5Hff +vLUWcU4IUmGkR5zB+eWc3/z+JXHyAK1O2Bt02R/2CX2Pw8N9frHOiJMcU2HPvSPxtCaslUZ6HZ/N +JkZLyE2OKd/P1W8bQpp8/QfrF7Vffd00YyhZRRmmyHm0sIz+uxEnCZPpnKvrG5armKyEUii0cmti +idOoNAjxY/dC++/GCHIDUVJwMV7x7atLsrzgycmIbtvn4ckR7XaLTzcZmzgjy8uaBKEJPZdWy8PV +giiKieMYU+QkScxyteL0/Irf/O4FaZqhnj3kYK/P/l6fIPA4PDjg80+3VgMCg2V8e44m8J26CcWz +1sGioiyzmrBRUAn148oju/ho7IDvf8NYLBYsl8uGbXI3hLDMDyllI2u+BbM7nQ7Pnj3jL//yL/nf +/j//G91etwFkt7F9/f6BZQbHcdywpufzecMKvry8bFi9W+AcqCVWZANMSynxfb8ByIW0PuJCCB48 +eECn08HzvAa038qwHx0dsbe3R7fb/YDtDSClbKS3Hz9+zJ//+Z83jPctw3m5XDKdTjHGUJblPYB/ +O2+6lpnsdrtorQnDEM/zfoA9c3+et6/fP9jns88+Y7lcAjS+27PZjOvr63tzs5U/chyneX+v12Nv +b49nz57x6aef8vjxYxzH4aditVoRRRFXV1dWUqku9N4dm9aaIAhot9v0+32ePXvGF198waefforv ++7iuy6NHj/jqq6/IssxK2S4WzTmfzWY/OX9KKcIwbMbjui5K/THyorvYxS52sYv7cddvWKK0xpSA +kGQl3Ew2ZPEaz3c4Ptjj+HAPrTWDvsfhqM/4sE9Z+8udX01tIqJV/Zo++3tD8jy3XoF5XjfJOfX9 +HoqiYLVes1yuOT2/4fT8hqvrGThthO4glEtpRM2CBd/VdNs+R/s9hNh6dxu2BYnt86OqKtabmOVq +zfnlDVc3M8umSCTKB+06d2TiPl5Qq7hl2+z3Q4Y9H99TFGXJarXm8nrCfLGmNALthmjfSo67QQ+9 +ZXzXna+VFGwymK1LNmmJMRWu4zIc9CiKopYNlMi4gqqWL/wgeZCNJ7tNnCqEFHd8vOWdhOojI6qs +N6AUFZ6u6IaKR0dd/vLLIw73WoS+i++56PpZHAY+x0cj+r0Omyi18oj6mqtJQlnMEcqpvb5pOt5/ +erlZsLs0tkAUJznrKGG1jpBa0e+12Bt06n2m+GAsVWXtakpTUhQlSZqyWK55c3rFN9+/4d35jE1i +5d+ldkF5RCmcXS1wZIVSDr1Oi9Gwi++7tFsBJ/We1n5+1ex1hBD1dxmSJKMsbUFLSSjLwp6P7bDu +MCREJT/oJv9gGoSwcmeVoTCCTVxS5hmrjS2Y5YX10dsWS6+uJ9xMZlburNJU0kXqmu2tLcu/FA6V +cNjUBSNZxWhtx3sw6jfjfVDP7VZ+/mPjzbKcsihJk4TKaMqy/AHZ/brog2zk/m6voF3sYhe72MUu +dvHHhBCgpFXF2R94DDoevqMoypzVOuLqZsp8uaEwEsdr4/gdXL+D9jo1K9SpPb4NlTJsMsNslbNJ +SkwJbuDQ7/XIc0O308JzFVIIyrIiSQrG8zVvT29wHVvvCX277z8+GJBmOXGSkxc5vucS+B5Kynov +RF2nuYWQ+aDYf3+PIO+wkoVQCHNXkeljwIS43Q83fr0S8RP7LlnvlbcM3vs5wO3vt/LYNQhRVWR5 +YUHQyuC5mr1ei8O9LlLavZOpa1dS3M9FosSqTs3ma04vbji9mNDvttnbG1iwphvy+ETCVsLcVLUN +1Z19aWmYL1csFivOr8bcjOfM5hvQIeiw9o9VIKApMYq77F2JqBQChcEyLKfzhJdvr9C1/VCvExIG +Pt12eG/vLaVs8q2yLK337GrDdLbk9GLCxfWCm2mE8pRlJNfH8sEZ+8geWdwDFz927rY5qmzWx20D +xc+6ikBUCKy9UKVcyjJnFeXMsw3Dizl7vQAtBWVpGhLTwV6Po32JVPLWq95SQKmqirK0e2VjDHlh +/byTNGe1ikgSm/Ne3Mz59uUlIGiFPvt7Xfb39zg+Omisn2zuaue3KEvK0rBeb5jMlownU85qtvJs +GeEEPjpooXRQe33r23pys3RFs26381VtgeztOfiYytmd1zeN1VJb4HudcH5Tsbc3YLWJ6LR9hsMe +B/vDWwstUzXXgDGGojDcT2Fv1yG1slZzrTZgu1WgKitFlJRc3az4RqZ2/gKPw4MBe8Meh/vDe/jE +3fkzpWETJ0znC6azBWcXYy5vFswXCTqwss71Ef+AVPyda+YeOPrekoLbe852TQtFhaasFLNlwuuz +CVIYXMdl0GvjOppet81g0EVJ2xhkgeWquXcIISxmYQxxvKEocuIoJktTyrKgwr1VgntfSQP50Ub+ +f0kYY8hziOOc9SZhsYrrOkxAvxda9TZ1p+5QXyMApr5GirIky3LGm4i3Z1d8++KUl28uWWwMym2B +tESEtJBcTVZ8/f0pUko6rRDfc3AczWivx8HIftf7uasdf1V/n7Vyy7KMOMkoioL3+9TvrXMEVXMf ++qGGrLppyCji1JBGCfN1SpRYv3KlbA0hihPGkxnX1xNWm4S8VOA4COXWJI47KgN31td9mjlNLl1R +UVSQZIabyZrv3tj3BJ6H62g67Rb9XqdZo7e30dvnURwnXGYx0zgmjhLSNCOOUy6vp3zz4gzluLTb +LcLARzsOo6F3O89CNNf13diuT2NK8jwjiWPyvLbMk259NnZVgD8kdsD3v4O4+1DZsq63zG7XdQnD +kE6nQ6/X4+TkhEePHvHpp5/y5Zdf0utb+XAlPw5QdjtdPvnkE9za96Lb7XJ6espkMmE2mzWM6a0v +N1i29vtA6y9+8Qs+//xzHj9+TL/ft3LqNZA9HA7Z29tjb2+POI5Zr9d4nsf+/j6ffPIJBwcHOI77 +o3OgpPWT3oLmYRgyHA4Zj8dMp1M2m03DVt/O2XaOrFy6bQh4/PgxX3zxBZ9//jmHh4d43s/3S+x2 +unz22WcNczsMQ969e9cwrouiIMuye00BWzn3g4MDnj17xieffMJ/+k//idFo1IDJPxWO49BqtRqZ ++6IoKGp/F6Bhmu/v7/P48WOeP3/OX/3VX3F0eHTbiCAEBwcHfPXVV838DQYDbm5umM1mDXP/7vxp +rZv563Q6tNttHj582Mzf0dHRR5sVdrGLXexiF39YCGFZtFJppDIIVdniVZ2UpSZjvsm4uNnw/Zsr +fN9pOkO7nRZPHx0T+D6XN3Mux3NW64T5csPoaspo0GY07OI4qpa+ss89YwybTcRqEzObL3nz7oLT +80u+f3XGZJGAbqGcDsrrYHCZb0rOrlcEvouSglE/pN3yCAPXNtMpjVRQ1kWZ1TpitdpwcTXh3fkV +b95d8+r0hrRQiBostFJ8Tl1M+TBRFMIW4hwtGhmtXjvAczV5ZgHKm+mS5SqlMArtttCN37Jl4Erp +QO2TZaqKKDEs1wVxUt7xrZb4vke/22F/b0hZrckr+z4bt1JsAiwzv5Yzl9Ie5M/zeKtZ1jVzRWAB +cN/TtEOPVmBZzbpuIBRCoJTCc53b0qUQBJ6DFFUtxV4zn+UfIGdVFwM3sZV1l8JgypTZdEq/69Pr +BHRaPmEQEAZeU4QCKM1tUWgTxdxMLHPm7GLMizcXnF6tWMVghIfruwjtorRLgWS6zCnNgpK3rNYR +x4d99vo99gZt+12hb/cs9WFmWUGW503zxHS+5LuX57x9d81w2OPpo/cUfGrPbqldpHGQlVMXWH+Y +USAQ1jewUhSVIisk66hgMtvQa/u0Qsu8Xyw3nF6MuR4vSXKsmoAO7HcpByGd2oPMRZmAgozJIiPP +Uiresd7EPDgYsDfosjfsEPgBrdBHa9Ws9SwvyLLteDfM5iu+fXXO63fX1vvwYXqHw24LslJqlPYo +KwdZ6fc62Xexi13sYhe72MUfE1JYSxVXS9q+Ztjx6bZdPFeSpRWbKGU8W7Fc22Y47Sq7/6yb4WyR +3UFUJZUwlFVBlJYsNwVxat7bg7r0Om1Gwz6FcchKjUExnm747fcXbNKcqJZv7nbb9LshQljWKyiS +JGW5XDdAveNowjCgFYY1gGtBNCF1rbak7u0TtiCSVHZPURgHKTRSaruPqsHp97cWSirrm6ocKulQ +1UpOt+DC+/t6y9xT2sVIh0rcsmbv7VtqoEOqei9XavLCMF8mnF7OCVyNkobDvS7tlk879K1PtnZQ +SlDW4OhqtWG9ibi4nvDu/IbzywnLdcYqyhj2E6R2KUtot3w6LQ/PdXC0wnG0BcLKkiRJWa03LJdr +Xr+75O3pBS9en3E1saC3dFtIHaKdwNoc3QFSrDqUtuMQLhIHhGtlsrXDMip4e7GiKCrSrGC5Wtt9 +4sCOy3NdtFbkRWHztihhuVwxW6x4c3rNu7Nr3pxNuZknSK+W2HcD25QpbxsftnMvlWMBW+MicOo1 +cQcEbYBt6qYGu4dWqh6Ddu17mvXwB4DfVCAUUhkq7SLdFlAxXmT883cXjGdrTg7HnBxdM+i1GfQ7 +dFohQeDh+x5SSIQUFEVZ16hTlquIxXLNfLlhMre2X79/cUmUGXKjGS9y5Ns5uZGkac6Dgy7DfptB +r4XrWE91IQVFXmKMYb6wjQ2X1xNevz3nzekF37++JM4E2us0qg6WPVrv/8V9uW6BQCqFUg6lqFUA +GnnlO4D4HcawuAPgyu16qfPMwghmq5QiTwnDGwLPIYlj+r0O/V67aVIuS0Oc2HnR2jKwlXt7XFIq +lK7P4b2c4ZZhzh0LqKIquJmnZGlEXgriJOXhUZ9hr8tw0MFzNZ7rIGv5f1NVzfxd3Ux5e3rFm9Mr +Xp2OiTKB8toop412g7oZqGzW0j1MW9hjtU0FGlHVICXi3jxbBQFl832tkJW9B0lt2b1RCheThMLM +yIqK+WLB3qDDaNCm123VymN+0yhkTEWaZWT1dbhYbji/mvD9q1Penl0zW2woSpp74v3lLW0dR1sP ++tu6xr8g6uaSOC2ZmJTK5AhKNuslw15Iv1vn6qFPK/DRjkYrjZACU9qGkCi2tg7zxYrzyzGX1xNe +vrnk1emM2aYkNy7a85rnQiUVi43BXCypxDlxnHF2cc1o2GU07NEKfYLA3pe25yovLKgexwmL5Ybl +asN3ry959faSoqx49ODgHp61lS23149AVPZZeU9N4v3nRr0uq0pRoMAoNnHJbLFhOl8RBj7tls96 +k3BxPeXsasYmKVBOgHAClPaaPH17j5Pbhgnx/nPNXgu2JidRuJSVZrbK4WxFXkIUp1yPJ+z12wwH +Vg5+S1qwzVYQxQnrzYbpbMXb00venF1yenlDkhkq4bKIDG8ul6AuSNKSy+spo4FVFmnm2XO3t2J7 +38ty4sSSDZarDS/eXPHyzQXn13NWm9tj/3HFu118LHbA979xVHc6j7Ysb9d1G/AyDEP29/d58OAB +jx8/5s/+7M/45S9/yfHxMd1ul0670xRPPxa9Xo8gDHjw4AH9fp/Dw0O++eYbvv32W169esVisWiO +Ycv83gK/w+GQo6MjHj16xJ/92Z/x1Vdf8fjx44bp7XmeZVINh4xGIwaDAUJY7wXP8zg4OODTTz+t +AegfB76lkhwfH7O3t8doNGJvb48HDx7w9ddf891333F9fc1yuWwY5Vuvc8/zaLVaHB4ecnxsZd3/ +7M/+jF/84heMRqM/CPju9Xp88cUXPHr0iHa7TbvdZjAY8P3331OW1nNyy5reMuoHgwHHx8f84he/ +4K//+q/567/+66YRwHGcn/VAdBynkYZPkoQ4jpFS3gO+Dw4O+Oyzz/ibv/kb/u7v/o6DgwP29/cb +trwQgqOjI4bDIQf7B4xGo2b+vv32W66urlgsFnV3YNFIs3ueRxiGHB0dcXx8zCeffMJXX33Fl19+ +yWg02gHfu9jFLnbxp4imuKCRyno/U9lkDiFIM0OUJ1yMN/TeXuNqe/8f9Dv0OjZ563Y6TJcx1zdL +ynLG5c2UYb/D88eHCFEx6HUIA1vUyfKCoihZrDZcXN3w9vSC3/z+JV9/+4rZMibJKtBtpNtG+z0K +4bFYF4jrFUpUVEXCYr/DwV6H0aBDqxWgwhBRSYwxpGnOdLbg4mrMt9+/4XffvubV20uiTJGWCukE +yAagrosC4sOCWgN8K0m75TIatOh3LViYJinrTcJkumK5TigqhXZ9lBOgtG+LPsoWdaraH6uoCqKs +QG8KNklJURoqBI7W+L5Ht9tmNBoQ55JlVDX2SPeOSXJbjFOO9feW1YdFux8IC3qXjcS4EhLPUbRb +Ae2W33zG9ncpJZ7r4tTzZBV2HJSs7niQW99uWxz4ycVmu6srwSbJ2WwSkmjNajnl4kI3fnkHez32 +Bl2ktPsQ6k7msrSJ12K55no85cXrU373zSu+f3nKYlOwjAqM8NCujxt0ENpBKpfC5EyWEdPFmvUm +4vrmmsfHQ549PqZ4fMSgbxk7fsNYgjRLWa0jxpM551djzi/GvDod8/bshsLAJkqbMdk5kwjl2MKO +cRCmLvBumxY+dn6E7dCvUBRGUiFZRhmzxZpFz0MpSSuQLJYbzi7GXI8XpDk4ni0uqtrbW9YFbqWt +t1aRG2armMmsbhAY3/DwaMDzJ8c8L27HG/he3T0PaZqxWm8YTxZ2vJcTXr+74e3ZDQf7QzbRfasZ +KcStr7yxRSzuFqj5Oc0Yu9jFLnaxi13s4v2wILKoJZUd9noB/ZaL1pI0qVhHCZPZurY/0WjPq5sv +3ds9olRUSERVUZqKOMtZbMxH96C9XpvR3oCkUCyjiigVTOYRq3XEJkoo8hxT5jw83sdz9e3+QWjL +/p0uKI3BdRS+51IBvufdYWXfAbJrYFreG69sgFFlXKTRCKEbYE98AGILhNJ1A6BngW/5XgPe+/su +KS1Qrn2E8CilW+cB9z+/aexTLrKyfsB5WTFfJZRmhhYGUyaslj3296zHc6fdot2yDZDbmtZsvuDy +esJ3L8/4/XdvePH6EpRHJV1G6xyEoiyNZQIPu/S7Ie0wsMB3WWGqkvUm4upmwvnFDb/+7Xf87puX +XI0XbOLSerrrEO21bUOkcpDy/hxJpSzIUzoN8K1qv9lltGGxtM2O682a2XzKs4dHlE+OqEwfOqC1 +taPKspzFcsXZ+RWn59f8/rtTvnlxynSZkVUe0unU/tNboMeW8rcQyG2z5Pb81s0Zd87VPY/37fE3 +jdlO3VT84fn6mVeUXUvSQSpTW1HBZLlhMptxdn7D2X6LhwcdHp0c8vjhAQejAYOqg9YaKStkJSiK +gs0mZrFcc3E15uJqzOnFmHcXU84up2xyTZxKjHAYLzKWyYIkzdmsN8zmPZ6c7CExNndVCokFbtOs +4PpmxvnFJd+/esfvvn7F19+/ISslSS6Rboj2urh+B6G2DdDbxoG7w9wquLlI4SDQ0CgxiEaW24Jv +d2SflW4+U0rHNm8rTW4k01XEbJ7geRpXGcoi5eGDA3zfQSsrb55mOcvVhsViTafTotOWuK5zqywt +pc0Xts3BRt8BAu/6fCuEcsiLkvEi5Wa8JopTVssFs2mfp48PkbKi0w6RUuFoyPLcvn4y4/T8ipdv +zvn627d88/1b0tIhqxyk20F5LbTbspLhZX7H5/u+sbNtAHDsPFfbe1D9GnnHCEFu71fSAt/CMnul +9omyjCiJWa4iloslF+fnPHm4z/PH+5wc7dHrdqxyrlZIJWvLt5T1JuL88obzyzHvzm949e6K07Mb +VrH1M9827N89ZoGwx6ochPmR+9/PvlLqX4UiyTLiKGGzzok3C64vFQ8Oezw86nE46rM36CIAv/LA +BVlJytLe/xbLNePJjLenl/zum9f8/rtXzJcZi6gkKxXKc9B+p8ljqQyLaMNssWETpUwmU84uOjx/ +8oDnT44ZDXsMKxq2vFKSLLPqJ/P5krMaXH/1bszrswlKa5bruK553K51uX1uaIE0de66raP8WK4u +FKVRlKVkk5TMlhHTmVVIDkOPzSbh8mrKxdWETVwgdYB0Q9uMdodoIZpmsPpZ+B7YLqRsyCgSl6xU +TFcFi3XKahOzmC+4GYc8fbjP02KfYb+NFALl27m3BJA1N+MZZ5c3fP/ylBevTrkcry3wLV3bAJet +iNOS2WLJ5dU1zx4d8ezxAaNhn0EFSkmUtKoXWyKAnecbLq/sHL8+m3I9jcgJqfBr9YVtU80O/v65 +sQO+/yfFcDjkyy+/JIoiVqsV6/W6AZq3sX1YbmW7t6zjvb09Dg4OODo64vnz5zx9+pRer4ejHZT+ +cUax0opAB7iOy8OHD5FS0mq1GAwGnJycsFwuWa/XpGmKMVaeMwxDgiBgMBhwcHDA4eEhT5484eTk +hH6/34ChUkqQsL+/zy9/+UvSNGU+n7NYLBiNRnz11Vf84he/4Ojo6CcBaCFEw3A/ODggz3N836fT +6XBwcMDNzQ2r1YrNZkNZlo3MeBAEDRv64OCAhw8f8uTJE/b3D2i3WyilGA6H/PKXv2SxWDQy4I8e +PeKLL75gOBzSbttOuu1ceZ7HkydPMMbQ6XSa8UdRZL0bjGnO0cHBAQcHBzx58qQBzbcNAT93sxiG +IScnJzx9+rQB1rfnAmjY9FsP76dPn97zAt/Gdv7K/ZLn+XM8z2t81rfzt1132zFs19gWSN8ex/7+ +Pu12G611M383NzcNaH5ycsIXX3zBYDAgCIKdJPoudrGLXdSxijJeX6wwpsSTGY7MObsY8+p0xnxV +kGQKhEZKywZASOszXZZsEji9Xtuu9FXO6eUcz9VQCdZRwnevr7mc2Oegt86ZSunKnQAAd/tJREFU +rQs2acV4HtMOfVxXoZWiKKx6yGKxYjJbcj2e8e5iwXSVkRaKSvs4yrdehV4boTyKSrOODdfTmKIo +uJlH9M5ndFoevqcJfLfpmM6LnNl0xXSx5PxizPn1hsWmopIa6QS2QOSEVnZK6R/coFdVRYWVNjy/ +WvKrry/o+JLKFKzXa759fcN0aY8Z5eLoLfDt1gwV1QC2VSWokJRGEmfw9mLBf/v1O3qhpqoK4ijm ++7djlpvSAqACQDGeR/z2+0uuZxtC34Gq5LuX51yMN6yjgqKSSK1qn+07TIkfDYExJUlaMFsWfP/m +BkVBO9RUpqwlrax8vNjKYWNZDkVZ8eJ0wXwZ3RYC/oDkWmBtvrc+bsZINmlJtchJkpQoMUwXEd3W +jE7NolFaWmZyI59Xsliu6saJCW/P54wXGXmpMTJAqhbKa1uP75r9U5XaJoOFYRVDVaUUZk6Uws0s +ssB/zdiR0p6zpJYDW6wiZvMV0/mam9mGZVzhzjN+/2qMcl+iZIUU8Ob0govx2jKlhINU3q0H3w+s +MSuxRlOoMqZkPIv57fdX3EyXtEMPz1V8/e1bXp/PmK5ycqPqNbxNpG8Tdsv6dqiMh5EZZVWwSsBM +E/JyTpzZ8XZaAe1WgLsdr4A4sZ3cy1XEdLZittjY8W4qpJvz9asxrv8KKSuUgHfn15zdrMhKh1I6 +COnW471bPNglvbvYxS52sYtd/KFRYT1Ms8JwMYn59fcT2n5FkSeslnO+eXnNbJnbPaj00DJEOX7D +LGskoevPqup9V5wZ3l4u+G//9Jpey4HKECcx3725YbEpyEtFJSVSS0oMaVExWxa8OV+Q5xWX4w2v +ziZ4jq733aZhgBlTopW2MqydFt1Om8l8zau319wsEqK0AuGwjg0v3s7wvTdIYZCi4sXLUy6nEXnl +YISHUF6jzGQqwcXNmv/xm3O+fzOmMgWbaMPLd1OiTFBJ+3qhfau0JDSLdc43r8ZoaTBFSWVyvv7u +lJtFCspHKB8tfaIUXp7N+M//+BJHC7QSvD294ny8Ii0VJRqhPZAOuVFEKVzPrE3jbBHTvZzTawf4 +nkMQuCgpqSpDURpm8yWz+YqLqxlnNxvmUYVyBMqxzMbzcURupraxuD0l9F0C386fMSVVZVivI6bz +JdPpnFenE25mCVEqMdLDCVy0G6CckDgXvDpf8J//8Q2OBq3g3dkNZ9cr0kJRohDSRdV+s5bNW1BS +EmWG61lKUc6JEpu39ToBQeDhubre/xvbHDlbMJksOb1cMluXxLlCuj7aaVt5fe0TpYZXp3P+yz+9 +QWKQGN6cXnA+Xts9o3AQyqvzMNtkvY5LLicRVVUxX6559dbh3dkFp6c2t0wygXJDpLJ7348x+n8y +tsCv0sjK1n9zU1BkGaukRM0y8mLJOoHxPKLfuyEMLKNVSIEUgrwoieOEKE6YzlfM5ivGszXj2YbZ +uqSSDkIFKOVihCYvNfNNibyJSPOS+Trj4maB7zp4rkaIiqIoyPOc8WTOZDLj4nrC+XjNOq5AeyjP +R7mW8a3cVs1aFcRpxatTu3alsCpeF1cTTi+XpEZTKQWyBrLrXGSTFFyMN1AZllHKu/MxpxfXnF/c +cDFesUmqWhXNrb3aJca4FKZkssh5dTZnkxScX694+e4GJa0sclGUbKKETZQQhiGtVmAJc0qilOB3 +Ly6ZrXLKyqmbD7YN4ttccnuKJFIoSqExaArjWEnscUyWVyyigsublb3efBcpbTNCURZMJgsm0wWX +N1POb9asEkA7CG0bQ7Qb2qaP0t4f06zizfmM//qPL2oGvmI8W/P6bEqUYe+FQpPlhneXS/7bb07x +FChZcTOe8vZiTpwLyspBSA+lbpn4BluTSIqC2dpKQudmyirOOb1YENY5rpQCWd9HozglSTJ7vc+W +TGZrbmYJm1xaZS1Hk1eaNxcL/vM/fIfnarSSTOf2mONMUFaSSm1VAP7IPEzQvNdUAlNJklwwXeak +SUqcFcyXEaeXCzotj3Y7wNEarW3+t2V8L1cbFqsN48mMN2cTrmcJSS4p8BCOj3Jtrr5tiDKmxJiS +sijZpIbreUpeVqSFZLZK6LQD2mGA77mN0kCa5XXDQMx0sWY2X3Mzi5ivcpQD37+d0fm/XxN4Gilh +Nl/w+mxWz5UDd2TIt835H942BNQNI0gN0mO+zvj2zZg4jum0PDotj9dvL/j+7YTreUKcy7oR6E6u +Xn8OtfpFVla8u5zz33/9mlbg4DqK9SblxdsJ66gkKzWV0Ajl2HkpBau4Qk5TsrxgkxjGsw2twCP0 +XRxH1XNvlQ/nyxXT2YLL6xmXk4hVApXw0F5gP1M6bDLJeJFTmDVJfs1sldBt3xCGvr2+hLg3z5tN +zGR735tHzFY5SSGplLLP4EZx5Y9bev9vDVG9by69i3+VmE6mvDt9x+XlJVmWkWXZPS3/uwygu8zv +u+Bkq9Wi1+vR7/dxHRepfn4nXlVVrNdrNpsNy+Wy+dkCwVug9a4E9pYN3Gq16Ha7jU/3FvTdxnw+ +5+zsjIuLC7IsI0nsw3g0GjEajeh2urQ7bVzX/VnHmqYpm7Xtitz6oEdR1Mzb9ji3cuNbOfgwDGm3 +2/R6PdrtdsPKns/mvH7zmtPT04Yx3u12OTk54fjoGD/w8VyvaSKoqorFYtH8rFYrVqtVI0O+/W4p +5b3zMhwOGQwGzXF97Nz8X//X/8X/8X/8H/yf/+f/2cz9w4cP+V//1/+V//1//9+bz9NaN8D3thGh +2+02IPjW9/tjHuZZlrHZbFiv18zn82b+8jwnTdMP5m8rtb6dv263S6fTaeZvsVjw+vVr3rx503iQ +dzodHj58yMnJSQP0/1QTxi52sYtd/GtFmqbEccxkvuZ3L6d8/WrC+STnegmL6H/usbQDRa/tELoV +RTQmjya2kzrKiLMKI0OMbFk/olpCrSxSyjzFkRmeLPB0ge9KPIfaJxiK0oLqq00BQjYd4J4rCTyF +FBWCsvautj9ZnpNlOUmaEyU5myQD4SB1aAFqN0Q5rcZmRStwRIGjCpQo0KJA1QUzQQ3W1uColaku +SNKSKC1JM4N0wlqGL6yLB0FdDPiYxxeAgcqgpaEXVPRCgawyijwmTTaMx1Mmkxm5EaDsZ7tBFyfo +1UWl2m8P7LEZW3hTlPRCQb8lkNjPy1LbRTtfrEnzCoOV6R502wz6LXxHILDvX64iVuuYrJQUwqeS +NZjvhkjlNT54H0ZlC2mmQFQZ0qTIKqPtVbR8kGSYPKUsM+sHWZnG58wCs5YBsElhsYFNKuz3eq2a +3fFD83j/GOz/BlNkmDJDmAxFijQZjipxlJ1zVRdxhLCM+e3+oDIVRVmSF2V9fguStKBSPkgf5dTn +1wkaULgyBWWeYPIEKVI0OY40uA54WqBlZdnz4s5cldTeZBV5QZ18C7K8wg8Chv0Og15IZXKqMme9 +jhhP58wXa/7/7Z3XexzHuebfqs49OWCQCYCkmETJlsOxdx/7+Nnn/Ld7u3u3lxssH9tKlihGEERO +A0ye6Vh70V01PcAABINE0P5+j2GQ1ExPd1V1T33f+wVu5MHM1NFiFWBYuXR8tDOOumQ84jhAFHoQ +kY+8GSFnC1g8BEMIiACdTh/d3gBeAITMgeA2DLuY9jG3VLlIOb9x6CP0h4iDITg8aAhg8AiWwWAZ +gM6TezfbGjCKBeJYIIwE/FAgDAEvArwAsC0b1UoBlVJOXW+/P0DzpI3TdhfMyKX3VtJjVDOT6+Xn +rvcnRggEvS2EvR3o0Qly2ggFc6T2p/KnWq2iVCqhWCwil8vDsW3YTlLqXk/Ly1G5doIgCEK23jtt +9/BDuoffOfZx2AFa/Z/uc1na69PQGCpFC9WiCYQ9jHpNDLpNNE8TcSSK9eQ7OBURDDuf7gXlnkz2 +Ww0gIh9cBCi5QNkFOEv2HoHvod0ZoN0dwI84BHcgmKXiGk0dsA0ByxAwdcDUADCRVPyJYwRBgCAI +IUTSXzppjWjCMHQEIdAfBRh6AqEwEAgTuZyLWjmHSsFCHHmIwhE67aSncavVAdNzYLoLzcxBt3Iw +bRe1golq0YTOQ4T+AN6oj6OjExw3TxBEDNzIp693oZs5FHI2qkUNJVdDGIwQ+gO0Wy0cHJ/itNVO +svE0G/lCDvVKEdVysr9BnGT0nrS66PSGgOYkJcU1A5qmQ9NYsn/UIug8gs5j6CxOBay07U9abjYI +Q/hBnFTO8iKMAiSVeoykXK9j6bAtDRqLobMIDOMfkfZQTdoZBvC8AP1RgMHARwgOrjlgaa9nzbCQ +c21UChbKeT3Zp8U++v0hTls9tLtDMCMpRc50V/V/j0MPUeCDwYfJA+gsgGkAtg7oGgNjAlzaVmlf +az+I4QUxRoHA0BNJlquZg2bmUoFHR87RUcnpKOc0ROEIUThEr9PFcfMEJ602mC73jDkYdgG6lYNr +aXAtDkuPYLIAGvPQ7XTQ7bTQHyY2qhcwmG4FVq4KzcwpAf/qJPMiRAQRBYijEFEwRBQMgNiHySPo +PEzWuC5gaMleWVN75WSfnLTVihGEyX7ZD4EgAvyIgevJvKh7kGswtRiGFsPSIhiagKUl9jBDCBFH +iKIwzar34XkBRn6IwTDEYBQkYr+RZCobVh6alVNXYxtQ93IcJTbEYDDCabuP084AmlkANwvJeOtJ +FQhbj2DrybmYegRTCzHoD9EfDDAcRRiFHKNQV/cR4xxx6ENEPiw9gqWFMPUYhpaMj9Ssk97eQBTF +0A0TumEmvZ/TqmfJ86qLkR+BGXkwXc59PhEH02oNQqR+/8jP+B8CGCy1Y/QYpg5wFie2vxj3Fff9 +AF4QwfMiDP0YQy9K1omZh2YkIqRm2IijJOPb1CIUU/s+8VEk/oN2P0BnEADcATQ3qYiRN1EtGEDs +Iw6HGAySKhfN03YSSKPnwA1X2ePJ/RuCiRAaPGjCh6mL5EeLwRnAeQyW9nqX91ccRcmaCpL15UUC +fsgQx0m1NNPSUSu5qJbdxEYWMbwgaWHR7gdgeh7MLKTPN+MN74/MnSLiJOA88hGHPhD70IQHLjwY +PIKuRTC4SIK/eRoiz2SpOvn8ixCGyfkNRiGGowCCWxCaDa476ZrOqcx/EceprT4EEx40+NB5BPuM +7Zrcj0nQtrRdozi5D8NQwIsY/FCAawaq5QJq5TwYIsRxAG80SuetA6E5me+NxGZnqe9r0p+RfofG +EaJghCjw4BohclYEx4jBEYKzCP3+EN3uEAMvRMQcxMxJA/Fz0AxblcyPgiFCfwANPkoOQynHoPGk +El8YJnPZ6UeIuQ1m5ABuQfq4dBZChw9DC2FqMSxNJPcCi8FElPoBYgRhgCCI4AcRRn7iB4ugI2ZJ +xRHZykDXYuhcwGAxTEPA1pHOaeKDkc+9KI7TbPI49REIeBFHEMugDBfcSHwS8vtFtbD4iRDhEEFv +B2FvGza6yBlDFG2hKjRLe79araJYLKJUKsF1Xdi2rVrySl3sQ9v8lPH9M1GtVVGtVfGLX/zig3w+ +YwyFQgGFQgFzc3Pv9djlchnlchmffvrpezmeZVmwLAvVWvW9HE+O/a9+9asrj5W8pp8Dx3Fw48YN +/ObXv8FMI8lcd133rY8nM78rlQqWl5ff+fwqlQoqlQq++OKLn2U8CIIgPma6gxDdvo8wGGBwuoNB +excAUmPQgWkzmI6ujCWZQcy4hsATGAxHCL0+Qr+P0B8AIobsySWzDhg3EmOOBYhScTMKPSXsiThx +lCWVndKeYroFzchD053U2HYTh0YabS6EQBBHGPoBosBD6A8Q+QOEwTAxYENfGWkAxn3hdCvJwjZy +4GmJM2n4cs28tFyeSHx6GIUh+r0hNv0BAq+HcNRDFAwhEAOCQzNs6IablJ82HBVZPyn4JVG+Ahx+ +FGHvuIet3V5mLIeq1Jumm+CGCY2bOOl6OOkMEYUB4miEOPTTHoUcmuHCsHUYuj29R+E5mOpRFoWA +H8QIfR/N4y4Cr5uO6RBROFL9z5CWnUtKJdrQTBuGmYduF2BYhYxj9Q367DGRZFxwXZXfDIIIUeAj +9L1EsA1HiXEZ+knAgEgyX+RIyuvluplkPutJFr9huuMy9rqcXw0iLf0WcR1RoGHoD9D1B4nxGQxT +ET5Qa0hAJOPMGLiWOBW1TG/4cAh0hx1s7JwmRno0Stc0B2MWdG7B0Ky0vJkOsLTXIbtgPFhSAjIW +Aq2eh6Nmd+I+k73mNMOBYVswpOCdljKV2dWqjKMO6GDqen2/j64fKAdfHAbnrpen6yop1565XsNG +6DF09zrY2G0hDkbJGomj9P61oLP0WtPylkz1w/vQTzyCIAiC+PhIBDoBLxbYPx5g76gPr9/EoL0L +r99U+y/NsGHqyf47+x08/gJOcr4Z4xCMI4gY9o972Bp1kj1GMEQUeKqctKY70O1kbyUDH/3Qw2CU +7BHDcIjQH0Kk7W6EEJd+32u6mbZmcWFYJnTLxGAk0N9rY2MnROgNEPl9xHGYBlza0LRkXy336rHg +ODgZYO+whdDvI/B6iPxB2r5HT/ZDupPswTUL4Dq6Ax+tjpfs87w+Qq+HOA6TEdHzan8z8jm29jvY +3BvvbwCkpdkN6LoFXbfT9k8ckQD80RDtwEPoJ6JuFAwhohBxFI57FLG0LLGmJ3aC7kAzHUC3wHQH +fszgDSKInjex/46C1F6KM/veNAiV61ayF0+Dg3XTUeW/R0GE7YMuXm2PEAWJfQQhxtdhGNA0Oy0N +nwpijAFcRxxoGPgxwsBD5CXvlTZbHEdyBaVlxxMbRTfc9BzcRFQ0nXRPyDH0QvR6HbwMhghHia0T +RX46NhZ0bqfj76js7f4oRG8QIfKHCEYdBKMuonCEMBiBMZ58luWq95wrUX0lZFsrDdAYeMbGjQIP +w6CPcBgg9ofpvHqJjRmHY6OQJSWgOedpr3RL2dCaYYPpDnTTSQW0ZF/thz5GvodW4KdrcZAef4Qo +DJSNo0qQ60Yyz3YuHePEfuVG8llJYEWMke+h2+niZXo/REGSMZ/0QjcBmODcGverZhoGowDdMLHl +k/uolyQ0iUQoTMRIO23bZapzirmBoddDx/fVGpF+ABUoLcuvM570FQeAdB2P+7Xb0LmVVq0yz9mR +0haGZiRdyBlH4DOMgiQzP/T6iIIBotBLbLc4SoOjZSuw5Hmgm4noqKXiqiw3Ldc9YwyeP8Rut4tX +o07qqwiSCremHHcNmq5jFACDox529r3xsyTyk/NkOjTNgqHJ9ZxWk+MAFwbiKEAQAKMgQrs3GM99 +6CEOPRVonvSgT8ZBtnxIbFwH3LDBtDQTXkTYPepje6+JOAqTYHbO1RoxNA2azE5+JzGPpfeGDqYJ +xAA8P0IUBAj9JJAoDkcZ/86krY70ecHSZ4Zm2tB0N/0eSJ8ZxtjPw5gGIRK7MtZ0RL4OL2DojwY4 +CQbp5/njNmtqzrlqk8FTW121nIt17Bz2sX3QTYJ8whHiKEg/z4CuJVn6XE8rpaS26/msb/kdypIK +e7pAf+Sj1eoiSNdj6A8y7Tps6LYJMxW8E3+anin7n2R8h36I/eMutna6SYBB+v2kG0nAl2Fx6NCh +yeAFxhCFHvwgQtRPvjdCr59+b4wQRz5ELCBElKluIX1hyfedbrowMutURCG8cIRB4CFsJz4C9R0U +hRBI7+/Ux8Ayzz3dcNJkEgdMc9R3C0t9HhQ/fnVI+CYIgiAIgnhPyFLTjGswrBwstwIAStzTTRdM +lmeWffKSDoGJsQ0kjo/UGJPVP8YOGSvtmZU65FKHgRKn43DszJGbcs5VhnkiMNpJKTKeMVBT55YG +2RdJR6Rb0AIXcRxkNufJ+fBUUGeamTjdlABuja/vtWXAxv3GpCGD1DHBDSsNABbQdCM9tpOW7btA +gE7HnQsBTbcSh1QqAGu6o16TZJWYYJoB2ehbM0LEoZVEjyvHSDJfUgBNsomBS9VGWa5L08FiE5oh +UocEh6bbiC0PUegrRwZSZxtjyZwnmSV26uiyoenZ8l1vIH6nTlim6dAggxXkWFgqYCIOAwhEyrma +uN5SQzztAzee38SoS9aSnpa7TA1Yrqe9JGXloqQkeBTY0EI3cVjGY2MaqTGdOLfSXnlpv3AZECLS +MdJ0C1HkjINAGE+dkul6uKxv2HhhJM5JIaAZSWS3us9MdxzskM65EvZlf7wJhxGS69WEahvGuQZN +MxEZNqLQTTJdUiezjM5PhP5EONd02UM+GVswpoIhYt2EFtqpc4OlDsnEocuzmf9k8RIEQRDEeyHR +KA0YVn5CJJPBlzLLSrXZmfgKTnsop3sDzbAgRE4FnsZmoKq0cMNSxxNpxhbX9LSftgUtcKAbXrKf +l4IdLt7jcG6k+8exQAjGARGDyz21pmeEXpZek5MK2smeLtmYscRBzFgmk1Gk9oOjhHKuGemeDErM +kKV0od6flP1mjKUVhWLEugUtctKXJKKNbtjghpMGMfJ0D8pT+8JCFNqIQxdxFEGIrPDNkh7lWtIr +XJYBT+wJI9nNphnzSa9gKxXDgkTAEHEiYKhr4Gnf5fFePCltLysbRWkQpRT/7FT41jLByVa6l03a +AHEpZqTzxzUdkWZCC52xzSYitYbkvntsr8kAyeS42XXANDOZq1SQ1KJQrYnEjpA2WVqGGwyCcQg9 +hm7mADDwyIZm+Ok6t5Qtl33P295LkLaBEGC63Icnok6sJ3vlOPIT+0DOaza4mo3nQ9lH+jiwIGml +ldhcmj4OXkhsShNx6KTBveGE8I00EDY7zzy9FzQ96YUsRAwmOKB6lQsl+CYCtq5EbDU36TlzXQdg +JmuDAWGmMhnjOnTDToNozExJeQ7OIujIgXGGMF0juummwbPSHtDGY5NmzEpbKStQJmKuk6zF1I4Z +T2VSaQBIxkEA0Iyx6KhpBiI5dpEMGhjPC+f6OIA39SnwVHxUQmx6xsoGZgxxlPgqwFgiXBrp+/Rx +kHxSxS35HWfWs27YygZKxlpXthVnUkBO7TzdQhy4SXU1dZ+ntphcI/IatPGzU9pVIo7AdRNRaKWV +5JK2ZFoayDPuJ6299f0xcY+kzxQAgClUcIGmm2NbXc5D5hmePPu19Jmhp88KM20JZ40DcGSrNvUM +EuOe76lAHKVBMtNs9aR/fXbeTWW/ygoCEBGi0IIWWkmgRDqfyp+hm+CalgSpX2arp+fEEKffoXkw +zhGl16WSObSkBQWfaEmW7eGdvEYYAnoqUsepf4wxqLmUQT7ZAHdoY7+bPI6ch/H9IJNSxs8bTTfA +jXFLvsS/oUHw9LtAs5J7y7DT4wVJ5ZBMwA+TQT/yOyD9TufquWdm7n+qdf4mkPBNEARBEATxvkiV +b851aGYOVi51DKVR2tlMALlx5dyAkI40rqWGeIA48tKo88RIYZqRlk7SMmWrktJhsgRzUn4vnhDN +kBrCycZbzzgMNGVkJog0EthIxLe0TJ2Ig8SZJeKM8ZtERidOCV2J6NLZo0S+y4aKJTKrNLzUcXVr +wuDlmjY2stLjY0omuertpOngsJLXpJkgcRQASgDWVeBBctkiFR1D5dgCWOqENFMnn6YixS/XG5nK +LoYulCNG063UeI3SIIJMZoN8jybnxxwbOGmQw3kn6xUWIuPgDIljlWsQcRJBrETZjIGbzeaX86Ac +cFyumcwcszPR7mLsTOCahlg3wSMHuhko4y6OwonPkcdIAigMlbmdrB8oY1DE2SwfabgbqqzgOPv/ +YiNQZuJDM6ClInRiOCeOt8SRzVPns5kGHEzL8mdjh4GWOrm5nlyv4UCPMtcbh0AcQ47q2OGmjbM2 +uJb2PRuXvRtf73h9aLoxDjiQWUTU4IsgCIIg3p1UhOC6CcMuJAGW4EpU4Hoq4mp6uj+a9v2bZrdq +ADds6EwD1+1kXxCHKtCTpaKGLPkKJIFvXDehRQ6Eme7RVCCrGDvFp50646oSkgwgRCo2Q8RpwKM9 +If4oWyCzH+apsMZ5snfWTSejfYwF2UQA0dJjpfs+rkEz7KQyT3JSqUgmbYy0BHZ6TclrMpmEerYt +ktzjRYgjG3okbZswFdbVrmosxKWiTLJHze6r0kxb3UQcOdDjsV0ztpWAsfAmRYp0j8mNZNx5UiaY +a/p4fxsFk9ehy+tIbCzONAjGAZHu9TQdwrCgGangm91/y+uRohzX032fHO/EzlLWWprxJ3giJmm6 +mRl7mQlvqj0758m5MJGZH02HntqOcswZ1zMBHu8SYCkDmwUETCX+xJqRrBMzSDKx1RhEaq7GghIf +z4WmpwGn6Vrl44ppchzG85aIZIndMWm/yvuWp4HbyZ4/a9skrSQFkkpoHAK6mQa36jZiM1T31HiO +0uOwbOsDpmyKJINcPiK4ChyR9p6sCiBEpOyhxAcwDtBAxvYei95j2z25Nq7uCSlQqvv73PMqtSkz +gbRc0yEMM3l2SfswSu+TiQBqTZVxZhm7n8tqVIwhZkm/doDBSG02cWatZd+f+DPSgJI0gzapipYO +28TrjbTNU3r1XKQBFUYSvBClwfrxuPJWcm9khW9NnbMMOpIHTMrAB9DT4OWk4gYy68RUdvA7Z3zL +54cKGpCBIdKmDNPnnnxeRRMVL2TQNk+fOcrXpGVt2LG/hAlAaCwdMy319diIzSAJhI8zAnsyGCpg +Rd136hmZ2MfyOaoZobLz2YStbp7pwY0LTVf5rOUwAF1ARxrIYcqgibHfQAauMG289iSJD0aOURKE +pQJEJuYy60eAOu/xejKhmWHGbxKMAymyGdryma/WqbwXkix7Hqe+DMOCFsnjjSu6jAP70+tPA3MY +H/vs1PeACsYn+/9NIOGbIAiCIAjivSE39Wk2KfLp36XgpU86FKQxwuRGH8qAj2NTZVQkh5CR3tq4 +BF0q2nIRI46TqFv5b2oTPSG4ZY6hjLax8QzNSLK5GYfgOriWKa2V6SuVbPYz0edcOgPT31fakEsx +l48zhlODRtPjzMsmjdSLM8nHx+NcB9MTI11oOnhsqmtMrjsdw9R+HEeEp5HUbCxGj4X814uNyZSw +9PixKiOXZNtYqjfU2Asiz5tlMqA1JXgr585bGDnJuXAwLgBhgLEk6l5wDTw1bJM1IwMaxDnnCtL5 +Ha+dsWMju3ZU5DoYwBk4CxODL3XAIM1mVg4+aUynzlHliGAcqjF2GvQBYYBpSak9NV7ZtcyvUPKL +AQypQ5XrgM7SSHUdPLaSccmUjVM9yCZKmU6OLdLjgTNo6fmL1Pkqr1c6pOQ9Lu/H7H08ztpI71uh +T1yvSM9ZOnOVc4tsXoIgCIJ4DyTiLE+Fv3FwpQzOlOKOlu4lphxB7YPSYNP0vdANxHE85XiaCsoT +IlZ7f6EZ4CogUQp28tjTPjgjBsg9kbIPhNrPi7QXcLKn4Jn9uwzshBKtNKYBwpj8jIl9i5Zuf9Is +4lSknNjbZgMk0wpOQhhKoJX7IbXvSm0VKfLL64m1CBAm4jiGJrL7ZzZx7UpozwjoTO5rZTap0DN7 +X4HJUudM2QbjsVTqSSLiqtfogG6kh85m+2uJHSSvAwyCCXCRHEfI84wNFbgssnZeZt+dnAMf7/sy +wk6SFZ/MgZbu67NjP7bL5DXwNMuXgXMB6Mlnje0ejOc3FYDfrM3SxfdVsuyT0seMMcSZvbJI7VYp +MCIT5DE5FvyMDXJ2H5zYpCJjV4g4uY/kPZQVvs/ZxOrYTK1XxjLVrNJAAX4uaGM8xkytRx3JLcgg +OE+rKWTviawtLu1weW7yfWmgSWylZcbP2PRqFQDn7MnsWGXvBTZ9bpIA9LTSVjrmQougCVPZbeMi +C3zSVuVnrkWOVcaul/fMubWmnida+tkMQvC0opw+3f+RvmfC/oQAuKH8Gown5y9iE1oaUJF8tryV +x6X0s/e7QggIriXPbGQCJrL2MNPV/f2uSL+BrAgmbXX57BMizqzjTJC6qnDAzo8nzzyDsn4eaQsz +kd6PHCINdOKaobK3hbK/BcbPWLl2x39PnmlJ72wRJ7Z69vwmfAcqiP/ySoByDXMY6s9qDM74Z7g6 +7pnvZFX5Qd4T2uR3qZrLzHnJ0vNI5gFMU2tRficjs56yz5KJZ3XWx5aK44LFyThzDVyLEMu1GcfI +ZnyP/UHZ5/ekj4Qyvd8OEr4JgiAIgiDeIyqiXDcz0a18nO3B5WZYbnJTo4cDHEn2t+A6eFaQhBTO +eMaZhFSrTBxB2dcLGTWfOsvGBtCk4H2uVGPGeSAyhgIgDd+M+KiMruw5SePh6mW5WTo+4HrSD45r +SixU4ykNL5Yduzc9XtZYTLPDJRPjBnWMrGPiatclS8glDpNE/I0gYj0tO5kdyzNrZmI82RnD6W2M +nPRcBBsHWUiHUGpAc2ncjiMAMmM0/mzZT26yt/Rk9kDSWzydK2Gkhp6WZJhMEdcnjDx1zRnnnnJK +ivQY2bdJx6scs7Pnc34s1Pml60LwJBNn3BaAjcc9vUdxoYE5eb0QDBwcgmng8piZ9ZQtsZd1Xkkj +VgBgaS9wiBgsW/IRSSACzjmwyPAlCIIgiPdDmvGc9vEe/9vkXhAXfv9OBnOyNMNWBqcqp77MLFaO +cwYmkgxTcWbPOrE/u6iizZm9lBRx1L5C7XWymYJyPz0OZGQ8yRKX++akxHLmMzJ7FrUXFqkgfPYz +AJwNkJT7v8l9dvacM8fliWgtuA6WCqPn9qtqLzUucT35mamRJAV6ZdNcdCxkxoNn9oIYnz830vGR ++7zJ60Ca1SgDLcEEmEhsKs6TsINkfseZflPtHUiBMROAwbJnkq4hkYqVaelndTV80jZjaTWypBWX +rtpVTQppfDyO/D0KLIxBmSJcB1djoI8rHWWDYtV4MmRtZ2TKNZ+1Q5SACwDcBGcRwOMp99FYXM7a +lWdFwjRuRK3LZM4zQQrn1lxmb57eC4ngpSXvO3NPjG3xsf0iIFQFOMGjdI1KW0J97Pg5Ml6V6trk +WpywmadUSMtMzthmTcdYy9iIZ9fnOdtfBe9k12jmXuI6RCoyjjNbcWZtZgRz9QzUMWEoy8CV7DMo +c07q2cHYmfPPCLhpxvf48yd9GOObS2TGP+OD4enzcsIWfh92WPbeztrqybNP3R/nnlnZdZh5Zp1d +j2fvYQb1TErKg2vQWJQERykfxaStPg7qGD9vk/sR43uMZ5+r6adKsXyaCH/RaDCWBgIAXGPpd0t6 +XLW+2KXHTT5bAxNp8NEZP5A6/yl+DbWGwSBSf9j4eX+V76CMjw3pc0l+x6f3Npt49k/zi0zxBcmg +DbL93woSvgniZ8QwDJRKJVSrVYRhiCAIUK1Wkc/nYZgGtIv6lhIEQRAfFbLcdVKODshGcmdepX6P +nWU8zfSUhq4UzrJOpMnjjMW18etF1ghO3nDmsy5xoLGkN/bY2D1zLtkjT3FAvMVgJZt6wcCgTVyH +Gif1cVfY9F9yPDUu075rz37uW1+bHGMtHUsN4JnPP6t6n5uXN7ze15zL+HAsNSgTY3VyXDLXrnwD +k+fyegFeftYV1tDY/3TmuGMHQGLjnhHkM3NzftyuOBbZtaE+KGMwT71Hr3i9amCn3b+Z4505f3aF +682OEUEQBEEQ7xmWlJoVPH7Lfa50yo/3WtK5fenxmEg12nHgH3Bmb3bZZ8o/TYjSIt2jaJfsoTJ7 +nlTtY+BX2IdLrvYZE1xlf5PupZLxG/dIPmeLTFQemm5fyf8mxNlg1/Pjm63sdG5vmW7Uzs7Rpddx +bp8IAInoO76m88Li+M9nryd7nmlA5aVjn7kOteQYIHi6B86KqmzKObwn1JrPjkHWPsjasTiz1i6Y +j8kPgMzkvGiOBMT512eCUaYeL4maSI93gQB9Zo7G9h9P9MdL52a83uX4jOcGZ9bpG+z+32gumVpP +ySNAm+6HwFnbf/JzLp4PTFlrU8YvidS5+no+95/lc1eef2b8MkE/yesvem7It2nnx/7S9fI+YGeu +A1PWwaQP4dz9cSU78fwzSbDxmE0Gw19ku2bGLV0jZ/0K8vVvbruOq0Scvxcu+h46K+7L9ZlWE3ij +uRyPJUPme+OcL+n8vE1bn9L/kvyPq++zs+M86RO5+HjE20HCN0H8jMzNzeHf//3fUSqVEEURoihC +o9HAw4cPMTPTQC7nQtfptiQIgvjnYNIgPp9dPe31IimVjbPGxkXHgTKQJv0z0w2Cq22gpQHMphip +58743UTv11xH9vzfzM4cR+xPHxc27S3v8dom5xLSqZCO7ZSLf8frfddxyVznlHX65udydg1lsgqm +Dum0dY1z43f2zW9nEGbG4MzFXeRkvPr1ynO+7Hqnj+ll13vWhCYIgiAI4v0iv4ez/ULfJQBSCExx +hE873kV7xqt/7tmyu9k9/OV7qCnvueI+/CqfMXWQZebdBbbK+O+X7CPPjPXln3t23zdxVRccb4rN +c8E+7fLrmLyes+N2seh6+fHebo980V41Mxo/6TbzNWPApr/nTfb6V5ujq1/v1cbqsnv5/Gde/LnT +3/tmU/K6z7jsfZfYMZNnOeGXeLPjTR/7i9fz2ddfLupOHb+pA8EuHB8GlpRoV8/g6ef803FFH4JM +hrjSfEz/HDlHADv/fXWFtZt971Q/yhuf02VjcP7zX/+8TQMJ3nIuX+9jO3s8dqXjXTzOb3Y84mqQ +wkYQPyPz8/P4j//4D/z+979XvTNM04TjOHAcBxrXwDX+7h9EEARBfHDeLlv37aI7316wu/g8xsf9 +OTbdlzkR3v6Yb36893mtFzivmLjw9T8PrxsXgfc1/sBFc3q1z7iacfuuY/D+jnl9r5cgCIIgiMt5 +nxlWb3qsqwmeb3QGb3G8N903v905v9nr3o9t8DY2wWXHeJvjvGvg5vs4l5/C3npfY/DGuc3vcVyu +cry3uZff1g/wPs79za8VeJ9r482O937mjb2THSbf9iZBRz8NV/EhvD9bHbjofvxpvjfebgze4tqU +tvx24vv7vbaLfGvvy+9CXAQJ3wTxM2KaJkzTRKVS+dCnQhAEQRDEB+G6Gzc/X6DDvw7/atdLEARB +EARBEFeF9srET8U/y9r6Oa/jn2XMrjs0zj81lFpKEARBEARBEARBEARBEARBEARBEARBfNSQ8E0Q +BEEQBEEQBEEQBEEQBEEQBEEQBEF81JDwTRAEQRAEQRAEQRAEQRAEQRAEQRAEQXzUkPBNEARBEARB +EARBEARBEARBEARBEARBfNSQ8E0QBEEQBEEQBEEQBEEQBEEQBEEQBEF81JDwTRAEQRAEQRAEQRAE +QRAEQRAEQRAEQXzUkPBNEARBEARBEARBvBOMsYnfBEEQBEEQBEEQBEF8nDBMt+8/BpufhG+CIAiC +IAiCIAjijZC2LmPsnOj9MRjCBEEQBEEQBEEQBEFcxMWi93W3+Un4JgiCIAiCIAiCIN6Y627sEgRB +EARBEARBEATxhpwx9bMB7x8D+oc+gdcRBAFGoxFGoxFarRba7TZ83wcAaJqGeq2Oaq0Kx3Fg6AY0 +XfvQp/xRE4UR9g8OcHh4gOFwCADgnKNcLqNUKiGXy8GyLFiW9aFPlSDeCM/z0Ol00O12MRwOMRwO +EQQBgOTBXavVUK/XkcvloOs6OKe4IIIgCIIgCIIgCIIgCIIgCIIg/jX5mARvybUXvj3PQ6vVwvHx +MV68eIH19XUMBgMAgGVZuHfvHh48eICZ+gzcnEvC9zsSRiFevlzH3//+d5yengJIAgzW1tZw8+ZN +zM/Po1KpkPBNfHQMh0Ps7Ozg1atXOD4+xvHxMYbDoYpWevDgAR4+fIjFxUUIIWiNEwRBEARBEARB +EARBEARBEARBfERcW+FbCAEA6Pf72N/bx/rLdfzjH//Ajz/+iH6/D13Xkc/n4TgOlpeXUSqVYMf2 +hz7tj54oirC9vY2vvvoKe3t7AADDMDAcDmFZFmzbhuu6H/o0CeKNCYIA3W4Xh4eH2NjYwMbGBlqt +FoQQEEIgDEPkcjmYpknBHQRBEARBEJcwLd77Y4wCJwiCIAiCIAiCIAjiLB9HL++LuJbCtxShoijC +zs4O/vb3v+Hrr7/G1tYW9vb2oGkaKpUKyuUyHMdBPp+H67rQ9Wt5OR8VURTh9PQUOzs72NraAgCY +polGo4G1tbWJ8tAE8TFh2zbm5uYAAN1uF5ubm+j3++rHNE2EYYhms4nPPvsM5XL5o32wEwRBEARB +/BRM7IzY2BCetmeifRRBEARBEARBEARBfCSwSZv/dSb9dbb5r61SHEURPM/D3t4evv76a/yf//N/ +0Ov10O12USwWk57ehgHLspDP55Fzc+Aa9eR9V+I4xunpKXZ3d7GzswMgKSl/cHCAVquFwWBAwjfx +UeI6Lubn51Gr1rCxsQFN09Dv93F0dIRms4kwDNHpdNDv91EqlfDZZ58BuN4PcIIgCIIgiJ8XlhjC +LPn9un0S57SPIgiCIAiCIAiCIIiPApZY/Uz+BUllbsY+rgzwayl8h2GIg4MD7O7u4scff8SrV69w +cnKCOI6h6zoqlQrW1tbw8OFDLC8vI5dLRO+PZdCvM5xzFAoF1Go1jEYjAEmp82q1ikKhANu2KbOe ++CjhGodjOzBNEzdu3MCDBw/geR6CIMDx8TG63S52dnbgui7W1tbw4sULlMtlFItFKntOEB8JLBuX +yMabM4IgCOJ9cfFzVWZ+cz4ZjMwy4jcjIZwgCIKYAstkGDHQPp4gCIIgCOKDcGYLlq3wxl5T9e06 +cS0VzCAIsLW1ha+//ho//PADDg4OMBqN4LouXNfFysoKvvjiC/z7v/87Go0GHMe59gP9saBpGmZn +Z3Hnzh0UCgUASanztbU1zMzMoFAowDTND32aBPHGMMaSABnOsLq6Ck3TYNs2BoMBdnd3EUURWq0W +dnZ28OLFCzx69AgrKyuqsgRBENeb8eZLbsAwvQktQRAE8e6IsTU8zeid9ney1wiCIIgs4++FcRUR ++q4gCIIgCIL40Mgs7w99Hm/PtRK+hRCIoxij4Qjb29v4xz/+gWfPnqHZbGI0GqFarWJubg63b9/G +559/ji9++cWlmd5xHCMIAkRRpH5E6qQRsVDv1TQNuq5D0zRomnbpZlsIASEEgiBQx47jGCIWKoOB +cw7OuTqurusq80EIAc/zMBqNEEcxYhGDMw7d0JWgHIYh4jhOfqIYXOMwTROapqnPFbE49155/mez +LOTnBkGAMAxV//QoitR1Z8+zXC5jeXkZtm1DCAFd1zE/P49KpQLXdc8J32pMz4yLnM/LxuWisX7X +cX6fa1IIARELBGFyLnJuRJw+ADibOA/5e9p1RWEEP/Dh+/7E9ei6DsMwpr7X9/1z82aa5oVzHoUR +PN+D7/sQQiCKImiaBsMwkvdwDVzjiKMYw9EQvucn94uI1ecbhjGxDuUYZO8Z0zRhGEYyBpc8BafN +pVwb2fGTYyjPddpcRmE09ZzltUVRNHnesZh6f8zPz6NWq4Exho2NDTx9+hTtdhudTgdHR0d4+fIl +vv/+e3DOUa/XUSqVyAAniI+BjOhNujdBEMRPRboHxvnobxItCIIgiDclG7jKGAMYZXsTBEEQBEH8 +7Ijk/4QQyqmatfOn2fwX/fuH5loJ30EQYDgc4rh5jK2tLayvr2N/fx+j0Qi2bWNhYQG//vWv8fnn +n2N2dva1Pb17vR5OTk5wenqKTqeDTqczFhyFgGVZsG0bxWIR9XodlUoFtm3Ddd2pkySFdM/zcHR4 +hIPDA3S7XQwGA3iep4Q1x3GQz+eRz+dRr9dRr9dh27a6xpcvX+L58+fodruI4xiccywsLGBhYQFC +CLRaLXQ6HQwGA4xGI+TzeczOzqJarWJ/fx8HBwcYDAbqvfPz81hcXESlUkE+n4fruufPPYpxeHiI +g4MDnJycqPMuFosol8uo1WqYnZ1FPp9HFEUIggCj0QhRFIFzDs/zwDmfKuhKUVOOy+HRoTr+aDSa +Oi7VahW1Wg2O7ZwLXojjGGEYwvd9HB4e4vDwcGI8ssfL5XLnxvl93mRxFMPzPXieh4ODAxwdHaHb +7WI4HMLzPCW453I5lMtlVKtVlEollEqlqVnC3V4Xr169wu7uLobDIYbDIRzHwczMDGZmZlAul1Gp +VNR7hRA4OjrGwcE+ms0mut0u+v0+FhcXsbi4iGq1ilwuNzHnnW4HGxsb2NzchO8nInsul8P8/DwW +FhaQz+eRy+XQ6XTw4sULvHz5UgUWuK6rzqXT6ahr9X0fQRDAtm04joNqpYrGbANzc3MXrovsXIZh +qNZfu90+N5e2bau5rNVqqNVqyOVy5+ay3W7jxYsXeLH+Qp1zPp9X90+r1UK73Uav18NgMMBwOMTc +3Bzm5+dRrVbV+tN4IpbXajXcvXsXx8fHePLkCV6+fAnP87Czs4OvvvoKuVwOt27dwvzcvBLoCYK4 +vjAAnAF6er8yFoPKJBIEQbxPYkBEYIjBGNJnLVO/syI4Z9SKiiAIgrganAEaZ9A4AyPhmyAIgiAI +4gMhwCBt/otF7+smdJ/lWgnfYRii1+uj2WxiZ2cHGxsbODo6QhiGcF0Xi4uL+Ld/+zc8fPgQs7Oz +r80yHQwG2Nvbw+bmJvb393F4eIjBYJBkTAuhRLCFhQWsra2BMw5RSgTxqdmmqSA8GAywvbONp0+f +4uDgAK1WC91uV2XhFotF1Go1NBoNxHGclAc3THCNIwxDbGxs4Msvv8Th4SHCMISu63jw4IHKhN3e +3sbe3h7a7Tba7TZqtRru3buHlZUV/Pjjj3jy5AlarRbiOMl2ffDgAcIwTLKzNX2q8B3FEY6OjvDk +yRNsbm7i8PAQp6enmJ+fx9zcHG7dugXbSsTHMAyV8O37PhhjCIJgwql1dlw8z0O/38f2zjaeP3+O +/f19NS6GYcAwjIlxWV1dheM4Kntc07WJ4/m+r0pQP336FHt7eyogQGb3FotFVKtVNBoNBEGAfC4P +y7TAtPd3w4VRiNFwhE63g83NTTx9+hSHh4mw3+v11JxXq1UsLy/jxo0bSkCeJnxLsfnbb79Ft9tF +p9NBoVDArVu38Mknn5x7rxACx8dHePz4sZq3k5MTPHz4UAUl6PrknHc6HTx9+hRfffUVBoMBBoMB +qtUqPv30U+i6jjiOYVkWOp0OHj16hL/85S9KnC6Xy7h16xZu376Ng4MDHB4eot1uo9/vYzgcKlF/ +aWkJsYhRrVZhmiaEENMz92OBMEzGcHd3F8+ePcPOzg5arRZarZYaPyl41+t13Lx5M2lr4Ljn5rLT +7eDRj4/w5z//WZ1zrVbDw4cPYRgGtre3sbOzg6OjI3X/3L17F59++imAxDmbz+eTKgpaMm/37t0D +Ywy+72N/fx/Hx8fY29tDv9/H8vIy2u02wiiEDh0g3ZsgrjcM4JxB0xg0ThnfBEEQ7xeRlDgXMSDi +TIbeBT98sg8YQRAEQVwEYwyazsA59fgmCIIgCIL4MEh7PwIywe7ZQPfsn6+z+H2thO9er4e9vV28 +ePECBwcH6Pf7Sgis1WpYWFjAysqKykyeRhAE2Nvbw8H+ATZebWB9fR1bW1s4OTlBq9WC53lKYJbZ +3RsbyesWFxdx584d3Lt3D9VqFYZhQNd1eF5SNvrg4ACbm5vY3NzEq1evsLm5idPTUyUKypLLruui +UCigWq1iZ2cH29vbuHHjBhYWFmDbNg4PD/HixQvs7u4iDEMwxtDv93FycoI4jnF0dIRms4l+v49+ +v4+FhQUUi0UUCgU0m00cHx/j6OgI/X5/QvxkjMG2bVRr1XPjEkUR9vf38cMPP2B9fV0dHwBKpVKS +PSuSTPh+v4/Dw0Ps7e0p4bvZbKqgAZkxL0u2Hx4eqvHY2trC9vY2Tk5O0Ov1JsbFcRwUi0VUKhX1 +2pUbK1haXsLc3Bx831dZ4682X2Fra0sd9+zxpNhbKBRQqVSwvb2N3d1dNc5zc3MA3s7RJoTAaDTC +cDDEweEBXr58iVevXqlrOz09VRnLsix4oVDA1tYWnj9/jtXVVdy8eROLi4tKzJUkwR09NJtNJSy7 +rquCGGzbxszMzPhcYoHDw0M8efJElf1vt9tKLJbCcaVSUSXFO50k4/u7777DaDSC53kq2zv7YOr3 ++9jd3cWTJ09UsEMul8PxcVJx4fT0FK1WC/1+H56XZL27rot8Po/t7W0cHR1hb28PKysrWF1dRblc +VuMt75lms3nunjk+Pkav10O/31cZ847joFAooFQqYWdnBzs7O1i5sYL5haSagZzL4XCIvb29iXMu +Fovodrs4ODhQ94YU6/v9vsrsrlQqqm+9PE/HcdS4PH36FMViEe12G77v4/T0VN0HR0dHKBaLKBaL +H/oxSRDEJXDGYGiAbTKYOgPn5DQjCIJ4r4gIED4YwiQ7L9NKZlr293U1ggmCIIjrQVIhhMHQGWxD +7uFp/04QBEEQBPFBEBEQ++A8BGeTLWqnCeDA9Qx2v3bCt8yqbTabCMNQiXpSzKzX6ygWi6q38Fk8 +z8OzZ8/wt7/9Dc+ePVPZ3p7nqR7XAFTvatmr2HVdVKtV/Pu//zvy+bwSMzVNw3A4RKvVwo8//ogv +v/wS3377rcpYlceVAjRj437Ntm3jxYsXWF5exsOHD/H73/8eN9du4vT0VIl7sh/x6ekpXrx4ASEE +hsMhRqORymiN4xjLy8uYn5/HYDCA7/vo9XpKQLQsS31eVjSVyFLke3t7+PHHH7G+vo5er4cwDDE3 +Nwdd12HbNnRdRxRFaLfb2N3dxfb2tipxfnCQlHWX/aaFEOj1ejg9PcWjR4/w5Zdf4ptvvlEl5bPn +P21c5ubmsLi4iIcPH+K//tf/irm5OQwGA7TbbTx5+gRffvklvvrqK1W6ejgcnjuenDvLsvDixQs8 +ffoUDx48wO9//3s0Zhpg/O2dbZ1OB8fHxyojWl5br9dT15adc3keruvi5s2buHPnDu7fv4/PPvts +QviWVQM8z8Pp6Sn29vZURrxpmqjX67h165Z6fSySQIinT5/iyZMnqoR3oVBAsVhUJdZXVlZUP+9e +r4ft7aQigRwzTdOUcC0rGnieh+PjY2xvb6t7g3OOnZ0dOI6jyptn16EUqguFAtbX1/H999/jD3/4 +A4qFIkqlEoDkQTcajdBut/Hs2TP8+c9/xt/+9jecnp6qucx+npxLOQbr6+t48uQJ7t+/j9/97neq +zDgAjEajc+dsGAb29/fx448/qvLmnuep867X61hbW0vWb9obXGJZFmZmZuC6OczNzaFcLuPo6Ai+ +72M4HE7cqwBI+CaIa4radDEG0+BwTQbLSEomEgRBEO8PJiKw2AdHAM40aJo5YQSfLXmu3ncNDWGC +IAjieqBpHKbO4JoMtgG8xyJ+BEEQBEEQxJURgAjAhA+GKLX5J4Pds3Y/cH1t/WsnfO/s7GB9fR0n +JycIgkAJ0svLy5ibm0OlUpnewzrtJdzr9bC+vo7//M//xPPnz3FwcIBms6kyY7OOGCGEel8QBMjn +82g0GvjFL36B2dlZJQafnp5ia2sLP/zwA/7617/im2++Uf2FWSbqQWbcSnFTCIGTkxOV2b2wsICZ +mRmcnp6i2Wyi2WxiNBohCAIcHh5C13V1nGzkRL/fV4I3AOi6roTng4MDOI4DXddRqVRw9+7dqeMy +HA5xcHCAjY0NbG1tqf7Uvu+rLF7DMNRxT05O1PkxxtBqtSYyvqMowunpKTY2NvD999/jr3/9K77+ ++mt17YwxGIYBTdPOjbUcl8PDQ3DOcfv2bcRRjFarhe3tbfzwww/4z//8T/z973+/dJzjOFZCfLPZ +xN7enspufvDggZrzN+nLLERSmvvk5ATr6+v4xz/+oeY8jmPVVz07V/I9sn98u93G6ekpgiBApVLB +7du3lbArex3K7OXT01OVqW/bNm7duqXWThwlPeWPj49VhvxoNMJgMECpVFKl41dXVwFAnUOn08HB +wQG2trbUddXrdURRhFwuB8uywBiD53lot9toNpsq2z6KIvUgM01zYg7lPToajWBZllpPlUoFDx48 +wOzcrHpPu93G1tYWHj16hL/97W/485//rMYPgBK7s3MpAwJOTk6wt7eHwWCARqOBzz//PJlLw4Tv +++fOWQryMhhG3j+apqnMdiWGR+HEfMs1Ui6XMTc3h1qthp2dHURRhH6/j3a7jb29Pezu7qJYLF5Y +0p0giA9Pct9zWDpDztZgGxyGFoMzASEY5X0TBEG8NUmJcyEiQATgIit8n/+5qPQZ7aEIgiCIs8jv +CdNI9/Am7eEJgiAIgiB+XtK2ZnEIJgLw2AOHgMZ1ZefLJGLOOTSunQt4v272/rUSvgeDAY6OjrCz +s4NOp4MoimDbNmZnZ3Hr1i00Gg0YunHhe1utFnZ2drC5ualKUkdRhHw+j5mZGczOzsJ13QlBTIpb +rVYLQiRbaik6xnGsMsi//PJLfPfdd9jd3UUURSgWiyiXyygUCigUCnBdV4m+p6enqsd1EARoNpvY +2trC+vo6isUiTk5OVOa5/Dwp/tl20mfbdV0YhgHLsrC6uoqFhQVUq1X4vg/OOUajEY6OjhDHMXq9 +Hvb391VfcCkUZ8XN3d1dnJycYDAYKHFTCn4ym/6i8vFZsiLlixcv8Oc//xnfffedKotuWZYqaV6r +1VCtVpV43e/3cXp6ik6nA8dxwBhTYqgf+Hjx4oXKrt7d3UUQBCiVSiiXyygWi8jn88jlcoiiCGEY +otVqYX9/X1UHODk5wfb2NtbX1/H06VPMzMyg0WjAcZwrrT85757nYX19Hf/3//5ffPfdd9jf31dl +wHO5HAqFAsrlMsrlMjzPw3A4RLvdxvHxMY6Pj9HpdLC1tYVcLskiXlpaQrVaQ61WhWEaqFQqmJ+f +x+bmpiqlLzP4O50OgiBAEAQTa7Pf78P3fRX8MBgMcHh4iP39fXS7XVUpQIruo9EIQiT96mVZ9Hw+ +n/SbN01oXJs6BrJUv8yEnpubQ7FYVEEHr169ws7ODvr9PoIgQLvdxv7+Pl6+fIlyuYx6vY5atab6 +2H/99dfY2tpSJcmzc5nP59VcttttHBwc4OjoCFEUqXv55cuXePz4MWZnZ9FoNC6cN7ku8/k8XNdV +/eNN08Tq6ioajQaKxeLUvusS2btcZpTL8v57e3vY2dnB/Pz8z/AUJAjiXeCcwbY0FFwdOceHYwqY +eowo4ghjcpwRBEG8DSKOICIPIhoB4QAcPnQew9CZqlwkW0QpQ3iKAE4QBEEQWcYJHwy2qSHv6sg7 +GhwrSPbwMUMYcwjaxBMEQRAEQfxkiDhMbP6gBx4NwVkAg2swjERDlFWAlc2vycxv7dra+9dK+JZi +7v7+Pnq9HoIgUGWxb926hZmZGRimceF7s5mxe3t7OD09VX2TFxcX8Ytf/AIzMzMq4/Xw8FD17ZbZ +4lKMjeIIUZT8vHjxAv/v//0/rK+vKzG7VCrh9u3buHHjBubn5zEzM4PRaITRaITnz5/j22+/Vf2V +e70ednd38fLlSxSLRSXUZpGZxLK0e6PRUOLg8vIyFhcXUa1WwTlHLpdDp9PB+vq6ykyVPby73a4S +tgGoDNqdnR3Vm9r3fVWau1qtYmlpCfPz89B1HcPh8NI5SgTGpL/3y5cv8ec//xnPnz9Hp9OB7/vI +5/MoFotYWlrC3bt3cevWLVVyen9/Hy9evMDOzo5yhElRPAgCdbynT5+i1WrB930UCgXcunULKysr +mJubw+zsrBKbNzY28O2336prkqXDNzY28Pz5c4hYoFwuv5HwHUZJ9r+sGvDs2TMl8jqOg5mZGSwt +LWFtbQ03btxAu91Gu93Gq1evlPje7XZV/2r52kSUzakxl33bNU1TInez2US321Xj1e12cXx8rLLt +e72eCs4YDAY4ODjA3t4eut0uAGA4HOLk5EQJ31EUqdLyUvTO5/KXloCXry8UCrh9+zZ++ctfYnFx +UWV+/+///b/VOg+CQPXWfvXqFer1OnRdR7lcxsbGBv7yl7/g+++/R7vdxmAwwMLCAlZXV7G6uqp6 +jstjbW1t4dtvv0Wv11NZ3fv7+9jY2MCzZ88ghECxULx07qIoguM4aDQaqFarKkjh1q1bmJ2dfa3w +XS6Xsby8jMPDQ5yenk6M8/7+Pjqdzk/27CMI4t1hjEHjDLbBE+Hb1uCYgMljeAKA0KjdN0EQxNsg +YohohNjvQYuH0OBD5xEMnU+I3tIIlpWRLip7ThAEQRBZOGOwDA0FR0fOSkqemzyGD44oFhCg7xGC +IAiCIIifChEHiIMBEPTA4iE0FkDX2Dmb/6y9z/m46q7kutj/10L4lmLeaDSa6BEdxzEsy0K9XseN +GzdQrVYv7O0thetWq4Vut6t6b0shr9Fo4M6dO1hYWIDjONA0TZUblxnPnU4H9+7dQ71eh8Y19Ho9 +9Pt97O7uTmRwZyeZMaZKiXueB9/3Vf9wwzBUlu5gMMDx8fFENrvEMAzMzMygWq1ibm4ON27cwOLi +IlzXheu6qNfrWFxcVGJ4tVrF/v4+8vk8TNNEGIZq7I6Pj3FwcKCyo/v9gRIQZfl40zRRLBbRaDRQ +q9VQKBRUMMDrSETJFtrtFvb29nB4eIh2u63E9EajgU8++QSffPIJbt26hdXVVSXkLi0todFoYG9v +D0EQwPd9LC4ugjGGvb097O3tTYxz1ok2bZzjOFaZ67IftRznra0tlSF/VWR2vAy+ODo6Qq/XQxzH +cF0Xc3Nz+Oyzz/DJJ59gaWkJCwsL6Ha76Ha7KJVKCIIAo9EI/X4f3W4XnU4HOzs7ePLkCWzbxvz8 +PCzLUhnfpVJJ9VWXPbG73S6GwyF6vR6Ojo6wubmJ09NTVdIbgMrk73a7qge67Ld+dHSEk5MTjEYj +AEkP61KphEqlgnw+D03XLh0Dx3GwsLCAmzdv4tNPP8Vnn32G+fn5pEw759je3sbGxoa6vk6ng8Fg +gGazif39fTiOA9u21XzKku9SOJfHiaLo3FxyzmFZlsp4Hw6HqlpCtVqF53tTz7lQKKBSqaBWq2Fp +aQk3btxAvV6H67rI5XKYn5/H0uISKpXKpUEQ+XxeBVdsbGxMrInT09PXBoUQBPHhkO0wGGPQdR2m +oaHoaKgVGAZDoDOM0R4KRDFHLBg5zwiCIK6AiEMgDhAHPQjvBMI7AY96MDXAtkzYlqWqC8l2N3LP +l/1RGX3s6u2HCIIgiH8NknKZHIbOYRoaCq6Gap6hVxRoD2N0hEAY0R6eIAiCIAjivSIEhAgh4gDC +b0OMTsCCE+jxEKauKZs/a+/L7O+zbc7U79Tmvw7i97UQvoFE/JYCsSzrHEURTNNUWcmFQmGi33AW +KZR1u13VI1lOgGVZqNVqWFtbw8rKiurdPRwOMRqNlGDY7/exurqKpaUlAFAlwo+OjtRxZe9qACrL +eDgc4vDwUGUvHx8fT4h9jDEEQYBWq4WDgwP0er1zwvfCwgI+//xz3LlzB6urq1heXoahG9ANXQl4 +ruPCDxIhfWNjA8ViEbZtq97OUizd2trC3NwcLMvCaDjE3t4eXr58qUqsyzFdWVlBo9FQovdVFqTn +eTg6OlJl6WWQgexTvbq6ij/96U+4f/9+Uva6VkMcx6oU9927d5Vg2m631Rhtbm7i8PBQZY4zxmCa +5sQ4y6x+Oc7NZhOe56ljcM4RBAE6nQ729/eV6H5VPM/D4eEhXr16hePjY9XTPFty/g9/+APu37+P +UqmEfD6vBPxSqQTP8zAajbC9va36Tx8eHuLZs2eYnZ3FaDRSgQthGKJYLELXdbV2gaSH9nA4RLfT +VT20W63WueuQpcxbrZYSv5vNJg4PD9FqteB5iUgshfZ6vX6lzHfXdXH79m386U9/wurqKtbW1lAu +l1Wf9IWFBSwuLuL09FQFFci+4sfHx7BtW1VTkAERjDG1xmQ2t+d5aDabql+8PGf54JT3TLfbxf7+ +PlZWVi4MYpDn/MUXX2BlZQVra2uoVqvqQZzP5ZEv5OE4zqUZ3/l8Ho1GAzMzM7BtWz1XZMl5OaYE +QVxfku99DsPQUXA1zJYYolCAnwp4vsBIaAA0ROLDb8AIgiCuOyL2IYIhhN+CGB4AowPoRgTT0OA4 +FmzbVj9SAM9Gg2eNYAllgBMEQRBZGGNgabaQYSSlzhtlhigEtNMYfiAwjGkPTxAEQRAE8X5JK7sF +I4jRCcRwLxG+jQi2baoER2nvW5albP2sHpdtb8b49dmrfXDhW/ZVjuJICd+dTkc5SSzLQrlcTsqc +pwN74XFSEU0KreN+QVz9ZMVw2e97cXERYRgiimI4tg3TMnFycqIydmWv3+FwCMdxlMDa7XaninHy +tVnCMFTlrAeDgcpyBwBd19FoNPDLX/4Sv/zlL5XwPY04jhGGIRqNhirjLcupD4dDlSVsGAZqtRr6 +g/45AVUKfGtra0rku6oDSn7WyckJms0mhsOhKgPuOA5u3LiB3/72t/j0009h27YSr7PzFIYhjo+P +cXR0hHa7PTHO/X5/6jhPE7CHw+G5cY6iCL1eD81mE71e742E79FopPqEN5tNVXVA15Pgg+XlZXzx +xRf47OFnYHzSiWfoBprNpgqEkMEPJycnePXqFe7du6dE+lKpBMY4isUiTNNUgrAsW9/v93HaSsq2 +y4zvKIrU/WAYhpoHWeVAjqcs0y2DERzHQa1WQ71eRy6Xe+0YqDn8zW8x00h6pLuuq9ae7LVdLpdx +cnICIBGHZca5rusQQqjAgcFgANd1oeu6qsogpjTo8jwP/X7/3Fz2+30cHx8ncxmEF57zysoK/st/ ++S9YW1vD6uoqKpXKlec9e5xqtYpqtaruCSl89/t9FWxAzlqCuH5wNt5oaVpSKSTn6KiXNIiIww9C +9IchGGJ4UQyW9goUYlz9/Oyz6fW9BOMPfdkEQRBvidzDnn2OpQ8+IZI/Bz0IvwvhnYAFTfDgBKZp +w7Edtfc/J3rrYwP4bLnz62QEEwRBEB8euYeXPSJ1XUfe0VEvcCC7hxfJHj6gPTxBEARBEMQVuMTm +F/J3COH3gKAH+E3woAk96sC07Ql7X/5kM7+lza9x7VpmewPXQPgGgCAM4HneRDlnXddVH2o5oLp2 +8enato1arYb5+XkUCgUltPm+j1arhW+++QbD4RCNRgOFQgGFQgHFYlGVgZY/8rOjKEK321XibhzH +iONYCd0nJyfwfX9qBrosPS6vKYoiJfiGYaiEeYnMInddF47jXFjOXb5W45oqy7y4uIijo6OJDOPn +z5+jWCxiZWUF/X4fh4eH2N7eRrvdVn3TG40Gbt++jUajcU6cvnSuUiH65OREjYtpmsjlcqhWqyiV +Ssjlkl7WGp+ena9pmhJhoyjC8fGxCgg4O87Z8vIXjbPv+xiNRgjDUPV6DoJA/f1Nrq3f76vscnlt +ruuiUqmgUCgkQvWUHtmGmQjajUYDuVwOmqap85I9rmXv9eSYjuq7XSwWVc9smdUuM8e3trbQbrdV +wEKpVEKhUEC73VZj02w2sb6+ju3tbRwcHCjhW94XMvP+KsL36+B83NdBCv9RdD5oRfadl/9NZqfL +DPqzyHLv8p6RAQvy/WEYIhbxhedkWRby+Twsy5oISHgTTNNEPp+H67rqHOXny/uYIIjri3SaJRnf +BixTR842MXJ9NIoRRBihM/DQG0YYegJ+GCOIBKI4+RGxUI4yIQQERPIdIv9tWoNw6hlOEMTHSHYb +K0VuEQMQYCICRACIECwagkdDaGIIXQtg5Gzk8276k1dVqbICuGGOxW+NU5lzgiAI4vUk7Yq01A9o +wHUsFIMQM8UIcRij0/fRG0UYeDH8IEYYCYS0hycIgiAIgjjPWd15wuaPxzZ/HIBHQ/A4tfmNGKZt +o5DPIZ9LbP1cLqc0Syl+Z6u8SZ3suojdWT648C2EUIKfzHqVQrCMIpDiN3BxxEBW+C4Wi9A0TWWM +yl7C6+vrSjysVCqYnZ3F3Nwc1tbWcPfuXZRKJQCJkBaGoRJ4pSArhTx5vu12e+r5nM08l8K3fK/8 ++8RE6LpyHF2U1S6vn2tcCd83btxQ5b1lGfKXL19iaWkJnudhMBjg8PAQu7u7KjvbsizMzc3hzp07 +qtT5VZHicFb4NgwDuVwOtVoN5XIZOTd3oZgub4R8Pg/bttHr9ZR4O22ch8MhWq3WVDEzO77yffLv +ci29ifAtM5Jl1rbsIS6vrVAoTAi+WUzTVGsqn89D0zTVk7zT6WA4HKqsbdM0IYRQorfruojjWPUo +l9nqOzs7qie8LFEv+1i/evUKg8EAvu+j2WxiY2MDh4eHqsS4FL4dx0G9Xlf94d8V2ac7Ow6yCsFw +OFT3jhS+4zhWgra8Z6aN39mKDWfvmcuCGOSY5nI5WKZ1YTuE1yGDT7LCt5wXKb4TBHF9GWd8y8ou +JlzHQhT5EJEPExFO9RFOMUI78jCIYgyiGEEYQ0QCIoohYiAWArEQECKGiJPNoZDRkCnjxxF5zQiC ++NhIbBfGkPT0kgawiJKfOAAXHljsQUMAznwYGoNtGbDtVPjO5ZHL5ZDP51UkuGVZSvzWjaTyBtf4 +tTWCCYIgiA+PdJbKCiG6rsMyDbiOiTgMEAUeTIRoGUOctNM9fBhjEMdAQHt4giAIgiCI85y1+eMJ +m5/FPljsgQkPHAE4C2AaGmzLgGM7yOdc5PM5JXxPq/Ym29XKPdx15IML38BYXMoKlSpdPlMr/jJM +00ShUECj0cCtW7fw4MEDFAoFdDod1TO83++rHuIyY/bg4ECViN7e3sbt27fxySefYDQaKcFcim6m +aSrHjswWvyw7O0upVML8/LzqBd1sNif++0RJ9tdkRDDG4Lou5ufnsba2hpOTE+zu7ioRdHNzE1tb +W9jZ2cHR0RFOT09VeXXHcVAsFjEzM4OFhQWUy+VLhfZpc+X7vhIChRAqC1gufE3XXjtfUqyUAnF2 +nKUAaVkWCoUCSqXSlbPSC4UCFhYWsLS0hLm5OdWr+SrIIIyz1yarD8gb+qLrMYzEIWgYBhhj54Rb +Wc1AzrXMki8Wi6pH+2AwwP7+Pjjn2NvbQ7PZVFn6pVIJS0tLuHPnjsrkD8MQp6enePXqlSq1Lntr +c87hui7q9TpmZmZUyfJ3Jdu3QY5b9lpldrQMisjOpexLfxVkcIfsK+44Dk5PTy8cf03TwLW3f9DK +Y+i6PnFtUtiX80cQxPWDcQYWM5VhaBgGTN1AYBpwbBthLgCTGYwihMYCWHoMi0fwghB+ECEII8RR +jDB9fsVxDMHixIEGMVE3UWT+nyAI4uODgU04/wXAYjAWg7EITATgWgiDC+iaDsvUYaflzqTgnc/n +zxnCpmnCNEzo+rjHt6Zpqsw5ieAEQRDEWaR/RNf1ceXHIEBoByiEVrqHj4AoBEcAS4thaRFG/uQe +PkpbKMo9vBACsfyOU9nf4/8nCIIgCIL45+Vim5+zEIyH4CyGwRl0rsO2k0B2WdUtl8uhUCioP9t2 +0vIsK3zrmj7Z4oxfL3v/WgjfUlzKlgDPCsFXwdCTrONGo4GHDx8iCAL8+OOPePHiBXZ2djAYDFRp +bCGE6pF8eHiInZ0dPH/+HLOzs/jjH/+IcrmM0XCkxHiJbdsoFouo1Wq4ffs27t+/j0KhcKXzy+Vy +KhO91WphfX39ncYsn89jcXER7XYbGxsb0DQN/X4frVYLjDFsbm5iY2MDu7u7KgNYliMolUqo1+uY +nZ1NsjLeQPg+m5mbnSvp4HqjuY/P92Z3HAeFQgHVahW3b9/GvXv3UC6Xr3Q813VRLBZRrVaxvLz8 +xmLv2XUI4EoBGOMSu+PXSUH47PHk6/P5PGZmZlCpVFTv616vh729PcRxjIODA7RaLZXNLPuy37t3 +D9vb2zBNE8PhEKenp9jc3ESn00G320Wn01Gl4h3HQaPRwMzMzHspdS7P/bK1kRWJpeBdLpfxySef +4O7du6jValf6HCn2l8tl3Lhx48r32rtcV7YXJTBeD9PmkCCI64XK+NY1aJEGwzRgRdZkFYk0fkXX +ANvw4BgxPC+E58fw/QBhGCEIQ8RS/I6iNHNEpFGSKWJK0UQhyI1GEMS1gwFpqPfZf2fpfxRg6Uvk +b841aEyDoXMYugbTNFSEd9YQluK367qTpc5lxveZfRVBEARBnOVsxrdMqpCVAOM4RhQnu2xdB/qm +B8eMMfIieF4MPwgRBiHt4QmCIAiC+Jdkms0//jeh/s5Yau8zgLM0cUhPfizLVMHurusqW1+2OHMc +B5adEb11PfG/poHunF0/u/9aCN/vA03X4OgONE3D3Tt34bruuCczY2i1WrBtG0EQKCd4t9uF53ng +nMO2bWxubqJer+PTTz8F5xxBEEwOlq7DdV3UajXcv38f/+2//TfMzs5e6fxkhnOv18O333575Uzx +i3BdF3NzcxiNRqjVajBNE61WC91uF1EUYXNzE8+ePcPx8bHKAJYZt7VaDbVaTfU0f1POZvwC46zf +txEHz94UnHM4joNKpYK7d+/iT3/6E5aWlt5onA3DUOXU3+e1XXZ9014z7XgAwBlXwnetVsPh4SHi +OEa/38fOzg7iOEaz2USr1cLs7CyKxSIWFhawsrKCO3fu4LvvvlPrqd1uY2dnB8PhUP2EYaiyrWu1 +Gur1OhzHeav5Pju+0+bs7PVKdF2H4zhK+P7jH/+ItbW1K3+WLKsuH7I/NW+7hgmC+PBwNo4yNAxj +ImglG7yiccDUGRyTYWDGGI1ieF4Mz+MIgghBIBCGcbpXGL8fgDrGtOcEPTsIgriuTNu3Zf9Nig7Z +QM5sexvTNJWwLTO8swJ4VviW+/BsFZ3raAQTBEEQHx75HRGzWH1vGIYxkRgh0TiDoQG2weAY6R7e +jOB5IYIACAKBKBJJ5jft4QmCIAiC+BfiMptf/pY2v7T3szZ/tnWZtPmlrS9/W5Z1zubPZntfdB4f +imshfEtxTJa+FkIgDENwzs/1an7d4GlcQ7FUxKJYBOcc1WoV9+7dQ6fTQbvdRrfbRa/XQ6fTwe7u +Lg4ODjAajRBFETzPw/HxMTY2NpDL5RAEAXK5nCpdHUURhsMhut0uhBDI5/NKPL4sCza7kR6NRu8l +88GyLFQqFXiep/pP93o91d96b28PX331lepZzTlHoVDA4uIilpaWUCwW3+pzNU1T2R4yqED2d263 +20kv6zC6+vH05Hi5XA6WZamAg9FohG63iziOkcvlUC6XLy2POE1wftPsc03T4DgO8vm8Ohff91Wf +brlOphGGIQaDAVqtlnqdfHhk13YWmcVdrVbhOA6EEOh0Otje3lbZ30AS5NBoNLC0tKRKuFcqFbiu +i9PTU4xGIxwfH6sS7VEUgTGmSlIUi0Xkcrkrl4t/WzRNU9nppmmCc656fMs+5bZtv/FcAvhZsoXC +IFQtDmQ/bzmHb1oZgSCInxfGGAQEGGPQdR1CiNRxZk48U7J9wKWY43keRqOReobKai9Zhxs5zQiC ++Ji5ihGczbiTBqxhGKrsrKwcJUVuWd5c/j1p9zNuDaRKnKdlzi86D4IgCOJfG8YZuEi+e+I4Tvp8 +WxaASd+A/H4yTVPt4eUP7eEJgiAIgvhX5SqB7sCk8C33VdLml6K2ZVkTdn72z5ZlJUK5biibP9u2 ++brZ+x9cyWGMwTAM1UNbioNBEChRVQrfrxs8IQS4xlEul1EoFDA3N4cHDx7A83z0el10u10cHx9j +f38f29vb+Otf/6pKoMsN88nJCV69eoVarQYhBFzXVcK3FGR7vd6EIHuZKCczzsIwRBiGygn0rpim +iUqlAsaYEr6bzSZGo5HqEy2vrd1uQ9M0FAoF3LhxA0tLSyiVSm/1uZqmqcgPwzAmhOpOp4PBYIAo +fgPhWxsL32fHWWavu66LSqXyRuMchqGKHr4qnPOJa2OMIY5jeJ6HbrebiPqXCN+yd/xwOEQcx+rh +IYMEskI84wyFQgEzMzOo1+uwLEsJ357nQQihypXL7P7l5WUsLi5ifn4elUoFjuOAc47hcKiCHmSm +t3ww5fN5FItF5PP5n/zhkx0/GRSRncswDN9qLoMgeOMghrchCJNzlYYzgInIp4v6uxMEcT1gnEGD +pkRv6SjjfFLUkQ4zuaELggCe5ynhW/6czRgHJh1j5DgjCOJj4KpGcFb4zvZazZadlQGd0uiVgncS +9W3BNA1lOKtjpdne180IJgiCIK4PsnKT3MNnye7h5feRZVnwfR+e5yEIgisL32f/fNm/EQRBEARB +fAy8qfB9tspb1uaXQe/TbH7pE9Azrc1koPt1tPc/uPANALqmAyZUVmVWAPU8D4PBAN1uVzmqp4lg +MrNZZuZ6ngfHcVCtVlEsFuHYNqrVKirlCsrlMvL5PHZ3d7GxsaEEYikcep4HxhJhMp/PI5/Pq+hT +3/fR6/Wws7ODR48eIY5j1Ot1VCoVaFwD18ZC8GAwwNHREZrNJsIwnOjJfJGAelWkM8lOr2t+fh6d +TgfNZlMJ0NJpL8u5l0olrKysYHl5+a37JctMcyEECoUCNE1DFEVK3Nzb28OzZ89gmibK5TKKxSJE +2o+pP0iE4U6nM1GW2/d9zM3NoVAoQNd1lbXe7/exu7uLx48fQ9M0zMzMoFqtqnGWPduHwyGOjo5w +fHyMIAhUBPDs7Czm5ubg6Fcr8W1ZlhpL2Y89CAKVzX5wcIBXr16hUCiiUMijUCjA930EQYDj42Ns +bm7i+fPnar51XVfidqlUgqFPlrfP5/OYnZ2dKEOeFV2jKALnSUn0hYUFLC0toVqtqt7X9XodR0dH +6Ha7KjteCKEic+Tat237ZxGODcNAqVTCwsICisWiKlEWBAH6/T729vbw+PFjOI6Den0GMzP1c3Mp +s9ePj48xGo1U6YzZxixm567WVuBtGY1GaLVaaLVa8H0fQggV6OE4zju3JyAI4qeFMQZwgMWTzjEA +ME3rXLa33NTJ57h0msm9QFb4npYlQk4zgiA+Bt5E+D6b9S2zt+V+TO6x5c/Y+DWV6J3N9uZsXPKM +IAiCIKYxbQ9/luzeXn7/yD18tvId7eEJgiAIgvhX421tftWre4rNny1rLn+kL/VsoPt15YML34wx +cI1Dh64G13Ec5YT2fR+dTgenp6coFopqYM/Sbnfw4sULvHz5Eqenp2i1WqjX67hz5w5u3bqlBGxd +12FaSRm+RqOBYrGohK5sT2/HcTAzM6OEXiniyVLlL1++xJdffol+v49PP/1UZYab3FSZ40eHR/j+ +h+/x+PFjxHGMarUK0zSxv79/rn/426JpGqrVKlZWVtButzEYDAAkAqrc7IdhqPosr66u4saNG2+d +8S0ziWW2u67rSrTknGNrawvfffcdGGO4ffs2XNdVBsfp6SmeP3+OjY0NNJtNnJycoFAoYG1tDTdu +3FDjLMV6xhhevXqFv/zlLxgOh3jw4IHKxjZgqHE+Pj7GDz/8gB9//BFBEKBaraJer+P+/fsoFUtX +7m1t2zbq9TqGwyHK5TIMw1DCtxACOzs7ePLkCVzXxY0bN5BzcxOC/8uXL/HkyRNVdtx1XZTLZSVY +G+ZYOJWlyLPCN2MMQggVFCGrHORyOczPz2NhYQGFQgGMMRSLRczNzeHo6EhFNkt0XUc+n0ej0UCp +VPrZSnSbpolarYaVlZXketNKDlLM39rawt/+9jeEYYgHDx6gUMgnD00k49xqtdBsNvHDDz/g8ePH +6Pf7qFarqNVquH//PgrFtwvWuCrD4RDNZlOVj5djKUt6yCx2giCuH/L5Kf+cdZqNBe9x/2/5E4bh +1ExvWWlG/r6sPKKEHGYEQVxXLtu/XGQEnxW/s89O0zRh6AYMM/mzDFyWr1eZ3tc08psgCIK4HmT3 +8BpP9u5n/RcycFV+15imqfyF0hcihW8petMeniAIgiCIfyXexObP9ua+zOaXtr60/2Wmd7bEubT5 +r6Pd/8GFbyDdyOqaEr1t2wYAVbpIZtsyxuDm3KmZl/1+TwmTu7u72N/fR6PRUOWpa7UaZuoz0A0d +YZBk1J7tYycnWGbUNhoN5HI51Ot15PN5dLtdVep8d3cX33zzjSrJni251Ol0cHBwgO3tbfz973/H +t99+C845lpeXUa1W0Wq1VFbvu8I5V2KjLOMuhFBGgIzekBnYy8vLSkB9G0zTRKFQQBzHqNVqqtl9 +VgD+/vvvAUCVG5clpra3t/GPf/wDT58+xdHREU5OTlTp9aWlJdTrdRQKBbiuO3G87777To1XtrRC +r9fDwcEB9vb21DhHUYSlpSWsrq6iXq/DD/w3urZqtQohhJpzeW2e56mMZcMwVGBBq9VCu93Gs2fP +8OLFC2xtbSkRxbZtzMzMYG1tTQVRZNd8LpcDZ8n8ycAJaaTJ1xiGgUKhgIWFBczPz6NQKIAzrjKr +9/f30Wq1Jow1XdeVMF4ul3/y3t4SwzCU0D8zM6N6i3uep+bq+++/V3Mp+7pbloXhcIj9/X0cHh7i +66+/xjfffIPRaITFxUXcuHEDtVoNN2/e/EnPv9/v4+joCEdHRxgOh+qaZK/07PwRBHE9kRs4WWlE +ijjZjV0URap/jczyzjrLwjBU31vSYXYVp9lV/jtBEMTPzesM0LNGcFYAz/btksJ2thSa/HvWWM5m +ep/tI04QBEEQ05BZ3xrG4nd2D58VvrNlzeXv1+3hARK/CYIgCIL45+Qqorf8c9beZ4ypPVfWtpc2 +f7b12USQ+0cS6H4thG+J4zio1Wqo1+sq63I0GuHg4AAvXrxQ2c1SGJ9GHMfodrvY39/H8fExms0m +Hj16pDK+pdN7MBjg6dOn2N3dVVnSSQnmuspAbsw0YJgGVlZWcP/+fei6rkqXdzodbG5uYjQaYXd3 +F3/961/VYpBCX7vdxt7eHvb391XJ6/eNpmlKFNzf38eTJ08m/rvsny2zoMvlMlzXfessYJkBa5om +lpeXcefOHQBQGdzNZhM//vgjms0mHj9+jJmZGWV8dDodHB4eotlsThghMtBgZWUF9+7dA2MMh4eH +ODw8RLfbxdbWFkajEfb29vDVV1+pm8/3fVXmW46z67rwPO+tx9JxHJRKJSwtLeH27duIogjNZhPH +x8c4PDzEN998g93dXfz9739HtVrFcDjEaDTC/v4+Xr58qT5blhpfXV3F559/jsXFxXOZ57quw3Zs +JayWSiX4vj+xHuXYlMtllIolWJYFxpOM78XFRezu7mJnZ+fccbPC98+V8S37tRcLRSwvL+PevXsI +w1AJyd1uFzs7O/A8D0dHR/j222/VwzMIAvR6PXS7XRW4YhgGarXaz3LuANBqtbC1tYXt7W10u101 +BzMzM5idnUU+n//ZzoUgiDdnIuubMzDBVIUYXdfV5i4MQ9WmQ7bXkM6ys329Xyd6k5OMIIiPkbPG +6bSs74my51yDpksRXIOuaxNitxTHGWPQuEaiN0EQBHFlrrqH1/j4u0fXdZU0kM30zu7lAZwTvyW0 +hycIgiAI4p+Vi0qfZ230bKW3s0Hv0s7XNV35AbJZ4h9LoPu1Er5t20atVsPs7CyCIFD9ug8ODrC+ +vo5qtYpbt25deoyswDocDvHq1auJ6AT5miiK4HkeRqMRGGOwLAuu66LRaODmzZtYWVlFoZAIXaur +q/jss89UqfPDw0Ocnp6i1+vh8PAQjx8/nohIleVRZQRqEASq5PP7RgYD3Lx5E69evUIul5v475Zl +KdG9VquhVCohl8u99YKU4nChUMCNGzdw//59eJ4H3/dVtmyn08GrV6+UQC2NimzPpWKxiHK5DCDJ +tJbC92effaZKp+/v70+M89OnTyf6BmbHWZarnZmZUWWq35Ss8+7GjRu4d+8eRqMRgiBQQnyr1cKz +Z8/OGVuyLL/necjn82otr62t4fPPP1fZ49M+L5fLoVgsolAooNfrwfd9NS65XA6lUgnVahWlUkn1 +SSyVSlhcXMT29va5OZfC9/z8PEql0s+W8c05T9ZbMVkbDx8+VGtjd3dXleI/OjrC8+fPL53LIAhQ +qVRU5vXPQbvdxvb2Nvb29tDv9wEkwnej0Uj6vheKP9u5EATxdmS/22Tm99metZqmTZRBzArd2bKI +lznMyFlGEMTHzuvEb2DcU/Xsc/Tsn9V//0gMYIIgCOJ6cZU9fBRF0GJtYs8+rbQ57eEJgiAIgiCm +2/zTKr5lbfxppdCz/+1jsvmvlfDtui7m5uawvLyMbreLw8NDJYCur69jeXkZo9EIcRyfqx0vMzMX +FxdxcHCAo6MjnJ6eKuHN87yJDC4gERbz+TyKxSLq9ToWFxdx9+5dzM3NoVhMShsLIbC4uIjPP/9c +TbxlWRgMBhgMBvA8D8PhUImV2VJMKgO2WMTCwgKWlpawsLCA7e1t1Xv0XWGMwXVd1Ot1NBoNVCoV +VCoV+L6P4XCo+i7fuHEDMzMzcBxnao/0N/k8KXAuLi7iF7/4xUTP1H6/j+FwiCAIMBqNJkq6y94A +tm2jUqlgcXERy8vLqrf3wsICHj58CCBxtJmmiX6/j8FgoK7n7DjLKBQp8M/NzWFpaUmVlTeNq4u+ +ck2ZhomFhYWJa+OcYzAYqHMYDAaqlDsw7n1eqVTQaDQwNzeHhw8f4vbt26hWq0rknvZ5Mqu7VqtB +CKGCMWTAQqVSgeu60PTk/UIIFArJmpqdnVV9z6Wxp+u6EsYrlcrPJnzLtaHrOubm5vDgwQMIIVRU +tpxLGXAiA0HkOMhgAllOf25uDouLi6pH+k9Ralz2k/d9H8fHxzg4OMDJyYnqMZ/P5zE/P4/FxUUU +SyR8E8THgAymyZZMlM4z+ZySTjIZwCQdY2f7AQK4MOObHGcEQXysvC4CPPt3zji4xs/1AM/+GcBH +ZQATBEEQ14/3sYcHcOFvCe3hCYIgCIL4Z+eyQPezv2Ww+7Tqb2eD3Kcd47pyrYRvKTKtrq5id3cX +mqZhNBrh6OgItm3j/v37GI1GSaRnOiHZ9y4uLiKOYyVO7u7u4vT0FJ1OB57nwfM85ZzRNE1lHS8v +L+P+/fu4f/8+PvnkE9RqNViWpUpELywswHEcJZI3Gg1sbW1hZ2cHrVZLZYfKTbhsAC/LXctM9Xv3 +7mF5eRkvXrx4r+WnbduGruuoVCqo1WqoVCro9XoIwxCWZWFmZga3bt3CzMzMexNBNU1T5btltnKl +UsH29jZ2d3fR6XRUf3UpUMsxLJfLuHnzJu7cuYO7d++iVqtB0zTMz83Dtmw1zrOzs9jc3MT29vbU +cZaBBbZtq3FeW1vDvXv3cOfOHczNzcG03vx6ucaxsLAA27ZRKBSQy+WQz+exs7ODvb09tZ6iKFI9 +DnK5HCqVCur1Ou7du4cHDx7g7t27WFtbU6+56GFgWRbK5TJmZ2cxGo3QarWgaZrql12r1c7NW1KN +YB5zc3MoFAowDEOV8JXC98LCAirlys9W6lzCGMPs7CxM00SxWESpVMLMzAw2Nzexs7ODk5MT9Pv9 +iQhtwzBg27bqs16tVrGysoIHDx7g/v376h583wRBgH6/j263i+PjYxwfH6PT6SAIAmiapp4rS0tL +Scb9NX+gEwSRMOE4A5TzjDOOKI7GvcAzmSHSiUZOMoIg/tW4aunzi6LDzx6D9ksEQRDE23Du+4P2 +8ARBEARBEO/Mm9j8E6J42srs7Hs+Bq6V8O26Lubn59Fut/HkyRNYlqV6ZXPOcXBwgGaziUqlAsdx +JjJAHcdRYpssdV2pVLC3t4ejoyP0+330+32VLW4YBmZmZtBoNHD//n38+te/xi9/+UvVa1mWRQeA +er2Oer2OYrGohO/Hjx/DcRwcHByg0+mg2+0qEc80Tbiui0KhgPn5eSwtLWFtbQ33799Ho9HA0tIS +Go0G4jhGEAQwTROVSkUJ2HIxXQXGmMqkrtfrWFpawsrKCk5OTmDbNubm5rC2toZPPvkEs7Ozr82a +ZYypnuBS4GWMoVwuw3Eclf3MOcfMzAxmZmZQLBZh2zbK5TIeP34M27ZxdHSEbreLfr+vRN9CoYBa +rYaZmRncvXsXDx48wNraGqrVGjjnqNaqqNaqKJaKSvh+9OgRXNdVgnOn05kQS13XRT6fx8LCAhYX +F5XwfevWLRQKhbcS+hlj6try+YL6nGfPnsGyLBwfH6vMdpmlXC6XVVb/F198gd/85je4ceOGGrPL +sG0bjUYDa2trCIIAnudB13XcuHEjM2/2xPnl3Bwsy8L8fCJ+z87OIgxDRFGkMs7n5+ZRLBUv/HzL +slQZdVlevFqtolAoQDf0c2K9vG+y5dfn5+dRq9VQLpeRz+fV+pKVB8ql5N9nZmbw6NEj5HI5VfZc +zqXss+u6LnK5HBYWFrCwsIC1tTXcvXsXt27eQrFUhGVZSkyX5xyGoTpn0zRVkMVV8X0fzWZT9YiX +pfUty0I+n0elUsHc3Bzm5ubOlZQnCOJ6c/b5JYQA0xgYZxBx4gCLRQwAF2Z5ZyGnGUEQ/6xclAGe +/bMqf86SfdbHavwSBEEQ15/sdwvt4QmCIAiCIN6Nt7X5P2Z7/9oJ33Nzc/B9H4uLi6hWq6rvb7/f +x9bWFr799luEYYiVlRXMz8+r92qaBtd1IYTAzZs34boubt26hVarNZHxLYVvmfFdKpUwPz+P5eVl +lIol2I59YRlyKcwbhoFSqYTV1VV0Oh0MBgNVgl2WmrYsS4nBlUoFMzMzmJ2dRT5fwO9+9zu4rotW +q6VEPykCl8tl2LZ91SGboNFo4He/+x3K5bIS+kulEm7evIm1tTXMzMy8Vvg2TROfffYZgiDA0dGR +KlV+//593LlzB7Va7VzmbS6Xw/LyMkzTVKJ2u91W5axlxIhtj7O5pZhYq9WQz08Kio7jYG5uDoZh +oFgsYnV1Fa1WC8PhEMPhUI2zLDufHed6vY7Z2VmUiiWYlgmNv1tJ+Xw+h5WVFViWhYWFBdy7dw/d +bleVx5Yl7V3XRblcRrVaxfLyMur1ZKyvUtK+XCrj/v37cBwHd+7cwdHRETRNU4L26urquTFinEGH +joWFBfzpT39CuVxWAQGNRgO/+MUvUCqXkmAKbfpt3mg08Ic//AH5fF7dZ/V6HZ999lkq+ufPnf/C +wgK++OILzMzM4NNPP8Xh4aFa24uLi1hYWJh4ve3YmJ2dha7rKBQKWFlZwenpqSobf3YuZfZ7uVwe +z2XmOmZmZvCHP/wBruuqc67Vanj48CEajQby+fwbZbh3Ol08fvwYX3/9NZ4/f47BYKAqNcgWAbVa +Da7rvjaAgSCI683Z8olAkkECACIWyoGWhRxlBEH8qzHNoD0rdp993cdoBBMEQRAfB7SHJwiCIAiC +eH9cZPOfTcb9mEVvAGDiGu0I4zhGGIY4PDzEf//v/x3/43/8D7x69QqdTgdRFOHhw4f49a9/jV/9 +6lf49a9/jU8//VS9N9vbJwxD+L6PKIwQRkkWbBwLCDHu7y17CmuaBsMwkmxSwwTj7MKMUZmhHUUR +fN9Xf46iGHEcjQeVMTDGoWlc9TfWdV1lSw+HQ9UjWpJzc3BcR53T2/Thln2wk97aEeI4EdUt04Lt +2OocLlusQgj0ej0MBgMEQaDGy3EcJf6dPb8ojBCEAYIgUJnDUpTMLi/ONei6pvp3G4aRXC/XVP/q +7DqQx/J9f+o4y2NyPu4PrWlacmzdUFEp73JzZq8t8AMEYYAwjM6tJSmAy17npmkm2ftX+PwojOD5 +nhLTgyAAYxyGkfS7lmszO0ZyrnzfR7/XR3/QV/8mKw44jjPRo2HaepEBEvL+kSX6HceBxjXV01Ei +A0h831f3mbxWea7Z4Iq3mUtN06BrOnQjWa/yvmSMqdLk2XOWGfkXnfNla/3x48f4n//zf+J//a// +he3tbezv70PTNKyuruL27dv44x//iP/4j//AJ5988s5riSCI68lFZREJgiCIhI/d4CUIgiD++aA9 +PEEQBEEQxPvhn9Hmv1YZ31IQLRaLWFlZwaeffgohBF6+fInj42McHR3h0aNHcBwH9XodjUZDCbLZ +ZuuGYfwk/YA550rUc133rY9jmiZKpdJ7Pz9Z8vxdji1LkhcKhSu/R9MT4fptM9XPItfB++pH/i68 +72u76DNc3X3jNcUYU0JztVZ948+VY1ypVK78nrPC9ut433P5Nuc8jZPmCY6bx/jhhx/w7NkzbG5u +otfrAQDy+TyWl5fxy1/+EmtraygWi28ViEIQxMfBuX7gBEEQxDnoGUkQBEFcJ+h7iSAIgiAI4v3w +z7ivulbCt8QwDKysrOC3v/0toihCq9XC0dERTk9PEQSBKjm+uLiIen1GZWwTBEFchf2DfTx69Ahf +f/011tfX0Ww2IYQA5xylUglra2v47W9/i4WFBertTRD/AvwzbvAIgiAIgiAI4p8d2scTBEEQBEEQ +Z7mWwreu61hcXIRhGGi321hfX8f29jY8z0O320U+n8ezZ88wPz+PIAjg2LYqXU6ZmQRBTCNbdn1r +awv/+Mc/8I9//APb29toNpuoVCooFAqYnZ3FrVu38PDhQ7iuS0E1BEEQBEEQBEEQBEEQBEEQBEEQ +HwHXUvjWNA25XA71+gxWV1fx6aefYjAYYH9/X2V+f/vttxiNRvjVr34FEQssLC6iVCoin89/6NMn +COIa4nkejo+P0Ww28ejRI3z//fdYX19Hu90GYwwzMzO4c+eOKnHuOI7qaU8QBEEQBEEQBEEQBEEQ +BEEQBEFcb66l8M05R87NwbZtrK2todlsIggCfPPNNzg5OcHJyQlGoxEODw8hhEC1WoVhGjAMnYRv +giCmIp8ZL1++xOPHj/Hjjz9ia2sLURSBc47Z2Vn85je/wb/927/h5s2bcB0XXONUOo0gCIIgCIIg +CIIgCIIgCIIgCOIj4FoK3wCg6Ro0aKjX6/jkk0/AGIPruqjVahgMBtB1HblcDrdu3UKtVoPrujAM +40OfNkEQ1xT5zKhUKrh9+zZ6vR4WFxeVsP3rX/8an3/+OVZXV1GpVKDplOlNEARBEARBEARBEARB +EARBEATxscCEEOJDn8RlDAYDnJ620G630Gw2VfY35xyGYWB+bh6zc7MoFotwHIf68RIEMRXf99Hv +99Hr9XBwcIDDw0P0ej0lfM/Pz2NhYQGVSgW2bcNxnA99ygRBEARBEARBEARBEARBEARBEMQVufbC +N0EQBEEQBEEQBEEQBEEQBEEQBEEQBEFcBv/QJ0AQBEEQBEEQBEEQBEEQBEEQBEEQBEEQ7wIJ3wRB +EARBEARBEARBEARBEARBEARBEMRHDQnfBEEQBEEQBEEQBEEQBEEQBEEQBEEQxEcNCd8EQRAEQRAE +QRAEQRAEQRAEQRAEQRDERw0J3wRBEARBEARBEARBEARBEARBEARBEMRHzf8HYUAUlGQscmUAAACK +ZVhJZk1NACoAAAAIAAQBGgAFAAAAAQAAAD4BGwAFAAAAAQAAAEYBKAADAAAAAQACAACHaQAEAAAA +AQAAAE4AAAAAAAAAkAAAAAEAAACQAAAAAQADkoYABwAAABIAAAB4oAIABAAAAAEAAAe+oAMABAAA +AAEAAAV4AAAAAEFTQ0lJAAAAU2NyZWVuc2hvdKvE71EAAAAldEVYdGRhdGU6Y3JlYXRlADIwMjEt +MTEtMzBUMDY6MTM6MDQrMDA6MDCEuH2lAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDIxLTExLTMwVDA2 +OjEzOjA1KzAwOjAwU5LOrQAAABJ0RVh0ZXhpZjpFeGlmT2Zmc2V0ADc4ydR7JwAAABl0RVh0ZXhp +ZjpQaXhlbFhEaW1lbnNpb24AMTk4Mu6YIC8AAAAZdEVYdGV4aWY6UGl4ZWxZRGltZW5zaW9uADE0 +MDB5bIOwAAAAXHRFWHRleGlmOlVzZXJDb21tZW50ADY1LCA4MywgNjcsIDczLCA3MywgMCwgMCwg +MCwgODMsIDk5LCAxMTQsIDEwMSwgMTAxLCAxMTAsIDExNSwgMTA0LCAxMTEsIDExNkC4H3IAAAAo +dEVYdGljYzpjb3B5cmlnaHQAQ29weXJpZ2h0IEFwcGxlIEluYy4sIDIwMjF9ve4mAAAAGXRFWHRp +Y2M6ZGVzY3JpcHRpb24ATEcgSERSIDRLavavUwAAABh0RVh0eG1wOlBpeGVsWERpbWVuc2lvbgAx +OTgyfYvQ/wAAABh0RVh0eG1wOlBpeGVsWURpbWVuc2lvbgAxNDAw6n9zYAAAABp0RVh0eG1wOlVz +ZXJDb21tZW50AFNjcmVlbnNob3TT4LJ8AAAAAElFTkSuQmCC" /> +</svg> diff --git a/content/en/blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md b/content/en/blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md new file mode 100644 index 0000000000..12eaaf3ff4 --- /dev/null +++ b/content/en/blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md @@ -0,0 +1,201 @@ +--- +layout: blog +title: "Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm" +slug: kubernetes-in-kubernetes-and-pxe-bootable-server-farm +date: 2021-12-22 +--- + +**Author**: Andrei Kvapil (WEDOS) + + +When you own two data centers, thousands of physical servers, virtual machines and hosting for hundreds of thousands sites, Kubernetes can actually simplify the management of all these things. As practice has shown, by using Kubernetes, you can declaratively describe and manage not only applications, but also the infrastructure itself. I work for the largest Czech hosting provider **WEDOS Internet a.s** and today I'll show you two of my projects — [Kubernetes-in-Kubernetes](https://github.com/kvaps/kubernetes-in-kubernetes) and [Kubefarm](https://github.com/kvaps/kubefarm). + +With their help you can deploy a fully working Kubernetes cluster inside another Kubernetes using Helm in just a couple of commands. How and why? + +Let me introduce you to how our infrastructure works. All our physical servers can be divided into two groups: **control-plane** and **compute** nodes. Control plane nodes are usually set up manually, have a stable OS installed, and designed to run all cluster services including Kubernetes control-plane. The main task of these nodes is to ensure the smooth operation of the cluster itself. Compute nodes do not have any operating system installed by default, instead they are booting the OS image over the network directly from the control plane nodes. Their work is to carry out the workload. + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme01.svg" alt="Kubernetes cluster layout" >}} + +Once nodes have downloaded their image, they can continue to work without keeping connection to the PXE server. That is, a PXE server is just keeping rootfs image and does not hold any other complex logic. After our nodes have booted, we can safely restart the PXE server, nothing critical will happen to them. + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme02.svg" alt="Kubernetes cluster after bootstrapping" >}} + +After booting, the first thing our nodes do is join to the existing Kubernetes cluster, namely, execute the **kubeadm join** command so that kube-scheduler could schedule some pods on them and launch various workloads afterwards. From the beginning we used the scheme when nodes were joined into the same cluster used for the control-plane nodes. + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme03.svg" alt="Kubernetes scheduling containers to the compute nodes" >}} + +This scheme worked stably for over two years. However later we decided to add containerized Kubernetes to it. And now we can spawn new Kubernetes-clusters very easily right on our control-plane nodes which are now member special admin-clusters. Now, compute nodes can be joined directly to their own clusters - depending on the configuration. + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme04.svg" alt="Multiple clusters are running in single Kubernetes, compute nodes joined to them" >}} + +## Kubefarm + +This project came with the goal of enabling anyone to deploy such an infrastructure in just a couple of commands using Helm and get about the same in the end. + +At this time, we moved away from the idea of a monocluster. Because it turned out to be not very convenient for managing work of several development teams in the same cluster. The fact is that Kubernetes was never designed as a multi-tenant solution and at the moment it does not provide sufficient means of isolation between projects. Therefore, running separate clusters for each team turned out to be a good idea. However, there should not be too many clusters, to let them be convenient to manage. Nor is it too small to have sufficient independence between development teams. + +The scalability of our clusters became noticeably better after that change. The more clusters you have per number of nodes, the smaller the failure domain and the more stable they work. And as a bonus, we got a fully declaratively described infrastructure. Thus, now you can deploy a new Kubernetes cluster in the same way as deploying any other application in Kubernetes. + +It uses [Kubernetes-in-Kubernetes](http://github.com/kvaps/kubernetes-in-kubernetes) as a basis, [LTSP](https://github.com/ltsp/ltsp/) as PXE-server from which the nodes are booted, and automates the DHCP server configuration using [dnsmasq-controller](https://github.com/kvaps/dnsmasq-controller): + + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/kubefarm.png" alt="Kubefarm" >}} + +## How it works + +Now let's see how it works. In general, if you look at Kubernetes as from an application perspective, you can note that it follows all the principles of [The Twelve-Factor App](https://12factor.net/), and is actually written very well. Thus, it means running Kubernetes as an app in a different Kubernetes shouldn't be a big deal. + +### Running Kubernetes in Kubernetes + +Now let's take a look at the [Kubernetes-in-Kubernetes](https://github.com/kvaps/kubernetes-in-kubernetes) project, which provides a ready-made Helm chart for running Kubernetes in Kubernetes. + +Here is the parameters that you can pass to Helm in the values file: + +* [**kubernetes/values.yaml**](https://github.com/kvaps/kubernetes-in-kubernetes/tree/v0.13.1/deploy/helm/kubernetes) + +<img alt="Kubernetes is just five binaries" style="float: right; max-height: 280px;" src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/5binaries.png"> + +Beside **persistence** (storage parameters for the cluster), the Kubernetes control-plane components are described here: namely: **etcd cluster**, **apiserver**, **controller-manager** and **scheduler**. These are pretty much standard Kubernetes components. There is a light-hearted saying that “Kubernetes is just five binaries”. So here is where the configuration for these binaries is located. + +If you ever tried to bootstrap a cluster using kubeadm, then this config will remind you it's configuration. But in addition to Kubernetes entities, you also have an admin container. In fact, it is a container which holds two binaries inside: **kubectl** and **kubeadm**. They are used to generate kubeconfig for the above components and to perform the initial configuration for the cluster. Also, in an emergency, you can always exec into it to check and manage your cluster. + +After the release [has been deployed](https://asciinema.org/a/407280), you can see a list of pods: **admin-container**, **apiserver** in two replicas, **controller-manager**, **etcd-cluster**, **scheduller** and the initial job that initializes the cluster. In the end you have a command, which allows you to get shell into the admin container, you can use it to see what is happening inside: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot01.svg)](https://asciinema.org/a/407280?autoplay=1) + +Also, let's take look at the certificates. If you've ever installed Kubernetes, then you know that it has a _scary_ directory `/etc/kubernetes/pki` with a bunch of some certificates. In case of Kubernetes-in-Kubernetes, you have fully automated management of them with cert-manager. Thus, it is enough to pass all certificates parameters to Helm during installation, and all the certificates will automatically be generated for your cluster. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot02.svg)](https://asciinema.org/a/407280?t=15&autoplay=1) + +Looking at one of the certificates, eg. apiserver, you can see that it has a list of DNS names and IP addresses. If you want to make this cluster accessible outside, then just describe the additional DNS names in the values file and update the release. This will update the certificate resource, and cert-manager will regenerate the certificate. You'll no longer need to think about this. If kubeadm certificates need to be renewed at least once a year, here the cert-manager will take care and automatically renew them. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot03.svg)](https://asciinema.org/a/407280?t=25&autoplay=1) + +Now let's log into the admin container and look at the cluster and nodes. Of course, there are no nodes, yet, because at the moment you have deployed just the blank control-plane for Kubernetes. But in kube-system namespace you can see some coredns pods waiting for scheduling and configmaps already appeared. That is, you can conclude that the cluster is working: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot04.svg)](https://asciinema.org/a/407280?t=30&autoplay=1) + +Here is the [diagram of the deployed cluster](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_kink_network.html). You can see services for all Kubernetes components: **apiserver**, **controller-manager**, **etcd-cluster** and **scheduler**. And the pods on right side to which they forward traffic. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/argocd01.png)](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_kink_network.html) + +*By the way, the diagram, is drawn in [ArgoCD](https://argoproj.github.io/argo-cd/) — the GitOps tool we use to manage our clusters, and cool diagrams are one of its features.* + +### Orchestrating physical servers + +OK, now you can see the way how is our Kubernetes control-plane deployed, but what about worker nodes, how are we adding them? As I already said, all our servers are bare metal. We do not use virtualization to run Kubernetes, but we orchestrate all physical servers by ourselves. + +Also, we do use Linux network boot feature very actively. Moreover, this is exactly the booting, not some kind of automation of the installation. When the nodes are booting, they just run a ready-made system image for them. That is, to update any node, we just need to reboot it - and it will download a new image. It is very easy, simple and convenient. + +For this, the [Kubefarm](https://github.com/kvaps/kubefarm) project was created, which allows you to automate this. The most commonly used examples can be found in the [examples](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples) directory. The most standard of them named [generic](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/generic). Let's take a look at values.yaml: + +* [**generic/values.yaml**](https://github.com/kvaps/kubefarm/blob/v0.13.1/examples/generic/values.yaml) + +Here you can specify the parameters which are passed into the upstream Kubernetes-in-Kubernetes chart. In order for you control-plane to be accessible from the outside, it is enough to specify the IP address here, but if you wish, you can specify some DNS name here. + +In the PXE server configuration you can specify a timezone. You can also add an SSH key for logging in without a password (but you can also specify a password), as well as kernel modules and parameters that should be applied during booting the system. + +Next comes the **nodePools** configuration, i.e. the nodes themselves. If you've ever used a terraform module for gke, then this logic will remind you of it. Here you statically describe all nodes with a set of parameters: + +- **Name** (hostname); + +- **MAC-addresses** — we have nodes with two network cards, and each one can boot from any of the MAC addresses specified here. + +- **IP-address**, which the DHCP server should issue to this node. + + +In this example, you have two pools: the first has five nodes, the second has only one, the second pool has also two tags assigned. Tags are the way to describe configuration for specific nodes. For example, you can add specific DHCP options for some pools, options for the PXE server for booting (e.g. here is debug option enabled) and set of **kubernetesLabels** and **kubernetesTaints** options. What does that mean? + +For example, in this configuration you have a second nodePool with one node. The pool has **debug** and **foo** tags assigned. Now see the options for **foo** tag in **kubernetesLabels**. This means that the m1c43 node will boot with these two labels and taint assigned. Everything seems to be simple. Now [let's try](https://asciinema.org/a/407282) this in practice. + +### Demo + +Go to [examples](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples) and update previously deployed chart to Kubefarm. Just use the [generic](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/generic) parameters and look at the pods. You can see that a PXE server and one more job were added. This job essentially goes to the deployed Kubernetes cluster and creates a new token. Now it will run repeatedly every 12 hours to generate a new token, so that the nodes can connect to your cluster. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot05.svg)](https://asciinema.org/a/407282?autoplay=1) + +In a [graphical representation](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_Applications_kubefarm-network.html), it looks about the same, but now apiserver started to be exposed outside. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/argocd02.png)](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_Applications_kubefarm-network.html) + +In the diagram, the IP is highlighted in green, the PXE server can be reached through it. At the moment, Kubernetes does not allow creating a single LoadBalancer service for TCP and UDP protocols by default, so you have to create two different services with the same IP address. One is for TFTP, and the second for HTTP, through which the system image is downloaded. + +But this simple example is not always enough, sometimes you might need to modify the logic at boot. For example, here is a directory [advanced_network](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/advanced_network), inside which there is a [values file](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/advanced_network) with a simple shell script. Let's call it `network.sh`: + +* [**network.sh**](https://github.com/kvaps/kubefarm/blob/v0.13.1/examples/advanced_network/values.yaml#L14-L78) + +All this script does is take environment variables at boot time, and generates a network configuration based on them. It creates a directory and puts the netplan config inside. For example, a bonding interface is created here. Basically, this script can contain everything you need. It can hold the network configuration or generate the system services, add some hooks or describe any other logic. Anything that can be described in bash or shell languages will work here, and it will be executed at boot time. + +Let's see how it can be [deployed](https://asciinema.org/a/407284). Let's pass the generic values file as the first parameter, and an additional values file as the second parameter. This is a standard Helm feature. This way you can also pass the secrets, but in this case, the configuration is just expanded by the second file: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot06.svg)](https://asciinema.org/a/407284?autoplay=1) + +Let's look at the configmap **foo-kubernetes-ltsp** for the netboot server and make sure that `network.sh` script is really there. These commands used to configure the network at boot time: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot07.svg)](https://asciinema.org/a/407284?t=15&autoplay=1) + +[Here](https://asciinema.org/a/407286) you can see how it works in principle. The chassis interface (we use HPE Moonshots 1500) have the nodes, you can enter `show node list` command to get a list of all the nodes. Now you can see the booting process. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot08.svg)](https://asciinema.org/a/407286?autoplay=1) + +You can also get their MAC addresses by `show node macaddr all` command. We have a clever operator that collects MAC-addresses from chassis automatically and passes them to the DHCP server. Actually, it's just creating custom configuration resources for dnsmasq-controller which is running in same admin Kubernetes cluster. Also, trough this interface you can control the nodes themselves, e.g. turn them on and off. + +If you have no such opportunity to enter the chassis through iLO and collect a list of MAC addresses for your nodes, you can consider using [catchall cluster](https://asciinema.org/a/407287) pattern. Purely speaking, it is just a cluster with a dynamic DHCP pool. Thus, all nodes that are not described in the configuration to other clusters will automatically join to this cluster. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot09.svg)](https://asciinema.org/a/407287?autoplay=1) + +For example, you can see a special cluster with some nodes. They are joined to the cluster with an auto-generated name based on their MAC address. Starting from this point you can connect to them and see what happens there. Here you can somehow prepare them, for example, set up the file system and then rejoin them to another cluster. + +Now let's try connecting to the node terminal and see how it is booting. After the BIOS, the network card is configured, here it sends a request to the DHCP server from a specific MAC address, which redirects it to a specific PXE server. Later the kernel and initrd image are downloaded from the server using the standard HTTP protocol: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot10.svg)](https://asciinema.org/a/407286?t=28&autoplay=1) + +After loading the kernel, the node downloads the rootfs image and transfers control to systemd. Then the booting proceeds as usual, and after that the node joins Kubernetes: + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot11.svg)](https://asciinema.org/a/407286?t=80&autoplay=1) + +If you take a look at **fstab**, you can see only two entries there: **/var/lib/docker** and **/var/lib/kubelet**, they are mounted as **tmpfs** (in fact, from RAM). At the same time, the root partition is mounted as **overlayfs**, so all changes that you make here on the system will be lost on the next reboot. + +Looking into the block devices on the node, you can see some nvme disk, but it has not yet been mounted anywhere. There is also a loop device - this is the exact rootfs image downloaded from the server. At the moment it is located in RAM, occupies 653 MB and mounted with the **loop** option. + +If you look in **/etc/ltsp**, you find the `network.sh` file that was executed at boot. From containers, you can see running `kube-proxy` and `pause` container for it. + +[![](/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/screenshot12.svg)](https://asciinema.org/a/407286?t=100&autoplay=1) + +## Details + +### Network Boot Image + +But where does the main image come from? There is a little trick here. The image for the nodes is built through the [Dockerfile](https://github.com/kvaps/kubefarm/tree/v0.13.1/build/ltsp) along with the server. The [Docker multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) feature allows you to easily add any packages and kernel modules exactly at the stage of the image build. It looks like this: + +* [**Dockerfile**](https://github.com/kvaps/kubefarm/blob/v0.13.1/build/ltsp/Dockerfile) + +What's going on here? First, we take a regular Ubuntu 20.04 and install all the packages we need. First of all we install the **kernel**, **lvm**, **systemd**, **ssh**. In general, everything that you want to see on the final node should be described here. Here we also install `docker` with `kubelet` and `kubeadm`, which are used to join the node to the cluster. + +And then we perform an additional configuration. In the last stage, we simply install `tftp` and `nginx` (which serves our image to clients), **grub** (bootloader). Then root of the previous stages copied into the final image and generate squashed image from it. That is, in fact, we get a docker image, which has both the server and the boot image for our nodes. At the same time, it can be easily updated by changing the Dockerfile. + +### Webhooks and API aggregation layer + +I want to pay special attention to the problem of webhooks and aggregation layer. In general, webhooks is a Kubernetes feature that allows you to respond to the creation or modification of any resources. Thus, you can add a handler so that when resources are applied, Kubernetes must send request to some pod and check if configuration of this resource is correct, or make additional changes to it. + +But the point is, in order for the webhooks to work, the apiserver must have direct access to the cluster for which it is running. And if it is started in a separate cluster, like our case, or even separately from any cluster, then Konnectivity service can help us here. Konnectivity is one of the optional but officially supported Kubernetes components. + +Let's take cluster of four nodes for example, each of them is running a `kubelet` and we have other Kubernetes components running outside: `kube-apiserver`, `kube-scheduler` and `kube-controller-manager`. By default, all these components interact with the apiserver directly - this is the most known part of the Kubernetes logic. But in fact, there is also a reverse connection. For example, when you want to view the logs or run a `kubectl exec command`, the API server establishes a connection to the specific kubelet independently: + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/konnectivity01.svg" alt="Kubernetes apiserver reaching kubelet" >}} + +But the problem is that if we have a webhook, then it usually runs as a standard pod with a service in our cluster. And when apiserver tries to reach it, it will fail because it will try to access an in-cluster service named **webhook.namespace.svc** being outside of the cluster where it is actually running: + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/konnectivity02.svg" alt="Kubernetes apiserver can't reach webhook" >}} + +And here Konnectivity comes to our rescue. Konnectivity is a tricky proxy server developed especially for Kubernetes. It can be deployed as a server next to the apiserver. And Konnectivity-agent is deployed in several replicas directly in the cluster you want to access. The agent establishes a connection to the server and sets up a stable channel to make apiserver able to access all webhooks and all kubelets in the cluster. Thus, now all communication with the cluster will take place through the Konnectivity-server: + +{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/konnectivity03.svg" alt="Kubernetes apiserver reaching webhook via konnectivity" >}} + +## Our plans + +Of course, we are not going to stop at this stage. People interested in the project often write to me. And if there will be a sufficient number of interested people, I hope to move Kubernetes-in-Kubernetes project under [Kubernetes SIGs](https://github.com/kubernetes-sigs), by representing it in form of the official Kubernetes Helm chart. Perhaps, by making this project independent we'll gather an even larger community. + +I am also thinking of integrating it with the Machine Controller Manager, which would allow creating worker nodes, not only of physical servers, but also, for example, for creating virtual machines using kubevirt and running them in the same Kubernetes cluster. By the way, it also allows to spawn virtual machines in the clouds, and have a control-plane deployed locally. + +I am also considering the option of integrating with the Cluster-API so that you can create physical Kubefarm clusters directly through the Kubernetes environment. But at the moment I'm not completely sure about this idea. If you have any thoughts on this matter, I'll be happy to listen to them. diff --git a/content/en/blog/_posts/2022-01-07-kubernetes-is-moving-on-from-dockershim.md b/content/en/blog/_posts/2022-01-07-kubernetes-is-moving-on-from-dockershim.md new file mode 100644 index 0000000000..9e7672f2b8 --- /dev/null +++ b/content/en/blog/_posts/2022-01-07-kubernetes-is-moving-on-from-dockershim.md @@ -0,0 +1,103 @@ +--- +layout: blog +title: "Kubernetes is Moving on From Dockershim: Commitments and Next Steps" +date: 2022-01-07 +slug: kubernetes-is-moving-on-from-dockershim +--- + +**Authors:** Sergey Kanzhelev (Google), Jim Angel (Google), Davanum Srinivas (VMware), Shannon Kularathna (Google), Chris Short (AWS), Dawn Chen (Google) + +Kubernetes is removing dockershim in the upcoming v1.24 release. We're excited +to reaffirm our community values by supporting open source container runtimes, +enabling a smaller kubelet, and increasing engineering velocity for teams using +Kubernetes. If you [use Docker Engine as a container runtime](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/) +for your Kubernetes cluster, get ready to migrate in 1.24! To check if you're +affected, refer to [Check whether dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/). + +## Why we’re moving away from dockershim + +Docker was the first container runtime used by Kubernetes. This is one of the +reasons why Docker is so familiar to many Kubernetes users and enthusiasts. +Docker support was hardcoded into Kubernetes – a component the project refers to +as dockershim. +As containerization became an industry standard, the Kubernetes project added support +for additional runtimes. This culminated in the implementation of the +container runtime interface (CRI), letting system components (like the kubelet) +talk to container runtimes in a standardized way. As a result, dockershim became +an anomaly in the Kubernetes project. +Dependencies on Docker and dockershim have crept into various tools +and projects in the CNCF ecosystem ecosystem, resulting in fragile code. + +By removing the +dockershim CRI, we're embracing the first value of CNCF: "[Fast is better than +slow](https://github.com/cncf/foundation/blob/master/charter.md#3-values)". +Stay tuned for future communications on the topic! + +## Deprecation timeline + +We [formally announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/) the dockershim deprecation in December 2020. Full removal is targeted +in Kubernetes 1.24, in April 2022. This timeline +aligns with our [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior), +which states that deprecated behaviors must function for at least 1 year +after their announced deprecation. + +We'll support Kubernetes version 1.23, which includes +dockershim, for another year in the Kubernetes project. For managed +Kubernetes providers, vendor support is likely to last even longer, but this is +dependent on the companies themselves. Regardless, we're confident all cluster operations will have +time to migrate. If you have more questions about the dockershim removal, refer +to the [Dockershim Deprecation FAQ](/dockershim). + +We asked you whether you feel prepared for the migration from dockershim in this +survey: [Are you ready for Dockershim removal](/blog/2021/11/12/are-you-ready-for-dockershim-removal/). +We had over 600 responses. To everybody who took time filling out the survey, +thank you. + +The results show that we still have a lot of ground to cover to help you to +migrate smoothly. Other container runtimes exist, and have been promoted +extensively. However, many users told us they still rely on dockershim, +and sometimes have dependencies that need to be re-worked. Some of these +dependencies are outside of your control. Based on your feedback, here are some +of the steps we are taking to help. + +## Our next steps + +Based on the feedback you provided: + +- CNCF and the 1.24 release team are committed to delivering documentation in + time for the 1.24 release. This includes more informative blog posts like this + one, updating existing code samples, tutorials, and tasks, and producing a + migration guide for cluster operators. +- We are reaching out to the rest of the CNCF community to help prepare them for + this change. + +If you're part of a project with dependencies on dockershim, or if you're +interested in helping with the migration effort, please join us! There's always +room for more contributors, whether to our transition tools or to our +documentation. To get started, say hello in the +[#sig-node](https://kubernetes.slack.com/archives/C0BP8PW9G) +channel on [Kubernetes Slack](https://slack.kubernetes.io/)! + +## Final thoughts + +As a project, we've already seen cluster operators increasingly adopt other +container runtimes through 2021. +We believe there are no major blockers to migration. The steps we're taking to +improve the migration experience will light the path more clearly for you. + +We understand that migration from dockershim is yet another action you may need to +do to keep your Kubernetes infrastructure up to date. For most of you, this step +will be straightforward and transparent. In some cases, you will encounter +hiccups or issues. The community has discussed at length whether postponing the +dockershim removal would be helpful. For example, we recently talked about it in +the [SIG Node discussion on November 11th](https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#bookmark=id.r77y11bgzid) +and in the [Kubernetes Steering committee meeting held on December 6th](https://docs.google.com/document/d/1qazwMIHGeF3iUh5xMJIJ6PDr-S3bNkT8tNLRkSiOkOU/edit#bookmark=id.m0ir406av7jx). +We already [postponed](https://github.com/kubernetes/enhancements/pull/2481/) it +once in 2021 because the adoption rate of other +runtimes was lower than we wanted, which also gave us more time to identify +potential blocking issues. + +At this point, we believe that the value that you (and Kubernetes) gain from +dockershim removal makes up for the migration effort you'll have. Start planning +now to avoid surprises. We'll have more updates and guides before Kubernetes +1.24 is released. diff --git a/content/en/blog/_posts/2022-01-10-meet-our-contributors-APAC-India-region-01.md b/content/en/blog/_posts/2022-01-10-meet-our-contributors-APAC-India-region-01.md new file mode 100644 index 0000000000..a1ac132c48 --- /dev/null +++ b/content/en/blog/_posts/2022-01-10-meet-our-contributors-APAC-India-region-01.md @@ -0,0 +1,105 @@ +--- +layout: blog +title: "Meet Our Contributors - APAC (India region)" +date: 2022-01-10 +slug: meet-our-contributors-india-ep-01 +canonicalUrl: https://www.kubernetes.dev/blog/2022/01/10/meet-our-contributors-india-ep-01/ +--- + +**Authors & Interviewers:** [Anubhav Vardhan](https://github.com/anubha-v-ardhan), [Atharva Shinde](https://github.com/Atharva-Shinde), [Avinesh Tripathi](https://github.com/AvineshTripathi), [Debabrata Panigrahi](https://github.com/Debanitrkl), [Kunal Verma](https://github.com/verma-kunal), [Pranshu Srivastava](https://github.com/PranshuSrivastava), [Pritish Samal](https://github.com/CIPHERTron), [Purneswar Prasad](https://github.com/PurneswarPrasad), [Vedant Kakde](https://github.com/vedant-kakde) + +**Editor:** [Priyanka Saggu](https://psaggu.com) + +--- + +Good day, everyone 👋 + +Welcome to the first episode of the APAC edition of the "Meet Our Contributors" blog post series. + + +In this post, we'll introduce you to five amazing folks from the India region who have been actively contributing to the upstream Kubernetes projects in a variety of ways, as well as being the leaders or maintainers of numerous community initiatives. + +💫 *Let's get started, so without further ado…* + + +## [Arsh Sharma](https://github.com/RinkiyaKeDad) + +Arsh is currently employed with Okteto as a Developer Experience engineer. As a new contributor, he realised that 1:1 mentorship opportunities were quite beneficial in getting him started with the upstream project. + +He is presently a CI Signal shadow on the Kubernetes 1.23 release team. He is also contributing to the SIG Testing and SIG Docs projects, as well as to the [cert-manager](https://github.com/cert-manager/infrastructure) tools development work that is being done under the aegis of SIG Architecture. + +To the newcomers, Arsh helps plan their early contributions sustainably. + +> _I would encourage folks to contribute in a way that's sustainable. What I mean by that +> is that it's easy to be very enthusiastic early on and take up more stuff than one can +> actually handle. This can often lead to burnout in later stages. It's much more sustainable +> to work on things iteratively._ + +## [Kunal Kushwaha](https://github.com/kunal-kushwaha) + +Kunal Kushwaha is a core member of the Kubernetes marketing council. He is also a CNCF ambassador and one of the founders of the [CNCF Students Program](https://community.cncf.io/cloud-native-students/).. He also served as a Communications role shadow during the 1.22 release cycle. + +At the end of his first year, Kunal began contributing to the [fabric8io kubernetes-client](https://github.com/fabric8io/kubernetes-client) project. He was then selected to work on the same project as part of Google Summer of Code. Kunal mentored people on the same project, first through Google Summer of Code then through Google Code-in. + +As an open-source enthusiast, he believes that diverse participation in the community is beneficial since it introduces new perspectives and opinions and respect for one's peers. He has worked on various open-source projects, and his participation in communities has considerably assisted his development as a developer. + + +> _I believe if you find yourself in a place where you do not know much about the +> project, that's a good thing because now you can learn while contributing and the +> community is there to help you. It has helped me a lot in gaining skills, meeting +> people from around the world and also helping them. You can learn on the go, +> you don't have to be an expert. Make sure to also check out no code contributions +> because being a beginner is a skill and you can bring new perspectives to the +> organisation._ + +## [Madhav Jivarajani](https://github.com/MadhavJivrajani) + + +Madhav Jivarajani works on the VMware Upstream Kubernetes stability team. He began contributing to the Kubernetes project in January 2021 and has since made significant contributions to several areas of work under SIG Architecture, SIG API Machinery, and SIG ContribEx (contributor experience). + +Among several significant contributions are his recent efforts toward the Archival of [design proposals](https://github.com/kubernetes/community/issues/6055), refactoring the ["groups" codebase](https://github.com/kubernetes/k8s.io/pull/2713) under k8s-infra repository to make it mockable and testable, and improving the functionality of the [GitHub k8s bot](https://github.com/kubernetes/test-infra/issues/23129). + +In addition to his technical efforts, Madhav oversees many projects aimed at assisting new contributors. He organises bi-weekly "KEP reading club" sessions to help newcomers understand the process of adding new features, deprecating old ones, and making other key changes to the upstream project. He has also worked on developing [Katacoda scenarios](https://github.com/kubernetes-sigs/contributor-katacoda) to assist new contributors to become acquainted with the process of contributing to k/k. In addition to his current efforts to meet with community members every week, he has organised several [new contributors workshops (NCW)](https://www.youtube.com/watch?v=FgsXbHBRYIc). + +> _I initially did not know much about Kubernetes. I joined because the community was +> super friendly. But what made me stay was not just the people, but the project itself. +> My solution to not feeling overwhelmed in the community was to gain as much context +> and knowledge into the topics that I was interested in and were being discussed. And +> as a result I continued to dig deeper into Kubernetes and the design of it. +> I am a systems nut & thus Kubernetes was an absolute goldmine for me._ + + +## [Rajas Kakodkar](https://github.com/rajaskakodkar) + +Rajas Kakodkar currently works at VMware as a Member of Technical Staff. He has been engaged in many aspects of the upstream Kubernetes project since 2019. + +He is now a key contributor to the Testing special interest group. He is also active in the SIG Network community. Lately, Rajas has contributed significantly to the [NetworkPolicy++](https://docs.google.com/document/d/1AtWQy2fNa4qXRag9cCp5_HsefD7bxKe3ea2RPn8jnSs/) and [`kpng`](https://github.com/kubernetes-sigs/kpng) sub-projects. + +One of the first challenges he ran across was that he was in a different time zone than the upstream project's regular meeting hours. However, async interactions on community forums progressively corrected that problem. + +> _I enjoy contributing to Kubernetes not just because I get to work on +> cutting edge tech but more importantly because I get to work with +> awesome people and help in solving real world problems._ + +## [Rajula Vineet Reddy](https://github.com/rajula96reddy) + +Rajula Vineet Reddy, a Junior Engineer at CERN, is a member of the Marketing Council team under SIG ContribEx . He also served as a release shadow for SIG Release during the 1.22 and 1.23 Kubernetes release cycles. + +He started looking at the Kubernetes project as part of a university project with the help of one of his professors. Over time, he spent a significant amount of time reading the project's documentation, Slack discussions, GitHub issues, and blogs, which helped him better grasp the Kubernetes project and piqued his interest in contributing upstream. One of his key contributions was his assistance with automation in the SIG ContribEx Upstream Marketing subproject. + +According to Rajula, attending project meetings and shadowing various project roles are vital for learning about the community. + +> _I find the community very helpful and it's always_ +> “you get back as much as you contribute”. +> _The more involved you are, the more you will understand, get to learn and +> contribute new things._ +> +> _The first step to_ “come forward and start” _is hard. But it's all gonna be +> smooth after that. Just take that jump._ + +--- + +If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. We're thrilled to have other folks assisting us in reaching out to even more wonderful individuals of the community. Your suggestions would be much appreciated. + + +We'll see you all in the next one. Everyone, till then, have a happy contributing! 👋 diff --git a/content/en/blog/_posts/2022-01-19-Securing-Admission-Controllers.md b/content/en/blog/_posts/2022-01-19-Securing-Admission-Controllers.md new file mode 100644 index 0000000000..002b2bd531 --- /dev/null +++ b/content/en/blog/_posts/2022-01-19-Securing-Admission-Controllers.md @@ -0,0 +1,44 @@ +--- +layout: blog +title: "Securing Admission Controllers" +date: 2022-01-19 +slug: secure-your-admission-controllers-and-webhooks +--- + +**Author:** Rory McCune (Aqua Security) + +[Admission control](/docs/reference/access-authn-authz/admission-controllers/) is a key part of Kubernetes security, alongside authentication and authorization. Webhook admission controllers are extensively used to help improve the security of Kubernetes clusters in a variety of ways including restricting the privileges of workloads and ensuring that images deployed to the cluster meet organization’s security requirements. + +However, as with any additional component added to a cluster, security risks can present themselves. A security risk example is if the deployment and management of the admission controller are not handled correctly. To help admission controller users and designers manage these risks appropriately, the [security documentation](https://github.com/kubernetes/community/tree/master/sig-security#security-docs) subgroup of SIG Security has spent some time developing a [threat model for admission controllers](https://github.com/kubernetes/sig-security/tree/main/sig-security-docs/papers/admission-control). This threat model looks at likely risks which may arise from the incorrect use of admission controllers, which could allow security policies to be bypassed, or even allow an attacker to get unauthorised access to the cluster. + +From the threat model, we developed a set of security best practices that should be adopted to ensure that cluster operators can get the security benefits of admission controllers whilst avoiding any risks from using them. + +## Admission controllers and good practices for security + +From the threat model, a couple of themes emerged around how to ensure the security of admission controllers. + +### Secure webhook configuration + +It’s important to ensure that any security component in a cluster is well configured and admission controllers are no different here. There are a couple of security best practices to consider when using admission controllers + +* **Correctly configured TLS for all webhook traffic**. Communications between the API server and the admission controller webhook should be authenticated and encrypted to ensure that attackers who may be in a network position to view or modify this traffic cannot do so. To achieve this access the API server and webhook must be using certificates from a trusted certificate authority so that they can validate their mutual identities +* **Only authenticated access allowed**. If an attacker can send an admission controller large numbers of requests, they may be able to overwhelm the service causing it to fail. Ensuring all access requires strong authentication should mitigate that risk. +* **Admission controller fails closed**. This is a security practice that has a tradeoff, so whether a cluster operator wants to configure it will depend on the cluster’s threat model. If an admission controller fails closed, when the API server can’t get a response from it, all deployments will fail. This stops attackers bypassing the admission controller by disabling it, but, can disrupt the cluster’s operation. As clusters can have multiple webhooks, one approach to hit a middle ground might be to have critical controls on a fail closed setups and less critical controls allowed to fail open. +* **Regular reviews of webhook configuration**. Configuration mistakes can lead to security issues, so it’s important that the admission controller webhook configuration is checked to make sure the settings are correct. This kind of review could be done automatically by an Infrastructure As Code scanner or manually by an administrator. + + +### Secure cluster configuration for admission control + +In most cases, the admission controller webhook used by a cluster will be installed as a workload in the cluster. As a result, it’s important to ensure that Kubernetes' security features that could impact its operation are well configured. + +* **Restrict [RBAC](/docs/reference/access-authn-authz/rbac/) rights**. Any user who has rights which would allow them to modify the configuration of the webhook objects or the workload that the admission controller uses could disrupt its operation. So it’s important to make sure that only cluster administrators have those rights. +* **Prevent privileged workloads**. One of the realities of container systems is that if a workload is given certain privileges, it will be possible to break out to the underlying cluster node and impact other containers on that node. Where admission controller services run in the cluster they’re protecting, it’s important to ensure that any requirement for privileged workloads is carefully reviewed and restricted as much as possible. +* **Strictly control external system access**. As a security service in a cluster admission controller systems will have access to sensitive information like credentials. To reduce the risk of this information being sent outside the cluster, [network policies](/docs/concepts/services-networking/network-policies/) should be used to restrict the admission controller services access to external networks. +* **Each cluster has a dedicated webhook**. Whilst it may be possible to have admission controller webhooks that serve multiple clusters, there is a risk when using that model that an attack on the webhook service would have a larger impact where it’s shared. Also where multiple clusters use an admission controller there will be increased complexity and access requirements, making it harder to secure. + + +### Admission controller rules + +A key element of any admission controller used for Kubernetes security is the rulebase it uses. The rules need to be able to accurately meet their goals avoiding false positive and false negative results. + +* **Regularly test and review rules**. Admission controller rules need to be tested to ensure their accuracy. They also need to be regularly reviewed as the Kubernetes API will change with each new version, and rules need to be assessed with each Kubernetes release to understand any changes that may be required to keep them up to date. diff --git a/content/en/blog/_posts/2022-02-07-sig-multicluster-spotlight/index.md b/content/en/blog/_posts/2022-02-07-sig-multicluster-spotlight/index.md new file mode 100644 index 0000000000..4f155ed54d --- /dev/null +++ b/content/en/blog/_posts/2022-02-07-sig-multicluster-spotlight/index.md @@ -0,0 +1,63 @@ +--- +layout: blog +title: "Spotlight on SIG Multicluster" +date: 2022-02-07 +slug: sig-multicluster-spotlight-2022 +canonicalUrl: https://www.kubernetes.dev/blog/2022/02/04/sig-multicluster-spotlight-2022/ +--- + +**Authors:** Dewan Ahmed (Aiven) and Chris Short (AWS) + +## Introduction + +[SIG Multicluster](https://github.com/kubernetes/community/tree/master/sig-multicluster) is the SIG focused on how Kubernetes concepts are expanded and used beyond the cluster boundary. Historically, Kubernetes resources only interacted within that boundary - KRU or Kubernetes Resource Universe (not an actual Kubernetes concept). Kubernetes clusters, even now, don't really know anything about themselves or, about other clusters. Absence of cluster identifiers is a case in point. With the growing adoption of multicloud and multicluster deployments, the work SIG Multicluster doing is gaining a lot of attention. In this blog, [Jeremy Olmsted-Thompson, Google](https://twitter.com/jeremyot) and [Chris Short, AWS](https://twitter.com/ChrisShort) discuss the interesting problems SIG Multicluster is solving and how you can get involved. Their initials **JOT** and **CS** will be used for brevity. + +## A summary of their conversation + +**CS**: How long has the SIG Multicluster existed and how was the SIG in its infancy? How long have you been with this SIG? + +**JOT**: I've been around for almost two years in the SIG Multicluster. All I know about the infancy years is from the lore but even in the early days, it was always about solving this same problem. Early efforts have been things like [KubeFed](https://github.com/kubernetes-sigs/kubefed). I think there are still folks using KubeFed but it's a smaller slice. Back then, I think people out there deploying large numbers of Kubernetes clusters were really not at a point where we had a ton of real concrete use cases. Projects like KubeFed and [Cluster Registry](https://github.com/kubernetes-retired/cluster-registry) were developed around that time and the need back then can be associated to these projects. The motivation for these projects were how do we solve the problems that we think people are **going to have**, when they start expanding to multiple clusters. Honestly, in some ways, it was trying to do too much at that time. + +**CS**: How does KubeFed differ from the current state of SIG Multicluster? How does the **lore** differ from the **now**? + +**JOT**: Yeah, it was like trying to get ahead of potential problems instead of addressing specific problems. I think towards the end of 2019, there was a slow down in SIG multicluster work and we kind of picked it back up with one of the most active recent projects that is the [SIG Multicluster services (MCS)](https://github.com/kubernetes-sigs/mcs-api). + +Now this is the shift to solving real specific problems. For example, + +> I've got workloads that are spread across multiple clusters and I need them to talk to each other. + +Okay, that's very straightforward and we know that we need to solve that. To get started, let's make sure that these projects can work together on a common API so you get the same kind of portability that you get with Kubernetes. + +There's a few implementations of the MCS API out there and more are being developed. But, we didn't build an implementation because depending on how you're deploying things there could be hundreds of implementations. As long as you only need the basic Multicluster service functionality, it'll just work on whatever background you want, whether it's Submariner, GKE, or a service mesh. + +My favorite example of "then vs. now" is cluster ID. A few years ago, there was an effort to define a cluster ID. A lot of really good thought went into this concept, for example, how do we make a cluster ID is unique across multiple clusters. How do we make this ID globally unique so it'll work in every contact? Let's say, there's an acquisition or merger of teams - does the cluster IDs still remain unique for those teams? + +With Multicluster services, we found the need for an actual cluster ID, and it has a very specific need. To address this specific need, we're no longer considering every single Kubernetes cluster out there rather the ClusterSets - a grouping of clusters that work together in some kind of bounds. That's a much narrower scope than considering clusters everywhere in time and space. It also leaves flexibility for an implementer to define the boundary (a ClusterSet) beyond which this cluster ID will no longer be unique. + + +**CS**: How do you feel about the current state of SIG Multicluster versus where you're hoping to be in future? + +**JOT**: There's a few projects that are kind of getting started, for example, Work API. In the future, I think that some common practices around how do we deploy things across clusters are going to develop. +> If I have clusters deployed in a bunch of different regions; what's the best way to actually do that? + +The answer is, almost always, "it depends". Why are you doing this? Is it because there's some kind of compliance that makes you care about locality? Is it performance? Is it availability? + +I think revisiting registry patterns will probably be a natural step after we have cluster IDs, that is, how do you actually associate these clusters together? Maybe you've got a distributed deployment that you run in your own data centers all over the world. I imagine that expanding the API in that space is going to be important as more multi cluster features develop. It really depends on what the community starts doing with these tools. + +**CS**: In the early days of Kubernetes, we used to have a few large Kubernetes clusters and now we're dealing with many small Kubernetes clusters - even multiple clusters for our own dev environments. How has this shift from a few large clusters to many small clusters affected the SIG? Has it accelerated the work or make it challenging in any way? + +**JOT**: I think that it has created a lot of ambiguity that needs solving. Originally, you'd have a dev cluster, a staging cluster, and a prod cluster. When the multi region thing came in, we started needing dev/staging/prod clusters, per region. And then, sometimes clusters really need more isolation due to compliance or some regulations issues. Thus, we're ending up with a lot of clusters. I think figuring out the right balance on how many clusters should you actually have is important. The power of Kubernetes is being able to deploy a lot of things managed by a single control plane. So, it's not like every single workload that gets deployed should be in its own cluster. But I think it's pretty clear that we can't put every single workload in a single cluster. + +**CS**: What are some of your favorite things about this SIG? + +**JOT**: The complexity of the problems, the people and the newness of the space. We don't have right answers and we have to figure this out. At the beginning, we couldn't even think about multi clusters because there was no way to connect services across clusters. Now there is and we're starting to go tackle those problems, I think that this is a really fun place to be in because I expect that the SIG is going to get a lot busier the next couple of years. It's a very collaborative group and we definitely would like more people to come join us, get involved, raise their problems and bring their ideas. + +**CS**: What do you think keeps people in this group? How has the pandemic affected you? + +**JOT**: I think it definitely got a little bit quieter during the pandemic. But for the most part; it's a very distributed group so whether you're calling in to our weekly meetings from a conference room or from your home, it doesn't make that huge of a difference. During the pandemic, a lot of people had time to focus on what's next for their scale and growth. I think that's what keeps people in the group - we have real problems that need to be solved which are very new in this space. And it's fun :) + +## Wrap up + +**CS**: That's all we have for today. Thanks Jeremy for your time. + +**JOT**: Thanks Chris. Everybody is welcome at our [bi-weekly meetings](https://github.com/kubernetes/community/tree/master/sig-multicluster#meetings). We love as many people to come as possible and welcome all questions and all ideas. It's a new space and it'd be great to grow the community. \ No newline at end of file diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md new file mode 100644 index 0000000000..86d31f5443 --- /dev/null +++ b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md @@ -0,0 +1,192 @@ +--- +layout: blog +title: 'SIG Node CI Subproject Celebrates Two Years of Test Improvements' +date: 2022-02-16 +slug: sig-node-ci-subproject-celebrates +canonicalUrl: https://www.kubernetes.dev/blog/2022/02/16/sig-node-ci-subproject-celebrates-two-years-of-test-improvements/ +--- + +**Authors:** Sergey Kanzhelev (Google), Elana Hashman (Red Hat) + +Ensuring the reliability of SIG Node upstream code is a continuous effort +that takes a lot of behind-the-scenes effort from many contributors. +There are frequent releases of Kubernetes, base operating systems, +container runtimes, and test infrastructure that result in a complex matrix that +requires attention and steady investment to "keep the lights on." +In May 2020, the Kubernetes node special interest group ("SIG Node") organized a new +subproject for continuous integration (CI) for node-related code and tests. Since its +inauguration, the SIG Node CI subproject has run a weekly meeting, and even the full hour +is often not enough to complete triage of all bugs, test-related PRs and issues, and discuss all +related ongoing work within the subgroup. + +Over the past two years, we've fixed merge-blocking and release-blocking tests, reducing time to merge Kubernetes contributors' pull requests thanks to reduced test flakes. When we started, Node test jobs only passed 42% of the time, and through our efforts, we now ensure a consistent >90% job pass rate. We've closed 144 test failure issues and merged 176 pull requests just in kubernetes/kubernetes. And we've helped subproject participants ascend the Kubernetes contributor ladder, with 3 new org members, 6 new reviewers, and 2 new approvers. + +The Node CI subproject is an approachable first stop to help new contributors +get started with SIG Node. There is a low barrier to entry for new contributors +to address high-impact bugs and test fixes, although there is a long +road before contributors can climb the entire contributor ladder: +it took over a year to establish two new approvers for the group. +The complexity of all the different components that power Kubernetes nodes +and its test infrastructure requires a sustained investment over a long period +for developers to deeply understand the entire system, +both at high and low levels of detail. + +We have several regular contributors at our meetings, however; our reviewers +and approvers pool is still small. It is our goal to continue to grow +contributors to ensure a sustainable distribution of work +that does not just fall to a few key approvers. + +It's not always obvious how subprojects within SIGs are formed, operate, +and work. Each is unique to its sponsoring SIG and tailored to the projects +that the group is intended to support. As a group that has welcomed many +first-time SIG Node contributors, we'd like to share some of the details and +accomplishments over the past two years, +helping to demystify our inner workings and celebrate the hard work +of all our dedicated contributors! + +## Timeline + +***May 2020.*** SIG Node CI group was formed on May 11, 2020, with more than +[30 volunteers](https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#bookmark=id.vsb8pqnf4gib) +signed up, to improve SIG Node CI signal and overall observability. +Victor Pickard focused on getting +[testgrid jobs](https://testgrid.k8s.io/sig-node) passing +when Ning Liao suggested forming a group around this effort and came up with +the [original group charter document](https://docs.google.com/document/d/1yS-XoUl6GjZdjrwxInEZVHhxxLXlTIX2CeWOARmD8tY/edit#heading=h.te6sgum6s8uf). +The SIG Node chairs sponsored group creation with Victor as a subproject lead. +Sergey Kanzhelev joined Victor shortly after as a co-lead. + +At the kick-off meeting, we discussed which tests to concentrate on fixing first +and discussed merge-blocking and release-blocking tests, many of which were failing due +to infrastructure issues or buggy test code. + +The subproject launched weekly hour-long meetings to discuss ongoing work +discussion and triage. + +***June 2020.*** Morgan Bauer, Karan Goel, and Jorge Alarcon Ochoa were +recognized as reviewers for the SIG Node CI group for their contributions, +helping significantly with the early stages of the subproject. +David Porter and Roy Yang also joined the SIG test failures GitHub team. + +***August 2020.*** All merge-blocking and release-blocking tests were passing, +with some flakes. However, only 42% of all SIG Node test jobs were green, as there +were many flakes and failing tests. + +***October 2020.*** Amim Knabben becomes a Kubernetes org member for his +contributions to the subproject. + +***January 2021.*** With healthy presubmit and critical periodic jobs passing, +the subproject discussed its goal for cleaning up the rest of periodic tests +and ensuring they passed without flakes. + +Elana Hashman joined the subproject, stepping up to help lead it after +Victor's departure. + +***February 2021.*** Artyom Lukianov becomes a Kubernetes org member for his +contributions to the subproject. + +***August 2021.*** After SIG Node successfully ran a [bug scrub](https://groups.google.com/g/kubernetes-dev/c/w2ghO4ihje0/m/VeEql1LJBAAJ) +to clean up its bug backlog, the scope of the meeting was extended to +include bug triage to increase overall reliability, anticipating issues +before they affect the CI signal. + +Subproject leads Elana Hashman and Sergey Kanzhelev are both recognized as +approvers on all node test code, supported by SIG Node and SIG Testing. + +***September 2021.*** After significant deflaking progress with serial tests in +the 1.22 release spearheaded by Francesco Romani, the subproject set a goal +for getting the serial job fully passing by the 1.23 release date. + +Mike Miranda becomes a Kubernetes org member for his contributions +to the subproject. + +***November 2021.*** Throughout 2021, SIG Node had no merge or +release-blocking test failures. Many flaky tests from past releases are removed +from release-blocking dashboards as they had been fully cleaned up. + +Danielle Lancashire was recognized as a reviewer for SIG Node's subgroup, test code. + +The final node serial tests were completely fixed. The serial tests consist of +many disruptive and slow tests which tend to be flakey and are hard +to troubleshoot. By the 1.23 release freeze, the last serial tests were +fixed and the job was passing without flakes. + +[![Slack announcement that Serial tests are green](serial-tests-green.png)](https://kubernetes.slack.com/archives/C0BP8PW9G/p1638211041322900) + +The 1.23 release got a special shout out for the tests quality and CI signal. +The SIG Node CI subproject was proud to have helped contribute to such +a high-quality release, in part due to our efforts in identifying +and fixing flakes in Node and beyond. + +[![Slack shoutout that release was mostly green](release-mostly-green.png)](https://kubernetes.slack.com/archives/C92G08FGD/p1637175755023200) + +***December 2021.*** An estimated 90% of test jobs were passing at the time of +the 1.23 release (up from 42% in August 2020). + +Dockershim code was removed from Kubernetes. This affected nearly half of SIG Node's +test jobs and the SIG Node CI subproject reacted quickly and retargeted all the +tests. SIG Node was the first SIG to complete test migrations off dockershim, +providing examples for other affected SIGs. The vast majority of new jobs passed +at the time of introduction without further fixes required. The [effort of +removing dockershim](https://k8s.io/dockershim)) from Kubernetes is ongoing. +There are still some wrinkles from the dockershim removal as we uncover more +dependencies on dockershim, but we plan to stabilize all test jobs +by the 1.24 release. + +## Statistics + +Our regular meeting attendees and subproject participants for the past few months: + +- Aditi Sharma +- Artyom Lukianov +- Arnaud Meukam +- Danielle Lancashire +- David Porter +- Davanum Srinivas +- Elana Hashman +- Francesco Romani +- Matthias Bertschy +- Mike Miranda +- Paco Xu +- Peter Hunt +- Ruiwen Zhao +- Ryan Phillips +- Sergey Kanzhelev +- Skyler Clark +- Swati Sehgal +- Wenjun Wu + +The [kubernetes/test-infra](https://github.com/kubernetes/test-infra/) source code repository contains test definitions. The number of +Node PRs just in that repository: +- 2020 PRs (since May): [183](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2020-05-01..2020-12-31+-author%3Ak8s-infra-ci-robot+) +- 2021 PRs: [264](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2021-01-01..2021-12-31+-author%3Ak8s-infra-ci-robot+) + +Triaged issues and PRs on CI board (including triaging away from the subgroup scope): + +- 2020 (since May): [132](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2020-05-01..2020-12-31) +- 2021: [532](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2021-01-01..2021-12-31+) + +## Future + +Just "keeping the lights on" is a bold task and we are committed to improving this experience. +We are working to simplify the triage and review processes for SIG Node. + +Specifically, we are working on better test organization, naming, +and tracking: + +- https://github.com/kubernetes/enhancements/pull/3042 +- https://github.com/kubernetes/test-infra/issues/24641 +- [Kubernetes SIG-Node CI Testgrid Tracker](https://docs.google.com/spreadsheets/d/1IwONkeXSc2SG_EQMYGRSkfiSWNk8yWLpVhPm-LOTbGM/edit#gid=0) + +We are also constantly making progress on improved tests debuggability and de-flaking. + +If any of this interests you, we'd love for you to join us! +There's plenty to learn in debugging test failures, and it will help you gain +familiarity with the code that SIG Node maintains. + +You can always find information about the group on the +[SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) page. +We give group updates at our maintainer track sessions, such as +[KubeCon + CloudNativeCon Europe 2021](https://kccnceu2021.sched.com/event/iE8E/kubernetes-sig-node-intro-and-deep-dive-elana-hashman-red-hat-sergey-kanzhelev-google) and +[KubeCon + CloudNative North America 2021](https://kccncna2021.sched.com/event/lV9D/kubenetes-sig-node-intro-and-deep-dive-elana-hashman-derek-carr-red-hat-sergey-kanzhelev-dawn-chen-google?iframe=no&w=100%&sidebar=yes&bg=no). +Join us in our mission to keep the kubelet and other SIG Node components reliable and ensure smooth and uneventful releases! diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png new file mode 100644 index 0000000000..c88eb3b37e Binary files /dev/null and b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png differ diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png new file mode 100644 index 0000000000..4ea64fedb7 Binary files /dev/null and b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png differ diff --git a/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md new file mode 100644 index 0000000000..4950f88097 --- /dev/null +++ b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md @@ -0,0 +1,224 @@ +--- +layout: blog +title: "Updated: Dockershim Removal FAQ" +linkTitle: "Dockershim Removal FAQ" +date: 2022-02-17 +slug: dockershim-faq +aliases: [ '/dockershim' ] +--- + +**This supersedes the original +[Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/) article, +published in late 2020. The article includes updates from the v1.24 +release of Kubernetes.** + +--- + +This document goes over some frequently asked questions regarding the +removal of _dockershim_ from Kubernetes. The removal was originally +[announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/) +as a part of the Kubernetes v1.20 release. The Kubernetes +[v1.24 release](/releases/#release-v1-24) actually removed the dockershim +from Kubernetes. + +For more on what that means, check out the blog post +[Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/). + +To determine the impact that the removal of dockershim would have for you or your organization, +you can read [Check whether dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/). + +In the months and days leading up to the Kubernetes 1.24 release, Kubernetes contributors worked hard to try to make this a smooth transition. + +- A blog post detailing our [commitment and next steps](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/). +- Checking if there were major blockers to migration to [other container runtimes](/docs/setup/production-environment/container-runtimes/#container-runtimes). +- Adding a [migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) guide. +- Creating a list of + [articles on dockershim removal and on using CRI-compatible runtimes](/docs/reference/node/topics-on-dockershim-and-cri-compatible-runtimes/). + That list includes some of the already mentioned docs, and also covers selected external sources + (including vendor guides). + +### Why was the dockershim removed from Kubernetes? + +Early versions of Kubernetes only worked with a specific container runtime: +Docker Engine. Later, Kubernetes added support for working with other container runtimes. +The CRI standard was [created](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) to +enable interoperability between orchestrators (like Kubernetes) and many different container +runtimes. +Docker Engine doesn't implement that interface (CRI), so the Kubernetes project created +special code to help with the transition, and made that _dockershim_ code part of Kubernetes +itself. + +The dockershim code was always intended to be a temporary solution (hence the name: shim). +You can read more about the community discussion and planning in the +[Dockershim Removal Kubernetes Enhancement Proposal][drkep]. +In fact, maintaining dockershim had become a heavy burden on the Kubernetes maintainers. + +Additionally, features that were largely incompatible with the dockershim, such +as cgroups v2 and user namespaces are being implemented in these newer CRI +runtimes. Removing the dockershim from Kubernetes allows further development in those areas. + +[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim + +### Are Docker and containers the same thing? + +Docker popularized the Linux containers pattern and has been instrumental in +developing the underlying technology, however containers in Linux have existed +for a long time. The container ecosystem has grown to be much broader than just +Docker. Standards like OCI and CRI have helped many tools grow and thrive in our +ecosystem, some replacing aspects of Docker while others enhance existing +functionality. + +### Will my existing container images still work? + +Yes, the images produced from `docker build` will work with all CRI implementations. +All your existing images will still work exactly the same. + +#### What about private images? + +Yes. All CRI runtimes support the same pull secrets configuration used in +Kubernetes, either via the PodSpec or ServiceAccount. + +### Can I still use Docker Engine in Kubernetes 1.23? + +Yes, the only thing changed in 1.20 is a single warning log printed at [kubelet] +startup if using Docker Engine as the runtime. You'll see this warning in all versions up to 1.23. The dockershim removal occurred +in Kubernetes 1.24. + +If you're running Kubernetes v1.24 or later, see [Can I still use Docker Engine as my container runtime?](#can-i-still-use-docker-engine-as-my-container-runtime). +(Remember, you can switch away from the dockershim if you're using any supported Kubernetes release; from release v1.24, you +**must** switch as Kubernetes no longer includes the dockershim). + +[kubelet]: /docs/reference/command-line-tools-reference/kubelet/ + +### Which CRI implementation should I use? + +That’s a complex question and it depends on a lot of factors. If Docker Engine is +working for you, moving to containerd should be a relatively easy swap and +will have strictly better performance and less overhead. However, we encourage you +to explore all the options from the [CNCF landscape] in case another would be an +even better fit for your environment. + +[CNCF landscape]: https://landscape.cncf.io/card-mode?category=container-runtime&grouping=category + +#### Can I still use Docker Engine as my container runtime? + +First off, if you use Docker on your own PC to develop or test containers: nothing changes. +You can still use Docker locally no matter what container runtime(s) you use for your +Kubernetes clusters. Containers make this kind of interoperability possible. + +Mirantis and Docker have [committed][mirantis] to maintaining a replacement adapter for +Docker Engine, and to maintain that adapter even after the in-tree dockershim is removed +from Kubernetes. The replacement adapter is named [`cri-dockerd`](https://github.com/Mirantis/cri-dockerd). + +You can install `cri-dockerd` and use it to connect the kubelet to Docker Engine. Read [Migrate Docker Engine nodes from dockershim to cri-dockerd](/docs/tasks/administer-cluster/migrating-from-dockershim/migrate-dockershim-dockerd/) to learn more. + +[mirantis]: https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/ + +### Are there examples of folks using other runtimes in production today? + +All Kubernetes project produced artifacts (Kubernetes binaries) are validated +with each release. + +Additionally, the [kind] project has been using containerd for some time and has +seen an improvement in stability for its use case. Kind and containerd are leveraged +multiple times every day to validate any changes to the Kubernetes codebase. Other +related projects follow a similar pattern as well, demonstrating the stability and +usability of other container runtimes. As an example, OpenShift 4.x has been +using the [CRI-O] runtime in production since June 2019. + +For other examples and references you can look at the adopters of containerd and +CRI-O, two container runtimes under the Cloud Native Computing Foundation ([CNCF]). + +- [containerd](https://github.com/containerd/containerd/blob/master/ADOPTERS.md) +- [CRI-O](https://github.com/cri-o/cri-o/blob/master/ADOPTERS.md) + +[CRI-O]: https://cri-o.io/ +[kind]: https://kind.sigs.k8s.io/ +[CNCF]: https://cncf.io + +### People keep referencing OCI, what is that? + +OCI stands for the [Open Container Initiative], which standardized many of the +interfaces between container tools and technologies. They maintain a standard +specification for packaging container images (OCI image-spec) and running containers +(OCI runtime-spec). They also maintain an actual implementation of the runtime-spec +in the form of [runc], which is the underlying default runtime for both +[containerd] and [CRI-O]. The CRI builds on these low-level specifications to +provide an end-to-end standard for managing containers. + +[Open Container Initiative]: https://opencontainers.org/about/overview/ +[runc]: https://github.com/opencontainers/runc +[containerd]: https://containerd.io/ + +### What should I look out for when changing CRI implementations? + +While the underlying containerization code is the same between Docker and most +CRIs (including containerd), there are a few differences around the edges. Some +common things to consider when migrating are: + +- Logging configuration +- Runtime resource limitations +- Node provisioning scripts that call docker or use Docker Engine via its control socket +- Plugins for `kubectl` that require the `docker` CLI or the Docker Engine control socket +- Tools from the Kubernetes project that require direct access to Docker Engine + (for example: the deprecated `kube-imagepuller` tool) +- Configuration of functionality like `registry-mirrors` and insecure registries +- Other support scripts or daemons that expect Docker Engine to be available and are run + outside of Kubernetes (for example, monitoring or security agents) +- GPUs or special hardware and how they integrate with your runtime and Kubernetes + +If you use Kubernetes resource requests/limits or file-based log collection +DaemonSets then they will continue to work the same, but if you've customized +your `dockerd` configuration, you’ll need to adapt that for your new container +runtime where possible. + +Another thing to look out for is anything expecting to run for system maintenance +or nested inside a container when building images will no longer work. For the +former, you can use the [`crictl`][cr] tool as a drop-in replacement (see +[mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) +and for the latter you can use newer container build options like [img], [buildah], +[kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. + +[cr]: https://github.com/kubernetes-sigs/cri-tools +[img]: https://github.com/genuinetools/img +[buildah]: https://github.com/containers/buildah +[kaniko]: https://github.com/GoogleContainerTools/kaniko +[buildkit-cli-for-kubectl]: https://github.com/vmware-tanzu/buildkit-cli-for-kubectl + +For containerd, you can start with their [documentation] to see what configuration +options are available as you migrate things over. + +[documentation]: https://github.com/containerd/cri/blob/master/docs/registry.md + +For instructions on how to use containerd and CRI-O with Kubernetes, see the +Kubernetes documentation on [Container Runtimes]. + +[Container Runtimes]: /docs/setup/production-environment/container-runtimes/ + +### What if I have more questions? + +If you use a vendor-supported Kubernetes distribution, you can ask them about +upgrade plans for their products. For end-user questions, please post them +to our end user community forum: https://discuss.kubernetes.io/. + +You can discuss the decision to remove dockershim via a dedicated +[GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917). + +You can also check out the excellent blog post +[Wait, Docker is deprecated in Kubernetes now?][dep] a more in-depth technical +discussion of the changes. + +[dep]: https://dev.to/inductor/wait-docker-is-deprecated-in-kubernetes-now-what-do-i-do-e4m + +### Is there any tooling that can help me find dockershim in use? + +Yes! The [Detector for Docker Socket (DDS)][dds] is a kubectl plugin that you can +install and then use to check your cluster. DDS can detect if active Kubernetes workloads +are mounting the Docker Engine socket (`docker.sock`) as a volume. +Find more details and usage patterns in the DDS project's [README][dds]. + +[dds]: https://github.com/aws-containers/kubectl-detector-for-docker-socket + +### Can I have a hug? + +Yes, we're still giving hugs as requested. 🤗🤗🤗 diff --git a/content/en/blog/_posts/2022-03-15-meet-our-contributors-APAC-AU-NZ-region-02.md b/content/en/blog/_posts/2022-03-15-meet-our-contributors-APAC-AU-NZ-region-02.md new file mode 100644 index 0000000000..e8d9cfdf89 --- /dev/null +++ b/content/en/blog/_posts/2022-03-15-meet-our-contributors-APAC-AU-NZ-region-02.md @@ -0,0 +1,72 @@ +--- +layout: blog +title: "Meet Our Contributors - APAC (Aus-NZ region)" +date: 2022-03-16 +slug: meet-our-contributors-au-nz-ep-02 +canonicalUrl: https://www.kubernetes.dev/blog/2022/03/14/meet-our-contributors-au-nz-ep-02/ +--- + +**Authors & Interviewers:** [Anubhav Vardhan](https://github.com/anubha-v-ardhan), [Atharva Shinde](https://github.com/Atharva-Shinde), [Avinesh Tripathi](https://github.com/AvineshTripathi), [Brad McCoy](https://github.com/bradmccoydev), [Debabrata Panigrahi](https://github.com/Debanitrkl), [Jayesh Srivastava](https://github.com/jayesh-srivastava), [Kunal Verma](https://github.com/verma-kunal), [Pranshu Srivastava](https://github.com/PranshuSrivastava), [Priyanka Saggu](github.com/Priyankasaggu11929/), [Purneswar Prasad](https://github.com/PurneswarPrasad), [Vedant Kakde](https://github.com/vedant-kakde) + +--- + +Good day, everyone 👋 + +Welcome back to the second episode of the "Meet Our Contributors" blog post series for APAC. + +This post will feature four outstanding contributors from the Australia and New Zealand regions, who have played diverse leadership and community roles in the Upstream Kubernetes project. + +So, without further ado, let's get straight to the blog. + +## [Caleb Woodbine](https://github.com/BobyMCbobs) + +Caleb Woodbine is currently a member of the ii.nz organisation. + +He began contributing to the Kubernetes project in 2018 as a member of the Kubernetes Conformance working group. His experience was positive, and he benefited from early guidance from [Hippie Hacker](https://github.com/hh), a fellow contributor from New Zealand. + +He has made major contributions to Kubernetes project since then through `SIG k8s-infra` and `k8s-conformance` working group. + +Caleb is also a co-organizer of the [CloudNative NZ](https://www.meetup.com/cloudnative-nz/) community events, which aim to expand the reach of Kubernetes project throughout New Zealand in order to encourage technical education and improved employment opportunities. + +> _There need to be more outreach in APAC and the educators and universities must pick up Kubernetes, as they are very slow and about 8+ years out of date. NZ tends to rather pay overseas than educate locals on the latest cloud tech Locally._ + +## [Dylan Graham](https://github.com/DylanGraham) + +Dylan Graham is a cloud engineer from Adeliade, Australia. He has been contributing to the upstream Kubernetes project since 2018. + +He stated that being a part of such a large-scale project was initially overwhelming, but that the community's friendliness and openness assisted him in getting through it. + +He began by contributing to the project documentation and is now mostly focused on the community support for the APAC region. + +He believes that consistent attendance at community/project meetings, taking on project tasks, and seeking community guidance as needed can help new aspiring developers become effective contributors. + +> _The feeling of being a part of a large community is really special. I've met some amazing people, even some before the pandemic in real life :)_ + +## [Hippie Hacker](https://github.com/hh) + +Hippie has worked for the CNCF.io as a Strategic Initiatives contractor from New Zealand for almost 5+ years. He is an active contributor to k8s-infra, API conformance testing, Cloud provider conformance submissions, and apisnoop.cncf.io domains of the upstream Kubernetes & CNCF projects. + +He recounts their early involvement with the Kubernetes project, which began roughly 5 years ago when their firm, ii.nz, demonstrated [network booting from a Raspberry Pi using PXE and running Gitlab in-cluster to install Kubernetes on servers](https://ii.nz/post/bringing-the-cloud-to-your-community/). + +He describes their own contributing experience as someone who, at first, tried to do all of the hard lifting on their own, but eventually saw the benefit of group contributions which reduced burnout and task division which allowed folks to keep moving forward on their own momentum. + +He recommends that new contributors use pair programming. + +> _The cross pollination of approaches and two pairs of eyes on the same work can often yield a much more amplified effect than a PR comment / approval alone can afford._ + +## [Nick Young](https://github.com/youngnick) + +Nick Young works at VMware as a technical lead for Contour, a CNCF ingress controller. He was active with the upstream Kubernetes project from the beginning, and eventually became the chair of the LTS working group, where he advocated user concerns. He is currently the SIG Network Gateway API subproject's maintainer. + +His contribution path was notable in that he began working on major areas of the Kubernetes project early on, skewing his trajectory. + +He asserts the best thing a new contributor can do is to "start contributing". Naturally, if it is relevant to their employment, that is excellent; however, investing non-work time in contributing can pay off in the long run in terms of work. He believes that new contributors, particularly those who are currently Kubernetes users, should be encouraged to participate in higher-level project discussions. + +> _Just being active and contributing will get you a long way. Once you've been active for a while, you'll find that you're able to answer questions, which will mean you're asked questions, and before you know it you are an expert._ + +--- + +If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. Your suggestions would be much appreciated. We're thrilled to have additional folks assisting us in reaching out to even more wonderful individuals of the community. + + +We'll see you all in the next one. Everyone, till then, have a happy contributing! 👋 diff --git a/content/en/blog/_posts/2022-03-31-ready-for-dockershim-removal.md b/content/en/blog/_posts/2022-03-31-ready-for-dockershim-removal.md new file mode 100644 index 0000000000..4362a0daa7 --- /dev/null +++ b/content/en/blog/_posts/2022-03-31-ready-for-dockershim-removal.md @@ -0,0 +1,34 @@ +--- +layout: blog +title: "Is Your Cluster Ready for v1.24?" +date: 2022-03-31 +slug: ready-for-dockershim-removal +--- + +**Author:** Kat Cosgrove + + +Way back in December of 2020, Kubernetes announced the [deprecation of Dockershim](/blog/2020/12/02/dont-panic-kubernetes-and-docker/). In Kubernetes, dockershim is a software shim that allows you to use the entire Docker engine as your container runtime within Kubernetes. In the upcoming v1.24 release, we are removing Dockershim - the delay between deprecation and removal in line with the [project’s policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) of supporting features for at least one year after deprecation. If you are a cluster operator, this guide includes the practical realities of what you need to know going into this release. Also, what do you need to do to ensure your cluster doesn’t fall over! + +## First, does this even affect you? + +If you are rolling your own cluster or are otherwise unsure whether or not this removal affects you, stay on the safe side and [check to see if you have any dependencies on Docker Engine](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/). Please note that using Docker Desktop to build your application containers is not a Docker dependency for your cluster. Container images created by Docker are compliant with the [Open Container Initiative (OCI)](https://opencontainers.org/), a Linux Foundation governance structure that defines industry standards around container formats and runtimes. They will work just fine on any container runtime supported by Kubernetes. + +If you are using a managed Kubernetes service from a cloud provider, and you haven’t explicitly changed the container runtime, there may be nothing else for you to do. Amazon EKS, Azure AKS, and Google GKE all default to containerd now, though you should make sure they do not need updating if you have any node customizations. To check the runtime of your nodes, follow [Find Out What Container Runtime is Used on a Node](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/). + +Regardless of whether you are rolling your own cluster or using a managed Kubernetes service from a cloud provider, you may need to [migrate telemetry or security agents that rely on Docker Engine](/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents/). + +## I have a Docker dependency. What now? + +If your Kubernetes cluster depends on Docker Engine and you intend to upgrade to Kubernetes v1.24 (which you should eventually do for security and similar reasons), you will need to change your container runtime from Docker Engine to something else or use [cri-dockerd](https://github.com/Mirantis/cri-dockerd). Since [containerd](https://containerd.io/) is a graduated CNCF project and the runtime within Docker itself, it’s a safe bet as an alternative container runtime. Fortunately, the Kubernetes project has already documented the process of [changing a node’s container runtime](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/), using containerd as an example. Instructions are similar for switching to one of the other supported runtimes. + +## I want to upgrade Kubernetes, and I need to maintain compatibility with Docker as a runtime. What are my options? + +Fear not, you aren’t being left out in the cold and you don’t have to take the security risk of staying on an old version of Kubernetes. Mirantis and Docker have jointly released, and are maintaining, a replacement for dockershim. That replacement is called [cri-dockerd](https://github.com/Mirantis/cri-dockerd). If you do need to maintain compatibility with Docker as a runtime, install cri-dockerd following the instructions in the project’s documentation. + +## Is that it? + + +Yes. As long as you go into this release aware of the changes being made and the details of your own clusters, and you make sure to communicate clearly with your development teams, it will be minimally dramatic. You may have some changes to make to your cluster, application code, or scripts, but all of these requirements are documented. Switching from using Docker Engine as your runtime to using [one of the other supported container runtimes](/docs/setup/production-environment/container-runtimes/) effectively means removing the middleman, since the purpose of dockershim is to access the container runtime used by Docker itself. From a practical perspective, this removal is better both for you and for Kubernetes maintainers in the long-run. + +If you still have questions, please first check the [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/). diff --git a/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md b/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md new file mode 100644 index 0000000000..6428e64ad5 --- /dev/null +++ b/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md @@ -0,0 +1,133 @@ +--- +layout: blog +title: "Kubernetes Removals and Deprecations In 1.24" +date: 2022-04-07 +slug: upcoming-changes-in-kubernetes-1-24 +--- + +**Author**: Mickey Boxell (Oracle) + +As Kubernetes evolves, features and APIs are regularly revisited and removed. New features may offer +an alternative or improved approach to solving existing problems, motivating the team to remove the +old approach. + +We want to make sure you are aware of the changes coming in the Kubernetes 1.24 release. The release will +**deprecate** several (beta) APIs in favor of stable versions of the same APIs. The major change coming +in the Kubernetes 1.24 release is the +[removal of Dockershim](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim). +This is discussed below and will be explored in more depth at release time. For an early look at the +changes coming in Kubernetes 1.24, take a look at the in-progress +[CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md). + +## A note about Dockershim + +It's safe to say that the removal receiving the most attention with the release of Kubernetes 1.24 +is Dockershim. Dockershim was deprecated in v1.20. As noted in the [Kubernetes 1.20 changelog](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation): +"Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet +uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance +issues in the Kubernetes community." With the upcoming release of Kubernetes 1.24, the Dockershim will +finally be removed. + +In the article [Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/), +the authors succinctly captured the change's impact and encouraged users to remain calm: +> Docker as an underlying runtime is being deprecated in favor of runtimes that use the +> Container Runtime Interface (CRI) created for Kubernetes. Docker-produced images +> will continue to work in your cluster with all runtimes, as they always have. + +Several guides have been created with helpful information about migrating from dockershim +to container runtimes that are directly compatible with Kubernetes. You can find them on the +[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) +page in the Kubernetes documentation. + +For more information about why Kubernetes is moving away from dockershim, check out the aptly +named: [Kubernetes is Moving on From Dockershim](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/) +and the [updated dockershim removal FAQ](/blog/2022/02/17/dockershim-faq/). + +Take a look at the [Is Your Cluster Ready for v1.24?](/blog/2022/03/31/ready-for-dockershim-removal/) post to learn about how to ensure your cluster continues to work after upgrading from v1.23 to v1.24. + +## The Kubernetes API removal and deprecation process + +Kubernetes contains a large number of components that evolve over time. In some cases, this +evolution results in APIs, flags, or entire features, being removed. To prevent users from facing +breaking changes, Kubernetes contributors adopted a feature [deprecation policy](/docs/reference/using-api/deprecation-policy/). +This policy ensures that stable APIs may only be deprecated when a newer stable version of that +same API is available and that APIs have a minimum lifetime as indicated by the following stability levels: + +* Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes. +* Beta or pre-release API versions must be supported for 3 releases after deprecation. +* Alpha or experimental API versions may be removed in any release without prior deprecation notice. + +Removals follow the same deprecation policy regardless of whether an API is removed due to a beta feature +graduating to stable or because that API was not proven to be successful. Kubernetes will continue to make +sure migration options are documented whenever APIs are removed. + +**Deprecated** APIs are those that have been marked for removal in a future Kubernetes release. **Removed** +APIs are those that are no longer available for use in current, supported Kubernetes versions after having +been deprecated. These removals have been superseded by newer, stable/generally available (GA) APIs. + +## API removals, deprecations, and other changes for Kubernetes 1.24 + +* [Dynamic kubelet configuration](https://github.com/kubernetes/enhancements/issues/281): `DynamicKubeletConfig` is used to enable the dynamic configuration of the kubelet. The `DynamicKubeletConfig` flag was deprecated in Kubernetes 1.22. In v1.24, this feature gate will be removed from the kubelet. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). Refer to the ["Dynamic kubelet config is removed" KEP](https://github.com/kubernetes/enhancements/issues/281) for more information. +* [Dynamic log sanitization](https://github.com/kubernetes/kubernetes/pull/107207): The experimental dynamic log sanitization feature is deprecated and will be removed in v1.24. This feature introduced a logging filter that could be applied to all Kubernetes system components logs to prevent various types of sensitive information from leaking via logs. Refer to [KEP-1753: Kubernetes system components logs sanitization](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#deprecation) for more information and an [alternative approach](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#alternatives=). +* In-tree provisioner to CSI driver migration: This applies to a number of in-tree plugins, including [Portworx](https://github.com/kubernetes/enhancements/issues/2589). Refer to the [In-tree Storage Plugin to CSI Migration Design Doc](https://git.k8s.io/design-proposals-archive/storage/csi-migration.md#background-and-motivations) for more information. +* [Removing Dockershim from kubelet](https://github.com/kubernetes/enhancements/issues/2221): the Container Runtime Interface (CRI) for Docker (i.e. Dockershim) is currently a built-in container runtime in the kubelet code base. It was deprecated in v1.20. As of v1.24, the kubelet will no longer have dockershim. Check out this blog on [what you need to do be ready for v1.24](/blog/2022/03/31/ready-for-dockershim-removal/). +* [Storage capacity tracking for pod scheduling](https://github.com/kubernetes/enhancements/issues/1472): The CSIStorageCapacity API supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding. In v1.24, the CSIStorageCapacity API will be stable. The API graduating to stable initates the deprecation of the v1beta1 CSIStorageCapacity API. Refer to the [Storage Capacity Constraints for Pod Scheduling KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1472-storage-capacity-tracking) for more information. +* [The `master` label is no longer present on kubeadm control plane nodes](https://github.com/kubernetes/kubernetes/pull/107533). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to [KEP-2067: Rename the kubeadm "master" label and taint](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint). +* [VolumeSnapshot v1beta1 CRD will be removed](https://github.com/kubernetes/enhancements/issues/177). Volume snapshot and restore functionality for Kubernetes and the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, moved to GA in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.20 and will become unsupported with the v1.24 release. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and the [Volume Snapshot GA blog](/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/) blog article for more information. + +## What to do + +### Dockershim removal + +As stated earlier, there are several guides about +[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/). +You can start with [Finding what container runtime are on your nodes](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/). +If your nodes are using dockershim, there are other possible Docker Engine dependencies such as +Pods or third-party tools executing Docker commands or private registries in the Docker configuration file. You can follow the +[Check whether Dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/) guide to review possible +Docker Engine dependencies. Before upgrading to v1.24, you decide to either remain using Docker Engine and +[Migrate Docker Engine nodes from dockershim to cri-dockerd](/docs/tasks/administer-cluster/migrating-from-dockershim/migrate-dockershim-dockerd/) or migrate to a CRI-compatible runtime. Here's a guide to +[change the container runtime on a node from Docker Engine to containerd](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/). + +### `kubectl convert` + +The [`kubectl convert`](/docs/tasks/tools/included/kubectl-convert-overview/) plugin for `kubectl` +can be helpful to address migrating off deprecated APIs. The plugin facilitates the conversion of +manifests between different API versions, for example, from a deprecated to a non-deprecated API +version. More general information about the API migration process can be found in the [Deprecated API Migration Guide](/docs/reference/using-api/deprecation-guide/). +Follow the [install `kubectl convert` plugin](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-convert-plugin) +documentation to download and install the `kubectl-convert` binary. + +### Looking ahead + +The Kubernetes 1.25 and 1.26 releases planned for later this year will stop serving beta versions +of several currently stable Kubernetes APIs. The v1.25 release will also remove PodSecurityPolicy, +which was deprecated with Kubernetes 1.21 and will not graduate to stable. See [PodSecurityPolicy +Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) for more information. + +The official [list of API removals planned for Kubernetes 1.25](/docs/reference/using-api/deprecation-guide/#v1-25) is: + +* The beta CronJob API (batch/v1beta1) +* The beta EndpointSlice API (discovery.k8s.io/v1beta1) +* The beta Event API (events.k8s.io/v1beta1) +* The beta HorizontalPodAutoscaler API (autoscaling/v2beta1) +* The beta PodDisruptionBudget API (policy/v1beta1) +* The beta PodSecurityPolicy API (policy/v1beta1) +* The beta RuntimeClass API (node.k8s.io/v1beta1) + + +The official [list of API removals planned for Kubernetes 1.26](/docs/reference/using-api/deprecation-guide/#v1-26) is: + +* The beta FlowSchema and PriorityLevelConfiguration APIs (flowcontrol.apiserver.k8s.io/v1beta1) +* The beta HorizontalPodAutoscaler API (autoscaling/v2beta2) + + +### Want to know more? +Deprecations are announced in the Kubernetes release notes. You can see the announcements of pending deprecations in the release notes for: +* [Kubernetes 1.21](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#deprecation) +* [Kubernetes 1.22](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#deprecation) +* [Kubernetes 1.23](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#deprecation) +* We will formally announce the deprecations that come with [Kubernetes 1.24](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#deprecation) as part of the CHANGELOG for that release. + +For information on the process of deprecation and removal, check out the official Kubernetes [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api) document. + diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md new file mode 100644 index 0000000000..6c21c5f3c1 --- /dev/null +++ b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md @@ -0,0 +1,155 @@ +--- +layout: blog +title: 'Increasing the security bar in Ingress-NGINX v1.2.0' +date: 2022-04-28 +slug: ingress-nginx-1-2-0 +--- + +**Authors:** Ricardo Katz (VMware), James Strong (Chainguard) + +The [Ingress](/docs/concepts/services-networking/ingress/) may be one of the most targeted components +of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing +multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to +TLS certificates and their private keys). + +While it is a risky component in your architecture, it is still the most popular way to properly expose your services. + +Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't +do all proper sanitization before turning the configuration into an `nginx.conf` file, which may lead to information +disclosure risks. + +While we understand this risk and the real need to fix this, it's not an easy process to do, so we took another approach to reduce (but not remove!) this risk in the current (v1.2.0) release. + +## Meet Ingress NGINX v1.2.0 and the chrooted NGINX process + +One of the main challenges is that Ingress-NGINX runs the web proxy server (NGINX) alongside the Ingress +controller (the component that has access to Kubernetes API that and that creates the `nginx.conf` file). + +So, NGINX does have the same access to the filesystem of the controller (and Kubernetes service account token, and other configurations from the container). While splitting those components is our end goal, the project needed a fast response; that lead us to the idea of using `chroot()`. + +Let's take a look into what an Ingress-NGINX container looked like before this change: + +![Ingress NGINX pre chroot](ingress-pre-chroot.png) + +As we can see, the same container (not the Pod, the container!) that provides HTTP Proxy is the one that watches Ingress objects and writes the Container Volume + +Now, meet the new architecture: + +![Ingress NGINX post chroot](ingress-post-chroot.png) + +What does all of this mean? A basic summary is: that we are isolating the NGINX service as a container inside the +controller container. + +While this is not strictly true, to understand what was done here, it's good to understand how +Linux containers (and underlying mechanisms such as kernel namespaces) work. +You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article +[What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/). +(As you read that, bear in mind that Linux kernel namespaces are a different thing from +[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/)). + +## Skip the talk, what do I need to use this new approach? + +While this increases the security, we made this feature an opt-in in this release so you can have +time to make the right adjustments in your environment(s). This new feature is only available from +release v1.2.0 of the Ingress-NGINX controller. + +There are two required changes in your deployments to use this feature: +* Append the suffix "-chroot" to the container image name. For example: `gcr.io/k8s-staging-ingress-nginx/controller-chroot:v1.2.0` +* In your Pod template for the Ingress controller, find where you add the capability `NET_BIND_SERVICE` and add the capability `SYS_CHROOT`. After you edit the manifest, you'll see a snippet like: + +```yaml +capabilities: + drop: + - ALL + add: + - NET_BIND_SERVICE + - SYS_CHROOT +``` + +If you deploy the controller using the official Helm chart then change the following setting in +`values.yaml`: + +```yaml +controller: + image: + chroot: true +``` + +Ingress controllers are normally set up cluster-wide (the IngressClass API is cluster scoped). If you manage the +Ingress-NGINX controller but you're not the overall cluster operator, then check with your cluster admin about +whether you can use the `SYS_CHROOT` capability, **before** you enable it in your deployment. + +## OK, but how does this increase the security of my Ingress controller? + +Take the following configuration snippet and imagine, for some reason it was added to your `nginx.conf`: +``` +location /randomthing/ { + alias /; + autoindex on; +} +``` + +If you deploy this configuration, someone can call `http://website.example/randomthing` and get some listing (and access) to the whole filesystem of the Ingress controller. + +Now, can you spot the difference between chrooted and non chrooted Nginx on the listings below? + +| Without extra `chroot()` | With extra `chroot()` | +|----------------------------|--------| +| `bin` | `bin` | +| `dev` | `dev` | +| `etc` | `etc` | +| `home` | | +| `lib` | `lib` | +| `media` | | +| `mnt` | | +| `opt` | `opt` | +| `proc` | `proc` | +| `root` | | +| `run` | `run` | +| `sbin` | | +| `srv` | | +| `sys` | | +| `tmp` | `tmp` | +| `usr` | `usr` | +| `var` | `var` | +| `dbg` | | +| `nginx-ingress-controller` | | +| `wait-shutdown` | | + +The one in left side is not chrooted. So NGINX has full access to the filesystem. The one in right side is chrooted, so a new filesystem with only the required files to make NGINX work is created. + +## What about other security improvements in this release? + +We know that the new `chroot()` mechanism helps address some portion of the risk, but still, someone +can try to inject commands to read, for example, the `nginx.conf` file and extract sensitive information. + +So, another change in this release (this is opt-out!) is the _deep inspector_. +We know that some directives or regular expressions may be dangerous to NGINX, so the deep inspector +checks all fields from an Ingress object (during its reconciliation, and also with a +[validating admission webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)) +to verify if any fields contains these dangerous directives. + +The ingress controller already does this for annotations, and our goal is to move this existing validation to happen inside +deep inspection as part of a future release. + +You can take a look into the existing rules in [https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go](https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go). + +Due to the nature of inspecting and matching all strings within relevant Ingress objects, this new feature may consume a bit more CPU. You can disable it by running the ingress controller with the command line argument `--deep-inspect=false`. + +## What's next? + +This is not our final goal. Our final goal is to split the control plane and the data plane processes. +In fact, doing so will help us also achieve a [Gateway](https://gateway-api.sigs.k8s.io/) API implementation, +as we may have a different controller as soon as it "knows" what to provide to the data plane +(we need some help here!!) + +Some other projects in Kubernetes already take this approach +(like [KPNG](https://github.com/kubernetes-sigs/kpng), the proposed replacement for `kube-proxy`), +and we plan to align with them and get the same experience for Ingress-NGINX. + +## Further reading + +If you want to take a look into how chrooting was done in Ingress NGINX, take a look +into [https://github.com/kubernetes/ingress-nginx/pull/8337](https://github.com/kubernetes/ingress-nginx/pull/8337) +The release v1.2.0 containing all the changes can be found at +[https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0) diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png new file mode 100644 index 0000000000..d5d588a3bb Binary files /dev/null and b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png differ diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-pre-chroot.png b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-pre-chroot.png new file mode 100644 index 0000000000..38c7d54a55 Binary files /dev/null and b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-pre-chroot.png differ diff --git a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md new file mode 100644 index 0000000000..c2532638d0 --- /dev/null +++ b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md @@ -0,0 +1,319 @@ +--- +layout: blog +title: "Frontiers, fsGroups and frogs: the Kubernetes 1.23 release interview" +date: 2022-04-29 +--- + +**Author**: Craig Box (Google) + +One of the highlights of hosting the weekly [Kubernetes Podcast from Google](https://kubernetespodcast.com/) is talking to the release managers for each new Kubernetes version. The release team is constantly refreshing. Many working their way from small documentation fixes, step up to shadow roles, and then eventually lead a release. + +As we prepare for the 1.24 release next week, [in accordance with long-standing tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the story of 1.23. The release was led by [Rey Lejano](https://twitter.com/reylejano), a Field Engineer at SUSE. [I spoke to Rey](https://kubernetespodcast.com/episode/167-kubernetes-1.23/) in December, as he was awaiting the birth of his first child. + +Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/), so you hear all our stories from the Cloud Native community, including the story of 1.24 next week. + +*This transcript has been lightly edited and condensed for clarity.* + +--- + +**CRAIG BOX: I'd like to start with what is, of course, on top of everyone's mind at the moment. Let's talk African clawed frogs!** + +REY LEJANO: [CHUCKLES] Oh, you mean [Xenopus lavis](https://en.wikipedia.org/wiki/African_clawed_frog), the scientific name for the African clawed frog? + +**CRAIG BOX: Of course.** + +REY LEJANO: Not many people know, but my background and my degree is actually in microbiology, from the University of California Davis. I did some research for about four years in biochemistry, in a biochemistry lab, and I [do have a research paper published](https://www.sciencedirect.com/science/article/pii/). It's actually on glycoproteins, particularly something called "cortical granule lectin". We used frogs, because they generate lots and lots of eggs, from which we can extract the protein. That protein prevents polyspermy. When the sperm goes into the egg, the egg releases a glycoprotein, cortical granule lectin, to the membrane, and prevents any other sperm from going inside the egg. + +**CRAIG BOX: Were you able to take anything from the testing that we did on frogs and generalize that to higher-order mammals, perhaps?** + +REY LEJANO: Yes. Since mammals also have cortical granule lectin, we were able to analyze both the convergence and the evolutionary pattern, not just from multiple species of frogs, but also into mammals as well. + +**CRAIG BOX: Now, there's a couple of different threads to unravel here. When you were young, what led you into the fields of biology, and perhaps more the technical side of it?** + +REY LEJANO: I think it was mostly from family, since I do have a family history in the medical field that goes back generations. So I kind of felt like that was the natural path going into college. + +**CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?** + +REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://www.axelos.com/certifications/itil-service-management/what-is-itil). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually. + +**CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.** + +REY LEJANO: Yes, absolutely. It's not very cloud native. [CHUCKLES] + +**CRAIG BOX: Not at all.** + +REY LEJANO: You don't really hear about it in the cloud native landscape. Definitely, you can tell someone's been in the field for a little bit, if they specialize or have worked with ITIL before. + +**CRAIG BOX: You mentioned that you wanted to get out of the basement. That is quite often where people put the programmers. Did they just give you a bit of light in the new basement?** + +REY LEJANO: [LAUGHS] They did give us much better lighting. Able to get some vitamin D sometimes, as well. + +**CRAIG BOX: To wrap up the discussion about your previous career — over the course of the last year, with all of the things that have happened in the world, I could imagine that microbiology skills may be more in demand than perhaps they were when you studied them?** + +REY LEJANO: Oh, absolutely. I could definitely see a big increase of numbers of people going into the field. Also, reading what's going on with the world currently kind of brings back all the education I've learned in the past, as well. + +**CRAIG BOX: Do you keep in touch with people you went through school with?** + +REY LEJANO: Just some close friends, but not in the microbiology field. + +**CRAIG BOX: One thing that I think will probably happen as a result of the pandemic is a renewed interest in some of these STEM fields. It will be interesting to see what impact that has on society at large.** + +REY LEJANO: Yeah. I think that'll be great. + +**CRAIG BOX: You mentioned working at a consultancy doing IT management, application performance monitoring, and so on. When did Kubernetes come into your professional life?** + +REY LEJANO: One of my good friends at the company I worked at, left in mid-2015. He went on to a company that was pretty heavily into Docker. He taught me a little bit. I did my first "docker run" around 2015, maybe 2016. Then, one of the applications we were using for the ITIL framework was containerized around 2018 or so, also in Kubernetes. At that time, it was pretty buggy. That was my initial introduction to Kubernetes and containerised applications. + +Then I left that company, and I actually joined my friend over at [RX-M](https://rx-m.com/), which is a cloud native consultancy and training firm. They specialize in Docker and Kubernetes. I was able to get my feet wet. I got my CKD, got my CKA as well. And they were really, really great at encouraging us to learn more about Kubernetes and also to be involved in the community. + +**CRAIG BOX: You will have seen, then, the life cycle of people adopting Kubernetes and containerization at large, through your own initial journey and then through helping customers. How would you characterize how that journey has changed from the early days to perhaps today?** + +REY LEJANO: I think the early days, there was a lot of questions of, why do I have to containerize? Why can't I just stay with virtual machines? + +**CRAIG BOX: It's a line item on your CV.** + +REY LEJANO: [CHUCKLES] It is. And nowadays, I think people know the value of using containers, of orchestrating containers with Kubernetes. I don't want to say "jumping on the bandwagon", but it's become the de-facto standard to orchestrate containers. + +**CRAIG BOX: It's not something that a consultancy needs to go out and pitch to customers that they should be doing. They're just taking it as, that will happen, and starting a bit further down the path, perhaps.** + +REY LEJANO: Absolutely. + +**CRAIG BOX: Working at a consultancy like that, how much time do you get to work on improving process, perhaps for multiple customers, and then looking at how you can upstream that work, versus paid work that you do for just an individual customer at a time?** + +REY LEJANO: Back then, it would vary. They helped me introduce myself, and I learned a lot about the cloud native landscape and Kubernetes itself. They helped educate me as to how the cloud native landscape, and the tools around it, can be used together. My boss at that company, Randy, he actually encouraged us to start contributing upstream, and encouraged me to join the release team. He just said, this is a great opportunity. Definitely helped me with starting with the contributions early on. + +**CRAIG BOX: Was the release team the way that you got involved with upstream Kubernetes contribution?** + +REY LEJANO: Actually, no. My first contribution was with SIG Docs. I met Taylor Dolezal — he was the release team lead for 1.19, but he is involved with SIG Docs as well. I met him at KubeCon 2019, I sat at his table during a luncheon. I remember Paris Pittman was hosting this luncheon at the Marriott. Taylor says he was involved with SIG Docs. He encouraged me to join. I started joining into meetings, started doing a few drive-by PRs. That's what we call them — drive-by — little typo fixes. Then did a little bit more, started to send better or higher quality pull requests, and also reviewing PRs. + +**CRAIG BOX: When did you first formally take your release team role?** + +REY LEJANO: That was in [1.18](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md), in December. My boss at the time encouraged me to apply. I did, was lucky enough to get accepted for the release notes shadow. Then from there, stayed in with release notes for a few cycles, then went into Docs, naturally then led Docs, then went to Enhancements, and now I'm the release lead for 1.23. + +**CRAIG BOX: I don't know that a lot of people think about what goes into a good release note. What would you say does?** + +REY LEJANO: [CHUCKLES] You have to tell the end user what has changed or what effect that they might see in the release notes. It doesn't have to be highly technical. It could just be a few lines, and just saying what has changed, what they have to do if they have to do anything as well. + +**CRAIG BOX: As you moved through the process of shadowing, how did you learn from the people who were leading those roles?** + +REY LEJANO: I said this a few times when I was the release lead for this cycle. You get out of the release team as much as you put in, or it directly aligns to how much you put in. I learned a lot. I went into the release team having that mindset of learning from the role leads, learning from the other shadows, as well. That's actually a saying that my first role lead told me. I still carry it to heart, and that was back in 1.18. That was Eddie, in the very first meeting we had, and I still carry it to heart. + +**CRAIG BOX: You, of course, were [the release lead for 1.23](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23). First of all, congratulations on the release.** + +REY LEJANO: Thank you very much. + +**CRAIG BOX: The theme for this release is [The Next Frontier](https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/). Tell me the story of how we came to the theme and then the logo.** + +REY LEJANO: The Next Frontier represents a few things. It not only represents the next enhancements in this release, but Kubernetes itself also has a history of Star Trek references. The original codename for Kubernetes was Project Seven, a reference to Seven of Nine, originally from Star Trek Voyager. Also the seven spokes in the helm in the logo of Kubernetes as well. And, of course, Borg, the predecessor to Kubernetes. + +The Next Frontier continues that Star Trek reference. It's a fusion of two titles in the Star Trek universe. One is [Star Trek V, the Final Frontier](https://en.wikipedia.org/wiki/Star_Trek_V:_The_Final_Frontier), and the Star Trek: The Next Generation. + +**CRAIG BOX: Do you have any opinion on the fact that Star Trek V was an odd-numbered movie, and they are [canonically referred to as being lesser than the even-numbered ones](https://screenrant.com/star-trek-movies-odd-number-curse-explained/)?** + +REY LEJANO: I can't say, because I am such a sci-fi nerd that I love all of them even though they're bad. Even the post-Next Generation movies, after the series, I still liked all of them, even though I know some weren't that great. + +**CRAIG BOX: Am I right in remembering that Star Trek V was the one directed by William Shatner?** + +REY LEJANO: Yes, that is correct. + +**CRAIG BOX: I think that says it all.** + +REY LEJANO: [CHUCKLES] Yes. + +**CRAIG BOX: Now, I understand that the theme comes from a part of the [SIG Release charter](https://github.com/kubernetes/community/blob/master/sig-release/charter.md)?** + +REY LEJANO: Yes. There's a line in the SIG Release charter, "ensure there is a consistent group of community members in place to support the release process across time." With the release team, we have new shadows that join every single release cycle. With this, we're growing with this community. We're growing the release team members. We're growing SIG Release. We're growing the Kubernetes community itself. For a lot of people, this is their first time contributing to open source, so that's why I say it's their new open source frontier. + +**CRAIG BOX: And the logo is obviously very Star Trek-inspired. It sort of surprised me that it took that long for someone to go this route.** + +REY LEJANO: I was very surprised as well. I had to relearn Adobe Illustrator to create the logo. + +**CRAIG BOX: This your own work, is it?** + +REY LEJANO: This is my own work. + +**CRAIG BOX: It's very nice.** + +REY LEJANO: Thank you very much. Funny, the galaxy actually took me the longest time versus the ship. Took me a few days to get that correct. I'm always fine-tuning it, so there might be a final change when this is actually released. + +**CRAIG BOX: No frontier is ever truly final.** + +REY LEJANO: True, very true. + +**CRAIG BOX: Moving now from the theme of the release to the substance, perhaps, what is new in 1.23?** + +REY LEJANO: We have 47 enhancements. I'm going to run through most of the stable ones, if not all of them, some of the key Beta ones, and a few of the Alpha enhancements for 1.23. + +One of the key enhancements is [dual-stack IPv4/IPv6](https://github.com/kubernetes/enhancements/issues/563), which went GA in 1.23. + +Some background info: dual-stack was introduced as Alpha in 1.15. You probably saw a keynote at KubeCon 2019. Back then, the way dual-stack worked was that you needed two services — you needed a service per IP family. You would need a service for IPv4 and a service for IPv6. It was refactored in 1.20. In 1.21, it was in Beta; clusters were enabled to be dual-stack by default. + +And then in 1.23 we did remove the IPv6 dual-stack feature flag. It's not mandatory to use dual-stack. It's actually not "default" still. The pods, the services still default to single-stack. There are some requirements to be able to use dual-stack. The nodes have to be routable on IPv4 and IPv6 network interfaces. You need a CNI plugin that supports dual-stack. The pods themselves have to be configured to be dual-stack. And the services need the ipFamilyPolicy field to specify prefer dual-stack, or require dual-stack. + +**CRAIG BOX: This sounds like there's an implication in this that v4 is still required. Do you see a world where we can actually move to v6-only clusters?** + +REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to come. I remember a long time ago, they kept saying "it's going to be all IPv6", and that was decades ago. + +**CRAIG BOX: I think I may have mentioned on the show before, but there was [a meeting in London that Vint Cerf attended](https://www.youtube.com/watch?v=AEaJtZVimqs), and he gave a public presentation at the time to say, now is the time of v6. And that was 10 years ago at least. It's still not the time of v6, and my desktop still doesn't have Linux on it. One day.** + +REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23. + +One of the other highlights of 1.23 is [pod security admission going to Beta](/blog/2021/12/09/pod-security-admission-beta/). I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running. + +There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed. + +**CRAIG BOX: You mentioned there that PodSecurityPolicy is due to be deprecated in two releases' time. Are we lining up these features so that pod security admission will be GA at that time?** + +REY LEJANO: Yes. Absolutely. I'll talk about that for another feature in a little bit as well. There's also another feature that went to GA. It was an API that went to GA, and therefore the Beta API is now deprecated. I'll talk about that a little bit. + +**CRAIG BOX: All right. Let's talk about what's next on the list.** + +REY LEJANO: Let's move on to more stable enhancements. One is the [TTL controller](https://github.com/kubernetes/enhancements/issues/592). This cleans up jobs and pods after the jobs are finished. There is a TTL timer that starts when the job or pod is finished. This TTL controller watches all the jobs, and ttlSecondsAfterFinished needs to be set. The controller will see if the ttlSecondsAfterFinished, combined with the last transition time, if it's greater than now. If it is, then it will delete the job and the pods of that job. + +**CRAIG BOX: Loosely, it could be called a garbage collector?** + +REY LEJANO: Yes. Garbage collector for pods and jobs, or jobs and pods. + +**CRAIG BOX: If Kubernetes is truly becoming a programming language, it of course has to have a garbage collector implemented.** + +REY LEJANO: Yeah. There's another one, too, coming in Alpha. [CHUCKLES] + +**CRAIG BOX: Tell me about that.** + +REY LEJANO: That one is coming in in Alpha. It's actually one of my favorite features, because there's only a few that I'm going to highlight today. [PVCs for StafeulSet will be cleaned up](https://github.com/kubernetes/enhancements/issues/1847). It will auto-delete PVCs created by StatefulSets, when you delete that StatefulSet. + +**CRAIG BOX: What's next on our tour of stable features?** + +REY LEJANO: Next one is, [skip volume ownership change goes to stable](https://github.com/kubernetes/enhancements/issues/695). This is from SIG Storage. There are times when you're running a stateful application, like many databases, they're sensitive to permission bits changing underneath. Currently, when a volume is bind mounted inside the container, the permissions of that volume will change recursively. It might take a really long time. + +Now, there's a field, the fsGroupChangePolicy, which allows you, as a user, to tell Kubernetes how you want the permission and ownership change for that volume to happen. You can set it to always, to always change permissions, or just on mismatch, to only do it when the permission ownership changes at the top level is different from what is expected. + +**CRAIG BOX: It does feel like a lot of these enhancements came from a very particular use case where someone said, "hey, this didn't work for me and I've plumbed in a feature that works with exactly the thing I need to have".** + +REY LEJANO: Absolutely. People create issues for these, then create Kubernetes enhancement proposals, and then get targeted for releases. + +**CRAIG BOX: Another GA feature in this release — ephemeral volumes.** + +REY LEJANO: We've always been able to use empty dir for ephemeral volumes, but now we could actually have [ephemeral inline volumes](https://github.com/kubernetes/enhancements/issues/1698), meaning that you could take your standard CSI driver and be able to use ephemeral volumes with it. + +**CRAIG BOX: And, a long time coming, [CronJobs](https://github.com/kubernetes/enhancements/issues/19).** + +REY LEJANO: CronJobs is a funny one, because it was stable before 1.23. For 1.23, it was still tracked,but it was just cleaning up some of the old controller. With CronJobs, there's a v2 controller. What was cleaned up in 1.23 is just the old v1 controller. + +**CRAIG BOX: Were there any other duplications or major cleanups of note in this release?** + +REY LEJANO: Yeah. There were a few you might see in the major themes. One's a little tricky, around FlexVolumes. This is one of the efforts from SIG Storage. They have an effort to migrate in-tree plugins to CSI drivers. This is a little tricky, because FlexVolumes were actually deprecated in November 2020. We're [formally announcing it in 1.23](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors). + +**CRAIG BOX: FlexVolumes, in my mind, predate CSI as a concept. So it's about time to get rid of them.** + +REY LEJANO: Yes, it is. There's another deprecation, just some [klog specific flags](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog), but other than that, there are no other big deprecations in 1.23. + +**CRAIG BOX: The buzzword of the last KubeCon, and in some ways the theme of the last 12 months, has been secure software supply chain. What work is Kubernetes doing to improve in this area?** + +REY LEJANO: For 1.23, Kubernetes is now SLSA compliant at Level 1, which means that provenance attestation files that describe the staging and release phases of the release process are satisfactory for the SLSA framework. + +**CRAIG BOX: What needs to happen to step up to further levels?** + +REY LEJANO: Level 1 means a few things — that the build is scripted; that the provenance is available, meaning that the artifacts are verified and they're handed over from one phase to the next; and describes how the artifact is produced. Level 2 means that the source is version-controlled, which it is, provenance is authenticated, provenance is service-generated, and there is a build service. There are four levels of SLSA compliance. + +**CRAIG BOX: It does seem like the levels were largely influenced by what it takes to build a big, secure project like this. It doesn't seem like it will take a lot of extra work to move up to verifiable provenance, for example. There's probably just a few lines of script required to meet many of those requirements.** + +REY LEJANO: Absolutely. I feel like we're almost there; we'll see what will come out of 1.24. And I do want to give a big shout-out to SIG Release and Release Engineering, primarily to Adolfo García Veytia, who is aka Puerco on GitHub and on Slack. He's been driving this forward. + +**CRAIG BOX: You've mentioned some APIs that are being graduated in time to replace their deprecated version. Tell me about the new HPA API.** + +REY LEJANO: The [horizontal pod autoscaler v2 API](https://github.com/kubernetes/enhancements/issues/2702), is now stable, which means that the v2beta2 API is deprecated. Just for everyone's knowledge, the v1 API is not being deprecated. The difference is that v2 adds support for multiple and custom metrics to be used for HPA. + +**CRAIG BOX: There's also now a facility to validate my CRDs with an expression language.** + +REY LEJANO: Yeah. You can use the [Common Expression Language, or CEL](https://github.com/google/cel-spec), to validate your CRDs, so you no longer need to use webhooks. This also makes the CRDs more self-contained and declarative, because the rules are now kept within the CRD object definition. + +**CRAIG BOX: What new features, perhaps coming in Alpha or Beta, have taken your interest?** + +REY LEJANO: Aside from pod security policies, I really love [ephemeral containers](https://github.com/kubernetes/enhancements/issues/277) supporting kubectl debug. It launches an ephemeral container and a running pod, shares those pod namespaces, and you can do all your troubleshooting with just running kubectl debug. + +**CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.** + +REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do `--watch`, and it will also sort with the `--watch` option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well. + +**CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?** + +REY LEJANO: More of what I look out for to help the customers. + +**CRAIG BOX: Right.** + +REY LEJANO: I really love kubectl events. Really love the PVCs being cleaned up with StatefulSets. Most of it's for selfish reasons that it will improve troubleshooting efforts. [CHUCKLES] + +**CRAIG BOX: I have always hoped that a release team lead would say to me, "yes, I have selfish reasons. And I finally got something I wanted in."** + +REY LEJANO: [LAUGHS] + +**CRAIG BOX: Perhaps I should run to be release team lead, just so I can finally get init containers fixed once and for all.** + +REY LEJANO: Oh, init containers, I've been looking for that for a while. I've actually created animated GIFs on how init containers will be run with that Kubernetes enhancement proposal, but it's halted currently. + +**CRAIG BOX: One day.** + +REY LEJANO: One day. Maybe I shouldn't stay halted. + +**CRAIG BOX: You mentioned there are obviously the things you look out for. Are there any things that are coming down the line, perhaps Alpha features or maybe even just proposals you've seen lately, that you're personally really looking forward to seeing which way they go?** + +REY LEJANO: Yeah. Oone is a very interesting one, it affects the whole community, so it's not just for personal reasons. As you may have known, Dockershim is deprecated. And we did release a blog that it will be removed in 1.24. + +**CRAIG BOX: Scared a bunch of people.** + +REY LEJANO: Scared a bunch of people. From a survey, we saw that a lot of people are still using Docker and Dockershim. One of the enhancements for 1.23 is, [kubelet CRI goes to Beta](https://github.com/kubernetes/enhancements/issues/2040). This promotes the CRI API, which is required. This had to be in Beta for Dockershim to be removed in 1.24. + +**CRAIG BOX: Now, in the last release team lead interview, [we spoke with Savitha Raghunathan](https://kubernetespodcast.com/episode/157-kubernetes-1.22/), and she talked about what she would advise you as her successor. It was to look out for the mental health of the team members. How were you able to take that advice on board?** + +REY LEJANO: That was great advice from Savitha. A few things I've made note of with each release team meeting. After each release team meeting, I stop the recording, because we do record all the meetings and post them on YouTube. And I open up the floor to anyone who wants to say anything that's not recorded, that's not going to be on the agenda. Also, I tell people not to work on weekends. I broke this rule once, but other than that, I told people it could wait. Just be mindful of your mental health. + +**CRAIG BOX: It's just been announced that [James Laverack from Jetstack](https://twitter.com/JamesLaverack/status/1466834312993644551) will be the release team lead for 1.24. James and I shared an interesting Mexican dinner at the last KubeCon in San Diego.** + +REY LEJANO: Oh, nice. I didn't know you knew James. + +**CRAIG BOX: The British tech scene. We're a very small world. What will your advice to James be?** + +REY LEJANO: What I would tell James for 1.24 is use teachable moments in the release team meetings. When you're a shadow for the first time, it's very daunting. It's very difficult, because you don't know the repos. You don't know the release process. Everyone around you seems like they know the release process, and very familiar with what the release process is. But as a first-time shadow, you don't know all the vernacular for the community. I just advise to use teachable moments. Take a few minutes in the release team meetings to make it a little easier for new shadows to ramp up and to be familiar with the release process. + +**CRAIG BOX: Has there been major evolution in the process in the time that you've been involved? Or do you think that it's effectively doing what it needs to do?** + +REY LEJANO: It's always evolving. I remember my first time in release notes, 1.18, we said that our goal was to automate and program our way out so that we don't have a release notes team anymore. That's changed [CHUCKLES] quite a bit. Although there's been significant advancements in the release notes process by Adolfo and also James, they've created a subcommand in krel to generate release notes. + +But nowadays, all their release notes are richer. Still not there at the automation process yet. Every release cycle, there is something a little bit different. For this release cycle, we had a production readiness review deadline. It was a soft deadline. A production readiness review is a review by several people in the community. It's actually been required since 1.21, and it ensures that the enhancements are observable, scalable, supportable, and it's safe to operate in production, and could also be disabled or rolled back. In 1.23, we had a deadline to have the production readiness review completed by a specific date. + +**CRAIG BOX: How have you found the change of schedule to three releases per year rather than four?** + +REY LEJANO: Moving to three releases a year from four, in my opinion, has been an improvement, because we support the last three releases, and now we can actually support the last releases in a calendar year instead of having 9 months out of 12 months of the year. + +**CRAIG BOX: The next event on the calendar is a [Kubernetes contributor celebration](https://www.kubernetes.dev/events/kcc2021/) starting next Monday. What can we expect from that event?** + +REY LEJANO: This is our second time running this virtual event. It's a virtual celebration to recognize the whole community and all of our accomplishments of the year, and also contributors. There's a number of events during this week of celebration. It starts the week of December 13. + +There's events like the Kubernetes Contributor Awards, where SIGs honor and recognize the hard work of the community and contributors. There's also a DevOps party game as well. There is a cloud native bake-off. I do highly suggest people to go to [kubernetes.dev/celebration](https://www.kubernetes.dev/events/past-events/2021/kcc2021/) to learn more. + +**CRAIG BOX: How exactly does one judge a virtual bake-off?** + +REY LEJANO: That I don't know. [CHUCKLES] + +**CRAIG BOX: I tasted my scones. I think they're the best. I rate them 10 out of 10.** + +REY LEJANO: Yeah. That is very difficult to do virtually. I would have to say, probably what the dish is, how closely it is tied with Kubernetes or open source or to CNCF. There's a few judges. I know Josh Berkus and Rin Oliver are a few of the judges running the bake-off. + +**CRAIG BOX: Yes. We spoke with Josh about his love of the kitchen, and so he seems like a perfect fit for that role.** + +REY LEJANO: He is. + +**CRAIG BOX: Finally, your wife and yourself are expecting your first child in January. Have you had a production readiness review for that?** + +REY LEJANO: I think we failed that review. [CHUCKLES] + +**CRAIG BOX: There's still time.** + +REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and `--apply` again. + +--- + +_[Rey Lejano](https://twitter.com/reylejano) is a field engineer at SUSE, by way of Rancher Labs, and was the release team lead for Kubernetes 1.23. He is now also a co-chair for SIG Docs. His son Liam is now 3 and a half months old._ + +_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._ diff --git a/content/en/blog/_posts/2022-05-03-dockershim-historical-context.md b/content/en/blog/_posts/2022-05-03-dockershim-historical-context.md new file mode 100644 index 0000000000..6bdfac570d --- /dev/null +++ b/content/en/blog/_posts/2022-05-03-dockershim-historical-context.md @@ -0,0 +1,25 @@ +--- +layout: blog +title: "Dockershim: The Historical Context" +date: 2022-05-03 +slug: dockershim-historical-context +--- + +**Author:** Kat Cosgrove + + +Dockershim has been removed as of Kubernetes v1.24, and this is a positive move for the project. However, context is important for fully understanding something, be it socially or in software development, and this deserves a more in-depth review. Alongside the dockershim removal in Kubernetes v1.24, we’ve seen some confusion (sometimes at a panic level) and dissatisfaction with this decision in the community, largely due to a lack of context around this removal. The decision to deprecate and eventually remove dockershim from Kubernetes was not made quickly or lightly. Still, it’s been in the works for so long that many of today’s users are newer than that decision, and certainly newer than the choices that led to the dockershim being necessary in the first place. + +So what is the dockershim, and why is it going away? + +In the early days of Kubernetes, we only supported one container runtime. That runtime was Docker Engine. Back then, there weren’t really a lot of other options out there and Docker was the dominant tool for working with containers, so this was not a controversial choice. Eventually, we started adding more container runtimes, like rkt and hypernetes, and it became clear that Kubernetes users want a choice of runtimes working best for them. So Kubernetes needed a way to allow cluster operators the flexibility to use whatever runtime they choose. + +The [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) (CRI) was released to allow that flexibility. The introduction of CRI was great for the project and users alike, but it did introduce a problem: Docker Engine’s use as a container runtime predates CRI, and Docker Engine is not CRI-compatible. To solve this issue, a small software shim (dockershim) was introduced as part of the kubelet component specifically to fill in the gaps between Docker Engine and CRI, allowing cluster operators to continue using Docker Engine as their container runtime largely uninterrupted. + +However, this little software shim was never intended to be a permanent solution. Over the course of years, its existence has introduced a lot of unnecessary complexity to the kubelet itself. Some integrations are inconsistently implemented for Docker because of this shim, resulting in an increased burden on maintainers, and maintaining vendor-specific code is not in line with our open source philosophy. To reduce this maintenance burden and move towards a more collaborative community in support of open standards, [KEP-2221 was introduced](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim), proposing the removal of the dockershim. With the release of Kubernetes v1.20, the deprecation was official. + +We didn’t do a great job communicating this, and unfortunately, the deprecation announcement led to some panic within the community. Confusion around what this meant for Docker as a company, if container images built by Docker would still run, and what Docker Engine actually is led to a conflagration on social media. This was our fault; we should have more clearly communicated what was happening and why at the time. To combat this, we released [a blog](/blog/2020/12/02/dont-panic-kubernetes-and-docker/) and [accompanying FAQ](/blog/2020/12/02/dockershim-faq/) to allay the community’s fears and correct some misconceptions about what Docker is and how containers work within Kubernetes. As a result of the community’s concerns, Docker and Mirantis jointly agreed to continue supporting the dockershim code in the form of [cri-dockerd](https://www.mirantis.com/blog/the-future-of-dockershim-is-cri-dockerd/), allowing you to continue using Docker Engine as your container runtime if need be. For the interest of users who want to try other runtimes, like containerd or cri-o, [migration documentation was written](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/). + +We later [surveyed the community](https://kubernetes.io/blog/2021/11/12/are-you-ready-for-dockershim-removal/) and [discovered that there are still many users with questions and concerns](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim). In response, Kubernetes maintainers and the CNCF committed to addressing these concerns by extending documentation and other programs. In fact, this blog post is a part of this program. With so many end users successfully migrated to other runtimes, and improved documentation, we believe that everyone has a paved way to migration now. + +Docker is not going away, either as a tool or as a company. It’s an important part of the cloud native community and the history of the Kubernetes project. We wouldn’t be where we are without them. That said, removing dockershim from kubelet is ultimately good for the community, the ecosystem, the project, and open source at large. This is an opportunity for all of us to come together to support open standards, and we’re glad to be doing so with the help of Docker and the community. diff --git a/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md b/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md new file mode 100644 index 0000000000..f29c6c92fe --- /dev/null +++ b/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md @@ -0,0 +1,242 @@ +--- +layout: blog +title: "Kubernetes 1.24: Stargazer" +date: 2022-05-03 +slug: kubernetes-1-24-release-announcement +--- + +**Authors**: [Kubernetes 1.24 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.24/release-team.md) + +We are excited to announce the release of Kubernetes 1.24, the first release of 2022! + +This release consists of 46 enhancements: fourteen enhancements have graduated to stable, +fifteen enhancements are moving to beta, and thirteen enhancements are entering alpha. +Also, two features have been deprecated, and two features have been removed. + +## Major Themes + +### Dockershim Removed from kubelet + +After its deprecation in v1.20, the dockershim component has been removed from the kubelet in Kubernetes v1.24. +From v1.24 onwards, you will need to either use one of the other [supported runtimes](/docs/setup/production-environment/container-runtimes/) (such as containerd or CRI-O) +or use cri-dockerd if you are relying on Docker Engine as your container runtime. +For more information about ensuring your cluster is ready for this removal, please +see [this guide](/blog/2022/03/31/ready-for-dockershim-removal/). + +### Beta APIs Off by Default + +[New beta APIs will not be enabled in clusters by default](https://github.com/kubernetes/enhancements/issues/3136). +Existing beta APIs and new versions of existing beta APIs will continue to be enabled by default. + +### Signing Release Artifacts + +Release artifacts are [signed](https://github.com/kubernetes/enhancements/issues/3031) using [cosign](https://github.com/sigstore/cosign) +signatures, +and there is experimental support for [verifying image signatures](/docs/tasks/administer-cluster/verify-signed-images/). +Signing and verification of release artifacts is part of [increasing software supply chain security for the Kubernetes release process](https://github.com/kubernetes/enhancements/issues/3027). + +### OpenAPI v3 + +Kubernetes 1.24 offers beta support for publishing its APIs in the [OpenAPI v3 format](https://github.com/kubernetes/enhancements/issues/2896). + +### Storage Capacity and Volume Expansion Are Generally Available + +[Storage capacity tracking](https://github.com/kubernetes/enhancements/issues/1472) +supports exposing currently available storage capacity via [CSIStorageCapacity objects](/docs/concepts/storage/storage-capacity/#api) +and enhances scheduling of pods that use CSI volumes with late binding. + +[Volume expansion](https://github.com/kubernetes/enhancements/issues/284) adds support +for resizing existing persistent volumes. + +### NonPreemptingPriority to Stable + +This feature adds [a new option to PriorityClasses](https://github.com/kubernetes/enhancements/issues/902), +which can enable or disable pod preemption. + +### Storage Plugin Migration + +Work is underway to [migrate the internals of in-tree storage plugins](https://github.com/kubernetes/enhancements/issues/625) to call out to CSI Plugins +while maintaining the original API. +The [Azure Disk](https://github.com/kubernetes/enhancements/issues/1490) +and [OpenStack Cinder](https://github.com/kubernetes/enhancements/issues/1489) plugins +have both been migrated. + +### gRPC Probes Graduate to Beta + +With Kubernetes 1.24, the [gRPC probes functionality](https://github.com/kubernetes/enhancements/issues/2727) +has entered beta and is available by default. You can now [configure startup, liveness, and readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) for your gRPC app +natively within Kubernetes without exposing an HTTP endpoint or +using an extra executable. + +### Kubelet Credential Provider Graduates to Beta + +Originally released as Alpha in Kubernetes 1.20, the kubelet's support for +[image credential providers](/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/) +has now graduated to Beta. +This allows the kubelet to dynamically retrieve credentials for a container image registry +using exec plugins rather than storing credentials on the node's filesystem. + +### Contextual Logging in Alpha + +Kubernetes 1.24 has introduced [contextual logging](https://github.com/kubernetes/enhancements/issues/3077) +that enables the caller of a function to control all aspects of logging (output formatting, verbosity, additional values, and names). + +### Avoiding Collisions in IP allocation to Services + +Kubernetes 1.24 introduces a new opt-in feature that allows you to +[soft-reserve a range for static IP address assignments](/docs/concepts/services-networking/service/#service-ip-static-sub-range) +to Services. +With the manual enablement of this feature, the cluster will prefer automatic assignment from +the pool of Service IP addresses, thereby reducing the risk of collision. + +A Service `ClusterIP` can be assigned: + +* dynamically, which means the cluster will automatically pick a free IP within the configured Service IP range. +* statically, which means the user will set one IP within the configured Service IP range. + +Service `ClusterIP` are unique; hence, trying to create a Service with a `ClusterIP` that has already been allocated will return an error. + +### Dynamic Kubelet Configuration is Removed from the Kubelet + +After being deprecated in Kubernetes 1.22, Dynamic Kubelet Configuration has been removed from the kubelet. The feature will be removed from the API server in Kubernetes 1.26. + +## CNI Version-Related Breaking Change + +Before you upgrade to Kubernetes 1.24, please verify that you are using/upgrading to a container +runtime that has been tested to work correctly with this release. + +For example, the following container runtimes are being prepared, or have already been prepared, for Kubernetes: + +* containerd v1.6.4 and later, v1.5.11 and later +* CRI-O 1.24 and later + +Service issues exist for pod CNI network setup and tear down in containerd +v1.6.0–v1.6.3 when the CNI plugins have not been upgraded and/or the CNI config +version is not declared in the CNI config files. The containerd team reports, "these issues are resolved in containerd v1.6.4." + +With containerd v1.6.0–v1.6.3, if you do not upgrade the CNI plugins and/or +declare the CNI config version, you might encounter the following "Incompatible +CNI versions" or "Failed to destroy network for sandbox" error conditions. + +## CSI Snapshot + +_This information was added after initial publication._ + +[VolumeSnapshot v1beta1 CRD has been removed](https://github.com/kubernetes/enhancements/issues/177). +Volume snapshot and restore functionality for Kubernetes and the Container Storage Interface (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, moved to GA in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.20 and is now unsupported. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and [Volume Snapshot GA blog](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/) for more information. + +## Other Updates + +### Graduations to Stable + +This release saw fourteen enhancements promoted to stable: + +* [Container Storage Interface (CSI) Volume Expansion](https://github.com/kubernetes/enhancements/issues/284) +* [Pod Overhead](https://github.com/kubernetes/enhancements/issues/688): Account for resources tied to the pod sandbox but not specific containers. +* [Add non-preempting option to PriorityClasses](https://github.com/kubernetes/enhancements/issues/902) +* [Storage Capacity Tracking](https://github.com/kubernetes/enhancements/issues/1472) +* [OpenStack Cinder In-Tree to CSI Driver Migration](https://github.com/kubernetes/enhancements/issues/1489) +* [Azure Disk In-Tree to CSI Driver Migration](https://github.com/kubernetes/enhancements/issues/1490) +* [Efficient Watch Resumption](https://github.com/kubernetes/enhancements/issues/1904): Watch can be efficiently resumed after kube-apiserver reboot. +* [Service Type=LoadBalancer Class Field](https://github.com/kubernetes/enhancements/issues/1959): Introduce a new Service annotation `service.kubernetes.io/load-balancer-class` that allows multiple implementations of `type: LoadBalancer` Services in the same cluster. +* [Indexed Job](https://github.com/kubernetes/enhancements/issues/2214): Add a completion index to Pods of Jobs with a fixed completion count. +* [Add Suspend Field to Jobs API](https://github.com/kubernetes/enhancements/issues/2232): Add a suspend field to the Jobs API to allow orchestrators to create jobs with more control over when pods are created. +* [Pod Affinity NamespaceSelector](https://github.com/kubernetes/enhancements/issues/2249): Add a `namespaceSelector` field for to pod affinity/anti-affinity spec. +* [Leader Migration for Controller Managers](https://github.com/kubernetes/enhancements/issues/2436): kube-controller-manager and cloud-controller-manager can apply new controller-to-controller-manager assignment in HA control plane without downtime. +* [CSR Duration](https://github.com/kubernetes/enhancements/issues/2784): Extend the CertificateSigningRequest API with a mechanism to allow clients to request a specific duration for the issued certificate. + +### Major Changes + +This release saw two major changes: + +* [Dockershim Removal](https://github.com/kubernetes/enhancements/issues/2221) +* [Beta APIs are off by Default](https://github.com/kubernetes/enhancements/issues/3136) + +### Release Notes + +Check out the full details of the Kubernetes 1.24 release in our [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md). + +### Availability + +Kubernetes 1.24 is available for download on [GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.24.0). +To get started with Kubernetes, check out these [interactive tutorials](/docs/tutorials/) or run local +Kubernetes clusters using containers as “nodes”, with [kind](https://kind.sigs.k8s.io/). +You can also easily install 1.24 using [kubeadm](/docs/setup/independent/create-cluster-kubeadm/). + +### Release Team + +This release would not have been possible without the combined efforts of committed individuals +comprising the Kubernetes 1.24 release team. This team came together to deliver all of the components +that go into each Kubernetes release, including code, documentation, release notes, and more. + +Special thanks to James Laverack, our release lead, for guiding us through a successful release cycle, +and to all of the release team members for the time and effort they put in to deliver the v1.24 +release for the Kubernetes community. + +### Release Theme and Logo + +**Kubernetes 1.24: Stargazer** + +{{< figure src="/images/blog/2022-05-03-kubernetes-release-1.24/kubernetes-1.24.png" alt="" class="release-logo" >}} + +The theme for Kubernetes 1.24 is _Stargazer_. + +Generations of people have looked to the stars in awe and wonder, from ancient astronomers to the +scientists who built the James Webb Space Telescope. The stars have inspired us, set our imagination +alight, and guided us through long nights on difficult seas. + +With this release we gaze upwards, to what is possible when our community comes together. Kubernetes +is the work of hundreds of contributors across the globe and thousands of end-users supporting +applications that serve millions. Every one is a star in our sky, helping us chart our course. + +The release logo is made by [Britnee Laverack](https://www.instagram.com/artsyfie/), and depicts a telescope set upon starry skies and the +[Pleiades](https://en.wikipedia.org/wiki/Pleiades), often known in mythology as the “Seven Sisters”. The number seven is especially auspicious +for the Kubernetes project, and is a reference back to our original “Project Seven” name. + +This release of Kubernetes is named for those that would look towards the night sky and wonder — for +all the stargazers out there. ✨ + +### User Highlights + +* Check out how leading retail e-commerce company [La Redoute used Kubernetes, alongside other CNCF projects, to transform and streamline its software delivery lifecycle](https://www.cncf.io/case-studies/la-redoute/) - from development to operations. +* Trying to ensure no change to an API call would cause any breaks, [Salt Security built its microservices entirely on Kubernetes, and it communicates via gRPC while Linkerd ensures messages are encrypted](https://www.cncf.io/case-studies/salt-security/). +* In their effort to migrate from private to public cloud, [Allainz Direct engineers redesigned its CI/CD pipeline in just three months while managing to condense 200 workflows down to 10-15](https://www.cncf.io/case-studies/allianz/). +* Check out how [Bink, a UK based fintech company, updated its in-house Kubernetes distribution with Linkerd to build a cloud-agnostic platform that scales as needed whilst allowing them to keep a close eye on performance and stability](https://www.cncf.io/case-studies/bink/). +* Using Kubernetes, the Dutch organization [Stichting Open Nederland](http://www.stichtingopennederland.nl/) created a testing portal in just one-and-a-half months to help safely reopen events in the Netherlands. The [Testing for Entry (Testen voor Toegang)](https://www.testenvoortoegang.org/) platform [leveraged the performance and scalability of Kubernetes to help individuals book over 400,000 COVID-19 testing appointments per day. ](https://www.cncf.io/case-studies/true/) +* Working alongside SparkFabrik and utilizing Backstage, [Santagostino created the developer platform Samaritan to centralize services and documentation, manage the entire lifecycle of services, and simplify the work of Santagostino developers](https://www.cncf.io/case-studies/santagostino/). + +### Ecosystem Updates + +* KubeCon + CloudNativeCon Europe 2022 will take place in Valencia, Spain, from 16 – 20 May 2022! You can find more information about the conference and registration on the [event site](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/). +* In the [2021 Cloud Native Survey](https://www.cncf.io/announcements/2022/02/10/cncf-sees-record-kubernetes-and-container-adoption-in-2021-cloud-native-survey/), the CNCF saw record Kubernetes and container adoption. Take a look at the [results of the survey](https://www.cncf.io/reports/cncf-annual-survey-2021/). +* The [Linux Foundation](https://www.linuxfoundation.org/) and [The Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF) announced the availability of a new [Cloud Native Developer Bootcamp](https://training.linuxfoundation.org/training/cloudnativedev-bootcamp/?utm_source=lftraining&utm_medium=pr&utm_campaign=clouddevbc0322) to provide participants with the knowledge and skills to design, build, and deploy cloud native applications. Check out the [announcement](https://www.cncf.io/announcements/2022/03/15/new-cloud-native-developer-bootcamp-provides-a-clear-path-to-cloud-native-careers/) to learn more. + +### Project Velocity + +The [CNCF K8s DevStats](https://k8s.devstats.cncf.io/d/12/dashboards?orgId=1&refresh=15m) project +aggregates a number of interesting data points related to the velocity of Kubernetes and various +sub-projects. This includes everything from individual contributions to the number of companies that +are contributing, and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem. + +In the v1.24 release cycle, which [ran for 17 weeks](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24) (January 10 to May 3), we saw contributions from [1029 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.23.0%20-%20v1.24.0&var-metric=contributions) and [1179 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.23.0%20-%20v1.24.0&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All&var-repo_name=kubernetes%2Fkubernetes). + +## Upcoming Release Webinar + +Join members of the Kubernetes 1.24 release team on Tue May 24, 2022 9:45am – 11am PT to learn about +the major features of this release, as well as deprecations and removals to help plan for upgrades. +For more information and registration, visit the [event page](https://community.cncf.io/e/mck3kd/) +on the CNCF Online Programs site. + +## Get Involved + +The simplest way to get involved with Kubernetes is by joining one of the many [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (SIGs) that align with your interests. +Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly [community meeting](https://github.com/kubernetes/community/tree/master/communication), and through the channels below: + +* Find out more about contributing to Kubernetes at the [Kubernetes Contributors](https://www.kubernetes.dev/) website +* Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for the latest updates +* Join the community discussion on [Discuss](https://discuss.kubernetes.io/) +* Join the community on [Slack](http://slack.k8s.io/) +* Post questions (or answer questions) on [Server Fault](https://serverfault.com/questions/tagged/kubernetes). +* Share your Kubernetes [story](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform) +* Read more about what’s happening with Kubernetes on the [blog](https://kubernetes.io/blog/) +* Learn more about the [Kubernetes Release Team](https://github.com/kubernetes/sig-release/tree/master/release-team) diff --git a/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md b/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md new file mode 100644 index 0000000000..c823ae8a2c --- /dev/null +++ b/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md @@ -0,0 +1,103 @@ +--- +layout: blog +title: "Kubernetes 1.24: Volume Expansion Now A Stable Feature" +date: 2022-05-05 +slug: volume-expansion-ga +--- + +**Author:** Hemant Kumar (Red Hat) + +Volume expansion was introduced as a alpha feature in Kubernetes 1.8 and it went beta in 1.11 and with Kubernetes 1.24 we are excited to announce general availability(GA) +of volume expansion. + +This feature allows Kubernetes users to simply edit their `PersistentVolumeClaim` objects and specify new size in PVC Spec and Kubernetes will automatically expand the volume +using storage backend and also expand the underlying file system in-use by the Pod without requiring any downtime at all if possible. + + +### How to use volume expansion + +You can trigger expansion for a PersistentVolume by editing the `spec` field of a PVC, specifying a different +(and larger) storage request. For example, given following PVC: + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: myclaim +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi # specify new size here +``` + +You can request expansion of the underlying PersistentVolume by specifying a new value instead of old `1Gi` size. +Once you've changed the requested size, watch the `status.conditions` field of the PVC to see if the +resize has completed. + +When Kubernetes starts expanding the volume - it will add `Resizing` condition to the PVC, which will be removed once expansion completes. More information about progress of +expansion operation can also be obtained by monitoring events associated with the PVC: + +``` +kubectl describe pvc <pvc> +``` + +### Storage driver support + +Not every volume type however is expandable by default. Some volume types such as - intree hostpath volumes are not expandable at all. For CSI volumes - the CSI driver +must have capability `EXPAND_VOLUME` in controller or node service (or both if appropriate). Please refer to documentation of your CSI driver, to find out +if it supports volume expansion. + +Please refer to volume expansion documentation for intree volume types which support volume expansion - [Expanding Persistent Volumes](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims). + + +In general to provide some degree of control over volumes that can be expanded, only dynamically provisioned PVCs whose storage class has `allowVolumeExpansion` parameter set to `true` are expandable. + +A Kubernetes cluster administrator must edit the appropriate StorageClass object and set +the `allowVolumeExpansion` field to `true`. For example: + +``` +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: gp2-default +provisioner: kubernetes.io/aws-ebs +parameters: + secretNamespace: "" + secretName: "" +allowVolumeExpansion: true +``` + +### Online expansion compared to offline expansion + +By default, Kubernetes attempts to expand volumes immediately after user requests a resize. +If one or more Pods are using the volume, Kubernetes tries to expands the volume using an online resize; +as a result volume expansion usually requires no application downtime. +Filesystem expansion on the node is also performed online and hence does not require shutting +down any Pod that was using the PVC. + +If you expand a PersistentVolume that is not in use, Kubernetes does an offline resize (and, +because the volume isn't in use, there is again no workload disruption). + +In some cases though - if underlying Storage Driver can only support offline expansion, users of the PVC must take down their Pod before expansion can succeed. Please refer to documentation of your storage +provider to find out - what mode of volume expansion it supports. + +When volume expansion was introduced as an alpha feature, Kubernetes only supported offline filesystem +expansion on the node and hence required users to restart their pods for file system resizing to finish. +His behaviour has been changed and Kubernetes tries its best to fulfil any resize request regardless +of whether the underlying PersistentVolume volume is online or offline. If your storage provider supports +online expansion then no Pod restart should be necessary for volume expansion to finish. + +## Next steps + +Although volume expansion is now stable as part of the recent v1.24 release, +SIG Storage are working to make it even simpler for users of Kubernetes to expand their persistent storage. +Kubernetes 1.23 introduced features for triggering recovery from failed volume expansion, allowing users +to attempt self-service healing after a failed resize. +See [Recovering from volume expansion failure](/docs/concepts/storage/persistent-volumes/#recovering-from-failure-when-expanding-volumes) for more details. + +The Kubernetes contributor community is also discussing the potential for StatefulSet-driven storage expansion. This proposed +feature would let you trigger expansion for all underlying PVs that are providing storage to a StatefulSet, +by directly editing the StatefulSet object. +See the [Support Volume Expansion Through StatefulSets](https://github.com/kubernetes/enhancements/issues/661) enhancement proposal for more details. diff --git a/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md new file mode 100644 index 0000000000..2bb85059e3 --- /dev/null +++ b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md @@ -0,0 +1,79 @@ +--- +layout: blog +title: "Kubernetes 1.24: Storage Capacity Tracking Now Generally Available" +date: 2022-05-06 +slug: storage-capacity-ga +--- + + **Authors:** Patrick Ohly (Intel) + +The v1.24 release of Kubernetes brings [storage capacity](/docs/concepts/storage/storage-capacity/) +tracking as a generally available feature. + +## Problems we have solved + +As explained in more detail in the [previous blog post about this +feature](/blog/2021/04/14/local-storage-features-go-beta/), storage capacity +tracking allows a CSI driver to publish information about remaining +capacity. The kube-scheduler then uses that information to pick suitable nodes +for a Pod when that Pod has volumes that still need to be provisioned. + +Without this information, a Pod may get stuck without ever being scheduled onto +a suitable node because kube-scheduler has to choose blindly and always ends up +picking a node for which the volume cannot be provisioned because the +underlying storage system managed by the CSI driver does not have sufficient +capacity left. + +Because CSI drivers publish storage capacity information that gets used at a +later time when it might not be up-to-date anymore, it can still happen that a +node is picked that doesn't work out after all. Volume provisioning recovers +from that by informing the scheduler that it needs to try again with a +different node. + +[Load +tests](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/storage-capacity-tracking.md) +that were done again for promotion to GA confirmed that all storage in a +cluster can be consumed by Pods with storage capacity tracking whereas Pods got +stuck without it. + +## Problems we have *not* solved + +Recovery from a failed volume provisioning attempt has one known limitation: if a Pod +uses two volumes and only one of them could be provisioned, then all future +scheduling decisions are limited by the already provisioned volume. If that +volume is local to a node and the other volume cannot be provisioned there, the +Pod is stuck. This problem pre-dates storage capacity tracking and while the +additional information makes it less likely to occur, it cannot be avoided in +all cases, except of course by only using one volume per Pod. + +An idea for solving this was proposed in a [KEP +draft](https://github.com/kubernetes/enhancements/pull/1703): volumes that were +provisioned and haven't been used yet cannot have any valuable data and +therefore could be freed and provisioned again elsewhere. SIG Storage is +looking for interested developers who want to continue working on this. + +Also not solved is support in Cluster Autoscaler for Pods with volumes. For CSI +drivers with storage capacity tracking, a prototype was developed and discussed +in [a PR](https://github.com/kubernetes/autoscaler/pull/3887). It was meant to +work with arbitrary CSI drivers, but that flexibility made it hard to configure +and slowed down scale up operations: because autoscaler was unable to simulate +volume provisioning, it only scaled the cluster by one node at a time, which +was seen as insufficient. + +Therefore that PR was not merged and a different approach with tighter coupling +between autoscaler and CSI driver will be needed. For this a better +understanding is needed about which local storage CSI drivers are used in +combination with cluster autoscaling. Should this lead to a new KEP, then users +will have to try out an implementation in practice before it can move to beta +or GA. So please reach out to SIG Storage if you have an interest in this +topic. + +## Acknowledgements + +Thanks a lot to the members of the community who have contributed to this +feature or given feedback including members of [SIG +Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling), +[SIG +Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling), +and of course [SIG +Storage](https://github.com/kubernetes/community/tree/master/sig-storage)! diff --git a/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md b/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md new file mode 100644 index 0000000000..5ff495410b --- /dev/null +++ b/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md @@ -0,0 +1,209 @@ +--- +layout: blog +title: "Kubernetes 1.24: gRPC container probes in beta" +date: 2022-05-13 +slug: grpc-probes-now-in-beta +--- + +**Author**: Sergey Kanzhelev (Google) + + +With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default. +Now you can configure startup, liveness, and readiness probes for your gRPC app +without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your your workload via gRPC and query its status. + +## Some history + +It's useful to let the system managing your workload check that the app is +healthy, has started OK, and whether the app considers itself good to accept +traffic. Before the gRPC support was added, Kubernetes already allowed you to +check for health based on running an executable from inside the container image, +by making an HTTP request, or by checking whether a TCP connection succeeded. + +For most apps, those checks are enough. If your app provides a gRPC endpoint +for a health (or readiness) check, it is easy +to repurpose the `exec` probe to use it for gRPC health checking. +In the blog article [Health checking gRPC servers on Kubernetes](/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/), +Ahmet Alp Balkan described how you can do that — a mechanism that still works today. + +There is a commonly used tool to enable this that was [created](https://github.com/grpc-ecosystem/grpc-health-probe/commit/2df4478982e95c9a57d5fe3f555667f4365c025d) +on August 21, 2018, and with +the first release at [Sep 19, 2018](https://github.com/grpc-ecosystem/grpc-health-probe/releases/tag/v0.1.0-alpha.1). + +This approach for gRPC apps health checking is very popular. There are [3,626 Dockerfiles](https://github.com/search?l=Dockerfile&q=grpc_health_probe&type=code) +with the `grpc_health_probe` and [6,621 yaml](https://github.com/search?l=YAML&q=grpc_health_probe&type=Code) files that are discovered with the +basic search on GitHub (at the moment of writing). This is good indication of the tool popularity +and the need to support this natively. + +Kubernetes v1.23 introduced an alpha-quality implementation of native support for +querying a workload status using gRPC. Because it was an alpha feature, +this was disabled by default for the v1.23 release. + +## Using the feature + +We built gRPC health checking in similar way with other probes and believe +it will be [easy to use](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) +if you are familiar with other probe types in Kubernetes. +The natively supported health probe has many benefits over the workaround involving `grpc_health_probe` executable. + +With the native gRPC support you don't need to download and carry `10MB` of an additional executable with your image. +Exec probes are generally slower than a gRPC call as they require instantiating a new process to run an executable. +It also makes the checks less sensible for edge cases when the pod is running at maximum resources and has troubles +instantiating new processes. + +There are a few limitations though. Since configuring a client certificate for probes is hard, +services that require client authentication are not supported. The built-in probes are also +not checking the server certificates and ignore related problems. + +Built-in checks also cannot be configured to ignore certain types of errors +(`grpc_health_probe` returns different exit codes for different errors), +and cannot be "chained" to run the health check on multiple services in a single probe. + +But all these limitations are quite standard for gRPC and there are easy workarounds +for those. + +## Try it for yourself + +### Cluster-level setup + +You can try this feature today. To try native gRPC probes, you can spin up a Kubernetes cluster +yourself with the `GRPCContainerProbe` feature gate enabled, there are many [tools available](/docs/tasks/tools/). + +Since the feature gate `GRPCContainerProbe` is enabled by default in 1.24, +many vendors will have this functionality working out of the box. +So you may just create an 1.24 cluster on platform of your choice. Some vendors +allow to enable alpha features on 1.23 clusters. + +For example, at the moment of writing, you can spin up the test cluster on GKE for a quick test. +Other vendors may also have similar capabilities, especially if you +are reading this blog post long after the Kubernetes 1.24 release. + +On GKE use the following command (note, version is `1.23` and `enable-kubernetes-alpha` are specified). + +```shell +gcloud container clusters create test-grpc \ + --enable-kubernetes-alpha \ + --no-enable-autorepair \ + --no-enable-autoupgrade \ + --release-channel=rapid \ + --cluster-version=1.23 +``` + +You will also need to configure `kubectl` to access the cluster: + +```shell +gcloud container clusters get-credentials test-grpc +``` + +### Trying the feature out + +Let's create the pod to test how gRPC probes work. For this test we will use the `agnhost` image. +This is a k8s maintained image with that can be used for all sorts of workload testing. +For example, it has a useful [grpc-health-checking](https://github.com/kubernetes/kubernetes/blob/b2c5bd2a278288b5ef19e25bf7413ecb872577a4/test/images/agnhost/README.md#grpc-health-checking) module +that exposes two ports - one is serving health checking service, +another - http port to react on commands `make-serving` and `make-not-serving`. + +Here is an example pod definition. It starts the `grpc-health-checking` module, +exposes ports `5000` and `8080`, and configures gRPC readiness probe: + +``` yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: test-grpc +spec: + containers: + - name: agnhost + image: k8s.gcr.io/e2e-test-images/agnhost:2.35 + command: ["/agnhost", "grpc-health-checking"] + ports: + - containerPort: 5000 + - containerPort: 8080 + readinessProbe: + grpc: + port: 5000 +``` + +If the file called `test.yaml`, you can create the pod and check it's status. +The pod will be in ready state as indicated by the snippet of the output. + +```shell +kubectl apply -f test.yaml +kubectl describe test-grpc +``` + +The output will contain something like this: + +``` +Conditions: + Type Status + Initialized True + Ready True + ContainersReady True + PodScheduled True +``` + +Now let's change the health checking endpoint status to NOT_SERVING. +In order to call the http port of the Pod, let's create a port forward: + +```shell +kubectl port-forward test-grpc 8080:8080 +``` + +You can `curl` to call the command... + +```shell +curl http://localhost:8080/make-not-serving +``` + +... and in a few seconds the port status will switch to not ready. + +```shell +kubectl describe pod test-grpc +``` + +The output now will have: + +``` +Conditions: + Type Status + Initialized True + Ready False + ContainersReady False + PodScheduled True + +... + + Warning Unhealthy 2s (x6 over 42s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING") +``` + +Once it is switched back, in about one second the Pod will get back to ready status: + +``` bsh +curl http://localhost:8080/make-serving +kubectl describe test-grpc +``` + +The output indicates that the Pod went back to being `Ready`: + +``` +Conditions: + Type Status + Initialized True + Ready True + ContainersReady True + PodScheduled True +``` + +This new built-in gRPC health probing on Kubernetes makes implementing a health-check via gRPC +much easier than the older approach that relied on using a separate `exec` probe. Read through +the official +[documentation](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) +to learn more and provide feedback before the feature will be promoted to GA. + +## Summary + +Kubernetes is a popular workload orchestration platform and we add features based on feedback and demand. +Features like gRPC probes support is a minor improvement that will make life of many app developers +easier and apps more resilient. Try it today and give feedback, before the feature went into GA. diff --git a/content/en/blog/_posts/2022-05-16-volume-populators-beta.md b/content/en/blog/_posts/2022-05-16-volume-populators-beta.md new file mode 100644 index 0000000000..4558f07eae --- /dev/null +++ b/content/en/blog/_posts/2022-05-16-volume-populators-beta.md @@ -0,0 +1,162 @@ +--- +layout: blog +title: "Kubernetes 1.24: Volume Populators Graduate to Beta" +date: 2022-05-16 +slug: volume-populators-beta +--- + +**Author:** +Ben Swartzlander (NetApp) + +The volume populators feature is now two releases old and entering beta! The `AnyVolumeDataSource` feature +gate defaults to enabled in Kubernetes v1.24, which means that users can specify any custom resource +as the data source of a PVC. + +An [earlier blog article](/blog/2021/08/30/volume-populators-redesigned/) detailed how the +volume populators feature works. In short, a cluster administrator can install a CRD and +associated populator controller in the cluster, and any user who can create instances of +the CR can create pre-populated volumes by taking advantage of the populator. + +Multiple populators can be installed side by side for different purposes. The SIG storage +community is already seeing some implementations in public, and more prototypes should +appear soon. + +Cluster administrations are **strongly encouraged** to install the +volume-data-source-validator controller and associated `VolumePopulator` CRD before installing +any populators so that users can get feedback about invalid PVC data sources. + +## New Features + +The [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library +on which populators are built now includes metrics to help operators monitor and detect +problems. This library is now beta and latest release is v1.0.1. + +The [volume data source validator](https://github.com/kubernetes-csi/volume-data-source-validator) +controller also has metrics support added, and is in beta. The `VolumePopulator` CRD is +beta and the latest release is v1.0.1. + +## Trying it out + +To see how this works, you can install the sample "hello" populator and try it +out. + +First install the volume-data-source-validator controller. + +```shell +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/client/config/crd/populator.storage.k8s.io_volumepopulators.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/rbac-data-source-validator.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/setup-data-source-validator.yaml +``` + +Next install the example populator. + +```shell +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/v1.0.1/example/hello-populator/crd.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/87a47467b86052819e9ad13d15036d65b9a32fbb/example/hello-populator/deploy.yaml +``` + +Your cluster now has a new CustomResourceDefinition that provides a test API named Hello. +Create an instance of the `Hello` custom resource, with some text: + +```yaml +apiVersion: hello.example.com/v1alpha1 +kind: Hello +metadata: + name: example-hello +spec: + fileName: example.txt + fileContents: Hello, world! +``` + +Create a PVC that refers to that CR as its data source. + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: example-pvc +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Mi + dataSourceRef: + apiGroup: hello.example.com + kind: Hello + name: example-hello + volumeMode: Filesystem +``` + +Next, run a Job that reads the file in the PVC. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: example-job +spec: + template: + spec: + containers: + - name: example-container + image: busybox:latest + command: + - cat + - /mnt/example.txt + volumeMounts: + - name: vol + mountPath: /mnt + restartPolicy: Never + volumes: + - name: vol + persistentVolumeClaim: + claimName: example-pvc +``` + +Wait for the job to complete (including all of its dependencies). + +```shell +kubectl wait --for=condition=Complete job/example-job +``` + +And last examine the log from the job. + +```shell +kubectl logs job/example-job +``` + +The output should be: + +```terminal +Hello, world! +``` + +Note that the volume already contained a text file with the string contents from +the CR. This is only the simplest example. Actual populators can set up the volume +to contain arbitrary contents. + +## How to write your own volume populator + +Developers interested in writing new poplators are encouraged to use the +[lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library +and to only supply a small controller wrapper around the library, and a pod image +capable of attaching to volumes and writing the appropriate data to the volume. + +Individual populators can be extremely generic such that they work with every type +of PVC, or they can do vendor specific things to rapidly fill a volume with data +if the volume was provisioned by a specific CSI driver from the same vendor, for +example, by communicating directly with the storage for that volume. + +## How can I learn more? + +The enhancement proposal, +[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation +of this feature. + +[Volume populators and data sources](/docs/concepts/storage/persistent-volumes/#volume-populators-and-data-sources), within the documentation topic about persistent volumes, +explains how to use this feature in your cluster. + +Please get involved by joining the Kubernetes storage SIG to help us enhance this +feature. There are a lot of good ideas already and we'd be thrilled to have more! + diff --git a/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md b/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md new file mode 100644 index 0000000000..920d578d01 --- /dev/null +++ b/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md @@ -0,0 +1,117 @@ +--- +layout: blog +title: 'Kubernetes 1.24: Prevent unauthorised volume mode conversion' +date: 2022-05-18 +slug: prevent-unauthorised-volume-mode-conversion-alpha +--- + +**Author:** Raunak Pradip Shah (Mirantis) + +Kubernetes v1.24 introduces a new alpha-level feature that prevents unauthorised users +from modifying the volume mode of a [`PersistentVolumeClaim`](/docs/concepts/storage/persistent-volumes/) created from an +existing [`VolumeSnapshot`](/docs/concepts/storage/volume-snapshots/) in the Kubernetes cluster. + + + +### The problem + +The [Volume Mode](/docs/concepts/storage/persistent-volumes/#volume-mode) determines whether a volume +is formatted into a filesystem or presented as a raw block device. + +Users can leverage the `VolumeSnapshot` feature, which has been stable since Kubernetes v1.20, +to create a `PersistentVolumeClaim` (shortened as PVC) from an existing `VolumeSnapshot` in +the Kubernetes cluster. The PVC spec includes a `dataSource` field, which can point to an +existing `VolumeSnapshot` instance. +Visit [Create a PersistentVolumeClaim from a Volume Snapshot](/docs/concepts/storage/persistent-volumes/#create-persistent-volume-claim-from-volume-snapshot) for more details. + +When leveraging the above capability, there is no logic that validates whether the mode of the +original volume, whose snapshot was taken, matches the mode of the newly created volume. + +This presents a security gap that allows malicious users to potentially exploit an +as-yet-unknown vulnerability in the host operating system. + +Many popular storage backup vendors convert the volume mode during the course of a +backup operation, for efficiency purposes, which prevents Kubernetes from blocking +the operation completely and presents a challenge in distinguishing trusted +users from malicious ones. + +### Preventing unauthorised users from converting the volume mode + +In this context, an authorised user is one who has access rights to perform `Update` +or `Patch` operations on `VolumeSnapshotContents`, which is a cluster-level resource. +It is upto the cluster administrator to provide these rights only to trusted users +or applications, like backup vendors. + +If the alpha feature is [enabled](https://kubernetes-csi.github.io/docs/) in +`snapshot-controller`, `snapshot-validation-webhook` and `external-provisioner`, +then unauthorised users will not be allowed to modify the volume mode of a PVC +when it is being created from a `VolumeSnapshot`. + +To convert the volume mode, an authorised user must do the following: + +1. Identify the `VolumeSnapshot` that is to be used as the data source for a newly +created PVC in the given namespace. +2. Identify the `VolumeSnapshotContent` bound to the above `VolumeSnapshot`. + + ```shell + kubectl get volumesnapshot -n <namespace> + ``` + +3. Add the annotation [`snapshot.storage.kubernetes.io/allowVolumeModeChange`](/docs/reference/labels-annotations-taints/#snapshot-storage-kubernetes-io-allowvolumemodechange) +to the `VolumeSnapshotContent`. + +4. This annotation can be added either via software or manually by the authorised +user. The `VolumeSnapshotContent` annotation must look like following manifest fragment: + + ```yaml + kind: VolumeSnapshotContent + metadata: + annotations: + - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true" + ... + ``` + +**Note**: For pre-provisioned `VolumeSnapshotContents`, you must take an extra +step of setting `spec.sourceVolumeMode` field to either `Filesystem` or `Block`, +depending on the mode of the volume from which this snapshot was taken. + +An example is shown below: + + ```yaml + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshotContent + metadata: + annotations: + - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true" + name: new-snapshot-content-test + spec: + deletionPolicy: Delete + driver: hostpath.csi.k8s.io + source: + snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002 + sourceVolumeMode: Filesystem + volumeSnapshotRef: + name: new-snapshot-test + namespace: default + ``` + +Repeat steps 1 to 3 for all `VolumeSnapshotContents` whose volume mode needs to be +converted during a backup or restore operation. + +If the annotation shown in step 4 above is present on a `VolumeSnapshotContent` +object, Kubernetes will not prevent the volume mode from being converted. +Users should keep this in mind before they attempt to add the annotation +to any `VolumeSnapshotContent`. + + +### What's next + +[Enable this feature](https://kubernetes-csi.github.io/docs/) and let us know +what you think! + +We hope this feature causes no disruption to existing workflows while preventing +malicious users from exploiting security vulnerabilities in their clusters. + +For any queries or issues, join [Kubernetes on Slack](https://slack.k8s.io/) and +create a thread in the #sig-storage channel. Alternately, create an issue in the +CSI external-snapshotter [repository](https://github.com/kubernetes-csi/external-snapshotter). \ No newline at end of file diff --git a/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md b/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md new file mode 100644 index 0000000000..f8f4876285 --- /dev/null +++ b/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md @@ -0,0 +1,96 @@ +--- +layout: blog +title: "Kubernetes 1.24: Introducing Non-Graceful Node Shutdown Alpha" +date: 2022-05-20 +slug: kubernetes-1-24-non-graceful-node-shutdown-alpha +--- + +**Authors** Xing Yang and Yassine Tijani (VMware) + +Kubernetes v1.24 introduces alpha support for [Non-Graceful Node Shutdown](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown). This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as hardware failure or broken OS. + +## How is this different from Graceful Node Shutdown + +You might have heard about the [Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) capability of Kubernetes, +and are wondering how the Non-Graceful Node Shutdown feature is different from that. Graceful Node Shutdown +allows Kubernetes to detect when a node is shutting down cleanly, and handles that situation appropriately. +A Node Shutdown can be "graceful" only if the node shutdown action can be detected by the kubelet ahead +of the actual shutdown. However, there are cases where a node shutdown action may not be detected by +the kubelet. This could happen either because the shutdown command does not trigger the systemd inhibitor +locks mechanism that kubelet relies upon, or because of a configuration error +(the `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are not configured properly). + +Graceful node shutdown relies on Linux-specific support. The kubelet does not watch for upcoming +shutdowns on Windows nodes (this may change in a future Kubernetes release). + +When a node is shutdown but without the kubelet detecting it, pods on that node +also shut down ungracefully. For stateless apps, that's often not a problem (a ReplicaSet adds a new pod once +the cluster detects that the affected node or pod has failed). For stateful apps, the story is more complicated. +If you use a StatefulSet and have a pod from that StatefulSet on a node that fails uncleanly, that affected pod +will be marked as terminating; the StatefulSet cannot create a replacement pod because the pod +still exists in the cluster. +As a result, the application running on the StatefulSet may be degraded or even offline. If the original, shut +down node comes up again, the kubelet on that original node reports in, deletes the existing pods, and +the control plane makes a replacement pod for that StatefulSet on a different running node. +If the original node has failed and does not come up, those stateful pods would be stuck in a +terminating status on that failed node indefinitely. + +``` +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +web-0 1/1 Running 0 100m 10.244.2.4 k8s-node-876-1639279816 <none> <none> +web-1 1/1 Terminating 0 100m 10.244.1.3 k8s-node-433-1639279804 <none> <none> +``` + +## Try out the new non-graceful shutdown handling + +To use the non-graceful node shutdown handling, you must enable the `NodeOutOfServiceVolumeDetach` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the `kube-controller-manager` +component. + +In the case of a node shutdown, you can manually taint that node as out of service. You should make certain that +the node is truly shutdown (not in the middle of restarting) before you add that taint. You could add that +taint following a shutdown that the kubelet did not detect and handle in advance; another case where you +can use that taint is when the node is in a non-recoverable state due to a hardware failure or a broken OS. +The values you set for that taint can be `node.kubernetes.io/out-of-service=nodeshutdown: "NoExecute"` +or `node.kubernetes.io/out-of-service=nodeshutdown:" NoSchedule"`. +Provided you have enabled the feature gate mentioned earlier, setting the out-of-service taint on a Node +means that pods on the node will be deleted unless if there are matching tolerations on the pods. +Persistent volumes attached to the shutdown node will be detached, and for StatefulSets, replacement pods will +be created successfully on a different running node. + +``` +$ kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute + +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +web-0 1/1 Running 0 150m 10.244.2.4 k8s-node-876-1639279816 <none> <none> +web-1 1/1 Running 0 10m 10.244.1.7 k8s-node-433-1639279804 <none> <none> +``` + +Note: Before applying the out-of-service taint, you **must** verify that a node is already in shutdown or power off state (not in the middle of restarting), either because the user intentionally shut it down or the node is down due to hardware failures, OS issues, etc. + +Once all the workload pods that are linked to the out-of-service node are moved to a new running node, and the shutdown node has been recovered, you should remove +that taint on the affected node after the node is recovered. +If you know that the node will not return to service, you could instead delete the node from the cluster. + +## What’s next? + +Depending on feedback and adoption, the Kubernetes team plans to push the Non-Graceful Node Shutdown implementation to Beta in either 1.25 or 1.26. + +This feature requires a user to manually add a taint to the node to trigger workloads failover and remove the taint after the node is recovered. In the future, we plan to find ways to automatically detect and fence nodes that are shutdown/failed and automatically failover workloads to another node. + +## How can I learn more? + +Check out the [documentation](/docs/concepts/architecture/nodes/#non-graceful-node-shutdown) +for non-graceful node shutdown. + +## How to get involved? + +This feature has a long story. Yassine Tijani ([yastij](https://github.com/yastij)) started the KEP more than two years ago. Xing Yang ([xing-yang](https://github.com/xing-yang)) continued to drive the effort. There were many discussions among SIG Storage, SIG Node, and API reviewers to nail down the design details. Ashutosh Kumar ([sonasingh46](https://github.com/sonasingh46)) did most of the implementation and brought it to Alpha in Kubernetes 1.24. + +We want to thank the following people for their insightful reviews: Tim Hockin ([thockin](https://github.com/thockin)) for his guidance on the design, Jing Xu ([jingxu97](https://github.com/jingxu97)), Hemant Kumar ([gnufied](https://github.com/gnufied)), and Michelle Au ([msau42](https://github.com/msau42)) for reviews from SIG Storage side, and Mrunal Patel ([mrunalp](https://github.com/mrunalp)), David Porter ([bobbypage](https://github.com/bobbypage)), Derek Carr ([derekwaynecarr](https://github.com/derekwaynecarr)), and Danielle Endocrimes ([endocrimes](https://github.com/endocrimes)) for reviews from SIG Node side. + +There are many people who have helped review the design and implementation along the way. We want to thank everyone who has contributed to this effort including the about 30 people who have reviewed the [KEP](https://github.com/kubernetes/enhancements/pull/1116) and implementation over the last couple of years. + +This feature is a collaboration between SIG Storage and SIG Node. For those interested in getting involved with the design and development of any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). For those interested in getting involved with the design and development of the components that support the controlled interactions between pods and host resources, join the [Kubernetes Node SIG](https://github.com/kubernetes/community/tree/master/sig-node). diff --git a/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md b/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md new file mode 100644 index 0000000000..c17453a6de --- /dev/null +++ b/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md @@ -0,0 +1,137 @@ +--- +layout: blog +title: "Kubernetes 1.24: Avoid Collisions Assigning IP Addresses to Services" +date: 2022-05-23 +slug: service-ip-dynamic-and-static-allocation +--- + +**Author:** Antonio Ojea (Red Hat) + + +In Kubernetes, [Services](/docs/concepts/services-networking/service/) are an abstract way to expose +an application running on a set of Pods. Services +can have a cluster-scoped virtual IP address (using a Service of `type: ClusterIP`). +Clients can connect using that virtual IP address, and Kubernetes then load-balances traffic to that +Service across the different backing Pods. + +## How Service ClusterIPs are allocated? + +A Service `ClusterIP` can be assigned: + +_dynamically_ +: the cluster's control plane automatically picks a free IP address from within the configured IP range for `type: ClusterIP` Services. + +_statically_ +: you specify an IP address of your choice, from within the configured IP range for Services. + +Across your whole cluster, every Service `ClusterIP` must be unique. +Trying to create a Service with a specific `ClusterIP` that has already +been allocated will return an error. + +## Why do you need to reserve Service Cluster IPs? + +Sometimes you may want to have Services running in well-known IP addresses, so other components and +users in the cluster can use them. + +The best example is the DNS Service for the cluster. Some Kubernetes installers assign the 10th address from +the Service IP range to the DNS service. Assuming you configured your cluster with Service IP range +10.96.0.0/16 and you want your DNS Service IP to be 10.96.0.10, you'd have to create a Service like +this: + +```yaml +apiVersion: v1 +kind: Service +metadata: + labels: + k8s-app: kube-dns + kubernetes.io/cluster-service: "true" + kubernetes.io/name: CoreDNS + name: kube-dns + namespace: kube-system +spec: + clusterIP: 10.96.0.10 + ports: + - name: dns + port: 53 + protocol: UDP + targetPort: 53 + - name: dns-tcp + port: 53 + protocol: TCP + targetPort: 53 + selector: + k8s-app: kube-dns + type: ClusterIP +``` + +but as I explained before, the IP address 10.96.0.10 has not been reserved; if other Services are created +before or in parallel with dynamic allocation, there is a chance they can allocate this IP, hence, +you will not be able to create the DNS Service because it will fail with a conflict error. + +## How can you avoid Service ClusterIP conflicts? {#avoid-ClusterIP-conflict} + +In Kubernetes 1.24, you can enable a new feature gate `ServiceIPStaticSubrange`. +Turning this on allows you to use a different IP +allocation strategy for Services, reducing the risk of collision. + +The `ClusterIP` range will be divided, based on the formula `min(max(16, cidrSize / 16), 256)`, +described as _never less than 16 or more than 256 with a graduated step between them_. + +Dynamic IP assignment will use the upper band by default, once this has been exhausted it will +use the lower range. This will allow users to use static allocations on the lower band with a low +risk of collision. + +Examples: + +#### Service IP CIDR block: 10.96.0.0/24 + +Range Size: 2<sup>8</sup> - 2 = 254 +Band Offset: `min(max(16, 256/16), 256)` = `min(16, 256)` = 16 +Static band start: 10.96.0.1 +Static band end: 10.96.0.16 +Range end: 10.96.0.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/24 + "Static" : 16 + "Dynamic" : 238 +{{< /mermaid >}} + +#### Service IP CIDR block: 10.96.0.0/20 + +Range Size: 2<sup>12</sup> - 2 = 4094 +Band Offset: `min(max(16, 4096/16), 256)` = `min(256, 256)` = 256 +Static band start: 10.96.0.1 +Static band end: 10.96.1.0 +Range end: 10.96.15.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/20 + "Static" : 256 + "Dynamic" : 3838 +{{< /mermaid >}} + +#### Service IP CIDR block: 10.96.0.0/16 + +Range Size: 2<sup>16</sup> - 2 = 65534 +Band Offset: `min(max(16, 65536/16), 256)` = `min(4096, 256)` = 256 +Static band start: 10.96.0.1 +Static band ends: 10.96.1.0 +Range end: 10.96.255.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/16 + "Static" : 256 + "Dynamic" : 65278 +{{< /mermaid >}} + +## Get involved with SIG Network + +The current SIG-Network [KEPs](https://github.com/orgs/kubernetes/projects/10) and [issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fnetwork) on GitHub illustrate the SIG’s areas of emphasis. + +[SIG Network meetings](https://github.com/kubernetes/community/tree/master/sig-network) are a friendly, welcoming venue for you to connect with the community and share your ideas. +Looking forward to hearing from you! + diff --git a/content/en/blog/_posts/2022-05-25-contextual-logging/index.md b/content/en/blog/_posts/2022-05-25-contextual-logging/index.md new file mode 100644 index 0000000000..2d5ef5c4c7 --- /dev/null +++ b/content/en/blog/_posts/2022-05-25-contextual-logging/index.md @@ -0,0 +1,251 @@ +--- +layout: blog +title: "Contextual Logging in Kubernetes 1.24" +date: 2022-05-25 +slug: contextual-logging +canonicalUrl: https://kubernetes.dev/blog/2022/05/25/contextual-logging/ +--- + + **Authors:** Patrick Ohly (Intel) + +The [Structured Logging Working +Group](https://github.com/kubernetes/community/blob/master/wg-structured-logging/README.md) +has added new capabilities to the logging infrastructure in Kubernetes +1.24. This blog post explains how developers can take advantage of those to +make log output more useful and how they can get involved with improving Kubernetes. + +## Structured logging + +The goal of [structured +logging](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1602-structured-logging/README.md) +is to replace C-style formatting and the resulting opaque log strings with log +entries that have a well-defined syntax for storing message and parameters +separately, for example as a JSON struct. + +When using the traditional klog text output format for structured log calls, +strings were originally printed with `\n` escape sequences, except when +embedded inside a struct. For structs, log entries could still span multiple +lines, with no clean way to split the log stream into individual entries: + +``` +I1112 14:06:35.783529 328441 structured_logging.go:51] "using InfoS" longData={Name:long Data:Multiple +lines +with quite a bit +of text. internal:0} +I1112 14:06:35.783549 328441 structured_logging.go:52] "using InfoS with\nthe message across multiple lines" int=1 stringData="long: Multiple\nlines\nwith quite a bit\nof text." str="another value" +``` + +Now, the `<` and `>` markers along with indentation are used to ensure that splitting at a +klog header at the start of a line is reliable and the resulting output is human-readable: + +``` +I1126 10:31:50.378204 121736 structured_logging.go:59] "using InfoS" longData=< + {Name:long Data:Multiple + lines + with quite a bit + of text. internal:0} + > +I1126 10:31:50.378228 121736 structured_logging.go:60] "using InfoS with\nthe message across multiple lines" int=1 stringData=< + long: Multiple + lines + with quite a bit + of text. + > str="another value" +``` + +Note that the log message itself is printed with quoting. It is meant to be a +fixed string that identifies a log entry, so newlines should be avoided there. + +Before Kubernetes 1.24, some log calls in kube-scheduler still used `klog.Info` +for multi-line strings to avoid the unreadable output. Now all log calls have +been updated to support structured logging. + +## Contextual logging + +[Contextual logging](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/3077-contextual-logging/README.md) +is based on the [go-logr API](https://github.com/go-logr/logr#a-minimal-logging-api-for-go). The key +idea is that libraries are passed a logger instance by their caller and use +that for logging instead of accessing a global logger. The binary decides about +the logging implementation, not the libraries. The go-logr API is designed +around structured logging and supports attaching additional information to a +logger. + +This enables additional use cases: + +- The caller can attach additional information to a logger: + - [`WithName`](https://pkg.go.dev/github.com/go-logr/logr#Logger.WithName) adds a prefix + - [`WithValues`](https://pkg.go.dev/github.com/go-logr/logr#Logger.WithValues) adds key/value pairs + + When passing this extended logger into a function and a function uses it + instead of the global logger, the additional information is + then included in all log entries, without having to modify the code that + generates the log entries. This is useful in highly parallel applications + where it can become hard to identify all log entries for a certain operation + because the output from different operations gets interleaved. + +- When running unit tests, log output can be associated with the current test. + Then when a test fails, only the log output of the failed test gets shown + by `go test`. That output can also be more verbose by default because it + will not get shown for successful tests. Tests can be run in parallel + without interleaving their output. + +One of the design decisions for contextual logging was to allow attaching a +logger as value to a `context.Context`. Since the logger encapsulates all +aspects of the intended logging for the call, it is *part* of the context and +not just *using* it. A practical advantage is that many APIs already have a +`ctx` parameter or adding one has additional advantages, like being able to get +rid of `context.TODO()` calls inside the functions. + +Another decision was to not break compatibility with klog v2: + +- Libraries that use the traditional klog logging calls in a binary that has + set up contextual logging will work and log through the logging backend + chosen by the binary. However, such log output will not include the + additional information and will not work well in unit tests, so libraries + should be modified to support contextual logging. The [migration guide](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/migration-to-structured-logging.md) + for structured logging has been extended to also cover contextual logging. + +- When a library supports contextual logging and retrieves a logger from its + context, it will still work in a binary that does not initialize contextual + logging because it will get a logger that logs through klog. + +In Kubernetes 1.24, contextual logging is a new alpha feature with +`ContextualLogging` as feature gate. When disabled (the default), the new klog +API calls for contextual logging (see below) become no-ops to avoid performance +or functional regressions. + +No Kubernetes component has been converted yet. An [example program](https://github.com/kubernetes/kubernetes/blob/v1.24.0-beta.0/staging/src/k8s.io/component-base/logs/example/cmd/logger.go) +in the Kubernetes repository demonstrates how to enable contextual logging in a +binary and how the output depends on the binary's parameters: + +```console +$ cd $GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/ +$ go run . --help +... + --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: + AllAlpha=true|false (ALPHA - default=false) + AllBeta=true|false (BETA - default=false) + ContextualLogging=true|false (ALPHA - default=false) +$ go run . --feature-gates ContextualLogging=true +... +I0404 18:00:02.916429 451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s" +I0404 18:00:02.916447 451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s" +``` + +The `example` prefix and `foo="bar"` were added by the caller of the function +which logs the `runtime` message and `duration="1m0s"` value. + +The sample code for klog includes an +[example](https://github.com/kubernetes/klog/blob/v2.60.1/ktesting/example/example_test.go) +for a unit test with per-test output. + +## klog enhancements + +### Contextual logging API + +The following calls manage the lookup of a logger: + +[`FromContext`](https://pkg.go.dev/k8s.io/klog/v2#FromContext) +: from a `context` parameter, with fallback to the global logger + +[`Background`](https://pkg.go.dev/k8s.io/klog/v2#Background) +: the global fallback, with no intention to support contextual logging + +[`TODO`](https://pkg.go.dev/k8s.io/klog/v2#TODO) +: the global fallback, but only as a temporary solution until the function gets extended to accept + a logger through its parameters + +[`SetLoggerWithOptions`](https://pkg.go.dev/k8s.io/klog/v2#SetLoggerWithOptions) +: changes the fallback logger; when called with [`ContextualLogger(true)`](https://pkg.go.dev/k8s.io/klog/v2#ContextualLogger), + the logger is ready to be called directly, in which case logging will be done + without going through klog + +To support the feature gate mechanism in Kubernetes, klog has wrapper calls for +the corresponding go-logr calls and a global boolean controlling their behavior: + +- [`LoggerWithName`](https://pkg.go.dev/k8s.io/klog/v2#LoggerWithName) +- [`LoggerWithValues`](https://pkg.go.dev/k8s.io/klog/v2#LoggerWithValues) +- [`NewContext`](https://pkg.go.dev/k8s.io/klog/v2#NewContext) +- [`EnableContextualLogging`](https://pkg.go.dev/k8s.io/klog/v2#EnableContextualLogging) + +Usage of those functions in Kubernetes code is enforced with a linter +check. The klog default for contextual logging is to enable the functionality +because it is considered stable in klog. It is only in Kubernetes binaries +where that default gets overridden and (in some binaries) controlled via the +`--feature-gate` parameter. + +### ktesting logger + +The new [ktesting](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting) package +implements logging through `testing.T` using klog's text output format. It has +a [single API call](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting#NewTestContext) for +instrumenting a test case and [support for command line flags](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting/init). + +### klogr + +[`klog/klogr`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/klogr) continues to be +supported and it's default behavior is unchanged: it formats structured log +entries using its own, custom format and prints the result via klog. + +However, this usage is discouraged because that format is neither +machine-readable (in contrast to real JSON output as produced by zapr, the +go-logr implementation used by Kubernetes) nor human-friendly (in contrast to +the klog text format). + +Instead, a klogr instance should be created with +[`WithFormat(FormatKlog)`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/klogr#WithFormat) +which chooses the klog text format. A simpler construction method with the same +result is the new +[`klog.NewKlogr`](https://pkg.go.dev/k8s.io/klog/v2#NewKlogr). That is the +logger that klog returns as fallback when nothing else is configured. + +### Reusable output test + +A lot of go-logr implementations have very similar unit tests where they check +the result of certain log calls. If a developer didn't know about certain +caveats like for example a `String` function that panics when called, then it +is likely that both the handling of such caveats and the unit test are missing. + +[`klog.test`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/test) is a reusable set +of test cases that can be applied to a go-logr implementation. + +### Output flushing + +klog used to start a goroutine unconditionally during `init` which flushed +buffered data at a hard-coded interval. Now that goroutine is only started on +demand (i.e. when writing to files with buffering) and can be controlled with +[`StopFlushDaemon`](https://pkg.go.dev/k8s.io/klog/v2#StopFlushDaemon) and +[`StartFlushDaemon`](https://pkg.go.dev/k8s.io/klog/v2#StartFlushDaemon). + +When a go-logr implementation buffers data, flushing that data can be +integrated into [`klog.Flush`](https://pkg.go.dev/k8s.io/klog/v2#Flush) by +registering the logger with the +[`FlushLogger`](https://pkg.go.dev/k8s.io/klog/v2#FlushLogger) option. + +### Various other changes + +For a description of all other enhancements see in the [release notes](https://github.com/kubernetes/klog/releases). + +## logcheck + +Originally designed as a linter for structured log calls, the + [`logcheck`](https://github.com/kubernetes/klog/tree/788efcdee1e9be0bfbe5b076343d447314f2377e/hack/tools/logcheck) +tool has been enhanced to support also contextual logging and traditional klog +log calls. These enhanced checks already found bugs in Kubernetes, like calling +`klog.Info` instead of `klog.Infof` with a format string and parameters. + +It can be included as a plugin in a `golangci-lint` invocation, which is how +[Kubernetes uses it now](https://github.com/kubernetes/kubernetes/commit/17e3c555c5115f8c9176bae10ba45baa04d23a7b), +or get invoked stand-alone. + +We are in the process of [moving the tool](https://github.com/kubernetes/klog/issues/312) into a new repository because it isn't +really related to klog and its releases should be tracked and tagged properly. + +## Next steps + +The [Structured Logging WG](https://github.com/kubernetes/community/tree/master/wg-structured-logging) +is always looking for new contributors. The migration +away from C-style logging is now going to target structured, contextual logging +in one step to reduce the overall code churn and number of PRs. Changing log +calls is good first contribution to Kubernetes and an opportunity to get to +know code in various different areas. diff --git a/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md b/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md new file mode 100644 index 0000000000..aa6257eb3e --- /dev/null +++ b/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md @@ -0,0 +1,148 @@ +--- +layout: blog +title: 'Kubernetes 1.24: Maximum Unavailable Replicas for StatefulSet' +date: 2022-05-27 +slug: maxunavailable-for-statefulset +--- + +**Author:** Mayank Kumar (Salesforce) + +Kubernetes [StatefulSets](/docs/concepts/workloads/controllers/statefulset/), since their introduction in +1.5 and becoming stable in 1.9, have been widely used to run stateful applications. They provide stable pod identity, persistent +per pod storage and ordered graceful deployment, scaling and rolling updates. You can think of StatefulSet as the atomic building +block for running complex stateful applications. As the use of Kubernetes has grown, so has the number of scenarios requiring +StatefulSets. Many of these scenarios, require faster rolling updates than the currently supported one-pod-at-a-time updates, in the +case where you're using the `OrderedReady` Pod management policy for a StatefulSet. + + +Here are some examples: + +- I am using a StatefulSet to orchestrate a multi-instance, cache based application where the size of the cache is large. The cache + starts cold and requires some siginificant amount of time before the container can start. There could be more initial startup tasks + that are required. A RollingUpdate on this StatefulSet would take a lot of time before the application is fully updated. If the + StatefulSet supported updating more than one pod at a time, it would result in a much faster update. + +- My stateful application is composed of leaders and followers or one writer and multiple readers. I have multiple readers or + followers and my application can tolerate multiple pods going down at the same time. I want to update this application more than + one pod at a time so that i get the new updates rolled out quickly, especially if the number of instances of my application are + large. Note that my application still requires unique identity per pod. + + +In order to support such scenarios, Kubernetes 1.24 includes a new alpha feature to help. Before you can use the new feature you must +enable the `MaxUnavailableStatefulSet` feature flag. Once you enable that, you can specify a new field called `maxUnavailable`, part +of the `spec` for a StatefulSet. For example: + +``` +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: web + namespace: default +spec: + podManagementPolicy: OrderedReady # you must set OrderedReady + replicas: 5 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - image: k8s.gcr.io/nginx-slim:0.8 + imagePullPolicy: IfNotPresent + name: nginx + updateStrategy: + rollingUpdate: + maxUnavailable: 2 # this is the new alpha field, whose default value is 1 + partition: 0 + type: RollingUpdate +``` + +If you enable the new feature and you don't specify a value for `maxUnavailable` in a StatefulSet, Kubernetes applies a default +`maxUnavailable: 1`. This matches the behavior you would see if you don't enable the new feature. + +I'll run through a scenario based on that example manifest to demonstrate how this feature works. I will deploy a StatefulSet that +has 5 replicas, with `maxUnavailable` set to 2 and `partition` set to 0. + +I can trigger a rolling update by changing the image to `k8s.gcr.io/nginx-slim:0.9`. Once I initiate the rolling update, I can +watch the pods update 2 at a time as the current value of maxUnavailable is 2. The below output shows a span of time and is not +complete. The maxUnavailable can be an absolute number (for example, 2) or a percentage of desired Pods (for example, 10%). The +absolute number is calculated from percentage by rounding down. +``` +kubectl get pods --watch +``` + +``` +NAME READY STATUS RESTARTS AGE +web-0 1/1 Running 0 85s +web-1 1/1 Running 0 2m6s +web-2 1/1 Running 0 106s +web-3 1/1 Running 0 2m47s +web-4 1/1 Running 0 2m27s +web-4 1/1 Terminating 0 5m43s ----> start terminating 4 +web-3 1/1 Terminating 0 6m3s ----> start terminating 3 +web-3 0/1 Terminating 0 6m7s +web-3 0/1 Pending 0 0s +web-3 0/1 Pending 0 0s +web-4 0/1 Terminating 0 5m48s +web-4 0/1 Terminating 0 5m48s +web-3 0/1 ContainerCreating 0 2s +web-3 1/1 Running 0 2s +web-4 0/1 Pending 0 0s +web-4 0/1 Pending 0 0s +web-4 0/1 ContainerCreating 0 0s +web-4 1/1 Running 0 1s +web-2 1/1 Terminating 0 5m46s ----> start terminating 2 (only after both 4 and 3 are running) +web-1 1/1 Terminating 0 6m6s ----> start terminating 1 +web-2 0/1 Terminating 0 5m47s +web-1 0/1 Terminating 0 6m7s +web-1 0/1 Pending 0 0s +web-1 0/1 Pending 0 0s +web-1 0/1 ContainerCreating 0 1s +web-1 1/1 Running 0 2s +web-2 0/1 Pending 0 0s +web-2 0/1 Pending 0 0s +web-2 0/1 ContainerCreating 0 0s +web-2 1/1 Running 0 1s +web-0 1/1 Terminating 0 6m6s ----> start terminating 0 (only after 2 and 1 are running) +web-0 0/1 Terminating 0 6m7s +web-0 0/1 Pending 0 0s +web-0 0/1 Pending 0 0s +web-0 0/1 ContainerCreating 0 0s +web-0 1/1 Running 0 1s +``` +Note that as soon as the rolling update starts, both 4 and 3 (the two highest ordinal pods) start terminating at the same time. Pods +with ordinal 4 and 3 may become ready at their own pace. As soon as both pods 4 and 3 are ready, pods 2 and 1 start terminating at the +same time. When pods 2 and 1 are both running and ready, pod 0 starts terminating. + +In Kubernetes, updates to StatefulSets follow a strict ordering when updating Pods. In this example, the update starts at replica 4, then +replica 3, then replica 2, and so on, one pod at a time. When going one pod at a time, its not possible for 3 to be running and ready +before 4. When `maxUnavailable` is more than 1 (in the example scenario I set `maxUnavailable` to 2), it is possible that replica 3 becomes +ready and running before replica 4 is ready—and that is ok. If you're a developer and you set `maxUnavailable` to more than 1, you should +know that this outcome is possible and you must ensure that your application is able to handle such ordering issues that occur +if any. When you set `maxUnavailable` greater than 1, the ordering is guaranteed in between each batch of pods being updated. That guarantee +means that pods in update batch 2 (replicas 2 and 1) cannot start updating until the pods from batch 0 (replicas 4 and 3) are ready. + +Although Kubernetes refers to these as _replicas_, your stateful application may have a different view and each pod of the StatefulSet may +be holding completely different data than other pods. The important thing here is that updates to StatefulSets happen in batches, and you can +now have a batch size larger than 1 (as an alpha feature). + +Also note, that the above behavior is with `podManagementPolicy: OrderedReady`. If you defined a StatefulSet as `podManagementPolicy: Parallel`, +not only `maxUnavailable` number of replicas are terminated at the same time; `maxUnavailable` number of replicas start in `ContainerCreating` +phase at the same time as well. This is called bursting. + +So, now you may have a lot of questions about:- +- What is the behavior when you set `podManagementPolicy: Parallel`? +- What is the behavior when `partition` to a value other than `0`? + +It might be better to try and see it for yourself. This is an alpha feature, and the Kubernetes contributors are looking for feedback on this feature. Did +this help you achieve your stateful scenarios Did you find a bug or do you think the behavior as implemented is not intuitive or can +break applications or catch them by surprise? Please [open an issue](https://github.com/kubernetes/kubernetes/issues) to let us know. + +## Further reading and next steps {#next-steps} +- [Maximum unavailable Pods](/docs/concepts/workloads/controllers/statefulset/#maximum-unavailable-pods) +- [KEP for MaxUnavailable for StatefulSet](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/961-maxunavailable-for-statefulset) +- [Implementation](https://github.com/kubernetes/kubernetes/pull/82162/files) +- [Enhancement Tracking Issue](https://github.com/kubernetes/enhancements/issues/961) diff --git a/content/en/blog/_posts/2022-06-01-annual-report-2021.md b/content/en/blog/_posts/2022-06-01-annual-report-2021.md new file mode 100644 index 0000000000..e0a4130357 --- /dev/null +++ b/content/en/blog/_posts/2022-06-01-annual-report-2021.md @@ -0,0 +1,19 @@ +--- +layout: blog +title: "Annual Report Summary 2021" +date: 2022-06-01 +slug: annual-report-summary-2021 +--- + +**Author:** Paris Pittman (Steering Committee) + +Last year, we published our first [Annual Report Summary](/blog/2021/06/28/announcing-kubernetes-community-group-annual-reports/) for 2020 and it's already time for our second edition! + +[2021 Annual Report Summary](https://www.cncf.io/reports/kubernetes-annual-report-2021/) + +This summary reflects the work that has been done in 2021 and the initiatives on deck for the rest of 2022. Please forward to organizations and indidviduals participating in upstream activities, planning cloud native strategies, and/or those looking to help out. To find a specific community group's complete report, go to the [kubernetes/community repo](https://github.com/kubernetes/community) under the groups folder. Example: [sig-api-machinery/annual-report-2021.md](https://github.com/kubernetes/community/blob/master/sig-api-machinery/annual-report-2021.md) + +You’ll see that this report summary is a growth area in itself. It takes us roughly 6 months to prepare and execute, which isn’t helpful or valuable to anyone as a fast moving project with short and long term needs. How can we make this better? Provide your feedback here: https://github.com/kubernetes/steering/issues/242 + +Reference: +[Annual Report Documentation](https://github.com/kubernetes/community/blob/master/committee-steering/governance/annual-reports.md) diff --git a/content/en/blog/_posts/2022-07-13-gateway-api-in-beta.md b/content/en/blog/_posts/2022-07-13-gateway-api-in-beta.md new file mode 100644 index 0000000000..54457cd336 --- /dev/null +++ b/content/en/blog/_posts/2022-07-13-gateway-api-in-beta.md @@ -0,0 +1,178 @@ +--- +layout: blog +title: Kubernetes Gateway API Graduates to Beta +date: 2022-07-13 +slug: gateway-api-graduates-to-beta +canonicalUrl: https://gateway-api.sigs.k8s.io/blog/2022/graduating-to-beta/ +--- + +**Authors:** Shane Utt (Kong), Rob Scott (Google), Nick Young (VMware), Jeff Apple (HashiCorp) + +We are excited to announce the v0.5.0 release of Gateway API. For the first +time, several of our most important Gateway API resources are graduating to +beta. Additionally, we are starting a new initiative to explore how Gateway API +can be used for mesh and introducing new experimental concepts such as URL +rewrites. We'll cover all of this and more below. + +## What is Gateway API? + +Gateway API is a collection of resources centered around [Gateway][gw] resources +(which represent the underlying network gateways / proxy servers) to enable +robust Kubernetes service networking through expressive, extensible and +role-oriented interfaces that are implemented by many vendors and have broad +industry support. + +Originally conceived as a successor to the well known [Ingress][ing] API, the +benefits of Gateway API include (but are not limited to) explicit support for +many commonly used networking protocols (e.g. `HTTP`, `TLS`, `TCP`, `UDP`) as +well as tightly integrated support for Transport Layer Security (TLS). The +`Gateway` resource in particular enables implementations to manage the lifecycle +of network gateways as a Kubernetes API. + +If you're an end-user interested in some of the benefits of Gateway API we +invite you to jump in and find an implementation that suits you. At the time of +this release there are over a dozen [implementations][impl] for popular API +gateways and service meshes and guides are available to start exploring quickly. + +[gw]:https://gateway-api.sigs.k8s.io/api-types/gateway/ +[ing]:https://kubernetes.io/docs/reference/kubernetes-api/service-resources/ingress-v1/ +[impl]:https://gateway-api.sigs.k8s.io/implementations/ + +### Getting started + +Gateway API is an official Kubernetes API like +[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). +Gateway API represents a superset of Ingress functionality, enabling more +advanced concepts. Similar to Ingress, there is no default implementation of +Gateway API built into Kubernetes. Instead, there are many different +[implementations][impl] available, providing significant choice in terms of underlying +technologies while providing a consistent and portable experience. + +Take a look at the [API concepts documentation][concepts] and check out some of +the [Guides][guides] to start familiarizing yourself with the APIs and how they +work. When you're ready for a practical application open the [implementations +page][impl] and select an implementation that belongs to an existing technology +you may already be familiar with or the one your cluster provider uses as a +default (if applicable). Gateway API is a [Custom Resource Definition +(CRD)][crd] based API so you'll need to [install the CRDs][install-crds] onto a +cluster to use the API. + +If you're specifically interested in helping to contribute to Gateway API, we +would love to have you! Please feel free to [open a new issue][issue] on the +repository, or join in the [discussions][disc]. Also check out the [community +page][community] which includes links to the Slack channel and community meetings. + +[crd]:https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ +[concepts]:https://gateway-api.sigs.k8s.io/concepts/api-overview/ +[guides]:https://gateway-api.sigs.k8s.io/guides/getting-started/ +[impl]:https://gateway-api.sigs.k8s.io/implementations/ +[install-crds]:https://gateway-api.sigs.k8s.io/guides/getting-started/#install-the-crds +[issue]:https://github.com/kubernetes-sigs/gateway-api/issues/new/choose +[disc]:https://github.com/kubernetes-sigs/gateway-api/discussions +[community]:https://gateway-api.sigs.k8s.io/contributing/community/ + +## Release highlights + +### Graduation to beta + +The `v0.5.0` release is particularly historic because it marks the growth in +maturity to a beta API version (`v1beta1`) release for some of the key APIs: + +- [GatewayClass](https://gateway-api.sigs.k8s.io/api-types/gatewayclass/) +- [Gateway](https://gateway-api.sigs.k8s.io/api-types/gateway/) +- [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) + +This achievement was marked by the completion of several graduation criteria: + +- API has been [widely implemented][impl]. +- Conformance tests provide basic coverage for all resources and have multiple implementations passing tests. +- Most of the API surface is actively being used. +- Kubernetes SIG Network API reviewers have approved graduation to beta. + +For more information on Gateway API versioning, refer to the [official +documentation](https://gateway-api.sigs.k8s.io/concepts/versioning/). To see +what's in store for future releases check out the [next steps](#next-steps) +section. + +[impl]:https://gateway-api.sigs.k8s.io/implementations/ + +### Release channels + +This release introduces the `experimental` and `standard` [release channels][ch] +which enable a better balance of maintaining stability while still enabling +experimentation and iterative development. + +The `standard` release channel includes: + +- resources that have graduated to beta +- fields that have graduated to standard (no longer considered experimental) + +The `experimental` release channel includes everything in the `standard` release +channel, plus: + +- `alpha` API resources +- fields that are considered experimental and have not graduated to `standard` channel + +Release channels are used internally to enable iterative development with +quick turnaround, and externally to indicate feature stability to implementors +and end-users. + +For this release we've added the following experimental features: + +- [Routes can attach to Gateways by specifying port numbers](https://gateway-api.sigs.k8s.io/geps/gep-957/) +- [URL rewrites and path redirects](https://gateway-api.sigs.k8s.io/geps/gep-726/) + +[ch]:https://gateway-api.sigs.k8s.io/concepts/versioning/#release-channels-eg-experimental-standard + +### Other improvements + +For an exhaustive list of changes included in the `v0.5.0` release, please see +the [v0.5.0 release notes](https://github.com/kubernetes-sigs/gateway-api/releases/tag/v0.5.0). + +## Gateway API for service mesh: the GAMMA Initiative +Some service mesh projects have [already implemented support for the Gateway +API](https://gateway-api.sigs.k8s.io/implementations/). Significant overlap +between the Service Mesh Interface (SMI) APIs and the Gateway API has [inspired +discussion in the SMI +community](https://github.com/servicemeshinterface/smi-spec/issues/249) about +possible integration. + +We are pleased to announce that the service mesh community, including +representatives from Cilium Service Mesh, Consul, Istio, Kuma, Linkerd, NGINX +Service Mesh and Open Service Mesh, is coming together to form the [GAMMA +Initiative](https://gateway-api.sigs.k8s.io/contributing/gamma/), a dedicated +workstream within the Gateway API subproject focused on Gateway API for Mesh +Management and Administration. + +This group will deliver [enhancement +proposals](https://gateway-api.sigs.k8s.io/v1beta1/contributing/gep/) consisting +of resources, additions, and modifications to the Gateway API specification for +mesh and mesh-adjacent use-cases. + +This work has begun with [an exploration of using Gateway API for +service-to-service +traffic](https://docs.google.com/document/d/1T_DtMQoq2tccLAtJTpo3c0ohjm25vRS35MsestSL9QU/edit#heading=h.jt37re3yi6k5) +and will continue with enhancement in areas such as authentication and +authorization policy. + +## Next steps + +As we continue to mature the API for production use cases, here are some of the highlights of what we'll be working on for the next Gateway API releases: + +- [GRPCRoute][gep1016] for [gRPC][grpc] traffic routing +- [Route delegation][pr1085] +- Layer 4 API maturity: Graduating [TCPRoute][tcpr], [UDPRoute][udpr] and + [TLSRoute][tlsr] to beta +- [GAMMA Initiative](https://gateway-api.sigs.k8s.io/contributing/gamma/) - Gateway API for Service Mesh + +If there's something on this list you want to get involved in, or there's +something not on this list that you want to advocate for to get on the roadmap +please join us in the #sig-network-gateway-api channel on Kubernetes Slack or our weekly [community calls](https://gateway-api.sigs.k8s.io/contributing/community/#meetings). + +[gep1016]:https://github.com/kubernetes-sigs/gateway-api/blob/master/site-src/geps/gep-1016.md +[grpc]:https://grpc.io/ +[pr1085]:https://github.com/kubernetes-sigs/gateway-api/pull/1085 +[tcpr]:https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/tcproute_types.go +[udpr]:https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/udproute_types.go +[tlsr]:https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/tlsroute_types.go +[community]:https://gateway-api.sigs.k8s.io/contributing/community/ diff --git a/content/en/blog/_posts/2022-08-02-sig-docs-spotlight/index.md b/content/en/blog/_posts/2022-08-02-sig-docs-spotlight/index.md new file mode 100644 index 0000000000..63fc79e01c --- /dev/null +++ b/content/en/blog/_posts/2022-08-02-sig-docs-spotlight/index.md @@ -0,0 +1,114 @@ +--- +layout: blog +title: "Spotlight on SIG Docs" +date: 2022-08-02 +slug: sig-docs-spotlight-2022 +canonicalUrl: https://kubernetes.dev/blog/2022/08/02/sig-docs-spotlight-2022/ +--- + +**Author:** Purneswar Prasad + +## Introduction + +The official documentation is the go-to source for any open source project. For Kubernetes, +it's an ever-evolving Special Interest Group (SIG) with people constantly putting in their efforts +to make details about the project easier to consume for new contributors and users. SIG Docs publishes +the official documentation on [kubernetes.io](https://kubernetes.io) which includes, +but is not limited to, documentation of the core APIs, core architectural details, and CLI tools +shipped with the Kubernetes release. + +To learn more about the work of SIG Docs and its future ahead in shaping the community, I have summarised +my conversation with the co-chairs, [Divya Mohan](https://twitter.com/Divya_Mohan02) (DM), +[Rey Lejano](https://twitter.com/reylejano) (RL) and Natali Vlatko (NV), who ran through the +SIG's goals and how fellow contributors can help. + +## A summary of the conversation + +### Could you tell us a little bit about what SIG Docs does? + +SIG Docs is the special interest group for documentation for the Kubernetes project on kubernetes.io, +generating reference guides for the Kubernetes API, kubeadm and kubectl as well as maintaining the official +website’s infrastructure and analytics. The remit of their work also extends to docs releases, translation of docs, +improvement and adding new features to existing documentation, pushing and reviewing content for the official +Kubernetes blog and engaging with the Release Team for each cycle to get docs and blogs reviewed. + + +### There are 2 subprojects under Docs: blogs and localization. How has the community benefited from it and are there some interesting contributions by those teams you want to highlight? + +**Blogs**: This subproject highlights new or graduated Kubernetes enhancements, community reports, SIG updates +or any relevant news to the Kubernetes community such as thought leadership, tutorials and project updates, +such as the Dockershim removal and removal of PodSecurityPolicy, which is upcoming in the 1.25 release. +Tim Bannister, one of the SIG Docs tech leads, does awesome work and is a major force when pushing contributions +through to the docs and blogs. + +**Localization**: With this subproject, the Kubernetes community has been able to achieve greater inclusivity +and diversity among both users and contributors. This has also helped the project gain more contributors, +especially students, since a couple of years ago. +One of the major highlights and up-and-coming localizations are Hindi and Bengali. The efforts for Hindi +localization are currently being spearheaded by students in India. + +In addition to that, there are two other subprojects: [reference-docs](https://github.com/kubernetes-sigs/reference-docs) and the [website](https://github.com/kubernetes/website), which is built with Hugo and is an important ownership area. + +### Recently there has been a lot of buzz around the Kubernetes ecosystem as well as the industry regarding the removal of dockershim in the latest 1.24 release. How has SIG Docs helped the project to ensure a smooth change among the end-users? {#dockershim-removal} + +Documenting the removal of Dockershim was a mammoth task, requiring the revamping of existing documentation +and communicating to the various stakeholders regarding the deprecation efforts. It needed a community effort, +so ahead of the 1.24 release, SIG Docs partnered with Docs and Comms verticals, the Release Lead from the +Release Team, and also the CNCF to help put the word out. Weekly meetings and a GitHub project board were +set up to track progress, review issues and approve PRs and keep the Kubernetes website updated. This has +also helped new contributors know about the depreciation, so that if any good-first-issue pops up, they could chip in. +A dedicated Slack channel was used to communicate meeting updates, invite feedback or to solicit help on +outstanding issues and PRs. The weekly meeting also continued for a month after the 1.24 release to review related issues and fix them. +A huge shoutout to [Celeste Horgan](https://twitter.com/celeste_horgan), who kept the ball rolling on this +conversation throughout the deprecation process. + +### Why should new and existing contributors consider joining this SIG? + +Kubernetes is a vast project and can be intimidating at first for a lot of folks to find a place to start. +Any open source project is defined by its quality of documentation and SIG Docs aims to be a welcoming, +helpful place for new contributors to get onboard. One gets the perks of working with the project docs +as well as learning by reading it. They can also bring their own, new perspective to create and improve +the documentation. In the long run if they stick to SIG Docs, they can rise up the ladder to be maintainers. +This will help make a big project like Kubernetes easier to parse and navigate. + +### How do you help new contributors get started? Are there any prerequisites to join? + +There are no such prerequisites to get started with contributing to Docs. But there is certainly a fantastic +Contribution to Docs guide which is always kept as updated and relevant as possible and new contributors +are urged to read it and keep it handy. Also, there are a lot of useful pins and bookmarks in the +community Slack channel [#sig-docs](https://kubernetes.slack.com/archives/C1J0BPD2M). GitHub issues with +the good-first-issue labels in the kubernetes/website repo is a great place to create your first PR. +Now, SIG Docs has a monthly New Contributor Meet and Greet on the first Tuesday of the month with the +first occupant of the New Contributor Ambassador role, [Arsh Sharma](https://twitter.com/RinkiyaKeDad). +This has helped in making a more accessible point of contact within the SIG for new contributors. + +### Any SIG related accomplishment that you’re really proud of? + +**DM & RL** : The formalization of the localization subproject in the last few months has been a big win +for SIG Docs, given all the great work put in by contributors from different countries. Earlier the +localization efforts didn’t have any streamlined process and focus was given to provide a structure by +drafting a KEP over the past couple of months for localization to be formalized as a subproject, which +is planned to be pushed through by the end of third quarter. + +**DM** : Another area where there has been a lot of success is the New Contributor Ambassador role, +which has helped in making a more accessible point of contact for the onboarding of new contributors into the project. + +**NV** : For each release cycle, SIG Docs have to review release docs and feature blogs highlighting +release updates within a short window. This is always a big effort for the docs and blogs reviewers. + +### Is there something exciting coming up for the future of SIG Docs that you want the community to know? + +SIG Docs is now looking forward to establishing a roadmap, having a steady pipeline of folks being able +to push improvements to the documentation and streamlining community involvement in triaging issues and +reviewing PRs being filed. To build one such contributor and reviewership base, a mentorship program is +being set up to help current contributors become reviewers. This definitely is a space to watch out for more! + + +## Wrap Up + +SIG Docs hosted a [deep dive talk](https://www.youtube.com/watch?v=GDfcBF5et3Q) +during on KubeCon + CloudNativeCon North America 2021, covering their awesome SIG. +They are very welcoming and have been the starting ground into Kubernetes +for a lot of new folks who want to contribute to the project. +Join the [SIG's meetings](https://github.com/kubernetes/community/blob/master/sig-docs/README.md) to find out +about the most recent research results, their plans for the forthcoming year, and how to get involved in the upstream Docs team as a contributor! diff --git a/content/en/blog/_posts/2022-08-04-kubernetes-1.24-deprecations-and-removals.md b/content/en/blog/_posts/2022-08-04-kubernetes-1.24-deprecations-and-removals.md new file mode 100644 index 0000000000..eccfa35a0a --- /dev/null +++ b/content/en/blog/_posts/2022-08-04-kubernetes-1.24-deprecations-and-removals.md @@ -0,0 +1,78 @@ +--- +layout: blog +title: "Kubernetes Removals and Major Changes In 1.25" +date: 2022-08-04 +slug: upcoming-changes-in-kubernetes-1-25 +--- + +**Authors**: Kat Cosgrove, Frederico Muñoz, Debabrata Panigrahi + +As Kubernetes grows and matures, features may be deprecated, removed, or replaced with improvements for the health of the project. Kubernetes v1.25 includes several major changes and one major removal. + +## The Kubernetes API Removal and Deprecation process + +The Kubernetes project has a well-documented [deprecation policy](/docs/reference/using-api/deprecation-policy/) for features. This policy states that stable APIs may only be deprecated when a newer, stable version of that same API is available and that APIs have a minimum lifetime for each stability level. A deprecated API is one that has been marked for removal in a future Kubernetes release; it will continue to function until removal (at least one year from the deprecation), but usage will result in a warning being displayed. Removed APIs are no longer available in the current version, at which point you must migrate to using the replacement. + +* Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes. +* Beta or pre-release API versions must be supported for 3 releases after deprecation. +* Alpha or experimental API versions may be removed in any release without prior deprecation notice. + +Whether an API is removed as a result of a feature graduating from beta to stable or because that API simply did not succeed, all removals comply with this deprecation policy. Whenever an API is removed, migration options are communicated in the documentation. + +## A Note About PodSecurityPolicy + +In Kubernetes v1.25, we will be removing PodSecurityPolicy [after its deprecation in v1.21](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). PodSecurityPolicy has served us honorably, but its complex and often confusing usage necessitated changes, which unfortunately would have been breaking changes. To address this, it is being removed in favor of a replacement, Pod Security Admission, which is graduating to stable in this release as well. If you are currently relying on PodSecurityPolicy, follow the instructions for [migration to Pod Security Admission](/docs/tasks/configure-pod-container/migrate-from-psp/). + +## Major Changes for Kubernetes v1.25 + +Kubernetes v1.25 includes several major changes, in addition to the removal of PodSecurityPolicy. + +### [CSI Migration](https://github.com/kubernetes/enhancements/issues/625) + +The effort to move the in-tree volume plugins to out-of-tree CSI drivers continues, with the core CSI Migration feature going GA in v1.25. This is an important step towards removing the in-tree volume plugins entirely. + +### Volume Plugin Deprecations and Removals + +Several volume are being deprecated or removed. + +[GlusterFS will be deprecated in v1.25](https://github.com/kubernetes/enhancements/issues/3446). While a CSI driver was built for it, it has not been maintained. The possibility of migration to a compatible CSI driver [was discussed](https://github.com/kubernetes/kubernetes/issues/100897), but a decision was ultimately made to begin the deprecation of the GlusterFS plugin from in-tree drivers. The [Portworx in-tree volume plugin](https://github.com/kubernetes/enhancements/issues/2589) is also being deprecated with this release. The Flocker, Quobyte, and StorageOS in-tree volume plugins are being removed. + +### [Declare Unsupported vSphere Versions](https://github.com/kubernetes/kubernetes/pull/111255) + +From Kubernetes v1.25, the in-tree vSphere volume driver will not support any vSphere release before 7.0u2. Check the v1.25 detailed release notes for more advice on how to handle this. + +### [Signing Release Artifacts](https://github.com/kubernetes/enhancements/issues/3031) + +An additional step in improving the security posture of the release process, the signing of Kubernetes release artifacts will graduate to Beta in this release. This is in line with the proposed enhancement of targeting SLSA Level 3 compliance for the Kubernetes release process. + +### [Support for cgroup v2 Graduating to Stable](https://github.com/kubernetes/enhancements/issues/2254) + +The new kernel cgroups v2 API was declared stable more than two years ago, and in this release we're taking solid steps towards full adoption of it. While cgroup v1 will continue to be supported, this change makes us ready to deal with the eventual deprecation of cgroup v1 and its replacement by cgroup v2. + +### [Cleaning up IPTables Chain Ownership](https://github.com/kubernetes/enhancements/issues/3178) + +From the Kubernetes 1.25 release, the iptables chains created by Kubernetes will only support for internal Kubernetes use cases. Starting with v1.25, the Kubelet will gradually move towards not creating the following iptables chains in the `nat` table: + + - `KUBE-MARK-DROP` + - `KUBE-MARK-MASQ` + - `KUBE-POSTROUTING` + +This change will be phased in via the `IPTablesCleanup` feature gate. + +## Looking ahead + +The official [list of API removals planned for Kubernetes 1.26](/docs/reference/using-api/deprecation-guide/#v1-26) is: + +* The beta FlowSchema and PriorityLevelConfiguration APIs (flowcontrol.apiserver.k8s.io/v1beta1) +* The beta HorizontalPodAutoscaler API (autoscaling/v2beta2) + + +### Want to know more? +Deprecations are announced in the Kubernetes release notes. You can see the announcements of pending deprecations in the release notes for: +* [Kubernetes 1.21](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#deprecation) +* [Kubernetes 1.22](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#deprecation) +* [Kubernetes 1.23](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#deprecation) +* [Kubernetes 1.24](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#deprecation) +* We will formally announce the deprecations that come with [Kubernetes 1.25](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#deprecation) as part of the CHANGELOG for that release. + +For information on the process of deprecation and removal, check out the official Kubernetes [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api) document. diff --git a/content/en/case-studies/peardeck/index.html b/content/en/case-studies/peardeck/index.html index 4398277677..77d94873fd 100644 --- a/content/en/case-studies/peardeck/index.html +++ b/content/en/case-studies/peardeck/index.html @@ -20,7 +20,7 @@ case_study_details: <h2>Solution</h2> -<p>In 2016, the company began moving their code from Heroku to <a href="https://www.docker.com/">Docker</a> containers running on <a href="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <a href="http://kubernetes.io/">Kubernetes</a> and monitored with <a href="https://prometheus.io/">Prometheus</a>.</p> +<p>In 2016, the company began moving their code from Heroku to containers running on <a href="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <a href="http://kubernetes.io/">Kubernetes</a> and monitored with <a href="https://prometheus.io/">Prometheus</a>.</p> <h2>Impact</h2> @@ -42,7 +42,7 @@ With the speed befitting a startup, Pear Deck delivered its first prototype to c <p>On top of that, many of Pear Deck's customers are behind government firewalls and connect through Firebase, not Pear Deck's servers, making troubleshooting even more difficult.</p> -<p>The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to <a href="https://www.docker.com/">Docker</a> containers running on <a href="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <a href="http://kubernetes.io/">Kubernetes</a> and monitored with <a href="https://prometheus.io/">Prometheus</a>.</p> +<p>The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to containers running on <a href="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <a href="http://kubernetes.io/">Kubernetes</a> and monitored with <a href="https://prometheus.io/">Prometheus</a>.</p> {{< case-studies/quote image="/images/case-studies/peardeck/banner1.jpg" >}} "When it became clear that Google Kubernetes Engine was going to have a lot of support from Google and be a fully-managed Kubernetes platform, it seemed very obvious to us that was the way to go," says Eynon-Lynch. diff --git a/content/en/case-studies/prowise/index.html b/content/en/case-studies/prowise/index.html index b8b584aa41..8e2a89c385 100644 --- a/content/en/case-studies/prowise/index.html +++ b/content/en/case-studies/prowise/index.html @@ -70,7 +70,7 @@ If you haven't set foot in a school in awhile, you might be surprised by what yo <p>Recently, the team launched a new single sign-on solution for use in an internal application. "Due to the resource based architecture of the Kubernetes platform, we were able to bring that application into an entirely new production environment in less than a day, most of that time used for testing after applying the already well-known resource definitions from staging to the new environment," says van den Bosch. "On a traditional VM this would have likely cost a day or two, and then probably a few weeks to iron out the kinks in our provisioning scripts as we apply updates."</p> -<p>Legacy applications are also being moved to Kubernetes. Not long ago, the team needed to set up a Java-based application for compiling and running a frontend. "On a traditional VM, it would have taken quite a bit of time to set it up and keep it up to date, not to mention maintenance for that setup down the line," says van den Bosch. Instead, it took less than half a day to Dockerize it and get it running on Kubernetes. "It was much easier, and we were able to save costs too because we didn't have to spin up new VMs specially for it."</p> +<p>Legacy applications are also being moved to Kubernetes. Not long ago, the team needed to set up a Java-based application for compiling and running a frontend. "On a traditional VM, it would have taken quite a bit of time to set it up and keep it up to date, not to mention maintenance for that setup down the line," says van den Bosch. Instead, it took less than half a day to containerize it and get it running on Kubernetes. "It was much easier, and we were able to save costs too because we didn't have to spin up new VMs specially for it."</p> {{< case-studies/quote author="VICTOR VAN DEN BOSCH, SENIOR DEVOPS ENGINEER, PROWISE" >}} "We're really trying to deliver integrated solutions with our hardware and software and making it as easy as possible for users to use and collaborate from different places," says van den Bosch. And, says Haalstra, "We cannot do it without Kubernetes." diff --git a/content/en/case-studies/squarespace/index.html b/content/en/case-studies/squarespace/index.html index 461e466d8c..d39ce0b6f6 100644 --- a/content/en/case-studies/squarespace/index.html +++ b/content/en/case-studies/squarespace/index.html @@ -46,7 +46,7 @@ Since it was started in a dorm room in 2003, Squarespace has made it simple for After experimenting with another container orchestration platform and "breaking it in very painful ways," Lynch says, the team began experimenting with Kubernetes in mid-2016 and found that it "answered all the questions that we had." {{< /case-studies/quote >}} -<p>Within a couple months, they had a stable cluster for their internal use, and began rolling out Kubernetes for production. They also added Zipkin and CNCF projects <a href="https://prometheus.io/">Prometheus</a> and <a href="https://www.fluentd.org/">fluentd</a> to their cloud native stack. "We switched to Kubernetes, a new world, and we revamped all our other tooling as well," says Lynch. "It allowed us to streamline our process, so we can now easily create an entire microservice project from templates, generate the code and deployment pipeline for that, generate the Docker file, and then immediately just ship a workable, deployable project to Kubernetes." Deployments across Dev/QA/Stage/Prod were also "simplified drastically," Lynch adds. "Now there is little configuration variation."</p> +<p>Within a couple months, they had a stable cluster for their internal use, and began rolling out Kubernetes for production. They also added Zipkin and CNCF projects <a href="https://prometheus.io/">Prometheus</a> and <a href="https://www.fluentd.org/">fluentd</a> to their cloud native stack. "We switched to Kubernetes, a new world, and we revamped all our other tooling as well," says Lynch. "It allowed us to streamline our process, so we can now easily create an entire microservice project from templates, generate the code and deployment pipeline for that, generate the Dockerfile, and then immediately just ship a workable, deployable project to Kubernetes." Deployments across Dev/QA/Stage/Prod were also "simplified drastically," Lynch adds. "Now there is little configuration variation."</p> <p>And the whole process takes only five minutes, an almost 85% reduction in time compared to their VM deployment. "From end to end that probably took half an hour, and that's not accounting for the fact that an infrastructure engineer would be responsible for doing that, so there's some business delay in there as well."</p> diff --git a/content/en/case-studies/wink/index.html b/content/en/case-studies/wink/index.html index 1f8ea4c89d..7b5a3f8541 100644 --- a/content/en/case-studies/wink/index.html +++ b/content/en/case-studies/wink/index.html @@ -58,9 +58,9 @@ How many people does it take to turn on a light bulb? <p>In addition, Wink had other requirements: horizontal scalability, the ability to encrypt everything quickly, connections that could be easily brought back up if something went wrong. "Looking at this whole structure we started, we decided to make a secure socket-based service," says Klein. "We've always used, I would say, some sort of clustering technology to deploy our services and so the decision we came to was, this thing is going to be containerized, running on Docker."</p> -<p>At the time – just over two years ago – Docker wasn't yet widely used, but as Klein points out, "it was certainly understood by the people who were on the frontier of technology. We started looking at potential technologies that existed. One of the limiting factors was that we needed to deploy multi-port non-http/https services. It wasn't really appropriate for some of the early cluster technology. We liked the project a lot and we ended up using it on other stuff for a while, but initially it was too targeted toward http workloads."</p> +<p>In 2015, Docker wasn't yet widely used, but as Klein points out, "it was certainly understood by the people who were on the frontier of technology. We started looking at potential technologies that existed. One of the limiting factors was that we needed to deploy multi-port non-http/https services. It wasn't really appropriate for some of the early cluster technology. We liked the project a lot and we ended up using it on other stuff for a while, but initially it was too targeted toward http workloads."</p> -<p>Once Wink's backend engineering team decided on a Dockerized workload, they had to make decisions about the OS and the container orchestration platform. "Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure."</p> +<p>Once Wink's backend engineering team decided on a containerized workload, they had to make decisions about the OS and the container orchestration platform. "Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure."</p> {{< case-studies/quote image="/images/case-studies/wink/banner4.jpg" >}} "Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure." @@ -68,7 +68,7 @@ How many people does it take to turn on a light bulb? <p>Wink considered building directly on a general purpose Linux distro like Ubuntu (which would have required installing tools to run a containerized workload) and cluster management systems like Mesos (which was targeted toward enterprises with larger teams/workloads), but ultimately set their sights on CoreOS Container Linux. "A container-optimized Linux distribution system was exactly what we needed," he says. "We didn't have to futz around with trying to take something like a Linux distro and install everything. It's got a built-in container orchestration system, which is Fleet, and an easy-to-use API. It's not as feature-rich as some of the heavier solutions, but we realized that, at that moment, it was exactly what we needed."</p> -<p>Wink's hub (along with a revamped app) was introduced in July 2014 with a short-term deployment, and within the first month, they had moved the service to the Dockerized CoreOS deployment. Since then, they've moved almost every other piece of their infrastructure – from third-party cloud-to-cloud integrations to their customer service and payment portals – onto CoreOS Container Linux clusters.</p> +<p>Wink's hub (along with a revamped app) was introduced in July 2014 with a short-term deployment, and within the first month, they had moved the service to the containerized CoreOS deployment. Since then, they've moved almost every other piece of their infrastructure – from third-party cloud-to-cloud integrations to their customer service and payment portals – onto CoreOS Container Linux clusters.</p> <p>Using this setup did require some customization. "Fleet is really nice as a basic container orchestration system, but it doesn't take care of routing, sharing configurations, secrets, et cetera, among instances of a service," Klein says. "All of those layers of functionality can be implemented, of course, but if you don't want to spend a lot of time writing unit files manually – which of course nobody does – you need to create a tool to automate some of that, which we did."</p> diff --git a/content/en/community/_index.html b/content/en/community/_index.html index b41323c69e..de7fcee9a4 100644 --- a/content/en/community/_index.html +++ b/content/en/community/_index.html @@ -1,256 +1,183 @@ ---- -title: Community -layout: basic -cid: community ---- - -<div class="newcommunitywrapper"> -<div class="banner1"> - <img src="/images/community/kubernetes-community-final-02.jpg" alt="Kubernetes Conference Gallery" style="width:100%;padding-left:0px" class="desktop"> - <img src="/images/community/kubernetes-community-02-mobile.jpg" alt="Kubernetes Conference Gallery" style="width:100%;padding-left:0px" class="mobile"> -</div> - -<div class="intro"> -<br class="mobile"> -<p>The Kubernetes community -- users, contributors, and the culture we've built together -- is one of the biggest reasons for the meteoric rise of this open source project. Our culture and values continue to grow and change as the project itself grows and changes. We all work together toward constant improvement of the project and the ways we work on it. -<br><br>We are the people who file issues and pull requests, attend SIG meetings, Kubernetes meetups, and KubeCon, advocate for its adoption and innovation, run <code>kubectl get pods</code>, and contribute in a thousand other vital ways. Read on to learn how you can get involved and become part of this amazing community.</p> -<br class="mobile"> -</div> - -<div class="community__navbar"> - -<a href="#values">Community Values</a>      -<a href="#conduct">Code of conduct </a>      -<a href="#videos">Videos</a>      -<a href="#discuss">Discussions</a>      -<a href="#events">Events and meetups</a>      -<a href="#news">News</a>      -<a href="/releases">Releases</a> - -</div> -<br class="mobile"><br class="mobile"> -<div class="imagecols"> -<br class="mobile"> - <div class="imagecol"> - <img src="/images/community/kubernetes-community-final-03.jpg" alt="Kubernetes Conference Gallery" style="width:100%" class="desktop"> - </div> - - <div class="imagecol"> - <img src="/images/community/kubernetes-community-final-04.jpg" alt="Kubernetes Conference Gallery" style="width:100%" class="desktop"> - </div> - - <div class="imagecol" style="margin-right:0% important"> - <img src="/images/community/kubernetes-community-final-05.jpg" alt="Kubernetes Conference Gallery" style="width:100%;margin-right:0% important" class="desktop"> - </div> - <img src="/images/community/kubernetes-community-04-mobile.jpg" alt="Kubernetes Conference Gallery" style="width:100%;margin-bottom:3%" class="mobile"> -<a name="values"></a> -</div> - -<div><a name="values"></a></div> -<div class="conduct"> -<div class="conducttext"> -<br class="mobile"><br class="mobile"> -<br class="tablet"><br class="tablet"> -<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Community Values</h1> -The Kubernetes Community values are the keystone to the ongoing success of the project.<br> -These principles guide every aspect of the Kubernetes project. -<br> -<a href="/community/values/"> -<br class="mobile"><br class="mobile"> -<span class="fullbutton"> - READ MORE - </span> - </a> - </div><a name="conduct"></a> - </div> - </div> - - - -<div class="conduct"> -<div class="conducttext"> -<br class="mobile"><br class="mobile"> -<br class="tablet"><br class="tablet"> -<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Code of Conduct</h1> -The Kubernetes community values respect and inclusiveness, and enforces a Code of Conduct in all interactions. If you notice a violation of the Code of Conduct at an event or meeting, in Slack, or in another communication mechanism, reach out to the Kubernetes Code of Conduct Committee at <a href="mailto:conduct@kubernetes.io" style="color:#0662EE;font-weight:300">conduct@kubernetes.io</a>. All reports are kept confidential. You can read about the committee <a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct" style="color:#0662EE;font-weight:300">here</a>. -<br> -<a href="https://kubernetes.io/community/code-of-conduct/"> -<br class="mobile"><br class="mobile"> - -<span class="fullbutton"> -READ MORE -</span> -</a> -</div><a name="videos"></a> -</div> -</div> - - - -<div class="videos"> -<br class="mobile"><br class="mobile"> -<br class="tablet"><br class="tablet"> -<h1 style="margin-top:0px">Videos</h1> - -<div style="margin-bottom:4%;font-weight:300;text-align:center;padding-left:10%;padding-right:10%">We're on YouTube, a lot. Subscribe for a wide range of topics.</div> - -<div class="videocontainer"> - -<div class="video"> - - <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg" title="Monthly office hours" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> - -<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg"> -<div class="videocta"> -Watch monthly office hours ▶</div> -</a> -</div> - -<div class="video"> - <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ" title="Weekly community meetings" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ"> -<div class="videocta"> -Watch weekly community meetings ▶ -</div> -</a> -</div> - -<div class="video"> - -<iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F" title="Talk from a community member" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> - -<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F"> -<div class="videocta"> -Watch a talk from a community member ▶ -</div> - -</a> -<a name="discuss"></a> -</div> -</div> -</div> - - -<div class="resources"> -<br class="mobile"><br class="mobile"> -<br class="tablet"><br class="tablet"> -<h1 style="padding-top:1%">Discussions</h1> - -<div style="font-weight:300;text-align:center">We talk a lot. Find us and join the conversation on any of these platforms.</div> - -<div class="resourcecontainer"> - -<div class="resourcebox"> -<img src="/images/community/discuss.png" alt=Forum" style="width:80%;padding-bottom:2%"> -<a href="https://discuss.kubernetes.io/" style="color:#0662EE;display:block;margin-top:1%"> -forum ▶ -</a> -<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%"> -Topic-based technical discussions that bridge docs, StackOverflow, and so much more -</div> -</div> - -<div class="resourcebox"> -<img src="/images/community/twitter.png" alt="Twitter" style="width:80%;padding-bottom:2%"> -<a href="https://twitter.com/kubernetesio" style="color:#0662EE;display:block;margin-top:1%"> -twitter ▶ -</a> -<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">Real-time announcements of blog posts, events, news, ideas -</div> -</div> - -<div class="resourcebox"> -<img src="/images/community/github.png" alt="GitHub" style="width:80%;padding-bottom:2%"> -<a href="https://github.com/kubernetes/kubernetes" style="color:#0662EE;display:block;margin-top:1%"> -github ▶ -</a> -<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%"> -All the project and issue tracking, plus of course code -</div> -</div> - -<div class="resourcebox"> -<img src="/images/community/stack.png" alt="Stack Overflow" style="width:80%;padding-bottom:2%"> -<a href="https://stackoverflow.com/search?q=kubernetes" style="color:#0662EE;display:block;margin-top:1%"> -stack overflow ▶ -</a> -<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%"> - Technical troubleshooting for any use case - <a name="events"></a> -</div> -</div> - -<!-- -<div class="resourcebox"> - -<img src="/images/community/slack.png" style="width:80%"> - -slack ▶ - -<div class="resourceboxtext" style="font-size:11px;text-transform:none !important;font-weight:200;line-height:1.4em;color:#333333;margin-top:4%"> -With 170+ channels, you'll find one that fits your needs. -</div> - -</div>--> - -</div> -</div> -<div class="events"> - <br class="mobile"><br class="mobile"> - <br class="tablet"><br class="tablet"> -<div class="eventcontainer"> - <h1 style="color:white !important">Upcoming Events</h1> - {{< upcoming-events >}} -</div> -</div> - -<div class="meetups"> -<div class="meetupcol"> -<div class="meetuptext"> -<h1 style="text-align:left">Global Community</h1> -With over 150 meetups in the world and growing, go find your local kube people. If one isn't near, take charge and create your own. -</div> -<a href="https://www.meetup.com/topics/kubernetes/"> -<div class="button"> -FIND A MEETUP -</div> -</a> -<a name="news"></a> -</div> -</div> - - -<!-- -<div class="contributor"> -<div class="contributortext"> -<br> -<h1 style="text-align:left"> -New Contributors Site - </h1> -Text about new contributors site. - -<br><br> - -<div class="button"> -VISIT SITE -</div> -</div> -</div> - ---> - -<div class="news"> -<br class="mobile"><br class="mobile"> -<br class="tablet"><br class="tablet"> -<h1 style="margin-bottom:2%">Recent News</h1> - -<br> -<div class="twittercol1"> -<a class="twitter-timeline" data-tweet-limit="1" href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets by kubernetesio</a> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> -</div> - -<br> -<br><br><br><br> -</div> - -</div> +--- +title: Community +layout: basic +cid: community +community_styles_migrated: true +--- +<img + id="banner" + srcset="/images/community/kubernetes-community-final-02.jpg 1500w, /images/community/kubernetes-community-02-mobile.jpg 900w" + sizes="(max-width: 900px) 900px, (max-width: 1920px) 1500px" + src="/images/community/kubernetes-community-final-02.jpg" + alt="Kubernetes conference photo"> + +<div class="community-section" id="introduction"> + <p>The Kubernetes community — users, contributors, and the culture we've + built together — is one of the biggest reasons for the meteoric rise of + this open source project. Our culture and values continue to grow and change + as the project itself grows and changes. We all work together toward constant + improvement of the project and the ways we work on it.</p> + <p> We are the people who file issues and pull requests, attend SIG meetings, + Kubernetes meetups, and KubeCon, advocate for its adoption and innovation, + run <code>kubectl get pods</code>, and contribute in a thousand other vital + ways. Read on to learn how you can get involved and become part of this amazing + community.</p> +</div> + +<div id="navigation-items"> + <div class="community-nav-item external-link"> + <a href="https://www.kubernetes.dev/">Contributor community</a> + </div> + <div class="community-nav-item"> + <a href="#values">Community values</a> + </div> + <div class="community-nav-item"> + <a href="#conduct">Code of conduct</a> + </div> + <div class="community-nav-item"> + <a href="#videos">Videos</a> + </div> + <div class="community-nav-item"> + <a href="#discuss">Discussions</a> + </div> + <div class="community-nav-item"> + <a href="#meetups">Meetups</a> + </div> + <div class="community-nav-item"> + <a href="#news">News</a> + </div> + <div class="community-nav-item"> + <a href="/releases">Releases</a> + </div> +</div> + +<div class="community-section" id="gallery"> + <img src="/images/community/kubernetes-community-final-03.jpg" alt="Kubernetes conference gallery photo" class="community-gallery-desktop"> + <img src="/images/community/kubernetes-community-final-04.jpg" alt="Kubernetes conference gallery photo" class="community-gallery-desktop"> + <img src="/images/community/kubernetes-community-final-05.jpg" alt="Kubernetes conference gallery photo" class="community-gallery-desktop"> + <img src="/images/community/kubernetes-community-04-mobile.jpg" alt="Kubernetes conference gallery photo" class="community-gallery-mobile"> +</div> + +<div class="community-section" id="values"> + <h2>Community Values</h2> + <p>The Kubernetes Community values are the keystone to the ongoing success of the project.<br class="optional"/> + These principles guide every aspect of the Kubernetes project.</p> + <a href="https://www.kubernetes.dev/community/values/" class="community-cta-button"> + <span class="community-cta">Read more</span> + </a> +</div> + +<div class="community-section" id="conduct"> + <h2>Code of Conduct</h2> + <p>The Kubernetes community values respect and inclusiveness, and enforces a Code of Conduct in all interactions.</p> + <p>If you notice a violation of the Code of Conduct at an event or meeting, in <a href="#slack">Slack</a>, or in another communication mechanism, reach out to the Kubernetes Code of Conduct Committee at <a href="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. All reports are kept confidential. You can read <a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct">about the committee</a> in the Kubernetes community repository on GitHub.</p> + <a href="/community/code-of-conduct/" class="community-cta-button"> + <span class="community-cta">Read more</span> + </a> +</div> + +<div id="videos" class="community-section"> + <h2>Videos</h2> + + <p class="community-simple">Kubernetes is on YouTube, a lot. Subscribe for a wide range of topics.</p> + + <div class="container"> + <div class="video youtube"> + <iframe src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg" title="Monthly office hours" allow="camera 'none'; microphone 'none'; geolocation 'none'; fullscreen https://www.youtube.com/" ></iframe> + <a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg"> + <span class="videocta">Watch monthly office hours ▶</span> + </a> + </div> + + <div class="video youtube"> + <iframe src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ" title="Weekly community meetings" allow="camera 'none'; microphone 'none'; geolocation 'none'; fullscreen https://www.youtube.com/"></iframe> + <a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ"> + <span class="videocta">Watch weekly community meetings ▶</span> + </a> + </div> + + <div class="video youtube" id="discuss"> + <iframe src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F" title="Talk from a community member" allow="camera 'none'; microphone 'none'; geolocation 'none'; fullscreen https://www.youtube.com/"></iframe> + <a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F"> + <span class="videocta">Watch a talk from a community member ▶</span> + </a> + </div> + </div> +</div> + +<div id="resources" class="community-section"> + <h2>Discussions</h2> + + <p class="community-simple">We talk a lot. Find us and join the conversation on any of these platforms.</p> + + <div class="container"> + <div class="community-resource"> + <a href="https://discuss.kubernetes.io/"> + <img src="/images/community/discuss.png" alt="Forum"> + </a> + <a href="https://discuss.kubernetes.io/">Community forums ▶</a> + <p>Topic-based technical discussions that bridge docs, + troubleshooting, and so much more.</p> + </div> + + <div id="twitter" class="community-resource"> + <a href="https://twitter.com/kubernetesio"> + <img src="/images/community/twitter.png" alt="Twitter"> + </a> + <a href="https://twitter.com/kubernetesio">Twitter ▶</a> + <p><em>#kubernetesio</em></p> + <p>Real-time announcements of blog posts, events, news, ideas.</p> + </div> + + <div id="github" class="community-resource"> + <a href="https://github.com/kubernetes/kubernetes"> + <img src="/images/community/github.png" alt="GitHub"> + </a> + <a href="https://github.com/kubernetes/kubernetes">GitHub ▶</a> + <p>All the project and issue tracking, plus of course code.</p> + </div> + + <div id="server-fault" class="community-resource"> + <a href="https://serverfault.com/questions/tagged/kubernetes"> + <img src="/images/community/serverfault.png" alt="Server Fault"> + </a> + <a href="https://serverfault.com/questions/tagged/kubernetes">Server Fault ▶</a> + <p>Kubernetes-related discussion on Server Fault. Ask a question, or answer one.</p> + </div> + + <div id="slack" class="community-resource"> + <a href="https://kubernetes.slack.com/"> + <img src="/images/community/slack.png" alt="Slack"> + </a> + <a href="https://kubernetes.slack.com/">Slack ▶</a> + <p>With 170+ channels, you'll find one that fits your needs.</p> + <details><summary><em>Need an invitation?</em></summary> + Visit <a href="https://slack.k8s.io/">https://slack.k8s.io/</a> + for an invitation.</details> + </div> + </div> +</div> + +<div class="community-section" id="events"> + <div class="container"> + <h2>Upcoming Events</h2> + {{< upcoming-events >}} + </div> +</div> + +<div class="community-section" id="meetups"> + <h2>Global community</h2> + <p> + With over 150 meetups in the world and growing, go find your local kube people. If one isn't near, take charge and create your own. + </p> + <a href="https://www.meetup.com/topics/kubernetes/" class="community-cta-button"> + <span class="community-cta">Find a meetup</span> + </a> +</div> + +<div class="community-section community-frame" id="news"> + <h2>Recent News</h2> + <div class="twittercol1"> + <a class="twitter-timeline" data-tweet-limit="1" href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets by kubernetesio</a> + </div> +</div> diff --git a/content/en/community/code-of-conduct.md b/content/en/community/code-of-conduct.md index 5dd0cb28e8..84c95370a3 100644 --- a/content/en/community/code-of-conduct.md +++ b/content/en/community/code-of-conduct.md @@ -1,27 +1,29 @@ --- -title: Community +title: Kubernetes Community Code of Conduct layout: basic cid: community -css: /css/community.css +community_styles_migrated: true --- -<div class="community_main"> -<h1>Kubernetes Community Code of Conduct</h1> - +<div class="community-section" id="cncf-code-of-conduct-intro"> +<p> Kubernetes follows the -<a href="https://github.com/cncf/foundation/blob/master/code-of-conduct.md">CNCF Code of Conduct</a>. +<a href="https://github.com/cncf/foundation/blob/main/code-of-conduct.md">CNCF Code of Conduct</a>. The text of the CNCF CoC is replicated below, as of -<a href="https://github.com/cncf/foundation/blob/214585e24aab747fb85c2ea44fbf4a2442e30de6/code-of-conduct.md">commit 214585e</a>. +<a href="https://github.com/cncf/foundation/blob/71b12a2f8b4589788ef2d69b351a3d035c68d927/code-of-conduct.md">commit 71b12a2</a>. If you notice that this is out of date, please <a href="https://github.com/kubernetes/website/issues/new">file an issue</a>. +</p> +<p> If you notice a violation of the Code of Conduct at an event or meeting, in Slack, or in another communication mechanism, reach out to the <a href="https://git.k8s.io/community/committee-code-of-conduct">Kubernetes Code of Conduct Committee</a>. You can reach us by email at <a href="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. Your anonymity will be protected. +</p> +</div> -<div class="cncf_coc_container"> +<div id="cncf-code-of-conduct"> {{< include "/static/cncf-code-of-conduct.md" >}} </div> -</div> diff --git a/content/en/community/static/OWNERS b/content/en/community/static/OWNERS new file mode 100644 index 0000000000..3db354af14 --- /dev/null +++ b/content/en/community/static/OWNERS @@ -0,0 +1,7 @@ +# See the OWNERS docs at https://go.k8s.io/owners + +# Disable inheritance to encourage careful review of any changes here. +options: + no_parent_owners: true +approvers: +- sig-docs-leads diff --git a/content/en/community/static/README.md b/content/en/community/static/README.md index ef8e8d5a3e..bc44990c07 100644 --- a/content/en/community/static/README.md +++ b/content/en/community/static/README.md @@ -1,2 +1,5 @@ The files in this directory have been imported from other sources. Do not -edit them directly, except by replacing them with new versions. \ No newline at end of file +edit them directly, except by replacing them with new versions. + +Localization note: you do not need to create localized versions of any of + the files in this directory. \ No newline at end of file diff --git a/content/en/community/static/cncf-code-of-conduct.md b/content/en/community/static/cncf-code-of-conduct.md index d07444c418..fb3202b24a 100644 --- a/content/en/community/static/cncf-code-of-conduct.md +++ b/content/en/community/static/cncf-code-of-conduct.md @@ -1,45 +1,72 @@ <!-- Do not edit this file directly. Get the latest from - https://github.com/cncf/foundation/blob/master/code-of-conduct.md --> -## CNCF Community Code of Conduct v1.0 + https://github.com/cncf/foundation/blob/main/code-of-conduct.md --> +## CNCF Community Code of Conduct v1.1 ### Contributor Code of Conduct -As contributors and maintainers of this project, and in the interest of fostering +As contributors and maintainers in the CNCF community, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. -We are committed to making participation in this project a harassment-free experience for -everyone, regardless of level of experience, gender, gender identity and expression, +We are committed to making participation in the CNCF community a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. -Examples of unacceptable behavior by participants include: +## Scope -* The use of sexualized language or imagery -* Personal attacks -* Trolling or insulting/derogatory comments +This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. + +### CNCF Events + +CNCF events, or events run by the Linux Foundation with professional events staff, are governed by the Linux Foundation [Events Code of Conduct](https://events.linuxfoundation.org/code-of-conduct/) available on the event page. This is designed to be used in conjunction with the CNCF Code of Conduct. + +## Our Standards + +Examples of behavior that contributes to a positive environment include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment -* Publishing other's private information, such as physical or electronic addresses, - without explicit permission -* Other unethical or unprofessional conduct. +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting -Project maintainers have the right and responsibility to remove, edit, or reject -comments, commits, code, wiki edits, issues, and other contributions that are not -aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers -commit themselves to fairly and consistently applying these principles to every aspect -of managing this project. Project maintainers who do not follow or enforce the Code of +Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. +By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect +of managing this project. +Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team. -This code of conduct applies both within project spaces and in public spaces -when an individual is representing the project or its community. +## Reporting -Instances of abusive, harassing, or otherwise unacceptable behavior in Kubernetes may be reported by contacting the [Kubernetes Code of Conduct Committee](https://git.k8s.io/community/committee-code-of-conduct) via <conduct@kubernetes.io>. For other projects, please contact a CNCF project maintainer or our mediator, Mishi Choudhary <mishi@linux.com>. +For incidents occurring in the Kubernetes community, contact the [Kubernetes Code of Conduct Committee](https://git.k8s.io/community/committee-code-of-conduct) via <conduct@kubernetes.io>. You can expect a response within three business days. + +For other projects, please contact the CNCF staff via <conduct@cncf.io>. You can expect a response within three business days. + +In matters that require an outside mediator, CNCF has retained Mishi Choudhary (mishi@linux.com). Use of an outside mediator can be requested when reporting or used at CNCF staff's discretion. In general, contacting <conduct@cncf.io> directly is preferred. + + +## Enforcement + +The Kubernetes project's [Code of Conduct Committee](https://github.com/kubernetes/community/tree/master/committee-code-of-conduct) enforces code of conduct issues. For all other projects, the CNCF enforces code of conduct issues. + +Both bodies try to resolve incidents without punishment, but may remove people from the project or CNCF communities at their discretion. + +## Acknowledgements This Code of Conduct is adapted from the Contributor Covenant -(https://contributor-covenant.org), version 1.2.0, available at -https://contributor-covenant.org/version/1/2/0/ - -### CNCF Events Code of Conduct - -CNCF events are governed by the Linux Foundation [Code of Conduct](https://events.linuxfoundation.org/code-of-conduct/) available on the event page. This is designed to be compatible with the above policy and also includes more details on responding to incidents. +(http://contributor-covenant.org), version 2.0 available at +http://contributor-covenant.org/version/2/0/code_of_conduct/ \ No newline at end of file diff --git a/content/en/community/static/community-values.md b/content/en/community/static/community-values.md index f6469a3e61..6fd1a1a06c 100644 --- a/content/en/community/static/community-values.md +++ b/content/en/community/static/community-values.md @@ -3,26 +3,26 @@ # Kubernetes Community Values -Kubernetes Community culture is frequently cited as a substantial contributor to the meteoric rise of this Open Source project. Below are the distilled values which have evolved over the last many years in our community pushing our project and peers toward constant improvement. +Kubernetes Community culture contributes substantially to the project's success. The following values have evolved over time, pushing our project and peers toward constant improvement. ## Distribution is better than centralization -The scale of the Kubernetes project is only viable through high-trust and high-visibility distribution of work, which includes delegation of authority, decision making, technical design, code ownership, and documentation. Distributed asynchronous ownership, collaboration, communication and decision making are the cornerstone of our world-wide community. +The scale of the Kubernetes project is only viable through high-trust and high-visibility distribution of work, which includes delegation of authority, decision making, technical design, code ownership, and documentation. Distributed asynchronous ownership, collaboration, communication and decision making are the cornerstones of our world-wide community. ## Community over product or company -We are here as a community first, our allegiance is to the intentional stewardship of the Kubernetes project for the benefit of all its members and users everywhere. We support working together publicly for the common goal of a vibrant interoperable ecosystem providing an excellent experience for our users. Individuals gain status through work, companies gain status through their commitments to support this community and fund the resources necessary for the project to operate. +We are here as a community first. Our allegiance is to the intentional stewardship of the Kubernetes project for the benefit of all its members and users everywhere. We support working together publicly for the common goal of a vibrant interoperable ecosystem, providing an excellent experience for our users. Individuals gain status through work. Companies gain status through their commitments to support this community and fund the resources necessary for the project to operate. ## Automation over process -Large projects have a lot of less exciting, yet, hard work. We value time spent automating repetitive work more highly than toil. Where that work cannot be automated, it is our culture to recognize and reward all types of contributions. However, heroism is not sustainable. +Large projects have a lot of hard yet less exciting work. We value time spent automating repetitive work more highly than toil. Where work cannot be automated, our culture recognizes and rewards all types of contributions while recognizing that heroism is not sustainable. ## Inclusive is better than exclusive -Broadly successful and useful technology requires different perspectives and skill sets which can only be heard in a welcoming and respectful environment. Community membership is a privilege, not a right. Community Leadership is earned through effort, scope, quality, quantity, and duration of contributions. Our community shows respect for the time and effort put into a discussion regardless of where a contributor is on their growth path. +Broadly successful and useful technologies require different perspectives and skill sets, which can only be heard in a welcoming and respectful environment. Community membership is a privilege, not a right. Community members earn leadership through effort, scope, quality, quantity, and duration of contributions. Our community respects the time and effort put into a discussion, regardless of where a contributor is on their growth path. ## Evolution is better than stagnation -Openness to new ideas and studied technological evolution make Kubernetes a stronger project. Continual improvement, servant leadership, mentorship and respect are the foundations of the Kubernetes project culture. It is the duty for leaders in the Kubernetes community to find, sponsor, and promote new community members. Leaders should expect to step aside. Community members should expect to step up. +Openness to new ideas and studied technological evolution make Kubernetes a stronger project. Continual improvement, servant leadership, mentorship, and respect are the foundations of Kubernetes culture. Kubernetes community leaders have a duty to find, sponsor, and promote new community members. Leaders should expect to step aside. Community members should expect to step up. **"Culture eats strategy for breakfast." --Peter Drucker** diff --git a/content/en/community/values.md b/content/en/community/values.md index 4ae1fe30b6..2974dc1e43 100644 --- a/content/en/community/values.md +++ b/content/en/community/values.md @@ -1,13 +1,26 @@ --- -title: Community +title: Kubernetes Community Values layout: basic cid: community -css: /css/community.css +community_styles_migrated: true + +# this page is deprecated +# canonical page is https://www.kubernetes.dev/community/values/ +sitemap: + priority: 0.1 --- -<div class="community_main"> - -<div class="cncf_coc_container"> +<div class="community-section" id="values-legacy"> +<p> +This page is a replicated version of +<a href="https://www.kubernetes.dev/community/values/">Kubernetes Community Values</a>, as of +<a href="https://github.com/kubernetes/community/blob/5c642749a030c5f7b2363a6bb9bad00a56a92161/values.md">commit 5c64274</a>. +If you notice that this is out of date, please +<a href="https://github.com/kubernetes/website/issues/new">file an issue</a>. +</p> {{< include "/static/community-values.md" >}} </div> -</div> + +<!-- no need to localize this file, nor the contents of the static directory --> +<!-- if localizing, find the appropriate localized version of the CNCF code of + conduct, and link directly to that --> diff --git a/content/en/docs/concepts/architecture/cloud-controller.md b/content/en/docs/concepts/architecture/cloud-controller.md index 229cc489f9..e6eadaa56f 100644 --- a/content/en/docs/concepts/architecture/cloud-controller.md +++ b/content/en/docs/concepts/architecture/cloud-controller.md @@ -43,11 +43,11 @@ The controllers inside the cloud controller manager include: ### Node controller -The node controller is responsible for creating {{< glossary_tooltip text="Node" term_id="node" >}} objects +The node controller is responsible for updating {{< glossary_tooltip text="Node" term_id="node" >}} objects when new servers are created in your cloud infrastructure. The node controller obtains information about the hosts running inside your tenancy with the cloud provider. The node controller performs the following functions: -1. Initialize a Node object for each server that the controller discovers through the cloud provider API. +1. Update a Node object with the corresponding server's unique identifier obtained from the cloud provider API. 2. Annotating and labelling the Node object with cloud-specific information, such as the region the node is deployed into and the resources (CPU, memory, etc) that it has available. 3. Obtain the node's hostname and network addresses. @@ -78,7 +78,7 @@ when you declare a Service resource that requires them. ## Authorization -This section breaks down the access that the cloud controller managers requires +This section breaks down the access that the cloud controller manager requires on various API objects, in order to perform its operations. ### Node controller {#authorization-node-controller} diff --git a/content/en/docs/concepts/architecture/control-plane-node-communication.md b/content/en/docs/concepts/architecture/control-plane-node-communication.md index a4814aab4b..785040cda3 100644 --- a/content/en/docs/concepts/architecture/control-plane-node-communication.md +++ b/content/en/docs/concepts/architecture/control-plane-node-communication.md @@ -2,7 +2,7 @@ reviewers: - dchen1107 - liggitt -title: Control Plane-Node Communication +title: Communication between Nodes and the Control Plane content_type: concept weight: 20 aliases: @@ -11,62 +11,109 @@ aliases: <!-- overview --> -This document catalogs the communication paths between the control plane (apiserver) and the Kubernetes cluster. The intent is to allow users to customize their installation to harden the network configuration such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud provider). - - +This document catalogs the communication paths between the API server and the Kubernetes cluster. +The intent is to allow users to customize their installation to harden the network configuration +such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud +provider). <!-- body --> ## Node to Control Plane -Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run) terminates at the apiserver. None of the other control plane components are designed to expose remote services. The apiserver is configured to listen for remote connections on a secure HTTPS port (typically 443) with one or more forms of client [authentication](/docs/reference/access-authn-authz/authentication/) enabled. -One or more forms of [authorization](/docs/reference/access-authn-authz/authorization/) should be enabled, especially if [anonymous requests](/docs/reference/access-authn-authz/authentication/#anonymous-requests) or [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens) are allowed. -Nodes should be provisioned with the public root certificate for the cluster such that they can connect securely to the apiserver along with valid client credentials. A good approach is that the client credentials provided to the kubelet are in the form of a client certificate. See [kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) for automated provisioning of kubelet client certificates. +Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run) +terminates at the API server. None of the other control plane components are designed to expose +remote services. The API server is configured to listen for remote connections on a secure HTTPS +port (typically 443) with one or more forms of client +[authentication](/docs/reference/access-authn-authz/authentication/) enabled. +One or more forms of [authorization](/docs/reference/access-authn-authz/authorization/) should be +enabled, especially if [anonymous requests](/docs/reference/access-authn-authz/authentication/#anonymous-requests) +or [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens) +are allowed. -Pods that wish to connect to the apiserver can do so securely by leveraging a service account so that Kubernetes will automatically inject the public root certificate and a valid bearer token into the pod when it is instantiated. -The `kubernetes` service (in `default` namespace) is configured with a virtual IP address that is redirected (via kube-proxy) to the HTTPS endpoint on the apiserver. +Nodes should be provisioned with the public root certificate for the cluster such that they can +connect securely to the API server along with valid client credentials. A good approach is that the +client credentials provided to the kubelet are in the form of a client certificate. See +[kubelet TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) +for automated provisioning of kubelet client certificates. -The control plane components also communicate with the cluster apiserver over the secure port. +Pods that wish to connect to the API server can do so securely by leveraging a service account so +that Kubernetes will automatically inject the public root certificate and a valid bearer token +into the pod when it is instantiated. +The `kubernetes` service (in `default` namespace) is configured with a virtual IP address that is +redirected (via `kube-proxy`) to the HTTPS endpoint on the API server. -As a result, the default operating mode for connections from the nodes and pods running on the nodes to the control plane is secured by default and can run over untrusted and/or public networks. +The control plane components also communicate with the API server over the secure port. -## Control Plane to node +As a result, the default operating mode for connections from the nodes and pods running on the +nodes to the control plane is secured by default and can run over untrusted and/or public +networks. -There are two primary communication paths from the control plane (apiserver) to the nodes. The first is from the apiserver to the kubelet process which runs on each node in the cluster. The second is from the apiserver to any node, pod, or service through the apiserver's proxy functionality. +## Control plane to node -### apiserver to kubelet +There are two primary communication paths from the control plane (the API server) to the nodes. +The first is from the API server to the kubelet process which runs on each node in the cluster. +The second is from the API server to any node, pod, or service through the API server's _proxy_ +functionality. -The connections from the apiserver to the kubelet are used for: +### API server to kubelet + +The connections from the API server to the kubelet are used for: * Fetching logs for pods. -* Attaching (through kubectl) to running pods. +* Attaching (usually through `kubectl`) to running pods. * Providing the kubelet's port-forwarding functionality. -These connections terminate at the kubelet's HTTPS endpoint. By default, the apiserver does not verify the kubelet's serving certificate, which makes the connection subject to man-in-the-middle attacks and **unsafe** to run over untrusted and/or public networks. +These connections terminate at the kubelet's HTTPS endpoint. By default, the API server does not +verify the kubelet's serving certificate, which makes the connection subject to man-in-the-middle +attacks and **unsafe** to run over untrusted and/or public networks. -To verify this connection, use the `--kubelet-certificate-authority` flag to provide the apiserver with a root certificate bundle to use to verify the kubelet's serving certificate. +To verify this connection, use the `--kubelet-certificate-authority` flag to provide the API +server with a root certificate bundle to use to verify the kubelet's serving certificate. -If that is not possible, use [SSH tunneling](#ssh-tunnels) between the apiserver and kubelet if required to avoid connecting over an +If that is not possible, use [SSH tunneling](#ssh-tunnels) between the API server and kubelet if +required to avoid connecting over an untrusted or public network. -Finally, [Kubelet authentication and/or authorization](/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) should be enabled to secure the kubelet API. -### apiserver to nodes, pods, and services +Finally, [Kubelet authentication and/or authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/) +should be enabled to secure the kubelet API. -The connections from the apiserver to a node, pod, or service default to plain HTTP connections and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will not validate the certificate provided by the HTTPS endpoint nor provide client credentials. So while the connection will be encrypted, it will not provide any guarantees of integrity. These connections **are not currently safe** to run over untrusted or public networks. +### API server to nodes, pods, and services + +The connections from the API server to a node, pod, or service default to plain HTTP connections +and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS +connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will +not validate the certificate provided by the HTTPS endpoint nor provide client credentials. So +while the connection will be encrypted, it will not provide any guarantees of integrity. These +connections **are not currently safe** to run over untrusted or public networks. ### SSH tunnels -Kubernetes supports SSH tunnels to protect the control plane to nodes communication paths. In this configuration, the apiserver initiates an SSH tunnel to each node in the cluster (connecting to the ssh server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or service through the tunnel. -This tunnel ensures that the traffic is not exposed outside of the network in which the nodes are running. +Kubernetes supports SSH tunnels to protect the control plane to nodes communication paths. In this +configuration, the API server initiates an SSH tunnel to each node in the cluster (connecting to +the SSH server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or +service through the tunnel. +This tunnel ensures that the traffic is not exposed outside of the network in which the nodes are +running. -SSH tunnels are currently deprecated, so you shouldn't opt to use them unless you know what you are doing. The Konnectivity service is a replacement for this communication channel. +{{< note >}} +SSH tunnels are currently deprecated, so you shouldn't opt to use them unless you know what you +are doing. The [Konnectivity service](#konnectivity-service) is a replacement for this +communication channel. +{{< /note >}} ### Konnectivity service {{< feature-state for_k8s_version="v1.18" state="beta" >}} -As a replacement to the SSH tunnels, the Konnectivity service provides TCP level proxy for the control plane to cluster communication. The Konnectivity service consists of two parts: the Konnectivity server in the control plane network and the Konnectivity agents in the nodes network. The Konnectivity agents initiate connections to the Konnectivity server and maintain the network connections. -After enabling the Konnectivity service, all control plane to nodes traffic goes through these connections. +As a replacement to the SSH tunnels, the Konnectivity service provides TCP level proxy for the +control plane to cluster communication. The Konnectivity service consists of two parts: the +Konnectivity server in the control plane network and the Konnectivity agents in the nodes network. +The Konnectivity agents initiate connections to the Konnectivity server and maintain the network +connections. +After enabling the Konnectivity service, all control plane to nodes traffic goes through these +connections. + +Follow the [Konnectivity service task](/docs/tasks/extend-kubernetes/setup-konnectivity/) to set +up the Konnectivity service in your cluster. -Follow the [Konnectivity service task](/docs/tasks/extend-kubernetes/setup-konnectivity/) to set up the Konnectivity service in your cluster. diff --git a/content/en/docs/concepts/architecture/cri.md b/content/en/docs/concepts/architecture/cri.md new file mode 100644 index 0000000000..02f6d8a199 --- /dev/null +++ b/content/en/docs/concepts/architecture/cri.md @@ -0,0 +1,51 @@ +--- +title: Container Runtime Interface (CRI) +content_type: concept +weight: 50 +--- + +<!-- overview --> + +The CRI is a plugin interface which enables the kubelet to use a wide variety of +container runtimes, without having a need to recompile the cluster components. + +You need a working +{{<glossary_tooltip text="container runtime" term_id="container-runtime">}} on +each Node in your cluster, so that the +{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} can launch +{{< glossary_tooltip text="Pods" term_id="pod" >}} and their containers. + +{{< glossary_definition prepend="The Container Runtime Interface (CRI) is" term_id="container-runtime-interface" length="all" >}} + +<!-- body --> + +## The API {#api} + +{{< feature-state for_k8s_version="v1.23" state="stable" >}} + +The kubelet acts as a client when connecting to the container runtime via gRPC. +The runtime and image service endpoints have to be available in the container +runtime, which can be configured separately within the kubelet by using the +`--image-service-endpoint` and `--container-runtime-endpoint` [command line +flags](/docs/reference/command-line-tools-reference/kubelet) + +For Kubernetes v{{< skew currentVersion >}}, the kubelet prefers to use CRI `v1`. +If a container runtime does not support `v1` of the CRI, then the kubelet tries to +negotiate any older supported version. +The v{{< skew currentVersion >}} kubelet can also negotiate CRI `v1alpha2`, but +this version is considered as deprecated. +If the kubelet cannot negotiate a supported CRI version, the kubelet gives up +and doesn't register as a node. + +## Upgrading + +When upgrading Kubernetes, then the kubelet tries to automatically select the +latest CRI version on restart of the component. If that fails, then the fallback +will take place as mentioned above. If a gRPC re-dial was required because the +container runtime has been upgraded, then the container runtime must also +support the initially selected version or the redial is expected to fail. This +requires a restart of the kubelet. + +## {{% heading "whatsnext" %}} + +- Learn more about the CRI [protocol definition](https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto) diff --git a/content/en/docs/concepts/architecture/garbage-collection.md b/content/en/docs/concepts/architecture/garbage-collection.md index 7c70675fff..a6fa885bfc 100644 --- a/content/en/docs/concepts/architecture/garbage-collection.md +++ b/content/en/docs/concepts/architecture/garbage-collection.md @@ -13,7 +13,7 @@ allows the clean up of resources like the following: * [Objects without owner references](#owners-dependents) * [Unused containers and container images](#containers-images) * [Dynamically provisioned PersistentVolumes with a StorageClass reclaim policy of Delete](/docs/concepts/storage/persistent-volumes/#delete) - * [Stale or expired CertificateSigningRequests (CSRs)](/reference/access-authn-authz/certificate-signing-requests/#request-signing-process) + * [Stale or expired CertificateSigningRequests (CSRs)](/docs/reference/access-authn-authz/certificate-signing-requests/#request-signing-process) * {{<glossary_tooltip text="Nodes" term_id="node">}} deleted in the following scenarios: * On a cloud when the cluster uses a [cloud controller manager](/docs/concepts/architecture/cloud-controller/) * On-premises when the cluster uses an addon similar to a cloud controller @@ -124,7 +124,8 @@ resource type. ### Container image lifecycle Kubernetes manages the lifecycle of all images through its *image manager*, -which is part of the kubelet, with the cooperation of cadvisor. The kubelet +which is part of the kubelet, with the cooperation of +{{< glossary_tooltip text="cadvisor" term_id="cadvisor" >}}. The kubelet considers the following disk usage limits when making garbage collection decisions: @@ -136,7 +137,7 @@ collection, which deletes images in order based on the last time they were used, starting with the oldest first. The kubelet deletes images until disk usage reaches the `LowThresholdPercent` value. -### Container image garbage collection {#container-image-garbage-collection} +### Container garbage collection {#container-image-garbage-collection} The kubelet garbage collects unused containers based on the following variables, which you can define: @@ -151,11 +152,11 @@ which you can define: In addition to these variables, the kubelet garbage collects unidentified and deleted containers, typically starting with the oldest first. -`MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other +`MaxPerPodContainer` and `MaxContainers` may potentially conflict with each other in situations where retaining the maximum number of containers per Pod (`MaxPerPodContainer`) would go outside the allowable total of global dead containers (`MaxContainers`). In this situation, the kubelet adjusts -`MaxPodPerContainer` to address the conflict. A worst-case scenario would be to +`MaxPerPodContainer` to address the conflict. A worst-case scenario would be to downgrade `MaxPerPodContainer` to `1` and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. @@ -179,4 +180,4 @@ configure garbage collection: * Learn more about [ownership of Kubernetes objects](/docs/concepts/overview/working-with-objects/owners-dependents/). * Learn more about Kubernetes [finalizers](/docs/concepts/overview/working-with-objects/finalizers/). -* Learn about the [TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) (beta) that cleans up finished Jobs. \ No newline at end of file +* Learn about the [TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) (beta) that cleans up finished Jobs. diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md index 1d4f6455b7..f57179df8e 100644 --- a/content/en/docs/concepts/architecture/nodes.md +++ b/content/en/docs/concepts/architecture/nodes.md @@ -33,9 +33,9 @@ There are two main ways to have Nodes added to the {{< glossary_tooltip text="AP 1. The kubelet on a node self-registers to the control plane 2. You (or another human user) manually add a Node object -After you create a Node object, or the kubelet on a node self-registers, the -control plane checks whether the new Node object is valid. For example, if you -try to create a Node from the following JSON manifest: +After you create a Node {{< glossary_tooltip text="object" term_id="object" >}}, +or the kubelet on a node self-registers, the control plane checks whether the new Node object is +valid. For example, if you try to create a Node from the following JSON manifest: ```json { @@ -72,7 +72,8 @@ The name of a Node object must be a valid The [name](/docs/concepts/overview/working-with-objects/names#names) identifies a Node. Two Nodes cannot have the same name at the same time. Kubernetes also assumes that a resource with the same name is the same object. In case of a Node, it is implicitly assumed that an instance using the -same name will have the same state (e.g. network settings, root disk contents). This may lead to +same name will have the same state (e.g. network settings, root disk contents) +and attributes like node labels. This may lead to inconsistencies if an instance was modified without changing its name. If the Node needs to be replaced or updated significantly, the existing Node object needs to be removed from API server first and re-added after the update. @@ -84,19 +85,38 @@ register itself with the API server. This is the preferred pattern, used by mos For self-registration, the kubelet is started with the following options: - - `--kubeconfig` - Path to credentials to authenticate itself to the API server. - - `--cloud-provider` - How to talk to a {{< glossary_tooltip text="cloud provider" term_id="cloud-provider" >}} to read metadata about itself. - - `--register-node` - Automatically register with the API server. - - `--register-with-taints` - Register the node with the given list of {{< glossary_tooltip text="taints" term_id="taint" >}} (comma separated `<key>=<value>:<effect>`). +- `--kubeconfig` - Path to credentials to authenticate itself to the API server. +- `--cloud-provider` - How to talk to a {{< glossary_tooltip text="cloud provider" term_id="cloud-provider" >}} + to read metadata about itself. +- `--register-node` - Automatically register with the API server. +- `--register-with-taints` - Register the node with the given list of + {{< glossary_tooltip text="taints" term_id="taint" >}} (comma separated `<key>=<value>:<effect>`). - No-op if `register-node` is false. - - `--node-ip` - IP address of the node. - - `--node-labels` - {{< glossary_tooltip text="Labels" term_id="label" >}} to add when registering the node in the cluster (see label restrictions enforced by the [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction)). - - `--node-status-update-frequency` - Specifies how often kubelet posts node status to master. + No-op if `register-node` is false. +- `--node-ip` - IP address of the node. +- `--node-labels` - {{< glossary_tooltip text="Labels" term_id="label" >}} to add when registering the node + in the cluster (see label restrictions enforced by the + [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction)). +- `--node-status-update-frequency` - Specifies how often kubelet posts its node status to the API server. When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and -[NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) are enabled, -kubelets are only authorized to create/modify their own Node resource. +[NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) +are enabled, kubelets are only authorized to create/modify their own Node resource. + +{{< note >}} +As mentioned in the [Node name uniqueness](#node-name-uniqueness) section, +when Node configuration needs to be updated, it is a good practice to re-register +the node with the API server. For example, if the kubelet being restarted with +the new set of `--node-labels`, but the same Node name is used, the change will +not take an effect, as labels are being set on the Node registration. + +Pods already scheduled on the Node may misbehave or cause issues if the Node +configuration will be changed on kubelet restart. For example, already running +Pod may be tainted against the new labels assigned to the Node, while other +Pods, that are incompatible with that Pod will be scheduled based on this new +label. Node re-registration ensures all Pods will be drained and properly +re-scheduled. +{{< /note >}} ### Manual Node administration @@ -152,8 +172,10 @@ Each section of the output is described below. The usage of these fields varies depending on your cloud provider or bare metal configuration. -* HostName: The hostname as reported by the node's kernel. Can be overridden via the kubelet `--hostname-override` parameter. -* ExternalIP: Typically the IP address of the node that is externally routable (available from outside the cluster). +* HostName: The hostname as reported by the node's kernel. Can be overridden via the kubelet + `--hostname-override` parameter. +* ExternalIP: Typically the IP address of the node that is externally routable (available from + outside the cluster). * InternalIP: Typically the IP address of the node that is routable only within the cluster. @@ -273,7 +295,6 @@ and for updating their related Leases. updates to the Node's `.status`. If the Lease update fails, the kubelet retries, using exponential backoff that starts at 200 milliseconds and capped at 7 seconds. - ## Node controller The node {{< glossary_tooltip text="controller" term_id="controller" >}} is a @@ -290,16 +311,19 @@ controller deletes the node from its list of nodes. The third is monitoring the nodes' health. The node controller is responsible for: -- In the case that a node becomes unreachable, updating the NodeReady condition - of within the Node's `.status`. In this case the node controller sets the - NodeReady condition to `ConditionUnknown`. + +- In the case that a node becomes unreachable, updating the `Ready` condition + in the Node's `.status` field. In this case the node controller sets the + `Ready` condition to `Unknown`. - If a node remains unreachable: triggering [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/) for all of the Pods on the unreachable node. By default, the node controller - waits 5 minutes between marking the node as `ConditionUnknown` and submitting + waits 5 minutes between marking the node as `Unknown` and submitting the first eviction request. -The node controller checks the state of each node every `--node-monitor-period` seconds. +By default, the node controller checks the state of each node every 5 seconds. +This period can be configured using the `--node-monitor-period` flag on the +`kube-controller-manager` component. ### Rate limits on eviction @@ -309,8 +333,9 @@ from more than 1 node per 10 seconds. The node eviction behavior changes when a node in a given availability zone becomes unhealthy. The node controller checks what percentage of nodes in the zone -are unhealthy (NodeReady condition is `ConditionUnknown` or `ConditionFalse`) at +are unhealthy (the `Ready` condition is `Unknown` or `False`) at the same time: + - If the fraction of unhealthy nodes is at least `--unhealthy-zone-threshold` (default 0.55), then the eviction rate is reduced. - If the cluster is small (i.e. has less than or equal to @@ -319,7 +344,7 @@ the same time: (default 0.01) per second. The reason these policies are implemented per availability zone is because one -availability zone might become partitioned from the master while the others remain +availability zone might become partitioned from the control plane while the others remain connected. If your cluster does not span multiple cloud provider availability zones, then the eviction mechanism does not take per-zone unavailability into account. @@ -361,7 +386,7 @@ If you want to explicitly reserve resources for non-Pod processes, see ## Node topology -{{< feature-state state="alpha" for_k8s_version="v1.16" >}} +{{< feature-state state="beta" for_k8s_version="v1.18" >}} If you have enabled the `TopologyManager` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), then @@ -375,7 +400,9 @@ for more information. The kubelet attempts to detect node system shutdown and terminates pods running on the node. -Kubelet ensures that pods follow the normal [pod termination process](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) during the node shutdown. +Kubelet ensures that pods follow the normal +[pod termination process](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) +during the node shutdown. The Graceful node shutdown feature depends on systemd since it takes advantage of [systemd inhibitor locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to @@ -386,42 +413,190 @@ Graceful node shutdown is controlled with the `GracefulNodeShutdown` enabled by default in 1.21. Note that by default, both configuration options described below, -`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero, -thus not activating Graceful node shutdown functionality. -To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values. +`shutdownGracePeriod` and `shutdownGracePeriodCriticalPods` are set to zero, +thus not activating the graceful node shutdown functionality. +To activate the feature, the two kubelet config settings should be configured appropriately and +set to non-zero values. During a graceful shutdown, kubelet terminates pods in two phases: 1. Terminate regular pods running on the node. -2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node. +2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) + running on the node. -Graceful node shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options: -* `ShutdownGracePeriod`: - * Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical). -* `ShutdownGracePeriodCriticalPods`: - * Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `ShutdownGracePeriod`. +Graceful node shutdown feature is configured with two +[`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options: -For example, if `ShutdownGracePeriod=30s`, and -`ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by +* `shutdownGracePeriod`: + * Specifies the total duration that the node should delay the shutdown by. This is the total + grace period for pod termination for both regular and + [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical). +* `shutdownGracePeriodCriticalPods`: + * Specifies the duration used to terminate + [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) + during a node shutdown. This value should be less than `shutdownGracePeriod`. + +For example, if `shutdownGracePeriod=30s`, and +`shutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical). {{< note >}} -When pods were evicted during the graceful node shutdown, they are marked as failed. -Running `kubectl get pods` shows the status of the the evicted pods as `Shutdown`. +When pods were evicted during the graceful node shutdown, they are marked as shutdown. +Running `kubectl get pods` shows the status of the the evicted pods as `Terminated`. And `kubectl describe pod` indicates that the pod was evicted because of node shutdown: ``` -Status: Failed -Reason: Shutdown -Message: Node is shutting, evicting pods +Reason: Terminated +Message: Pod was terminated in response to imminent node shutdown. ``` -Failed pod objects will be preserved until explicitly deleted or [cleaned up by the GC](/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection). -This is a change of behavior compared to abrupt node termination. {{< /note >}} +## Non Graceful node shutdown {#non-graceful-node-shutdown} + +{{< feature-state state="alpha" for_k8s_version="v1.24" >}} + +A node shutdown action may not be detected by kubelet's Node Shutdown Manager, +either because the command does not trigger the inhibitor locks mechanism used by +kubelet or because of a user error, i.e., the ShutdownGracePeriod and +ShutdownGracePeriodCriticalPods are not configured properly. Please refer to above +section [Graceful Node Shutdown](#graceful-node-shutdown) for more details. + +When a node is shutdown but not detected by kubelet's Node Shutdown Manager, the pods +that are part of a StatefulSet will be stuck in terminating status on +the shutdown node and cannot move to a new running node. This is because kubelet on +the shutdown node is not available to delete the pods so the StatefulSet cannot +create a new pod with the same name. If there are volumes used by the pods, the +VolumeAttachments will not be deleted from the original shutdown node so the volumes +used by these pods cannot be attached to a new running node. As a result, the +application running on the StatefulSet cannot function properly. If the original +shutdown node comes up, the pods will be deleted by kubelet and new pods will be +created on a different running node. If the original shutdown node does not come up, +these pods will be stuck in terminating status on the shutdown node forever. + +To mitigate the above situation, a user can manually add the taint `node +kubernetes.io/out-of-service` with either `NoExecute` or `NoSchedule` effect to +a Node marking it out-of-service. +If the `NodeOutOfServiceVolumeDetach`[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +is enabled on `kube-controller-manager`, and a Node is marked out-of-service with this taint, the +pods on the node will be forcefully deleted if there are no matching tolerations on it and volume +detach operations for the pods terminating on the node will happen immediately. This allows the +Pods on the out-of-service node to recover quickly on a different node. + +During a non-graceful shutdown, Pods are terminated in the two phases: + +1. Force delete the Pods that do not have matching `out-of-service` tolerations. +2. Immediately perform detach volume operation for such pods. + +{{< note >}} +- Before adding the taint `node.kubernetes.io/out-of-service` , it should be verified + that the node is already in shutdown or power off state (not in the middle of + restarting). +- The user is required to manually remove the out-of-service taint after the pods are + moved to a new node and the user has checked that the shutdown node has been + recovered since the user was the one who originally added the taint. +{{< /note >}} + +### Pod Priority based graceful node shutdown {#pod-priority-graceful-node-shutdown} + +{{< feature-state state="alpha" for_k8s_version="v1.23" >}} + +To provide more flexibility during graceful node shutdown around the ordering +of pods during shutdown, graceful node shutdown honors the PriorityClass for +Pods, provided that you enabled this feature in your cluster. The feature +allows cluster administers to explicitly define the ordering of pods +during graceful node shutdown based on +[priority classes](/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass). + +The [Graceful Node Shutdown](#graceful-node-shutdown) feature, as described +above, shuts down pods in two phases, non-critical pods, followed by critical +pods. If additional flexibility is needed to explicitly define the ordering of +pods during shutdown in a more granular way, pod priority based graceful +shutdown can be used. + +When graceful node shutdown honors pod priorities, this makes it possible to do +graceful node shutdown in multiple phases, each phase shutting down a +particular priority class of pods. The kubelet can be configured with the exact +phases and shutdown time per phase. + +Assuming the following custom pod +[priority classes](/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass) +in a cluster, + +|Pod priority class name|Pod priority class value| +|-------------------------|------------------------| +|`custom-class-a` | 100000 | +|`custom-class-b` | 10000 | +|`custom-class-c` | 1000 | +|`regular/unset` | 0 | + +Within the [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) +the settings for `shutdownGracePeriodByPodPriority` could look like: + +|Pod priority class value|Shutdown period| +|------------------------|---------------| +| 100000 |10 seconds | +| 10000 |180 seconds | +| 1000 |120 seconds | +| 0 |60 seconds | + +The corresponding kubelet config YAML configuration would be: + +```yaml +shutdownGracePeriodByPodPriority: + - priority: 100000 + shutdownGracePeriodSeconds: 10 + - priority: 10000 + shutdownGracePeriodSeconds: 180 + - priority: 1000 + shutdownGracePeriodSeconds: 120 + - priority: 0 + shutdownGracePeriodSeconds: 60 +``` + +The above table implies that any pod with `priority` value >= 100000 will get +just 10 seconds to stop, any pod with value >= 10000 and < 100000 will get 180 +seconds to stop, any pod with value >= 1000 and < 10000 will get 120 seconds to stop. +Finally, all other pods will get 60 seconds to stop. + +One doesn't have to specify values corresponding to all of the classes. For +example, you could instead use these settings: + +|Pod priority class value|Shutdown period| +|------------------------|---------------| +| 100000 |300 seconds | +| 1000 |120 seconds | +| 0 |60 seconds | + + +In the above case, the pods with `custom-class-b` will go into the same bucket +as `custom-class-c` for shutdown. + +If there are no pods in a particular range, then the kubelet does not wait +for pods in that priority range. Instead, the kubelet immediately skips to the +next priority class value range. + +If this feature is enabled and no configuration is provided, then no ordering +action will be taken. + +Using this feature requires enabling the `GracefulNodeShutdownBasedOnPodPriority` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +, and setting `ShutdownGracePeriodByPodPriority` in the +[kubelet config](/docs/reference/config-api/kubelet-config.v1beta1/) +to the desired configuration containing the pod priority class values and +their respective shutdown periods. + +{{< note >}} +The ability to take Pod priority into account during graceful node shutdown was introduced +as an Alpha feature in Kubernetes v1.23. In Kubernetes {{< skew currentVersion >}} +the feature is Beta and is enabled by default. +{{< /note >}} + +Metrics `graceful_shutdown_start_time_seconds` and `graceful_shutdown_end_time_seconds` +are emitted under the kubelet subsystem to monitor node shutdowns. + ## Swap memory management {#swap-memory} {{< feature-state state="alpha" for_k8s_version="v1.22" >}} @@ -435,6 +610,11 @@ the kubelet, and the `--fail-swap-on` command line flag or `failSwapOn` [configuration setting](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) must be set to false. +{{< warning >}} +When the memory swap feature is turned on, Kubernetes data such as the content +of Secret objects that were written to tmpfs now could be swapped to disk. +{{< /warning >}} + A user can also optionally configure `memorySwap.swapBehavior` in order to specify how a node will use swap memory. For example, @@ -469,6 +649,7 @@ see [KEP-2400](https://github.com/kubernetes/enhancements/issues/2400) and its * Learn about the [components](/docs/concepts/overview/components/#node-components) that make up a node. * Read the [API definition for Node](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#node-v1-core). -* Read the [Node](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md#the-kubernetes-node) +* Read the [Node](https://git.k8s.io/design-proposals-archive/architecture/architecture.md#the-kubernetes-node) section of the architecture design document. * Read about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/). + diff --git a/content/en/docs/concepts/cluster-administration/_index.md b/content/en/docs/concepts/cluster-administration/_index.md index 7d5aec5078..d5d6a273e2 100644 --- a/content/en/docs/concepts/cluster-administration/_index.md +++ b/content/en/docs/concepts/cluster-administration/_index.md @@ -11,31 +11,37 @@ no_list: true --- <!-- overview --> + The cluster administration overview is for anyone creating or administering a Kubernetes cluster. It assumes some familiarity with core Kubernetes [concepts](/docs/concepts/). - <!-- body --> + ## Planning a cluster -See the guides in [Setup](/docs/setup/) for examples of how to plan, set up, and configure Kubernetes clusters. The solutions listed in this article are called *distros*. +See the guides in [Setup](/docs/setup/) for examples of how to plan, set up, and configure +Kubernetes clusters. The solutions listed in this article are called *distros*. - {{< note >}} - Not all distros are actively maintained. Choose distros which have been tested with a recent version of Kubernetes. - {{< /note >}} +{{< note >}} +Not all distros are actively maintained. Choose distros which have been tested with a recent +version of Kubernetes. +{{< /note >}} Before choosing a guide, here are some considerations: - - Do you want to try out Kubernetes on your computer, or do you want to build a high-availability, multi-node cluster? Choose distros best suited for your needs. - - Will you be using **a hosted Kubernetes cluster**, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), or **hosting your own cluster**? - - Will your cluster be **on-premises**, or **in the cloud (IaaS)**? Kubernetes does not directly support hybrid clusters. Instead, you can set up multiple clusters. - - **If you are configuring Kubernetes on-premises**, consider which [networking model](/docs/concepts/cluster-administration/networking/) fits best. - - Will you be running Kubernetes on **"bare metal" hardware** or on **virtual machines (VMs)**? - - Do you **want to run a cluster**, or do you expect to do **active development of Kubernetes project code**? If the - latter, choose an actively-developed distro. Some distros only use binary releases, but - offer a greater variety of choices. - - Familiarize yourself with the [components](/docs/concepts/overview/components/) needed to run a cluster. - +- Do you want to try out Kubernetes on your computer, or do you want to build a high-availability, + multi-node cluster? Choose distros best suited for your needs. +- Will you be using **a hosted Kubernetes cluster**, such as + [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), or **hosting your own cluster**? +- Will your cluster be **on-premises**, or **in the cloud (IaaS)**? Kubernetes does not directly + support hybrid clusters. Instead, you can set up multiple clusters. +- **If you are configuring Kubernetes on-premises**, consider which + [networking model](/docs/concepts/cluster-administration/networking/) fits best. +- Will you be running Kubernetes on **"bare metal" hardware** or on **virtual machines (VMs)**? +- Do you **want to run a cluster**, or do you expect to do **active development of Kubernetes project code**? + If the latter, choose an actively-developed distro. Some distros only use binary releases, but + offer a greater variety of choices. +- Familiarize yourself with the [components](/docs/concepts/overview/components/) needed to run a cluster. ## Managing a cluster @@ -45,29 +51,43 @@ Before choosing a guide, here are some considerations: ## Securing a cluster -* [Generate Certificates](/docs/tasks/administer-cluster/certificates/) describes the steps to generate certificates using different tool chains. +* [Generate Certificates](/docs/tasks/administer-cluster/certificates/) describes the steps to + generate certificates using different tool chains. -* [Kubernetes Container Environment](/docs/concepts/containers/container-environment/) describes the environment for Kubelet managed containers on a Kubernetes node. +* [Kubernetes Container Environment](/docs/concepts/containers/container-environment/) describes + the environment for Kubelet managed containers on a Kubernetes node. -* [Controlling Access to the Kubernetes API](/docs/concepts/security/controlling-access) describes how Kubernetes implements access control for its own API. +* [Controlling Access to the Kubernetes API](/docs/concepts/security/controlling-access) describes + how Kubernetes implements access control for its own API. -* [Authenticating](/docs/reference/access-authn-authz/authentication/) explains authentication in Kubernetes, including the various authentication options. +* [Authenticating](/docs/reference/access-authn-authz/authentication/) explains authentication in + Kubernetes, including the various authentication options. -* [Authorization](/docs/reference/access-authn-authz/authorization/) is separate from authentication, and controls how HTTP calls are handled. +* [Authorization](/docs/reference/access-authn-authz/authorization/) is separate from + authentication, and controls how HTTP calls are handled. -* [Using Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/) explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization. +* [Using Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/) + explains plug-ins which intercepts requests to the Kubernetes API server after authentication + and authorization. -* [Using Sysctls in a Kubernetes Cluster](/docs/tasks/administer-cluster/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters . +* [Using Sysctls in a Kubernetes Cluster](/docs/tasks/administer-cluster/sysctl-cluster/) + describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters +. -* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs. +* [Auditing](/docs/tasks/debug/debug-cluster/audit/) describes how to interact with Kubernetes' + audit logs. ### Securing the kubelet - * [Control Plane-Node communication](/docs/concepts/architecture/control-plane-node-communication/) - * [TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) - * [Kubelet authentication/authorization](/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) + +* [Control Plane-Node communication](/docs/concepts/architecture/control-plane-node-communication/) +* [TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) +* [Kubelet authentication/authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/) ## Optional Cluster Services -* [DNS Integration](/docs/concepts/services-networking/dns-pod-service/) describes how to resolve a DNS name directly to a Kubernetes service. +* [DNS Integration](/docs/concepts/services-networking/dns-pod-service/) describes how to resolve + a DNS name directly to a Kubernetes service. + +* [Logging and Monitoring Cluster Activity](/docs/concepts/cluster-administration/logging/) + explains how logging in Kubernetes works and how to implement it. -* [Logging and Monitoring Cluster Activity](/docs/concepts/cluster-administration/logging/) explains how logging in Kubernetes works and how to implement it. diff --git a/content/en/docs/concepts/cluster-administration/addons.md b/content/en/docs/concepts/cluster-administration/addons.md index 5ed93ad20b..5f7df077c9 100644 --- a/content/en/docs/concepts/cluster-administration/addons.md +++ b/content/en/docs/concepts/cluster-administration/addons.md @@ -18,19 +18,19 @@ This page lists some of the available add-ons and links to their respective inst * [ACI](https://www.github.com/noironetworks/aci-containers) provides integrated container networking and network security with Cisco ACI. * [Antrea](https://antrea.io/) operates at Layer 3/4 to provide networking and security services for Kubernetes, leveraging Open vSwitch as the networking data plane. * [Calico](https://docs.projectcalico.org/latest/introduction/) is a networking and network policy provider. Calico supports a flexible set of networking options so you can choose the most efficient option for your situation, including non-overlay and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio & Envoy) applications at the service mesh layer. -* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) unites Flannel and Calico, providing networking and network policy. +* [Canal](https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel) unites Flannel and Calico, providing networking and network policy. * [Cilium](https://github.com/cilium/cilium) is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported, and it can work on top of other CNI plugins. -* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave. -* [Contiv](https://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](https://github.com/contiv). The [installer](https://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options. +* [CNI-Genie](https://github.com/cni-genie/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, or Weave. +* [Contiv](https://contivpp.io/) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](https://github.com/contiv). The [installer](https://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options. * [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads. * [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes. * [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod. -* [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes. +* [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes. * [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) is a networking provider for Kubernetes based on [OVN (Open Virtual Network)](https://github.com/ovn-org/ovn/), a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy. -* [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking -* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift. +* [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking. +* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/index.html) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift. * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring. -* [Romana](https://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize). +* [Romana](https://github.com/romana) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy](/docs/concepts/services-networking/network-policies/) API. * [Weave Net](https://www.weave.works/docs/net/latest/kubernetes/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database. ## Service Discovery @@ -45,6 +45,11 @@ This page lists some of the available add-ons and links to their respective inst ## Infrastructure * [KubeVirt](https://kubevirt.io/user-guide/#/installation/installation) is an add-on to run virtual machines on Kubernetes. Usually run on bare-metal clusters. +* The + [node problem detector](https://github.com/kubernetes/node-problem-detector) + runs on Linux nodes and reports system issues as either + [Events](/docs/reference/kubernetes-api/cluster-resources/event-v1/) or + [Node conditions](/docs/concepts/architecture/nodes/#condition). ## Legacy Add-ons diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md index 46f0a1eadc..ccf1eb5887 100644 --- a/content/en/docs/concepts/cluster-administration/flow-control.md +++ b/content/en/docs/concepts/cluster-administration/flow-control.md @@ -26,6 +26,10 @@ fair queuing technique so that, for example, a poorly-behaved {{< glossary_tooltip text="controller" term_id="controller" >}} need not starve others (even at the same priority level). +This feature is designed to work well with standard controllers, which +use informers and react to failures of API requests with exponential +back-off, and other clients that also work this way. + {{< caution >}} Requests classified as "long-running" — primarily watches — are not subject to the API Priority and Fairness filter. This is also true for @@ -38,21 +42,21 @@ Fairness feature enabled. ## Enabling/Disabling API Priority and Fairness The API Priority and Fairness feature is controlled by a feature gate -and is enabled by default. See -[Feature Gates](/docs/reference/command-line-tools-reference/feature-gates/) +and is enabled by default. See [Feature +Gates](/docs/reference/command-line-tools-reference/feature-gates/) for a general explanation of feature gates and how to enable and disable them. The name of the feature gate for APF is "APIPriorityAndFairness". This feature also involves an {{< glossary_tooltip term_id="api-group" text="API Group" >}} with: (a) a -`v1alpha1` version, disabled by default, and (b) a `v1beta1` -version, enabled by default. You can disable the feature -gate and API group v1beta1 version by adding the following +`v1alpha1` version, disabled by default, and (b) `v1beta1` and +`v1beta2` versions, enabled by default. You can disable the feature +gate and API group beta versions by adding the following command-line flags to your `kube-apiserver` invocation: ```shell kube-apiserver \ --feature-gates=APIPriorityAndFairness=false \ ---runtime-config=flowcontrol.apiserver.k8s.io/v1beta1=false \ +--runtime-config=flowcontrol.apiserver.k8s.io/v1beta1=false,flowcontrol.apiserver.k8s.io/v1beta2=false \ # …and other flags as usual ``` @@ -102,6 +106,8 @@ name of the matching FlowSchema plus a _flow distinguisher_ — which is either the requesting user, the target resource's namespace, or nothing — and the system attempts to give approximately equal weight to requests in different flows of the same priority level. +To enable distinct handling of distinct instances, controllers that have +many instances should authenticate with distinct usernames After classifying a request into a flow, the API Priority and Fairness feature then may assign the request to a queue. This assignment uses @@ -121,86 +127,13 @@ any of the limitations imposed by this feature. These exemptions prevent an improperly-configured flow control configuration from totally disabling an API server. -## Defaults - -The Priority and Fairness feature ships with a suggested configuration that -should suffice for experimentation; if your cluster is likely to -experience heavy load then you should consider what configuration will work -best. The suggested configuration groups requests into five priority -classes: - -* The `system` priority level is for requests from the `system:nodes` group, - i.e. Kubelets, which must be able to contact the API server in order for - workloads to be able to schedule on them. - -* The `leader-election` priority level is for leader election requests from - built-in controllers (in particular, requests for `endpoints`, `configmaps`, - or `leases` coming from the `system:kube-controller-manager` or - `system:kube-scheduler` users and service accounts in the `kube-system` - namespace). These are important to isolate from other traffic because failures - in leader election cause their controllers to fail and restart, which in turn - causes more expensive traffic as the new controllers sync their informers. - -* The `workload-high` priority level is for other requests from built-in - controllers. - -* The `workload-low` priority level is for requests from any other service - account, which will typically include all requests from controllers running in - Pods. - -* The `global-default` priority level handles all other traffic, e.g. - interactive `kubectl` commands run by nonprivileged users. - -Additionally, there are two PriorityLevelConfigurations and two FlowSchemas that -are built in and may not be overwritten: - -* The special `exempt` priority level is used for requests that are not subject - to flow control at all: they will always be dispatched immediately. The - special `exempt` FlowSchema classifies all requests from the `system:masters` - group into this priority level. You may define other FlowSchemas that direct - other requests to this priority level, if appropriate. - -* The special `catch-all` priority level is used in combination with the special - `catch-all` FlowSchema to make sure that every request gets some kind of - classification. Typically you should not rely on this catch-all configuration, - and should create your own catch-all FlowSchema and PriorityLevelConfiguration - (or use the `global-default` configuration that is installed by default) as - appropriate. To help catch configuration errors that miss classifying some - requests, the mandatory `catch-all` priority level only allows one concurrency - share and does not queue requests, making it relatively likely that traffic - that only matches the `catch-all` FlowSchema will be rejected with an HTTP 429 - error. - -## Health check concurrency exemption - -The suggested configuration gives no special treatment to the health -check requests on kube-apiservers from their local kubelets --- which -tend to use the secured port but supply no credentials. With the -suggested config, these requests get assigned to the `global-default` -FlowSchema and the corresponding `global-default` priority level, -where other traffic can crowd them out. - -If you add the following additional FlowSchema, this exempts those -requests from rate limiting. - -{{< caution >}} -Making this change also allows any hostile party to then send -health-check requests that match this FlowSchema, at any volume they -like. If you have a web traffic filter or similar external security -mechanism to protect your cluster's API server from general internet -traffic, you can configure rules to block any health check requests -that originate from outside your cluster. -{{< /caution >}} - -{{< codenew file="priority-and-fairness/health-for-strangers.yaml" >}} - ## Resources The flow control API involves two kinds of resources. -[PriorityLevelConfigurations](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#prioritylevelconfiguration-v1beta1-flowcontrol-apiserver-k8s-io) +[PriorityLevelConfigurations](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#prioritylevelconfiguration-v1beta2-flowcontrol-apiserver-k8s-io) define the available isolation classes, the share of the available concurrency budget that each can handle, and allow for fine-tuning queuing behavior. -[FlowSchemas](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#flowschema-v1beta1-flowcontrol-apiserver-k8s-io) +[FlowSchemas](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#flowschema-v1beta2-flowcontrol-apiserver-k8s-io) are used to classify individual inbound requests, matching each to a single PriorityLevelConfiguration. There is also a `v1alpha1` version of the same API group, and it has the same Kinds with the same syntax and @@ -241,7 +174,7 @@ to balance progress between request flows. The queuing configuration allows tuning the fair queuing algorithm for a priority level. Details of the algorithm can be read in the -[enhancement proposal](#whats-next), but in short: +[enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness), but in short: * Increasing `queues` reduces the rate of collisions between different flows, at the cost of increased memory usage. A value of 1 here effectively disables the @@ -323,6 +256,153 @@ omitted entirely), in which case all requests matched by this FlowSchema will be considered part of a single flow. The correct choice for a given FlowSchema depends on the resource and your particular environment. +## Defaults + +Each kube-apiserver maintains two sorts of APF configuration objects: +mandatory and suggested. + +### Mandatory Configuration Objects + +The four mandatory configuration objects reflect fixed built-in +guardrail behavior. This is behavior that the servers have before +those objects exist, and when those objects exist their specs reflect +this behavior. The four mandatory objects are as follows. + +* The mandatory `exempt` priority level is used for requests that are + not subject to flow control at all: they will always be dispatched + immediately. The mandatory `exempt` FlowSchema classifies all + requests from the `system:masters` group into this priority + level. You may define other FlowSchemas that direct other requests + to this priority level, if appropriate. + +* The mandatory `catch-all` priority level is used in combination with + the mandatory `catch-all` FlowSchema to make sure that every request + gets some kind of classification. Typically you should not rely on + this catch-all configuration, and should create your own catch-all + FlowSchema and PriorityLevelConfiguration (or use the suggested + `global-default` priority level that is installed by default) as + appropriate. Because it is not expected to be used normally, the + mandatory `catch-all` priority level has a very small concurrency + share and does not queue requests. + +### Suggested Configuration Objects + +The suggested FlowSchemas and PriorityLevelConfigurations constitute a +reasonable default configuration. You can modify these and/or create +additional configuration objects if you want. If your cluster is +likely to experience heavy load then you should consider what +configuration will work best. + +The suggested configuration groups requests into six priority levels: + +* The `node-high` priority level is for health updates from nodes. + +* The `system` priority level is for non-health requests from the + `system:nodes` group, i.e. Kubelets, which must be able to contact + the API server in order for workloads to be able to schedule on + them. + +* The `leader-election` priority level is for leader election requests from + built-in controllers (in particular, requests for `endpoints`, `configmaps`, + or `leases` coming from the `system:kube-controller-manager` or + `system:kube-scheduler` users and service accounts in the `kube-system` + namespace). These are important to isolate from other traffic because failures + in leader election cause their controllers to fail and restart, which in turn + causes more expensive traffic as the new controllers sync their informers. + +* The `workload-high` priority level is for other requests from built-in + controllers. + +* The `workload-low` priority level is for requests from any other service + account, which will typically include all requests from controllers running in + Pods. + +* The `global-default` priority level handles all other traffic, e.g. + interactive `kubectl` commands run by nonprivileged users. + +The suggested FlowSchemas serve to steer requests into the above +priority levels, and are not enumerated here. + +### Maintenance of the Mandatory and Suggested Configuration Objects + +Each `kube-apiserver` independently maintains the mandatory and +suggested configuration objects, using initial and periodic behavior. +Thus, in a situation with a mixture of servers of different versions +there may be thrashing as long as different servers have different +opinions of the proper content of these objects. + +Each `kube-apiserver` makes an initial maintenance pass over the +mandatory and suggested configuration objects, and after that does +periodic maintenance (once per minute) of those objects. + +For the mandatory configuration objects, maintenance consists of +ensuring that the object exists and, if it does, has the proper spec. +The server refuses to allow a creation or update with a spec that is +inconsistent with the server's guardrail behavior. + +Maintenance of suggested configuration objects is designed to allow +their specs to be overridden. Deletion, on the other hand, is not +respected: maintenance will restore the object. If you do not want a +suggested configuration object then you need to keep it around but set +its spec to have minimal consequences. Maintenance of suggested +objects is also designed to support automatic migration when a new +version of the `kube-apiserver` is rolled out, albeit potentially with +thrashing while there is a mixed population of servers. + +Maintenance of a suggested configuration object consists of creating +it --- with the server's suggested spec --- if the object does not +exist. OTOH, if the object already exists, maintenance behavior +depends on whether the `kube-apiservers` or the users control the +object. In the former case, the server ensures that the object's spec +is what the server suggests; in the latter case, the spec is left +alone. + +The question of who controls the object is answered by first looking +for an annotation with key `apf.kubernetes.io/autoupdate-spec`. If +there is such an annotation and its value is `true` then the +kube-apiservers control the object. If there is such an annotation +and its value is `false` then the users control the object. If +neither of those condtions holds then the `metadata.generation` of the +object is consulted. If that is 1 then the kube-apiservers control +the object. Otherwise the users control the object. These rules were +introduced in release 1.22 and their consideration of +`metadata.generation` is for the sake of migration from the simpler +earlier behavior. Users who wish to control a suggested configuration +object should set its `apf.kubernetes.io/autoupdate-spec` annotation +to `false`. + +Maintenance of a mandatory or suggested configuration object also +includes ensuring that it has an `apf.kubernetes.io/autoupdate-spec` +annotation that accurately reflects whether the kube-apiservers +control the object. + +Maintenance also includes deleting objects that are neither mandatory +nor suggested but are annotated +`apf.kubernetes.io/autoupdate-spec=true`. + +## Health check concurrency exemption + +The suggested configuration gives no special treatment to the health +check requests on kube-apiservers from their local kubelets --- which +tend to use the secured port but supply no credentials. With the +suggested config, these requests get assigned to the `global-default` +FlowSchema and the corresponding `global-default` priority level, +where other traffic can crowd them out. + +If you add the following additional FlowSchema, this exempts those +requests from rate limiting. + +{{< caution >}} +Making this change also allows any hostile party to then send +health-check requests that match this FlowSchema, at any volume they +like. If you have a web traffic filter or similar external security +mechanism to protect your cluster's API server from general internet +traffic, you can configure rules to block any health check requests +that originate from outside your cluster. +{{< /caution >}} + +{{< codenew file="priority-and-fairness/health-for-strangers.yaml" >}} + ## Diagnostics Every HTTP response from an API server with the priority and fairness feature @@ -391,11 +471,15 @@ poorly-behaved workloads that may be harming system health. requests, broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on the values `mutating` and `readOnly`). The observations are made - periodically at a high rate. + periodically at a high rate. Each observed value is a ratio, + between 0 and 1, of a number of requests divided by the + corresponding limit on the number of requests (queue length limit + for waiting and concurrency limit for executing). * `apiserver_flowcontrol_read_vs_write_request_count_watermarks` is a histogram vector of high or low water marks of the number of - requests broken down by the labels `phase` (which takes on the + requests (divided by the corresponding limit to get a ratio in the + range 0 to 1) broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on the values `mutating` and `readOnly`); the label `mark` takes on values `high` and `low`. The water marks are accumulated over @@ -422,11 +506,15 @@ poorly-behaved workloads that may be harming system health. values `waiting` and `executing`) and `priority_level`. Each histogram gets observations taken periodically, up through the last activity of the relevant sort. The observations are made at a high - rate. + rate. Each observed value is a ratio, between 0 and 1, of a number + of requests divided by the corresponding limit on the number of + requests (queue length limit for waiting and concurrency limit for + executing). * `apiserver_flowcontrol_priority_level_request_count_watermarks` is a histogram vector of high or low water marks of the number of - requests broken down by the labels `phase` (which takes on the + requests (divided by the corresponding limit to get a ratio in the + range 0 to 1) broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `priority_level`; the label `mark` takes on values `high` and `low`. The water marks are accumulated over windows bounded by the times when an observation @@ -434,6 +522,31 @@ poorly-behaved workloads that may be harming system health. `apiserver_flowcontrol_priority_level_request_count_samples`. These water marks show the range of values that occurred between samples. +* `apiserver_flowcontrol_priority_level_seat_count_samples` is a + histogram vector of observations of the utilization of a priority + level's concurrency limit, broken down by `priority_level`. This + utilization is the fraction (number of seats occupied) / + (concurrency limit). This metric considers all stages of execution + (both normal and the extra delay at the end of a write to cover for + the corresponding notification work) of all requests except WATCHes; + for those it considers only the initial stage that delivers + notifications of pre-existing objects. Each histogram in the vector + is also labeled with `phase: executing` (there is no seat limit for + the waiting phase). Each histogram gets observations taken + periodically, up through the last activity of the relevant sort. + The observations + are made at a high rate. + +* `apiserver_flowcontrol_priority_level_seat_count_watermarks` is a + histogram vector of high or low water marks of the utilization of a + priority level's concurrency limit, broken down by `priority_level` + and `mark` (which takes on values `high` and `low`). Each histogram + in the vector is also labeled with `phase: executing` (there is no + seat limit for the waiting phase). The water marks are accumulated + over windows bounded by the times when an observation was added to + `apiserver_flowcontrol_priority_level_seat_count_samples`. These + water marks show the range of values that occurred between samples. + * `apiserver_flowcontrol_request_queue_length_after_enqueue` is a histogram vector of queue lengths for the queues, broken down by the labels `priority_level` and `flow_schema`, as sampled by the @@ -476,6 +589,22 @@ poorly-behaved workloads that may be harming system health. and `priority_level` (indicating the one to which the request was assigned). +* `apiserver_flowcontrol_watch_count_samples` is a histogram vector of + the number of active WATCH requests relevant to a given write, + broken down by `flow_schema` and `priority_level`. + +* `apiserver_flowcontrol_work_estimated_seats` is a histogram vector + of the number of estimated seats (maximum of initial and final stage + of execution) associated with requests, broken down by `flow_schema` + and `priority_level`. + +* `apiserver_flowcontrol_request_dispatch_no_accommodation_total` is a + counter vec of the number of events that in principle could have led + to a request being dispatched but did not, due to lack of available + concurrency, broken down by `flow_schema` and `priority_level`. The + relevant sorts of events are arrival of a request and completion of + a request. + ### Debug endpoints When you enable the API Priority and Fairness feature, the `kube-apiserver` diff --git a/content/en/docs/concepts/cluster-administration/logging.md b/content/en/docs/concepts/cluster-administration/logging.md index 1bf057f23e..4916548a73 100644 --- a/content/en/docs/concepts/cluster-administration/logging.md +++ b/content/en/docs/concepts/cluster-administration/logging.md @@ -12,7 +12,9 @@ weight: 60 Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams. However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. -For example, you may want access your application's logs if a container crashes; a pod gets evicted; or a node dies. + +For example, you may want to access your application's logs if a container crashes, a pod gets evicted, or a node dies. + In a cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level logging_. <!-- body --> @@ -55,7 +57,15 @@ The output is: ... ``` -You can use `kubectl logs --previous` to retrieve logs from a previous instantiation of a container. If your pod has multiple containers, specify which container's logs you want to access by appending a container name to the command. See the [`kubectl logs` documentation](/docs/reference/generated/kubectl/kubectl-commands#logs) for more details. +You can use `kubectl logs --previous` to retrieve logs from a previous instantiation of a container. +If your pod has multiple containers, specify which container's logs you want to access by +appending a container name to the command, with a `-c` flag, like so: + +```console +kubectl logs counter -c count +``` + +See the [`kubectl logs` documentation](/docs/reference/generated/kubectl/kubectl-commands#logs) for more details. ## Logging at the node level @@ -141,7 +151,7 @@ as a `DaemonSet`. Node-level logging creates only one agent per node and doesn't require any changes to the applications running on the node. -Containers write stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation. +Containers write to stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation. ### Using a sidecar container with the logging agent {#sidecar-container-with-logging-agent} diff --git a/content/en/docs/concepts/cluster-administration/manage-deployment.md b/content/en/docs/concepts/cluster-administration/manage-deployment.md index 4d98cf820c..c90715da09 100644 --- a/content/en/docs/concepts/cluster-administration/manage-deployment.md +++ b/content/en/docs/concepts/cluster-administration/manage-deployment.md @@ -154,7 +154,7 @@ deployment.apps/my-deployment created persistentvolumeclaim/my-pvc created ``` -If you're interested in learning more about `kubectl`, go ahead and read [kubectl Overview](/docs/reference/kubectl/overview/). +If you're interested in learning more about `kubectl`, go ahead and read [Command line tool (kubectl)](/docs/reference/kubectl/). ## Using labels effectively @@ -461,7 +461,7 @@ That's it! The Deployment will declaratively update the deployed nginx applicati ## {{% heading "whatsnext" %}} -- Learn about [how to use `kubectl` for application introspection and debugging](/docs/tasks/debug-application-cluster/debug-application-introspection/). +- Learn about [how to use `kubectl` for application introspection and debugging](/docs/tasks/debug/debug-application/debug-running-pod/). - See [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/). diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md index c517b13175..b780ef15ca 100644 --- a/content/en/docs/concepts/cluster-administration/networking.md +++ b/content/en/docs/concepts/cluster-administration/networking.md @@ -30,46 +30,7 @@ insert dynamic port numbers into configuration blocks, services have to know how to find each other, etc. Rather than deal with this, Kubernetes takes a different approach. -## The Kubernetes network model - -Every `Pod` gets its own IP address. This means you do not need to explicitly -create links between `Pods` and you almost never need to deal with mapping -container ports to host ports. This creates a clean, backwards-compatible -model where `Pods` can be treated much like VMs or physical hosts from the -perspectives of port allocation, naming, service discovery, load balancing, -application configuration, and migration. - -Kubernetes imposes the following fundamental requirements on any networking -implementation (barring any intentional network segmentation policies): - - * pods on a node can communicate with all pods on all nodes without NAT - * agents on a node (e.g. system daemons, kubelet) can communicate with all - pods on that node - -Note: For those platforms that support `Pods` running in the host network (e.g. -Linux): - - * pods in the host network of a node can communicate with all pods on all - nodes without NAT - -This model is not only less complex overall, but it is principally compatible -with the desire for Kubernetes to enable low-friction porting of apps from VMs -to containers. If your job previously ran in a VM, your VM had an IP and could -talk to other VMs in your project. This is the same basic model. - -Kubernetes IP addresses exist at the `Pod` scope - containers within a `Pod` -share their network namespaces - including their IP address and MAC address. -This means that containers within a `Pod` can all reach each other's ports on -`localhost`. This also means that containers within a `Pod` must coordinate port -usage, but this is no different from processes in a VM. This is called the -"IP-per-pod" model. - -How this is implemented is a detail of the particular container runtime in use. - -It is possible to request ports on the `Node` itself which forward to your `Pod` -(called host ports), but this is a very niche operation. How that forwarding is -implemented is also a detail of the container runtime. The `Pod` itself is -blind to the existence or non-existence of host ports. +To learn about the Kubernetes networking model, see [here](/docs/concepts/services-networking/). ## How to implement the Kubernetes networking model @@ -91,18 +52,6 @@ imply any preferential status. Project [Antrea](https://github.com/vmware-tanzu/antrea) is an opensource Kubernetes networking solution intended to be Kubernetes native. It leverages Open vSwitch as the networking data plane. Open vSwitch is a high-performance programmable virtual switch that supports both Linux and Windows. Open vSwitch enables Antrea to implement Kubernetes Network Policies in a high-performance and efficient manner. Thanks to the "programmable" characteristic of Open vSwitch, Antrea is able to implement an extensive set of networking and security features and services on top of Open vSwitch. -### AOS from Apstra - -[AOS](https://www.apstra.com/products/aos/) is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform. AOS leverages a highly scalable distributed design to eliminate network outages while minimizing costs. - -The AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems. These Layer-3 hosts can be Linux servers (Debian, Ubuntu, CentOS) that create BGP neighbor relationships directly with the top of rack switches (TORs). AOS automates the routing adjacencies and then provides fine grained control over the route health injections (RHI) that are common in a Kubernetes deployment. - -AOS has a rich set of REST API endpoints that enable Kubernetes to quickly change the network policy based on application requirements. Further enhancements will integrate the AOS Graph model used for the network design with the workload provisioning, enabling an end to end management system for both private and public clouds. - -AOS supports the use of common vendor equipment from manufacturers including Cisco, Arista, Dell, Mellanox, HPE, and a large number of white-box systems and open network operating systems like Microsoft SONiC, Dell OPX, and Cumulus Linux. - -Details on how the AOS system works can be accessed here: https://www.apstra.com/products/how-it-works/ - ### AWS VPC CNI for Kubernetes The [AWS VPC CNI](https://github.com/aws/amazon-vpc-cni-k8s) offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters. This CNI plugin offers high throughput and availability, low latency, and minimal network jitter. Additionally, users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters. This includes the ability to use VPC flow logs, VPC routing policies, and security groups for network traffic isolation. @@ -116,18 +65,9 @@ Additionally, the CNI can be run alongside [Calico for network policy enforcemen Azure CNI is available natively in the [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni). - -### Big Cloud Fabric from Big Switch Networks - -[Big Cloud Fabric](https://www.bigswitch.com/container-network-automation) is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments. Using unified physical & virtual SDN, Big Cloud Fabric tackles inherent container networking problems such as load balancing, visibility, troubleshooting, security policies & container traffic monitoring. - -With the help of the Big Cloud Fabric's virtual pod multi-tenant architecture, container orchestration systems such as Kubernetes, RedHat OpenShift, Mesosphere DC/OS & Docker Swarm will be natively integrated alongside with VM orchestration systems such as VMware, OpenStack & Nutanix. Customers will be able to securely inter-connect any number of these clusters and enable inter-tenant communication between them if needed. - -BCF was recognized by Gartner as a visionary in the latest [Magic Quadrant](https://go.bigswitch.com/17GatedDocuments-MagicQuadrantforDataCenterNetworking_Reg.html). One of the BCF Kubernetes on-premises deployments (which includes Kubernetes, DC/OS & VMware running on multiple DCs across different geographic regions) is also referenced [here](https://portworx.com/architects-corner-kubernetes-satya-komala-nio/). - ### Calico -[Calico](https://docs.projectcalico.org/) is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports multiple data planes including: a pure Linux eBPF dataplane, a standard Linux networking dataplane, and a Windows HNS dataplane. Calico provides a full networking stack but can also be used in conjunction with [cloud provider CNIs](https://docs.projectcalico.org/networking/determine-best-networking#calico-compatible-cni-plugins-and-cloud-provider-integrations) to provide network policy enforcement. +[Calico](https://projectcalico.docs.tigera.io/about/about-calico/) is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports multiple data planes including: a pure Linux eBPF dataplane, a standard Linux networking dataplane, and a Windows HNS dataplane. Calico provides a full networking stack but can also be used in conjunction with [cloud provider CNIs](https://projectcalico.docs.tigera.io/networking/determine-best-networking#calico-compatible-cni-plugins-and-cloud-provider-integrations) to provide network policy enforcement. ### Cilium @@ -139,9 +79,9 @@ addressing, and it can be used in combination with other CNI plugins. ### CNI-Genie from Huawei -[CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) is a CNI plugin that enables Kubernetes to [simultaneously have access to different implementations](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-cni-plugins/README.md#what-cni-genie-feature-1-multiple-cni-plugins-enables) of the [Kubernetes network model](/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) in runtime. This includes any implementation that runs as a [CNI plugin](https://github.com/containernetworking/cni#3rd-party-plugins), such as [Flannel](https://github.com/coreos/flannel#flannel), [Calico](https://docs.projectcalico.org/), [Romana](https://romana.io), [Weave-net](https://www.weave.works/products/weave-net/). +[CNI-Genie](https://github.com/cni-genie/CNI-Genie) is a CNI plugin that enables Kubernetes to [simultaneously have access to different implementations](https://github.com/cni-genie/CNI-Genie/blob/master/docs/multiple-cni-plugins/README.md#what-cni-genie-feature-1-multiple-cni-plugins-enables) of the [Kubernetes network model](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model) in runtime. This includes any implementation that runs as a [CNI plugin](https://github.com/containernetworking/cni#3rd-party-plugins), such as [Flannel](https://github.com/flannel-io/flannel#flannel), [Calico](https://projectcalico.docs.tigera.io/about/about-calico/), [Weave-net](https://www.weave.works/oss/net/). -CNI-Genie also supports [assigning multiple IP addresses to a pod](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-ips/README.md#feature-2-extension-cni-genie-multiple-ip-addresses-per-pod), each from a different CNI plugin. +CNI-Genie also supports [assigning multiple IP addresses to a pod](https://github.com/cni-genie/CNI-Genie/blob/master/docs/multiple-ips/README.md#feature-2-extension-cni-genie-multiple-ip-addresses-per-pod), each from a different CNI plugin. ### cni-ipvlan-vpc-k8s [cni-ipvlan-vpc-k8s](https://github.com/lyft/cni-ipvlan-vpc-k8s) contains a set @@ -164,9 +104,10 @@ network complexity required to deploy Kubernetes at scale within AWS. [Coil](https://github.com/cybozu-go/coil) is a CNI plugin designed for ease of integration, providing flexible egress networking. Coil operates with a low overhead compared to bare metal, and allows you to define arbitrary egress NAT gateways for external networks. -### Contiv +### Contiv-VPP -[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](https://contiv.io) is all open sourced. +[Contiv-VPP](https://contivpp.io/) is a user-space, performance-oriented network plugin for +Kubernetes, using the [fd.io](https://fd.io/) data plane. ### Contrail / Tungsten Fabric @@ -186,52 +127,13 @@ With this toolset DANM is able to provide multiple separated network interfaces, ### Flannel -[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay +[Flannel](https://github.com/flannel-io/flannel#flannel) is a very simple overlay network that satisfies the Kubernetes requirements. Many people have reported success with Flannel and Kubernetes. -### Google Compute Engine (GCE) +### Hybridnet -For the Google Compute Engine cluster configuration scripts, [advanced -routing](https://cloud.google.com/vpc/docs/routes) is used to -assign each VM a subnet (default is `/24` - 254 IPs). Any traffic bound for that -subnet will be routed directly to the VM by the GCE network fabric. This is in -addition to the "main" IP address assigned to the VM, which is NAT'ed for -outbound internet access. A linux bridge (called `cbr0`) is configured to exist -on that subnet, and is passed to docker's `--bridge` flag. - -Docker is started with: - -```shell -DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false" -``` - -This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet` -flag) according to the `Node`'s `.spec.podCIDR`. - -Docker will now allocate IPs from the `cbr-cidr` block. Containers can reach -each other and `Nodes` over the `cbr0` bridge. Those IPs are all routable -within the GCE project network. - -GCE itself does not know anything about these IPs, though, so it will not NAT -them for outbound internet traffic. To achieve that an iptables rule is used -to masquerade (aka SNAT - to make it seem as if packets came from the `Node` -itself) traffic that is bound for IPs outside the GCE project network -(10.0.0.0/8). - -```shell -iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE -``` - -Lastly IP forwarding is enabled in the kernel (so the kernel will process -packets for bridged containers): - -```shell -sysctl net.ipv4.ip_forward=1 -``` - -The result of all this is that all `Pods` can reach each other and can egress -traffic to the internet. +[Hybridnet](https://github.com/alibaba/hybridnet) is an open source CNI plugin designed for hybrid clouds which provides both overlay and underlay networking for containers in one or more clusters. Overlay and underlay containers can run on the same node and have cluster-wide bidirectional network connectivity. ### Jaguar @@ -267,9 +169,9 @@ Lars Kellogg-Stedman. ### Multus (a Multi Network plugin) -[Multus](https://github.com/Intel-Corp/multus-cni) is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes. +Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes. -Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes. +Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/cni.dev/blob/main/content/plugins/v0.9/meta/flannel.md), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes. ### OVN4NFV-K8s-Plugin (OVN based CNI controller & plugin) @@ -281,18 +183,6 @@ Multus supports all [reference plugins](https://github.com/containernetworking/p [NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift. -### Nuage Networks VCS (Virtualized Cloud Services) - -[Nuage](https://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards. - -The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications. - -### OpenVSwitch - -[OpenVSwitch](https://www.openvswitch.org/) is a somewhat more mature but also -complicated way to build an overlay network. This is endorsed by several of the -"Big Shops" for networking. - ### OVN (Open Virtual Networking) OVN is an opensource network virtualization solution developed by the @@ -301,13 +191,9 @@ stateful ACLs, load-balancers etc to build different virtual networking topologies. The project has a specific Kubernetes plugin and documentation at [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes). -### Romana - -[Romana](https://romana.io) is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network. Romana supports Kubernetes [Network Policy](/docs/concepts/services-networking/network-policies/) to provide isolation across network namespaces. - ### Weave Net from Weaveworks -[Weave Net](https://www.weave.works/products/weave-net/) is a +[Weave Net](https://www.weave.works/oss/net/) is a resilient and simple to use network for Kubernetes and its hosted applications. Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/) or stand-alone. In either version, it doesn't require any configuration or extra code @@ -317,4 +203,4 @@ to run, and in both cases, the network provides one IP address per pod - as is s The early design of the networking model and its rationale, and some future plans are described in more detail in the -[networking design document](https://git.k8s.io/community/contributors/design-proposals/network/networking.md). +[networking design document](https://git.k8s.io/design-proposals-archive/network/networking.md). diff --git a/content/en/docs/concepts/cluster-administration/system-logs.md b/content/en/docs/concepts/cluster-administration/system-logs.md index 0466837356..f468de32b7 100644 --- a/content/en/docs/concepts/cluster-administration/system-logs.md +++ b/content/en/docs/concepts/cluster-administration/system-logs.md @@ -22,14 +22,62 @@ generates log messages for the Kubernetes system components. For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/). -An example of the klog native format: +Kubernetes is in the process of simplifying logging in its components. The +following klog command line flags [are +deprecated](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +starting with Kubernetes 1.23 and will be removed in a future release: + +- `--add-dir-header` +- `--alsologtostderr` +- `--log-backtrace-at` +- `--log-dir` +- `--log-file` +- `--log-file-max-size` +- `--logtostderr` +- `--one-output` +- `--skip-headers` +- `--skip-log-headers` +- `--stderrthreshold` + +Output will always be written to stderr, regardless of the output +format. Output redirection is expected to be handled by the component which +invokes a Kubernetes component. This can be a POSIX shell or a tool like +systemd. + +In some cases, for example a distroless container or a Windows system service, +those options are not available. Then the +[`kube-log-runner`](https://github.com/kubernetes/kubernetes/blob/d2a8a81639fcff8d1221b900f66d28361a170654/staging/src/k8s.io/component-base/logs/kube-log-runner/README.md) +binary can be used as wrapper around a Kubernetes component to redirect +output. A prebuilt binary is included in several Kubernetes base images under +its traditional name as `/go-runner` and as `kube-log-runner` in server and +node release archives. + +This table shows how `kube-log-runner` invocations correspond to shell redirection: + +| Usage | POSIX shell (such as bash) | `kube-log-runner <options> <cmd>` | +| -----------------------------------------|----------------------------|-------------------------------------------------------------| +| Merge stderr and stdout, write to stdout | `2>&1` | `kube-log-runner` (default behavior) | +| Redirect both into log file | `1>>/tmp/log 2>&1` | `kube-log-runner -log-file=/tmp/log` | +| Copy into log file and to stdout | `2>&1 \| tee -a /tmp/log` | `kube-log-runner -log-file=/tmp/log -also-stdout` | +| Redirect only stdout into log file | `>/tmp/log` | `kube-log-runner -log-file=/tmp/log -redirect-stderr=false` | + +### Klog output + +An example of the traditional klog native format: ``` I1025 00:15:15.525108 1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756] ``` +The message string may contain line breaks: +``` +I1025 00:15:15.525108 1 example.go:79] This is a message +which has a line break. +``` + + ### Structured Logging -{{< feature-state for_k8s_version="v1.19" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} {{< warning >}} Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages. @@ -38,9 +86,11 @@ Log formatting and value serialization are subject to change. {{< /warning>}} Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost. -New message format is backward compatible and enabled by default. +The code which generates a log message determines whether it uses the traditional unstructured klog output +or structured logging. -Format of structured logs: +The default formatting of structured log messages is as text, with a format that +is backward compatible with traditional klog: ```ini <klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ... @@ -52,6 +102,62 @@ Example: I1025 00:15:15.525108 1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready" ``` +Strings are quoted. Other values are formatted with +[`%+v`](https://pkg.go.dev/fmt#hdr-Printing), which may cause log messages to +continue on the next line [depending on the data](https://github.com/kubernetes/kubernetes/issues/106428). +``` +I1025 00:15:15.525108 1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line, +second line.} +``` + +### Contextual Logging + +{{< feature-state for_k8s_version="v1.24" state="alpha" >}} + +Contextual logging builds on top of structured logging. It is primarily about +how developers use logging calls: code based on that concept is more flexible +and supports additional use cases as described in the [Contextual Logging +KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging). + +If developers use additional functions like `WithValues` or `WithName` in +their components, then log entries contain additional information that gets +passed into functions by their caller. + +Currently this is gated behind the `StructuredLogging` feature gate and +disabled by default. The infrastructure for this was added in 1.24 without +modifying components. The +[`component-base/logs/example`](https://github.com/kubernetes/kubernetes/blob/v1.24.0-beta.0/staging/src/k8s.io/component-base/logs/example/cmd/logger.go) +command demonstrates how to use the new logging calls and how a component +behaves that supports contextual logging. + +```console +$ cd $GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/ +$ go run . --help +... + --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: + AllAlpha=true|false (ALPHA - default=false) + AllBeta=true|false (BETA - default=false) + ContextualLogging=true|false (ALPHA - default=false) +$ go run . --feature-gates ContextualLogging=true +... +I0404 18:00:02.916429 451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s" +I0404 18:00:02.916447 451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s" +``` + +The `example` prefix and `foo="bar"` were added by the caller of the function +which logs the `runtime` message and `duration="1m0s"` value, without having to +modify that function. + +With contextual logging disable, `WithValues` and `WithName` do nothing and log +calls go through the global klog logger. Therefore this additional information +is not in the log output anymore: + +```console +$ go run . --feature-gates ContextualLogging=false +... +I0404 18:03:31.171945 452150 logger.go:94] "runtime" duration="1m0s" +I0404 18:03:31.171962 452150 logger.go:95] "another runtime" duration="1m0s" +``` ### JSON log format @@ -82,7 +188,7 @@ Example of JSON log format (pretty printed): Keys with special meaning: * `ts` - timestamp as Unix time (required, float) -* `v` - verbosity (required, int, default 0) +* `v` - verbosity (only for info and not for error messages, int) * `err` - error string (optional, string) * `msg` - message (required, string) @@ -93,27 +199,6 @@ List of components currently supporting JSON format: * {{< glossary_tooltip term_id="kube-scheduler" text="kube-scheduler" >}} * {{< glossary_tooltip term_id="kubelet" text="kubelet" >}} -### Log sanitization - -{{< feature-state for_k8s_version="v1.20" state="alpha" >}} - -{{<warning >}} -Log sanitization might incur significant computation overhead and therefore should not be enabled in production. -{{< /warning >}} - -The `--experimental-logging-sanitization` flag enables the klog sanitization filter. -If enabled all log arguments are inspected for fields tagged as sensitive data (e.g. passwords, keys, tokens) and logging of these fields will be prevented. - -List of components currently supporting log sanitization: -* kube-controller-manager -* kube-apiserver -* kube-scheduler -* kubelet - -{{< note >}} -The Log sanitization filter does not prevent user workload logs from leaking sensitive data. -{{< /note >}} - ### Log verbosity level The `-v` flag controls log verbosity. Increasing the value increases the number of logged events. Decreasing the value decreases the number of logged events. @@ -125,7 +210,8 @@ There are two types of system components: those that run in a container and thos that do not run in a container. For example: * The Kubernetes scheduler and kube-proxy run in a container. -* The kubelet and container runtime, for example Docker, do not run in containers. +* The kubelet and {{<glossary_tooltip term_id="container-runtime" text="container runtime">}} + do not run in containers. On machines with systemd, the kubelet and container runtime write to journald. Otherwise, they write to `.log` files in the `/var/log` directory. @@ -139,4 +225,6 @@ The `logrotate` tool rotates logs daily, or once the log size is greater than 10 * Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/) * Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging) +* Read about [Contextual Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging) +* Read about [deprecation of klog flags](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) * Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) diff --git a/content/en/docs/concepts/configuration/configmap.md b/content/en/docs/concepts/configuration/configmap.md index 47ecaedba6..4f5ace4348 100644 --- a/content/en/docs/concepts/configuration/configmap.md +++ b/content/en/docs/concepts/configuration/configmap.md @@ -15,7 +15,6 @@ If the data you want to store are confidential, use a or use additional (third party) tools to keep your data private. {{< /caution >}} - <!-- body --> ## Motivation @@ -42,7 +41,7 @@ that lets you store configuration for other objects to use. Unlike most Kubernetes objects that have a `spec`, a ConfigMap has `data` and `binaryData` fields. These fields accept key-value pairs as their values. Both the `data` field and the `binaryData` are optional. The `data` field is designed to -contain UTF-8 byte sequences while the `binaryData` field is designed to +contain UTF-8 strings while the `binaryData` field is designed to contain binary data as base64-encoded strings. The name of a ConfigMap must be a valid @@ -239,6 +238,10 @@ propagation delay, where the cache propagation delay depends on the chosen cache ConfigMaps consumed as environment variables are not updated automatically and require a pod restart. +{{< note >}} +A container using a ConfigMap as a [subPath](/docs/concepts/storage/volumes#using-subpath) volume mount will not receive ConfigMap updates. +{{< /note >}} + ## Immutable ConfigMaps {#configmap-immutable} {{< feature-state for_k8s_version="v1.21" state="stable" >}} @@ -276,7 +279,6 @@ to the deleted ConfigMap, it is recommended to recreate these pods. * Read about [Secrets](/docs/concepts/configuration/secret/). * Read [Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/). +* Read about [changing a ConfigMap (or any other Kubernetes object)](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/) * Read [The Twelve-Factor App](https://12factor.net/) to understand the motivation for separating code from configuration. - - diff --git a/content/en/docs/concepts/configuration/manage-resources-containers.md b/content/en/docs/concepts/configuration/manage-resources-containers.md index cbcfff67ef..9428da09e3 100644 --- a/content/en/docs/concepts/configuration/manage-resources-containers.md +++ b/content/en/docs/concepts/configuration/manage-resources-containers.md @@ -1,22 +1,24 @@ --- -title: Managing Resources for Containers +title: Resource Management for Pods and Containers content_type: concept weight: 40 feature: title: Automatic bin packing description: > - Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. Mix critical and best-effort workloads in order to drive up utilization and save even more resources. + Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. + Mix critical and best-effort workloads in order to drive up utilization and save even more resources. --- <!-- overview --> When you specify a {{< glossary_tooltip term_id="pod" >}}, you can optionally specify how -much of each resource a {{< glossary_tooltip text="Container" term_id="container" >}} needs. +much of each resource a {{< glossary_tooltip text="container" term_id="container" >}} needs. The most common resources to specify are CPU and memory (RAM); there are others. -When you specify the resource _request_ for Containers in a Pod, the scheduler uses this +When you specify the resource _request_ for containers in a Pod, the +{{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}} uses this information to decide which node to place the Pod on. When you specify a resource _limit_ -for a Container, the kubelet enforces those limits so that the running container is not +for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. The kubelet also reserves at least the _request_ amount of that system resource specifically for that container to use. @@ -33,7 +35,7 @@ For example, if you set a `memory` request of 256 MiB for a container, and that a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use more RAM. -If you set a `memory` limit of 4GiB for that Container, the kubelet (and +If you set a `memory` limit of 4GiB for that container, the kubelet (and {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}) enforce the limit. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, @@ -45,10 +47,9 @@ or by enforcement (the system prevents the container from ever exceeding the lim runtimes can have different ways to implement the same restrictions. {{< note >}} -If a Container specifies its own memory limit, but does not specify a memory request, Kubernetes -automatically assigns a memory request that matches the limit. Similarly, if a Container specifies its own -CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches -the limit. +If you specify a limit for a resource, but do not specify any request, and no admission-time +mechanism has applied a default request for that resource, then Kubernetes copies the limit +you specified and uses it as the requested value for the resource. {{< /note >}} ## Resource types @@ -56,7 +57,7 @@ the limit. *CPU* and *memory* are each a *resource type*. A resource type has a base unit. CPU represents compute processing and is specified in units of [Kubernetes CPUs](#meaning-of-cpu). Memory is specified in units of bytes. -If you're using Kubernetes v1.14 or newer, you can specify _huge page_ resources. +For Linux workloads, you can specify _huge page_ resources. Huge pages are a Linux-specific feature where the node kernel allocates blocks of memory that are much larger than the default page size. @@ -76,9 +77,10 @@ consumed. They are distinct from [Services](/docs/concepts/services-networking/service/) are objects that can be read and modified through the Kubernetes API server. -## Resource requests and limits of Pod and Container +## Resource requests and limits of Pod and container -Each Container of a Pod can specify one or more of the following: +For each container, you can specify resource limits and requests, +including the following: * `spec.containers[].resources.limits.cpu` * `spec.containers[].resources.limits.memory` @@ -87,48 +89,64 @@ Each Container of a Pod can specify one or more of the following: * `spec.containers[].resources.requests.memory` * `spec.containers[].resources.requests.hugepages-<size>` -Although requests and limits can only be specified on individual Containers, it -is convenient to talk about Pod resource requests and limits. A -*Pod resource request/limit* for a particular resource type is the sum of the -resource requests/limits of that type for each Container in the Pod. +Although you can only specify requests and limits for individual containers, +it is also useful to think about the overall resource requests and limits for +a Pod. +For a particular resource, a *Pod resource request/limit* is the sum of the +resource requests/limits of that type for each container in the Pod. ## Resource units in Kubernetes -### Meaning of CPU +### CPU resource units {#meaning-of-cpu} Limits and requests for CPU resources are measured in *cpu* units. -One cpu, in Kubernetes, is equivalent to **1 vCPU/Core** for cloud providers and **1 hyperthread** on bare-metal Intel processors. +In Kubernetes, 1 CPU unit is equivalent to **1 physical CPU core**, +or **1 virtual core**, depending on whether the node is a physical host +or a virtual machine running inside a physical machine. -Fractional requests are allowed. A Container with -`spec.containers[].resources.requests.cpu` of `0.5` is guaranteed half as much -CPU as one that asks for 1 CPU. The expression `0.1` is equivalent to the +Fractional requests are allowed. When you define a container with +`spec.containers[].resources.requests.cpu` set to `0.5`, you are requesting half +as much CPU time compared to if you asked for `1.0` CPU. +For CPU resource units, the [quantity](/docs/reference/kubernetes-api/common-definitions/quantity/) expression `0.1` is equivalent to the expression `100m`, which can be read as "one hundred millicpu". Some people say -"one hundred millicores", and this is understood to mean the same thing. A -request with a decimal point, like `0.1`, is converted to `100m` by the API, and -precision finer than `1m` is not allowed. For this reason, the form `100m` might -be preferred. +"one hundred millicores", and this is understood to mean the same thing. -CPU is always requested as an absolute quantity, never as a relative quantity; -0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine. +CPU resource is always specified as an absolute amount of resource, never as a relative amount. For example, +`500m` CPU represents the roughly same amount of computing power whether that container +runs on a single-core, dual-core, or 48-core machine. -### Meaning of memory +{{< note >}} +Kubernetes doesn't allow you to specify CPU resources with a precision finer than +`1m`. Because of this, it's useful to specify CPU units less than `1.0` or `1000m` using +the milliCPU form; for example, `5m` rather than `0.005`. +{{< /note >}} + +### Memory resource units {#meaning-of-memory} Limits and requests for `memory` are measured in bytes. You can express memory as -a plain integer or as a fixed-point number using one of these suffixes: +a plain integer or as a fixed-point number using one of these +[quantity](/docs/reference/kubernetes-api/common-definitions/quantity/) suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following represent roughly the same value: ```shell -128974848, 129e6, 129M, 123Mi +128974848, 129e6, 129M, 128974848000m, 123Mi ``` -Here's an example. -The following Pod has two Containers. Each Container has a request of 0.25 cpu -and 64MiB (2<sup>26</sup> bytes) of memory. Each Container has a limit of 0.5 -cpu and 128MiB of memory. You can say the Pod has a request of 0.5 cpu and 128 -MiB of memory, and a limit of 1 cpu and 256MiB of memory. +Pay attention to the case of the suffixes. If you request `400m` of memory, this is a request +for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (`400Mi`) +or 400 megabytes (`400M`). + +## Container resources example {#example-1} + +The following Pod has two containers. Both containers are defined with a request for +0.25 CPU +and 64MiB (2<sup>26</sup> bytes) of memory. Each container has a limit of 0.5 +CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128 +MiB of memory, and a limit of 1 CPU and 256MiB of memory. ```yaml +--- apiVersion: v1 kind: Pod metadata: @@ -161,60 +179,58 @@ When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled -Containers is less than the capacity of the node. Note that although actual memory +containers is less than the capacity of the node. +Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails. This protects against a resource shortage on a node when resource usage later increases, for example, during a daily peak in request rate. -## How Pods with resource limits are run +## How Kubernetes applies resource requests and limits {#how-pods-with-resource-limits-are-run} -When the kubelet starts a Container of a Pod, it passes the CPU and memory limits -to the container runtime. +When the kubelet starts a container as part of a Pod, the kubelet passes that container's +requests and limits for memory and CPU to the container runtime. -When using Docker: +On Linux, the container runtime typically configures +kernel {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} that apply and enforce the +limits you defined. -- The `spec.containers[].resources.requests.cpu` is converted to its core value, - which is potentially fractional, and multiplied by 1024. The greater of this number - or 2 is used as the value of the - [`--cpu-shares`](https://docs.docker.com/engine/reference/run/#cpu-share-constraint) - flag in the `docker run` command. +- The CPU limit defines a hard ceiling on how much CPU time that the container can use. + During each scheduling interval (time slice), the Linux kernel checks to see if this + limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution. +- The CPU request typically defines a weighting. If several different containers (cgroups) + want to run on a contended system, workloads with larger CPU requests are allocated more + CPU time than workloads with small requests. +- The memory request is mainly used during (Kubernetes) Pod scheduling. On a node that uses + cgroups v2, the container runtime might use the memory request as a hint to set + `memory.min` and `memory.low`. +- The memory limit defines a memory limit for that cgroup. If the container tries to + allocate more memory than this limit, the Linux kernel out-of-memory subsystem activates + and, typically, intervenes by stopping one of the processes in the container that tried + to allocate memory. If that process is the container's PID 1, and the container is marked + as restartable, Kubernetes restarts the container. +- The memory limit for the Pod or container can also apply to pages in memory backed + volumes, such as an `emptyDir`. The kubelet tracks `tmpfs` emptyDir volumes as container + memory use, rather than as local ephemeral storage. -- The `spec.containers[].resources.limits.cpu` is converted to its millicore value and - multiplied by 100. The resulting value is the total amount of CPU time in microseconds - that a container can use every 100ms. A container cannot use more than its share of - CPU time during this interval. +If a container exceeds its memory request and the node that it runs on becomes short of +memory overall, it is likely that the Pod the container belongs to will be +{{< glossary_tooltip text="evicted" term_id="eviction" >}}. - {{< note >}} - The default quota period is 100ms. The minimum resolution of CPU quota is 1ms. - {{</ note >}} +A container might or might not be allowed to exceed its CPU limit for extended periods of time. +However, container runtimes don't terminate Pods or containers for excessive CPU usage. -- The `spec.containers[].resources.limits.memory` is converted to an integer, and - used as the value of the - [`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints) - flag in the `docker run` command. - -If a Container exceeds its memory limit, it might be terminated. If it is -restartable, the kubelet will restart it, as with any other type of runtime -failure. - -If a Container exceeds its memory request, it is likely that its Pod will -be evicted whenever the node runs out of memory. - -A Container might or might not be allowed to exceed its CPU limit for extended -periods of time. However, it will not be killed for excessive CPU usage. - -To determine whether a Container cannot be scheduled or is being killed due to -resource limits, see the -[Troubleshooting](#troubleshooting) section. +To determine whether a container cannot be scheduled or is being killed due to resource limits, +see the [Troubleshooting](#troubleshooting) section. ### Monitoring compute & memory resource usage -The resource usage of a Pod is reported as part of the Pod status. +The kubelet reports the resource usage of a Pod as part of the Pod +[`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status). -If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) +If optional [tools for monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) are available in your cluster, then Pod resource usage can be retrieved either -from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api) +from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api) directly or from your monitoring tools. ## Local ephemeral storage @@ -236,7 +252,7 @@ The kubelet also uses this kind of storage to hold container images, and the writable layers of running containers. {{< caution >}} -If a node fails, the data in its ephemeral storage can be lost. +If a node fails, the data in its ephemeral storage can be lost. Your applications cannot expect any performance SLAs (disk IOPS for example) from local ephemeral storage. {{< /caution >}} @@ -308,21 +324,30 @@ than as local ephemeral storage. ### Setting requests and limits for local ephemeral storage -You can use _ephemeral-storage_ for managing local ephemeral storage. Each Container of a Pod can specify one or more of the following: +You can specify `ephemeral-storage` for managing local ephemeral storage. Each +container of a Pod can specify either or both of the following: * `spec.containers[].resources.limits.ephemeral-storage` * `spec.containers[].resources.requests.ephemeral-storage` -Limits and requests for `ephemeral-storage` are measured in bytes. You can express storage as -a plain integer or as a fixed-point number using one of these suffixes: -E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, -Mi, Ki. For example, the following represent roughly the same value: +Limits and requests for `ephemeral-storage` are measured in byte quantities. +You can express storage as a plain integer or as a fixed-point number using one of these suffixes: +E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, +Mi, Ki. For example, the following quantities all represent roughly the same value: -```shell -128974848, 129e6, 129M, 123Mi -``` +- `128974848` +- `129e6` +- `129M` +- `123Mi` -In the following example, the Pod has two Containers. Each Container has a request of 2GiB of local ephemeral storage. Each Container has a limit of 4GiB of local ephemeral storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. +Pay attention to the case of the suffixes. If you request `400m` of ephemeral-storage, this is a request +for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (`400Mi`) +or 400 megabytes (`400M`). + +In the following example, the Pod has two containers. Each container has a request of +2GiB of local ephemeral storage. Each container has a limit of 4GiB of local ephemeral +storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and +a limit of 8GiB of local ephemeral storage. ```yaml apiVersion: v1 @@ -359,9 +384,11 @@ spec: ### How Pods with ephemeral-storage requests are scheduled When you create a Pod, the Kubernetes scheduler selects a node for the Pod to -run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods. For more information, see [Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). +run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods. +For more information, see +[Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). -The scheduler ensures that the sum of the resource requests of the scheduled Containers is less than the capacity of the node. +The scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. ### Ephemeral storage consumption management {#resource-emphemeralstorage-consumption} @@ -375,7 +402,7 @@ kubelet measures storage use in: If a Pod is using more ephemeral storage than you allow it to, the kubelet sets an eviction signal that triggers Pod eviction. -For container-level isolation, if a Container's writable layer and log +For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction. For pod-level isolation the kubelet works out an overall Pod storage limit by @@ -440,7 +467,7 @@ Kubernetes does not use them. Quotas are faster and more accurate than directory scanning. When a directory is assigned to a project, all files created under a directory are created in that project, and the kernel merely has to -keep track of how many blocks are in use by files in that project. +keep track of how many blocks are in use by files in that project. If a file is created and deleted, but has an open file descriptor, it continues to consume space. Quota tracking records that space accurately whereas directory scans overlook the storage used by deleted files. @@ -492,15 +519,19 @@ Plugin](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for how to advertise device plugin managed resources on each node. ##### Other resources + To advertise a new node-level extended resource, the cluster operator can submit a `PATCH` HTTP request to the API server to specify the available quantity in the `status.capacity` for a node in the cluster. After this operation, the node's `status.capacity` will include a new resource. The `status.allocatable` field is updated automatically with the new resource -asynchronously by the kubelet. Note that because the scheduler uses the node -`status.allocatable` value when evaluating Pod fitness, there may be a short -delay between patching the node capacity with a new resource and the first Pod -that requests the resource to be scheduled on that node. +asynchronously by the kubelet. + +Because the scheduler uses the node's `status.allocatable` value when +evaluating Pod fitness, the scheduler only takes account of the new value after +that asynchronous update. There may be a short delay between patching the +node capacity with a new resource and the time when the first Pod that requests +the resource can be scheduled on that node. **Example:** @@ -528,7 +559,7 @@ Cluster-level extended resources are not tied to nodes. They are usually managed by scheduler extenders, which handle the resource consumption and resource quota. You can specify the extended resources that are handled by scheduler extenders -in [scheduler policy configuration](/docs/reference/config-api/kube-scheduler-policy-config.v1/) +in [scheduler configuration](/docs/reference/config-api/kube-scheduler-config.v1beta3/) **Example:** @@ -610,27 +641,32 @@ spec: ## PID limiting -Process ID (PID) limits allow for the configuration of a kubelet to limit the number of PIDs that a given Pod can consume. See [Pid Limiting](/docs/concepts/policy/pid-limiting/) for information. +Process ID (PID) limits allow for the configuration of a kubelet +to limit the number of PIDs that a given Pod can consume. See +[PID Limiting](/docs/concepts/policy/pid-limiting/) for information. ## Troubleshooting -### My Pods are pending with event message failedScheduling +### My Pods are pending with event message `FailedScheduling` If the scheduler cannot find any node where a Pod can fit, the Pod remains -unscheduled until a place can be found. An event is produced each time the -scheduler fails to find a place for the Pod, like this: +unscheduled until a place can be found. An +[Event](/docs/reference/kubernetes-api/cluster-resources/event-v1/) is produced +each time the scheduler fails to find a place for the Pod. You can use `kubectl` +to view the events for a Pod; for example: ```shell -kubectl describe pod frontend | grep -A 3 Events +kubectl describe pod frontend | grep -A 9999999999 Events ``` ``` Events: - FirstSeen LastSeen Count From Subobject PathReason Message - 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 23s default-scheduler 0/42 nodes available: insufficient cpu ``` In the preceding example, the Pod named "frontend" fails to be scheduled due to -insufficient CPU resource on the node. Similar error messages can also suggest +insufficient CPU resource on any node. Similar error messages can also suggest failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod is pending with a message of this type, there are several things to try: @@ -639,6 +675,9 @@ is pending with a message of this type, there are several things to try: - Check that the Pod is not larger than all the nodes. For example, if all the nodes have a capacity of `cpu: 1`, then a Pod with a request of `cpu: 1.1` will never be scheduled. +- Check for node taints. If most of your nodes are tainted, and the new Pod does + not tolerate that taint, the scheduler only considers placements onto the + remaining nodes that don't have that taint. You can check node capacities and amounts allocated with the `kubectl describe nodes` command. For example: @@ -673,31 +712,46 @@ Allocated resources: 680m (34%) 400m (20%) 920Mi (11%) 1070Mi (13%) ``` -In the preceding output, you can see that if a Pod requests more than 1120m -CPUs or 6.23Gi of memory, it will not fit on the node. +In the preceding output, you can see that if a Pod requests more than 1.120 CPUs +or more than 6.23Gi of memory, that Pod will not fit on the node. -By looking at the `Pods` section, you can see which Pods are taking up space on +By looking at the “Pods” section, you can see which Pods are taking up space on the node. -The amount of resources available to Pods is less than the node capacity, because -system daemons use a portion of the available resources. The `allocatable` field -[NodeStatus](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#nodestatus-v1-core) -gives the amount of resources that are available to Pods. For more information, see -[Node Allocatable Resources](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md). +The amount of resources available to Pods is less than the node capacity because +system daemons use a portion of the available resources. Within the Kubernetes API, +each Node has a `.status.allocatable` field +(see [NodeStatus](/docs/reference/kubernetes-api/cluster-resources/node-v1/#NodeStatus) +for details). -The [resource quota](/docs/concepts/policy/resource-quotas/) feature can be configured -to limit the total amount of resources that can be consumed. If used in conjunction -with namespaces, it can prevent one team from hogging all the resources. +The `.status.allocatable` field describes the amount of resources that are available +to Pods on that node (for example: 15 virtual CPUs and 7538 MiB of memory). +For more information on node allocatable resources in Kubernetes, see +[Reserve Compute Resources for System Daemons](/docs/tasks/administer-cluster/reserve-compute-resources/). -### My Container is terminated +You can configure [resource quotas](/docs/concepts/policy/resource-quotas/) +to limit the total amount of resources that a namespace can consume. +Kubernetes enforces quotas for objects in particular namespace when there is a +ResourceQuota in that namespace. +For example, if you assign specific namespaces to different teams, you +can add ResourceQuotas into those namespaces. Setting resource quotas helps to +prevent one team from using so much of any resource that this over-use affects other teams. -Your Container might get terminated because it is resource-starved. To check -whether a Container is being killed because it is hitting a resource limit, call +You should also consider what access you grant to that namespace: +**full** write access to a namespace allows someone with that access to remove any +resource, including a configured ResourceQuota. + +### My container is terminated + +Your container might get terminated because it is resource-starved. To check +whether a container is being killed because it is hitting a resource limit, call `kubectl describe pod` on the Pod of interest: ```shell kubectl describe pod simmemleak-hra99 ``` + +The output is similar to: ``` Name: simmemleak-hra99 Namespace: default @@ -708,57 +762,48 @@ Status: Running Reason: Message: IP: 10.244.2.75 -Replication Controllers: simmemleak (1/1 replicas created) Containers: simmemleak: - Image: saadali/simmemleak + Image: saadali/simmemleak:latest Limits: - cpu: 100m - memory: 50Mi - State: Running - Started: Tue, 07 Jul 2015 12:54:41 -0700 - Last Termination State: Terminated - Exit Code: 1 - Started: Fri, 07 Jul 2015 12:54:30 -0700 - Finished: Fri, 07 Jul 2015 12:54:33 -0700 - Ready: False - Restart Count: 5 + cpu: 100m + memory: 50Mi + State: Running + Started: Tue, 07 Jul 2019 12:54:41 -0700 + Last State: Terminated + Reason: OOMKilled + Exit Code: 137 + Started: Fri, 07 Jul 2019 12:54:30 -0700 + Finished: Fri, 07 Jul 2019 12:54:33 -0700 + Ready: False + Restart Count: 5 Conditions: Type Status Ready False Events: - FirstSeen LastSeen Count From SubobjectPath Reason Message - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "k8s.gcr.io/pause:0.8.0" already present on machine - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 42s default-scheduler Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f + Normal Pulled 41s kubelet Container image "saadali/simmemleak:latest" already present on machine + Normal Created 41s kubelet Created container simmemleak + Normal Started 40s kubelet Started container simmemleak + Normal Killing 32s kubelet Killing container with id ead3fb35-5cf5-44ed-9ae1-488115be66c6: Need to kill Pod ``` In the preceding example, the `Restart Count: 5` indicates that the `simmemleak` -Container in the Pod was terminated and restarted five times. +container in the Pod was terminated and restarted five times (so far). +The `OOMKilled` reason shows that the container tried to use more memory than its limit. -You can call `kubectl get pod` with the `-o go-template=...` option to fetch the status -of previously terminated Containers: - -```shell -kubectl get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-hra99 -``` -``` -Container Name: simmemleak -LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]] -``` - -You can see that the Container was terminated because of `reason:OOM Killed`, where `OOM` stands for Out Of Memory. +Your next step might be to check the application code for a memory leak. If you +find that the application is behaving how you expect, consider setting a higher +memory limit (and possibly request) for that container. ## {{% heading "whatsnext" %}} -* Get hands-on experience [assigning Memory resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/). -* Get hands-on experience [assigning CPU resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/). -* For more details about the difference between requests and limits, see - [Resource QoS](https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md). -* Read the [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) API reference -* Read the [ResourceRequirements](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcerequirements-v1-core) API reference -* Read about [project quotas](https://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs-quotas.html) in XFS -* Read more about the [kube-scheduler Policy reference (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) +* Get hands-on experience [assigning Memory resources to containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/). +* Get hands-on experience [assigning CPU resources to containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/). +* Read how the API reference defines a [container](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container) + and its [resource requirements](/docs/reference/kubernetes-api/workload-resources/pod-v1/#resources) +* Read about [project quotas](https://xfs.org/index.php/XFS_FAQ#Q:_Quota:_Do_quotas_work_on_XFS.3F) in XFS +* Read more about the [kube-scheduler configuration reference (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) diff --git a/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md b/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md index b27fcdee61..87c4737ad7 100644 --- a/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md +++ b/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md @@ -18,7 +18,7 @@ It does not mean that there is a file named `kubeconfig`. {{< /note >}} {{< warning >}} -Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. +Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script. {{< /warning>}} @@ -53,7 +53,7 @@ clusters and namespaces. A *context* element in a kubeconfig file is used to group access parameters under a convenient name. Each context has three parameters: cluster, namespace, and user. By default, the `kubectl` command-line tool uses parameters from -the *current context* to communicate with the cluster. +the *current context* to communicate with the cluster. To choose the current context: ``` @@ -148,7 +148,27 @@ File references on the command line are relative to the current working director In `$HOME/.kube/config`, relative paths are stored relatively, and absolute paths are stored absolutely. +## Proxy +You can configure `kubectl` to use a proxy per cluster using `proxy-url` in your kubeconfig file, like this: + +```yaml +apiVersion: v1 +kind: Config + +clusters: +- cluster: + proxy-url: http://proxy.example.org:3128 + server: https://k8s.example.org/k8s/clusters/c-xxyyzz + name: development + +users: +- name: developer + +contexts: +- context: + name: development +``` ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/concepts/configuration/overview.md b/content/en/docs/concepts/configuration/overview.md index 36eebe3abc..5dc5a64826 100644 --- a/content/en/docs/concepts/configuration/overview.md +++ b/content/en/docs/concepts/configuration/overview.md @@ -55,7 +55,7 @@ DNS server watches the Kubernetes API for new `Services` and creates a set of DN If you only need access to the port for debugging purposes, you can use the [apiserver proxy](/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls) or [`kubectl port-forward`](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/). - If you explicitly need to expose a Pod's port on the node, consider using a [NodePort](/docs/concepts/services-networking/service/#nodeport) Service before resorting to `hostPort`. + If you explicitly need to expose a Pod's port on the node, consider using a [NodePort](/docs/concepts/services-networking/service/#type-nodeport) Service before resorting to `hostPort`. - Avoid using `hostNetwork`, for the same reasons as `hostPort`. @@ -63,7 +63,7 @@ DNS server watches the Kubernetes API for new `Services` and creates a set of DN ## Using Labels -- Define and use [labels](/docs/concepts/overview/working-with-objects/labels/) that identify __semantic attributes__ of your application or Deployment, such as `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`. You can use these labels to select the appropriate Pods for other resources; for example, a Service that selects all `tier: frontend` Pods, or all `phase: test` components of `app: myapp`. See the [guestbook](https://github.com/kubernetes/examples/tree/master/guestbook/) app for examples of this approach. +- Define and use [labels](/docs/concepts/overview/working-with-objects/labels/) that identify __semantic attributes__ of your application or Deployment, such as `{ app.kubernetes.io/name: MyApp, tier: frontend, phase: test, deployment: v3 }`. You can use these labels to select the appropriate Pods for other resources; for example, a Service that selects all `tier: frontend` Pods, or all `phase: test` components of `app.kubernetes.io/name: MyApp`. See the [guestbook](https://github.com/kubernetes/examples/tree/master/guestbook/) app for examples of this approach. A Service can be made to span multiple Deployments by omitting release-specific labels from its selector. When you need to update a running service without downtime, use a [Deployment](/docs/concepts/workloads/controllers/deployment/). diff --git a/content/en/docs/concepts/configuration/secret.md b/content/en/docs/concepts/configuration/secret.md index e1c7dfb79b..f83372532f 100644 --- a/content/en/docs/concepts/configuration/secret.md +++ b/content/en/docs/concepts/configuration/secret.md @@ -6,7 +6,8 @@ content_type: concept feature: title: Secret and configuration management description: > - Deploy and update secrets and application configuration without rebuilding your image and without exposing secrets in your stack configuration. + Deploy and update secrets and application configuration without rebuilding your image + and without exposing secrets in your stack configuration. weight: 30 --- @@ -22,8 +23,8 @@ application code. Because Secrets can be created independently of the Pods that use them, there is less risk of the Secret (and its data) being exposed during the workflow of creating, viewing, and editing Pods. Kubernetes, and applications that run in -your cluster, can also take additional precautions with Secrets, such as -avoiding writing confidential data to nonvolatile storage. +your cluster, can also take additional precautions with Secrets, such as avoiding +writing secret data to nonvolatile storage. Secrets are similar to {{< glossary_tooltip text="ConfigMaps" term_id="configmap" >}} but are specifically intended to hold confidential data. @@ -35,19 +36,21 @@ Additionally, anyone who is authorized to create a Pod in a namespace can use th In order to safely use Secrets, take at least the following steps: 1. [Enable Encryption at Rest](/docs/tasks/administer-cluster/encrypt-data/) for Secrets. -2. Enable or configure [RBAC rules](/docs/reference/access-authn-authz/authorization/) that - restrict reading data in Secrets (including via indirect means). -3. Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones. +1. [Enable or configure RBAC rules](/docs/reference/access-authn-authz/authorization/) that + restrict reading and writing the Secret. Be aware that secrets can be obtained + implicitly by anyone with the permission to create a Pod. +1. Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed + to create new Secrets or replace existing ones. {{< /caution >}} +See [Information security for Secrets](#information-security-for-secrets) for more details. + <!-- body --> -## Overview of Secrets - -To use a Secret, a Pod needs to reference the Secret. -A Secret can be used with a Pod in three ways: +## Uses for Secrets +There are three main ways for a Pod to use a Secret: - As [files](#using-secrets-as-files-from-a-pod) in a {{< glossary_tooltip text="volume" term_id="volume" >}} mounted on one or more of its containers. @@ -58,8 +61,50 @@ The Kubernetes control plane also uses Secrets; for example, [bootstrap token Secrets](#bootstrap-token-secrets) are a mechanism to help automate node registration. +### Alternatives to Secrets + +Rather than using a Secret to protect confidential data, you can pick from alternatives. + +Here are some of your options: + +- if your cloud-native component needs to authenticate to another application that you + know is running within the same Kubernetes cluster, you can use a + [ServiceAccount](/docs/reference/access-authn-authz/authentication/#service-account-tokens) + and its tokens to identify your client. +- there are third-party tools that you can run, either within or outside your cluster, + that provide secrets management. For example, a service that Pods access over HTTPS, + that reveals a secret if the client correctly authenticates (for example, with a ServiceAccount + token). +- for authentication, you can implement a custom signer for X.509 certificates, and use + [CertificateSigningRequests](/docs/reference/access-authn-authz/certificate-signing-requests/) + to let that custom signer issue certificates to Pods that need them. +- you can use a [device plugin](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) + to expose node-local encryption hardware to a specific Pod. For example, you can schedule + trusted Pods onto nodes that provide a Trusted Platform Module, configured out-of-band. + +You can also combine two or more of those options, including the option to use Secret objects themselves. + +For example: implement (or deploy) an {{< glossary_tooltip text="operator" term_id="operator-pattern" >}} +that fetches short-lived session tokens from an external service, and then creates Secrets based +on those short-lived session tokens. Pods running in your cluster can make use of the session tokens, +and operator ensures they are valid. This separation means that you can run Pods that are unaware of +the exact mechanisms for issuing and refreshing those session tokens. + +## Working with Secrets + +### Creating a Secret + +There are several options to create a Secret: + +- [create Secret using `kubectl` command](/docs/tasks/configmap-secret/managing-secret-using-kubectl/) +- [create Secret from config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/) +- [create Secret using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/) + +#### Constraints on Secret names and data {#restriction-names-data} + The name of a Secret object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). + You can specify the `data` and/or the `stringData` field when creating a configuration file for a Secret. The `data` and the `stringData` fields are optional. The values for all keys in the `data` field have to be base64-encoded strings. @@ -72,21 +117,734 @@ merged into the `data` field. If a key appears in both the `data` and the `stringData` field, the value specified in the `stringData` field takes precedence. +#### Size limit {#restriction-data-size} + +Individual secrets are limited to 1MiB in size. This is to discourage creation +of very large secrets that could exhaust the API server and kubelet memory. +However, creation of many smaller secrets could also exhaust memory. You can +use a [resource quota](/docs/concepts/policy/resource-quotas/) to limit the +number of Secrets (or other resources) in a namespace. + +### Editing a Secret + +You can edit an existing Secret using kubectl: + +```shell +kubectl edit secrets mysecret +``` + +This opens your default editor and allows you to update the base64 encoded Secret +values in the `data` field; for example: + +```yaml +# Please edit the object below. Lines beginning with a '#' will be ignored, +# and an empty file will abort the edit. If an error occurs while saving this file, it will be +# reopened with the relevant failures. +# +apiVersion: v1 +data: + username: YWRtaW4= + password: MWYyZDFlMmU2N2Rm +kind: Secret +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: { ... } + creationTimestamp: 2020-01-22T18:41:56Z + name: mysecret + namespace: default + resourceVersion: "164619" + uid: cfee02d6-c137-11e5-8d73-42010af00002 +type: Opaque +``` + +That example manifest defines a Secret with two keys in the `data` field: `username` and `password`. +The values are Base64 strings in the manifest; however, when you use the Secret with a Pod +then the kubelet provides the _decoded_ data to the Pod and its containers. + +You can package many keys and values into one Secret, or use many Secrets, whichever is convenient. + +### Using a Secret + +Secrets can be mounted as data volumes or exposed as +{{< glossary_tooltip text="environment variables" term_id="container-env-variables" >}} +to be used by a container in a Pod. Secrets can also be used by other parts of the +system, without being directly exposed to the Pod. For example, Secrets can hold +credentials that other parts of the system should use to interact with external +systems on your behalf. + +Secret volume sources are validated to ensure that the specified object +reference actually points to an object of type Secret. Therefore, a Secret +needs to be created before any Pods that depend on it. + +If the Secret cannot be fetched (perhaps because it does not exist, or +due to a temporary lack of connection to the API server) the kubelet +periodically retries running that Pod. The kubelet also reports an Event +for that Pod, including details of the problem fetching the Secret. + +#### Optional Secrets {#restriction-secret-must-exist} + +When you define a container environment variable based on a Secret, +you can mark it as _optional_. The default is for the Secret to be +required. + +None of a Pod's containers will start until all non-optional Secrets are +available. + +If a Pod references a specific key in a Secret and that Secret does exist, but +is missing the named key, the Pod fails during startup. + +### Using Secrets as files from a Pod {#using-secrets-as-files-from-a-pod} + +If you want to access data from a Secret in a Pod, one way to do that is to +have Kubernetes make the value of that Secret be available as a file inside +the filesystem of one or more of the Pod's containers. + +To configure that, you: + +1. Create a secret or use an existing one. Multiple Pods can reference the same secret. +1. Modify your Pod definition to add a volume under `.spec.volumes[]`. Name the volume anything, and have a `.spec.volumes[].secret.secretName` field equal to the name of the Secret object. +1. Add a `.spec.containers[].volumeMounts[]` to each container that needs the secret. Specify `.spec.containers[].volumeMounts[].readOnly = true` and `.spec.containers[].volumeMounts[].mountPath` to an unused directory name where you would like the secrets to appear. +1. Modify your image or command line so that the program looks for files in that directory. Each key in the secret `data` map becomes the filename under `mountPath`. + +This is an example of a Pod that mounts a Secret named `mysecret` in a volume: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mypod +spec: + containers: + - name: mypod + image: redis + volumeMounts: + - name: foo + mountPath: "/etc/foo" + readOnly: true + volumes: + - name: foo + secret: + secretName: mysecret + optional: false # default setting; "mysecret" must exist +``` + +Each Secret you want to use needs to be referred to in `.spec.volumes`. + +If there are multiple containers in the Pod, then each container needs its +own `volumeMounts` block, but only one `.spec.volumes` is needed per Secret. + +{{< note >}} +Versions of Kubernetes before v1.22 automatically created credentials for accessing +the Kubernetes API. This older mechanism was based on creating token Secrets that +could then be mounted into running Pods. +In more recent versions, including Kubernetes v{{< skew currentVersion >}}, API credentials +are obtained directly by using the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) API, +and are mounted into Pods using a [projected volume](/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume). +The tokens obtained using this method have bounded lifetimes, and are automatically +invalidated when the Pod they are mounted into is deleted. + +You can still [manually create](/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-service-account-api-token) +a service account token Secret; for example, if you need a token that never expires. +However, using the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +subresource to obtain a token to access the API is recommended instead. +You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-) +command to obtain a token from the `TokenRequest` API. +{{< /note >}} + +#### Projection of Secret keys to specific paths + +You can also control the paths within the volume where Secret keys are projected. +You can use the `.spec.volumes[].secret.items` field to change the target path of each key: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mypod +spec: + containers: + - name: mypod + image: redis + volumeMounts: + - name: foo + mountPath: "/etc/foo" + readOnly: true + volumes: + - name: foo + secret: + secretName: mysecret + items: + - key: username + path: my-group/my-username +``` + +What will happen: + +* the `username` key from `mysecret` is available to the container at the path + `/etc/foo/my-group/my-username` instead of at `/etc/foo/username`. +* the `password` key from that Secret object is not projected. + +If `.spec.volumes[].secret.items` is used, only keys specified in `items` are projected. +To consume all keys from the Secret, all of them must be listed in the `items` field. + +If you list keys explicitly, then all listed keys must exist in the corresponding Secret. +Otherwise, the volume is not created. + +#### Secret files permissions + +You can set the POSIX file access permission bits for a single Secret key. +If you don't specify any permissions, `0644` is used by default. +You can also set a default mode for the entire Secret volume and override per key if needed. + +For example, you can specify a default mode like this: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mypod +spec: + containers: + - name: mypod + image: redis + volumeMounts: + - name: foo + mountPath: "/etc/foo" + volumes: + - name: foo + secret: + secretName: mysecret + defaultMode: 0400 +``` + +The secret is mounted on `/etc/foo`; all the files created by the +secret volume mount have permission `0400`. + + +{{< note >}} +If you're defining a Pod or a Pod template using JSON, beware that the JSON +specification doesn't support octal notation. You can use the decimal value +for the `defaultMode` (for example, 0400 in octal is 256 in decimal) instead. +If you're writing YAML, you can write the `defaultMode` in octal. +{{< /note >}} + +#### Consuming Secret values from volumes + +Inside the container that mounts a secret volume, the secret keys appear as +files. The secret values are base64 decoded and stored inside these files. + +This is the result of commands executed inside the container from the example above: + +```shell +ls /etc/foo/ +``` + +The output is similar to: + +``` +username +password +``` + +```shell +cat /etc/foo/username +``` + +The output is similar to: + +``` +admin +``` + +```shell +cat /etc/foo/password +``` + +The output is similar to: + +``` +1f2d1e2e67df +``` + +The program in a container is responsible for reading the secret data from these +files, as needed. + +#### Mounted Secrets are updated automatically + +When a volume contains data from a Secret, and that Secret is updated, Kubernetes tracks +this and updates the data in the volume, using an eventually-consistent approach. + +{{< note >}} +A container using a Secret as a +[subPath](/docs/concepts/storage/volumes#using-subpath) volume mount does not receive +automated Secret updates. +{{< /note >}} + +The kubelet keeps a cache of the current keys and values for the Secrets that are used in +volumes for pods on that node. +You can configure the way that the kubelet detects changes from the cached values. The `configMapAndSecretChangeDetectionStrategy` field in +the [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/) controls which strategy the kubelet uses. The default strategy is `Watch`. + +Updates to Secrets can be either propagated by an API watch mechanism (the default), based on +a cache with a defined time-to-live, or polled from the cluster API server on each kubelet +synchronisation loop. + +As a result, the total delay from the moment when the Secret is updated to the moment +when new keys are projected to the Pod can be as long as the kubelet sync period + cache +propagation delay, where the cache propagation delay depends on the chosen cache type +(following the same order listed in the previous paragraph, these are: +watch propagation delay, the configured cache TTL, or zero for direct polling). + +### Using Secrets as environment variables + +To use a Secret in an {{< glossary_tooltip text="environment variable" term_id="container-env-variables" >}} +in a Pod: + +1. Create a Secret (or use an existing one). Multiple Pods can reference the same Secret. +1. Modify your Pod definition in each container that you wish to consume the value of a secret + key to add an environment variable for each secret key you wish to consume. The environment + variable that consumes the secret key should populate the secret's name and key in `env[].valueFrom.secretKeyRef`. +1. Modify your image and/or command line so that the program looks for values in the specified + environment variables. + +This is an example of a Pod that uses a Secret via environment variables: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: secret-env-pod +spec: + containers: + - name: mycontainer + image: redis + env: + - name: SECRET_USERNAME + valueFrom: + secretKeyRef: + name: mysecret + key: username + optional: false # same as default; "mysecret" must exist + # and include a key named "username" + - name: SECRET_PASSWORD + valueFrom: + secretKeyRef: + name: mysecret + key: password + optional: false # same as default; "mysecret" must exist + # and include a key named "password" + restartPolicy: Never +``` + + +#### Invalid environment variables {#restriction-env-from-invalid} + +Secrets used to populate environment variables by the `envFrom` field that have keys +that are considered invalid environment variable names will have those keys +skipped. The Pod is allowed to start. + +If you define a Pod with an invalid variable name, the failed Pod startup includes +an event with the reason set to `InvalidVariableNames` and a message that lists the +skipped invalid keys. The following example shows a Pod that refers to a Secret +named `mysecret`, where `mysecret` contains 2 invalid keys: `1badkey` and `2alsobad`. + +```shell +kubectl get events +``` + +The output is similar to: + +``` +LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON +0s 0s 1 dapi-test-pod Pod Warning InvalidEnvironmentVariableNames kubelet, 127.0.0.1 Keys [1badkey, 2alsobad] from the EnvFrom secret default/mysecret were skipped since they are considered invalid environment variable names. +``` + + +#### Consuming Secret values from environment variables + +Inside a container that consumes a Secret using environment variables, the secret keys appear +as normal environment variables. The values of those variables are the base64 decoded values +of the secret data. + +This is the result of commands executed inside the container from the example above: + +```shell +echo "$SECRET_USERNAME" +``` + +The output is similar to: + +``` +admin +``` + +```shell +echo "$SECRET_PASSWORD" +``` + +The output is similar to: + +``` +1f2d1e2e67df +``` + +{{< note >}} +If a container already consumes a Secret in an environment variable, +a Secret update will not be seen by the container unless it is +restarted. There are third party solutions for triggering restarts when +secrets change. +{{< /note >}} + +### Container image pull secrets {#using-imagepullsecrets} + +If you want to fetch container images from a private repository, you need a way for +the kubelet on each node to authenticate to that repository. You can configure +_image pull secrets_ to make this possible. These secrets are configured at the Pod +level. + +The `imagePullSecrets` field for a Pod is a list of references to Secrets in the same namespace +as the Pod. +You can use an `imagePullSecrets` to pass image registry access credentials to +the kubelet. The kubelet uses this information to pull a private image on behalf of your Pod. +See `PodSpec` in the [Pod API reference](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) +for more information about the `imagePullSecrets` field. + +#### Using imagePullSecrets + +The `imagePullSecrets` field is a list of references to secrets in the same namespace. +You can use an `imagePullSecrets` to pass a secret that contains a Docker (or other) image registry +password to the kubelet. The kubelet uses this information to pull a private image on behalf of your Pod. +See the [PodSpec API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core) for more information about the `imagePullSecrets` field. + +##### Manually specifying an imagePullSecret + +You can learn how to specify `imagePullSecrets` from the [container images](/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) +documentation. + +##### Arranging for imagePullSecrets to be automatically attached + +You can manually create `imagePullSecrets`, and reference these from +a ServiceAccount. Any Pods created with that ServiceAccount +or created with that ServiceAccount by default, will get their `imagePullSecrets` +field set to that of the service account. +See [Add ImagePullSecrets to a service account](/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account) + for a detailed explanation of that process. + +### Using Secrets with static Pods {#restriction-static-pod} + +You cannot use ConfigMaps or Secrets with +{{< glossary_tooltip text="static Pods" term_id="static-pod" >}}. + +## Use cases + +### Use case: As container environment variables + +Create a secret +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mysecret +type: Opaque +data: + USER_NAME: YWRtaW4= + PASSWORD: MWYyZDFlMmU2N2Rm +``` + +Create the Secret: +```shell +kubectl apply -f mysecret.yaml +``` + +Use `envFrom` to define all of the Secret's data as container environment variables. The key from the Secret becomes the environment variable name in the Pod. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: secret-test-pod +spec: + containers: + - name: test-container + image: k8s.gcr.io/busybox + command: [ "/bin/sh", "-c", "env" ] + envFrom: + - secretRef: + name: mysecret + restartPolicy: Never +``` + +### Use case: Pod with SSH keys + +Create a Secret containing some SSH keys: + +```shell +kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub +``` + +The output is similar to: + +``` +secret "ssh-key-secret" created +``` + +You can also create a `kustomization.yaml` with a `secretGenerator` field containing ssh keys. + +{{< caution >}} +Think carefully before sending your own SSH keys: other users of the cluster may have access +to the Secret. + +You could instead create an SSH private key representing a service identity that you want to be +accessible to all the users with whom you share the Kubernetes cluster, and that you can revoke +if the credentials are compromised. +{{< /caution >}} + +Now you can create a Pod which references the secret with the SSH key and +consumes it in a volume: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: secret-test-pod + labels: + name: secret-test +spec: + volumes: + - name: secret-volume + secret: + secretName: ssh-key-secret + containers: + - name: ssh-test-container + image: mySshImage + volumeMounts: + - name: secret-volume + readOnly: true + mountPath: "/etc/secret-volume" +``` + +When the container's command runs, the pieces of the key will be available in: + +``` +/etc/secret-volume/ssh-publickey +/etc/secret-volume/ssh-privatekey +``` + +The container is then free to use the secret data to establish an SSH connection. + +### Use case: Pods with prod / test credentials + +This example illustrates a Pod which consumes a secret containing production +credentials and another Pod which consumes a secret with test environment +credentials. + +You can create a `kustomization.yaml` with a `secretGenerator` field or run +`kubectl create secret`. + +```shell +kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11 +``` + +The output is similar to: + +``` +secret "prod-db-secret" created +``` + +You can also create a secret for test environment credentials. + +```shell +kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests +``` + +The output is similar to: + +``` +secret "test-db-secret" created +``` + +{{< note >}} +Special characters such as `$`, `\`, `*`, `=`, and `!` will be interpreted by your [shell](https://en.wikipedia.org/wiki/Shell_(computing)) and require escaping. + +In most shells, the easiest way to escape the password is to surround it with single quotes (`'`). +For example, if your actual password is `S!B\*d$zDsb=`, you should execute the command this way: + +```shell +kubectl create secret generic dev-db-secret --from-literal=username=devuser --from-literal=password='S!B\*d$zDsb=' +``` + +You do not need to escape special characters in passwords from files (`--from-file`). +{{< /note >}} + +Now make the Pods: + +```shell +cat <<EOF > pod.yaml +apiVersion: v1 +kind: List +items: +- kind: Pod + apiVersion: v1 + metadata: + name: prod-db-client-pod + labels: + name: prod-db-client + spec: + volumes: + - name: secret-volume + secret: + secretName: prod-db-secret + containers: + - name: db-client-container + image: myClientImage + volumeMounts: + - name: secret-volume + readOnly: true + mountPath: "/etc/secret-volume" +- kind: Pod + apiVersion: v1 + metadata: + name: test-db-client-pod + labels: + name: test-db-client + spec: + volumes: + - name: secret-volume + secret: + secretName: test-db-secret + containers: + - name: db-client-container + image: myClientImage + volumeMounts: + - name: secret-volume + readOnly: true + mountPath: "/etc/secret-volume" +EOF +``` + +Add the pods to the same `kustomization.yaml`: + +```shell +cat <<EOF >> kustomization.yaml +resources: +- pod.yaml +EOF +``` + +Apply all those objects on the API server by running: + +```shell +kubectl apply -k . +``` + +Both containers will have the following files present on their filesystems with the values +for each container's environment: + +``` +/etc/secret-volume/username +/etc/secret-volume/password +``` + +Note how the specs for the two Pods differ only in one field; this facilitates +creating Pods with different capabilities from a common Pod template. + +You could further simplify the base Pod specification by using two service accounts: + +1. `prod-user` with the `prod-db-secret` +1. `test-user` with the `test-db-secret` + +The Pod specification is shortened to: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: prod-db-client-pod + labels: + name: prod-db-client +spec: + serviceAccount: prod-db-client + containers: + - name: db-client-container + image: myClientImage +``` + +### Use case: dotfiles in a secret volume + +You can make your data "hidden" by defining a key that begins with a dot. +This key represents a dotfile or "hidden" file. For example, when the following secret +is mounted into a volume, `secret-volume`: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: dotfile-secret +data: + .secret-file: dmFsdWUtMg0KDQo= +--- +apiVersion: v1 +kind: Pod +metadata: + name: secret-dotfiles-pod +spec: + volumes: + - name: secret-volume + secret: + secretName: dotfile-secret + containers: + - name: dotfile-test-container + image: k8s.gcr.io/busybox + command: + - ls + - "-l" + - "/etc/secret-volume" + volumeMounts: + - name: secret-volume + readOnly: true + mountPath: "/etc/secret-volume" +``` + +The volume will contain a single file, called `.secret-file`, and +the `dotfile-test-container` will have this file present at the path +`/etc/secret-volume/.secret-file`. + +{{< note >}} +Files beginning with dot characters are hidden from the output of `ls -l`; +you must use `ls -la` to see them when listing directory contents. +{{< /note >}} + +### Use case: Secret visible to one container in a Pod + +Consider a program that needs to handle HTTP requests, do some complex business +logic, and then sign some messages with an HMAC. Because it has complex +application logic, there might be an unnoticed remote file reading exploit in +the server, which could expose the private key to an attacker. + +This could be divided into two processes in two containers: a frontend container +which handles user interaction and business logic, but which cannot see the +private key; and a signer container that can see the private key, and responds +to simple signing requests from the frontend (for example, over localhost networking). + +With this partitioned approach, an attacker now has to trick the application +server into doing something rather arbitrary, which may be harder than getting +it to read a file. + ## Types of Secret {#secret-types} When creating a Secret, you can specify its type using the `type` field of -a Secret resource, or certain equivalent `kubectl` command line flags (if available). -The `type` of a Secret is used to facilitate programmatic handling of different -kinds of confidential data. +the [Secret](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/) +resource, or certain equivalent `kubectl` command line flags (if available). +The Secret type is used to facilitate programmatic handling of the Secret data. -Kubernetes provides several builtin types for some common usage scenarios. +Kubernetes provides several built-in types for some common usage scenarios. These types vary in terms of the validations performed and the constraints Kubernetes imposes on them. -| Builtin Type | Usage | +| Built-in Type | Usage | |--------------|-------| | `Opaque` | arbitrary user-defined data | -| `kubernetes.io/service-account-token` | service account token | +| `kubernetes.io/service-account-token` | ServiceAccount token | | `kubernetes.io/dockercfg` | serialized `~/.dockercfg` file | | `kubernetes.io/dockerconfigjson` | serialized `~/.docker/config.json` file | | `kubernetes.io/basic-auth` | credentials for basic authentication | @@ -95,11 +853,16 @@ Kubernetes imposes on them. | `bootstrap.kubernetes.io/token` | bootstrap token data | You can define and use your own Secret type by assigning a non-empty string as the -`type` value for a Secret object. An empty string is treated as an `Opaque` type. +`type` value for a Secret object (an empty string is treated as an `Opaque` type). + Kubernetes doesn't impose any constraints on the type name. However, if you -are using one of the builtin types, you must meet all the requirements defined +are using one of the built-in types, you must meet all the requirements defined for that type. +If you are defining a type of secret that's for public use, follow the convention +and structure the secret type to have your domain name before the name, separated +by a `/`. For example: `cloud-hosting.example.net/cloud-api-credentials`. + ### Opaque secrets `Opaque` is the default Secret type if omitted from a Secret configuration file. @@ -120,16 +883,35 @@ empty-secret Opaque 0 2m6s ``` The `DATA` column shows the number of data items stored in the Secret. -In this case, `0` means we have created an empty Secret. +In this case, `0` means you have created an empty Secret. -### Service account token Secrets +### Service account token Secrets A `kubernetes.io/service-account-token` type of Secret is used to store a -token that identifies a service account. When using this Secret type, you need -to ensure that the `kubernetes.io/service-account.name` annotation is set to an -existing service account name. A Kubernetes controller fills in some other -fields such as the `kubernetes.io/service-account.uid` annotation and the -`token` key in the `data` field set to actual token content. +token credential that identifies a +{{< glossary_tooltip text="service account" term_id="service-account" >}}. + +Since 1.22, this type of Secret is no longer used to mount credentials into Pods, +and obtaining tokens via the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +API is recommended instead of using service account token Secret objects. +Tokens obtained from the `TokenRequest` API are more secure than ones stored in Secret objects, +because they have a bounded lifetime and are not readable by other API clients. +You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-) +command to obtain a token from the `TokenRequest` API. + +You should only create a service account token Secret object +if you can't use the `TokenRequest` API to obtain a token, +and the security exposure of persisting a non-expiring token credential +in a readable API object is acceptable to you. + +When using this Secret type, you need to ensure that the +`kubernetes.io/service-account.name` annotation is set to an existing +service account name. If you are creating both the ServiceAccount and +the Secret objects, you should create the ServiceAccount object first. + +After the Secret is created, a Kubernetes {{< glossary_tooltip text="controller" term_id="controller" >}} +fills in some other fields such as the `kubernetes.io/service-account.uid` annotation, and the +`token` key in the `data` field, which is populated with an authentication token. The following example configuration declares a service account token Secret: @@ -146,25 +928,19 @@ data: extra: YmFyCg== ``` -When creating a `Pod`, Kubernetes automatically creates a service account Secret -and automatically modifies your Pod to use this Secret. The service account token -Secret contains credentials for accessing the API. - -The automatic creation and use of API credentials can be disabled or -overridden if desired. However, if all you need to do is securely access the -API server, this is the recommended workflow. +After creating the Secret, wait for Kubernetes to populate the `token` key in the `data` field. See the [ServiceAccount](/docs/tasks/configure-pod-container/configure-service-account/) -documentation for more information on how service accounts work. +documentation for more information on how service accounts work. You can also check the `automountServiceAccountToken` field and the `serviceAccountName` field of the [`Pod`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) -for information on referencing service account from Pods. +for information on referencing service account credentials from within Pods. ### Docker config Secrets You can use one of the following `type` values to create a Secret to -store the credentials for accessing a Docker registry for images. +store the credentials for accessing a container image registry: - `kubernetes.io/dockercfg` - `kubernetes.io/dockerconfigjson` @@ -201,52 +977,64 @@ If you do not want to perform the base64 encoding, you can choose to use the {{< /note >}} When you create these types of Secrets using a manifest, the API -server checks whether the expected key does exists in the `data` field, and +server checks whether the expected key exists in the `data` field, and it verifies if the value provided can be parsed as a valid JSON. The API server doesn't validate if the JSON actually is a Docker config file. When you do not have a Docker config file, or you want to use `kubectl` -to create a Docker registry Secret, you can do: +to create a Secret for accessing a container registry, you can do: ```shell kubectl create secret docker-registry secret-tiger-docker \ + --docker-email=tiger@acme.example \ --docker-username=tiger \ - --docker-password=pass113 \ - --docker-email=tiger@acme.com + --docker-password=pass1234 \ + --docker-server=my-registry.example:5000 ``` -This command creates a Secret of type `kubernetes.io/dockerconfigjson`. -If you dump the `.dockerconfigjson` content from the `data` field, you will -get the following JSON content which is a valid Docker configuration created -on the fly: +That command creates a Secret of type `kubernetes.io/dockerconfigjson`. +If you dump the `.data.dockerconfigjson` field from that new Secret and then +decode it from base64: + +```shell +kubectl get secret secret-tiger-docker -o jsonpath='{.data.*}' | base64 -d +``` + +then the output is equivalent to this JSON document (which is also a valid +Docker configuration file): ```json { "auths": { - "https://index.docker.io/v1/": { + "my-registry.example:5000": { "username": "tiger", - "password": "pass113", - "email": "tiger@acme.com", - "auth": "dGlnZXI6cGFzczExMw==" + "password": "pass1234", + "email": "tiger@acme.example", + "auth": "dGlnZXI6cGFzczEyMzQ=" } } } ``` +{{< note >}} +The `auth` value there is base64 encoded; it is obscured but not secret. +Anyone who can read that Secret can learn the registry access bearer token. +{{< /note >}} + ### Basic authentication Secret The `kubernetes.io/basic-auth` type is provided for storing credentials needed for basic authentication. When using this Secret type, the `data` field of the -Secret must contain the following two keys: +Secret must contain one of the following two keys: -- `username`: the user name for authentication; -- `password`: the password or token for authentication. +- `username`: the user name for authentication +- `password`: the password or token for authentication Both values for the above two keys are base64 encoded strings. You can, of course, provide the clear text content using the `stringData` for Secret creation. -The following YAML is an example config for a basic authentication Secret: +The following manifest is an example of a basic authentication Secret: ```yaml apiVersion: v1 @@ -255,15 +1043,17 @@ metadata: name: secret-basic-auth type: kubernetes.io/basic-auth stringData: - username: admin - password: t0p-Secret + username: admin # required field for kubernetes.io/basic-auth + password: t0p-Secret # required field for kubernetes.io/basic-auth ``` -The basic authentication Secret type is provided only for user's convenience. -You can create an `Opaque` for credentials used for basic authentication. -However, using the builtin Secret type helps unify the formats of your credentials -and the API server does verify if the required keys are provided in a Secret -configuration. +The basic authentication Secret type is provided only for convenience. +You can create an `Opaque` type for credentials used for basic authentication. +However, using the defined and public Secret type (`kubernetes.io/basic-auth`) helps other +people to understand the purpose of your Secret, and sets a convention for what key names +to expect. +The Kubernetes API verifies that the required keys are set for a Secret +of this type. ### SSH authentication secrets @@ -272,7 +1062,8 @@ SSH authentication. When using this Secret type, you will have to specify a `ssh-privatekey` key-value pair in the `data` (or `stringData`) field as the SSH credential to use. -The following YAML is an example config for a SSH authentication Secret: +The following manifest is an example of a Secret used for SSH public/private +key authentication: ```yaml apiVersion: v1 @@ -287,8 +1078,10 @@ data: ``` The SSH authentication Secret type is provided only for user's convenience. -You can create an `Opaque` for credentials used for SSH authentication. -However, using the builtin Secret type helps unify the formats of your credentials +You could instead create an `Opaque` type Secret for credentials used for SSH authentication. +However, using the defined and public Secret type (`kubernetes.io/ssh-auth`) helps other +people to understand the purpose of your Secret, and sets a convention for what key names +to expect. and the API server does verify if the required keys are provided in a Secret configuration. @@ -302,9 +1095,11 @@ ConfigMap. ### TLS secrets Kubernetes provides a builtin Secret type `kubernetes.io/tls` for storing -a certificate and its associated key that are typically used for TLS . This -data is primarily used with TLS termination of the Ingress resource, but may -be used with other resources or directly by a workload. +a certificate and its associated key that are typically used for TLS. + +One common use for TLS secrets is to configure encryption in transit for +an [Ingress](/docs/concepts/services-networking/ingress/), but you can also use it +with other resources or directly in your workload. When using this type of Secret, the `tls.key` and the `tls.crt` key must be provided in the `data` (or `stringData`) field of the Secret configuration, although the API server doesn't actually validate the values for each key. @@ -339,20 +1134,28 @@ kubectl create secret tls my-tls-secret \ --key=path/to/key/file ``` -The public/private key pair must exist beforehand. The public key certificate -for `--cert` must be .PEM encoded (Base64-encoded DER format), and match the -given private key for `--key`. -The private key must be in what is commonly called PEM private key format, -unencrypted. In both cases, the initial and the last lines from PEM (for -example, `--------BEGIN CERTIFICATE-----` and `-------END CERTIFICATE----` for -a certificate) are *not* included. +The public/private key pair must exist before hand. The public key certificate +for `--cert` must be DER format as per +[Section 5.1 of RFC 7468](https://datatracker.ietf.org/doc/html/rfc7468#section-5.1), +and must match the given private key for `--key` (PKCS #8 in DER format; +[Section 11 of RFC 7468](https://datatracker.ietf.org/doc/html/rfc7468#section-11)). + +{{< note >}} +A kubernetes.io/tls Secret stores the Base64-encoded DER data for keys and +certificates. If you're familiar with PEM format for private keys and for certificates, +the base64 data are the same as that format except that you omit +the initial and the last lines that are used in PEM. + +For example, for a certificate, you do **not** include `--------BEGIN CERTIFICATE-----` +and `-------END CERTIFICATE----`. +{{< /note >}} ### Bootstrap token Secrets A bootstrap token Secret can be created by explicitly specifying the Secret `type` to `bootstrap.kubernetes.io/token`. This type of Secret is designed for tokens used during the node bootstrap process. It stores tokens used to sign -well known ConfigMaps. +well-known ConfigMaps. A bootstrap token Secret is usually created in the `kube-system` namespace and named in the form `bootstrap-token-<token-id>` where `<token-id>` is a 6 character @@ -414,367 +1217,23 @@ stringData: usage-bootstrap-signing: "true" ``` -## Creating a Secret - -There are several options to create a Secret: - -- [create Secret using `kubectl` command](/docs/tasks/configmap-secret/managing-secret-using-kubectl/) -- [create Secret from config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/) -- [create Secret using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/) - -## Editing a Secret - -An existing Secret may be edited with the following command: - -```shell -kubectl edit secrets mysecret -``` - -This will open the default configured editor and allow for updating the base64 encoded Secret values in the `data` field: - -```yaml -# Please edit the object below. Lines beginning with a '#' will be ignored, -# and an empty file will abort the edit. If an error occurs while saving this file will be -# reopened with the relevant failures. -# -apiVersion: v1 -data: - username: YWRtaW4= - password: MWYyZDFlMmU2N2Rm -kind: Secret -metadata: - annotations: - kubectl.kubernetes.io/last-applied-configuration: { ... } - creationTimestamp: 2016-01-22T18:41:56Z - name: mysecret - namespace: default - resourceVersion: "164619" - uid: cfee02d6-c137-11e5-8d73-42010af00002 -type: Opaque -``` - -## Using Secrets - -Secrets can be mounted as data volumes or exposed as -{{< glossary_tooltip text="environment variables" term_id="container-env-variables" >}} -to be used by a container in a Pod. Secrets can also be used by other parts of the -system, without being directly exposed to the Pod. For example, Secrets can hold -credentials that other parts of the system should use to interact with external -systems on your behalf. - -### Using Secrets as files from a Pod - -To consume a Secret in a volume in a Pod: - -1. Create a secret or use an existing one. Multiple Pods can reference the same secret. -1. Modify your Pod definition to add a volume under `.spec.volumes[]`. Name the volume anything, and have a `.spec.volumes[].secret.secretName` field equal to the name of the Secret object. -1. Add a `.spec.containers[].volumeMounts[]` to each container that needs the secret. Specify `.spec.containers[].volumeMounts[].readOnly = true` and `.spec.containers[].volumeMounts[].mountPath` to an unused directory name where you would like the secrets to appear. -1. Modify your image or command line so that the program looks for files in that directory. Each key in the secret `data` map becomes the filename under `mountPath`. - -This is an example of a Pod that mounts a Secret in a volume: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - containers: - - name: mypod - image: redis - volumeMounts: - - name: foo - mountPath: "/etc/foo" - readOnly: true - volumes: - - name: foo - secret: - secretName: mysecret -``` - -Each Secret you want to use needs to be referred to in `.spec.volumes`. - -If there are multiple containers in the Pod, then each container needs its -own `volumeMounts` block, but only one `.spec.volumes` is needed per Secret. - -You can package many files into one secret, or use many secrets, whichever is convenient. - -#### Projection of Secret keys to specific paths - -You can also control the paths within the volume where Secret keys are projected. -You can use the `.spec.volumes[].secret.items` field to change the target path of each key: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - containers: - - name: mypod - image: redis - volumeMounts: - - name: foo - mountPath: "/etc/foo" - readOnly: true - volumes: - - name: foo - secret: - secretName: mysecret - items: - - key: username - path: my-group/my-username -``` - -What will happen: - -* `username` secret is stored under `/etc/foo/my-group/my-username` file instead of `/etc/foo/username`. -* `password` secret is not projected. - -If `.spec.volumes[].secret.items` is used, only keys specified in `items` are projected. -To consume all keys from the secret, all of them must be listed in the `items` field. -All listed keys must exist in the corresponding secret. Otherwise, the volume is not created. - -#### Secret files permissions - -You can set the file access permission bits for a single Secret key. -If you don't specify any permissions, `0644` is used by default. -You can also set a default mode for the entire Secret volume and override per key if needed. - -For example, you can specify a default mode like this: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - containers: - - name: mypod - image: redis - volumeMounts: - - name: foo - mountPath: "/etc/foo" - volumes: - - name: foo - secret: - secretName: mysecret - defaultMode: 0400 -``` - -Then, the secret will be mounted on `/etc/foo` and all the files created by the -secret volume mount will have permission `0400`. - -Note that the JSON spec doesn't support octal notation, so use the value 256 for -0400 permissions. If you use YAML instead of JSON for the Pod, you can use octal -notation to specify permissions in a more natural way. - -Note if you `kubectl exec` into the Pod, you need to follow the symlink to find -the expected file mode. For example, - -Check the secrets file mode on the pod. -``` -kubectl exec mypod -it sh - -cd /etc/foo -ls -l -``` - -The output is similar to this: -``` -total 0 -lrwxrwxrwx 1 root root 15 May 18 00:18 password -> ..data/password -lrwxrwxrwx 1 root root 15 May 18 00:18 username -> ..data/username -``` - -Follow the symlink to find the correct file mode. - -``` -cd /etc/foo/..data -ls -l -``` - -The output is similar to this: -``` -total 8 --r-------- 1 root root 12 May 18 00:18 password --r-------- 1 root root 5 May 18 00:18 username -``` - -You can also use mapping, as in the previous example, and specify different -permissions for different files like this: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - containers: - - name: mypod - image: redis - volumeMounts: - - name: foo - mountPath: "/etc/foo" - volumes: - - name: foo - secret: - secretName: mysecret - items: - - key: username - path: my-group/my-username - mode: 0777 -``` - -In this case, the file resulting in `/etc/foo/my-group/my-username` will have -permission value of `0777`. If you use JSON, owing to JSON limitations, you -must specify the mode in decimal notation, `511`. - -Note that this permission value might be displayed in decimal notation if you -read it later. - -#### Consuming Secret values from volumes - -Inside the container that mounts a secret volume, the secret keys appear as -files and the secret values are base64 decoded and stored inside these files. -This is the result of commands executed inside the container from the example above: - -```shell -ls /etc/foo/ -``` - -The output is similar to: - -``` -username -password -``` - -```shell -cat /etc/foo/username -``` - -The output is similar to: - -``` -admin -``` - -```shell -cat /etc/foo/password -``` - -The output is similar to: - -``` -1f2d1e2e67df -``` - -The program in a container is responsible for reading the secrets from the -files. - -#### Mounted Secrets are updated automatically - -When a secret currently consumed in a volume is updated, projected keys are eventually updated as well. -The kubelet checks whether the mounted secret is fresh on every periodic sync. -However, the kubelet uses its local cache for getting the current value of the Secret. -The type of the cache is configurable using the `ConfigMapAndSecretChangeDetectionStrategy` field in -the [KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/). -A Secret can be either propagated by watch (default), ttl-based, or by redirecting -all requests directly to the API server. -As a result, the total delay from the moment when the Secret is updated to the moment -when new keys are projected to the Pod can be as long as the kubelet sync period + cache -propagation delay, where the cache propagation delay depends on the chosen cache type -(it equals to watch propagation delay, ttl of cache, or zero correspondingly). - -{{< note >}} -A container using a Secret as a -[subPath](/docs/concepts/storage/volumes#using-subpath) volume mount will not receive -Secret updates. -{{< /note >}} - -### Using Secrets as environment variables - -To use a secret in an {{< glossary_tooltip text="environment variable" term_id="container-env-variables" >}} -in a Pod: - -1. Create a secret or use an existing one. Multiple Pods can reference the same secret. -1. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in `env[].valueFrom.secretKeyRef`. -1. Modify your image and/or command line so that the program looks for values in the specified environment variables. - -This is an example of a Pod that uses secrets from environment variables: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: secret-env-pod -spec: - containers: - - name: mycontainer - image: redis - env: - - name: SECRET_USERNAME - valueFrom: - secretKeyRef: - name: mysecret - key: username - - name: SECRET_PASSWORD - valueFrom: - secretKeyRef: - name: mysecret - key: password - restartPolicy: Never -``` - -#### Consuming Secret Values from environment variables - -Inside a container that consumes a secret in the environment variables, the secret keys appear as -normal environment variables containing the base64 decoded values of the secret data. -This is the result of commands executed inside the container from the example above: - -```shell -echo $SECRET_USERNAME -``` - -The output is similar to: - -``` -admin -``` - -```shell -echo $SECRET_PASSWORD -``` - -The output is similar to: - -``` -1f2d1e2e67df -``` - -#### Environment variables are not updated after a secret update - -If a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. -There are third party solutions for triggering restarts when secrets change. ## Immutable Secrets {#secret-immutable} {{< feature-state for_k8s_version="v1.21" state="stable" >}} -The Kubernetes feature _Immutable Secrets and ConfigMaps_ provides an option to set -individual Secrets and ConfigMaps as immutable. For clusters that extensively use Secrets -(at least tens of thousands of unique Secret to Pod mounts), preventing changes to their -data has the following advantages: +Kubernetes lets you mark specific Secrets (and ConfigMaps) as _immutable_. +Preventing changes to the data of an existing Secret has the following benefits: - protects you from accidental (or unwanted) updates that could cause applications outages -- improves performance of your cluster by significantly reducing load on kube-apiserver, by -closing watches for secrets marked as immutable. +- (for clusters that extensively use Secrets - at least tens of thousands of unique Secret + to Pod mounts), switching to immutable Secrets improves the performance of your cluster + by significantly reducing load on kube-apiserver. The kubelet does not need to maintain + a [watch] on any Secrets that are marked as immutable. -This feature is controlled by the `ImmutableEphemeralVolumes` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/), -which is enabled by default since v1.19. You can create an immutable -Secret by setting the `immutable` field to `true`. For example, +### Marking a Secret as immutable {#secret-immutable-create} + +You can create an immutable Secret by setting the `immutable` field to `true`. For example, ```yaml apiVersion: v1 kind: Secret @@ -785,6 +1244,8 @@ data: immutable: true ``` +You can also update any existing mutable Secret to make it immutable. + {{< note >}} Once a Secret or ConfigMap is marked as immutable, it is _not_ possible to revert this change nor to mutate the contents of the `data` field. You can only delete and recreate the Secret. @@ -792,388 +1253,10 @@ Existing Pods maintain a mount point to the deleted Secret - it is recommended t these pods. {{< /note >}} -### Using imagePullSecrets +## Information security for Secrets -The `imagePullSecrets` field is a list of references to secrets in the same namespace. -You can use an `imagePullSecrets` to pass a secret that contains a Docker (or other) image registry -password to the kubelet. The kubelet uses this information to pull a private image on behalf of your Pod. -See the [PodSpec API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core) for more information about the `imagePullSecrets` field. - -#### Manually specifying an imagePullSecret - -You can learn how to specify `ImagePullSecrets` from the [container images documentation](/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod). - -### Arranging for imagePullSecrets to be automatically attached - -You can manually create `imagePullSecrets`, and reference it from -a ServiceAccount. Any Pods created with that ServiceAccount -or created with that ServiceAccount by default, will get their `imagePullSecrets` -field set to that of the service account. -See [Add ImagePullSecrets to a service account](/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account) - for a detailed explanation of that process. - -## Details - -### Restrictions - -Secret volume sources are validated to ensure that the specified object -reference actually points to an object of type Secret. Therefore, a secret -needs to be created before any Pods that depend on it. - -Secret resources reside in a {{< glossary_tooltip text="namespace" term_id="namespace" >}}. -Secrets can only be referenced by Pods in that same namespace. - -Individual secrets are limited to 1MiB in size. This is to discourage creation -of very large secrets which would exhaust the API server and kubelet memory. -However, creation of many smaller secrets could also exhaust memory. More -comprehensive limits on memory usage due to secrets is a planned feature. - -The kubelet only supports the use of secrets for Pods where the secrets -are obtained from the API server. -This includes any Pods created using `kubectl`, or indirectly via a replication -controller. It does not include Pods created as a result of the kubelet -`--manifest-url` flag, its `--config` flag, or its REST API (these are -not common ways to create Pods). -The `spec` of a {{< glossary_tooltip text="static Pod" term_id="static-pod" >}} cannot refer to a Secret -or any other API objects. - - -Secrets must be created before they are consumed in Pods as environment -variables unless they are marked as optional. References to secrets that do -not exist will prevent the Pod from starting. - -References (`secretKeyRef` field) to keys that do not exist in a named Secret -will prevent the Pod from starting. - -Secrets used to populate environment variables by the `envFrom` field that have keys -that are considered invalid environment variable names will have those keys -skipped. The Pod will be allowed to start. There will be an event whose -reason is `InvalidVariableNames` and the message will contain the list of -invalid keys that were skipped. The example shows a pod which refers to the -default/mysecret that contains 2 invalid keys: `1badkey` and `2alsobad`. - -```shell -kubectl get events -``` - -The output is similar to: - -``` -LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON -0s 0s 1 dapi-test-pod Pod Warning InvalidEnvironmentVariableNames kubelet, 127.0.0.1 Keys [1badkey, 2alsobad] from the EnvFrom secret default/mysecret were skipped since they are considered invalid environment variable names. -``` - -### Secret and Pod lifetime interaction - -When a Pod is created by calling the Kubernetes API, there is no check if a referenced -secret exists. Once a Pod is scheduled, the kubelet will try to fetch the -secret value. If the secret cannot be fetched because it does not exist or -because of a temporary lack of connection to the API server, the kubelet will -periodically retry. It will report an event about the Pod explaining the -reason it is not started yet. Once the secret is fetched, the kubelet will -create and mount a volume containing it. None of the Pod's containers will -start until all the Pod's volumes are mounted. - -## Use cases - -### Use-Case: As container environment variables - -Create a secret - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: mysecret -type: Opaque -data: - USER_NAME: YWRtaW4= - PASSWORD: MWYyZDFlMmU2N2Rm -``` - -Create the Secret: - -```shell -kubectl apply -f mysecret.yaml -``` - -Use `envFrom` to define all of the Secret's data as container environment variables. The key from the Secret becomes the environment variable name in the Pod. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: secret-test-pod -spec: - containers: - - name: test-container - image: k8s.gcr.io/busybox - command: [ "/bin/sh", "-c", "env" ] - envFrom: - - secretRef: - name: mysecret - restartPolicy: Never -``` - -### Use-Case: Pod with ssh keys - -Create a secret containing some ssh keys: - -```shell -kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub -``` - -The output is similar to: - -``` -secret "ssh-key-secret" created -``` - -You can also create a `kustomization.yaml` with a `secretGenerator` field containing ssh keys. - -{{< caution >}} -Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret. Use a service account which you want to be accessible to all the users with whom you share the Kubernetes cluster, and can revoke this account if the users are compromised. -{{< /caution >}} - -Now you can create a Pod which references the secret with the ssh key and -consumes it in a volume: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: secret-test-pod - labels: - name: secret-test -spec: - volumes: - - name: secret-volume - secret: - secretName: ssh-key-secret - containers: - - name: ssh-test-container - image: mySshImage - volumeMounts: - - name: secret-volume - readOnly: true - mountPath: "/etc/secret-volume" -``` - -When the container's command runs, the pieces of the key will be available in: - -``` -/etc/secret-volume/ssh-publickey -/etc/secret-volume/ssh-privatekey -``` - -The container is then free to use the secret data to establish an ssh connection. - -### Use-Case: Pods with prod / test credentials - -This example illustrates a Pod which consumes a secret containing production -credentials and another Pod which consumes a secret with test environment -credentials. - -You can create a `kustomization.yaml` with a `secretGenerator` field or run -`kubectl create secret`. - -```shell -kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11 -``` - -The output is similar to: - -``` -secret "prod-db-secret" created -``` - -You can also create a secret for test environment credentials. - -```shell -kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests -``` - -The output is similar to: - -``` -secret "test-db-secret" created -``` - -{{< note >}} -Special characters such as `$`, `\`, `*`, `=`, and `!` will be interpreted by your [shell](https://en.wikipedia.org/wiki/Shell_(computing)) and require escaping. -In most shells, the easiest way to escape the password is to surround it with single quotes (`'`). -For example, if your actual password is `S!B\*d$zDsb=`, you should execute the command this way: - -```shell -kubectl create secret generic dev-db-secret --from-literal=username=devuser --from-literal=password='S!B\*d$zDsb=' -``` - -You do not need to escape special characters in passwords from files (`--from-file`). -{{< /note >}} - -Now make the Pods: - -```shell -cat <<EOF > pod.yaml -apiVersion: v1 -kind: List -items: -- kind: Pod - apiVersion: v1 - metadata: - name: prod-db-client-pod - labels: - name: prod-db-client - spec: - volumes: - - name: secret-volume - secret: - secretName: prod-db-secret - containers: - - name: db-client-container - image: myClientImage - volumeMounts: - - name: secret-volume - readOnly: true - mountPath: "/etc/secret-volume" -- kind: Pod - apiVersion: v1 - metadata: - name: test-db-client-pod - labels: - name: test-db-client - spec: - volumes: - - name: secret-volume - secret: - secretName: test-db-secret - containers: - - name: db-client-container - image: myClientImage - volumeMounts: - - name: secret-volume - readOnly: true - mountPath: "/etc/secret-volume" -EOF -``` - -Add the pods to the same kustomization.yaml: - -```shell -cat <<EOF >> kustomization.yaml -resources: -- pod.yaml -EOF -``` - -Apply all those objects on the API server by running: - -```shell -kubectl apply -k . -``` - -Both containers will have the following files present on their filesystems with the values for each container's environment: - -``` -/etc/secret-volume/username -/etc/secret-volume/password -``` - -Note how the specs for the two Pods differ only in one field; this facilitates -creating Pods with different capabilities from a common Pod template. - -You could further simplify the base Pod specification by using two service accounts: - -1. `prod-user` with the `prod-db-secret` -1. `test-user` with the `test-db-secret` - -The Pod specification is shortened to: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: prod-db-client-pod - labels: - name: prod-db-client -spec: - serviceAccount: prod-db-client - containers: - - name: db-client-container - image: myClientImage -``` - -### Use-case: dotfiles in a secret volume - -You can make your data "hidden" by defining a key that begins with a dot. -This key represents a dotfile or "hidden" file. For example, when the following secret -is mounted into a volume, `secret-volume`: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: dotfile-secret -data: - .secret-file: dmFsdWUtMg0KDQo= ---- -apiVersion: v1 -kind: Pod -metadata: - name: secret-dotfiles-pod -spec: - volumes: - - name: secret-volume - secret: - secretName: dotfile-secret - containers: - - name: dotfile-test-container - image: k8s.gcr.io/busybox - command: - - ls - - "-l" - - "/etc/secret-volume" - volumeMounts: - - name: secret-volume - readOnly: true - mountPath: "/etc/secret-volume" -``` - -The volume will contain a single file, called `.secret-file`, and -the `dotfile-test-container` will have this file present at the path -`/etc/secret-volume/.secret-file`. - -{{< note >}} -Files beginning with dot characters are hidden from the output of `ls -l`; -you must use `ls -la` to see them when listing directory contents. -{{< /note >}} - -### Use-case: Secret visible to one container in a Pod - -Consider a program that needs to handle HTTP requests, do some complex business -logic, and then sign some messages with an HMAC. Because it has complex -application logic, there might be an unnoticed remote file reading exploit in -the server, which could expose the private key to an attacker. - -This could be divided into two processes in two containers: a frontend container -which handles user interaction and business logic, but which cannot see the -private key; and a signer container that can see the private key, and responds -to simple signing requests from the frontend (for example, over localhost networking). - -With this partitioned approach, an attacker now has to trick the application -server into doing something rather arbitrary, which may be harder than getting -it to read a file. - -<!-- TODO: explain how to do this while still using automation. --> - -## Best practices - -### Clients that use the Secret API - -When deploying applications that interact with the Secret API, you should -limit access using [authorization policies]( -/docs/reference/access-authn-authz/authorization/) such as [RBAC]( -/docs/reference/access-authn-authz/rbac/). +Although ConfigMap and Secret work similarly, Kubernetes applies some additional +protection for Secret objects. Secrets often hold values that span a spectrum of importance, many of which can cause escalations within Kubernetes (e.g. service account tokens) and to @@ -1181,78 +1264,78 @@ external systems. Even if an individual app can reason about the power of the Secrets it expects to interact with, other apps within the same namespace can render those assumptions invalid. -For these reasons `watch` and `list` requests for secrets within a namespace are -extremely powerful capabilities and should be avoided, since listing secrets allows -the clients to inspect the values of all secrets that are in that namespace. The ability to -`watch` and `list` all secrets in a cluster should be reserved for only the most -privileged, system-level components. +A Secret is only sent to a node if a Pod on that node requires it. +For mounting secrets into Pods, the kubelet stores a copy of the data into a `tmpfs` +so that the confidential data is not written to durable storage. +Once the Pod that depends on the Secret is deleted, the kubelet deletes its local copy +of the confidential data from the Secret. -Applications that need to access the Secret API should perform `get` requests on -the secrets they need. This lets administrators restrict access to all secrets -while [white-listing access to individual instances](/docs/reference/access-authn-authz/rbac/#referring-to-resources) that -the app needs. +There may be several containers in a Pod. By default, containers you define +only have access to the default ServiceAccount and its related Secret. +You must explicitly define environment variables or map a volume into a +container in order to provide access to any other Secret. -For improved performance over a looping `get`, clients can design resources that -reference a secret then `watch` the resource, re-requesting the secret when the -reference changes. Additionally, a ["bulk watch" API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/bulk_watch.md) -to let clients `watch` individual resources has also been proposed, and will likely -be available in future releases of Kubernetes. +There may be Secrets for several Pods on the same node. However, only the +Secrets that a Pod requests are potentially visible within its containers. +Therefore, one Pod does not have access to the Secrets of another Pod. -## Security properties +{{< warning >}} +Any privileged containers on a node are liable to have access to all Secrets used +on that node. +{{< /warning >}} -### Protections -Because secrets can be created independently of the Pods that use -them, there is less risk of the secret being exposed during the workflow of -creating, viewing, and editing Pods. The system can also take additional -precautions with Secrets, such as avoiding writing them to disk where -possible. +### Security recommendations for developers -A secret is only sent to a node if a Pod on that node requires it. -The kubelet stores the secret into a `tmpfs` so that the secret is not written -to disk storage. Once the Pod that depends on the secret is deleted, the kubelet -will delete its local copy of the secret data as well. +- Applications still need to protect the value of confidential information after reading it + from an environment variable or volume. For example, your application must avoid logging + the secret data in the clear or transmitting it to an untrusted party. +- If you are defining multiple containers in a Pod, and only one of those + containers needs access to a Secret, define the volume mount or environment + variable configuration so that the other containers do not have access to that + Secret. +- If you configure a Secret through a {{< glossary_tooltip text="manifest" term_id="manifest" >}}, + with the secret data encoded as base64, sharing this file or checking it in to a + source repository means the secret is available to everyone who can read the manifest. + Base64 encoding is _not_ an encryption method, it provides no additional confidentiality + over plain text. +- When deploying applications that interact with the Secret API, you should + limit access using + [authorization policies](/docs/reference/access-authn-authz/authorization/) such as + [RBAC](/docs/reference/access-authn-authz/rbac/). +- In the Kubernetes API, `watch` and `list` requests for Secrets within a namespace + are extremely powerful capabilities. Avoid granting this access where feasible, since + listing Secrets allows the clients to inspect the values of every Secret in that + namespace. -There may be secrets for several Pods on the same node. However, only the -secrets that a Pod requests are potentially visible within its containers. -Therefore, one Pod does not have access to the secrets of another Pod. +### Security recommendations for cluster administrators -There may be several containers in a Pod. However, each container in a Pod has -to request the secret volume in its `volumeMounts` for it to be visible within -the container. This can be used to construct useful [security partitions at the -Pod level](#use-case-secret-visible-to-one-container-in-a-pod). +{{< caution >}} +A user who can create a Pod that uses a Secret can also see the value of that Secret. Even +if cluster policies do not allow a user to read the Secret directly, the same user could +have access to run a Pod that then exposes the Secret. +{{< /caution >}} -On most Kubernetes distributions, communication between users -and the API server, and from the API server to the kubelets, is protected by SSL/TLS. -Secrets are protected when transmitted over these channels. - -{{< feature-state for_k8s_version="v1.13" state="beta" >}} - -You can enable [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/) -for secret data, so that the secrets are not stored in the clear into {{< glossary_tooltip term_id="etcd" >}}. - -### Risks - - - In the API server, secret data is stored in {{< glossary_tooltip term_id="etcd" >}}; - therefore: - - Administrators should enable encryption at rest for cluster data (requires v1.13 or later). - - Administrators should limit access to etcd to admin users. - - Administrators may want to wipe/shred disks used by etcd when no longer in use. - - If running etcd in a cluster, administrators should make sure to use SSL/TLS - for etcd peer-to-peer communication. - - If you configure the secret through a manifest (JSON or YAML) file which has - the secret data encoded as base64, sharing this file or checking it in to a - source repository means the secret is compromised. Base64 encoding is _not_ an - encryption method and is considered the same as plain text. - - Applications still need to protect the value of secret after reading it from the volume, - such as not accidentally logging it or transmitting it to an untrusted party. - - A user who can create a Pod that uses a secret can also see the value of that secret. Even - if the API server policy does not allow that user to read the Secret, the user could - run a Pod which exposes the secret. +- Reserve the ability to `watch` or `list` all secrets in a cluster (using the Kubernetes + API), so that only the most privileged, system-level components can perform this action. +- When deploying applications that interact with the Secret API, you should + limit access using + [authorization policies](/docs/reference/access-authn-authz/authorization/) such as + [RBAC](/docs/reference/access-authn-authz/rbac/). +- In the API server, objects (including Secrets) are persisted into + {{< glossary_tooltip term_id="etcd" >}}; therefore: + - only allow cluster admistrators to access etcd (this includes read-only access); + - enable [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/) + for Secret objects, so that the data of these Secrets are not stored in the clear + into {{< glossary_tooltip term_id="etcd" >}}; + - consider wiping / shredding the durable storage used by etcd once it is + no longer in use; + - if there are multiple etcd instances, make sure that etcd is + using SSL/TLS for communication between etcd peers. ## {{% heading "whatsnext" %}} -- Learn how to [manage Secret using `kubectl`](/docs/tasks/configmap-secret/managing-secret-using-kubectl/) -- Learn how to [manage Secret using config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/) -- Learn how to [manage Secret using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/) +- Learn how to [manage Secrets using `kubectl`](/docs/tasks/configmap-secret/managing-secret-using-kubectl/) +- Learn how to [manage Secrets using config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/) +- Learn how to [manage Secrets using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/) - Read the [API reference](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/) for `Secret` diff --git a/content/en/docs/concepts/configuration/windows-resource-management.md b/content/en/docs/concepts/configuration/windows-resource-management.md new file mode 100644 index 0000000000..955fea194a --- /dev/null +++ b/content/en/docs/concepts/configuration/windows-resource-management.md @@ -0,0 +1,79 @@ +--- +reviewers: +- jayunit100 +- jsturtevant +- marosset +- perithompson +title: Resource Management for Windows nodes +content_type: concept +weight: 75 +--- + +<!-- overview --> + +This page outlines the differences in how resources are managed between Linux and Windows. + +<!-- body --> + +On Linux nodes, {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} are used +as a pod boundary for resource control. Containers are created within that boundary +for network, process and file system isolation. The Linux cgroup APIs can be used to +gather CPU, I/O, and memory use statistics. + +In contrast, Windows uses a [_job object_](https://docs.microsoft.com/windows/win32/procthread/job-objects) per container with a system namespace filter +to contain all processes in a container and provide logical isolation from the +host. +(Job objects are a Windows process isolation mechanism and are different from +what Kubernetes refers to as a {{< glossary_tooltip term_id="job" text="Job" >}}). + +There is no way to run a Windows container without the namespace filtering in +place. This means that system privileges cannot be asserted in the context of the +host, and thus privileged containers are not available on Windows. +Containers cannot assume an identity from the host because the Security Account Manager +(SAM) is separate. + +## Memory management {#resource-management-memory} + +Windows does not have an out-of-memory process killer as Linux does. Windows always +treats all user-mode memory allocations as virtual, and pagefiles are mandatory. + +Windows nodes do not overcommit memory for processes. The +net effect is that Windows won't reach out of memory conditions the same way Linux +does, and processes page to disk instead of being subject to out of memory (OOM) +termination. If memory is over-provisioned and all physical memory is exhausted, +then paging can slow down performance. + +## CPU management {#resource-management-cpu} + +Windows can limit the amount of CPU time allocated for different processes but cannot +guarantee a minimum amount of CPU time. + +On Windows, the kubelet supports a command-line flag to set the +[scheduling priority](https://docs.microsoft.com/windows/win32/procthread/scheduling-priorities) of the +kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get +more CPU time slices when compared to other processes running on the Windows host. +More information on the allowable values and their meaning is available at +[Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). +To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above. + +## Resource reservation {#resource-reservation} + +To account for memory and CPU used by the operating system, the container runtime, and by +Kubernetes host processes such as the kubelet, you can (and should) reserve +memory and CPU resources with the `--kube-reserved` and/or `--system-reserved` kubelet flags. +On Windows these values are only used to calculate the node's +[allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) resources. + +{{< caution >}} +As you deploy workloads, set resource memory and CPU limits on containers. +This also subtracts from `NodeAllocatable` and helps the cluster-wide scheduler in determining which pods to place on which nodes. + +Scheduling pods without limits may over-provision the Windows nodes and in extreme +cases can cause the nodes to become unhealthy. +{{< /caution >}} + +On Windows, a good practice is to reserve at least 2GiB of memory. + +To determine how much CPU to reserve, +identify the maximum pod density for each node and monitor the CPU usage of +the system services running there, then choose a value that meets your workload needs. diff --git a/content/en/docs/concepts/containers/container-environment.md b/content/en/docs/concepts/containers/container-environment.md index 3c4c153927..f773178b54 100644 --- a/content/en/docs/concepts/containers/container-environment.md +++ b/content/en/docs/concepts/containers/container-environment.md @@ -35,13 +35,12 @@ The Pod name and namespace are available as environment variables through the [downward API](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/). User defined environment variables from the Pod definition are also available to the Container, -as are any environment variables specified statically in the Docker image. +as are any environment variables specified statically in the container image. ### Cluster information A list of all services that were running when a Container was created is available to that Container as environment variables. This list is limited to services within the same namespace as the new Container's Pod and Kubernetes control plane services. -Those environment variables match the syntax of Docker links. For a service named *foo* that maps to a Container named *bar*, the following variables are defined: diff --git a/content/en/docs/concepts/containers/container-lifecycle-hooks.md b/content/en/docs/concepts/containers/container-lifecycle-hooks.md index 96569f9518..cb953eecbc 100644 --- a/content/en/docs/concepts/containers/container-lifecycle-hooks.md +++ b/content/en/docs/concepts/containers/container-lifecycle-hooks.md @@ -59,7 +59,7 @@ Resources consumed by the command are counted against the Container. ### Hook handler execution When a Container lifecycle management hook is called, -the Kubernetes management system execute the handler according to the hook action, +the Kubernetes management system executes the handler according to the hook action, `httpGet` and `tcpSocket` are executed by the kubelet process, and `exec` is executed in the container. Hook handler calls are synchronous within the context of the Pod containing the Container. @@ -105,22 +105,22 @@ The logs for a Hook handler are not exposed in Pod events. If a handler fails for some reason, it broadcasts an event. For `PostStart`, this is the `FailedPostStartHook` event, and for `PreStop`, this is the `FailedPreStopHook` event. -You can see these events by running `kubectl describe pod <pod_name>`. -Here is some example output of events from running this command: +To generate a failed `FailedPreStopHook` event yourself, modify the [lifecycle-events.yaml](https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/lifecycle-events.yaml) file to change the postStart command to "badcommand" and apply it. +Here is some example output of the resulting events you see from running `kubectl describe pod lifecycle-demo`: ``` Events: - FirstSeen LastSeen Count From SubObjectPath Type Reason Message - --------- -------- ----- ---- ------------- -------- ------ ------- - 1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned test-1730497541-cq1d2 to gke-test-cluster-default-pool-a07e5d30-siqd - 1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulling pulling image "test:1.0" - 1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Created Created container with docker id 5c6a256a2567; Security:[seccomp=unconfined] - 1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulled Successfully pulled image "test:1.0" - 1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Started Started container with docker id 5c6a256a2567 - 38s 38s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 5c6a256a2567: PostStart handler: Error executing in Docker Container: 1 - 37s 37s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 8df9fdfd7054: PostStart handler: Error executing in Docker Container: 1 - 38s 37s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "main" with RunContainerError: "PostStart handler: Error executing in Docker Container: 1" - 1m 22s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Warning FailedPostStartHook + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 7s default-scheduler Successfully assigned default/lifecycle-demo to ip-XXX-XXX-XX-XX.us-east-2... + Normal Pulled 6s kubelet Successfully pulled image "nginx" in 229.604315ms + Normal Pulling 4s (x2 over 6s) kubelet Pulling image "nginx" + Normal Created 4s (x2 over 5s) kubelet Created container lifecycle-demo-container + Normal Started 4s (x2 over 5s) kubelet Started container lifecycle-demo-container + Warning FailedPostStartHook 4s (x2 over 5s) kubelet Exec lifecycle hook ([badcommand]) for Container "lifecycle-demo-container" in Pod "lifecycle-demo_default(30229739-9651-4e5a-9a32-a8f1688862db)" failed - error: command 'badcommand' exited with 126: , message: "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: \"badcommand\": executable file not found in $PATH: unknown\r\n" + Normal Killing 4s (x2 over 5s) kubelet FailedPostStartHook + Normal Pulled 4s kubelet Successfully pulled image "nginx" in 215.66395ms + Warning BackOff 2s (x2 over 3s) kubelet Back-off restarting failed container ``` diff --git a/content/en/docs/concepts/containers/images.md b/content/en/docs/concepts/containers/images.md index 9300561e46..c07880a458 100644 --- a/content/en/docs/concepts/containers/images.md +++ b/content/en/docs/concepts/containers/images.md @@ -29,8 +29,7 @@ and possibly a port number as well; for example: `fictional.registry.example:104 If you don't specify a registry hostname, Kubernetes assumes that you mean the Docker public registry. -After the image name part you can add a _tag_ (as also using with commands such -as `docker` and `podman`). +After the image name part you can add a _tag_ (in the same way you would when using with commands like `docker` or `podman`). Tags let you identify different versions of the same series of images. Image tags consist of lowercase and uppercase letters, digits, underscores (`_`), @@ -91,7 +90,7 @@ the image's digest; replace `<image-name>:<tag>` with `<image-name>@<digest>` (for example, `image@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2`). -When using image tags, if the image registry were to change the code that the tag on that image represents, you might end up with a mix of Pods running the old and new code. An image digest uniquely identifies a specific version of the image, so Kubernetes runs the same code every time it starts a container with that image name and digest specified. Specifying an image fixes the code that you run so that a change at the registry cannot lead to that mix of versions. +When using image tags, if the image registry were to change the code that the tag on that image represents, you might end up with a mix of Pods running the old and new code. An image digest uniquely identifies a specific version of the image, so Kubernetes runs the same code every time it starts a container with that image name and digest specified. Specifying an image by digest fixes the code that you run so that a change at the registry cannot lead to that mix of versions. There are third-party [admission controllers](/docs/reference/access-authn-authz/admission-controllers/) that mutate Pods (and pod templates) when they are created, so that the @@ -108,7 +107,7 @@ When you (or a controller) submit a new Pod to the API server, your cluster sets `:latest`, `imagePullPolicy` is automatically set to `Always`; - if you omit the `imagePullPolicy` field, and you don't specify the tag for the container image, `imagePullPolicy` is automatically set to `Always`; -- if you omit the `imagePullPolicy` field, and you don't specify the tag for the +- if you omit the `imagePullPolicy` field, and you specify the tag for the container image that isn't `:latest`, the `imagePullPolicy` is automatically set to `IfNotPresent`. @@ -175,95 +174,79 @@ These options are explained in more detail below. ### Configuring nodes to authenticate to a private registry -If you run Docker on your nodes, you can configure the Docker container -runtime to authenticate to a private container registry. +Specific instructions for setting credentials depends on the container runtime and registry you chose to use. You should refer to your solution's documentation for the most accurate information. -This approach is suitable if you can control node configuration. +For an example of configuring a private container image registry, see the +[Pull an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry) +task. That example uses a private registry in Docker Hub. -{{< note >}} -Default Kubernetes only supports the `auths` and `HttpHeaders` section in Docker configuration. -Docker credential helpers (`credHelpers` or `credsStore`) are not supported. -{{< /note >}} +### Interpretation of config.json {#config-json} +The interpretation of `config.json` varies between the original Docker +implementation and the Kubernetes interpretation. In Docker, the `auths` keys +can only specify root URLs, whereas Kubernetes allows glob URLs as well as +prefix-matched paths. This means that a `config.json` like this is valid: -Docker stores keys for private registries in the `$HOME/.dockercfg` or `$HOME/.docker/config.json` file. If you put the same file -in the search paths list below, kubelet uses it as the credential provider when pulling images. - -* `{--root-dir:-/var/lib/kubelet}/config.json` -* `{cwd of kubelet}/config.json` -* `${HOME}/.docker/config.json` -* `/.docker/config.json` -* `{--root-dir:-/var/lib/kubelet}/.dockercfg` -* `{cwd of kubelet}/.dockercfg` -* `${HOME}/.dockercfg` -* `/.dockercfg` - -{{< note >}} -You may have to set `HOME=/root` explicitly in the environment of the kubelet process. -{{< /note >}} - -Here are the recommended steps to configuring your nodes to use a private registry. In this -example, run these on your desktop/laptop: - - 1. Run `docker login [server]` for each set of credentials you want to use. This updates `$HOME/.docker/config.json` on your PC. - 1. View `$HOME/.docker/config.json` in an editor to ensure it contains only the credentials you want to use. - 1. Get a list of your nodes; for example: - - if you want the names: `nodes=$( kubectl get nodes -o jsonpath='{range.items[*].metadata}{.name} {end}' )` - - if you want to get the IP addresses: `nodes=$( kubectl get nodes -o jsonpath='{range .items[*].status.addresses[?(@.type=="ExternalIP")]}{.address} {end}' )` - 1. Copy your local `.docker/config.json` to one of the search paths list above. - - for example, to test this out: `for n in $nodes; do scp ~/.docker/config.json root@"$n":/var/lib/kubelet/config.json; done` - -{{< note >}} -For production clusters, use a configuration management tool so that you can apply this -setting to all the nodes where you need it. -{{< /note >}} - -Verify by creating a Pod that uses a private image; for example: - -```shell -kubectl apply -f - <<EOF -apiVersion: v1 -kind: Pod -metadata: - name: private-image-test-1 -spec: - containers: - - name: uses-private-image - image: $PRIVATE_IMAGE_NAME - imagePullPolicy: Always - command: [ "echo", "SUCCESS" ] -EOF -``` -``` -pod/private-image-test-1 created +```json +{ + "auths": { + "*my-registry.io/images": { + "auth": "…" + } + } +} ``` -If everything is working, then, after a few moments, you can run: +The root URL (`*my-registry.io`) is matched by using the following syntax: -```shell -kubectl logs private-image-test-1 ``` -and see that the command outputs: -``` -SUCCESS +pattern: + { term } + +term: + '*' matches any sequence of non-Separator characters + '?' matches any single non-Separator character + '[' [ '^' ] { character-range } ']' + character class (must be non-empty) + c matches character c (c != '*', '?', '\\', '[') + '\\' c matches character c + +character-range: + c matches character c (c != '\\', '-', ']') + '\\' c matches character c + lo '-' hi matches character c for lo <= c <= hi ``` -If you suspect that the command failed, you can run: -```shell -kubectl describe pods/private-image-test-1 | grep 'Failed' -``` -In case of failure, the output is similar to: -``` - Fri, 26 Jun 2015 15:36:13 -0700 Fri, 26 Jun 2015 15:39:13 -0700 19 {kubelet node-i2hq} spec.containers{uses-private-image} failed Failed to pull image "user/privaterepo:v1": Error: image user/privaterepo:v1 not found +Image pull operations would now pass the credentials to the CRI container +runtime for every valid pattern. For example the following container image names +would match successfully: + +- `my-registry.io/images` +- `my-registry.io/images/my-image` +- `my-registry.io/images/another-image` +- `sub.my-registry.io/images/my-image` +- `a.sub.my-registry.io/images/my-image` + +The kubelet performs image pulls sequentially for every found credential. This +means, that multiple entries in `config.json` are possible, too: + +```json +{ + "auths": { + "my-registry.io/images": { + "auth": "…" + }, + "my-registry.io/images/subpath": { + "auth": "…" + } + } +} ``` +If now a container specifies an image `my-registry.io/images/subpath/my-image` +to be pulled, then the kubelet will try to download them from both +authentication sources if one of them fails. -You must ensure all nodes in the cluster have the same `.docker/config.json`. Otherwise, pods will run on -some nodes and fail to run on others. For example, if you use node autoscaling, then each instance -template needs to include the `.docker/config.json` or mount a drive that contains it. - -All pods will have read access to images in any private registry once private -registry keys are added to the `.docker/config.json`. ### Pre-pulled images @@ -295,6 +278,8 @@ Kubernetes supports specifying container image registry keys on a Pod. #### Creating a Secret with a Docker config +You need to know the username, registry password and client email address for authenticating +to the registry, as well as its hostname. Run the following command, substituting the appropriate uppercase values: ```shell @@ -359,14 +344,13 @@ There are a number of solutions for configuring private registries. Here are so common use cases and suggested solutions. 1. Cluster running only non-proprietary (e.g. open-source) images. No need to hide images. - - Use public images on the Docker hub. + - Use public images from a public registry - No configuration required. - Some cloud providers automatically cache or mirror public images, which improves availability and reduces the time to pull images. 1. Cluster running some proprietary images which should be hidden to those outside the company, but visible to all cluster users. - - Use a hosted private [Docker registry](https://docs.docker.com/registry/). - - It may be hosted on the [Docker Hub](https://hub.docker.com/signup), or elsewhere. - - Manually configure .docker/config.json on each node as described above. + - Use a hosted private registry + - Manual configuration may be required on the nodes that need to access to private registry - Or, run an internal private registry behind your firewall with open read access. - No Kubernetes configuration is required. - Use a hosted container image registry service that controls image access @@ -383,10 +367,9 @@ common use cases and suggested solutions. If you need access to multiple registries, you can create one secret for each registry. -Kubelet will merge any `imagePullSecrets` into a single virtual `.docker/config.json` - ## {{% heading "whatsnext" %}} * Read the [OCI Image Manifest Specification](https://github.com/opencontainers/image-spec/blob/master/manifest.md). * Learn about [container image garbage collection](/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection). +* Learn more about [pulling an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry). diff --git a/content/en/docs/concepts/containers/runtime-class.md b/content/en/docs/concepts/containers/runtime-class.md index 96858d32af..cd8f74aa29 100644 --- a/content/en/docs/concepts/containers/runtime-class.md +++ b/content/en/docs/concepts/containers/runtime-class.md @@ -1,7 +1,7 @@ --- reviewers: -- tallclair -- dchen1107 + - tallclair + - dchen1107 title: Runtime Class content_type: concept weight: 20 @@ -16,9 +16,6 @@ This page describes the RuntimeClass resource and runtime selection mechanism. RuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod's containers. - - - <!-- body --> ## Motivation @@ -62,12 +59,15 @@ The RuntimeClass resource currently only has 2 significant fields: the RuntimeCl (`metadata.name`) and the handler (`handler`). The object definition looks like this: ```yaml -apiVersion: node.k8s.io/v1 # RuntimeClass is defined in the node.k8s.io API group +# RuntimeClass is defined in the node.k8s.io API group +apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: - name: myclass # The name the RuntimeClass will be referenced by - # RuntimeClass is a non-namespaced resource -handler: myconfiguration # The name of the corresponding CRI configuration + # The name the RuntimeClass will be referenced by. + # RuntimeClass is a non-namespaced resource. + name: myclass +# The name of the corresponding CRI configuration +handler: myconfiguration ``` The name of a RuntimeClass object must be a valid @@ -75,14 +75,14 @@ The name of a RuntimeClass object must be a valid {{< note >}} It is recommended that RuntimeClass write operations (create/update/patch/delete) be -restricted to the cluster administrator. This is typically the default. See [Authorization -Overview](/docs/reference/access-authn-authz/authorization/) for more details. +restricted to the cluster administrator. This is typically the default. See +[Authorization Overview](/docs/reference/access-authn-authz/authorization/) for more details. {{< /note >}} ## Usage -Once RuntimeClasses are configured for the cluster, using them is very simple. Specify a -`runtimeClassName` in the Pod spec. For example: +Once RuntimeClasses are configured for the cluster, you can specify a +`runtimeClassName` in the Pod spec to use it. For example: ```yaml apiVersion: v1 @@ -97,7 +97,7 @@ spec: This will instruct the kubelet to use the named RuntimeClass to run this pod. If the named RuntimeClass does not exist, or the CRI cannot run the corresponding handler, the pod will enter the `Failed` terminal [phase](/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase). Look for a -corresponding [event](/docs/tasks/debug-application-cluster/debug-application-introspection/) for an +corresponding [event](/docs/tasks/debug/debug-application/debug-running-pod/) for an error message. If no `runtimeClassName` is specified, the default RuntimeHandler will be used, which is equivalent @@ -107,11 +107,6 @@ to the behavior when the RuntimeClass feature is disabled. For more details on setting up CRI runtimes, see [CRI installation](/docs/setup/production-environment/container-runtimes/). -#### dockershim - -RuntimeClasses with dockershim must set the runtime handler to `docker`. Dockershim does not support -custom configurable runtime handlers. - #### {{< glossary_tooltip term_id="containerd" >}} Runtime handlers are configured through containerd's configuration at @@ -121,14 +116,14 @@ Runtime handlers are configured through containerd's configuration at [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}] ``` -See containerd's config documentation for more details: -https://github.com/containerd/cri/blob/master/docs/config.md +See containerd's [config documentation](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) +for more details: #### {{< glossary_tooltip term_id="cri-o" >}} Runtime handlers are configured through CRI-O's configuration at `/etc/crio/crio.conf`. Valid -handlers are configured under the [crio.runtime -table](https://github.com/cri-o/cri-o/blob/master/docs/crio.conf.5.md#crioruntime-table): +handlers are configured under the +[crio.runtime table](https://github.com/cri-o/cri-o/blob/master/docs/crio.conf.5.md#crioruntime-table): ``` [crio.runtime.runtimes.${HANDLER_NAME}] @@ -156,27 +151,24 @@ can add `tolerations` to the RuntimeClass. As with the `nodeSelector`, the toler with the pod's tolerations in admission, effectively taking the union of the set of nodes tolerated by each. -To learn more about configuring the node selector and tolerations, see [Assigning Pods to -Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/). +To learn more about configuring the node selector and tolerations, see +[Assigning Pods to Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/). ### Pod Overhead -{{< feature-state for_k8s_version="v1.18" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} You can specify _overhead_ resources that are associated with running a Pod. Declaring overhead allows the cluster (including the scheduler) to account for it when making decisions about Pods and resources. -To use Pod overhead, you must have the PodOverhead [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -enabled (it is on by default). -Pod overhead is defined in RuntimeClass through the `overhead` fields. Through the use of these fields, +Pod overhead is defined in RuntimeClass through the `overhead` field. Through the use of this field, you can specify the overhead of running pods utilizing this RuntimeClass and ensure these overheads are accounted for in Kubernetes. - ## {{% heading "whatsnext" %}} - - [RuntimeClass Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/585-runtime-class/README.md) - [RuntimeClass Scheduling Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/585-runtime-class/README.md#runtimeclass-scheduling) - Read about the [Pod Overhead](/docs/concepts/scheduling-eviction/pod-overhead/) concept - [PodOverhead Feature Design](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/688-pod-overhead) + diff --git a/content/en/docs/concepts/extend-kubernetes/_index.md b/content/en/docs/concepts/extend-kubernetes/_index.md index 083dca6964..5404f8c463 100644 --- a/content/en/docs/concepts/extend-kubernetes/_index.md +++ b/content/en/docs/concepts/extend-kubernetes/_index.md @@ -17,44 +17,61 @@ no_list: true <!-- overview --> -Kubernetes is highly configurable and extensible. As a result, -there is rarely a need to fork or submit patches to the Kubernetes -project code. +Kubernetes is highly configurable and extensible. As a result, there is rarely a need to fork or +submit patches to the Kubernetes project code. -This guide describes the options for customizing a Kubernetes -cluster. It is aimed at {{< glossary_tooltip text="cluster operators" term_id="cluster-operator" >}} who want to -understand how to adapt their Kubernetes cluster to the needs of -their work environment. Developers who are prospective {{< glossary_tooltip text="Platform Developers" term_id="platform-developer" >}} or Kubernetes Project {{< glossary_tooltip text="Contributors" term_id="contributor" >}} will also find it -useful as an introduction to what extension points and patterns -exist, and their trade-offs and limitations. +This guide describes the options for customizing a Kubernetes cluster. It is aimed at +{{< glossary_tooltip text="cluster operators" term_id="cluster-operator" >}} who want to understand +how to adapt their Kubernetes cluster to the needs of their work environment. Developers who are +prospective {{< glossary_tooltip text="Platform Developers" term_id="platform-developer" >}} or +Kubernetes Project {{< glossary_tooltip text="Contributors" term_id="contributor" >}} will also +find it useful as an introduction to what extension points and patterns exist, and their +trade-offs and limitations. <!-- body --> ## Overview -Customization approaches can be broadly divided into *configuration*, which only involves changing flags, local configuration files, or API resources; and *extensions*, which involve running additional programs or services. This document is primarily about extensions. +Customization approaches can be broadly divided into *configuration*, which only involves changing +flags, local configuration files, or API resources; and *extensions*, which involve running +additional programs or services. This document is primarily about extensions. ## Configuration -*Configuration files* and *flags* are documented in the Reference section of the online documentation, under each binary: +*Configuration files* and *flags* are documented in the Reference section of the online +documentation, under each binary: * [kubelet](/docs/reference/command-line-tools-reference/kubelet/) +* [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/) * [kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/) * [kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/) * [kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/). -Flags and configuration files may not always be changeable in a hosted Kubernetes service or a distribution with managed installation. When they are changeable, they are usually only changeable by the cluster administrator. Also, they are subject to change in future Kubernetes versions, and setting them may require restarting processes. For those reasons, they should be used only when there are no other options. +Flags and configuration files may not always be changeable in a hosted Kubernetes service or a +distribution with managed installation. When they are changeable, they are usually only changeable +by the cluster administrator. Also, they are subject to change in future Kubernetes versions, and +setting them may require restarting processes. For those reasons, they should be used only when +there are no other options. -*Built-in Policy APIs*, such as [ResourceQuota](/docs/concepts/policy/resource-quotas/), [PodSecurityPolicies](/docs/concepts/policy/pod-security-policy/), [NetworkPolicy](/docs/concepts/services-networking/network-policies/) and Role-based Access Control ([RBAC](/docs/reference/access-authn-authz/rbac/)), are built-in Kubernetes APIs. APIs are typically used with hosted Kubernetes services and with managed Kubernetes installations. They are declarative and use the same conventions as other Kubernetes resources like pods, so new cluster configuration can be repeatable and be managed the same way as applications. And, where they are stable, they enjoy a [defined support policy](/docs/reference/using-api/deprecation-policy/) like other Kubernetes APIs. For these reasons, they are preferred over *configuration files* and *flags* where suitable. +*Built-in Policy APIs*, such as [ResourceQuota](/docs/concepts/policy/resource-quotas/), +[PodSecurityPolicies](/docs/concepts/security/pod-security-policy/), +[NetworkPolicy](/docs/concepts/services-networking/network-policies/) and Role-based Access Control +([RBAC](/docs/reference/access-authn-authz/rbac/)), are built-in Kubernetes APIs. +APIs are typically used with hosted Kubernetes services and with managed Kubernetes installations. +They are declarative and use the same conventions as other Kubernetes resources like pods, +so new cluster configuration can be repeatable and be managed the same way as applications. +And, where they are stable, they enjoy a +[defined support policy](/docs/reference/using-api/deprecation-policy/) like other Kubernetes APIs. +For these reasons, they are preferred over *configuration files* and *flags* where suitable. ## Extensions Extensions are software components that extend and deeply integrate with Kubernetes. They adapt it to support new types and new kinds of hardware. -Most cluster administrators will use a hosted or distribution -instance of Kubernetes. As a result, most Kubernetes users will not need to -install extensions and fewer will need to author new ones. +Many cluster administrators use a hosted or distribution instance of Kubernetes. +These clusters come with extensions pre-installed. As a result, most Kubernetes +users will not need to install extensions and even fewer users will need to author new ones. ## Extension Patterns @@ -69,15 +86,14 @@ There is a specific pattern for writing client programs that work well with Kubernetes called the *Controller* pattern. Controllers typically read an object's `.spec`, possibly do things, and then update the object's `.status`. -A controller is a client of Kubernetes. When Kubernetes is the client and -calls out to a remote service, it is called a *Webhook*. The remote service -is called a *Webhook Backend*. Like Controllers, Webhooks do add a point of -failure. +A controller is a client of Kubernetes. When Kubernetes is the client and calls out to a remote +service, it is called a *Webhook*. The remote service is called a *Webhook Backend*. Like +Controllers, Webhooks do add a point of failure. In the webhook model, Kubernetes makes a network request to a remote service. In the *Binary Plugin* model, Kubernetes executes a binary (program). Binary plugins are used by the kubelet (e.g. -[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexVolume) +[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexvolume) and [Network Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)) and by kubectl. @@ -94,15 +110,35 @@ This diagram shows the extension points in a Kubernetes system. <!-- image source diagrams: https://docs.google.com/drawings/d/1k2YdJgNTtNfW7_A8moIIkij-DmVgEhNrn3y2OODwqQQ/view --> ![Extension Points](/docs/concepts/extend-kubernetes/extension-points.png) -1. Users often interact with the Kubernetes API using `kubectl`. [Kubectl plugins](/docs/tasks/extend-kubectl/kubectl-plugins/) extend the kubectl binary. They only affect the individual user's local environment, and so cannot enforce site-wide policies. -2. The apiserver handles all requests. Several types of extension points in the apiserver allow authenticating requests, or blocking them based on their content, editing content, and handling deletion. These are described in the [API Access Extensions](#api-access-extensions) section. -3. The apiserver serves various kinds of *resources*. *Built-in resource kinds*, like `pods`, are defined by the Kubernetes project and can't be changed. You can also add resources that you define, or that other projects have defined, called *Custom Resources*, as explained in the [Custom Resources](#user-defined-types) section. Custom Resources are often used with API Access Extensions. -4. The Kubernetes scheduler decides which nodes to place pods on. There are several ways to extend scheduling. These are described in the [Scheduler Extensions](#scheduler-extensions) section. -5. Much of the behavior of Kubernetes is implemented by programs called Controllers which are clients of the API-Server. Controllers are often used in conjunction with Custom Resources. -6. The kubelet runs on servers, and helps pods appear like virtual servers with their own IPs on the cluster network. [Network Plugins](#network-plugins) allow for different implementations of pod networking. -7. The kubelet also mounts and unmounts volumes for containers. New types of storage can be supported via [Storage Plugins](#storage-plugins). +1. Users often interact with the Kubernetes API using `kubectl`. + [Kubectl plugins](/docs/tasks/extend-kubectl/kubectl-plugins/) extend the kubectl binary. + They only affect the individual user's local environment, and so cannot enforce site-wide policies. -If you are unsure where to start, this flowchart can help. Note that some solutions may involve several types of extensions. +1. The API server handles all requests. Several types of extension points in the API server allow + authenticating requests, or blocking them based on their content, editing content, and handling + deletion. These are described in the [API Access Extensions](#api-access-extensions) section. + +1. The API server serves various kinds of *resources*. *Built-in resource kinds*, like `pods`, are + defined by the Kubernetes project and can't be changed. You can also add resources that you + define, or that other projects have defined, called *Custom Resources*, as explained in the + [Custom Resources](#user-defined-types) section. Custom Resources are often used with API access + extensions. + +1. The Kubernetes scheduler decides which nodes to place pods on. There are several ways to extend + scheduling. These are described in the [Scheduler Extensions](#scheduler-extensions) section. + +1. Much of the behavior of Kubernetes is implemented by programs called Controllers which are + clients of the API server. Controllers are often used in conjunction with Custom Resources. + +1. The kubelet runs on servers, and helps pods appear like virtual servers with their own IPs on + the cluster network. [Network Plugins](#network-plugins) allow for different implementations of + pod networking. + +1. The kubelet also mounts and unmounts volumes for containers. New types of storage can be + supported via [Storage Plugins](#storage-plugins). + +If you are unsure where to start, this flowchart can help. Note that some solutions may involve +several types of extensions. <!-- image source drawing: https://docs.google.com/drawings/d/1sdviU6lDz4BpnzJNHfNpQrqI9F19QZ07KnhnxVrp2yg/edit --> ![Flowchart for Extension](/docs/concepts/extend-kubernetes/flowchart.png) @@ -111,58 +147,86 @@ If you are unsure where to start, this flowchart can help. Note that some soluti ### User-Defined Types -Consider adding a Custom Resource to Kubernetes if you want to define new controllers, application configuration objects or other declarative APIs, and to manage them using Kubernetes tools, such as `kubectl`. +Consider adding a Custom Resource to Kubernetes if you want to define new controllers, application +configuration objects or other declarative APIs, and to manage them using Kubernetes tools, such +as `kubectl`. Do not use a Custom Resource as data storage for application, user, or monitoring data. -For more about Custom Resources, see the [Custom Resources concept guide](/docs/concepts/extend-kubernetes/api-extension/custom-resources/). +For more about Custom Resources, see the +[Custom Resources concept guide](/docs/concepts/extend-kubernetes/api-extension/custom-resources/). ### Combining New APIs with Automation -The combination of a custom resource API and a control loop is called the [Operator pattern](/docs/concepts/extend-kubernetes/operator/). The Operator pattern is used to manage specific, usually stateful, applications. These custom APIs and control loops can also be used to control other resources, such as storage or policies. +The combination of a custom resource API and a control loop is called the +[Operator pattern](/docs/concepts/extend-kubernetes/operator/). The Operator pattern is used to manage +specific, usually stateful, applications. These custom APIs and control loops can also be used to +control other resources, such as storage or policies. ### Changing Built-in Resources -When you extend the Kubernetes API by adding custom resources, the added resources always fall into a new API Groups. You cannot replace or change existing API groups. -Adding an API does not directly let you affect the behavior of existing APIs (e.g. Pods), but API Access Extensions do. +When you extend the Kubernetes API by adding custom resources, the added resources always fall +into a new API Groups. You cannot replace or change existing API groups. +Adding an API does not directly let you affect the behavior of existing APIs (e.g. Pods), but API +Access Extensions do. ### API Access Extensions -When a request reaches the Kubernetes API Server, it is first Authenticated, then Authorized, then subject to various types of Admission Control. See [Controlling Access to the Kubernetes API](/docs/concepts/security/controlling-access/) for more on this flow. +When a request reaches the Kubernetes API Server, it is first Authenticated, then Authorized, then +subject to various types of Admission Control. See +[Controlling Access to the Kubernetes API](/docs/concepts/security/controlling-access/) +for more on this flow. Each of these steps offers extension points. -Kubernetes has several built-in authentication methods that it supports. It can also sit behind an authenticating proxy, and it can send a token from an Authorization header to a remote service for verification (a webhook). All of these methods are covered in the [Authentication documentation](/docs/reference/access-authn-authz/authentication/). +Kubernetes has several built-in authentication methods that it supports. It can also sit behind an +authenticating proxy, and it can send a token from an Authorization header to a remote service for +verification (a webhook). All of these methods are covered in the +[Authentication documentation](/docs/reference/access-authn-authz/authentication/). ### Authentication -[Authentication](/docs/reference/access-authn-authz/authentication/) maps headers or certificates in all requests to a username for the client making the request. - -Kubernetes provides several built-in authentication methods, and an [Authentication webhook](/docs/reference/access-authn-authz/authentication/#webhook-token-authentication) method if those don't meet your needs. +[Authentication](/docs/reference/access-authn-authz/authentication/) maps headers or certificates +in all requests to a username for the client making the request. +Kubernetes provides several built-in authentication methods, and an +[Authentication webhook](/docs/reference/access-authn-authz/authentication/#webhook-token-authentication) +method if those don't meet your needs. ### Authorization -[Authorization](/docs/reference/access-authn-authz/webhook/) determines whether specific users can read, write, and do other operations on API resources. It works at the level of whole resources -- it doesn't discriminate based on arbitrary object fields. If the built-in authorization options don't meet your needs, and [Authorization webhook](/docs/reference/access-authn-authz/webhook/) allows calling out to user-provided code to make an authorization decision. - +[Authorization](/docs/reference/access-authn-authz/authorization/) determines whether specific +users can read, write, and do other operations on API resources. It works at the level of whole +resources -- it doesn't discriminate based on arbitrary object fields. If the built-in +authorization options don't meet your needs, [Authorization webhook](/docs/reference/access-authn-authz/webhook/) +allows calling out to user-provided code to make an authorization decision. ### Dynamic Admission Control -After a request is authorized, if it is a write operation, it also goes through [Admission Control](/docs/reference/access-authn-authz/admission-controllers/) steps. In addition to the built-in steps, there are several extensions: +After a request is authorized, if it is a write operation, it also goes through +[Admission Control](/docs/reference/access-authn-authz/admission-controllers/) steps. +In addition to the built-in steps, there are several extensions: -* The [Image Policy webhook](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook) restricts what images can be run in containers. -* To make arbitrary admission control decisions, a general [Admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) can be used. Admission Webhooks can reject creations or updates. +* The [Image Policy webhook](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook) + restricts what images can be run in containers. +* To make arbitrary admission control decisions, a general + [Admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) + can be used. Admission Webhooks can reject creations or updates. ## Infrastructure Extensions ### Storage Plugins -[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/flexvolume-deployment.md -) allow users to mount volume types without built-in support by having the -Kubelet call a Binary Plugin to mount the volume. +[Flex Volumes](https://git.k8s.io/design-proposals-archive/storage/flexvolume-deployment.md) +allow users to mount volume types without built-in support by having the kubelet call a binary +plugin to mount the volume. +FlexVolume is deprecated since Kubernetes v1.23. The out-of-tree CSI driver is the recommended way +to write volume drivers in Kubernetes. See +[Kubernetes Volume Plugin FAQ for Storage Vendors](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) +for more information. ### Device Plugins @@ -170,7 +234,6 @@ Device plugins allow a node to discover new Node resources (in addition to the builtin ones like cpu and memory) via a [Device Plugin](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/). - ### Network Plugins Different networking fabrics can be supported via node-level @@ -188,7 +251,7 @@ This is a significant undertaking, and almost all Kubernetes users find they do not need to modify the scheduler. The scheduler also supports a -[webhook](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md) +[webhook](https://git.k8s.io/design-proposals-archive/scheduling/scheduler_extender.md) that permits a webhook backend (scheduler extension) to filter and prioritize the nodes chosen for a pod. diff --git a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md index 3d72f279b6..258014a432 100644 --- a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md +++ b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md @@ -26,7 +26,7 @@ many core Kubernetes functions are now built using custom resources, making Kube Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using -[kubectl](/docs/reference/kubectl/overview/), just as they do for built-in resources like +[kubectl](/docs/reference/kubectl/), just as they do for built-in resources like *Pods*. ## Custom controllers @@ -35,11 +35,11 @@ On their own, custom resources let you store and retrieve structured data. When you combine a custom resource with a *custom controller*, custom resources provide a true _declarative API_. -A [declarative API](/docs/concepts/overview/kubernetes-api/) -allows you to _declare_ or specify the desired state of your resource and tries to -keep the current state of Kubernetes objects in sync with the desired state. -The controller interprets the structured data as a record of the user's -desired state, and continually maintains this state. +The Kubernetes [declarative API](/docs/concepts/overview/kubernetes-api/) +enforces a separation of responsibilities. You declare the desired state of +your resource. The Kubernetes controller keeps the current state of Kubernetes +objects in sync with your declared desired state. This is in contrast to an +imperative API, where you *instruct* a server what to do. You can deploy and update a custom controller on a running cluster, independently of the cluster's lifecycle. Custom controllers can work with any kind of resource, @@ -148,8 +148,8 @@ and use a controller to handle events. Usually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like *pods* and *services*, and can also generically handle custom resources through [CRDs](#customresourcedefinitions). The [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) allows you to provide specialized -implementations for your custom resources by writing and deploying your own standalone API server. -The main API server delegates requests to you for the custom resources that you handle, +implementations for your custom resources by writing and deploying your own API server. +The main API server delegates requests to your API server for the custom resources that you handle, making them available to all of its clients. ## Choosing a method for adding custom resources diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 868d8d56e8..6f265f57f3 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -1,7 +1,6 @@ --- -reviewers: title: Device Plugins -description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup. +description: Device plugins let you configure your cluster with support for devices or resources that require vendor-specific setup, such as GPUs, NICs, FPGAs, or non-volatile main memory. content_type: concept weight: 20 --- @@ -9,7 +8,7 @@ weight: 20 <!-- overview --> {{< feature-state for_k8s_version="v1.10" state="beta" >}} -Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md) +Kubernetes provides a [device plugin framework](https://git.k8s.io/design-proposals-archive/resource-management/device-plugin.md) that you can use to advertise system hardware resources to the {{< glossary_tooltip term_id="kubelet" >}}. @@ -48,12 +47,14 @@ For example, after a device plugin registers `hardware-vendor.example/foo` with and reports two healthy devices on a node, the node status is updated to advertise that the node has 2 "Foo" devices installed and available. -Then, users can request devices in a -[Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) -specification as they request other types of resources, with the following limitations: - +Then, users can request devices as part of a Pod specification +(see [`container`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container)). +Requesting extended resources is similar to how you manage requests and limits for +other resources, with the following differences: * Extended resources are only supported as integer resources and cannot be overcommitted. -* Devices cannot be shared among Containers. +* Devices cannot be shared between containers. + +### Example {#example-pod} Suppose a Kubernetes cluster is running a device plugin that advertises resource `hardware-vendor.example/foo` on certain nodes. Here is an example of a pod requesting this resource to run a demo workload: @@ -174,7 +175,7 @@ a Kubernetes release with a newer device plugin API version, upgrade your device to support both versions before upgrading these nodes. Taking that approach will ensure the continuous functioning of the device allocations during the upgrade. -## Monitoring Device Plugin Resources +## Monitoring device plugin resources {{< feature-state for_k8s_version="v1.15" state="beta" >}} @@ -197,6 +198,8 @@ service PodResourcesLister { } ``` +### `List` gRPC endpoint {#grpc-endpoint-list} + The `List` endpoint provides information on resources of running pods, with details such as the id of exclusively allocated CPUs, device id as it was reported by device plugins and id of the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container. @@ -246,10 +249,35 @@ message ContainerDevices { TopologyInfo topology = 3; } ``` +{{< note >}} +cpu_ids in the `ContainerResources` in the `List` endpoint correspond to exclusive CPUs allocated +to a partilar container. If the goal is to evaluate CPUs that belong to the shared pool, the `List` +endpoint needs to be used in conjunction with the `GetAllocatableResources` endpoint as explained +below: +1. Call `GetAllocatableResources` to get a list of all the allocatable CPUs +2. Call `GetCpuIds` on all `ContainerResources` in the system +3. Subtract out all of the CPUs from the `GetCpuIds` calls from the `GetAllocatableResources` call +{{< /note >}} + +### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources} + +{{< feature-state state="beta" for_k8s_version="v1.23" >}} GetAllocatableResources provides information on resources initially available on the worker node. It provides more information than kubelet exports to APIServer. +{{< note >}} +`GetAllocatableResources` should only be used to evaluate [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +resources on a node. If the goal is to evaluate free/unallocated resources it should be used in +conjunction with the List() endpoint. The result obtained by `GetAllocatableResources` would remain +the same unless the underlying resources exposed to kubelet change. This happens rarely but when +it does (for example: hotplug/hotunplug, device health changes), client is expected to call +`GetAlloctableResources` endpoint. +However, calling `GetAllocatableResources` endpoint is not sufficient in case of cpu and/or memory +update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable. +{{< /note >}} + + ```gRPC // AllocatableResourcesResponses contains informations about all the devices known by the kubelet message AllocatableResourcesResponse { @@ -259,6 +287,13 @@ message AllocatableResourcesResponse { } ``` +Starting from Kubernetes v1.23, the `GetAllocatableResources` is enabled by default. +You can disable it by turning off the +`KubeletPodResourcesGetAllocatable` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). + +Preceding Kubernetes v1.23, to enable this feature `kubelet` must be started with the following flag: + +`--feature-gates=KubeletPodResourcesGetAllocatable=true` `ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to what device @@ -276,7 +311,7 @@ DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a Support for the `PodResourcesLister service` requires `KubeletPodResources` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled. It is enabled by default starting with Kubernetes 1.15 and is v1 since Kubernetes 1.20. -## Device Plugin integration with the Topology Manager +## Device plugin integration with the Topology Manager {{< feature-state for_k8s_version="v1.18" state="beta" >}} @@ -285,7 +320,7 @@ The Topology Manager is a Kubelet component that allows resources to be co-ordin ```gRPC message TopologyInfo { - repeated NUMANode nodes = 1; + repeated NUMANode nodes = 1; } message NUMANode { @@ -304,15 +339,16 @@ pluginapi.Device{ID: "25102017", Health: pluginapi.Healthy, Topology:&pluginapi. ## Device plugin examples {#examples} +{{% thirdparty-content %}} + Here are some examples of device plugin implementations: * The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) -* The [Intel device plugins](https://github.com/intel/intel-device-plugins-for-kubernetes) for Intel GPU, FPGA and QuickAssist devices +* The [Intel device plugins](https://github.com/intel/intel-device-plugins-for-kubernetes) for Intel GPU, FPGA, QAT, VPU, SGX, DSA, DLB and IAA devices * The [KubeVirt device plugins](https://github.com/kubevirt/kubernetes-device-plugins) for hardware-assisted virtualization -* The [NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin) - * Requires [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) 2.0, which allows you to run GPU-enabled Docker containers. * The [NVIDIA GPU device plugin for Container-Optimized OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu) * The [RDMA device plugin](https://github.com/hustcat/k8s-rdma-device-plugin) +* The [SocketCAN device plugin](https://github.com/collabora/k8s-socketcan) * The [Solarflare device plugin](https://github.com/vikaschoudhary16/sfc-device-plugin) * The [SR-IOV Network device plugin](https://github.com/intel/sriov-network-device-plugin) * The [Xilinx FPGA device plugins](https://github.com/Xilinx/FPGA_as_a_Service/tree/master/k8s-fpga-device-plugin) for Xilinx FPGA devices @@ -323,5 +359,5 @@ Here are some examples of device plugin implementations: * Learn about [scheduling GPU resources](/docs/tasks/manage-gpus/scheduling-gpus/) using device plugins * Learn about [advertising extended resources](/docs/tasks/administer-cluster/extended-resource-node/) on a node -* Read about using [hardware acceleration for TLS ingress](https://kubernetes.io/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes * Learn about the [Topology Manager](/docs/tasks/administer-cluster/topology-manager/) +* Read about using [hardware acceleration for TLS ingress](/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md index 0ec8bf81b1..38d2c7afbc 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md @@ -11,36 +11,52 @@ weight: 10 <!-- overview --> -Network plugins in Kubernetes come in a few flavors: +Kubernetes {{< skew currentVersion >}} supports [Container Network Interface](https://github.com/containernetworking/cni) +(CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem. -* CNI plugins: adhere to the [Container Network Interface](https://github.com/containernetworking/cni) (CNI) specification, designed for interoperability. - * Kubernetes follows the [v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) release of the CNI specification. -* Kubenet plugin: implements basic `cbr0` using the `bridge` and `host-local` CNI plugins +A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). + +You must use a CNI plugin that is compatible with the +[v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) or later +releases of the CNI specification. The Kubernetes project recommends using a plugin that is +compatible with the [v1.0.0](https://github.com/containernetworking/cni/blob/spec-v1.0.0/SPEC.md) +CNI specification (plugins can be compatible with multiple spec versions). <!-- body --> ## Installation -The kubelet has a single default network plugin, and a default network common to the entire cluster. It probes for plugins when it starts up, remembers what it finds, and executes the selected plugin at appropriate times in the pod lifecycle (this is only true for Docker, as CRI manages its own CNI plugins). There are two Kubelet command line parameters to keep in mind when using plugins: +A Container Runtime, in the networking context, is a daemon on a node configured to provide CRI Services for kubelet. In particular, the Container Runtime must be configured to load the CNI plugins required to implement the Kubernetes network model. -* `cni-bin-dir`: Kubelet probes this directory for plugins on startup -* `network-plugin`: The network plugin to use from `cni-bin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is `cni`. +{{< note >}} +Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the `cni-bin-dir` and `network-plugin` command-line parameters. +These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet. + +See [Troubleshooting CNI plugin-related errors](/docs/tasks/administer-cluster/migrating-from-dockershim/troubleshooting-cni-plugin-related-errors/) +if you are facing issues following the removal of dockershim. +{{< /note >}} + +For specific information about how a Container Runtime manages the CNI plugins, see the documentation for that Container Runtime, for example: +- [containerd](https://github.com/containerd/containerd/blob/main/script/setup/install-cni) +- [CRI-O](https://github.com/cri-o/cri-o/blob/main/contrib/cni/README.md) + +For specific information about how to install and manage a CNI plugin, see the documentation for that plugin or [networking provider](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model). ## Network Plugin Requirements -Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/dockershim/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. +For plugin developers and users who regularly build or deploy Kubernetes, the plugin may also need specific configuration to support kube-proxy. +The iptables proxy depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. +For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. +If the plugin does not use a Linux bridge, but uses something like Open vSwitch or some other mechanism instead, it should ensure container traffic is appropriately routed for the proxy. -By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. +By default, if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. -### CNI +### Loopback CNI -The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`). +In addition to the CNI plugin installed on the nodes for implementing the Kubernetes network model, Kubernetes also requires the container runtimes to provide a loopback interface `lo`, which is used for each sandbox (pod sandboxes, vm sandboxes, ...). +Implementing the loopback interface can be accomplished by re-using the [CNI loopback plugin.](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) or by developing your own code to achieve this (see [this example from CRI-O](https://github.com/cri-o/ocicni/blob/release-1.24/pkg/ocicni/util_linux.go#L91)). -If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order. - -In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 - -#### Support hostPort +### Support hostPort The CNI networking plugin supports `hostPort`. You can use the official [portmap](https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap) plugin offered by the CNI plugin team or use your own plugin with portMapping functionality. @@ -51,7 +67,7 @@ For example: ```json { "name": "k8s-pod-network", - "cniVersion": "0.3.0", + "cniVersion": "0.4.0", "plugins": [ { "type": "calico", @@ -77,7 +93,7 @@ For example: } ``` -#### Support traffic shaping +### Support traffic shaping **Experimental Feature** @@ -90,7 +106,7 @@ If you want to enable traffic shaping support, you must add the `bandwidth` plug ```json { "name": "k8s-pod-network", - "cniVersion": "0.3.0", + "cniVersion": "0.4.0", "plugins": [ { "type": "calico", @@ -129,37 +145,4 @@ metadata: ... ``` -### kubenet - -Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments. - -Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host. - -The plugin requires a few things: - -* The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `cni-bin-dir` to supply additional search path. The first found match will take effect. -* Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin -* Kubelet should also be run with the `--non-masquerade-cidr=<clusterCidr>` argument to ensure traffic to IPs outside this range will use IP masquerade. -* The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=<cidr>` controller-manager command-line options. - -### Customizing the MTU (with kubenet) - -The MTU should always be configured correctly to get the best networking performance. Network plugins will usually try -to infer a sensible MTU, but sometimes the logic will not result in an optimal MTU. For example, if the -Docker bridge or another interface has a small MTU, kubenet will currently select that MTU. Or if you are -using IPSEC encapsulation, the MTU must be reduced, and this calculation is out-of-scope for -most network plugins. - -Where needed, you can specify the MTU explicitly with the `network-plugin-mtu` kubelet option. For example, -on AWS the `eth0` MTU is typically 9001, so you might specify `--network-plugin-mtu=9001`. If you're using IPSEC you -might reduce it to allow for encapsulation overhead; for example: `--network-plugin-mtu=8873`. - -This option is provided to the network-plugin; currently **only kubenet supports `network-plugin-mtu`**. - -## Usage Summary - -* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`). -* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge`, `lo` and `host-local` plugins placed in `/opt/cni/bin` or `cni-bin-dir`. -* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin. - ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/concepts/extend-kubernetes/operator.md b/content/en/docs/concepts/extend-kubernetes/operator.md index 72fe12f1e7..133814e98a 100644 --- a/content/en/docs/concepts/extend-kubernetes/operator.md +++ b/content/en/docs/concepts/extend-kubernetes/operator.md @@ -31,9 +31,7 @@ built-in automation from the core of Kubernetes. You can use Kubernetes to automate deploying and running workloads, *and* you can automate how Kubernetes does that. -Kubernetes' {{< glossary_tooltip text="controllers" term_id="controller" >}} -concept lets you extend the cluster's behaviour without modifying the code -of Kubernetes itself. +Kubernetes' {{< glossary_tooltip text="operator pattern" term_id="operator-pattern" >}} concept lets you extend the cluster's behaviour without modifying the code of Kubernetes itself by linking {{< glossary_tooltip text="controllers" term_id="controller" >}} to one or more custom resources. Operators are clients of the Kubernetes API that act as controllers for a [Custom Resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/). @@ -113,6 +111,9 @@ Operator. {{% thirdparty-content %}} * [Charmed Operator Framework](https://juju.is/) +* [Java Operator SDK](https://github.com/java-operator-sdk/java-operator-sdk) +* [Kopf](https://github.com/nolar/kopf) (Kubernetes Operator Pythonic Framework) +* [kube-rs](https://kube.rs/) (Rust) * [kubebuilder](https://book.kubebuilder.io/) * [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (.NET operator SDK) * [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator) diff --git a/content/en/docs/concepts/extend-kubernetes/service-catalog.md b/content/en/docs/concepts/extend-kubernetes/service-catalog.md deleted file mode 100644 index 26517de9c6..0000000000 --- a/content/en/docs/concepts/extend-kubernetes/service-catalog.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -title: Service Catalog -reviewers: -- chenopis -content_type: concept -weight: 40 ---- - -<!-- overview --> -{{< glossary_definition term_id="service-catalog" length="all" prepend="Service Catalog is" >}} - -A service broker, as defined by the [Open service broker API spec](https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md), is an endpoint for a set of managed services offered and maintained by a third-party, which could be a cloud provider such as AWS, GCP, or Azure. -Some examples of managed services are Microsoft Azure Cloud Queue, Amazon Simple Queue Service, and Google Cloud Pub/Sub, but they can be any software offering that can be used by an application. - -Using Service Catalog, a {{< glossary_tooltip text="cluster operator" term_id="cluster-operator" >}} can browse the list of managed services offered by a service broker, provision an instance of a managed service, and bind with it to make it available to an application in the Kubernetes cluster. - - - - -<!-- body --> -## Example use case - -An {{< glossary_tooltip text="application developer" term_id="application-developer" >}} wants to use message queuing as part of their application running in a Kubernetes cluster. -However, they do not want to deal with the overhead of setting such a service up and administering it themselves. -Fortunately, there is a cloud provider that offers message queuing as a managed service through its service broker. - -A cluster operator can setup Service Catalog and use it to communicate with the cloud provider's service broker to provision an instance of the message queuing service and make it available to the application within the Kubernetes cluster. -The application developer therefore does not need to be concerned with the implementation details or management of the message queue. -The application can access the message queue as a service. - -## Architecture - -Service Catalog uses the [Open service broker API](https://github.com/openservicebrokerapi/servicebroker) to communicate with service brokers, acting as an intermediary for the Kubernetes API Server to negotiate the initial provisioning and retrieve the credentials necessary for the application to use a managed service. - -It is implemented using a [CRDs-based](/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-resources) architecture. - -<br> - -![Service Catalog Architecture](/images/docs/service-catalog-architecture.svg) - - -### API Resources - -Service Catalog installs the `servicecatalog.k8s.io` API and provides the following Kubernetes resources: - -* `ClusterServiceBroker`: An in-cluster representation of a service broker, encapsulating its server connection details. -These are created and managed by cluster operators who wish to use that broker server to make new types of managed services available within their cluster. -* `ClusterServiceClass`: A managed service offered by a particular service broker. -When a new `ClusterServiceBroker` resource is added to the cluster, the Service Catalog controller connects to the service broker to obtain a list of available managed services. It then creates a new `ClusterServiceClass` resource corresponding to each managed service. -* `ClusterServicePlan`: A specific offering of a managed service. For example, a managed service may have different plans available, such as a free tier or paid tier, or it may have different configuration options, such as using SSD storage or having more resources. Similar to `ClusterServiceClass`, when a new `ClusterServiceBroker` is added to the cluster, Service Catalog creates a new `ClusterServicePlan` resource corresponding to each Service Plan available for each managed service. -* `ServiceInstance`: A provisioned instance of a `ClusterServiceClass`. -These are created by cluster operators to make a specific instance of a managed service available for use by one or more in-cluster applications. -When a new `ServiceInstance` resource is created, the Service Catalog controller connects to the appropriate service broker and instruct it to provision the service instance. -* `ServiceBinding`: Access credentials to a `ServiceInstance`. -These are created by cluster operators who want their applications to make use of a `ServiceInstance`. -Upon creation, the Service Catalog controller creates a Kubernetes `Secret` containing connection details and credentials for the Service Instance, which can be mounted into Pods. - -### Authentication - -Service Catalog supports these methods of authentication: - -* Basic (username/password) -* [OAuth 2.0 Bearer Token](https://tools.ietf.org/html/rfc6750) - -## Usage - -A cluster operator can use Service Catalog API Resources to provision managed services and make them available within a Kubernetes cluster. The steps involved are: - -1. Listing the managed services and Service Plans available from a service broker. -1. Provisioning a new instance of the managed service. -1. Binding to the managed service, which returns the connection credentials. -1. Mapping the connection credentials into the application. - -### Listing managed services and Service Plans - -First, a cluster operator must create a `ClusterServiceBroker` resource within the `servicecatalog.k8s.io` group. This resource contains the URL and connection details necessary to access a service broker endpoint. - -This is an example of a `ClusterServiceBroker` resource: - -```yaml -apiVersion: servicecatalog.k8s.io/v1beta1 -kind: ClusterServiceBroker -metadata: - name: cloud-broker -spec: - # Points to the endpoint of a service broker. (This example is not a working URL.) - url: https://servicebroker.somecloudprovider.com/v1alpha1/projects/service-catalog/brokers/default - ##### - # Additional values can be added here, which may be used to communicate - # with the service broker, such as bearer token info or a caBundle for TLS. - ##### -``` - -The following is a sequence diagram illustrating the steps involved in listing managed services and Plans available from a service broker: - -![List Services](/images/docs/service-catalog-list.svg) - -1. Once the `ClusterServiceBroker` resource is added to Service Catalog, it triggers a call to the external service broker for a list of available services. -1. The service broker returns a list of available managed services and a list of Service Plans, which are cached locally as `ClusterServiceClass` and `ClusterServicePlan` resources respectively. -1. A cluster operator can then get the list of available managed services using the following command: - - kubectl get clusterserviceclasses -o=custom-columns=SERVICE\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName - - It should output a list of service names with a format similar to: - - SERVICE NAME EXTERNAL NAME - 4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468 cloud-provider-service - ... ... - - They can also view the Service Plans available using the following command: - - kubectl get clusterserviceplans -o=custom-columns=PLAN\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName - - It should output a list of plan names with a format similar to: - - PLAN NAME EXTERNAL NAME - 86064792-7ea2-467b-af93-ac9694d96d52 service-plan-name - ... ... - - -### Provisioning a new instance - -A cluster operator can initiate the provisioning of a new instance by creating a `ServiceInstance` resource. - -This is an example of a `ServiceInstance` resource: - -```yaml -apiVersion: servicecatalog.k8s.io/v1beta1 -kind: ServiceInstance -metadata: - name: cloud-queue-instance - namespace: cloud-apps -spec: - # References one of the previously returned services - clusterServiceClassExternalName: cloud-provider-service - clusterServicePlanExternalName: service-plan-name - ##### - # Additional parameters can be added here, - # which may be used by the service broker. - ##### -``` - -The following sequence diagram illustrates the steps involved in provisioning a new instance of a managed service: - -![Provision a Service](/images/docs/service-catalog-provision.svg) - -1. When the `ServiceInstance` resource is created, Service Catalog initiates a call to the external service broker to provision an instance of the service. -1. The service broker creates a new instance of the managed service and returns an HTTP response. -1. A cluster operator can then check the status of the instance to see if it is ready. - -### Binding to a managed service - -After a new instance has been provisioned, a cluster operator must bind to the managed service to get the connection credentials and service account details necessary for the application to use the service. This is done by creating a `ServiceBinding` resource. - -The following is an example of a `ServiceBinding` resource: - -```yaml -apiVersion: servicecatalog.k8s.io/v1beta1 -kind: ServiceBinding -metadata: - name: cloud-queue-binding - namespace: cloud-apps -spec: - instanceRef: - name: cloud-queue-instance - ##### - # Additional information can be added here, such as a secretName or - # service account parameters, which may be used by the service broker. - ##### -``` - -The following sequence diagram illustrates the steps involved in binding to a managed service instance: - -![Bind to a managed service](/images/docs/service-catalog-bind.svg) - -1. After the `ServiceBinding` is created, Service Catalog makes a call to the external service broker requesting the information necessary to bind with the service instance. -1. The service broker enables the application permissions/roles for the appropriate service account. -1. The service broker returns the information necessary to connect and access the managed service instance. This is provider and service-specific so the information returned may differ between Service Providers and their managed services. - -### Mapping the connection credentials - -After binding, the final step involves mapping the connection credentials and service-specific information into the application. -These pieces of information are stored in secrets that the application in the cluster can access and use to connect directly with the managed service. - -<br> - -![Map connection credentials](/images/docs/service-catalog-map.svg) - -#### Pod configuration File - -One method to perform this mapping is to use a declarative Pod configuration. - -The following example describes how to map service account credentials into the application. A key called `sa-key` is stored in a volume named `provider-cloud-key`, and the application mounts this volume at `/var/secrets/provider/key.json`. The environment variable `PROVIDER_APPLICATION_CREDENTIALS` is mapped from the value of the mounted file. - -```yaml -... - spec: - volumes: - - name: provider-cloud-key - secret: - secretName: sa-key - containers: -... - volumeMounts: - - name: provider-cloud-key - mountPath: /var/secrets/provider - env: - - name: PROVIDER_APPLICATION_CREDENTIALS - value: "/var/secrets/provider/key.json" -``` - -The following example describes how to map secret values into application environment variables. In this example, the messaging queue topic name is mapped from a secret named `provider-queue-credentials` with a key named `topic` to the environment variable `TOPIC`. - - -```yaml -... - env: - - name: "TOPIC" - valueFrom: - secretKeyRef: - name: provider-queue-credentials - key: topic -``` - - - - -## {{% heading "whatsnext" %}} - -* If you are familiar with {{< glossary_tooltip text="Helm Charts" term_id="helm-chart" >}}, [install Service Catalog using Helm](/docs/tasks/service-catalog/install-service-catalog-using-helm/) into your Kubernetes cluster. Alternatively, you can [install Service Catalog using the SC tool](/docs/tasks/service-catalog/install-service-catalog-using-sc/). -* View [sample service brokers](https://github.com/openservicebrokerapi/servicebroker/blob/master/gettingStarted.md#sample-service-brokers). -* Explore the [kubernetes-sigs/service-catalog](https://github.com/kubernetes-sigs/service-catalog) project. -* View [svc-cat.io](https://svc-cat.io/docs/). - - - - - diff --git a/content/en/docs/concepts/overview/components.md b/content/en/docs/concepts/overview/components.md index 6c5c0eefa1..387fc157d9 100644 --- a/content/en/docs/concepts/overview/components.md +++ b/content/en/docs/concepts/overview/components.md @@ -19,11 +19,7 @@ When you deploy Kubernetes, you get a cluster. This document outlines the various components you need to have for a complete and working Kubernetes cluster. -Here's the diagram of a Kubernetes cluster with all the components tied together. - -![Components of Kubernetes](/images/docs/components-of-kubernetes.svg) - - +{{< figure src="/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes" caption="The components of a Kubernetes cluster" class="diagram-large" >}} <!-- body --> ## Control Plane Components @@ -34,7 +30,7 @@ Control plane components can be run on any machine in the cluster. However, for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine. See [Creating Highly Available clusters with kubeadm](/docs/setup/production-environment/tools/kubeadm/high-availability/) -for an example control plane setup that runs across multiple VMs. +for an example control plane setup that runs across multiple machines. ### kube-apiserver @@ -118,7 +114,7 @@ Containers started by Kubernetes automatically include this DNS server in their ### Container Resource Monitoring -[Container Resource Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) records generic time-series metrics +[Container Resource Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) records generic time-series metrics about containers in a central database, and provides a UI for browsing that data. ### Cluster-level Logging diff --git a/content/en/docs/concepts/overview/kubernetes-api.md b/content/en/docs/concepts/overview/kubernetes-api.md index 07b5d559d7..b72d1aab1f 100644 --- a/content/en/docs/concepts/overview/kubernetes-api.md +++ b/content/en/docs/concepts/overview/kubernetes-api.md @@ -23,7 +23,7 @@ The Kubernetes API lets you query and manipulate the state of API objects in Kub (for example: Pods, Namespaces, ConfigMaps, and Events). Most operations can be performed through the -[kubectl](/docs/reference/kubectl/overview/) command-line interface or other +[kubectl](/docs/reference/kubectl/) command-line interface or other command-line tools, such as [kubeadm](/docs/reference/setup-tools/kubeadm/), which in turn use the API. However, you can also access the API directly using REST calls. @@ -37,8 +37,11 @@ if you are writing an application using the Kubernetes API. Complete API details are documented using [OpenAPI](https://www.openapis.org/). -The Kubernetes API server serves an OpenAPI spec via the `/openapi/v2` endpoint. -You can request the response format using request headers as follows: +### OpenAPI V2 + +The Kubernetes API server serves an aggregated OpenAPI v2 spec via the +`/openapi/v2` endpoint. You can request the response format using +request headers as follows: <table> <caption style="display:none">Valid request header values for OpenAPI v2 queries</caption> @@ -73,10 +76,80 @@ You can request the response format using request headers as follows: Kubernetes implements an alternative Protobuf based serialization format that is primarily intended for intra-cluster communication. For more information -about this format, see the [Kubernetes Protobuf serialization](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md) design proposal and the +about this format, see the [Kubernetes Protobuf serialization](https://git.k8s.io/design-proposals-archive/api-machinery/protobuf.md) design proposal and the Interface Definition Language (IDL) files for each schema located in the Go packages that define the API objects. +### OpenAPI V3 + +{{< feature-state state="beta" for_k8s_version="v1.24" >}} + +Kubernetes {{< param "version" >}} offers beta support for publishing its APIs as OpenAPI v3; this is a +beta feature that is enabled by default. +You can disable the beta feature by turning off the +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) named `OpenAPIV3` +for the kube-apiserver component. + +A discovery endpoint `/openapi/v3` is provided to see a list of all +group/versions available. This endpoint only returns JSON. These group/versions +are provided in the following format: +```json +{ + "paths": { + ... + "api/v1": { + "serverRelativeURL": "/openapi/v3/api/v1?hash=CC0E9BFD992D8C59AEC98A1E2336F899E8318D3CF4C68944C3DEC640AF5AB52D864AC50DAA8D145B3494F75FA3CFF939FCBDDA431DAD3CA79738B297795818CF" + }, + "apis/admissionregistration.k8s.io/v1": { + "serverRelativeURL": "/openapi/v3/apis/admissionregistration.k8s.io/v1?hash=E19CC93A116982CE5422FC42B590A8AFAD92CDE9AE4D59B5CAAD568F083AD07946E6CB5817531680BCE6E215C16973CD39003B0425F3477CFD854E89A9DB6597" + }, + ... +} +``` + +The relative URLs are pointing to immutable OpenAPI descriptions, in +order to improve client-side caching. The proper HTTP caching headers +are also set by the API server for that purpose (`Expires` to 1 year in +the future, and `Cache-Control` to `immutable`). When an obsolete URL is +used, the API server returns a redirect to the newest URL. + +The Kubernetes API server publishes an OpenAPI v3 spec per Kubernetes +group version at the `/openapi/v3/apis/<group>/<version>?hash=<hash>` +endpoint. + +Refer to the table below for accepted request headers. + +<table> + <caption style="display:none">Valid request header values for OpenAPI v3 queries</caption> + <thead> + <tr> + <th>Header</th> + <th style="min-width: 50%;">Possible values</th> + <th>Notes</th> + </tr> + </thead> + <tbody> + <tr> + <td><code>Accept-Encoding</code></td> + <td><code>gzip</code></td> + <td><em>not supplying this header is also acceptable</em></td> + </tr> + <tr> + <td rowspan="3"><code>Accept</code></td> + <td><code>application/com.github.proto-openapi.spec.v3@v1.0+protobuf</code></td> + <td><em>mainly for intra-cluster use</em></td> + </tr> + <tr> + <td><code>application/json</code></td> + <td><em>default</em></td> + </tr> + <tr> + <td><code>*</code></td> + <td><em>serves </em><code>application/json</code></td> + </tr> + </tbody> +</table> + ## Persistence Kubernetes stores the serialized state of objects by writing them into diff --git a/content/en/docs/concepts/overview/what-is-kubernetes.md b/content/en/docs/concepts/overview/what-is-kubernetes.md index d72f1beb48..b9cc6bf6be 100644 --- a/content/en/docs/concepts/overview/what-is-kubernetes.md +++ b/content/en/docs/concepts/overview/what-is-kubernetes.md @@ -4,7 +4,7 @@ reviewers: - mikedanese title: What is Kubernetes? description: > - Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. + Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. content_type: concept weight: 10 card: @@ -19,7 +19,7 @@ This page is an overview of Kubernetes. <!-- body --> -Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. +Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the "K" and the "s". Google open-sourced the Kubernetes project in 2014. Kubernetes combines [over 15 years of Google's experience](/blog/2015/04/borg-predecessor-to-kubernetes/) running production workloads at scale with best-of-breed ideas and practices from the community. diff --git a/content/en/docs/concepts/overview/working-with-objects/common-labels.md b/content/en/docs/concepts/overview/working-with-objects/common-labels.md index 3053544cd2..210d33dcd4 100644 --- a/content/en/docs/concepts/overview/working-with-objects/common-labels.md +++ b/content/en/docs/concepts/overview/working-with-objects/common-labels.md @@ -42,9 +42,10 @@ on every resource object. | `app.kubernetes.io/managed-by` | The tool being used to manage the operation of an application | `helm` | string | | `app.kubernetes.io/created-by` | The controller/user who created this resource | `controller-manager` | string | -To illustrate these labels in action, consider the following StatefulSet object: +To illustrate these labels in action, consider the following {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} object: ```yaml +# This is an excerpt apiVersion: apps/v1 kind: StatefulSet metadata: diff --git a/content/en/docs/concepts/overview/working-with-objects/finalizers.md b/content/en/docs/concepts/overview/working-with-objects/finalizers.md index 9516b935c9..9241ae7e2a 100644 --- a/content/en/docs/concepts/overview/working-with-objects/finalizers.md +++ b/content/en/docs/concepts/overview/working-with-objects/finalizers.md @@ -21,18 +21,21 @@ your own. When you create a resource using a manifest file, you can specify finalizers in the `metadata.finalizers` field. When you attempt to delete the resource, the -controller that manages it notices the values in the `finalizers` field and does -the following: +API server handling the delete request notices the values in the `finalizers` field +and does the following: * Modifies the object to add a `metadata.deletionTimestamp` field with the time you started the deletion. - * Marks the object as read-only until its `metadata.finalizers` field is empty. + * Prevents the object from being removed until its `metadata.finalizers` field is empty. + * Returns a `202` status code (HTTP "Accepted") +The controller managing that finalizer notices the update to the object setting the +`metadata.deletionTimestamp`, indicating deletion of the object has been requested. The controller then attempts to satisfy the requirements of the finalizers specified for that resource. Each time a finalizer condition is satisfied, the controller removes that key from the resource's `finalizers` field. When the -field is empty, garbage collection continues. You can also use finalizers to -prevent deletion of unmanaged resources. +`finalizers` field is emptied, an object with a `deletionTimestamp` field set +is automatically deleted. You can also use finalizers to prevent deletion of unmanaged resources. A common example of a finalizer is `kubernetes.io/pv-protection`, which prevents accidental deletion of `PersistentVolume` objects. When a `PersistentVolume` @@ -44,7 +47,8 @@ and the controller deletes the volume. ## Owner references, labels, and finalizers {#owners-labels-finalizers} -Like {{<glossary_tooltip text="labels" term_id="label">}}, [owner references](/concepts/overview/working-with-objects/owners-dependents/) +Like {{<glossary_tooltip text="labels" term_id="label">}}, +[owner references](/docs/concepts/overview/working-with-objects/owners-dependents/) describe the relationships between objects in Kubernetes, but are used for a different purpose. When a {{<glossary_tooltip text="controller" term_id="controller">}} manages objects @@ -62,16 +66,18 @@ Kubernetes also processes finalizers when it identifies owner references on a resource targeted for deletion. In some situations, finalizers can block the deletion of dependent objects, -which can cause the targeted owner object to remain in a read-only state for +which can cause the targeted owner object to remain for longer than expected without being fully deleted. In these situations, you should check finalizers and owner references on the target owner and dependent objects to troubleshoot the cause. {{<note>}} -In cases where objects are stuck in a deleting state, try to avoid manually +In cases where objects are stuck in a deleting state, avoid manually removing finalizers to allow deletion to continue. Finalizers are usually added to resources for a reason, so forcefully removing them can lead to issues in -your cluster. +your cluster. This should only be done when the purpose of the finalizer is +understood and is accomplished in another way (for example, manually cleaning +up some dependent object). {{</note>}} ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/concepts/overview/working-with-objects/kubernetes-objects.md b/content/en/docs/concepts/overview/working-with-objects/kubernetes-objects.md index c763b40e05..8e4214991c 100644 --- a/content/en/docs/concepts/overview/working-with-objects/kubernetes-objects.md +++ b/content/en/docs/concepts/overview/working-with-objects/kubernetes-objects.md @@ -8,21 +8,29 @@ card: --- <!-- overview --> -This page explains how Kubernetes objects are represented in the Kubernetes API, and how you can express them in `.yaml` format. - +This page explains how Kubernetes objects are represented in the Kubernetes API, and how you can +express them in `.yaml` format. <!-- body --> ## Understanding Kubernetes objects {#kubernetes-objects} -*Kubernetes objects* are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe: +*Kubernetes objects* are persistent entities in the Kubernetes system. Kubernetes uses these +entities to represent the state of your cluster. Specifically, they can describe: * What containerized applications are running (and on which nodes) * The resources available to those applications * The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance -A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that object exists. By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's *desired state*. +A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system +will constantly work to ensure that object exists. By creating an object, you're effectively +telling the Kubernetes system what you want your cluster's workload to look like; this is your +cluster's *desired state*. -To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the [Kubernetes API](/docs/concepts/overview/kubernetes-api/). When you use the `kubectl` command-line interface, for example, the CLI makes the necessary Kubernetes API calls for you. You can also use the Kubernetes API directly in your own programs using one of the [Client Libraries](/docs/reference/using-api/client-libraries/). +To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the +[Kubernetes API](/docs/concepts/overview/kubernetes-api/). When you use the `kubectl` command-line +interface, for example, the CLI makes the necessary Kubernetes API calls for you. You can also use +the Kubernetes API directly in your own programs using one of the +[Client Libraries](/docs/reference/using-api/client-libraries/). ### Object Spec and Status @@ -48,11 +56,17 @@ the status to match your spec. If any of those instances should fail between spec and status by making a correction--in this case, starting a replacement instance. -For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md). +For more information on the object spec, status, and metadata, see the +[Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md). ### Describing a Kubernetes object -When you create an object in Kubernetes, you must provide the object spec that describes its desired state, as well as some basic information about the object (such as a name). When you use the Kubernetes API to create the object (either directly or via `kubectl`), that API request must include that information as JSON in the request body. **Most often, you provide the information to `kubectl` in a .yaml file.** `kubectl` converts the information to JSON when making the API request. +When you create an object in Kubernetes, you must provide the object spec that describes its +desired state, as well as some basic information about the object (such as a name). When you use +the Kubernetes API to create the object (either directly or via `kubectl`), that API request must +include that information as JSON in the request body. **Most often, you provide the information to +`kubectl` in a .yaml file.** `kubectl` converts the information to JSON when making the API +request. Here's an example `.yaml` file that shows the required fields and object spec for a Kubernetes Deployment: @@ -63,7 +77,7 @@ One way to create a Deployment using a `.yaml` file like the one above is to use in the `kubectl` command-line interface, passing the `.yaml` file as an argument. Here's an example: ```shell -kubectl apply -f https://k8s.io/examples/application/deployment.yaml --record +kubectl apply -f https://k8s.io/examples/application/deployment.yaml ``` The output is similar to this: @@ -81,12 +95,23 @@ In the `.yaml` file for the Kubernetes object you want to create, you'll need to * `metadata` - Data that helps uniquely identify the object, including a `name` string, `UID`, and optional `namespace` * `spec` - What state you desire for the object -The precise format of the object `spec` is different for every Kubernetes object, and contains nested fields specific to that object. The [Kubernetes API Reference](https://kubernetes.io/docs/reference/kubernetes-api/) can help you find the spec format for all of the objects you can create using Kubernetes. - -For example, the reference for Pod details the [`spec` field](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) -for a Pod in the API, and the reference for Deployment details the [`spec` field](/docs/reference/kubernetes-api/workload-resources/deployment-v1/#DeploymentSpec) for Deployments. -In those API reference pages you'll see mention of PodSpec and DeploymentSpec. These names are implementation details of the Golang code that Kubernetes uses to implement its API. +The precise format of the object `spec` is different for every Kubernetes object, and contains +nested fields specific to that object. The [Kubernetes API Reference](/docs/reference/kubernetes-api/) +can help you find the spec format for all of the objects you can create using Kubernetes. +For example, see the [`spec` field](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) +for the Pod API reference. +For each Pod, the `.spec` field specifies the pod and its desired state (such as the container image name for +each container within that pod). +Another example of an object specification is the +[`spec` field](/docs/reference/kubernetes-api/workload-resources/stateful-set-v1/#StatefulSetSpec) +for the StatefulSet API. For StatefulSet, the `.spec` field specifies the StatefulSet and +its desired state. +Within the `.spec` of a StatefulSet is a [template](/docs/concepts/workloads/pods/#pod-templates) +for Pod objects. That template describes Pods that the StatefulSet controller will create in order to +satisfy the StatefulSet specification. +Different kinds of object can also have different `.status`; again, the API reference pages +detail the structure of that `.status` field, and its content for each different type of object. ## {{% heading "whatsnext" %}} @@ -94,5 +119,3 @@ In those API reference pages you'll see mention of PodSpec and DeploymentSpec. T * Learn about [controllers](/docs/concepts/architecture/controller/) in Kubernetes. * [Using the Kubernetes API](/docs/reference/using-api/) explains some more API concepts. - - diff --git a/content/en/docs/concepts/overview/working-with-objects/names.md b/content/en/docs/concepts/overview/working-with-objects/names.md index 9bafb1584c..7b6b380e35 100644 --- a/content/en/docs/concepts/overview/working-with-objects/names.md +++ b/content/en/docs/concepts/overview/working-with-objects/names.md @@ -100,4 +100,4 @@ UUIDs are standardized as ISO/IEC 9834-8 and as ITU-T X.667. ## {{% heading "whatsnext" %}} * Read about [labels](/docs/concepts/overview/working-with-objects/labels/) in Kubernetes. -* See the [Identifiers and Names in Kubernetes](https://git.k8s.io/community/contributors/design-proposals/architecture/identifiers.md) design document. +* See the [Identifiers and Names in Kubernetes](https://git.k8s.io/design-proposals-archive/architecture/identifiers.md) design document. diff --git a/content/en/docs/concepts/overview/working-with-objects/namespaces.md b/content/en/docs/concepts/overview/working-with-objects/namespaces.md index 6664a2ad4c..77998a74c4 100644 --- a/content/en/docs/concepts/overview/working-with-objects/namespaces.md +++ b/content/en/docs/concepts/overview/working-with-objects/namespaces.md @@ -10,8 +10,7 @@ weight: 30 <!-- overview --> -Kubernetes supports multiple virtual clusters backed by the same physical cluster. -These virtual clusters are called namespaces. +In Kubernetes, _namespaces_ provides a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects _(e.g. Deployments, Services, etc)_ and not for cluster-wide objects _(e.g. StorageClass, Nodes, PersistentVolumes, etc)_. <!-- body --> @@ -99,6 +98,24 @@ is local to a namespace. This is useful for using the same configuration across multiple namespaces such as Development, Staging and Production. If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN). +As a result, all namespace names must be valid +[RFC 1123 DNS labels](/docs/concepts/overview/working-with-objects/names/#dns-label-names). + +{{< warning >}} +By creating namespaces with the same name as [public top-level +domains](https://data.iana.org/TLD/tlds-alpha-by-domain.txt), Services in these +namespaces can have short DNS names that overlap with public DNS records. +Workloads from any namespace performing a DNS lookup without a [trailing dot](https://datatracker.ietf.org/doc/html/rfc1034#page-8) will +be redirected to those services, taking precedence over public DNS. + +To mitigate this, limit privileges for creating namespaces to trusted users. If +required, you could additionally configure third-party security controls, such +as [admission +webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/), +to block creating any namespace with the name of [public +TLDs](https://data.iana.org/TLD/tlds-alpha-by-domain.txt). +{{< /warning >}} + ## Not All Objects are in a Namespace Most Kubernetes resources (e.g. pods, services, replication controllers, and others) are diff --git a/content/en/docs/concepts/overview/working-with-objects/object-management.md b/content/en/docs/concepts/overview/working-with-objects/object-management.md index b85c622823..10b6dacf85 100644 --- a/content/en/docs/concepts/overview/working-with-objects/object-management.md +++ b/content/en/docs/concepts/overview/working-with-objects/object-management.md @@ -169,9 +169,9 @@ Disadvantages compared to imperative object configuration: ## {{% heading "whatsnext" %}} - [Managing Kubernetes Objects Using Imperative Commands](/docs/tasks/manage-kubernetes-objects/imperative-command/) -- [Managing Kubernetes Objects Using Object Configuration (Imperative)](/docs/tasks/manage-kubernetes-objects/imperative-config/) -- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/tasks/manage-kubernetes-objects/declarative-config/) -- [Managing Kubernetes Objects Using Kustomize (Declarative)](/docs/tasks/manage-kubernetes-objects/kustomization/) +- [Imperative Management of Kubernetes Objects Using Configuration Files](/docs/tasks/manage-kubernetes-objects/imperative-config/) +- [Declarative Management of Kubernetes Objects Using Configuration Files](/docs/tasks/manage-kubernetes-objects/declarative-config/) +- [Declarative Management of Kubernetes Objects Using Kustomize](/docs/tasks/manage-kubernetes-objects/kustomization/) - [Kubectl Command Reference](/docs/reference/generated/kubectl/kubectl-commands/) - [Kubectl Book](https://kubectl.docs.kubernetes.io) - [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/) diff --git a/content/en/docs/concepts/policy/limit-range.md b/content/en/docs/concepts/policy/limit-range.md index 8158b78437..f5a7e66cac 100644 --- a/content/en/docs/concepts/policy/limit-range.md +++ b/content/en/docs/concepts/policy/limit-range.md @@ -53,7 +53,7 @@ Neither contention nor changes to a LimitRange will affect already created resou ## {{% heading "whatsnext" %}} -Refer to the [LimitRanger design document](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) for more information. +Refer to the [LimitRanger design document](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_limit_range.md) for more information. For examples on using limits, see: diff --git a/content/en/docs/concepts/policy/resource-quotas.md b/content/en/docs/concepts/policy/resource-quotas.md index d31efd09bc..8d9490b828 100644 --- a/content/en/docs/concepts/policy/resource-quotas.md +++ b/content/en/docs/concepts/policy/resource-quotas.md @@ -22,8 +22,7 @@ be consumed by resources in that namespace. Resource quotas work like this: -- Different teams work in different namespaces. Currently this is voluntary, but - support for making this mandatory via ACLs is planned. +- Different teams work in different namespaces. This can be enforced with [RBAC](/docs/reference/access-authn-authz/rbac/). - The administrator creates one ResourceQuota for each namespace. @@ -442,7 +441,7 @@ pods 0 10 ### Cross-namespace Pod Affinity Quota -{{< feature-state for_k8s_version="v1.22" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} Operators can use `CrossNamespacePodAffinity` quota scope to limit which namespaces are allowed to have pods with affinity terms that cross namespaces. Specifically, it controls which pods are allowed @@ -493,10 +492,6 @@ With the above configuration, pods can use `namespaces` and `namespaceSelector` if the namespace where they are created have a resource quota object with `CrossNamespaceAffinity` scope and a hard limit greater than or equal to the number of pods using those fields. -This feature is beta and enabled by default. You can disable it using the -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`PodAffinityNamespaceSelector` in both kube-apiserver and kube-scheduler. - ## Requests compared to Limits {#requests-vs-limits} When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. @@ -702,7 +697,7 @@ and it is to be created in a namespace other than `kube-system`. ## {{% heading "whatsnext" %}} -- See [ResourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) for more information. +- See [ResourceQuota design doc](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_resource_quota.md) for more information. - See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/). -- Read [Quota support for priority class design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-resourcequota.md). +- Read [Quota support for priority class design doc](https://git.k8s.io/design-proposals-archive/scheduling/pod-priority-resourcequota.md). - See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) diff --git a/content/en/docs/concepts/scheduling-eviction/_index.md b/content/en/docs/concepts/scheduling-eviction/_index.md index 21e9371f03..fd1c0bbf00 100644 --- a/content/en/docs/concepts/scheduling-eviction/_index.md +++ b/content/en/docs/concepts/scheduling-eviction/_index.md @@ -23,6 +23,7 @@ of terminating one or more Pods on Nodes. * [Kubernetes Scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/) * [Assigning Pods to Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/) * [Pod Overhead](/docs/concepts/scheduling-eviction/pod-overhead/) +* [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) * [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) * [Scheduling Framework](/docs/concepts/scheduling-eviction/scheduling-framework) * [Scheduler Performance Tuning](/docs/concepts/scheduling-eviction/scheduler-perf-tuning/) diff --git a/content/en/docs/concepts/scheduling-eviction/api-eviction.md b/content/en/docs/concepts/scheduling-eviction/api-eviction.md index e7f1942df2..b4e92c40bf 100644 --- a/content/en/docs/concepts/scheduling-eviction/api-eviction.md +++ b/content/en/docs/concepts/scheduling-eviction/api-eviction.md @@ -6,14 +6,117 @@ weight: 70 {{< glossary_definition term_id="api-eviction" length="short" >}} </br> -You can request eviction by directly calling the Eviction API -using a client of the kube-apiserver, like the `kubectl drain` command. -This creates an `Eviction` object, which causes the API server to terminate the Pod. +You can request eviction by calling the Eviction API directly, or programmatically +using a client of the {{<glossary_tooltip term_id="kube-apiserver" text="API server">}}, like the `kubectl drain` command. This +creates an `Eviction` object, which causes the API server to terminate the Pod. API-initiated evictions respect your configured [`PodDisruptionBudgets`](/docs/tasks/run-application/configure-pdb/) and [`terminationGracePeriodSeconds`](/docs/concepts/workloads/pods/pod-lifecycle#pod-termination). +Using the API to create an Eviction object for a Pod is like performing a +policy-controlled [`DELETE` operation](/docs/reference/kubernetes-api/workload-resources/pod-v1/#delete-delete-a-pod) +on the Pod. + +## Calling the Eviction API + +You can use a [Kubernetes language client](/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api) +to access the Kubernetes API and create an `Eviction` object. To do this, you +POST the attempted operation, similar to the following example: + +{{< tabs name="Eviction_example" >}} +{{% tab name="policy/v1" %}} +{{< note >}} +`policy/v1` Eviction is available in v1.22+. Use `policy/v1beta1` with prior releases. +{{< /note >}} + +```json +{ + "apiVersion": "policy/v1", + "kind": "Eviction", + "metadata": { + "name": "quux", + "namespace": "default" + } +} +``` +{{% /tab %}} +{{% tab name="policy/v1beta1" %}} +{{< note >}} +Deprecated in v1.22 in favor of `policy/v1` +{{< /note >}} + +```json +{ + "apiVersion": "policy/v1beta1", + "kind": "Eviction", + "metadata": { + "name": "quux", + "namespace": "default" + } +} +``` +{{% /tab %}} +{{< /tabs >}} + +Alternatively, you can attempt an eviction operation by accessing the API using +`curl` or `wget`, similar to the following example: + +```bash +curl -v -H 'Content-type: application/json' https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json +``` + +## How API-initiated eviction works + +When you request an eviction using the API, the API server performs admission +checks and responds in one of the following ways: + +* `200 OK`: the eviction is allowed, the `Eviction` subresource is created, and + the Pod is deleted, similar to sending a `DELETE` request to the Pod URL. +* `429 Too Many Requests`: the eviction is not currently allowed because of the + configured {{<glossary_tooltip term_id="pod-disruption-budget" text="PodDisruptionBudget">}}. + You may be able to attempt the eviction again later. You might also see this + response because of API rate limiting. +* `500 Internal Server Error`: the eviction is not allowed because there is a + misconfiguration, like if multiple PodDisruptionBudgets reference the same Pod. + +If the Pod you want to evict isn't part of a workload that has a +PodDisruptionBudget, the API server always returns `200 OK` and allows the +eviction. + +If the API server allows the eviction, the Pod is deleted as follows: + +1. The `Pod` resource in the API server is updated with a deletion timestamp, + after which the API server considers the `Pod` resource to be terminated. The + `Pod` resource is also marked with the configured grace period. +1. The {{<glossary_tooltip term_id="kubelet" text="kubelet">}} on the node where the local Pod is running notices that the `Pod` + resource is marked for termination and starts to gracefully shut down the + local Pod. +1. While the kubelet is shutting the Pod down, the control plane removes the Pod + from {{<glossary_tooltip term_id="endpoint" text="Endpoint">}} and + {{<glossary_tooltip term_id="endpoint-slice" text="EndpointSlice">}} + objects. As a result, controllers no longer consider the Pod as a valid object. +1. After the grace period for the Pod expires, the kubelet forcefully terminates + the local Pod. +1. The kubelet tells the API server to remove the `Pod` resource. +1. The API server deletes the `Pod` resource. + +## Troubleshooting stuck evictions + +In some cases, your applications may enter a broken state, where the Eviction +API will only return `429` or `500` responses until you intervene. This can +happen if, for example, a ReplicaSet creates pods for your application but new +pods do not enter a `Ready` state. You may also notice this behavior in cases +where the last evicted Pod had a long termination grace period. + +If you notice stuck evictions, try one of the following solutions: + +* Abort or pause the automated operation causing the issue. Investigate the stuck + application before you restart the operation. +* Wait a while, then directly delete the Pod from your cluster control plane + instead of using the Eviction API. + ## {{% heading "whatsnext" %}} -* Learn about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/) -* Learn about [Pod Priority and Preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) +* Learn how to protect your applications with a [Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/). +* Learn about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/). +* Learn about [Pod Priority and Preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/). diff --git a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md index 9216ec2ff9..9cd70f9e97 100644 --- a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md +++ b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -11,163 +11,189 @@ weight: 20 <!-- overview --> -You can constrain a {{< glossary_tooltip text="Pod" term_id="pod" >}} so that it can only run on particular set of -{{< glossary_tooltip text="Node(s)" term_id="node" >}}. +You can constrain a {{< glossary_tooltip text="Pod" term_id="pod" >}} so that it is +_restricted_ to run on particular {{< glossary_tooltip text="node(s)" term_id="node" >}}, +or to _prefer_ to run on particular nodes. There are several ways to do this and the recommended approaches all use [label selectors](/docs/concepts/overview/working-with-objects/labels/) to facilitate the selection. -Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement -(e.g. spread your pods across nodes so as not place the pod on a node with insufficient free resources, etc.) -but there are some circumstances where you may want to control which node the pod deploys to - for example to ensure -that a pod ends up on a machine with an SSD attached to it, or to co-locate pods from two different -services that communicate a lot into the same availability zone. - +Often, you do not need to set any such constraints; the +{{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} will automatically do a reasonable placement +(for example, spreading your Pods across nodes so as not place Pods on a node with insufficient free resources). +However, there are some circumstances where you may want to control which node +the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it, +or to co-locate Pods from two different services that communicate a lot into the same availability zone. <!-- body --> +You can use any of the following methods to choose where Kubernetes schedules +specific Pods: + + * [nodeSelector](#nodeselector) field matching against [node labels](#built-in-node-labels) + * [Affinity and anti-affinity](#affinity-and-anti-affinity) + * [nodeName](#nodename) field + * [Pod topology spread constraints](#pod-topology-spread-constraints) + +## Node labels {#built-in-node-labels} + +Like many other Kubernetes objects, nodes have +[labels](/docs/concepts/overview/working-with-objects/labels/). You can [attach labels manually](/docs/tasks/configure-pod-container/assign-pods-nodes/#add-a-label-to-a-node). +Kubernetes also populates a standard set of labels on all nodes in a cluster. See [Well-Known Labels, Annotations and Taints](/docs/reference/labels-annotations-taints/) +for a list of common node labels. + +{{<note>}} +The value of these labels is cloud provider specific and is not guaranteed to be reliable. +For example, the value of `kubernetes.io/hostname` may be the same as the node name in some environments +and a different value in other environments. +{{</note>}} + +### Node isolation/restriction + +Adding labels to nodes allows you to target Pods for scheduling on specific +nodes or groups of nodes. You can use this functionality to ensure that specific +Pods only run on nodes with certain isolation, security, or regulatory +properties. + +If you use labels for node isolation, choose label keys that the {{<glossary_tooltip text="kubelet" term_id="kubelet">}} +cannot modify. This prevents a compromised node from setting those labels on +itself so that the scheduler schedules workloads onto the compromised node. + +The [`NodeRestriction` admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) +prevents the kubelet from setting or modifying labels with a +`node-restriction.kubernetes.io/` prefix. + +To make use of that label prefix for node isolation: + +1. Ensure you are using the [Node authorizer](/docs/reference/access-authn-authz/node/) and have _enabled_ the `NodeRestriction` admission plugin. +2. Add labels with the `node-restriction.kubernetes.io/` prefix to your nodes, and use those labels in your [node selectors](#nodeselector). + For example, `example.com.node-restriction.kubernetes.io/fips=true` or `example.com.node-restriction.kubernetes.io/pci-dss=true`. + ## nodeSelector `nodeSelector` is the simplest recommended form of node selection constraint. -`nodeSelector` is a field of PodSpec. It specifies a map of key-value pairs. For the pod to be eligible -to run on a node, the node must have each of the indicated key-value pairs as labels (it can have -additional labels as well). The most common usage is one key-value pair. +You can add the `nodeSelector` field to your Pod specification and specify the +[node labels](#built-in-node-labels) you want the target node to have. +Kubernetes only schedules the Pod onto nodes that have each of the labels you +specify. -Let's walk through an example of how to use `nodeSelector`. - -### Step Zero: Prerequisites - -This example assumes that you have a basic understanding of Kubernetes pods and that you have [set up a Kubernetes cluster](/docs/setup/). - -### Step One: Attach label to the node - -Run `kubectl get nodes` to get the names of your cluster's nodes. Pick out the one that you want to add a label to, and then run `kubectl label nodes <node-name> <label-key>=<label-value>` to add a label to the node you've chosen. For example, if my node name is 'kubernetes-foo-node-1.c.a-robinson.internal' and my desired label is 'disktype=ssd', then I can run `kubectl label nodes kubernetes-foo-node-1.c.a-robinson.internal disktype=ssd`. - -You can verify that it worked by re-running `kubectl get nodes --show-labels` and checking that the node now has a label. You can also use `kubectl describe node "nodename"` to see the full list of labels of the given node. - -### Step Two: Add a nodeSelector field to your pod configuration - -Take whatever pod config file you want to run, and add a nodeSelector section to it, like this. For example, if this is my pod config: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: nginx - labels: - env: test -spec: - containers: - - name: nginx - image: nginx -``` - -Then add a nodeSelector like so: - -{{< codenew file="pods/pod-nginx.yaml" >}} - -When you then run `kubectl apply -f https://k8s.io/examples/pods/pod-nginx.yaml`, -the Pod will get scheduled on the node that you attached the label to. You can -verify that it worked by running `kubectl get pods -o wide` and looking at the -"NODE" that the Pod was assigned to. - -## Interlude: built-in node labels {#built-in-node-labels} - -In addition to labels you [attach](#step-one-attach-label-to-the-node), nodes come pre-populated -with a standard set of labels. See [Well-Known Labels, Annotations and Taints](/docs/reference/labels-annotations-taints/) for a list of these. - -{{< note >}} -The value of these labels is cloud provider specific and is not guaranteed to be reliable. -For example, the value of `kubernetes.io/hostname` may be the same as the Node name in some environments -and a different value in other environments. -{{< /note >}} - -## Node isolation/restriction - -Adding labels to Node objects allows targeting pods to specific nodes or groups of nodes. -This can be used to ensure specific pods only run on nodes with certain isolation, security, or regulatory properties. -When using labels for this purpose, choosing label keys that cannot be modified by the kubelet process on the node is strongly recommended. -This prevents a compromised node from using its kubelet credential to set those labels on its own Node object, -and influencing the scheduler to schedule workloads to the compromised node. - -The `NodeRestriction` admission plugin prevents kubelets from setting or modifying labels with a `node-restriction.kubernetes.io/` prefix. -To make use of that label prefix for node isolation: - -1. Ensure you are using the [Node authorizer](/docs/reference/access-authn-authz/node/) and have _enabled_ the [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction). -2. Add labels under the `node-restriction.kubernetes.io/` prefix to your Node objects, and use those labels in your node selectors. -For example, `example.com.node-restriction.kubernetes.io/fips=true` or `example.com.node-restriction.kubernetes.io/pci-dss=true`. +See [Assign Pods to Nodes](/docs/tasks/configure-pod-container/assign-pods-nodes) for more +information. ## Affinity and anti-affinity -`nodeSelector` provides a very simple way to constrain pods to nodes with particular labels. The affinity/anti-affinity -feature, greatly expands the types of constraints you can express. The key enhancements are +`nodeSelector` is the simplest way to constrain Pods to nodes with specific +labels. Affinity and anti-affinity expands the types of constraints you can +define. Some of the benefits of affinity and anti-affinity include: -1. The affinity/anti-affinity language is more expressive. The language offers more matching rules - besides exact matches created with a logical AND operation; -2. you can indicate that the rule is "soft"/"preference" rather than a hard requirement, so if the scheduler - can't satisfy it, the pod will still be scheduled; -3. you can constrain against labels on other pods running on the node (or other topological domain), - rather than against labels on the node itself, which allows rules about which pods can and cannot be co-located +* The affinity/anti-affinity language is more expressive. `nodeSelector` only + selects nodes with all the specified labels. Affinity/anti-affinity gives you + more control over the selection logic. +* You can indicate that a rule is *soft* or *preferred*, so that the scheduler + still schedules the Pod even if it can't find a matching node. +* You can constrain a Pod using labels on other Pods running on the node (or other topological domain), + instead of just node labels, which allows you to define rules for which Pods + can be co-located on a node. -The affinity feature consists of two types of affinity, "node affinity" and "inter-pod affinity/anti-affinity". -Node affinity is like the existing `nodeSelector` (but with the first two benefits listed above), -while inter-pod affinity/anti-affinity constrains against pod labels rather than node labels, as -described in the third item listed above, in addition to having the first and second properties listed above. +The affinity feature consists of two types of affinity: + +* *Node affinity* functions like the `nodeSelector` field but is more expressive and + allows you to specify soft rules. +* *Inter-pod affinity/anti-affinity* allows you to constrain Pods against labels + on other Pods. ### Node affinity -Node affinity is conceptually similar to `nodeSelector` -- it allows you to constrain which nodes your -pod is eligible to be scheduled on, based on labels on the node. +Node affinity is conceptually similar to `nodeSelector`, allowing you to constrain which nodes your +Pod can be scheduled on based on node labels. There are two types of node +affinity: -There are currently two types of node affinity, called `requiredDuringSchedulingIgnoredDuringExecution` and -`preferredDuringSchedulingIgnoredDuringExecution`. You can think of them as "hard" and "soft" respectively, -in the sense that the former specifies rules that *must* be met for a pod to be scheduled onto a node (similar to -`nodeSelector` but using a more expressive syntax), while the latter specifies *preferences* that the scheduler -will try to enforce but will not guarantee. The "IgnoredDuringExecution" part of the names means that, similar -to how `nodeSelector` works, if labels on a node change at runtime such that the affinity rules on a pod are no longer -met, the pod continues to run on the node. In the future we plan to offer -`requiredDuringSchedulingRequiredDuringExecution` which will be identical to `requiredDuringSchedulingIgnoredDuringExecution` -except that it will evict pods from nodes that cease to satisfy the pods' node affinity requirements. + * `requiredDuringSchedulingIgnoredDuringExecution`: The scheduler can't + schedule the Pod unless the rule is met. This functions like `nodeSelector`, + but with a more expressive syntax. + * `preferredDuringSchedulingIgnoredDuringExecution`: The scheduler tries to + find a node that meets the rule. If a matching node is not available, the + scheduler still schedules the Pod. -Thus an example of `requiredDuringSchedulingIgnoredDuringExecution` would be "only run the pod on nodes with Intel CPUs" -and an example `preferredDuringSchedulingIgnoredDuringExecution` would be "try to run this set of pods in failure -zone XYZ, but if it's not possible, then allow some to run elsewhere". +{{<note>}} +In the preceding types, `IgnoredDuringExecution` means that if the node labels +change after Kubernetes schedules the Pod, the Pod continues to run. +{{</note>}} -Node affinity is specified as field `nodeAffinity` of field `affinity` in the PodSpec. +You can specify node affinities using the `.spec.affinity.nodeAffinity` field in +your Pod spec. -Here's an example of a pod that uses node affinity: +For example, consider the following Pod spec: {{< codenew file="pods/pod-with-node-affinity.yaml" >}} -This node affinity rule says the pod can only be placed on a node with a label whose key is -`kubernetes.io/e2e-az-name` and whose value is either `e2e-az1` or `e2e-az2`. In addition, -among nodes that meet that criteria, nodes with a label whose key is `another-node-label-key` and whose -value is `another-node-label-value` should be preferred. +In this example, the following rules apply: -You can see the operator `In` being used in the example. The new node affinity syntax supports the following operators: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`. -You can use `NotIn` and `DoesNotExist` to achieve node anti-affinity behavior, or use -[node taints](/docs/concepts/scheduling-eviction/taint-and-toleration/) to repel pods from specific nodes. + * The node *must* have a label with the key `topology.kubernetes.io/zone` and + the value of that label *must* be either `antarctica-east1` or `antarctica-west1`. + * The node *preferably* has a label with the key `another-node-label-key` and + the value `another-node-label-value`. -If you specify both `nodeSelector` and `nodeAffinity`, *both* must be satisfied for the pod -to be scheduled onto a candidate node. +You can use the `operator` field to specify a logical operator for Kubernetes to use when +interpreting the rules. You can use `In`, `NotIn`, `Exists`, `DoesNotExist`, +`Gt` and `Lt`. -If you specify multiple `nodeSelectorTerms` associated with `nodeAffinity` types, then the pod can be scheduled onto a node **if one of the** `nodeSelectorTerms` can be satisfied. +`NotIn` and `DoesNotExist` allow you to define node anti-affinity behavior. +Alternatively, you can use [node taints](/docs/concepts/scheduling-eviction/taint-and-toleration/) +to repel Pods from specific nodes. -If you specify multiple `matchExpressions` associated with `nodeSelectorTerms`, then the pod can be scheduled onto a node **only if all** `matchExpressions` is satisfied. +{{<note>}} +If you specify both `nodeSelector` and `nodeAffinity`, *both* must be satisfied +for the Pod to be scheduled onto a node. -If you remove or change the label of the node where the pod is scheduled, the pod won't be removed. In other words, the affinity selection works only at the time of scheduling the pod. +If you specify multiple `nodeSelectorTerms` associated with `nodeAffinity` +types, then the Pod can be scheduled onto a node if one of the specified `nodeSelectorTerms` can be +satisfied. -The `weight` field in `preferredDuringSchedulingIgnoredDuringExecution` is in the range 1-100. For each node that meets all of the scheduling requirements (resource request, RequiredDuringScheduling affinity expressions, etc.), the scheduler will compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding MatchExpressions. This score is then combined with the scores of other priority functions for the node. The node(s) with the highest total score are the most preferred. +If you specify multiple `matchExpressions` associated with a single `nodeSelectorTerms`, +then the Pod can be scheduled onto a node only if all the `matchExpressions` are +satisfied. +{{</note>}} + +See [Assign Pods to Nodes using Node Affinity](/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/) +for more information. + +#### Node affinity weight + +You can specify a `weight` between 1 and 100 for each instance of the +`preferredDuringSchedulingIgnoredDuringExecution` affinity type. When the +scheduler finds nodes that meet all the other scheduling requirements of the Pod, the +scheduler iterates through every preferred rule that the node satisfies and adds the +value of the `weight` for that expression to a sum. + +The final sum is added to the score of other priority functions for the node. +Nodes with the highest total score are prioritized when the scheduler makes a +scheduling decision for the Pod. + +For example, consider the following Pod spec: + +{{< codenew file="pods/pod-with-affinity-anti-affinity.yaml" >}} + +If there are two possible nodes that match the +`preferredDuringSchedulingIgnoredDuringExecution` rule, one with the +`label-1:key-1` label and another with the `label-2:key-2` label, the scheduler +considers the `weight` of each node and adds the weight to the other scores for +that node, and schedules the Pod onto the node with the highest final score. + +{{<note>}} +If you want Kubernetes to successfully schedule the Pods in this example, you +must have existing nodes with the `kubernetes.io/os=linux` label. +{{</note>}} #### Node affinity per scheduling profile {{< feature-state for_k8s_version="v1.20" state="beta" >}} When configuring multiple [scheduling profiles](/docs/reference/scheduling/config/#multiple-profiles), you can associate -a profile with a Node affinity, which is useful if a profile only applies to a specific set of Nodes. -To do so, add an `addedAffinity` to the args of the [`NodeAffinity` plugin](/docs/reference/scheduling/config/#scheduling-plugins) +a profile with a node affinity, which is useful if a profile only applies to a specific set of nodes. +To do so, add an `addedAffinity` to the `args` field of the [`NodeAffinity` plugin](/docs/reference/scheduling/config/#scheduling-plugins) in the [scheduler configuration](/docs/reference/scheduling/config/). For example: ```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 +apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: @@ -188,29 +214,41 @@ profiles: The `addedAffinity` is applied to all Pods that set `.spec.schedulerName` to `foo-scheduler`, in addition to the NodeAffinity specified in the PodSpec. -That is, in order to match the Pod, Nodes need to satisfy `addedAffinity` and the Pod's `.spec.NodeAffinity`. +That is, in order to match the Pod, nodes need to satisfy `addedAffinity` and +the Pod's `.spec.NodeAffinity`. -Since the `addedAffinity` is not visible to end users, its behavior might be unexpected to them. We -recommend to use node labels that have clear correlation with the profile's scheduler name. +Since the `addedAffinity` is not visible to end users, its behavior might be +unexpected to them. Use node labels that have a clear correlation to the +scheduler profile name. {{< note >}} -The DaemonSet controller, which [creates Pods for DaemonSets](/docs/concepts/workloads/controllers/daemonset/#scheduled-by-default-scheduler) -is not aware of scheduling profiles. For this reason, it is recommended that you keep a scheduler profile, such as the -`default-scheduler`, without any `addedAffinity`. Then, the Daemonset's Pod template should use this scheduler name. -Otherwise, some Pods created by the Daemonset controller might remain unschedulable. +The DaemonSet controller, which [creates Pods for DaemonSets](/docs/concepts/workloads/controllers/daemonset/#scheduled-by-default-scheduler), +does not support scheduling profiles. When the DaemonSet controller creates +Pods, the default Kubernetes scheduler places those Pods and honors any +`nodeAffinity` rules in the DaemonSet controller. {{< /note >}} ### Inter-pod affinity and anti-affinity -Inter-pod affinity and anti-affinity allow you to constrain which nodes your pod is eligible to be scheduled *based on -labels on pods that are already running on the node* rather than based on labels on nodes. The rules are of the form -"this pod should (or, in the case of anti-affinity, should not) run in an X if that X is already running one or more pods that meet rule Y". -Y is expressed as a LabelSelector with an optional associated list of namespaces; unlike nodes, because pods are namespaced -(and therefore the labels on pods are implicitly namespaced), -a label selector over pod labels must specify which namespaces the selector should apply to. Conceptually X is a topology domain -like node, rack, cloud provider zone, cloud provider region, etc. You express it using a `topologyKey` which is the -key for the node label that the system uses to denote such a topology domain; for example, see the label keys listed above -in the section [Interlude: built-in node labels](#built-in-node-labels). +Inter-pod affinity and anti-affinity allow you to constrain which nodes your +Pods can be scheduled on based on the labels of **Pods** already running on that +node, instead of the node labels. + +Inter-pod affinity and anti-affinity rules take the form "this +Pod should (or, in the case of anti-affinity, should not) run in an X if that X +is already running one or more Pods that meet rule Y", where X is a topology +domain like node, rack, cloud provider zone or region, or similar and Y is the +rule Kubernetes tries to satisfy. + +You express these rules (Y) as [label selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) +with an optional associated list of namespaces. Pods are namespaced objects in +Kubernetes, so Pod labels also implicitly have namespaces. Any label selectors +for Pod labels should specify the namespaces in which Kubernetes should look for those +labels. + +You express the topology domain (X) using a `topologyKey`, which is the key for +the node label that the system uses to denote the domain. For examples, see +[Well-Known Labels, Annotations and Taints](/docs/reference/labels-annotations-taints/). {{< note >}} Inter-pod affinity and anti-affinity require substantial amount of @@ -219,80 +257,101 @@ not recommend using them in clusters larger than several hundred nodes. {{< /note >}} {{< note >}} -Pod anti-affinity requires nodes to be consistently labelled, in other words every node in the cluster must have an appropriate label matching `topologyKey`. If some or all nodes are missing the specified `topologyKey` label, it can lead to unintended behavior. +Pod anti-affinity requires nodes to be consistently labelled, in other words, +every node in the cluster must have an appropriate label matching `topologyKey`. +If some or all nodes are missing the specified `topologyKey` label, it can lead +to unintended behavior. {{< /note >}} -As with node affinity, there are currently two types of pod affinity and anti-affinity, called `requiredDuringSchedulingIgnoredDuringExecution` and -`preferredDuringSchedulingIgnoredDuringExecution` which denote "hard" vs. "soft" requirements. -See the description in the node affinity section earlier. -An example of `requiredDuringSchedulingIgnoredDuringExecution` affinity would be "co-locate the pods of service A and service B -in the same zone, since they communicate a lot with each other" -and an example `preferredDuringSchedulingIgnoredDuringExecution` anti-affinity would be "spread the pods from this service across zones" -(a hard requirement wouldn't make sense, since you probably have more pods than zones). +#### Types of inter-pod affinity and anti-affinity -Inter-pod affinity is specified as field `podAffinity` of field `affinity` in the PodSpec. -And inter-pod anti-affinity is specified as field `podAntiAffinity` of field `affinity` in the PodSpec. +Similar to [node affinity](#node-affinity) are two types of Pod affinity and +anti-affinity as follows: -#### An example of a pod that uses pod affinity: + * `requiredDuringSchedulingIgnoredDuringExecution` + * `preferredDuringSchedulingIgnoredDuringExecution` + +For example, you could use +`requiredDuringSchedulingIgnoredDuringExecution` affinity to tell the scheduler to +co-locate Pods of two services in the same cloud provider zone because they +communicate with each other a lot. Similarly, you could use +`preferredDuringSchedulingIgnoredDuringExecution` anti-affinity to spread Pods +from a service across multiple cloud provider zones. + +To use inter-pod affinity, use the `affinity.podAffinity` field in the Pod spec. +For inter-pod anti-affinity, use the `affinity.podAntiAffinity` field in the Pod +spec. + +#### Pod affinity example {#an-example-of-a-pod-that-uses-pod-affinity} + +Consider the following Pod spec: {{< codenew file="pods/pod-with-pod-affinity.yaml" >}} -The affinity on this pod defines one pod affinity rule and one pod anti-affinity rule. In this example, the -`podAffinity` is `requiredDuringSchedulingIgnoredDuringExecution` -while the `podAntiAffinity` is `preferredDuringSchedulingIgnoredDuringExecution`. The -pod affinity rule says that the pod can be scheduled onto a node only if that node is in the same zone -as at least one already-running pod that has a label with key "security" and value "S1". (More precisely, the pod is eligible to run -on node N if node N has a label with key `topology.kubernetes.io/zone` and some value V -such that there is at least one node in the cluster with key `topology.kubernetes.io/zone` and -value V that is running a pod that has a label with key "security" and value "S1".) The pod anti-affinity -rule says that the pod should not be scheduled onto a node if that node is in the same zone as a pod with -label having key "security" and value "S2". See the -[design doc](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md) -for many more examples of pod affinity and anti-affinity, both the `requiredDuringSchedulingIgnoredDuringExecution` -flavor and the `preferredDuringSchedulingIgnoredDuringExecution` flavor. +This example defines one Pod affinity rule and one Pod anti-affinity rule. The +Pod affinity rule uses the "hard" +`requiredDuringSchedulingIgnoredDuringExecution`, while the anti-affinity rule +uses the "soft" `preferredDuringSchedulingIgnoredDuringExecution`. -The legal operators for pod affinity and anti-affinity are `In`, `NotIn`, `Exists`, `DoesNotExist`. +The affinity rule says that the scheduler can only schedule a Pod onto a node if +the node is in the same zone as one or more existing Pods with the label +`security=S1`. More precisely, the scheduler must place the Pod on a node that has the +`topology.kubernetes.io/zone=V` label, as long as there is at least one node in +that zone that currently has one or more Pods with the Pod label `security=S1`. -In principle, the `topologyKey` can be any legal label-key. However, -for performance and security reasons, there are some constraints on topologyKey: +The anti-affinity rule says that the scheduler should try to avoid scheduling +the Pod onto a node that is in the same zone as one or more Pods with the label +`security=S2`. More precisely, the scheduler should try to avoid placing the Pod on a node that has the +`topology.kubernetes.io/zone=R` label if there are other nodes in the +same zone currently running Pods with the `Security=S2` Pod label. -1. For pod affinity, empty `topologyKey` is not allowed in both `requiredDuringSchedulingIgnoredDuringExecution` -and `preferredDuringSchedulingIgnoredDuringExecution`. -2. For pod anti-affinity, empty `topologyKey` is also not allowed in both `requiredDuringSchedulingIgnoredDuringExecution` -and `preferredDuringSchedulingIgnoredDuringExecution`. -3. For `requiredDuringSchedulingIgnoredDuringExecution` pod anti-affinity, the admission controller `LimitPodHardAntiAffinityTopology` was introduced to limit `topologyKey` to `kubernetes.io/hostname`. If you want to make it available for custom topologies, you may modify the admission controller, or disable it. -4. Except for the above cases, the `topologyKey` can be any legal label-key. +To get yourself more familiar with the examples of Pod affinity and anti-affinity, +refer to the [design proposal](https://git.k8s.io/design-proposals-archive/scheduling/podaffinity.md). -In addition to `labelSelector` and `topologyKey`, you can optionally specify a list `namespaces` -of namespaces which the `labelSelector` should match against (this goes at the same level of the definition as `labelSelector` and `topologyKey`). -If omitted or empty, it defaults to the namespace of the pod where the affinity/anti-affinity definition appears. +You can use the `In`, `NotIn`, `Exists` and `DoesNotExist` values in the +`operator` field for Pod affinity and anti-affinity. -All `matchExpressions` associated with `requiredDuringSchedulingIgnoredDuringExecution` affinity and anti-affinity -must be satisfied for the pod to be scheduled onto a node. +In principle, the `topologyKey` can be any allowed label key with the following +exceptions for performance and security reasons: + +* For Pod affinity and anti-affinity, an empty `topologyKey` field is not allowed in both `requiredDuringSchedulingIgnoredDuringExecution` + and `preferredDuringSchedulingIgnoredDuringExecution`. +* For `requiredDuringSchedulingIgnoredDuringExecution` Pod anti-affinity rules, + the admission controller `LimitPodHardAntiAffinityTopology` limits + `topologyKey` to `kubernetes.io/hostname`. You can modify or disable the + admission controller if you want to allow custom topologies. + +In addition to `labelSelector` and `topologyKey`, you can optionally specify a list +of namespaces which the `labelSelector` should match against using the +`namespaces` field at the same level as `labelSelector` and `topologyKey`. +If omitted or empty, `namespaces` defaults to the namespace of the Pod where the +affinity/anti-affinity definition appears. #### Namespace selector -{{< feature-state for_k8s_version="v1.22" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -Users can also select matching namespaces using `namespaceSelector`, which is a label query over the set of namespaces. -The affinity term is applied to the union of the namespaces selected by `namespaceSelector` and the ones listed in the `namespaces` field. +You can also select matching namespaces using `namespaceSelector`, which is a label query over the set of namespaces. +The affinity term is applied to namespaces selected by both `namespaceSelector` and the `namespaces` field. Note that an empty `namespaceSelector` ({}) matches all namespaces, while a null or empty `namespaces` list and -null `namespaceSelector` means "this pod's namespace". +null `namespaceSelector` matches the namespace of the Pod where the rule is defined. -This feature is beta and enabled by default. You can disable it via the -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`PodAffinityNamespaceSelector` in both kube-apiserver and kube-scheduler. +#### More practical use-cases -#### More Practical Use-cases +Inter-pod affinity and anti-affinity can be even more useful when they are used with higher +level collections such as ReplicaSets, StatefulSets, Deployments, etc. These +rules allow you to configure that a set of workloads should +be co-located in the same defined topology; for example, preferring to place two related +Pods onto the same node. -Interpod Affinity and AntiAffinity can be even more useful when they are used with higher -level collections such as ReplicaSets, StatefulSets, Deployments, etc. One can easily configure that a set of workloads should -be co-located in the same defined topology, eg., the same node. +For example: imagine a three-node cluster. You use the cluster to run a web application +and also an in-memory cache (such as Redis). For this example, also assume that latency between +the web application and the memory cache should be as low as is practical. You could use inter-pod +affinity and anti-affinity to co-locate the web servers with the cache as much as possible. -##### Always co-located in the same node - -In a three node cluster, a web application has in-memory cache such as redis. We want the web-servers to be co-located with the cache as much as possible. - -Here is the yaml snippet of a simple redis deployment with three replicas and selector label `app=store`. The deployment has `PodAntiAffinity` configured to ensure the scheduler does not co-locate replicas on a single node. +In the following example Deployment for the Redis cache, the replicas get the label `app=store`. The +`podAntiAffinity` rule tells the scheduler to avoid placing multiple replicas +with the `app=store` label on a single node. This creates each cache in a +separate node. ```yaml apiVersion: apps/v1 @@ -324,7 +383,10 @@ spec: image: redis:3.2-alpine ``` -The below yaml snippet of the webserver deployment has `podAntiAffinity` and `podAffinity` configured. This informs the scheduler that all its replicas are to be co-located with pods that have selector label `app=store`. This will also ensure that each web-server replica does not co-locate on a single node. +The following example Deployment for the web servers creates replicas with the label `app=web-store`. +The Pod affinity rule tells the scheduler to place each replica on a node that has a Pod +with the label `app=store`. The Pod anti-affinity rule tells the scheduler never to place +multiple `app=web-store` servers on a single node. ```yaml apiVersion: apps/v1 @@ -365,56 +427,41 @@ spec: image: nginx:1.16-alpine ``` -If we create the above two deployments, our three node cluster should look like below. +Creating the two preceding Deployments results in the following cluster layout, +where each web server is co-located with a cache, on three separate nodes. | node-1 | node-2 | node-3 | |:--------------------:|:-------------------:|:------------------:| | *webserver-1* | *webserver-2* | *webserver-3* | | *cache-1* | *cache-2* | *cache-3* | -As you can see, all the 3 replicas of the `web-server` are automatically co-located with the cache as expected. +The overall effect is that each cache instance is likely to be accessed by a single client, that +is running on the same node. This approach aims to minimize both skew (imbalanced load) and latency. -``` -kubectl get pods -o wide -``` -The output is similar to this: -``` -NAME READY STATUS RESTARTS AGE IP NODE -redis-cache-1450370735-6dzlj 1/1 Running 0 8m 10.192.4.2 kube-node-3 -redis-cache-1450370735-j2j96 1/1 Running 0 8m 10.192.2.2 kube-node-1 -redis-cache-1450370735-z73mh 1/1 Running 0 8m 10.192.3.1 kube-node-2 -web-server-1287567482-5d4dz 1/1 Running 0 7m 10.192.2.3 kube-node-1 -web-server-1287567482-6f7v5 1/1 Running 0 7m 10.192.4.3 kube-node-3 -web-server-1287567482-s330j 1/1 Running 0 7m 10.192.3.2 kube-node-2 -``` - -##### Never co-located in the same node - -The above example uses `PodAntiAffinity` rule with `topologyKey: "kubernetes.io/hostname"` to deploy the redis cluster so that -no two instances are located on the same host. -See [ZooKeeper tutorial](/docs/tutorials/stateful-application/zookeeper/#tolerating-node-failure) -for an example of a StatefulSet configured with anti-affinity for high availability, using the same technique. +You might have other reasons to use Pod anti-affinity. +See the [ZooKeeper tutorial](/docs/tutorials/stateful-application/zookeeper/#tolerating-node-failure) +for an example of a StatefulSet configured with anti-affinity for high +availability, using the same technique as this example. ## nodeName -`nodeName` is the simplest form of node selection constraint, but due -to its limitations it is typically not used. `nodeName` is a field of -PodSpec. If it is non-empty, the scheduler ignores the pod and the -kubelet running on the named node tries to run the pod. Thus, if -`nodeName` is provided in the PodSpec, it takes precedence over the -above methods for node selection. +`nodeName` is a more direct form of node selection than affinity or +`nodeSelector`. `nodeName` is a field in the Pod spec. If the `nodeName` field +is not empty, the scheduler ignores the Pod and the kubelet on the named node +tries to place the Pod on that node. Using `nodeName` overrules using +`nodeSelector` or affinity and anti-affinity rules. Some of the limitations of using `nodeName` to select nodes are: -- If the named node does not exist, the pod will not be run, and in +- If the named node does not exist, the Pod will not run, and in some cases may be automatically deleted. - If the named node does not have the resources to accommodate the - pod, the pod will fail and its reason will indicate why, + Pod, the Pod will fail and its reason will indicate why, for example OutOfmemory or OutOfcpu. - Node names in cloud environments are not always predictable or stable. -Here is an example of a pod config file using the `nodeName` field: +Here is an example of a Pod spec using the `nodeName` field: ```yaml apiVersion: v1 @@ -428,21 +475,26 @@ spec: nodeName: kube-01 ``` -The above pod will run on the node kube-01. +The above Pod will only run on the node `kube-01`. +## Pod topology spread constraints +You can use _topology spread constraints_ to control how {{< glossary_tooltip text="Pods" term_id="Pod" >}} +are spread across your cluster among failure-domains such as regions, zones, nodes, or among any other +topology domains that you define. You might do this to improve performance, expected availability, or +overall utilization. + +Read [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) +to learn more about how these work. ## {{% heading "whatsnext" %}} - -[Taints](/docs/concepts/scheduling-eviction/taint-and-toleration/) allow a Node to *repel* a set of Pods. - -The design documents for -[node affinity](https://git.k8s.io/community/contributors/design-proposals/scheduling/nodeaffinity.md) -and for [inter-pod affinity/anti-affinity](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md) contain extra background information about these features. - -Once a Pod is assigned to a Node, the kubelet runs the Pod and allocates node-local resources. -The [topology manager](/docs/tasks/administer-cluster/topology-manager/) can take part in node-level -resource allocation decisions. +* Read more about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) . +* Read the design docs for [node affinity](https://git.k8s.io/design-proposals-archive/scheduling/nodeaffinity.md) + and for [inter-pod affinity/anti-affinity](https://git.k8s.io/design-proposals-archive/scheduling/podaffinity.md). +* Learn about how the [topology manager](/docs/tasks/administer-cluster/topology-manager/) takes part in node-level + resource allocation decisions. +* Learn how to use [nodeSelector](/docs/tasks/configure-pod-container/assign-pods-nodes/). +* Learn how to use [affinity and anti-affinity](/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/). diff --git a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md index 916f050513..c27013f6b7 100644 --- a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md +++ b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md @@ -83,9 +83,9 @@ of the scheduler: ## {{% heading "whatsnext" %}} * Read about [scheduler performance tuning](/docs/concepts/scheduling-eviction/scheduler-perf-tuning/) -* Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) +* Read about [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) * Read the [reference documentation](/docs/reference/command-line-tools-reference/kube-scheduler/) for kube-scheduler -* Read the [kube-scheduler config (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) reference +* Read the [kube-scheduler config (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) reference * Learn about [configuring multiple schedulers](/docs/tasks/extend-kubernetes/configure-multiple-schedulers/) * Learn about [topology management policies](/docs/tasks/administer-cluster/topology-manager/) * Learn about [Pod Overhead](/docs/concepts/scheduling-eviction/pod-overhead/) diff --git a/content/en/docs/concepts/scheduling-eviction/node-pressure-eviction.md b/content/en/docs/concepts/scheduling-eviction/node-pressure-eviction.md index e832d1d48c..244298d150 100644 --- a/content/en/docs/concepts/scheduling-eviction/node-pressure-eviction.md +++ b/content/en/docs/concepts/scheduling-eviction/node-pressure-eviction.md @@ -66,8 +66,8 @@ the signal. The value for `memory.available` is derived from the cgroupfs instead of tools like `free -m`. This is important because `free -m` does not work in a -container, and if users use the [node -allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) feature, out of resource decisions +container, and if users use the [node allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +feature, out of resource decisions are made local to the end user Pod part of the cgroup hierarchy as well as the root node. This [script](/examples/admin/resource/memory-available.sh) reproduces the same set of steps that the kubelet performs to calculate @@ -85,10 +85,15 @@ The kubelet supports the following filesystem partitions: Kubelet auto-discovers these filesystems and ignores other filesystems. Kubelet does not support other configurations. -{{<note>}} -Some kubelet garbage collection features are deprecated in favor of eviction. -For a list of the deprecated features, see [kubelet garbage collection deprecation](/docs/concepts/cluster-administration/kubelet-garbage-collection/#deprecation). -{{</note>}} +Some kubelet garbage collection features are deprecated in favor of eviction: + +| Existing Flag | New Flag | Rationale | +| ------------- | -------- | --------- | +| `--image-gc-high-threshold` | `--eviction-hard` or `--eviction-soft` | existing eviction signals can trigger image garbage collection | +| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior | +| `--maximum-dead-containers` | - | deprecated once old logs are stored outside of container's context | +| `--maximum-dead-containers-per-container` | - | deprecated once old logs are stored outside of container's context | +| `--minimum-container-ttl-duration` | - | deprecated once old logs are stored outside of container's context | ### Eviction thresholds @@ -211,7 +216,7 @@ the kubelet frees up disk space in the following order: If the kubelet's attempts to reclaim node-level resources don't bring the eviction signal below the threshold, the kubelet begins to evict end-user pods. -The kubelet uses the following parameters to determine pod eviction order: +The kubelet uses the following parameters to determine the pod eviction order: 1. Whether the pod's resource usage exceeds requests 1. [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/) @@ -234,8 +239,8 @@ so the above scenario will not apply if the node is, for example, under `DiskPre `Guaranteed` pods are guaranteed only when requests and limits are specified for all the containers and they are equal. These pods will never be evicted because -of another pod's resource consumption. If a system daemon (such as `kubelet`, -`docker`, and `journald`) is consuming more resources than were reserved via +of another pod's resource consumption. If a system daemon (such as `kubelet` +and `journald`) is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations, and the node only has `Guaranteed` or `Burstable` pods using less resources than requests left on it, then the kubelet must choose to evict one of these pods to preserve node stability @@ -314,7 +319,7 @@ The kubelet sets an `oom_score_adj` value for each container based on the QoS fo {{<note>}} The kubelet also sets an `oom_score_adj` value of `-997` for containers in Pods that have -`system-node-critical` {{<glossary_tooltip text="Priority" term_id="pod-priority">}} +`system-node-critical` {{<glossary_tooltip text="Priority" term_id="pod-priority">}}. {{</note>}} If the kubelet can't reclaim memory before a node experiences OOM, the @@ -396,7 +401,7 @@ counted as `active_file`. If enough of these kernel block buffers are on the active LRU list, the kubelet is liable to observe this as high resource use and taint the node as experiencing memory pressure - triggering pod eviction. -For more more details, see [https://github.com/kubernetes/kubernetes/issues/43916](https://github.com/kubernetes/kubernetes/issues/43916) +For more details, see [https://github.com/kubernetes/kubernetes/issues/43916](https://github.com/kubernetes/kubernetes/issues/43916) You can work around that behavior by setting the memory limit and memory request the same for containers likely to perform intensive I/O activity. You will need diff --git a/content/en/docs/concepts/scheduling-eviction/pod-overhead.md b/content/en/docs/concepts/scheduling-eviction/pod-overhead.md index eebc235084..2610ea80ad 100644 --- a/content/en/docs/concepts/scheduling-eviction/pod-overhead.md +++ b/content/en/docs/concepts/scheduling-eviction/pod-overhead.md @@ -10,17 +10,12 @@ weight: 30 <!-- overview --> -{{< feature-state for_k8s_version="v1.18" state="beta" >}} - +{{< feature-state for_k8s_version="v1.24" state="stable" >}} When you run a Pod on a Node, the Pod itself takes an amount of system resources. These resources are additional to the resources needed to run the container(s) inside the Pod. -_Pod Overhead_ is a feature for accounting for the resources consumed by the Pod infrastructure -on top of the container requests & limits. - - - - +In Kubernetes, _Pod Overhead_ is a way to account for the resources consumed by the Pod +infrastructure on top of the container requests & limits. <!-- body --> @@ -29,33 +24,30 @@ In Kubernetes, the Pod's overhead is set at time according to the overhead associated with the Pod's [RuntimeClass](/docs/concepts/containers/runtime-class/). -When Pod Overhead is enabled, the overhead is considered in addition to the sum of container -resource requests when scheduling a Pod. Similarly, the kubelet will include the Pod overhead when sizing -the Pod cgroup, and when carrying out Pod eviction ranking. +A pod's overhead is considered in addition to the sum of container resource requests when +scheduling a Pod. Similarly, the kubelet will include the Pod overhead when sizing the Pod cgroup, +and when carrying out Pod eviction ranking. -## Enabling Pod Overhead {#set-up} +## Configuring Pod overhead {#set-up} -You need to make sure that the `PodOverhead` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled (it is on by default as of 1.18) -across your cluster, and a `RuntimeClass` is utilized which defines the `overhead` field. +You need to make sure a `RuntimeClass` is utilized which defines the `overhead` field. ## Usage example -To use the PodOverhead feature, you need a RuntimeClass that defines the `overhead` field. As -an example, you could use the following RuntimeClass definition with a virtualizing container runtime -that uses around 120MiB per Pod for the virtual machine and the guest OS: +To work with Pod overhead, you need a RuntimeClass that defines the `overhead` field. As +an example, you could use the following RuntimeClass definition with a virtualization container +runtime that uses around 120MiB per Pod for the virtual machine and the guest OS: ```yaml ---- -kind: RuntimeClass apiVersion: node.k8s.io/v1 +kind: RuntimeClass metadata: - name: kata-fc + name: kata-fc handler: kata-fc overhead: - podFixed: - memory: "120Mi" - cpu: "250m" + podFixed: + memory: "120Mi" + cpu: "250m" ``` Workloads which are created which specify the `kata-fc` RuntimeClass handler will take the memory and @@ -72,7 +64,7 @@ spec: runtimeClassName: kata-fc containers: - name: busybox-ctr - image: busybox + image: busybox:1.28 stdin: true tty: true resources: @@ -92,62 +84,70 @@ updates the workload's PodSpec to include the `overhead` as described in the Run the Pod will be rejected. In the given example, since only the RuntimeClass name is specified, the admission controller mutates the Pod to include an `overhead`. -After the RuntimeClass admission controller, you can check the updated PodSpec: +After the RuntimeClass admission controller has made modifications, you can check the updated +Pod overhead value: ```bash kubectl get pod test-pod -o jsonpath='{.spec.overhead}' ``` The output is: + ``` map[cpu:250m memory:120Mi] ``` -If a ResourceQuota is defined, the sum of container requests as well as the +If a [ResourceQuota](/docs/concepts/policy/resource-quotas/) is defined, the sum of container requests as well as the `overhead` field are counted. When the kube-scheduler is deciding which node should run a new Pod, the scheduler considers that Pod's `overhead` as well as the sum of container requests for that Pod. For this example, the scheduler adds the requests and the overhead, then looks for a node that has 2.25 CPU and 320 MiB of memory available. -Once a Pod is scheduled to a node, the kubelet on that node creates a new {{< glossary_tooltip text="cgroup" term_id="cgroup" >}} -for the Pod. It is within this pod that the underlying container runtime will create containers. +Once a Pod is scheduled to a node, the kubelet on that node creates a new {{< glossary_tooltip +text="cgroup" term_id="cgroup" >}} for the Pod. It is within this pod that the underlying +container runtime will create containers. -If the resource has a limit defined for each container (Guaranteed QoS or Bustrable QoS with limits defined), +If the resource has a limit defined for each container (Guaranteed QoS or Burstable QoS with limits defined), the kubelet will set an upper limit for the pod cgroup associated with that resource (cpu.cfs_quota_us for CPU and memory.limit_in_bytes memory). This upper limit is based on the sum of the container limits plus the `overhead` defined in the PodSpec. -For CPU, if the Pod is Guaranteed or Burstable QoS, the kubelet will set `cpu.shares` based on the sum of container -requests plus the `overhead` defined in the PodSpec. +For CPU, if the Pod is Guaranteed or Burstable QoS, the kubelet will set `cpu.shares` based on the +sum of container requests plus the `overhead` defined in the PodSpec. Looking at our example, verify the container requests for the workload: + ```bash kubectl get pod test-pod -o jsonpath='{.spec.containers[*].resources.limits}' ``` The total container requests are 2000m CPU and 200MiB of memory: + ``` map[cpu: 500m memory:100Mi] map[cpu:1500m memory:100Mi] ``` Check this against what is observed by the node: + ```bash kubectl describe node | grep test-pod -B2 ``` -The output shows 2250m CPU and 320MiB of memory are requested, which includes PodOverhead: +The output shows requests for 2250m CPU, and for 320MiB of memory. The requests include Pod overhead: + ``` - Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE - --------- ---- ------------ ---------- --------------- ------------- --- - default test-pod 2250m (56%) 2250m (56%) 320Mi (1%) 320Mi (1%) 36m + Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE + --------- ---- ------------ ---------- --------------- ------------- --- + default test-pod 2250m (56%) 2250m (56%) 320Mi (1%) 320Mi (1%) 36m ``` ## Verify Pod cgroup limits -Check the Pod's memory cgroups on the node where the workload is running. In the following example, [`crictl`](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md) +Check the Pod's memory cgroups on the node where the workload is running. In the following example, +[`crictl`](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md) is used on the node, which provides a CLI for CRI-compatible container runtimes. This is an -advanced example to show PodOverhead behavior, and it is not expected that users should need to check +advanced example to show Pod overhead behavior, and it is not expected that users should need to check cgroups directly on the node. First, on the particular node, determine the Pod identifier: @@ -158,17 +158,21 @@ POD_ID="$(sudo crictl pods --name test-pod -q)" ``` From this, you can determine the cgroup path for the Pod: + ```bash # Run this on the node where the Pod is scheduled sudo crictl inspectp -o=json $POD_ID | grep cgroupsPath ``` The resulting cgroup path includes the Pod's `pause` container. The Pod level cgroup is one directory above. + ``` - "cgroupsPath": "/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/7ccf55aee35dd16aca4189c952d83487297f3cd760f1bbf09620e206e7d0c27a" + "cgroupsPath": "/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/7ccf55aee35dd16aca4189c952d83487297f3cd760f1bbf09620e206e7d0c27a" ``` -In this specific case, the pod cgroup path is `kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2`. Verify the Pod level cgroup setting for memory: +In this specific case, the pod cgroup path is `kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2`. +Verify the Pod level cgroup setting for memory: + ```bash # Run this on the node where the Pod is scheduled. # Also, change the name of the cgroup to match the cgroup allocated for your pod. @@ -176,22 +180,20 @@ In this specific case, the pod cgroup path is `kubepods/podd7f4b509-cf94-4951-94 ``` This is 320 MiB, as expected: + ``` 335544320 ``` ### Observability -A `kube_pod_overhead` metric is available in [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) -to help identify when PodOverhead is being utilized and to help observe stability of workloads -running with a defined Overhead. This functionality is not available in the 1.9 release of -kube-state-metrics, but is expected in a following release. Users will need to build kube-state-metrics -from source in the meantime. - - +Some `kube_pod_overhead_*` metrics are available in [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) +to help identify when Pod overhead is being utilized and to help observe stability of workloads +running with a defined overhead. ## {{% heading "whatsnext" %}} +* Learn more about [RuntimeClass](/docs/concepts/containers/runtime-class/) +* Read the [PodOverhead Design](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/688-pod-overhead) + enhancement proposal for extra context -* [RuntimeClass](/docs/concepts/containers/runtime-class/) -* [PodOverhead Design](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/688-pod-overhead) diff --git a/content/en/docs/concepts/scheduling-eviction/pod-priority-preemption.md b/content/en/docs/concepts/scheduling-eviction/pod-priority-preemption.md index fff925b6c5..cee15e966e 100644 --- a/content/en/docs/concepts/scheduling-eviction/pod-priority-preemption.md +++ b/content/en/docs/concepts/scheduling-eviction/pod-priority-preemption.md @@ -104,9 +104,9 @@ description: "This priority class should be used for XYZ service pods only." ## Non-preempting PriorityClass {#non-preempting-priority-class} -{{< feature-state for_k8s_version="v1.19" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -Pods with `PreemptionPolicy: Never` will be placed in the scheduling queue +Pods with `preemptionPolicy: Never` will be placed in the scheduling queue ahead of lower-priority pods, but they cannot preempt other pods. A non-preempting pod waiting to be scheduled will stay in the scheduling queue, @@ -122,16 +122,16 @@ allowing other pods with lower priority to be scheduled before them. Non-preempting pods may still be preempted by other, high-priority pods. -`PreemptionPolicy` defaults to `PreemptLowerPriority`, +`preemptionPolicy` defaults to `PreemptLowerPriority`, which will allow pods of that PriorityClass to preempt lower-priority pods (as is existing default behavior). -If `PreemptionPolicy` is set to `Never`, +If `preemptionPolicy` is set to `Never`, pods in that PriorityClass will be non-preempting. An example use case is for data science workloads. A user may submit a job that they want to be prioritized above other workloads, but do not wish to discard existing work by preempting running pods. -The high priority job with `PreemptionPolicy: Never` will be scheduled +The high priority job with `preemptionPolicy: Never` will be scheduled ahead of other queued pods, as soon as sufficient cluster resources "naturally" become free. @@ -203,9 +203,10 @@ resources reserved for Pod P and also gives users information about preemptions in their clusters. Please note that Pod P is not necessarily scheduled to the "nominated Node". +The scheduler always tries the "nominated Node" before iterating over any other nodes. After victim Pods are preempted, they get their graceful termination period. If another node becomes available while scheduler is waiting for the victim Pods to -terminate, scheduler will use the other node to schedule Pod P. As a result +terminate, scheduler may use the other node to schedule Pod P. As a result `nominatedNodeName` and `nodeName` of Pod spec are not always the same. Also, if scheduler preempts Pods on Node N, but then a higher priority Pod than Pod P arrives, scheduler may give Node N to the new higher priority Pod. In such a diff --git a/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md b/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md index 17a426e35b..951d3f273d 100644 --- a/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md +++ b/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md @@ -3,70 +3,100 @@ reviewers: - bsalamat - k82cn - ahg-g -title: Resource Bin Packing for Extended Resources +title: Resource Bin Packing content_type: concept weight: 80 --- <!-- overview --> -{{< feature-state for_k8s_version="v1.16" state="alpha" >}} - -The kube-scheduler can be configured to enable bin packing of resources along -with extended resources using `RequestedToCapacityRatioResourceAllocation` -priority function. Priority functions can be used to fine-tune the -kube-scheduler as per custom needs. +In the [scheduling-plugin](/docs/reference/scheduling/config/#scheduling-plugins) `NodeResourcesFit` of kube-scheduler, there are two +scoring strategies that support the bin packing of resources: `MostAllocated` and `RequestedToCapacityRatio`. <!-- body --> -## Enabling Bin Packing using RequestedToCapacityRatioResourceAllocation +## Enabling bin packing using MostAllocated strategy +The `MostAllocated` strategy scores the nodes based on the utilization of resources, favoring the ones with higher allocation. +For each resource type, you can set a weight to modify its influence in the node score. -Kubernetes allows the users to specify the resources along with weights for +To set the `MostAllocated` strategy for the `NodeResourcesFit` plugin, use a +[scheduler configuration](/docs/reference/scheduling/config) similar to the following: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta3 +kind: KubeSchedulerConfiguration +profiles: +- pluginConfig: + - args: + scoringStrategy: + resources: + - name: cpu + weight: 1 + - name: memory + weight: 1 + - name: intel.com/foo + weight: 3 + - name: intel.com/bar + weight: 3 + type: MostAllocated + name: NodeResourcesFit +``` + +To learn more about other parameters and their default configuration, see the API documentation for +[`NodeResourcesFitArgs`](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs). + +## Enabling bin packing using RequestedToCapacityRatio + +The `RequestedToCapacityRatio` strategy allows the users to specify the resources along with weights for each resource to score nodes based on the request to capacity ratio. This allows users to bin pack extended resources by using appropriate parameters -and improves the utilization of scarce resources in large clusters. The -behavior of the `RequestedToCapacityRatioResourceAllocation` priority function -can be controlled by a configuration option called `RequestedToCapacityRatioArgs`. -This argument consists of two parameters `shape` and `resources`. The `shape` +to improve the utilization of scarce resources in large clusters. It favors nodes according to a +configured function of the allocated resources. The behavior of the `RequestedToCapacityRatio` in +the `NodeResourcesFit` score function can be controlled by the +[scoringStrategy](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy) field. +Within the `scoringStrategy` field, you can configure two parameters: `requestedToCapacityRatioParam` and +`resources`. The `shape` in `requestedToCapacityRatioParam` parameter allows the user to tune the function as least requested or most requested based on `utilization` and `score` values. The `resources` parameter consists of `name` of the resource to be considered during scoring and `weight` specify the weight of each resource. Below is an example configuration that sets -`requestedToCapacityRatioArguments` to bin packing behavior for extended -resources `intel.com/foo` and `intel.com/bar`. +the bin packing behavior for extended resources `intel.com/foo` and `intel.com/bar` +using the `requestedToCapacityRatio` field. ```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 +apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: -# ... - pluginConfig: - - name: RequestedToCapacityRatio - args: - shape: - - utilization: 0 - score: 10 - - utilization: 100 - score: 0 - resources: - - name: intel.com/foo - weight: 3 - - name: intel.com/bar - weight: 5 +- pluginConfig: + - args: + scoringStrategy: + resources: + - name: intel.com/foo + weight: 3 + - name: intel.com/bar + weight: 3 + requestedToCapacityRatioParam: + shape: + - utilization: 0 + score: 0 + - utilization: 100 + score: 10 + type: RequestedToCapacityRatio + name: NodeResourcesFit ``` Referencing the `KubeSchedulerConfiguration` file with the kube-scheduler flag `--config=/path/to/config/file` will pass the configuration to the scheduler. -**This feature is disabled by default** +To learn more about other parameters and their default configuration, see the API documentation for +[`NodeResourcesFitArgs`](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs). -### Tuning the Priority Function +### Tuning the score function -`shape` is used to specify the behavior of the -`RequestedToCapacityRatioPriority` function. +`shape` is used to specify the behavior of the `RequestedToCapacityRatio` function. ```yaml shape: diff --git a/content/en/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md b/content/en/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md index 5894398c9b..00302166ce 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md @@ -43,7 +43,7 @@ If you set `percentageOfNodesToScore` above 100, kube-scheduler acts as if you had set a value of 100. To change the value, edit the -[kube-scheduler configuration file](/docs/reference/config-api/kube-scheduler-config.v1beta2/) +[kube-scheduler configuration file](/docs/reference/config-api/kube-scheduler-config.v1beta3/) and then restart the scheduler. In many cases, the configuration file can be found at `/etc/kubernetes/config/kube-scheduler.yaml`. @@ -161,5 +161,5 @@ After going over all the Nodes, it goes back to Node 1. ## {{% heading "whatsnext" %}} -* Check the [kube-scheduler configuration reference (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) +* Check the [kube-scheduler configuration reference (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index e08052c017..52db68bf3d 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -52,7 +52,7 @@ equivalent to "Predicate" and "Scoring" is equivalent to "Priority function". One plugin may register at multiple extension points to perform more complex or stateful tasks. -{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" >}} +{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" class="diagram-large">}} ### QueueSort {#queue-sort} diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index 030f28e7d1..82e812c53b 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -15,15 +15,15 @@ is a property of {{< glossary_tooltip text="Pods" term_id="pod" >}} that *attrac a set of {{< glossary_tooltip text="nodes" term_id="node" >}} (either as a preference or a hard requirement). _Taints_ are the opposite -- they allow a node to repel a set of pods. -_Tolerations_ are applied to pods, and allow (but do not require) the pods to schedule -onto nodes with matching taints. +_Tolerations_ are applied to pods. Tolerations allow the scheduler to schedule pods with matching +taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also +[evaluates other parameters](/docs/concepts/scheduling-eviction/pod-priority-preemption/) +as part of its function. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. - - <!-- body --> ## Concepts @@ -267,7 +267,8 @@ This ensures that DaemonSet pods are never evicted due to these problems. ## Taint Nodes by Condition The control plane, using the node {{<glossary_tooltip text="controller" term_id="controller">}}, -automatically creates taints with a `NoSchedule` effect for [node conditions](/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions). +automatically creates taints with a `NoSchedule` effect for +[node conditions](/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions). The scheduler checks taints, not node conditions, when it makes scheduling decisions. This ensures that node conditions don't directly affect scheduling. @@ -298,7 +299,7 @@ arbitrary tolerations to DaemonSets. ## {{% heading "whatsnext" %}} -* Read about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/) and how you can configure it +* Read about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/) + and how you can configure it * Read about [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/) - diff --git a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md new file mode 100644 index 0000000000..77f4d1ea55 --- /dev/null +++ b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md @@ -0,0 +1,570 @@ +--- +title: Pod Topology Spread Constraints +content_type: concept +weight: 40 +--- + + +<!-- overview --> + +You can use _topology spread constraints_ to control how +{{< glossary_tooltip text="Pods" term_id="Pod" >}} are spread across your cluster +among failure-domains such as regions, zones, nodes, and other user-defined topology +domains. This can help to achieve high availability as well as efficient resource +utilization. + +You can set [cluster-level constraints](#cluster-level-default-constraints) as a default, +or configure topology spread constraints for individual workloads. + +<!-- body --> + +## Motivation + +Imagine that you have a cluster of up to twenty nodes, and you want to run a +{{< glossary_tooltip text="workload" term_id="workload" >}} +that automatically scales how many replicas it uses. There could be as few as +two Pods or as many as fifteen. +When there are only two Pods, you'd prefer not to have both of those Pods run on the +same node: you would run the risk that a single node failure takes your workload +offline. + +In addition to this basic usage, there are some advanced usage examples that +enable your workloads to benefit on high availability and cluster utilization. + +As you scale up and run more Pods, a different concern becomes important. Imagine +that you have three nodes running five Pods each. The nodes have enough capacity +to run that many replicas; however, the clients that interact with this workload +are split across three different datacenters (or infrastructure zones). Now you +have less concern about a single node failure, but you notice that latency is +higher than you'd like, and you are paying for network costs associated with +sending network traffic between the different zones. + +You decide that under normal operation you'd prefer to have a similar number of replicas +[scheduled](/docs/concepts/scheduling-eviction/) into each infrastructure zone, +and you'd like the cluster to self-heal in the case that there is a problem. + +Pod topology spread constraints offer you a declarative way to configure that. + + +## `topologySpreadConstraints` field + +The Pod API includes a field, `spec.topologySpreadConstraints`. Here is an example: + +```yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: example-pod +spec: + # Configure a topology spread constraint + topologySpreadConstraints: + - maxSkew: <integer> + minDomains: <integer> # optional; alpha since v1.24 + topologyKey: <string> + whenUnsatisfiable: <string> + labelSelector: <object> + ### other Pod fields go here +``` + +You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`. + +### Spread constraint definition + +You can define one or multiple `topologySpreadConstraints` entries to instruct the +kube-scheduler how to place each incoming Pod in relation to the existing Pods across +your cluster. Those fields are: + +- **maxSkew** describes the degree to which Pods may be unevenly distributed. You must + specify this field and the number must be greater than zero. Its semantics differ + according to the value of `whenUnsatisfiable`: + + - if you select `whenUnsatisfiable: DoNotSchedule`, then `maxSkew` defines the + maximum permitted difference between the number of matching pods in the target + topology and the _global minimum_ + (the minimum number of pods that match the label selector in a topology domain). + For example, if you have 3 zones with 2, 4 and 5 matching pods respectively, + then the global minimum is 2 and `maxSkew` is compared relative to that number. + - if you select `whenUnsatisfiable: ScheduleAnyway`, the scheduler gives higher + precedence to topologies that would help reduce the skew. + +- **minDomains** indicates a minimum number of eligible domains. This field is optional. + A domain is a particular instance of a topology. An eligible domain is a domain whose + nodes match the node selector. + + {{< note >}} + The `minDomains` field is an alpha field added in 1.24. You have to enable the + `MinDomainsInPodToplogySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) + in order to use it. + {{< /note >}} + + - The value of `minDomains` must be greater than 0, when specified. + You can only specify `minDomains` in conjunction with `whenUnsatisfiable: DoNotSchedule`. + - When the number of eligible domains with match topology keys is less than `minDomains`, + Pod topology spread treats global minimum as 0, and then the calculation of `skew` is performed. + The global minimum is the minimum number of matching Pods in an eligible domain, + or zero if the number of eligible domains is less than `minDomains`. + - When the number of eligible domains with matching topology keys equals or is greater than + `minDomains`, this value has no effect on scheduling. + - If you do not specify `minDomains`, the constraint behaves as if `minDomains` is 1. + +- **topologyKey** is the key of [node labels](#node-labels). If two Nodes are labelled + with this key and have identical values for that label, the scheduler treats both + Nodes as being in the same topology. The scheduler tries to place a balanced number + of Pods into each topology domain. + +- **whenUnsatisfiable** indicates how to deal with a Pod if it doesn't satisfy the spread constraint: + - `DoNotSchedule` (default) tells the scheduler not to schedule it. + - `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew. + +- **labelSelector** is used to find matching Pods. Pods + that match this label selector are counted to determine the + number of Pods in their corresponding topology domain. + See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) + for more details. + +When a Pod defines more than one `topologySpreadConstraint`, those constraints are +combined using a logical AND operation: the kube-scheduler looks for a node for the incoming Pod +that satisfies all the configured constraints. + +### Node labels + +Topology spread constraints rely on node labels to identify the topology +domain(s) that each {{< glossary_tooltip text="node" term_id="node" >}} is in. +For example, a node might have labels: +```yaml + region: us-east-1 + zone: us-east-1a +``` + +{{< note >}} +For brevity, this example doesn't use the +[well-known](/docs/reference/labels-annotations-taints/) label keys +`topology.kubernetes.io/zone` and `topology.kubernetes.io/region`. However, +those registered label keys are nonetheless recommended rather than the private +(unqualified) label keys `region` and `zone` that are used here. + +You can't make a reliable assumption about the meaning of a private label key +between different contexts. +{{< /note >}} + + +Suppose you have a 4-node cluster with the following labels: + +``` +NAME STATUS ROLES AGE VERSION LABELS +node1 Ready <none> 4m26s v1.16.0 node=node1,zone=zoneA +node2 Ready <none> 3m58s v1.16.0 node=node2,zone=zoneA +node3 Ready <none> 3m17s v1.16.0 node=node3,zone=zoneB +node4 Ready <none> 2m43s v1.16.0 node=node4,zone=zoneB +``` + +Then the cluster is logically viewed as below: + +{{<mermaid>}} +graph TB + subgraph "zoneB" + n3(Node3) + n4(Node4) + end + subgraph "zoneA" + n1(Node1) + n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4 k8s; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +## Consistency + +You should set the same Pod topology spread constraints on all pods in a group. + +Usually, if you are using a workload controller such as a Deployment, the pod template +takes care of this for you. If you mix different spread constraints then Kubernetes +follows the API definition of the field; however, the behavior is more likely to become +confusing and troubleshooting is less straightforward. + +You need a mechanism to ensure that all the nodes in a topology domain (such as a +cloud provider region) are labelled consistently. +To avoid you needing to manually label nodes, most clusters automatically +populate well-known labels such as `topology.kubernetes.io/hostname`. Check whether +your cluster supports this. + +## Topology spread constraint examples + +### Example: one topology spread constraint {#example-one-topologyspreadconstraint} + +Suppose you have a 4-node cluster where 3 Pods labelled `foo: bar` are located in +node1, node2 and node3 respectively: + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p3(Pod) --> n3(Node3) + n4(Node4) + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4,p1,p2,p3 k8s; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +If you want an incoming Pod to be evenly spread with existing Pods across zones, you +can use a manifest similar to: + +{{< codenew file="pods/topology-spread-constraints/one-constraint.yaml" >}} + +From that manifest, `topologyKey: zone` implies the even distribution will only be applied +to nodes that are labelled `zone: <any value>` (nodes that don't have a `zone` label +are skipped). The field `whenUnsatisfiable: DoNotSchedule` tells the scheduler to let the +incoming Pod stay pending if the scheduler can't find a way to satisfy the constraint. + +If the scheduler placed this incoming Pod into zone `A`, the distribution of Pods would +become `[3, 1]`. That means the actual skew is then 2 (calculated as `3 - 1`), which +violates `maxSkew: 1`. To satisfy the constraints and context for this example, the +incoming Pod can only be placed onto a node in zone `B`: + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p3(Pod) --> n3(Node3) + p4(mypod) --> n4(Node4) + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4,p1,p2,p3 k8s; + class p4 plain; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +OR + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p3(Pod) --> n3(Node3) + p4(mypod) --> n3 + n4(Node4) + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4,p1,p2,p3 k8s; + class p4 plain; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +You can tweak the Pod spec to meet various kinds of requirements: + +- Change `maxSkew` to a bigger value - such as `2` - so that the incoming Pod can + be placed into zone `A` as well. +- Change `topologyKey` to `node` so as to distribute the Pods evenly across nodes + instead of zones. In the above example, if `maxSkew` remains `1`, the incoming + Pod can only be placed onto the node `node4`. +- Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` + to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs + are satisfied). However, it's preferred to be placed into the topology domain which + has fewer matching Pods. (Be aware that this preference is jointly normalized + with other internal scheduling priorities such as resource usage ratio). + +### Example: multiple topology spread constraints {#example-multiple-topologyspreadconstraints} + +This builds upon the previous example. Suppose you have a 4-node cluster where 3 +existing Pods labeled `foo: bar` are located on node1, node2 and node3 respectively: + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p3(Pod) --> n3(Node3) + n4(Node4) + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4,p1,p2,p3 k8s; + class p4 plain; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +You can combine two topology spread constraints to control the spread of Pods both +by node and by zone: + +{{< codenew file="pods/topology-spread-constraints/two-constraints.yaml" >}} + +In this case, to match the first constraint, the incoming Pod can only be placed onto +nodes in zone `B`; while in terms of the second constraint, the incoming Pod can only be +scheduled to the node `node4`. The scheduler only considers options that satisfy all +defined constraints, so the only valid placement is onto node `node4`. + +### Example: conflicting topology spread constraints {#example-conflicting-topologyspreadconstraints} + +Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones: + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p4(Pod) --> n3(Node3) + p5(Pod) --> n3 + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n1 + p3(Pod) --> n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4,p1,p2,p3,p4,p5 k8s; + class zoneA,zoneB cluster; +{{< /mermaid >}} + +If you were to apply +[`two-constraints.yaml`](https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/topology-spread-constraints/two-constraints.yaml) +(the manifest from the previous example) +to **this** cluster, you would see that the Pod `mypod` stays in the `Pending` state. +This happens because: to satisfy the first constraint, the Pod `mypod` can only +be placed into zone `B`; while in terms of the second constraint, the Pod `mypod` +can only schedule to node `node2`. The intersection of the two constraints returns +an empty set, and the scheduler cannot place the Pod. + +To overcome this situation, you can either increase the value of `maxSkew` or modify +one of the constraints to use `whenUnsatisfiable: ScheduleAnyway`. Depending on +circumstances, you might also decide to delete an existing Pod manually - for example, +if you are troubleshooting why a bug-fix rollout is not making progress. + +#### Interaction with node affinity and node selectors + +The scheduler will skip the non-matching nodes from the skew calculations if the +incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. + +### Example: topology spread constraints with node affinity {#example-topologyspreadconstraints-with-nodeaffinity} + +Suppose you have a 5-node cluster ranging across zones A to C: + +{{<mermaid>}} +graph BT + subgraph "zoneB" + p3(Pod) --> n3(Node3) + n4(Node4) + end + subgraph "zoneA" + p1(Pod) --> n1(Node1) + p2(Pod) --> n2(Node2) + end + +classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; +classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; +class n1,n2,n3,n4,p1,p2,p3 k8s; +class p4 plain; +class zoneA,zoneB cluster; +{{< /mermaid >}} + +{{<mermaid>}} +graph BT + subgraph "zoneC" + n5(Node5) + end + +classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; +classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; +class n5 k8s; +class zoneC cluster; +{{< /mermaid >}} + +and you know that zone `C` must be excluded. In this case, you can compose a manifest +as below, so that Pod `mypod` will be placed into zone `B` instead of zone `C`. +Similarly, Kubernetes also respects `spec.nodeSelector`. + +{{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} + +## Implicit conventions + +There are some implicit conventions worth noting here: + +- Only the Pods holding the same namespace as the incoming Pod can be matching candidates. + +- The scheduler bypasses any nodes that don't have any `topologySpreadConstraints[*].topologyKey` + present. This implies that: + + 1. any Pods located on those bypassed nodes do not impact `maxSkew` calculation - in the + above example, suppose the node `node1` does not have a label "zone", then the 2 Pods will + be disregarded, hence the incoming Pod will be scheduled into zone `A`. + 2. the incoming Pod has no chances to be scheduled onto this kind of nodes - + in the above example, suppose a node `node5` has the **mistyped** label `zone-typo: zoneC` + (and no `zone` label set). After node `node5` joins the cluster, it will be bypassed and + Pods for this workload aren't scheduled there. + +- Be aware of what will happen if the incoming Pod's + `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the + above example, if you remove the incoming Pod's labels, it can still be placed onto + nodes in zone `B`, since the constraints are still satisfied. However, after that + placement, the degree of imbalance of the cluster remains unchanged - it's still zone `A` + having 2 Pods labelled as `foo: bar`, and zone `B` having 1 Pod labelled as + `foo: bar`. If this is not what you expect, update the workload's + `topologySpreadConstraints[*].labelSelector` to match the labels in the pod template. + +## Cluster-level default constraints + +It is possible to set default topology spread constraints for a cluster. Default +topology spread constraints are applied to a Pod if, and only if: + +- It doesn't define any constraints in its `.spec.topologySpreadConstraints`. +- It belongs to a Service, ReplicaSet, StatefulSet or ReplicationController. + +Default constraints can be set as part of the `PodTopologySpread` plugin +arguments in a [scheduling profile](/docs/reference/scheduling/config/#profiles). +The constraints are specified with the same [API above](#api), except that +`labelSelector` must be empty. The selectors are calculated from the Services, +ReplicaSets, StatefulSets or ReplicationControllers that the Pod belongs to. + +An example configuration might look like follows: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta3 +kind: KubeSchedulerConfiguration + +profiles: + - schedulerName: default-scheduler + pluginConfig: + - name: PodTopologySpread + args: + defaultConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway + defaultingType: List +``` + +{{< note >}} +The [`SelectorSpread` plugin](/docs/reference/scheduling/config/#scheduling-plugins) +is disabled by default. The Kubernetes project recommends using `PodTopologySpread` +to achieve similar behavior. +{{< /note >}} + +### Built-in default constraints {#internal-default-constraints} + +{{< feature-state for_k8s_version="v1.24" state="stable" >}} + +If you don't configure any cluster-level default constraints for pod topology spreading, +then kube-scheduler acts as if you specified the following default topology constraints: + +```yaml +defaultConstraints: + - maxSkew: 3 + topologyKey: "kubernetes.io/hostname" + whenUnsatisfiable: ScheduleAnyway + - maxSkew: 5 + topologyKey: "topology.kubernetes.io/zone" + whenUnsatisfiable: ScheduleAnyway +``` + +Also, the legacy `SelectorSpread` plugin, which provides an equivalent behavior, +is disabled by default. + +{{< note >}} +The `PodTopologySpread` plugin does not score the nodes that don't have +the topology keys specified in the spreading constraints. This might result +in a different default behavior compared to the legacy `SelectorSpread` plugin when +using the default topology constraints. + +If your nodes are not expected to have **both** `kubernetes.io/hostname` and +`topology.kubernetes.io/zone` labels set, define your own constraints +instead of using the Kubernetes defaults. +{{< /note >}} + +If you don't want to use the default Pod spreading constraints for your cluster, +you can disable those defaults by setting `defaultingType` to `List` and leaving +empty `defaultConstraints` in the `PodTopologySpread` plugin configuration: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta3 +kind: KubeSchedulerConfiguration + +profiles: + - schedulerName: default-scheduler + pluginConfig: + - name: PodTopologySpread + args: + defaultConstraints: [] + defaultingType: List +``` + +## Comparison with podAffinity and podAntiAffinity {#comparison-with-podaffinity-podantiaffinity} + +In Kubernetes, [inter-Pod affinity and anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) +control how Pods are scheduled in relation to one another - either more packed +or more scattered. + +`podAffinity` +: attracts Pods; you can try to pack any number of Pods into qualifying + topology domain(s) +`podAntiAffinity` +: repels Pods. If you set this to `requiredDuringSchedulingIgnoredDuringExecution` mode then + only a single Pod can be scheduled into a single topology domain; if you choose + `preferredDuringSchedulingIgnoredDuringExecution` then you lose the ability to enforce the + constraint. + +For finer control, you can specify topology spread constraints to distribute +Pods across different topology domains - to achieve either high availability or +cost-saving. This can also help on rolling update workloads and scaling out +replicas smoothly. + +For more context, see the +[Motivation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/895-pod-topology-spread#motivation) +section of the enhancement proposal about Pod topology spread constraints. + +## Known limitations + +- There's no guarantee that the constraints remain satisfied when Pods are removed. For + example, scaling down a Deployment may result in imbalanced Pods distribution. + + You can use a tool such as the [Descheduler](https://github.com/kubernetes-sigs/descheduler) + to rebalance the Pods distribution. +- Pods matched on tainted nodes are respected. + See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921). +- The scheduler doesn't have prior knowledge of all the zones or other topology + domains that a cluster has. They are determined from the existing nodes in the + cluster. This could lead to a problem in autoscaled clusters, when a node pool (or + node group) is scaled to zero nodes, and you're expecting the cluster to scale up, + because, in this case, those topology domains won't be considered until there is + at least one node in them. + You can work around this by using an cluster autoscaling tool that is aware of + Pod topology spread constraints and is also aware of the overall set of topology + domains. + + +## {{% heading "whatsnext" %}} + +- The blog article [Introducing PodTopologySpread](/blog/2020/05/introducing-podtopologyspread/) + explains `maxSkew` in some detail, as well as covering some advanced usage examples. +- Read the [scheduling](/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling) section of + the API reference for Pod. diff --git a/content/en/docs/concepts/security/controlling-access.md b/content/en/docs/concepts/security/controlling-access.md index 1a0c93d8cf..1718aa4a54 100644 --- a/content/en/docs/concepts/security/controlling-access.md +++ b/content/en/docs/concepts/security/controlling-access.md @@ -4,6 +4,7 @@ reviewers: - lavalamp title: Controlling Access to the Kubernetes API content_type: concept +weight: 50 --- <!-- overview --> @@ -22,10 +23,11 @@ following diagram: ## Transport security -In a typical Kubernetes cluster, the API serves on port 443, protected by TLS. +By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS. In a typical production Kubernetes cluster, the API serves on port 443. The port can be changed with the `--secure-port`, and the listening IP address with the `--bind-address` flag. + The API server presents a certificate. This certificate may be signed using a private certificate authority (CA), or based on a public key infrastructure linked -to a generally recognized CA. +to a generally recognized CA. The certificate and corresponding private key can be set by using the `--tls-cert-file` and `--tls-private-key-file` flags. If your cluster uses a private certificate authority, you need a copy of that CA certificate configured into your `~/.kube/config` on the client, so that you can @@ -129,34 +131,12 @@ The available Admission Control modules are described in [Admission Controllers] Once a request passes all admission controllers, it is validated using the validation routines for the corresponding API object, and then written to the object store (shown as step **4**). +## Auditing -## API server ports and IPs +Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. +The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself. -The previous discussion applies to requests sent to the secure port of the API server -(the typical case). The API server can actually serve on 2 ports: - -By default, the Kubernetes API server serves HTTP on 2 ports: - - 1. `localhost` port: - - - is intended for testing and bootstrap, and for other components of the master node - (scheduler, controller-manager) to talk to the API - - no TLS - - default is port 8080 - - default IP is localhost, change with `--insecure-bind-address` flag. - - request **bypasses** authentication and authorization modules. - - request handled by admission control module(s). - - protected by need to have host access - - 2. “Secure port”: - - - use whenever possible - - uses TLS. Set cert with `--tls-cert-file` and key with `--tls-private-key-file` flag. - - default is port 6443, change with `--secure-port` flag. - - default IP is first non-localhost network interface, change with `--bind-address` flag. - - request handled by authentication and authorization modules. - - request handled by admission control module(s). - - authentication and authorization modules run. +For more information, see [Auditing](/docs/tasks/debug/debug-cluster/audit/). ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/concepts/security/multi-tenancy.md b/content/en/docs/concepts/security/multi-tenancy.md new file mode 100755 index 0000000000..a1c70fe370 --- /dev/null +++ b/content/en/docs/concepts/security/multi-tenancy.md @@ -0,0 +1,271 @@ +--- +title: Multi-tenancy +content_type: concept +weight: 70 +--- + +<!-- overview --> + +This page provides an overview of available configuration options and best practices for cluster multi-tenancy. + +Sharing clusters saves costs and simplifies administration. However, sharing clusters also presents challenges such as security, fairness, and managing _noisy neighbors_. + +Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term _multi-tenancy_. + +While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below. + +<!-- body --> +## Use cases + +The first step to determining how to share your cluster is understanding your use case, so you can evaluate the patterns and tools available. In general, multi-tenancy in Kubernetes clusters falls into two broad categories, though many variations and hybrids are also possible. + +### Multiple teams + +A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. These workloads frequently need to communicate with each other, and with other workloads located on the same or different clusters. + +In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as `kubectl`, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters. + +### Multiple customers + +The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." However, a better term might be "multi-customer tenancy,” since SaaS vendors may also use other deployment models, and this deployment model can also be used outside of SaaS. + + +In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other. + + +## Terminology + +### Tenants + +When discussing multi-tenancy in Kubernetes, there is no single definition for a "tenant". Rather, the definition of a tenant will vary depending on whether multi-team or multi-customer tenancy is being discussed. + +In multi-team usage, a tenant is typically a team, where each team typically deploys a small number of workloads that scales with the complexity of the service. However, the definition of "team" may itself be fuzzy, as teams may be organized into higher-level divisions or subdivided into smaller teams. + + +By contrast, if each team deploys dedicated workloads for each new client, they are using a multi-customer model of tenancy. In this case, a "tenant" is simply a group of users who share a single workload. This may be as large as an entire company, or as small as a single team at that company. + +In many cases, the same organization may use both definitions of "tenants" in different contexts. For example, a platform team may offer shared services such as security tools and databases to multiple internal “customers” and a SaaS vendor may also have multiple teams sharing a development cluster. Finally, hybrid architectures are also possible, such as a SaaS provider using a combination of per-customer workloads for sensitive data, combined with multi-tenant shared services. + + +{{< figure src="/images/docs/multi-tenancy.png" title="A cluster showing coexisting tenancy models" class="diagram-large" >}} + + +### Isolation + +There are several ways to design and build multi-tenant solutions with Kubernetes. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service. + + +A Kubernetes cluster consists of a control plane which runs Kubernetes software, and a data plane consisting of worker nodes where tenant workloads are executed as pods. Tenant isolation can be applied in both the control plane and the data plane based on organizational requirements. + +The level of isolation offered is sometimes described using terms like “hard” multi-tenancy, which implies strong isolation, and “soft” multi-tenancy, which implies weaker isolation. In particular, "hard" multi-tenancy is often used to describe cases where the tenants do not trust each other, often from security and resource sharing perspectives (e.g. guarding against attacks such as data exfiltration or DoS). Since data planes typically have much larger attack surfaces, "hard" multi-tenancy often requires extra attention to isolating the data-plane, though control plane isolation also remains critical. + +However, the terms "hard" and "soft" can often be confusing, as there is no single definition that will apply to all users. Rather, "hardness" or "softness" is better understood as a broad spectrum, with many different techniques that can be used to maintain different types of isolation in your clusters, based on your requirements. + + +In more extreme cases, it may be easier or necessary to forgo any cluster-level sharing at all and assign each tenant their dedicated cluster, possibly even running on dedicated hardware if VMs are not considered an adequate security boundary. This may be easier with managed Kubernetes clusters, where the overhead of creating and operating clusters is at least somewhat taken on by a cloud provider. The benefit of stronger tenant isolation must be evaluated against the cost and complexity of managing multiple clusters. The [Multi-cluster SIG](https://git.k8s.io/community/sig-multicluster/README.md) is responsible for addressing these types of use cases. + + + +The remainder of this page focuses on isolation techniques used for shared Kubernetes clusters. However, even if you are considering dedicated clusters, it may be valuable to review these recommendations, as it will give you the flexibility to shift to shared clusters in the future if your needs or capabilities change. + + +## Control plane isolation + +Control plane isolation ensures that different tenants cannot access or affect each others' Kubernetes API resources. + +### Namespaces + +In Kubernetes, a {{< glossary_tooltip text="Namespace" term_id="namespace" >}} provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions: + +1. Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing. + +2. Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace. + +In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. In fact, a common practice is to isolate every workload in its own namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy. + +The namespace isolation model requires configuration of several other Kubernetes resources, networking plugins, and adherence to security best practices to properly isolate tenant workloads. These considerations are discussed below. + +### Access controls + +The most important type of isolation for the control plane is authorization. If teams or their workloads can access or modify each others' API resources, they can change or disable all other types of policies thereby negating any protection those policies may offer. As a result, it is critical to ensure that each tenant has the appropriate access to only the namespaces they need, and no more. This is known as the "Principle of Least Privilege." + + +Role-based access control (RBAC) is commonly used to enforce authorization in the Kubernetes control plane, for both users and workloads (service accounts). [Roles](/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) and [role bindings](/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) are Kubernetes objects that are used at a namespace level to enforce access control in your application; similar objects exist for authorizing access to cluster-level objects, though these are less useful for multi-tenant clusters. + +In a multi-team environment, RBAC must be used to restrict tenants' access to the appropriate namespaces, and ensure that cluster-wide resources can only be accessed or modified by privileged users such as cluster administrators. + +If a policy ends up granting a user more permissions than they need, this is likely a signal that the namespace containing the affected resources should be refactored into finer-grained namespaces. Namespace management tools may simplify the management of these finer-grained namespaces by applying common RBAC policies to different namespaces, while still allowing fine-grained policies where necessary. + +### Quotas + +Kubernetes workloads consume node resources, like CPU and memory. In a multi-tenant environment, you can use +[Resource Quotas](/docs/concepts/policy/resource-quotas/) to manage resource usage of tenant workloads. +For the multiple teams use case, where tenants have access to the Kubernetes API, you can use resource quotas +to limit the number of API resources (for example: the number of Pods, or the number of ConfigMaps) +that a tenant can create. Limits on object count ensure fairness and aim to avoid _noisy neighbor_ issues from +affecting other tenants that share a control plane. + +Resource quotas are namespaced objects. By mapping tenants to namespaces, cluster admins can use quotas to ensure that a tenant cannot monopolize a cluster's resources or overwhelm its control plane. Namespace management tools simplify the administration of quotas. In addition, while Kubernetes quotas only apply within a single namespace, some namespace management tools allow groups of namespaces to share quotas, giving administrators far more flexibility with less effort than built-in quotas. + +Quotas prevent a single tenant from consuming greater than their allocated share of resources hence minimizing the “noisy neighbor” issue, where one tenant negatively impacts the performance of other tenants' workloads. + +When you apply a quota to namespace, Kubernetes requires you to also specify resource requests and limits for each container. Limits are the upper bound for the amount of resources that a container can consume. Containers that attempt to consume resources that exceed the configured limits will either be throttled or killed, based on the resource type. When resource requests are set lower than limits, each container is guaranteed the requested amount but there may still be some potential for impact across workloads. + +Quotas cannot protect against all kinds of resource sharing, such as network traffic. Node isolation (described below) may be a better solution for this problem. + +## Data Plane Isolation + +Data plane isolation ensures that pods and workloads for different tenants are sufficiently isolated. + +### Network isolation + +By default, all pods in a Kubernetes cluster are allowed to communicate with each other, and all network traffic is unencrypted. This can lead to security vulnerabilities where traffic is accidentally or maliciously sent to an unintended destination, or is intercepted by a workload on a compromised node. + +Pod-to-pod communication can be controlled using [Network Policies](/docs/concepts/services-networking/network-policies/), which restrict communication between pods using namespace labels or IP address ranges. In a multi-tenant environment where strict network isolation between tenants is required, starting with a default policy that denies communication between pods is recommended with another rule that allows all pods to query the DNS server for name resolution. With such a default policy in place, you can begin adding more permissive rules that allow for communication within a namespace. This scheme can be further refined as required. Note that this only applies to pods within a single control plane; pods that belong to different virtual control planes cannot talk to each other via Kubernetes networking. + +Namespace management tools may simplify the creation of default or common network policies. In addition, some of these tools allow you to enforce a consistent set of namespace labels across your cluster, ensuring that they are a trusted basis for your policies. + +{{< warning >}} +Network policies require a [CNI plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni) that supports the implementation of network policies. Otherwise, NetworkPolicy resources will be ignored. +{{< /warning >}} + +More advanced network isolation may be provided by service meshes, which provide OSI Layer 7 policies based on workload identity, in addition to namespaces. These higher-level policies can make it easier to manage namespace-based multi-tenancy, especially when multiple namespaces are dedicated to a single tenant. They frequently also offer encryption using mutual TLS, protecting your data even in the presence of a compromised node, and work across dedicated or virtual clusters. However, they can be significantly more complex to manage and may not be appropriate for all users. + +### Storage isolation + +Kubernetes offers several types of volumes that can be used as persistent storage for workloads. For security and data-isolation, [dynamic volume provisioning](/docs/concepts/storage/dynamic-provisioning/) is recommended and volume types that use node resources should be avoided. + +[StorageClasses](/docs/concepts/storage/storage-classes/) allow you to describe custom "classes" of storage offered by your cluster, based on quality-of-service levels, backup policies, or custom policies determined by the cluster administrators. + +Pods can request storage using a [PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/). A PersistentVolumeClaim is a namespaced resource, which enables isolating portions of the storage system and dedicating it to tenants within the shared Kubernetes cluster. However, it is important to note that a PersistentVolume is a cluster-wide resource and has a lifecycle independent of workloads and namespaces. + +For example, you can configure a separate StorageClass for each tenant and use this to strengthen isolation. +If a StorageClass is shared, you should set a [reclaim policy of `Delete`](/docs/concepts/storage/storage-classes/#reclaim-policy) +to ensure that a PersistentVolume cannot be reused across different namespaces. + +### Sandboxing containers + +{{% thirdparty-content %}} + +Kubernetes pods are composed of one or more containers that execute on worker nodes. Containers utilize OS-level virtualization and hence offer a weaker isolation boundary than virtual machines that utilize hardware-based virtualization. + +In a shared environment, unpatched vulnerabilities in the application and system layers can be exploited by attackers for container breakouts and remote code execution that allow access to host resources. In some applications, like a Content Management System (CMS), customers may be allowed the ability to upload and execute untrusted scripts or code. In either case, mechanisms to further isolate and protect workloads using strong isolation are desirable. + +Sandboxing provides a way to isolate workloads running in a shared cluster. It typically involves running each pod in a separate execution environment such as a virtual machine or a userspace kernel. Sandboxing is often recommended when you are running untrusted code, where workloads are assumed to be malicious. Part of the reason this type of isolation is necessary is because containers are processes running on a shared kernel; they mount file systems like /sys and /proc from the underlying host, making them less secure than an application that runs on a virtual machine which has its own kernel. While controls such as seccomp, AppArmor, and SELinux can be used to strengthen the security of containers, it is hard to apply a universal set of rules to all workloads running in a shared cluster. Running workloads in a sandbox environment helps to insulate the host from container escapes, where an attacker exploits a vulnerability to gain access to the host system and all the processes/files running on that host. + +Virtual machines and userspace kernels are 2 popular approaches to sandboxing. The following sandboxing implementations are available: +* [gVisor](https://gvisor.dev/) intercepts syscalls from containers and runs them through a userspace kernel, written in Go, with limited access to the underlying host. +* [Kata Containers](https://katacontainers.io/) is an OCI compliant runtime that allows you to run containers in a VM. The hardware virtualization available in Kata offers an added layer of security for containers running untrusted code. + +### Node Isolation + +Node isolation is another technique that you can use to isolate tenant workloads from each other. With node isolation, a set of nodes is dedicated to running pods from a particular tenant and co-mingling of tenant pods is prohibited. This configuration reduces the noisy tenant issue, as all pods running on a node will belong to a single tenant. The risk of information disclosure is slightly lower with node isolation because an attacker that manages to escape from a container will only have access to the containers and volumes mounted to that node. + +Although workloads from different tenants are running on different nodes, it is important to be aware that the kubelet and (unless using virtual control planes) the API service are still shared services. A skilled attacker could use the permissions assigned to the kubelet or other pods running on the node to move laterally within the cluster and gain access to tenant workloads running on other nodes. If this is a major concern, consider implementing compensating controls such as seccomp, AppArmor or SELinux or explore using sandboxed containers or creating separate clusters for each tenant. + +Node isolation is a little easier to reason about from a billing standpoint than sandboxing containers since you can charge back per node rather than per pod. It also has fewer compatibility and performance issues and may be easier to implement than sandboxing containers. For example, nodes for each tenant can be configured with taints so that only pods with the corresponding toleration can run on them. A mutating webhook could then be used to automatically add tolerations and node affinities to pods deployed into tenant namespaces so that they run on a specific set of nodes designated for that tenant. + +Node isolation can be implemented using an [pod node selectors](/docs/concepts/scheduling-eviction/assign-pod-node/) or a [Virtual Kubelet](https://github.com/virtual-kubelet). + +## Additional Considerations + +This section discusses other Kubernetes constructs and patterns that are relevant for multi-tenancy. + +### API Priority and Fairness + +[API priority and fairness](/docs/concepts/cluster-administration/flow-control/) is a Kubernetes feature that allows you to assign a priority to certain pods running within the cluster. When an application calls the Kubernetes API, the API server evaluates the priority assigned to pod. Calls from pods with higher priority are fulfilled before those with a lower priority. When contention is high, lower priority calls can be queued until the server is less busy or you can reject the requests. + +Using API priority and fairness will not be very common in SaaS environments unless you are allowing customers to run applications that interface with the Kubernetes API, e.g. a controller. + +### Quality-of-Service (QoS) {#qos} + +When you’re running a SaaS application, you may want the ability to offer different Quality-of-Service (QoS) tiers of service to different tenants. For example, you may have freemium service that comes with fewer performance guarantees and features and a for-fee service tier with specific performance guarantees. Fortunately, there are several Kubernetes constructs that can help you accomplish this within a shared cluster, including network QoS, storage classes, and pod priority and preemption. The idea with each of these is to provide tenants with the quality of service that they paid for. Let’s start by looking at networking QoS. + +Typically, all pods on a node share a network interface. Without network QoS, some pods may consume an unfair share of the available bandwidth at the expense of other pods. The Kubernetes [bandwidth plugin](https://www.cni.dev/plugins/current/meta/bandwidth/) creates an [extended resource](/docs/concepts/configuration/manage-resources-containers/#extended-resources) for networking that allows you to use Kubernetes resources constructs, i.e. requests/limits, to apply rate limits to pods by using Linux tc queues. Be aware that the plugin is considered experimental as per the [Network Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping) documentation and should be thoroughly tested before use in production environments. + +For storage QoS, you will likely want to create different storage classes or profiles with different performance characteristics. Each storage profile can be associated with a different tier of service that is optimized for different workloads such IO, redundancy, or throughput. Additional logic might be necessary to allow the tenant to associate the appropriate storage profile with their workload. + +Finally, there’s [pod priority and preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) where you can assign priority values to pods. When scheduling pods, the scheduler will try evicting pods with lower priority when there are insufficient resources to schedule pods that are assigned a higher priority. If you have a use case where tenants have different service tiers in a shared cluster e.g. free and paid, you may want to give higher priority to certain tiers using this feature. + +### DNS + +Kubernetes clusters include a Domain Name System (DNS) service to provide translations from names to IP addresses, for all Services and Pods. By default, the Kubernetes DNS service allows lookups across all namespaces in the cluster. + +In multi-tenant environments where tenants can access pods and other Kubernetes resources, or where +stronger isolation is required, it may be necessary to prevent pods from looking up services in other +Namespaces. +You can restrict cross-namespace DNS lookups by configuring security rules for the DNS service. +For example, CoreDNS (the default DNS service for Kubernetes) can leverage Kubernetes metadata +to restrict queries to Pods and Services within a namespace. For more information, read an +[example](https://github.com/coredns/policy#kubernetes-metadata-multi-tenancy-policy) of configuring +this within the CoreDNS documentation. + +When a [Virtual Control Plane per tenant](#virtual-control-plane-per-tenant) model is used, a DNS service must be configured per tenant or a multi-tenant DNS service must be used. Here is an example of a [customized version of CoreDNS](https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/tenant-dns.md) that supports multiple tenants. + +### Operators + +[Operators](/docs/concepts/extend-kubernetes/operator/) are Kubernetes controllers that manage applications. Operators can simplify the management of multiple instances of an application, like a database service, which makes them a common building block in the multi-consumer (SaaS) multi-tenancy use case. + +Operators used in a multi-tenant environment should follow a stricter set of guidelines. Specifically, the Operator should: +* Support creating resources within different tenant namespaces, rather than just in the namespace in which the Operator is deployed. +* Ensure that the Pods are configured with resource requests and limits, to ensure scheduling and fairness. +* Support configuration of Pods for data-plane isolation techniques such as node isolation and sandboxed containers. + +## Implementations + +{{% thirdparty-content %}} + +There are two primary ways to share a Kubernetes cluster for multi-tenancy: using Namespaces (i.e. a Namespace per tenant) or by virtualizing the control plane (i.e. Virtual control plane per tenant). + +In both cases, data plane isolation, and management of additional considerations such as API Priority and Fairness, is also recommended. + +Namespace isolation is well-supported by Kubernetes, has a negligible resource cost, and provides mechanisms to allow tenants to interact appropriately, such as by allowing service-to-service communication. However, it can be difficult to configure, and doesn't apply to Kubernetes resources that can't be namespaced, such as Custom Resource Definitions, Storage Classes, and Webhooks. + +Control plane virtualization allows for isolation of non-namespaced resources at the cost of somewhat higher resource usage and more difficult cross-tenant sharing. It is a good option when namespace isolation is insufficient but dedicated clusters are undesirable, due to the high cost of maintaining them (especially on-prem) or due to their higher overhead and lack of resource sharing. However, even within a virtualized control plane, you will likely see benefits by using namespaces as well. + +The two options are discussed in more detail in the following sections: + +### Namespace per tenant + +As previously mentioned, you should consider isolating each workload in its own namespace, even if you are using dedicated clusters or virtualized control planes. This ensures that each workload only has access to its own resources, such as Config Maps and Secrets, and allows you to tailor dedicated security policies for each workload. In addition, it is a best practice to give each namespace names that are unique across your entire fleet (i.e., even if they are in separate clusters), as this gives you the flexibility to switch between dedicated and shared clusters in the future, or to use multi-cluster tooling such as service meshes. + +Conversely, there are also advantages to assigning namespaces at the tenant level, not just the workload level, since there are often policies that apply to all workloads owned by a single tenant. However, this raises its own problems. Firstly, this makes it difficult or impossible to customize policies to individual workloads, and secondly, it may be challenging to come up with a single level of "tenancy" that should be given a namespace. For example, an organization may have divisions, teams, and subteams - which should be assigned a namespace? + +To solve this, Kubernetes provides the [Hierarchical Namespace Controller (HNC)](https://github.com/kubernetes-sigs/hierarchical-namespaces), which allows you to organize your namespaces into hierarchies, and share certain policies and resources between them. It also helps you manage namespace labels, namespace lifecycles, and delegated management, and share resource quotas across related namespaces. These capabilities can be useful in both multi-team and multi-customer scenarios. + +Other projects that provide similar capabilities and aid in managing namespaced resources are listed below: + +#### Multi-team tenancy + +* [Capsule](https://github.com/clastix/capsule) +* [Kiosk](https://github.com/loft-sh/kiosk) + +#### Multi-customer tenancy + +* [Kubeplus](https://github.com/cloud-ark/kubeplus) + +#### Policy engines + +Policy engines provide features to validate and generate tenant configurations: + +* [Kyverno](https://kyverno.io/) +* [OPA/Gatekeeper](https://github.com/open-policy-agent/gatekeeper) + +### Virtual control plane per tenant + +Another form of control-plane isolation is to use Kubernetes extensions to provide each tenant a virtual control-plane that enables segmentation of cluster-wide API resources. [Data plane isolation](#data-plane-isolation) techniques can be used with this model to securely manage worker nodes across tenants. + +The virtual control plane based multi-tenancy model extends namespace-based multi-tenancy by providing each tenant with dedicated control plane components, and hence complete control over cluster-wide resources and add-on services. Worker nodes are shared across all tenants, and are managed by a Kubernetes cluster that is normally inaccessible to tenants. This cluster is often referred to as a _super-cluster_ (or sometimes as a _host-cluster_). Since a tenant’s control-plane is not directly associated with underlying compute resources it is referred to as a _virtual control plane_. + +A virtual control plane typically consists of the Kubernetes API server, the controller manager, and the etcd data store. It interacts with the super cluster via a metadata synchronization controller which coordinates changes across tenant control planes and the control plane of the super--cluster. + +By using per-tenant dedicated control planes, most of the isolation problems due to sharing one API server among all tenants are solved. Examples include noisy neighbors in the control plane, uncontrollable blast radius of policy misconfigurations, and conflicts between cluster scope objects such as webhooks and CRDs. Hence, the virtual control plane model is particularly suitable for cases where each tenant requires access to a Kubernetes API server and expects the full cluster manageability. + +The improved isolation comes at the cost of running and maintaining an individual virtual control plane per tenant. In addition, per-tenant control planes do not solve isolation problems in the data plane, such as node-level noisy neighbors or security threats. These must still be addressed separately. + +The Kubernetes [Cluster API - Nested (CAPN)](https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster) project provides an implementation of virtual control planes. + +#### Other implementations +* [Kamaji](https://github.com/clastix/kamaji) +* [vcluster](https://github.com/loft-sh/vcluster) + diff --git a/content/en/docs/concepts/security/overview.md b/content/en/docs/concepts/security/overview.md index ce75653dfd..9373e78ea9 100644 --- a/content/en/docs/concepts/security/overview.md +++ b/content/en/docs/concepts/security/overview.md @@ -29,7 +29,7 @@ computing approach to security, which is widely regarded as a best practice for software systems. {{< /note >}} -{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" >}} +{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" class="diagram-large" >}} Each layer of the Cloud Native security model builds upon the next outermost layer. The Code layer benefits from strong base (Cloud, Cluster, Container) security layers. @@ -60,6 +60,7 @@ Amazon Web Services | https://aws.amazon.com/security/ | Google Cloud Platform | https://cloud.google.com/security/ | IBM Cloud | https://www.ibm.com/cloud/security | Microsoft Azure | https://docs.microsoft.com/en-us/azure/security/azure-security | +Oracle Cloud Infrastructure | https://www.oracle.com/security/ | VMWare VSphere | https://www.vmware.com/security/hardening-guides.html | {{< /table >}} @@ -73,10 +74,10 @@ Suggestions for securing your infrastructure in a Kubernetes cluster: Area of Concern for Kubernetes Infrastructure | Recommendation | --------------------------------------------- | -------------- | Network access to API Server (Control plane) | All access to the Kubernetes control plane is not allowed publicly on the internet and is controlled by network access control lists restricted to the set of IP addresses needed to administer the cluster.| -Network access to Nodes (nodes) | Nodes should be configured to _only_ accept connections (via network access control lists)from the control plane on the specified ports, and accept connections for services in Kubernetes of type NodePort and LoadBalancer. If possible, these nodes should not be exposed on the public internet entirely. +Network access to Nodes (nodes) | Nodes should be configured to _only_ accept connections (via network access control lists) from the control plane on the specified ports, and accept connections for services in Kubernetes of type NodePort and LoadBalancer. If possible, these nodes should not be exposed on the public internet entirely. Kubernetes access to Cloud Provider API | Each cloud provider needs to grant a different set of permissions to the Kubernetes control plane and nodes. It is best to provide the cluster with cloud provider access that follows the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) for the resources it needs to administer. The [Kops documentation](https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md#iam-roles) provides information about IAM policies and roles. Access to etcd | Access to etcd (the datastore of Kubernetes) should be limited to the control plane only. Depending on your configuration, you should attempt to use etcd over TLS. More information can be found in the [etcd documentation](https://github.com/etcd-io/etcd/tree/master/Documentation). -etcd Encryption | Wherever possible it's a good practice to encrypt all drives at rest, but since etcd holds the state of the entire cluster (including Secrets) its disk should especially be encrypted at rest. +etcd Encryption | Wherever possible it's a good practice to encrypt all storage at rest, and since etcd holds the state of the entire cluster (including Secrets) its disk should especially be encrypted at rest. {{< /table >}} @@ -98,7 +99,7 @@ good information practices, read and follow the advice about Depending on the attack surface of your application, you may want to focus on specific aspects of security. For example: If you are running a service (Service A) that is critical in a chain of other resources and a separate workload (Service B) which is -vulnerable to a resource exhaustion attack then the risk of compromising Service A +vulnerable to a resource exhaustion attack, then the risk of compromising Service A is high if you do not limit the resources of Service B. The following table lists areas of security concerns and recommendations for securing workloads running in Kubernetes: @@ -107,10 +108,10 @@ Area of Concern for Workload Security | Recommendation | RBAC Authorization (Access to the Kubernetes API) | https://kubernetes.io/docs/reference/access-authn-authz/rbac/ Authentication | https://kubernetes.io/docs/concepts/security/controlling-access/ Application secrets management (and encrypting them in etcd at rest) | https://kubernetes.io/docs/concepts/configuration/secret/ <br> https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ -Pod Security Policies | https://kubernetes.io/docs/concepts/policy/pod-security-policy/ +Ensuring that pods meet defined Pod Security Standards | https://kubernetes.io/docs/concepts/security/pod-security-standards/#policy-instantiation Quality of Service (and Cluster resource management) | https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/ Network Policies | https://kubernetes.io/docs/concepts/services-networking/network-policies/ -TLS For Kubernetes Ingress | https://kubernetes.io/docs/concepts/services-networking/ingress/#tls +TLS for Kubernetes Ingress | https://kubernetes.io/docs/concepts/services-networking/ingress/#tls ## Container @@ -122,7 +123,7 @@ Area of Concern for Containers | Recommendation | Container Vulnerability Scanning and OS Dependency Security | As part of an image build step, you should scan your containers for known vulnerabilities. Image Signing and Enforcement | Sign container images to maintain a system of trust for the content of your containers. Disallow privileged users | When constructing containers, consult your documentation for how to create users inside of the containers that have the least level of operating system privilege necessary in order to carry out the goal of the container. -Use container runtime with stronger isolation | Select [container runtime classes](/docs/concepts/containers/runtime-class/) that provider stronger isolation +Use container runtime with stronger isolation | Select [container runtime classes](/docs/concepts/containers/runtime-class/) that provide stronger isolation ## Code @@ -136,7 +137,7 @@ are recommendations to protect application code: Area of Concern for Code | Recommendation | -------------------------| -------------- | -Access over TLS only | If your code needs to communicate by TCP, perform a TLS handshake with the client ahead of time. With the exception of a few cases, encrypt everything in transit. Going one step further, it's a good idea to encrypt network traffic between services. This can be done through a process known as mutual or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) which performs a two sided verification of communication between two certificate holding services. | +Access over TLS only | If your code needs to communicate by TCP, perform a TLS handshake with the client ahead of time. With the exception of a few cases, encrypt everything in transit. Going one step further, it's a good idea to encrypt network traffic between services. This can be done through a process known as mutual TLS authentication or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) which performs a two sided verification of communication between two certificate holding services. | Limiting port ranges of communication | This recommendation may be a bit self-explanatory, but wherever possible you should only expose the ports on your service that are absolutely essential for communication or metric gathering. | 3rd Party Dependency Security | It is a good practice to regularly scan your application's third party libraries for known security vulnerabilities. Each programming language has a tool for performing this check automatically. | Static Code Analysis | Most languages provide a way for a snippet of code to be analyzed for any potentially unsafe coding practices. Whenever possible you should perform checks using automated tooling that can scan codebases for common security errors. Some of the tools can be found at: https://owasp.org/www-community/Source_Code_Analysis_Tools | diff --git a/content/en/docs/concepts/security/pod-security-admission.md b/content/en/docs/concepts/security/pod-security-admission.md index a1c87767c9..60f87c04d9 100644 --- a/content/en/docs/concepts/security/pod-security-admission.md +++ b/content/en/docs/concepts/security/pod-security-admission.md @@ -13,15 +13,15 @@ min-kubernetes-server-version: v1.22 <!-- overview --> -{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} The Kubernetes [Pod Security Standards](/docs/concepts/security/pod-security-standards/) define different isolation levels for Pods. These standards let you define how you want to restrict the behavior of pods in a clear, consistent fashion. -As an Alpha feature, Kubernetes offers a built-in _Pod Security_ {{< glossary_tooltip +As a beta feature, Kubernetes offers a built-in _Pod Security_ {{< glossary_tooltip text="admission controller" term_id="admission-controller" >}}, the successor -to [PodSecurityPolicies](/docs/concepts/policy/pod-security-policy/). Pod security restrictions +to [PodSecurityPolicies](/docs/concepts/security/pod-security-policy/). Pod security restrictions are applied at the {{< glossary_tooltip text="namespace" term_id="namespace" >}} level when pods are created. @@ -30,17 +30,40 @@ The PodSecurityPolicy API is deprecated and will be [removed](/docs/reference/using-api/deprecation-guide/#v1-25) from Kubernetes in v1.25. {{< /note >}} -<!-- body --> -## Enabling the Alpha feature +## {{% heading "prerequisites" %}} -Setting pod security controls by namespace is an alpha feature. You must enable the `PodSecurity` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) in order to use it. +To use this mechanism, your cluster must enforce Pod Security admission. +### Built-in Pod Security admission enforcement + +From Kubernetes v1.23, the `PodSecurity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is a beta feature and is enabled by default. +This page is part of the documentation for Kubernetes v{{< skew currentVersion >}}. +If you are running a different version of Kubernetes, consult the documentation for that release. + +### Alternative: installing the `PodSecurity` admission webhook {#webhook} + +The `PodSecurity` admission logic is also available as a [validating admission webhook](https://git.k8s.io/pod-security-admission/webhook). This implementation is also beta. +For environments where the built-in `PodSecurity` admission plugin cannot be enabled, you can instead enable that logic via a validating admission webhook. + +A pre-built container image, certificate generation scripts, and example manifests +are available at [https://git.k8s.io/pod-security-admission/webhook](https://git.k8s.io/pod-security-admission/webhook). + +To install: ```shell ---feature-gates="...,PodSecurity=true" +git clone https://github.com/kubernetes/pod-security-admission.git +cd pod-security-admission/webhook +make certs +kubectl apply -k . ``` +{{< note >}} +The generated certificate is valid for 2 years. Before it expires, +regenerate the certificate or remove the webhook in favor of the built-in admission plugin. +{{< /note >}} + +<!-- body --> + ## Pod Security levels Pod Security admission places requirements on a Pod's [Security @@ -52,7 +75,7 @@ page for an in-depth look at those requirements. ## Pod Security Admission labels for namespaces -Provided that you have enabled this feature, you can configure namespaces to define the admission +Once the feature is enabled or the webhook is installed, you can configure namespaces to define the admission control mode you want to use for pod security in each namespace. Kubernetes defines a set of {{< glossary_tooltip term_id="label" text="labels" >}} that you can set to define which of the predefined Pod Security Standard levels you want to use for a namespace. The label you select @@ -63,7 +86,7 @@ takes if a potential violation is detected: Mode | Description :---------|:------------ **enforce** | Policy violations will cause the pod to be rejected. -**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug-application-cluster/audit/), but are otherwise allowed. +**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug/debug-cluster/audit/), but are otherwise allowed. **warn** | Policy violations will trigger a user-facing warning, but are otherwise allowed. {{< /table >}} @@ -79,7 +102,7 @@ For each mode, there are two labels that determine the policy used: pod-security.kubernetes.io/<MODE>: <LEVEL> # Optional: per-mode version label that can be used to pin the policy to the -# version that shipped with a given Kubernetes minor version (for example v{{< skew latestVersion >}}). +# version that shipped with a given Kubernetes minor version (for example v{{< skew currentVersion >}}). # # MODE must be one of `enforce`, `audit`, or `warn`. # VERSION must be a valid Kubernetes minor version, or `latest`. @@ -100,7 +123,7 @@ applied to workload resources, only to the resulting pod objects. ## Exemptions -You can define _exemptions_ from pod security enforcement in order allow the creation of pods that +You can define _exemptions_ from pod security enforcement in order to allow the creation of pods that would have otherwise been prohibited due to the policy associated with a given namespace. Exemptions can be statically configured in the [Admission Controller configuration](/docs/tasks/configure-pod-container/enforce-standards-admission-controller/#configure-the-admission-controller). diff --git a/content/en/docs/concepts/policy/pod-security-policy.md b/content/en/docs/concepts/security/pod-security-policy.md similarity index 79% rename from content/en/docs/concepts/policy/pod-security-policy.md rename to content/en/docs/concepts/security/pod-security-policy.md index 36172faba5..c296f31cc2 100644 --- a/content/en/docs/concepts/policy/pod-security-policy.md +++ b/content/en/docs/concepts/security/pod-security-policy.md @@ -11,8 +11,13 @@ weight: 30 {{< feature-state for_k8s_version="v1.21" state="deprecated" >}} -PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. For more information on the deprecation, +{{< caution >}} +PodSecurityPolicy is deprecated as of Kubernetes v1.21, and **will be removed in v1.25**. We recommend migrating to +[Pod Security Admission](/docs/concepts/security/pod-security-admission/), or a 3rd party admission plugin. +For a migration guide, see [Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/). +For more information on the deprecation, see [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). +{{< /caution >}} Pod Security Policies enable fine-grained authorization of pod creation and updates. @@ -22,7 +27,8 @@ updates. ## What is a Pod Security Policy? A _Pod Security Policy_ is a cluster-level resource that controls security -sensitive aspects of the pod specification. The [PodSecurityPolicy](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicy-v1beta1-policy) objects +sensitive aspects of the pod specification. The +[PodSecurityPolicy](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicy-v1beta1-policy) objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. They allow an administrator to control the following: @@ -49,10 +55,10 @@ administrator to control the following: ## Enabling Pod Security Policies -Pod security policy control is implemented as an optional [admission -controller](/docs/reference/access-authn-authz/admission-controllers/#podsecuritypolicy). -PodSecurityPolicies are enforced by [enabling the admission -controller](/docs/reference/access-authn-authz/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in), +Pod security policy control is implemented as an optional +[admission controller](/docs/reference/access-authn-authz/admission-controllers/#podsecuritypolicy). +PodSecurityPolicies are enforced by +[enabling the admission controller](/docs/reference/access-authn-authz/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in), but doing so without authorizing any policies **will prevent any pods from being created** in the cluster. @@ -64,9 +70,9 @@ controller. ## Authorizing Policies When a PodSecurityPolicy resource is created, it does nothing. In order to use -it, the requesting user or target pod's [service -account](/docs/tasks/configure-pod-container/configure-service-account/) must be -authorized to use the policy, by allowing the `use` verb on the policy. +it, the requesting user or target pod's +[service account](/docs/tasks/configure-pod-container/configure-service-account/) +must be authorized to use the policy, by allowing the `use` verb on the policy. Most Kubernetes pods are not created directly by users. Instead, they are typically created indirectly as part of a @@ -127,6 +133,7 @@ subjects: If a `RoleBinding` (not a `ClusterRoleBinding`) is used, it will only grant usage for pods being run in the same namespace as the binding. This can be paired with system groups to grant access to all pods run in the namespace: + ```yaml # Authorize all service accounts in a namespace: - kind: Group @@ -138,45 +145,47 @@ paired with system groups to grant access to all pods run in the namespace: name: system:authenticated ``` -For more examples of RBAC bindings, see [Role Binding -Examples](/docs/reference/access-authn-authz/rbac#role-binding-examples). -For a complete example of authorizing a PodSecurityPolicy, see -[below](#example). +For more examples of RBAC bindings, see +[RoleBinding examples](/docs/reference/access-authn-authz/rbac#role-binding-examples). +For a complete example of authorizing a PodSecurityPolicy, see [below](#example). ### Recommended Practice -PodSecurityPolicy is being replaced by a new, simplified `PodSecurity` {{< glossary_tooltip -text="admission controller" term_id="admission-controller" >}}. For more details on this change, see -[PodSecurityPolicy Deprecation: Past, Present, and -Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). Follow these -guidelines to simplify migration from PodSecurityPolicy to the new admission controller: +PodSecurityPolicy is being replaced by a new, simplified `PodSecurity` +{{< glossary_tooltip text="admission controller" term_id="admission-controller" >}}. +For more details on this change, see +[PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). +Follow these guidelines to simplify migration from PodSecurityPolicy to the +new admission controller: -1. Limit your PodSecurityPolicies to the policies defined by the [Pod Security Standards](/docs/concepts/security/pod-security-standards): - - {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}} - - {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}} - - {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}} +1. Limit your PodSecurityPolicies to the policies defined by the + [Pod Security Standards](/docs/concepts/security/pod-security-standards): -2. Only bind PSPs to entire namespaces, by using the `system:serviceaccounts:<namespace>` group + - {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}} + - {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}} + - {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}} + +1. Only bind PSPs to entire namespaces, by using the `system:serviceaccounts:<namespace>` group (where `<namespace>` is the target namespace). For example: - ```yaml - apiVersion: rbac.authorization.k8s.io/v1 - # This cluster role binding allows all pods in the "development" namespace to use the baseline PSP. - kind: ClusterRoleBinding - metadata: - name: psp-baseline-namespaces - roleRef: - kind: ClusterRole - name: psp-baseline - apiGroup: rbac.authorization.k8s.io - subjects: - - kind: Group - name: system:serviceaccounts:development - apiGroup: rbac.authorization.k8s.io - - kind: Group - name: system:serviceaccounts:canary - apiGroup: rbac.authorization.k8s.io - ``` + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + # This cluster role binding allows all pods in the "development" namespace to use the baseline PSP. + kind: ClusterRoleBinding + metadata: + name: psp-baseline-namespaces + roleRef: + kind: ClusterRole + name: psp-baseline + apiGroup: rbac.authorization.k8s.io + subjects: + - kind: Group + name: system:serviceaccounts:development + apiGroup: rbac.authorization.k8s.io + - kind: Group + name: system:serviceaccounts:canary + apiGroup: rbac.authorization.k8s.io + ``` ### Troubleshooting @@ -205,6 +214,9 @@ controller selects policies according to the following criteria: 2. If the pod must be defaulted or mutated, the first PodSecurityPolicy (ordered by name) to allow the pod is selected. +When a Pod is validated against a PodSecurityPolicy, [a `kubernetes.io/psp` annotation](/docs/reference/labels-annotations-taints/#kubernetes-io-psp) +is added to the Pod, with the name of the PodSecurityPolicy as the annotation value. + {{< note >}} During update operations (during which mutations to pod specs are disallowed) only non-mutating PodSecurityPolicies are used to validate the pod. @@ -212,8 +224,8 @@ only non-mutating PodSecurityPolicies are used to validate the pod. ## Example -_This example assumes you have a running cluster with the PodSecurityPolicy -admission controller enabled and you have cluster admin privileges._ +This example assumes you have a running cluster with the PodSecurityPolicy +admission controller enabled and you have cluster admin privileges. ### Set up @@ -236,8 +248,7 @@ alias kubectl-user='kubectl --as=system:serviceaccount:psp-example:fake-user -n ### Create a policy and a pod -Define the example PodSecurityPolicy object in a file. This is a policy that -prevents the creation of privileged pods. +This is a policy that prevents the creation of privileged pods. The name of a PodSecurityPolicy object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). @@ -246,7 +257,7 @@ The name of a PodSecurityPolicy object must be a valid And create it with kubectl: ```shell -kubectl-admin create -f example-psp.yaml +kubectl-admin create -f https://k8s.io/examples/policy/example-psp.yaml ``` Now, as the unprivileged user, try to create a simple pod: @@ -275,6 +286,11 @@ pod's service account nor `fake-user` have permission to use the new policy: ```shell kubectl-user auth can-i use podsecuritypolicy/example +``` + +The output is similar to this: + +``` no ``` @@ -291,14 +307,27 @@ kubectl-admin create role psp:unprivileged \ --verb=use \ --resource=podsecuritypolicy \ --resource-name=example -role "psp:unprivileged" created +``` +``` +role "psp:unprivileged" created +``` + +```shell kubectl-admin create rolebinding fake-user:psp:unprivileged \ --role=psp:unprivileged \ --serviceaccount=psp-example:fake-user -rolebinding "fake-user:psp:unprivileged" created +``` +``` +rolebinding "fake-user:psp:unprivileged" created +``` + +```shell kubectl-user auth can-i use podsecuritypolicy/example +``` + +``` yes ``` @@ -323,7 +352,20 @@ The output is similar to this pod "pause" created ``` -It works as expected! But any attempts to create a privileged pod should still +It works as expected! You can verify that the pod was validated against the +newly created PodSecurityPolicy: + +```shell +kubectl-user get pod pause -o yaml | grep kubernetes.io/psp +``` + +The output is similar to this + +``` +kubernetes.io/psp: example +``` + +But any attempts to create a privileged pod should still be denied: ```shell @@ -359,12 +401,24 @@ Let's try that again, slightly differently: ```shell kubectl-user create deployment pause --image=k8s.gcr.io/pause +``` + +```none deployment "pause" created - +``` +```shell kubectl-user get pods -No resources found. +``` +``` +No resources found. +``` + +```shell kubectl-user get events | head -n 2 +``` + +``` LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE 1m 2m 15 pause-7774d79b5 ReplicaSet Warning FailedCreate replicaset-controller Error creating: pods "pause-7774d79b5-" is forbidden: no providers available to validate pod request ``` @@ -385,6 +439,9 @@ is `default`: kubectl-admin create rolebinding default:psp:unprivileged \ --role=psp:unprivileged \ --serviceaccount=psp-example:default +``` + +```none rolebinding "default:psp:unprivileged" created ``` @@ -393,6 +450,9 @@ eventually succeed in creating the pod: ```shell kubectl-user get pods --watch +``` + +```none NAME READY STATUS RESTARTS AGE pause-7774d79b5-qrgcb 0/1 Pending 0 1s pause-7774d79b5-qrgcb 0/1 Pending 0 1s @@ -406,6 +466,9 @@ Delete the namespace to clean up most of the example resources: ```shell kubectl-admin delete ns psp-example +``` + +``` namespace "psp-example" deleted ``` @@ -414,6 +477,9 @@ up separately: ```shell kubectl-admin delete psp example +``` + +``` podsecuritypolicy "example" deleted ``` @@ -430,7 +496,8 @@ several security mechanisms. {{< codenew file="policy/restricted-psp.yaml" >}} -See [Pod Security Standards](/docs/concepts/security/pod-security-standards/#policy-instantiation) for more examples. +See [Pod Security Standards](/docs/concepts/security/pod-security-standards/#policy-instantiation) +for more examples. ## Policy Reference @@ -466,17 +533,17 @@ and `max`(inclusive). Defaults to no allowed host ports. **Volumes** - Provides a list of allowed volume types. The allowable values correspond to the volume sources that are defined when creating a volume. For the complete list of volume types, see [Types of -Volumes](/docs/concepts/storage/volumes/#types-of-volumes). Additionally, `*` -may be used to allow all volume types. +Volumes](/docs/concepts/storage/volumes/#types-of-volumes). Additionally, +`*` may be used to allow all volume types. The **recommended minimum set** of allowed volumes for new PSPs are: -- configMap -- downwardAPI -- emptyDir -- persistentVolumeClaim -- secret -- projected +- `configMap` +- `downwardAPI` +- `emptyDir` +- `persistentVolumeClaim` +- `secret` +- `projected` {{< warning >}} PodSecurityPolicy does not limit the types of `PersistentVolume` objects that @@ -488,10 +555,10 @@ should be granted permission to create `PersistentVolume` objects. **FSGroup** - Controls the supplemental group applied to some volumes. - *MustRunAs* - Requires at least one `range` to be specified. Uses the -minimum value of the first range as the default. Validates against all ranges. + minimum value of the first range as the default. Validates against all ranges. - *MayRunAs* - Requires at least one `range` to be specified. Allows -`FSGroups` to be left unset without providing a default. Validates against -all ranges if `FSGroups` is set. + `FSGroups` to be left unset without providing a default. Validates against + all ranges if `FSGroups` is set. - *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified. **AllowedHostPaths** - This specifies a list of host paths that are allowed @@ -510,7 +577,8 @@ For example: readOnly: true # only allow read-only mounts ``` -{{< warning >}}There are many ways a container with unrestricted access to the host +{{< warning >}} +There are many ways a container with unrestricted access to the host filesystem can escalate privileges, including reading data from other containers, and abusing the credentials of system services, such as Kubelet. @@ -551,33 +619,33 @@ spec: **RunAsUser** - Controls which user ID the containers are run with. - *MustRunAs* - Requires at least one `range` to be specified. Uses the -minimum value of the first range as the default. Validates against all ranges. + minimum value of the first range as the default. Validates against all ranges. - *MustRunAsNonRoot* - Requires that the pod be submitted with a non-zero -`runAsUser` or have the `USER` directive defined (using a numeric UID) in the -image. Pods which have specified neither `runAsNonRoot` nor `runAsUser` settings -will be mutated to set `runAsNonRoot=true`, thus requiring a defined non-zero -numeric `USER` directive in the container. No default provided. Setting -`allowPrivilegeEscalation=false` is strongly recommended with this strategy. + `runAsUser` or have the `USER` directive defined (using a numeric UID) in the + image. Pods which have specified neither `runAsNonRoot` nor `runAsUser` settings + will be mutated to set `runAsNonRoot=true`, thus requiring a defined non-zero + numeric `USER` directive in the container. No default provided. Setting + `allowPrivilegeEscalation=false` is strongly recommended with this strategy. - *RunAsAny* - No default provided. Allows any `runAsUser` to be specified. **RunAsGroup** - Controls which primary group ID the containers are run with. - *MustRunAs* - Requires at least one `range` to be specified. Uses the -minimum value of the first range as the default. Validates against all ranges. + minimum value of the first range as the default. Validates against all ranges. - *MayRunAs* - Does not require that RunAsGroup be specified. However, when RunAsGroup -is specified, they have to fall in the defined range. + is specified, they have to fall in the defined range. - *RunAsAny* - No default provided. Allows any `runAsGroup` to be specified. **SupplementalGroups** - Controls which group IDs containers add. - *MustRunAs* - Requires at least one `range` to be specified. Uses the -minimum value of the first range as the default. Validates against all ranges. + minimum value of the first range as the default. Validates against all ranges. - *MayRunAs* - Requires at least one `range` to be specified. Allows -`supplementalGroups` to be left unset without providing a default. -Validates against all ranges if `supplementalGroups` is set. + `supplementalGroups` to be left unset without providing a default. + Validates against all ranges if `supplementalGroups` is set. - *RunAsAny* - No default provided. Allows any `supplementalGroups` to be -specified. + specified. ### Privilege Escalation @@ -622,9 +690,8 @@ added. Capabilities listed in `RequiredDropCapabilities` must not be included in `AllowedCapabilities` or `DefaultAddCapabilities`. **DefaultAddCapabilities** - The capabilities which are added to containers by -default, in addition to the runtime defaults. See the [Docker -documentation](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) -for the default list of capabilities when using the Docker runtime. +default, in addition to the runtime defaults. See the +documentation for your container runtime for information on working with Linux capabilities. ### SELinux @@ -650,16 +717,17 @@ denoted as the string `Unmasked`. ### AppArmor -Controlled via annotations on the PodSecurityPolicy. Refer to the [AppArmor -documentation](/docs/tutorials/clusters/apparmor/#podsecuritypolicy-annotations). +Controlled via annotations on the PodSecurityPolicy. Refer to the +[AppArmor documentation](/docs/tutorials/security/apparmor/#podsecuritypolicy-annotations). ### Seccomp As of Kubernetes v1.19, you can use the `seccompProfile` field in the -`securityContext` of Pods or containers to [control use of seccomp -profiles](/docs/tutorials/clusters/seccomp). In prior versions, seccomp was -controlled by adding annotations to a Pod. The same PodSecurityPolicies can be -used with either version to enforce how these fields or annotations are applied. +`securityContext` of Pods or containers to +[control use of seccomp profiles](/docs/tutorials/security/seccomp/). +In prior versions, seccomp was controlled by adding annotations to a Pod. The +same PodSecurityPolicies can be used with either version to enforce how these +fields or annotations are applied. **seccomp.security.alpha.kubernetes.io/defaultProfileName** - Annotation that specifies the default seccomp profile to apply to containers. Possible values @@ -676,10 +744,10 @@ are: flag is not defined, the default path will be used, which is `<root-dir>/seccomp` where `<root-dir>` is specified by the `--root-dir` flag. -{{< note >}} + {{< note >}} The `--seccomp-profile-root` flag is deprecated since Kubernetes v1.19. Users are encouraged to use the default path. -{{< /note >}} + {{< /note >}} **seccomp.security.alpha.kubernetes.io/allowedProfileNames** - Annotation that specifies which values are allowed for the pod seccomp annotations. Specified as @@ -691,18 +759,22 @@ default cannot be changed. By default, all safe sysctls are allowed. -- `forbiddenSysctls` - excludes specific sysctls. You can forbid a combination of safe and unsafe sysctls in the list. To forbid setting any sysctls, use `*` on its own. -- `allowedUnsafeSysctls` - allows specific sysctls that had been disallowed by the default list, so long as these are not listed in `forbiddenSysctls`. +- `forbiddenSysctls` - excludes specific sysctls. You can forbid a combination + of safe and unsafe sysctls in the list. To forbid setting any sysctls, use + `*` on its own. +- `allowedUnsafeSysctls` - allows specific sysctls that had been disallowed by + the default list, so long as these are not listed in `forbiddenSysctls`. -Refer to the [Sysctl documentation]( -/docs/tasks/administer-cluster/sysctl-cluster/#podsecuritypolicy). +Refer to the [Sysctl documentation](/docs/tasks/administer-cluster/sysctl-cluster/#podsecuritypolicy). ## {{% heading "whatsnext" %}} -- See [PodSecurityPolicy Deprecation: Past, Present, and - Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) to learn about - the future of pod security policy. +- See [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) + to learn about the future of pod security policy. -- See [Pod Security Standards](/docs/concepts/security/pod-security-standards/) for policy recommendations. +- See [Pod Security Standards](/docs/concepts/security/pod-security-standards/) + for policy recommendations. + +- Refer to [PodSecurityPolicy reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicy-v1beta1-policy) + for the API details. -- Refer to [Pod Security Policy Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicy-v1beta1-policy) for the api details. diff --git a/content/en/docs/concepts/security/pod-security-standards.md b/content/en/docs/concepts/security/pod-security-standards.md index f3b43344bf..d60f3b0ae1 100644 --- a/content/en/docs/concepts/security/pod-security-standards.md +++ b/content/en/docs/concepts/security/pod-security-standards.md @@ -29,10 +29,9 @@ This guide outlines the requirements of each policy. **The _Privileged_ policy is purposely-open, and entirely unrestricted.** This type of policy is typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users. -The Privileged policy is defined by an absence of restrictions. For allow-by-default enforcement -mechanisms (such as gatekeeper), the Privileged policy may be an absence of applied constraints -rather than an instantiated profile. In contrast, for a deny-by-default mechanism (such as Pod -Security Policy) the Privileged policy should enable all controls (disable all restrictions). +The Privileged policy is defined by an absence of restrictions. Allow-by-default +mechanisms (such as gatekeeper) may be Privileged by default. In contrast, for a deny-by-default mechanism (such as Pod +Security Policy) the Privileged policy should disable all restrictions. ### Baseline @@ -52,13 +51,13 @@ fail validation. <caption style="display:none">Baseline policy specification</caption> <tbody> <tr> - <td><strong>Control</strong></td> - <td><strong>Policy</strong></td> + <th>Control</th> + <th>Policy</th> </tr> <tr> <td style="white-space: nowrap">HostProcess</td> <td> - <p>Windows pods offer the ability to run <a href="/docs/tasks/configure-pod-container/create-hostprocess-pod">HostProcess containers</a> which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy. HostProcess pods are an <strong>alpha</strong> feature as of Kubernetes <strong>v1.22</strong>.</p> + <p>Windows pods offer the ability to run <a href="/docs/tasks/configure-pod-container/create-hostprocess-pod">HostProcess containers</a> which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy. {{< feature-state for_k8s_version="v1.23" state="beta" >}}</p> <p><strong>Restricted Fields</strong></p> <ul> <li><code>spec.securityContext.windowsOptions.hostProcess</code></li> @@ -305,34 +304,22 @@ fail validation. <tr> <td style="white-space: nowrap">Volume Types</td> <td> - <p>In addition to restricting HostPath volumes, the restricted policy limits usage of non-core volume types to those defined through PersistentVolumes.</p> + <p>The restricted policy only permits the following volume types.</p> <p><strong>Restricted Fields</strong></p> <ul> - <li><code>spec.volumes[*].hostPath</code></li> - <li><code>spec.volumes[*].gcePersistentDisk</code></li> - <li><code>spec.volumes[*].awsElasticBlockStore</code></li> - <li><code>spec.volumes[*].gitRepo</code></li> - <li><code>spec.volumes[*].nfs</code></li> - <li><code>spec.volumes[*].iscsi</code></li> - <li><code>spec.volumes[*].glusterfs</code></li> - <li><code>spec.volumes[*].rbd</code></li> - <li><code>spec.volumes[*].flexVolume</code></li> - <li><code>spec.volumes[*].cinder</code></li> - <li><code>spec.volumes[*].cephfs</code></li> - <li><code>spec.volumes[*].flocker</code></li> - <li><code>spec.volumes[*].fc</code></li> - <li><code>spec.volumes[*].azureFile</code></li> - <li><code>spec.volumes[*].vsphereVolume</code></li> - <li><code>spec.volumes[*].quobyte</code></li> - <li><code>spec.volumes[*].azureDisk</code></li> - <li><code>spec.volumes[*].portworxVolume</code></li> - <li><code>spec.volumes[*].scaleIO</code></li> - <li><code>spec.volumes[*].storageos</code></li> - <li><code>spec.volumes[*].photonPersistentDisk</code></li> + <li><code>spec.volumes[*]</code></li> </ul> <p><strong>Allowed Values</strong></p> + Every item in the <code>spec.volumes[*]</code> list must set one of the following fields to a non-null value: <ul> - <li>Undefined/nil</li> + <li><code>spec.volumes[*].configMap</code></li> + <li><code>spec.volumes[*].csi</code></li> + <li><code>spec.volumes[*].downwardAPI</code></li> + <li><code>spec.volumes[*].emptyDir</code></li> + <li><code>spec.volumes[*].ephemeral</code></li> + <li><code>spec.volumes[*].persistentVolumeClaim</code></li> + <li><code>spec.volumes[*].projected</code></li> + <li><code>spec.volumes[*].secret</code></li> </ul> </td> </tr> @@ -374,22 +361,20 @@ fail validation. </td> </tr> <tr> - <td style="white-space: nowrap">Non-root groups <em>(optional)</em></td> + <td style="white-space: nowrap">Running as Non-root user (v1.23+)</td> <td> - <p>Containers should be forbidden from running with a root primary or supplementary GID.</p> + <p>Containers must not set <tt>runAsUser</tt> to 0</p> <p><strong>Restricted Fields</strong></p> <ul> - <li><code>spec.securityContext.runAsGroup</code></li> - <li><code>spec.securityContext.supplementalGroups[*]</code></li> - <li><code>spec.securityContext.fsGroup</code></li> - <li><code>spec.containers[*].securityContext.runAsGroup</code></li> - <li><code>spec.initContainers[*].securityContext.runAsGroup</code></li> - <li><code>spec.ephemeralContainers[*].securityContext.runAsGroup</code></li> + <li><code>spec.securityContext.runAsUser</code></li> + <li><code>spec.containers[*].securityContext.runAsUser</code></li> + <li><code>spec.initContainers[*].securityContext.runAsUser</code></li> + <li><code>spec.ephemeralContainers[*].securityContext.runAsUser</code></li> </ul> <p><strong>Allowed Values</strong></p> <ul> - <li>Undefined/nil (except for <code>*.runAsGroup</code>)</li> - <li>Non-zero</li> + <li>any non-zero value</li> + <li><code>undefined/null</code></li> </ul> </td> </tr> @@ -466,12 +451,22 @@ of individual policies are not defined here. - {{< example file="security/podsecurity-baseline.yaml" >}}Baseline namespace{{< /example >}} - {{< example file="security/podsecurity-restricted.yaml" >}}Restricted namespace{{< /example >}} -[**PodSecurityPolicy**](/docs/concepts/profile/pod-security-profile/) (Deprecated) +[**PodSecurityPolicy**](/docs/concepts/security/pod-security-policy/) (Deprecated) - {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}} - {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}} - {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}} +### Alternatives + +{{% thirdparty-content %}} + +Other alternatives for enforcing policies are being developed in the Kubernetes ecosystem, such as: + +- [Kubewarden](https://github.com/kubewarden) +- [Kyverno](https://kyverno.io/policies/pod-security/) +- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) + ## FAQ ### Why isn't there a profile between privileged and baseline? @@ -492,23 +487,15 @@ in the Pod manifest, and represent parameters to the container runtime. Security profiles are control plane mechanisms to enforce specific settings in the Security Context, as well as other related parameters outside the Security Context. As of July 2021, -[Pod Security Policies](/docs/concepts/profile/pod-security-profile/) are deprecated in favor of the +[Pod Security Policies](/docs/concepts/security/pod-security-policy/) are deprecated in favor of the built-in [Pod Security Admission Controller](/docs/concepts/security/pod-security-admission/). -{{% thirdparty-content %}} - -Other alternatives for enforcing security profiles are being developed in the Kubernetes -ecosystem, such as: -- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper). -- [Kubewarden](https://github.com/kubewarden). -- [Kyverno](https://kyverno.io/policies/pod-security/). - ### What profiles should I apply to my Windows Pods? Windows in Kubernetes has some limitations and differentiators from standard Linux-based -workloads. Specifically, many of the Pod SecurityContext fields [have no effect on -Windows](/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#v1-podsecuritycontext). As -such, no standardized Pod Security profiles currently exist. +workloads. Specifically, many of the Pod SecurityContext fields +[have no effect on Windows](/docs/concepts/windows/intro/#compatibility-v1-pod-spec-containers-securitycontext). +As such, no standardized Pod Security profiles currently exist. If you apply the restricted profile for a Windows pod, this **may** have an impact on the pod at runtime. The restricted profile requires enforcing Linux-specific restrictions (such as seccomp @@ -517,7 +504,9 @@ these Linux-specific values, then the Windows pod should still work normally wit profile. However, the lack of enforcement means that there is no additional restriction, for Pods that use Windows containers, compared to the baseline profile. -The use of the HostProcess flag to create a HostProcess pod should only be done in alignment with the privileged policy. Creation of a Windows HostProcess pod is blocked under the baseline and restricted policies, so any HostProcess pod should be considered privileged. +The use of the HostProcess flag to create a HostProcess pod should only be done in alignment with the privileged policy. +Creation of a Windows HostProcess pod is blocked under the baseline and restricted policies, +so any HostProcess pod should be considered privileged. ### What about sandboxed Pods? @@ -531,3 +520,4 @@ kernel. This allows for workloads requiring heightened permissions to still be i Additionally, the protection of sandboxed workloads is highly dependent on the method of sandboxing. As such, no single recommended profile is recommended for all sandboxed workloads. + diff --git a/content/en/docs/concepts/security/rbac-good-practices.md b/content/en/docs/concepts/security/rbac-good-practices.md new file mode 100644 index 0000000000..fe858bba72 --- /dev/null +++ b/content/en/docs/concepts/security/rbac-good-practices.md @@ -0,0 +1,189 @@ +--- +reviewers: +title: Role Based Access Control Good Practices +description: > + Principles and practices for good RBAC design for cluster operators. +content_type: concept +weight: 60 +--- + +<!-- overview --> + +Kubernetes {{< glossary_tooltip text="RBAC" term_id="rbac" >}} is a key security control +to ensure that cluster users and workloads have only the access to resources required to +execute their roles. It is important to ensure that, when designing permissions for cluster +users, the cluster administrator understands the areas where privilge escalation could occur, +to reduce the risk of excessive access leading to security incidents. + +The good practices laid out here should be read in conjunction with the general +[RBAC documentation](/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update). + +<!-- body --> + +## General good practice + +### Least privilege + +Ideally, minimal RBAC rights should be assigned to users and service accounts. Only permissions +explicitly required for their operation should be used. While each cluster will be different, +some general rules that can be applied are : + + - Assign permissions at the namespace level where possible. Use RoleBindings as opposed to + ClusterRoleBindings to give users rights only within a specific namespace. + - Avoid providing wildcard permissions when possible, especially to all resources. + As Kubernetes is an extensible system, providing wildcard access gives rights + not just to all object types that currently exist in the cluster, but also to all future object types + which are created in the future. + - Administrators should not use `cluster-admin` accounts except where specifically needed. + Providing a low privileged account with + [impersonation rights](/docs/reference/access-authn-authz/authentication/#user-impersonation) + can avoid accidental modification of cluster resources. + - Avoid adding users to the `system:masters` group. Any user who is a member of this group + bypasses all RBAC rights checks and will always have unrestricted superuser access, which cannot be + revoked by removing RoleBindings or ClusterRoleBindings. As an aside, if a cluster is + using an authorization webhook, membership of this group also bypasses that webhook (requests + from users who are members of that group are never sent to the webhook) + +### Minimize distribution of privileged tokens + +Ideally, pods shouldn't be assigned service accounts that have been granted powerful permissions +(for example, any of the rights listed under [privilege escalation risks](#privilege-escalation-risks)). +In cases where a workload requires powerful permissions, consider the following practices: + + - Limit the number of nodes running powerful pods. Ensure that any DaemonSets you run + are necessary and are run with least privilege to limit the blast radius of container escapes. + - Avoid running powerful pods alongside untrusted or publicly-exposed ones. Consider using + [Taints and Toleration](/docs/concepts/scheduling-eviction/taint-and-toleration/), + [NodeAffinity](/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity), or + [PodAntiAffinity](/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) + to ensure pods don't run alongside untrusted or less-trusted Pods. Pay especial attention to + situations where less-trustworthy Pods are not meeting the **Restricted** Pod Security Standard. + +### Hardening + +Kubernetes defaults to providing access which may not be required in every cluster. Reviewing +the RBAC rights provided by default can provide opportunities for security hardening. +In general, changes should not be made to rights provided to `system:` accounts some options +to harden cluster rights exist: + +- Review bindings for the `system:unauthenticated` group and remove them where possible, as this gives + access to anyone who can contact the API server at a network level. +- Avoid the default auto-mounting of service account tokens by setting + `automountServiceAccountToken: false`. For more details, see + [using default service account token](/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server). + Setting this value for a Pod will overwrite the service account setting, workloads + which require service account tokens can still mount them. + +### Periodic review + +It is vital to periodically review the Kubernetes RBAC settings for redundant entries and +possible privilege escalations. +If an attacker is able to create a user account with the same name as a deleted user, +they can automatically inherit all the rights of the deleted user, especially the +rights assigned to that user. + +## Kubernetes RBAC - privilege escalation risks {#privilege-escalation-risks} + +Within Kubernetes RBAC there are a number of privileges which, if granted, can allow a user or a service account +to escalate their privileges in the cluster or affect systems outside the cluster. + +This section is intended to provide visibility of the areas where cluster operators +should take care, to ensure that they do not inadvertently allow for more access to clusters than intended. + +### Listing secrets + +It is generally clear that allowing `get` access on Secrets will allow a user to read their contents. +It is also important to note that `list` and `watch` access also effectively allow for users to reveal the Secret contents. +For example, when a List response is returned (for example, via `kubectl get secrets -A -o yaml`), the response +includes the contents of all Secrets. + +### Workload creation + +Users who are able to create workloads (either Pods, or +[workload resources](/docs/concepts/workloads/controllers/) that manage Pods) will +be able to gain access to the underlying node unless restrictions based on the Kubernetes +[Pod Security Standards](/docs/concepts/security/pod-security-standards/) are in place. + +Users who can run privileged Pods can use that access to gain node access and potentially to +further elevate their privileges. Where you do not fully trust a user or other principal +with the ability to create suitably secure and isolated Pods, you should enforce either the +**Baseline** or **Restricted** Pod Security Standard. +You can use [Pod Security admission](/docs/concepts/security/pod-security-admission/) +or other (third party) mechanisms to implement that enforcement. + +You can also use the deprecated [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) mechanism +to restrict users' abilities to create privileged Pods (N.B. PodSecurityPolicy is scheduled for removal +in version 1.25). + +Creating a workload in a namespace also grants indirect access to Secrets in that namespace. +Creating a pod in kube-system or a similarly privileged namespace can grant a user access to +Secrets they would not have through RBAC directly. + +### Persistent volume creation + +As noted in the [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/#volumes-and-file-systems) +documentation, access to create PersistentVolumes can allow for escalation of access to the underlying host. +Where access to persistent storage is required trusted administrators should create +PersistentVolumes, and constrained users should use PersistentVolumeClaims to access that storage. + +### Access to `proxy` subresource of Nodes + +Users with access to the proxy sub-resource of node objects have rights to the Kubelet API, +which allows for command execution on every pod on the node(s) to which they have rights. +This access bypasses audit logging and admission control, so care should be taken before +granting rights to this resource. + +### Escalate verb + +Generally, the RBAC system prevents users from creating clusterroles with more rights than the user possesses. +The exception to this is the `escalate` verb. As noted in the [RBAC documentation](/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update), +users with this right can effectively escalate their privileges. + +### Bind verb + +Similar to the `escalate` verb, granting users this right allows for the bypass of Kubernetes +in-built protections against privilege escalation, allowing users to create bindings to +roles with rights they do not already have. + +### Impersonate verb + +This verb allows users to impersonate and gain the rights of other users in the cluster. +Care should be taken when granting it, to ensure that excessive permissions cannot be gained +via one of the impersonated accounts. + +### CSRs and certificate issuing + +The CSR API allows for users with `create` rights to CSRs and `update` rights on `certificatesigningrequests/approval` +where the signer is `kubernetes.io/kube-apiserver-client` to create new client certificates +which allow users to authenticate to the cluster. Those client certificates can have arbitrary +names including duplicates of Kubernetes system components. This will effectively allow for privilege escalation. + +### Token request + +Users with `create` rights on `serviceaccounts/token` can create TokenRequests to issue +tokens for existing service accounts. + +### Control admission webhooks + +Users with control over `validatingwebhookconfigurations` or `mutatingwebhookconfigurations` +can control webhooks that can read any object admitted to the cluster, and in the case of +mutating webhooks, also mutate admitted objects. + + +## Kubernetes RBAC - denial of service risks {#denial-of-service-risks} + +### Object creation denial-of-service {#object-creation-dos} +Users who have rights to create objects in a cluster may be able to create sufficient large +objects to create a denial of service condition either based on the size or number of objects, as discussed in +[etcd used by Kubernetes is vulnerable to OOM attack](https://github.com/kubernetes/kubernetes/issues/107325). This may be +specifically relevant in multi-tenant clusters if semi-trusted or untrusted users +are allowed limited access to a system. + +One option for mitigation of this issue would be to use +[resource quotas](/docs/concepts/policy/resource-quotas/#object-count-quota) +to limit the quantity of objects which can be created. + +## {{% heading "whatsnext" %}} + +* To learn more about RBAC, see the [RBAC documentation](/docs/reference/access-authn-authz/rbac/). + diff --git a/content/en/docs/concepts/security/windows-security.md b/content/en/docs/concepts/security/windows-security.md new file mode 100644 index 0000000000..c6523e3a43 --- /dev/null +++ b/content/en/docs/concepts/security/windows-security.md @@ -0,0 +1,62 @@ +--- +reviewers: +- jayunit100 +- jsturtevant +- marosset +- perithompson +title: Security For Windows Nodes +content_type: concept +weight: 40 +--- + +<!-- overview --> + +This page describes security considerations and best practices specific to the Windows operating system. + +<!-- body --> + +## Protection for Secret data on nodes + +On Windows, data from Secrets are written out in clear text onto the node's local +storage (as compared to using tmpfs / in-memory filesystems on Linux). As a cluster +operator, you should take both of the following additional measures: + +1. Use file ACLs to secure the Secrets' file location. +1. Apply volume-level encryption using + [BitLocker](https://docs.microsoft.com/windows/security/information-protection/bitlocker/bitlocker-how-to-deploy-on-windows-server). + +## Container users + +[RunAsUsername](/docs/tasks/configure-pod-container/configure-runasusername) +can be specified for Windows Pods or containers to execute the container +processes as specific user. This is roughly equivalent to +[RunAsUser](/docs/concepts/security/pod-security-policy/#users-and-groups). + +Windows containers offer two default user accounts, ContainerUser and ContainerAdministrator. +The differences between these two user accounts are covered in +[When to use ContainerAdmin and ContainerUser user accounts](https://docs.microsoft.com/virtualization/windowscontainers/manage-containers/container-security#when-to-use-containeradmin-and-containeruser-user-accounts) +within Microsoft's _Secure Windows containers_ documentation. + +Local users can be added to container images during the container build process. + +{{< note >}} + +* [Nano Server](https://hub.docker.com/_/microsoft-windows-nanoserver) based images run as + `ContainerUser` by default +* [Server Core](https://hub.docker.com/_/microsoft-windows-servercore) based images run as + `ContainerAdministrator` by default + +{{< /note >}} + +Windows containers can also run as Active Directory identities by utilizing +[Group Managed Service Accounts](/docs/tasks/configure-pod-container/configure-gmsa/) + +## Pod-level security isolation + +Linux-specific pod security context mechanisms (such as SELinux, AppArmor, Seccomp, or custom +POSIX capabilities) are not supported on Windows nodes. + +Privileged containers are [not supported](/docs/concepts/windows/intro/#compatibility-v1-pod-spec-containers-securitycontext) +on Windows. +Instead [HostProcess containers](/docs/tasks/configure-pod-container/create-hostprocess-pod) +can be used on Windows to perform many of the tasks performed by privileged containers on Linux. diff --git a/content/en/docs/concepts/services-networking/_index.md b/content/en/docs/concepts/services-networking/_index.md index 2e7d91427e..b4f7861075 100644 --- a/content/en/docs/concepts/services-networking/_index.md +++ b/content/en/docs/concepts/services-networking/_index.md @@ -5,8 +5,49 @@ description: > Concepts and resources behind networking in Kubernetes. --- +## The Kubernetes network model + +Every [`Pod`](/docs/concepts/workloads/pods/) in a cluster gets its own unique cluster-wide IP address. +This means you do not need to explicitly create links between `Pods` and you +almost never need to deal with mapping container ports to host ports. +This creates a clean, backwards-compatible model where `Pods` can be treated +much like VMs or physical hosts from the perspectives of port allocation, +naming, service discovery, [load balancing](/docs/concepts/services-networking/ingress/#load-balancing), +application configuration, and migration. + +Kubernetes imposes the following fundamental requirements on any networking +implementation (barring any intentional network segmentation policies): + + * pods can communicate with all other pods on any other [node](/docs/concepts/architecture/nodes/) + without NAT + * agents on a node (e.g. system daemons, kubelet) can communicate with all + pods on that node + +Note: For those platforms that support `Pods` running in the host network (e.g. +Linux), when pods are attached to the host network of a node they can still communicate +with all pods on all nodes without NAT. + +This model is not only less complex overall, but it is principally compatible +with the desire for Kubernetes to enable low-friction porting of apps from VMs +to containers. If your job previously ran in a VM, your VM had an IP and could +talk to other VMs in your project. This is the same basic model. + +Kubernetes IP addresses exist at the `Pod` scope - containers within a `Pod` +share their network namespaces - including their IP address and MAC address. +This means that containers within a `Pod` can all reach each other's ports on +`localhost`. This also means that containers within a `Pod` must coordinate port +usage, but this is no different from processes in a VM. This is called the +"IP-per-pod" model. + +How this is implemented is a detail of the particular container runtime in use. + +It is possible to request ports on the `Node` itself which forward to your `Pod` +(called host ports), but this is a very niche operation. How that forwarding is +implemented is also a detail of the container runtime. The `Pod` itself is +blind to the existence or non-existence of host ports. + Kubernetes networking addresses four concerns: -- Containers within a Pod use networking to communicate via loopback. +- Containers within a Pod [use networking to communicate](/docs/concepts/services-networking/dns-pod-service/) via loopback. - Cluster networking provides communication between different Pods. -- The Service resource lets you expose an application running in Pods to be reachable from outside your cluster. -- You can also use Services to publish services only for consumption inside your cluster. +- The [Service resource](/docs/concepts/services-networking/service/) lets you [expose an application running in Pods](/docs/concepts/services-networking/connect-applications-service/) to be reachable from outside your cluster. +- You can also use Services to [publish services only for consumption inside your cluster](/docs/concepts/services-networking/service-traffic-policy/). diff --git a/content/en/docs/concepts/services-networking/connect-applications-service.md b/content/en/docs/concepts/services-networking/connect-applications-service.md index 89d2daddb2..4aa4c4d29b 100644 --- a/content/en/docs/concepts/services-networking/connect-applications-service.md +++ b/content/en/docs/concepts/services-networking/connect-applications-service.md @@ -13,11 +13,9 @@ weight: 30 ## The Kubernetes model for connecting containers -Now that you have a continuously running, replicated application you can expose it on a network. Before discussing the Kubernetes approach to networking, it is worthwhile to contrast it with the "normal" way networking works with Docker. +Now that you have a continuously running, replicated application you can expose it on a network. -By default, Docker uses host-private networking, so containers can talk to other containers only if they are on the same machine. In order for Docker containers to communicate across nodes, there must be allocated ports on the machine's own IP address, which are then forwarded or proxied to the containers. This obviously means that containers must either coordinate which ports they use very carefully or ports must be allocated dynamically. - -Coordinating port allocations across multiple developers or teams that provide containers is very difficult to do at scale, and exposes users to cluster-level issues outside of their control. Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. Kubernetes gives every pod its own cluster-private IP address, so you do not need to explicitly create links between pods or map container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document elaborates on how you can run reliable services on such a networking model. +Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. Kubernetes gives every pod its own cluster-private IP address, so you do not need to explicitly create links between pods or map container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document elaborates on how you can run reliable services on such a networking model. This guide uses a simple nginx server to demonstrate proof of concept. @@ -52,9 +50,9 @@ kubectl get pods -l run=my-nginx -o yaml | grep podIP podIP: 10.244.2.5 ``` -You should be able to ssh into any node in your cluster and curl both IPs. Note that the containers are *not* using port 80 on the node, nor are there any special NAT rules to route traffic to the pod. This means you can run multiple nginx pods on the same node all using the same containerPort and access them from any other pod or node in your cluster using IP. Like Docker, ports can still be published to the host node's interfaces, but the need for this is radically diminished because of the networking model. +You should be able to ssh into any node in your cluster and use a tool such as `curl` to make queries against both IPs. Note that the containers are *not* using port 80 on the node, nor are there any special NAT rules to route traffic to the pod. This means you can run multiple nginx pods on the same node all using the same `containerPort`, and access them from any other pod or node in your cluster using the assigned IP address for the Service. If you want to arrange for a specific port on the host Node to be forwarded to backing Pods, you can - but the networking model should mean that you do not need to do so. -You can read more about [how we achieve this](/docs/concepts/cluster-administration/networking/#how-to-achieve-this) if you're curious. +You can read more about the [Kubernetes Networking Model](/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) if you're curious. ## Creating a Service diff --git a/content/en/docs/concepts/services-networking/dns-pod-service.md b/content/en/docs/concepts/services-networking/dns-pod-service.md index f43eeff22b..939269f8ec 100644 --- a/content/en/docs/concepts/services-networking/dns-pod-service.md +++ b/content/en/docs/concepts/services-networking/dns-pod-service.md @@ -8,8 +8,8 @@ weight: 20 --- <!-- overview --> -Kubernetes creates DNS records for services and pods. You can contact -services with consistent DNS names instead of IP addresses. +Kubernetes creates DNS records for Services and Pods. You can contact +Services with consistent DNS names instead of IP addresses. <!-- body --> @@ -25,21 +25,21 @@ Pod's own namespace and the cluster's default domain. ### Namespaces of Services -A DNS query may return different results based on the namespace of the pod making -it. DNS queries that don't specify a namespace are limited to the pod's -namespace. Access services in other namespaces by specifying it in the DNS query. +A DNS query may return different results based on the namespace of the Pod making +it. DNS queries that don't specify a namespace are limited to the Pod's +namespace. Access Services in other namespaces by specifying it in the DNS query. -For example, consider a pod in a `test` namespace. A `data` service is in +For example, consider a Pod in a `test` namespace. A `data` Service is in the `prod` namespace. -A query for `data` returns no results, because it uses the pod's `test` namespace. +A query for `data` returns no results, because it uses the Pod's `test` namespace. A query for `data.prod` returns the intended result, because it specifies the namespace. -DNS queries may be expanded using the pod's `/etc/resolv.conf`. Kubelet -sets this file for each pod. For example, a query for just `data` may be -expanded to `data.test.cluster.local`. The values of the `search` option +DNS queries may be expanded using the Pod's `/etc/resolv.conf`. Kubelet +sets this file for each Pod. For example, a query for just `data` may be +expanded to `data.test.svc.cluster.local`. The values of the `search` option are used to expand queries. To learn more about DNS queries, see [the `resolv.conf` manual page.](https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html) @@ -49,7 +49,7 @@ search <namespace>.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` -In summary, a pod in the _test_ namespace can successfully resolve either +In summary, a Pod in the _test_ namespace can successfully resolve either `data.prod` or `data.prod.svc.cluster.local`. ### DNS Records @@ -70,14 +70,14 @@ For more up-to-date specification, see ### A/AAAA records "Normal" (not headless) Services are assigned a DNS A or AAAA record, -depending on the IP family of the service, for a name of the form +depending on the IP family of the Service, for a name of the form `my-svc.my-namespace.svc.cluster-domain.example`. This resolves to the cluster IP of the Service. "Headless" (without a cluster IP) Services are also assigned a DNS A or AAAA record, -depending on the IP family of the service, for a name of the form +depending on the IP family of the Service, for a name of the form `my-svc.my-namespace.svc.cluster-domain.example`. Unlike normal -Services, this resolves to the set of IPs of the pods selected by the Service. +Services, this resolves to the set of IPs of the Pods selected by the Service. Clients are expected to consume the set or else use standard round-robin selection from the set. @@ -87,37 +87,36 @@ SRV Records are created for named ports that are part of normal or [Headless Services](/docs/concepts/services-networking/service/#headless-services). For each named port, the SRV record would have the form `_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster-domain.example`. -For a regular service, this resolves to the port number and the domain name: +For a regular Service, this resolves to the port number and the domain name: `my-svc.my-namespace.svc.cluster-domain.example`. -For a headless service, this resolves to multiple answers, one for each pod -that is backing the service, and contains the port number and the domain name of the pod +For a headless Service, this resolves to multiple answers, one for each Pod +that is backing the Service, and contains the port number and the domain name of the Pod of the form `auto-generated-name.my-svc.my-namespace.svc.cluster-domain.example`. ## Pods ### A/AAAA records -In general a pod has the following DNS resolution: +In general a Pod has the following DNS resolution: `pod-ip-address.my-namespace.pod.cluster-domain.example`. -For example, if a pod in the `default` namespace has the IP address 172.17.0.3, +For example, if a Pod in the `default` namespace has the IP address 172.17.0.3, and the domain name for your cluster is `cluster.local`, then the Pod has a DNS name: `172-17-0-3.default.pod.cluster.local`. -Any pods created by a Deployment or DaemonSet exposed by a Service have the -following DNS resolution available: +Any Pods exposed by a Service have the following DNS resolution available: -`pod-ip-address.deployment-name.my-namespace.svc.cluster-domain.example`. +`pod-ip-address.service-name.my-namespace.svc.cluster-domain.example`. ### Pod's hostname and subdomain fields -Currently when a pod is created, its hostname is the Pod's `metadata.name` value. +Currently when a Pod is created, its hostname is the Pod's `metadata.name` value. The Pod spec has an optional `hostname` field, which can be used to specify the Pod's hostname. When specified, it takes precedence over the Pod's name to be -the hostname of the pod. For example, given a Pod with `hostname` set to +the hostname of the Pod. For example, given a Pod with `hostname` set to "`my-host`", the Pod will have its hostname set to "`my-host`". The Pod spec also has an optional `subdomain` field which can be used to specify @@ -174,14 +173,14 @@ spec: name: busybox ``` -If there exists a headless service in the same namespace as the pod and with +If there exists a headless Service in the same namespace as the Pod and with the same name as the subdomain, the cluster's DNS Server also returns an A or AAAA record for the Pod's fully qualified hostname. For example, given a Pod with the hostname set to "`busybox-1`" and the subdomain set to "`default-subdomain`", and a headless Service named "`default-subdomain`" in -the same namespace, the pod will see its own FQDN as +the same namespace, the Pod will see its own FQDN as "`busybox-1.default-subdomain.my-namespace.svc.cluster-domain.example`". DNS serves an -A or AAAA record at that name, pointing to the Pod's IP. Both pods "`busybox1`" and +A or AAAA record at that name, pointing to the Pod's IP. Both Pods "`busybox1`" and "`busybox2`" can have their distinct A or AAAA records. The Endpoints object can specify the `hostname` for any endpoint addresses, @@ -190,7 +189,7 @@ along with its IP. {{< note >}} Because A or AAAA records are not created for Pod names, `hostname` is required for the Pod's A or AAAA record to be created. A Pod with no `hostname` but with `subdomain` will only create the -A or AAAA record for the headless service (`default-subdomain.my-namespace.svc.cluster-domain.example`), +A or AAAA record for the headless Service (`default-subdomain.my-namespace.svc.cluster-domain.example`), pointing to the Pod's IP address. Also, Pod needs to become ready in order to have a record unless `publishNotReadyAddresses=True` is set on the Service. {{< /note >}} @@ -206,27 +205,28 @@ When you set `setHostnameAsFQDN: true` in the Pod spec, the kubelet writes the P {{< note >}} In Linux, the hostname field of the kernel (the `nodename` field of `struct utsname`) is limited to 64 characters. -If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment. +If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from Pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment. {{< /note >}} ### Pod's DNS Policy -DNS policies can be set on a per-pod basis. Currently Kubernetes supports the -following pod-specific DNS policies. These policies are specified in the +DNS policies can be set on a per-Pod basis. Currently Kubernetes supports the +following Pod-specific DNS policies. These policies are specified in the `dnsPolicy` field of a Pod Spec. - "`Default`": The Pod inherits the name resolution configuration from the node - that the pods run on. - See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node) + that the Pods run on. + See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers) for more details. - "`ClusterFirst`": Any DNS query that does not match the configured cluster domain suffix, such as "`www.kubernetes.io`", is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured. - See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#effects-on-pods) + See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers) for details on how DNS queries are handled in those cases. - "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should explicitly set its DNS policy "`ClusterFirstWithHostNet`". + - Note: This is not supported on Windows. See [below](#dns-windows) for details - "`None`": It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the `dnsConfig` field in the Pod Spec. @@ -307,7 +307,7 @@ For IPv6 setup, search path and name server should be setup like this: kubectl exec -it dns-example -- cat /etc/resolv.conf ``` The output is similar to this: -```shell +``` nameserver fd00:79:30::a search default.svc.cluster-domain.example svc.cluster-domain.example cluster-domain.example options ndots:5 @@ -324,19 +324,25 @@ If the feature gate `ExpandedDNSConfig` is enabled for the kube-apiserver and the kubelet, it is allowed for Kubernetes to have at most 32 search domains and a list of search domains of up to 2048 characters. -### Feature availability - -The availability of Pod DNS Config and DNS Policy "`None`" is shown as below. - -| k8s version | Feature support | -| :---------: |:-----------:| -| 1.14 | Stable | -| 1.10 | Beta (on by default)| -| 1.9 | Alpha | +## DNS resolution on Windows nodes {#dns-windows} +- ClusterFirstWithHostNet is not supported for Pods that run on Windows nodes. + Windows treats all names with a `.` as a FQDN and skips FQDN resolution. +- On Windows, there are multiple DNS resolvers that can be used. As these come with + slightly different behaviors, using the + [`Resolve-DNSName`](https://docs.microsoft.com/powershell/module/dnsclient/resolve-dnsname) + powershell cmdlet for name query resolutions is recommended. +- On Linux, you have a DNS suffix list, which is used after resolution of a name as fully + qualified has failed. + On Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that + Pod's namespace (example: `mydns.svc.cluster.local`). Windows can resolve FQDNs, Services, + or network name which can be resolved with this single suffix. For example, a Pod spawned + in the `default` namespace, will have the DNS suffix `default.svc.cluster.local`. + Inside a Windows Pod, you can resolve both `kubernetes.default.svc.cluster.local` + and `kubernetes`, but not the partially qualified names (`kubernetes.default` or + `kubernetes.default.svc`). ## {{% heading "whatsnext" %}} - For guidance on administering DNS configurations, check [Configure DNS Service](/docs/tasks/administer-cluster/dns-custom-nameservers/) diff --git a/content/en/docs/concepts/services-networking/dual-stack.md b/content/en/docs/concepts/services-networking/dual-stack.md index a85226beed..1716d7483b 100644 --- a/content/en/docs/concepts/services-networking/dual-stack.md +++ b/content/en/docs/concepts/services-networking/dual-stack.md @@ -1,28 +1,27 @@ --- -reviewers: -- lachie83 -- khenidak -- aramase -- bridgetkromhout title: IPv4/IPv6 dual-stack feature: title: IPv4/IPv6 dual-stack description: > Allocation of IPv4 and IPv6 addresses to Pods and Services - content_type: concept +reviewers: + - lachie83 + - khenidak + - aramase + - bridgetkromhout weight: 70 --- <!-- overview --> -{{< feature-state for_k8s_version="v1.21" state="beta" >}} - -IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to {{< glossary_tooltip text="Pods" term_id="pod" >}} and {{< glossary_tooltip text="Services" term_id="service" >}}. - -IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses. +{{< feature-state for_k8s_version="v1.23" state="stable" >}} +IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to +{{< glossary_tooltip text="Pods" term_id="pod" >}} and {{< glossary_tooltip text="Services" term_id="service" >}}. +IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in +1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses. <!-- body --> @@ -30,65 +29,78 @@ IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluste IPv4/IPv6 dual-stack on your Kubernetes cluster provides the following features: - * Dual-stack Pod networking (a single IPv4 and IPv6 address assignment per Pod) - * IPv4 and IPv6 enabled Services - * Pod off-cluster egress routing (eg. the Internet) via both IPv4 and IPv6 interfaces +* Dual-stack Pod networking (a single IPv4 and IPv6 address assignment per Pod) +* IPv4 and IPv6 enabled Services +* Pod off-cluster egress routing (eg. the Internet) via both IPv4 and IPv6 interfaces ## Prerequisites The following prerequisites are needed in order to utilize IPv4/IPv6 dual-stack Kubernetes clusters: - * Kubernetes 1.20 or later - For information about using dual-stack services with earlier - Kubernetes versions, refer to the documentation for that version - of Kubernetes. - * Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide Kubernetes nodes with routable IPv4/IPv6 network interfaces) - * A network plugin that supports dual-stack (such as Kubenet or Calico) +* Kubernetes 1.20 or later + + For information about using dual-stack services with earlier + Kubernetes versions, refer to the documentation for that version + of Kubernetes. + +* Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide + Kubernetes nodes with routable IPv4/IPv6 network interfaces) +* A [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) that + supports dual-stack networking. ## Configure IPv4/IPv6 dual-stack -To use IPv4/IPv6 dual-stack, ensure the `IPv6DualStack` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled for the relevant components of your cluster. (Starting in 1.21, IPv4/IPv6 dual-stack defaults to enabled.) - To configure IPv4/IPv6 dual-stack, set dual-stack cluster network assignments: - * kube-apiserver: - * `--service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>` - * kube-controller-manager: - * `--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR>` - * `--service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>` - * `--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6 - * kube-proxy: - * `--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR>` +* kube-apiserver: + * `--service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>` +* kube-controller-manager: + * `--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR>` + * `--service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>` + * `--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6 +* kube-proxy: + * `--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR>` +* kubelet: + * when there is no `--cloud-provider` the administrator can pass a comma-separated pair of IP + addresses via `--node-ip` to manually configure dual-stack `.status.addresses` for that Node. + If a Pod runs on that node in HostNetwork mode, the Pod reports these IP addresses in its + `.status.podIPs` field. + All `podIPs` in a node match the IP family preference defined by the `.status.addresses` + field for that Node. {{< note >}} An example of an IPv4 CIDR: `10.244.0.0/16` (though you would supply your own address range) -An example of an IPv6 CIDR: `fdXY:IJKL:MNOP:15::/64` (this shows the format but is not a valid address - see [RFC 4193](https://tools.ietf.org/html/rfc4193)) - -Starting in 1.21, IPv4/IPv6 dual-stack defaults to enabled. -You can disable it when necessary by specifying `--feature-gates="IPv6DualStack=false"` -on the kube-apiserver, kube-controller-manager, kubelet, and kube-proxy command line. +An example of an IPv6 CIDR: `fdXY:IJKL:MNOP:15::/64` (this shows the format but is not a valid +address - see [RFC 4193](https://tools.ietf.org/html/rfc4193)) {{< /note >}} ## Services You can create {{< glossary_tooltip text="Services" term_id="service" >}} which can use IPv4, IPv6, or both. -The address family of a Service defaults to the address family of the first service cluster IP range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver). +The address family of a Service defaults to the address family of the first service cluster IP +range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver). When you define a Service you can optionally configure it as dual stack. To specify the behavior you want, you set the `.spec.ipFamilyPolicy` field to one of the following values: -* `SingleStack`: Single-stack service. The control plane allocates a cluster IP for the Service, using the first configured service cluster IP range. +* `SingleStack`: Single-stack service. The control plane allocates a cluster IP for the Service, + using the first configured service cluster IP range. * `PreferDualStack`: - * Allocates IPv4 and IPv6 cluster IPs for the Service. (If the cluster has `--feature-gates="IPv6DualStack=false"`, this setting follows the same behavior as `SingleStack`.) + * Allocates IPv4 and IPv6 cluster IPs for the Service. * `RequireDualStack`: Allocates Service `.spec.ClusterIPs` from both IPv4 and IPv6 address ranges. - * Selects the `.spec.ClusterIP` from the list of `.spec.ClusterIPs` based on the address family of the first element in the `.spec.ipFamilies` array. + * Selects the `.spec.ClusterIP` from the list of `.spec.ClusterIPs` based on the address family + of the first element in the `.spec.ipFamilies` array. -If you would like to define which IP family to use for single stack or define the order of IP families for dual-stack, you can choose the address families by setting an optional field, `.spec.ipFamilies`, on the Service. +If you would like to define which IP family to use for single stack or define the order of IP +families for dual-stack, you can choose the address families by setting an optional field, +`.spec.ipFamilies`, on the Service. {{< note >}} -The `.spec.ipFamilies` field is immutable because the `.spec.ClusterIP` cannot be reallocated on a Service that already exists. If you want to change `.spec.ipFamilies`, delete and recreate the Service. +The `.spec.ipFamilies` field is immutable because the `.spec.ClusterIP` cannot be reallocated on a +Service that already exists. If you want to change `.spec.ipFamilies`, delete and recreate the +Service. {{< /note >}} You can set `.spec.ipFamilies` to any of the following array values: @@ -106,139 +118,197 @@ These examples demonstrate the behavior of various dual-stack Service configurat #### Dual-stack options on new Services -1. This Service specification does not explicitly define `.spec.ipFamilyPolicy`. When you create this Service, Kubernetes assigns a cluster IP for the Service from the first configured `service-cluster-ip-range` and sets the `.spec.ipFamilyPolicy` to `SingleStack`. ([Services without selectors](/docs/concepts/services-networking/service/#services-without-selectors) and [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors will behave in this same way.) +1. This Service specification does not explicitly define `.spec.ipFamilyPolicy`. When you create + this Service, Kubernetes assigns a cluster IP for the Service from the first configured + `service-cluster-ip-range` and sets the `.spec.ipFamilyPolicy` to `SingleStack`. ([Services + without selectors](/docs/concepts/services-networking/service/#services-without-selectors) and + [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors + will behave in this same way.) -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} -1. This Service specification explicitly defines `PreferDualStack` in `.spec.ipFamilyPolicy`. When you create this Service on a dual-stack cluster, Kubernetes assigns both IPv4 and IPv6 addresses for the service. The control plane updates the `.spec` for the Service to record the IP address assignments. The field `.spec.ClusterIPs` is the primary field, and contains both assigned IP addresses; `.spec.ClusterIP` is a secondary field with its value calculated from `.spec.ClusterIPs`. - - * For the `.spec.ClusterIP` field, the control plane records the IP address that is from the same address family as the first service cluster IP range. - * On a single-stack cluster, the `.spec.ClusterIPs` and `.spec.ClusterIP` fields both only list one address. - * On a cluster with dual-stack enabled, specifying `RequireDualStack` in `.spec.ipFamilyPolicy` behaves the same as `PreferDualStack`. +1. This Service specification explicitly defines `PreferDualStack` in `.spec.ipFamilyPolicy`. When + you create this Service on a dual-stack cluster, Kubernetes assigns both IPv4 and IPv6 + addresses for the service. The control plane updates the `.spec` for the Service to record the IP + address assignments. The field `.spec.ClusterIPs` is the primary field, and contains both assigned + IP addresses; `.spec.ClusterIP` is a secondary field with its value calculated from + `.spec.ClusterIPs`. -{{< codenew file="service/networking/dual-stack-preferred-svc.yaml" >}} + * For the `.spec.ClusterIP` field, the control plane records the IP address that is from the + same address family as the first service cluster IP range. + * On a single-stack cluster, the `.spec.ClusterIPs` and `.spec.ClusterIP` fields both only list + one address. + * On a cluster with dual-stack enabled, specifying `RequireDualStack` in `.spec.ipFamilyPolicy` + behaves the same as `PreferDualStack`. -1. This Service specification explicitly defines `IPv6` and `IPv4` in `.spec.ipFamilies` as well as defining `PreferDualStack` in `.spec.ipFamilyPolicy`. When Kubernetes assigns an IPv6 and IPv4 address in `.spec.ClusterIPs`, `.spec.ClusterIP` is set to the IPv6 address because that is the first element in the `.spec.ClusterIPs` array, overriding the default. + {{< codenew file="service/networking/dual-stack-preferred-svc.yaml" >}} -{{< codenew file="service/networking/dual-stack-preferred-ipfamilies-svc.yaml" >}} +1. This Service specification explicitly defines `IPv6` and `IPv4` in `.spec.ipFamilies` as well + as defining `PreferDualStack` in `.spec.ipFamilyPolicy`. When Kubernetes assigns an IPv6 and + IPv4 address in `.spec.ClusterIPs`, `.spec.ClusterIP` is set to the IPv6 address because that is + the first element in the `.spec.ClusterIPs` array, overriding the default. + + {{< codenew file="service/networking/dual-stack-preferred-ipfamilies-svc.yaml" >}} #### Dual-stack defaults on existing Services -These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster where Services already exist. (Upgrading an existing cluster to 1.21 will enable dual-stack unless `--feature-gates="IPv6DualStack=false"` is set.) +These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster +where Services already exist. (Upgrading an existing cluster to 1.21 or beyond will enable +dual-stack.) -1. When dual-stack is enabled on a cluster, existing Services (whether `IPv4` or `IPv6`) are configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set `.spec.ipFamilies` to the address family of the existing Service. The existing Service cluster IP will be stored in `.spec.ClusterIPs`. +1. When dual-stack is enabled on a cluster, existing Services (whether `IPv4` or `IPv6`) are + configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set + `.spec.ipFamilies` to the address family of the existing Service. The existing Service cluster IP + will be stored in `.spec.ClusterIPs`. -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} You can validate this behavior by using kubectl to inspect an existing service. -```shell -kubectl get svc my-service -o yaml -``` + ```shell + kubectl get svc my-service -o yaml + ``` -```yaml -apiVersion: v1 -kind: Service -metadata: - labels: - app: MyApp - name: my-service -spec: - clusterIP: 10.0.197.123 - clusterIPs: - - 10.0.197.123 - ipFamilies: - - IPv4 - ipFamilyPolicy: SingleStack - ports: - - port: 80 - protocol: TCP - targetPort: 80 - selector: - app: MyApp - type: ClusterIP -status: - loadBalancer: {} -``` + ```yaml + apiVersion: v1 + kind: Service + metadata: + labels: + app.kubernetes.io/name: MyApp + name: my-service + spec: + clusterIP: 10.0.197.123 + clusterIPs: + - 10.0.197.123 + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + targetPort: 80 + selector: + app.kubernetes.io/name: MyApp + type: ClusterIP + status: + loadBalancer: {} + ``` -1. When dual-stack is enabled on a cluster, existing [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors are configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set `.spec.ipFamilies` to the address family of the first service cluster IP range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver) even though `.spec.ClusterIP` is set to `None`. +1. When dual-stack is enabled on a cluster, existing + [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors are + configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set + `.spec.ipFamilies` to the address family of the first service cluster IP range (configured via the + `--service-cluster-ip-range` flag to the kube-apiserver) even though `.spec.ClusterIP` is set to + `None`. -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} You can validate this behavior by using kubectl to inspect an existing headless service with selectors. -```shell -kubectl get svc my-service -o yaml -``` + ```shell + kubectl get svc my-service -o yaml + ``` -```yaml -apiVersion: v1 -kind: Service -metadata: - labels: - app: MyApp - name: my-service -spec: - clusterIP: None - clusterIPs: - - None - ipFamilies: - - IPv4 - ipFamilyPolicy: SingleStack - ports: - - port: 80 - protocol: TCP - targetPort: 80 - selector: - app: MyApp -``` + ```yaml + apiVersion: v1 + kind: Service + metadata: + labels: + app.kubernetes.io/name: MyApp + name: my-service + spec: + clusterIP: None + clusterIPs: + - None + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + targetPort: 80 + selector: + app.kubernetes.io/name: MyApp + ``` #### Switching Services between single-stack and dual-stack Services can be changed from single-stack to dual-stack and from dual-stack to single-stack. -1. To change a Service from single-stack to dual-stack, change `.spec.ipFamilyPolicy` from `SingleStack` to `PreferDualStack` or `RequireDualStack` as desired. When you change this Service from single-stack to dual-stack, Kubernetes assigns the missing address family so that the Service now has IPv4 and IPv6 addresses. +1. To change a Service from single-stack to dual-stack, change `.spec.ipFamilyPolicy` from + `SingleStack` to `PreferDualStack` or `RequireDualStack` as desired. When you change this + Service from single-stack to dual-stack, Kubernetes assigns the missing address family so that the + Service now has IPv4 and IPv6 addresses. Edit the Service specification updating the `.spec.ipFamilyPolicy` from `SingleStack` to `PreferDualStack`. -Before: -```yaml -spec: - ipFamilyPolicy: SingleStack -``` -After: -```yaml -spec: - ipFamilyPolicy: PreferDualStack -``` + Before: -1. To change a Service from dual-stack to single-stack, change `.spec.ipFamilyPolicy` from `PreferDualStack` or `RequireDualStack` to `SingleStack`. When you change this Service from dual-stack to single-stack, Kubernetes retains only the first element in the `.spec.ClusterIPs` array, and sets `.spec.ClusterIP` to that IP address and sets `.spec.ipFamilies` to the address family of `.spec.ClusterIPs`. + ```yaml + spec: + ipFamilyPolicy: SingleStack + ``` + + After: + + ```yaml + spec: + ipFamilyPolicy: PreferDualStack + ``` + +1. To change a Service from dual-stack to single-stack, change `.spec.ipFamilyPolicy` from + `PreferDualStack` or `RequireDualStack` to `SingleStack`. When you change this Service from + dual-stack to single-stack, Kubernetes retains only the first element in the `.spec.ClusterIPs` + array, and sets `.spec.ClusterIP` to that IP address and sets `.spec.ipFamilies` to the address + family of `.spec.ClusterIPs`. ### Headless Services without selector -For [Headless Services without selectors](/docs/concepts/services-networking/service/#without-selectors) and without `.spec.ipFamilyPolicy` explicitly set, the `.spec.ipFamilyPolicy` field defaults to `RequireDualStack`. +For [Headless Services without selectors](/docs/concepts/services-networking/service/#without-selectors) +and without `.spec.ipFamilyPolicy` explicitly set, the `.spec.ipFamilyPolicy` field defaults to +`RequireDualStack`. ### Service type LoadBalancer To provision a dual-stack load balancer for your Service: - * Set the `.spec.type` field to `LoadBalancer` - * Set `.spec.ipFamilyPolicy` field to `PreferDualStack` or `RequireDualStack` + +* Set the `.spec.type` field to `LoadBalancer` +* Set `.spec.ipFamilyPolicy` field to `PreferDualStack` or `RequireDualStack` {{< note >}} -To use a dual-stack `LoadBalancer` type Service, your cloud provider must support IPv4 and IPv6 load balancers. +To use a dual-stack `LoadBalancer` type Service, your cloud provider must support IPv4 and IPv6 +load balancers. {{< /note >}} ## Egress traffic -If you want to enable egress traffic in order to reach off-cluster destinations (eg. the public Internet) from a Pod that uses non-publicly routable IPv6 addresses, you need to enable the Pod to use a publicly routed IPv6 address via a mechanism such as transparent proxying or IP masquerading. The [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent) project supports IP masquerading on dual-stack clusters. +If you want to enable egress traffic in order to reach off-cluster destinations (eg. the public +Internet) from a Pod that uses non-publicly routable IPv6 addresses, you need to enable the Pod to +use a publicly routed IPv6 address via a mechanism such as transparent proxying or IP +masquerading. The [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent) project +supports IP masquerading on dual-stack clusters. {{< note >}} Ensure your {{< glossary_tooltip text="CNI" term_id="cni" >}} provider supports IPv6. {{< /note >}} +## Windows support + +Kubernetes on Windows does not support single-stack "IPv6-only" networking. However, +dual-stack IPv4/IPv6 networking for pods and nodes with single-family services +is supported. + +You can use IPv4/IPv6 dual-stack networking with `l2bridge` networks. + +{{< note >}} +Overlay (VXLAN) networks on Windows **do not** support dual-stack networking. +{{< /note >}} + +You can read more about the different network modes for Windows within the +[Networking on Windows](/docs/concepts/services-networking/windows-networking#network-modes) topic. + ## {{% heading "whatsnext" %}} - * [Validate IPv4/IPv6 dual-stack](/docs/tasks/network/validate-dual-stack) networking -* [Enable dual-stack networking using kubeadm -](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/) +* [Enable dual-stack networking using kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/) + diff --git a/content/en/docs/concepts/services-networking/endpoint-slices.md b/content/en/docs/concepts/services-networking/endpoint-slices.md index da8d09d9d5..cd90ecdcfd 100644 --- a/content/en/docs/concepts/services-networking/endpoint-slices.md +++ b/content/en/docs/concepts/services-networking/endpoint-slices.md @@ -147,9 +147,9 @@ In the v1 API, the per endpoint `topology` was effectively removed in favor of the dedicated fields `nodeName` and `zone`. Setting arbitrary topology fields on the `endpoint` field of an `EndpointSlice` -resource has been deprecated and is not be supported in the v1 API. Instead, -the v1 API supports setting individual `nodeName` and `zone` fields. These -fields are automatically translated between API versions. For example, the +resource has been deprecated and is not supported in the v1 API. +Instead, the v1 API supports setting individual `nodeName` and `zone` fields. +These fields are automatically translated between API versions. For example, the value of the `"topology.kubernetes.io/zone"` key in the `topology` field in the v1beta1 API is accessible as the `zone` field in the v1 API. {{< /note >}} diff --git a/content/en/docs/concepts/services-networking/ingress-controllers.md b/content/en/docs/concepts/services-networking/ingress-controllers.md index 0ee1d53ef9..f2d593448e 100644 --- a/content/en/docs/concepts/services-networking/ingress-controllers.md +++ b/content/en/docs/concepts/services-networking/ingress-controllers.md @@ -23,16 +23,17 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet {{% thirdparty-content %}} -* [AKS Application Gateway Ingress Controller](https://azure.github.io/application-gateway-kubernetes-ingress/) is an ingress controller that configures the [Azure Application Gateway](https://docs.microsoft.com/azure/application-gateway/overview). +* [AKS Application Gateway Ingress Controller](https://docs.microsoft.com/azure/application-gateway/tutorial-ingress-controller-add-on-existing?toc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Ftoc.json&bc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json) is an ingress controller that configures the [Azure Application Gateway](https://docs.microsoft.com/azure/application-gateway/overview). * [Ambassador](https://www.getambassador.io/) API Gateway is an [Envoy](https://www.envoyproxy.io)-based ingress controller. * [Apache APISIX ingress controller](https://github.com/apache/apisix-ingress-controller) is an [Apache APISIX](https://github.com/apache/apisix)-based ingress controller. * [Avi Kubernetes Operator](https://github.com/vmware/load-balancer-and-ingress-services-for-kubernetes) provides L4-L7 load-balancing using [VMware NSX Advanced Load Balancer](https://avinetworks.com/). +* [BFE Ingress Controller](https://github.com/bfenetworks/ingress-bfe) is a [BFE](https://www.bfe-networks.net)-based ingress controller. * The [Citrix ingress controller](https://github.com/citrix/citrix-k8s-ingress-controller#readme) works with Citrix Application Delivery Controller. * [Contour](https://projectcontour.io/) is an [Envoy](https://www.envoyproxy.io/) based ingress controller. * [EnRoute](https://getenroute.io/) is an [Envoy](https://www.envoyproxy.io) based API gateway that can run as an ingress controller. -* [Easegress IngressController](https://github.com/megaease/easegress/blob/main/doc/ingresscontroller.md) is an [Easegress](https://megaease.com/easegress/) based API gateway that can run as an ingress controller. +* [Easegress IngressController](https://github.com/megaease/easegress/blob/main/doc/reference/ingresscontroller.md) is an [Easegress](https://megaease.com/easegress/) based API gateway that can run as an ingress controller. * F5 BIG-IP [Container Ingress Services for Kubernetes](https://clouddocs.f5.com/containers/latest/userguide/kubernetes/) lets you use an Ingress to configure F5 BIG-IP virtual servers. * [Gloo](https://gloo.solo.io) is an open-source ingress controller based on [Envoy](https://www.envoyproxy.io), @@ -45,8 +46,10 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet is an [Istio](https://istio.io/) based ingress controller. * The [Kong Ingress Controller for Kubernetes](https://github.com/Kong/kubernetes-ingress-controller#readme) is an ingress controller driving [Kong Gateway](https://konghq.com/kong/). +* [Kusk Gateway](https://kusk.kubeshop.io/) is an OpenAPI-driven ingress controller based on [Envoy](https://www.envoyproxy.io). * The [NGINX Ingress Controller for Kubernetes](https://www.nginx.com/products/nginx-ingress-controller/) works with the [NGINX](https://www.nginx.com/resources/glossary/nginx/) webserver (as a proxy). +* The [Pomerium Ingress Controller](https://www.pomerium.com/docs/k8s/ingress.html) is based on [Pomerium](https://pomerium.com/), which offers context-aware access policy. * [Skipper](https://opensource.zalando.com/skipper/kubernetes/ingress-controller/) HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy. * The [Traefik Kubernetes Ingress provider](https://doc.traefik.io/traefik/providers/kubernetes-ingress/) is an ingress controller for the [Traefik](https://traefik.io/traefik/) proxy. @@ -56,12 +59,11 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet ## Using multiple Ingress controllers -You may deploy [any number of ingress controllers](https://git.k8s.io/ingress-nginx/docs/user-guide/multiple-ingress.md#multiple-ingress-controllers) -within a cluster. When you create an ingress, you should annotate each ingress with the appropriate -[`ingress.class`](https://git.k8s.io/ingress-gce/docs/faq/README.md#how-do-i-run-multiple-ingress-controllers-in-the-same-cluster) -to indicate which ingress controller should be used if more than one exists within your cluster. +You may deploy any number of ingress controllers using [ingress class](/docs/concepts/services-networking/ingress/#ingress-class) +within a cluster. Note the `.metadata.name` of your ingress class resource. When you create an ingress you would need that name to specify the `ingressClassName` field on your Ingress object (refer to [IngressSpec v1 reference](/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec). `ingressClassName` is a replacement of the older [annotation method](/docs/concepts/services-networking/ingress/#deprecated-annotation). -If you do not define a class, your cloud provider may use a default ingress controller. +If you do not specify an IngressClass for an Ingress, and your cluster has exactly one IngressClass marked as default, then Kubernetes [applies](/docs/concepts/services-networking/ingress/#default-ingress-class) the cluster's default IngressClass to the Ingress. +You mark an IngressClass as default by setting the [`ingressclass.kubernetes.io/is-default-class` annotation](/docs/reference/labels-annotations-taints/#ingressclass-kubernetes-io-is-default-class) on that IngressClass, with the string value `"true"`. Ideally, all ingress controllers should fulfill this specification, but the various ingress controllers operate slightly differently. diff --git a/content/en/docs/concepts/services-networking/ingress.md b/content/en/docs/concepts/services-networking/ingress.md index 6879b998db..aebcc800c6 100644 --- a/content/en/docs/concepts/services-networking/ingress.md +++ b/content/en/docs/concepts/services-networking/ingress.md @@ -30,28 +30,13 @@ For clarity, this guide defines the following terms: Traffic routing is controlled by rules defined on the Ingress resource. Here is a simple example where an Ingress sends all its traffic to one Service: -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress]; - ingress-->|routing rule|service[Service]; - subgraph cluster - ingress; - service-->pod1[Pod]; - service-->pod2[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service,pod1,pod2 k8s; - class client plain; - class cluster cluster; -{{</ mermaid >}} +{{< figure src="/docs/images/ingress.svg" alt="ingress-diagram" class="diagram-large" caption="Figure. Ingress" link="https://mermaid.live/edit#pako:eNqNkstuwyAQRX8F4U0r2VHqPlSRKqt0UamLqlnaWWAYJygYLB59KMm_Fxcix-qmGwbuXA7DwAEzzQETXKutof0Ovb4vaoUQkwKUu6pi3FwXM_QSHGBt0VFFt8DRU2OWSGrKUUMlVQwMmhVLEV1Vcm9-aUksiuXRaO_CEhkv4WjBfAgG1TrGaLa-iaUw6a0DcwGI-WgOsF7zm-pN881fvRx1UDzeiFq7ghb1kgqFWiElyTjnuXVG74FkbdumefEpuNuRu_4rZ1pqQ7L5fL6YQPaPNiFuywcG9_-ihNyUkm6YSONWkjVNM8WUIyaeOJLO3clTB_KhL8NQDmVe-OJjxgZM5FhFiiFTK5zjDkxHBQ9_4zB4a-x20EGNSZhyaKmXrg7f5hSsvufUwTMXThtMWiot5Jh6p9ffimHijIezaSVoeN0uiqcfMJvf7w" >}} An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting. An [Ingress controller](/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic. An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically -uses a service of type [Service.Type=NodePort](/docs/concepts/services-networking/service/#nodeport) or +uses a service of type [Service.Type=NodePort](/docs/concepts/services-networking/service/#type-nodeport) or [Service.Type=LoadBalancer](/docs/concepts/services-networking/service/#loadbalancer). ## Prerequisites @@ -74,13 +59,13 @@ A minimal Ingress resource example: {{< codenew file="service/networking/minimal-ingress.yaml" >}} -As with all other Kubernetes resources, an Ingress needs `apiVersion`, `kind`, and `metadata` fields. +An Ingress needs `apiVersion`, `kind`, `metadata` and `spec` fields. The name of an Ingress object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). For general information about working with config files, see [deploying applications](/docs/tasks/run-application/run-stateless-application-deployment/), [configuring containers](/docs/tasks/configure-pod-container/configure-pod-configmap/), [managing resources](/docs/concepts/cluster-administration/manage-deployment/). Ingress frequently uses annotations to configure some options depending on the Ingress controller, an example of which is the [rewrite-target annotation](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md). -Different [Ingress controller](/docs/concepts/services-networking/ingress-controllers) support different annotations. Review the documentation for +Different [Ingress controllers](/docs/concepts/services-networking/ingress-controllers) support different annotations. Review the documentation for your choice of Ingress controller to learn which annotations are supported. The Ingress [spec](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) @@ -88,6 +73,15 @@ has all the information needed to configure a load balancer or proxy server. Mos contains a list of rules matched against all incoming requests. Ingress resource only supports rules for directing HTTP(S) traffic. +If the `ingressClassName` is omitted, a [default Ingress class](#default-ingress-class) +should be defined. + +There are some ingress controllers, that work without the definition of a +default `IngressClass`. For example, the Ingress-NGINX controller can be +configured with a [flag](https://kubernetes.github.io/ingress-nginx/#what-is-the-flag-watch-ingress-without-class) +`--watch-ingress-without-class`. It is [recommended](https://kubernetes.github.io/ingress-nginx/#i-have-only-one-instance-of-the-ingresss-nginx-controller-in-my-cluster-what-should-i-do) though, to specify the +default `IngressClass` as shown [below](#default-ingress-class). + ### Ingress rules Each HTTP rule contains the following information: @@ -109,8 +103,14 @@ match a path in the spec. ### DefaultBackend {#default-backend} -An Ingress with no rules sends all traffic to a single default backend. The `defaultBackend` is conventionally a configuration option -of the [Ingress controller](/docs/concepts/services-networking/ingress-controllers) and is not specified in your Ingress resources. +An Ingress with no rules sends all traffic to a single default backend and `.spec.defaultBackend` +is the backend that should handle requests in that case. +The `defaultBackend` is conventionally a configuration option of the +[Ingress controller](/docs/concepts/services-networking/ingress-controllers) and +is not specified in your Ingress resources. +If no `.spec.rules` are specified, `.spec.defaultBackend` must be specified. +If `defaultBackend` is not set, the handling of requests that do not match any of the rules will be up to the +ingress controller (consult the documentation for your ingress controller to find out how it handles this case). If none of the hosts or paths match the HTTP request in the Ingress objects, the traffic is routed to your default backend. @@ -219,25 +219,98 @@ of the controller that should implement the class. {{< codenew file="service/networking/external-lb.yaml" >}} -IngressClass resources contain an optional parameters field. This can be used to -reference additional implementation-specific configuration for this class. +The `.spec.parameters` field of an IngressClass lets you reference another +resource that provides configuration related to that IngressClass. -#### Namespace-scoped parameters +The specific type of parameters to use depends on the ingress controller +that you specify in the `.spec.controller` field of the IngressClass. -{{< feature-state for_k8s_version="v1.22" state="beta" >}} +### IngressClass scope -`Parameters` field has a `scope` and `namespace` field that can be used to -reference a namespace-specific resource for configuration of an Ingress class. -`Scope` field defaults to `Cluster`, meaning, the default is cluster-scoped -resource. Setting `Scope` to `Namespace` and setting the `Namespace` field -will reference a parameters resource in a specific namespace: +Depending on your ingress controller, you may be able to use parameters +that you set cluster-wide, or just for one namespace. -Namespace-scoped parameters avoid the need for a cluster-scoped CustomResourceDefinition -for a parameters resource. This further avoids RBAC-related resources -that would otherwise be required to grant permissions to cluster-scoped -resources. +{{< tabs name="tabs_ingressclass_parameter_scope" >}} +{{% tab name="Cluster" %}} +The default scope for IngressClass parameters is cluster-wide. -{{< codenew file="service/networking/namespaced-params.yaml" >}} +If you set the `.spec.parameters` field and don't set +`.spec.parameters.scope`, or if you set `.spec.parameters.scope` to +`Cluster`, then the IngressClass refers to a cluster-scoped resource. +The `kind` (in combination the `apiGroup`) of the parameters +refers to a cluster-scoped API (possibly a custom resource), and +the `name` of the parameters identifies a specific cluster scoped +resource for that API. + +For example: +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: IngressClass +metadata: + name: external-lb-1 +spec: + controller: example.com/ingress-controller + parameters: + # The parameters for this IngressClass are specified in a + # ClusterIngressParameter (API group k8s.example.net) named + # "external-config-1". This definition tells Kubernetes to + # look for a cluster-scoped parameter resource. + scope: Cluster + apiGroup: k8s.example.net + kind: ClusterIngressParameter + name: external-config-1 +``` +{{% /tab %}} +{{% tab name="Namespaced" %}} +{{< feature-state for_k8s_version="v1.23" state="stable" >}} + +If you set the `.spec.parameters` field and set +`.spec.parameters.scope` to `Namespace`, then the IngressClass refers +to a namespaced-scoped resource. You must also set the `namespace` +field within `.spec.parameters` to the namespace that contains +the parameters you want to use. + +The `kind` (in combination the `apiGroup`) of the parameters +refers to a namespaced API (for example: ConfigMap), and +the `name` of the parameters identifies a specific resource +in the namespace you specified in `namespace`. + +Namespace-scoped parameters help the cluster operator delegate control over the +configuration (for example: load balancer settings, API gateway definition) +that is used for a workload. If you used a cluster-scoped parameter then either: + +- the cluster operator team needs to approve a different team's changes every + time there's a new configuration change being applied. +- the cluster operator must define specific access controls, such as + [RBAC](/docs/reference/access-authn-authz/rbac/) roles and bindings, that let + the application team make changes to the cluster-scoped parameters resource. + +The IngressClass API itself is always cluster-scoped. + +Here is an example of an IngressClass that refers to parameters that are +namespaced: +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: IngressClass +metadata: + name: external-lb-2 +spec: + controller: example.com/ingress-controller + parameters: + # The parameters for this IngressClass are specified in an + # IngressParameter (API group k8s.example.com) named "external-config", + # that's in the "external-configuration" namespace. + scope: Namespace + apiGroup: k8s.example.com + kind: IngressParameter + namespace: external-configuration + name: external-config +``` + +{{% /tab %}} +{{< /tabs >}} ### Deprecated annotation @@ -266,6 +339,14 @@ an `ingressClassName` specified. You can resolve this by ensuring that at most 1 IngressClass is marked as default in your cluster. {{< /caution >}} +There are some ingress controllers, that work without the definition of a +default `IngressClass`. For example, the Ingress-NGINX controller can be +configured with a [flag](https://kubernetes.github.io/ingress-nginx/#what-is-the-flag-watch-ingress-without-class) +`--watch-ingress-without-class`. It is [recommended](https://kubernetes.github.io/ingress-nginx/#i-have-only-one-instance-of-the-ingresss-nginx-controller-in-my-cluster-what-should-i-do) though, to specify the +default `IngressClass`: + +{{< codenew file="service/networking/default-ingressclass.yaml" >}} + ## Types of Ingress ### Ingress backed by a single Service {#single-service-ingress} @@ -302,25 +383,8 @@ A fanout configuration routes traffic from a single IP address to more than one based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers down to a minimum. For example, a setup like: -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress, 178.91.123.132]; - ingress-->|/foo|service1[Service service1:4200]; - ingress-->|/bar|service2[Service service2:8080]; - subgraph cluster - ingress; - service1-->pod1[Pod]; - service1-->pod2[Pod]; - service2-->pod3[Pod]; - service2-->pod4[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s; - class client plain; - class cluster cluster; -{{</ mermaid >}} +{{< figure src="/docs/images/ingressFanOut.svg" alt="ingress-fanout-diagram" class="diagram-large" caption="Figure. Ingress Fan Out" link="https://mermaid.live/edit#pako:eNqNUslOwzAQ_RXLvYCUhMQpUFzUUzkgcUBwbHpw4klr4diR7bCo8O8k2FFbFomLPZq3jP00O1xpDpjijWHtFt09zAuFUCUFKHey8vf6NE7QrdoYsDZumGIb4Oi6NAskNeOoZJKpCgxK4oXwrFVgRyi7nCVXWZKRPMlysv5yD6Q4Xryf1Vq_WzDPooJs9egLNDbolKTpT03JzKgh3zWEztJZ0Niu9L-qZGcdmAMfj4cxvWmreba613z9C0B-AMQD-V_AdA-A4j5QZu0SatRKJhSqhZR0wjmPrDP6CeikrutQxy-Cuy2dtq9RpaU2dJKm6fzI5Glmg0VOLio4_5dLjx27hFSC015KJ2VZHtuQvY2fuHcaE43G0MaCREOow_FV5cMxHZ5-oPX75UM5avuXhXuOI9yAaZjg_aLuBl6B3RYaKDDtSw4166QrcKE-emrXcubghgunDaY1kxYizDqnH99UhakzHYykpWD9hjS--fEJoIELqQ" >}} + would require an Ingress such as: @@ -364,25 +428,7 @@ you are using, you may need to create a default-http-backend Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address. -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress, 178.91.123.132]; - ingress-->|Host: foo.bar.com|service1[Service service1:80]; - ingress-->|Host: bar.foo.com|service2[Service service2:80]; - subgraph cluster - ingress; - service1-->pod1[Pod]; - service1-->pod2[Pod]; - service2-->pod3[Pod]; - service2-->pod4[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s; - class client plain; - class cluster cluster; -{{</ mermaid >}} +{{< figure src="/docs/images/ingressNameBased.svg" alt="ingress-namebase-diagram" class="diagram-large" caption="Figure. Ingress Name Based Virtual hosting" link="https://mermaid.live/edit#pako:eNqNkl9PwyAUxb8KYS-atM1Kp05m9qSJJj4Y97jugcLtRqTQAPVPdN_dVlq3qUt8gZt7zvkBN7xjbgRgiteW1Rt0_zjLNUJcSdD-ZBn21WmcoDu9tuBcXDHN1iDQVWHnSBkmUMEU0xwsSuK5DK5l745QejFNLtMkJVmSZmT1Re9NcTz_uDXOU1QakxTMJtxUHw7ss-SQLhehQEODTsdH4l20Q-zFyc84-Y67pghv5apxHuweMuj9eS2_NiJdPhix-kMgvwQShOyYMNkJoEUYM3PuGkpUKyY1KqVSdCSEiJy35gnoqCzLvo5fpPAbOqlfI26UsXQ0Ho9nB5CnqesRGTnncPYvSqsdUvqp9KRdlI6KojjEkB0mnLgjDRONhqENBYm6oXbLV5V1y6S7-l42_LowlIN2uFm_twqOcAW2YlK0H_i9c-bYb6CCHNO2FFCyRvkc53rbWptaMA83QnpjMS2ZchBh1nizeNMcU28bGEzXkrV_pArN7Sc0rBTu" >}} The following Ingress tells the backing load balancer to route requests based on @@ -395,9 +441,7 @@ web traffic to the IP address of your Ingress controller can be matched without virtual host being required. For example, the following Ingress routes traffic -requested for `first.bar.com` to `service1`, `second.bar.com` to `service2`, and any traffic -to the IP address without a hostname defined in request (that is, without a request header being -presented) to `service3`. +requested for `first.bar.com` to `service1`, `second.bar.com` to `service2`, and any traffic whose request host header doesn't match `first.bar.com` and `second.bar.com` to `service3`. {{< codenew file="service/networking/name-virtual-host-ingress-no-third-host.yaml" >}} @@ -570,6 +614,6 @@ You can expose a Service in multiple ways that don't directly involve the Ingres ## {{% heading "whatsnext" %}} -* Learn about the [Ingress API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#ingress-v1beta1-networking-k8s-io) +* Learn about the [Ingress](/docs/reference/kubernetes-api/service-resources/ingress-v1/) API * Learn about [Ingress controllers](/docs/concepts/services-networking/ingress-controllers/) * [Set up Ingress on Minikube with the NGINX Controller](/docs/tasks/access-application-cluster/ingress-minikube/) diff --git a/content/en/docs/concepts/services-networking/network-policies.md b/content/en/docs/concepts/services-networking/network-policies.md index e3f3a203b7..9a97abfb5b 100644 --- a/content/en/docs/concepts/services-networking/network-policies.md +++ b/content/en/docs/concepts/services-networking/network-policies.md @@ -10,7 +10,7 @@ weight: 50 <!-- overview --> -If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), then you might consider using Kubernetes NetworkPolicies for particular applications in your cluster. NetworkPolicies are an application-centric construct which allow you to specify how a {{< glossary_tooltip text="pod" term_id="pod">}} is allowed to communicate with various network "entities" (we use the word "entity" here to avoid overloading the more common terms such as "endpoints" and "services", which have specific Kubernetes connotations) over the network. +If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), then you might consider using Kubernetes NetworkPolicies for particular applications in your cluster. NetworkPolicies are an application-centric construct which allow you to specify how a {{< glossary_tooltip text="pod" term_id="pod">}} is allowed to communicate with various network "entities" (we use the word "entity" here to avoid overloading the more common terms such as "endpoints" and "services", which have specific Kubernetes connotations) over the network. NetworkPolicies apply to a connection with a pod on one or both ends, and are not relevant to other connections. The entities that a Pod can communicate with are identified through a combination of the following 3 identifiers: @@ -27,15 +27,17 @@ Meanwhile, when IP based NetworkPolicies are created, we define policies based o Network policies are implemented by the [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). To use network policies, you must be using a networking solution which supports NetworkPolicy. Creating a NetworkPolicy resource without a controller that implements it will have no effect. -## Isolated and Non-isolated Pods +## The Two Sorts of Pod Isolation -By default, pods are non-isolated; they accept traffic from any source. +There are two sorts of isolation for a pod: isolation for egress, and isolation for ingress. They concern what connections may be established. "Isolation" here is not absolute, rather it means "some restrictions apply". The alternative, "non-isolated for $direction", means that no restrictions apply in the stated direction. The two sorts of isolation (or not) are declared independently, and are both relevant for a connection from one pod to another. -Pods become isolated by having a NetworkPolicy that selects them. Once there is any NetworkPolicy in a namespace selecting a particular pod, that pod will reject any connections that are not allowed by any NetworkPolicy. (Other pods in the namespace that are not selected by any NetworkPolicy will continue to accept all traffic.) +By default, a pod is non-isolated for egress; all outbound connections are allowed. A pod is isolated for egress if there is any NetworkPolicy that both selects the pod and has "Egress" in its `policyTypes`; we say that such a policy applies to the pod for egress. When a pod is isolated for egress, the only allowed connections from the pod are those allowed by the `egress` list of some NetworkPolicy that applies to the pod for egress. The effects of those `egress` lists combine additively. -Network policies do not conflict; they are additive. If any policy or policies select a pod, the pod is restricted to what is allowed by the union of those policies' ingress/egress rules. Thus, order of evaluation does not affect the policy result. +By default, a pod is non-isolated for ingress; all inbound connections are allowed. A pod is isolated for ingress if there is any NetworkPolicy that both selects the pod and has "Ingress" in its `policyTypes`; we say that such a policy applies to the pod for ingress. When a pod is isolated for ingress, the only allowed connections into the pod are those from the pod's node and those allowed by the `ingress` list of some NetworkPolicy that applies to the pod for ingress. The effects of those `ingress` lists combine additively. -For a network flow between two pods to be allowed, both the egress policy on the source pod and the ingress policy on the destination pod need to allow the traffic. If either the egress policy on the source, or the ingress policy on the destination denies the traffic, the traffic will be denied. +Network policies do not conflict; they are additive. If any policy or policies apply to a given pod for a given direction, the connections allowed in that direction from that pod is the union of what the applicable policies allow. Thus, order of evaluation does not affect the policy result. + +For a connection from a source pod to a destination pod to be allowed, both the egress policy on the source pod and the ingress policy on the destination pod need to allow the connection. If either side does not allow the connection, it will not happen. ## The NetworkPolicy resource {#networkpolicy-resource} @@ -43,42 +45,7 @@ See the [NetworkPolicy](/docs/reference/generated/kubernetes-api/{{< param "vers An example NetworkPolicy might look like this: -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: test-network-policy - namespace: default -spec: - podSelector: - matchLabels: - role: db - policyTypes: - - Ingress - - Egress - ingress: - - from: - - ipBlock: - cidr: 172.17.0.0/16 - except: - - 172.17.1.0/24 - - namespaceSelector: - matchLabels: - project: myproject - - podSelector: - matchLabels: - role: frontend - ports: - - protocol: TCP - port: 6379 - egress: - - to: - - ipBlock: - cidr: 10.0.0.0/24 - ports: - - protocol: TCP - port: 5978 -``` +{{< codenew file="service/networking/networkpolicy.yaml" >}} {{< note >}} POSTing this to the API server for your cluster will have no effect unless your chosen networking solution supports network policy. @@ -87,7 +54,7 @@ POSTing this to the API server for your cluster will have no effect unless your __Mandatory Fields__: As with all other Kubernetes config, a NetworkPolicy needs `apiVersion`, `kind`, and `metadata` fields. For general information about working with config files, see -[Configure Containers Using a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/), +[Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/), and [Object Management](/docs/concepts/overview/working-with-objects/object-management). __spec__: NetworkPolicy [spec](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) has all the information needed to define a particular network policy in the given namespace. @@ -176,18 +143,20 @@ in that namespace. ### Default deny all ingress traffic -You can create a "default" isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any ingress traffic to those pods. +You can create a "default" ingress isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any ingress traffic to those pods. {{< codenew file="service/networking/network-policy-default-deny-ingress.yaml" >}} -This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated. This policy does not change the default egress isolation behavior. +This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated for ingress. This policy does not affect isolation for egress from any pod. -### Default allow all ingress traffic +### Allow all ingress traffic -If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as "isolated"), you can create a policy that explicitly allows all traffic in that namespace. +If you want to allow all incoming connections to all pods in a namespace, you can create a policy that explicitly allows that. {{< codenew file="service/networking/network-policy-allow-all-ingress.yaml" >}} +With this policy in place, no additional policy or policies can cause any incoming connection to those pods to be denied. This policy has no effect on isolation for egress from any pod. + ### Default deny all egress traffic You can create a "default" egress isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any egress traffic from those pods. @@ -195,14 +164,16 @@ You can create a "default" egress isolation policy for a namespace by creating a {{< codenew file="service/networking/network-policy-default-deny-egress.yaml" >}} This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed egress traffic. This policy does not -change the default ingress isolation behavior. +change the ingress isolation behavior of any pod. -### Default allow all egress traffic +### Allow all egress traffic -If you want to allow all traffic from all pods in a namespace (even if policies are added that cause some pods to be treated as "isolated"), you can create a policy that explicitly allows all egress traffic in that namespace. +If you want to allow all connections from all pods in a namespace, you can create a policy that explicitly allows all outgoing connections from pods in that namespace. {{< codenew file="service/networking/network-policy-allow-all-egress.yaml" >}} +With this policy in place, no additional policy or policies can cause any outgoing connection from those pods to be denied. This policy has no effect on isolation for ingress to any pod. + ### Default deny all ingress and all egress traffic You can create a "default" policy for a namespace which prevents all ingress AND egress traffic by creating the following NetworkPolicy in that namespace. @@ -261,7 +232,7 @@ The following restrictions apply when using this field: at a cluster level, you (or your cluster administrator) need to disable the `NetworkPolicyEndPort` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the API server with `--feature-gates=NetworkPolicyEndPort=false,…`. -* The `endPort` field must be equal than or greater to the `port` field. +* The `endPort` field must be equal to or greater than the `port` field. * `endPort` can only be defined if `port` is also defined. * Both ports must be numeric. @@ -275,7 +246,7 @@ the policy will be applied only for the single `port` field. ## Targeting a Namespace by its name -{{< feature-state state="beta" for_k8s_version="1.21" >}} +{{< feature-state for_k8s_version="1.22" state="stable" >}} The Kubernetes control plane sets an immutable label `kubernetes.io/metadata.name` on all namespaces, provided that the `NamespaceDefaultLabelName` @@ -287,7 +258,7 @@ standardized label to target a specific namespace. ## What you can't do with network policies (at least, not yet) -As of Kubernetes {{< skew latestVersion >}}, the following functionality does not exist in the NetworkPolicy API, but you might be able to implement workarounds using Operating System components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress controllers, Service Mesh implementations) or admission controllers. In case you are new to network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be implemented using the NetworkPolicy API. +As of Kubernetes {{< skew currentVersion >}}, the following functionality does not exist in the NetworkPolicy API, but you might be able to implement workarounds using Operating System components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress controllers, Service Mesh implementations) or admission controllers. In case you are new to network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be implemented using the NetworkPolicy API. - Forcing internal cluster traffic to go through a common gateway (this might be best served with a service mesh or other proxy). - Anything TLS related (use a service mesh or ingress controller for this). diff --git a/content/en/docs/concepts/services-networking/service-traffic-policy.md b/content/en/docs/concepts/services-networking/service-traffic-policy.md index fb55a3d833..8755b5298b 100644 --- a/content/en/docs/concepts/services-networking/service-traffic-policy.md +++ b/content/en/docs/concepts/services-networking/service-traffic-policy.md @@ -9,7 +9,7 @@ weight: 45 <!-- overview --> -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} _Service Internal Traffic Policy_ enables internal traffic restrictions to only route internal traffic to endpoints within the node the traffic originated from. The @@ -20,9 +20,9 @@ cluster. This can help to reduce costs and improve performance. ## Using Service Internal Traffic Policy -Once you have enabled the `ServiceInternalTrafficPolicy` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/), -you can enable an internal-only traffic policy for a +The `ServiceInternalTrafficPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +is a Beta feature and enabled by default. +When the feature is enabled, you can enable the internal-only traffic policy for a {{< glossary_tooltip text="Services" term_id="service" >}}, by setting its `.spec.internalTrafficPolicy` to `Local`. This tells kube-proxy to only use node local endpoints for cluster internal traffic. @@ -43,7 +43,7 @@ metadata: name: my-service spec: selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: - protocol: TCP port: 80 @@ -60,14 +60,8 @@ considered. When the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `ServiceInternalTrafficPolicy` is enabled, `spec.internalTrafficPolicy` defaults to "Cluster". -## Constraints - -* Service Internal Traffic Policy is not used when `externalTrafficPolicy` is set - to `Local` on a Service. It is possible to use both features in the same cluster - on different Services, just not on the same Service. - ## {{% heading "whatsnext" %}} -* Read about [enabling Topology Aware Hints](/docs/tasks/administer-cluster/enabling-topology-aware-hints) +* Read about [Topology Aware Hints](/docs/concepts/services-networking/topology-aware-hints) * Read about [Service External Traffic Policy](/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) * Read [Connecting Applications with Services](/docs/concepts/services-networking/connect-applications-service/) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index ee1c7514da..eda3b6b9b3 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -25,7 +25,7 @@ and can load-balance across them. ## Motivation Kubernetes {{< glossary_tooltip term_id="pod" text="Pods" >}} are created and destroyed -to match the state of your cluster. Pods are nonpermanent resources. +to match the desired state of your cluster. Pods are nonpermanent resources. If you use a {{< glossary_tooltip term_id="deployment" >}} to run your app, it can create and destroy Pods dynamically. @@ -75,7 +75,7 @@ The name of a Service object must be a valid [RFC 1035 label name](/docs/concepts/overview/working-with-objects/names#rfc-1035-label-names). For example, suppose you have a set of Pods where each listens on TCP port 9376 -and contains a label `app=MyApp`: +and contains a label `app.kubernetes.io/name=MyApp`: ```yaml apiVersion: v1 @@ -84,7 +84,7 @@ metadata: name: my-service spec: selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: - protocol: TCP port: 80 @@ -92,7 +92,7 @@ spec: ``` This specification creates a new Service object named "my-service", which -targets TCP port 9376 on any Pod with the `app=MyApp` label. +targets TCP port 9376 on any Pod with the `app.kubernetes.io/name=MyApp` label. Kubernetes assigns this Service an IP address (sometimes called the "cluster IP"), which is used by the Service proxies @@ -109,12 +109,45 @@ field. {{< /note >}} Port definitions in Pods have names, and you can reference these names in the -`targetPort` attribute of a Service. This works even if there is a mixture -of Pods in the Service using a single configured name, with the same network -protocol available via different port numbers. -This offers a lot of flexibility for deploying and evolving your Services. -For example, you can change the port numbers that Pods expose in the next -version of your backend software, without breaking clients. +`targetPort` attribute of a Service. For example, we can bind the `targetPort` +of the Service to the Pod port in the following way: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx + labels: + app.kubernetes.io/name: proxy +spec: + containers: + - name: nginx + image: nginx:stable + ports: + - containerPort: 80 + name: http-web-svc + +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-service +spec: + selector: + app.kubernetes.io/name: proxy + ports: + - name: name-of-service-port + protocol: TCP + port: 80 + targetPort: http-web-svc +``` + + +This works even if there is a mixture of Pods in the Service using a single +configured name, with the same network protocol available via different +port numbers. This offers a lot of flexibility for deploying and evolving +your Services. For example, you can change the port numbers that Pods expose +in the next version of your backend software, without breaking clients. The default protocol for Services is TCP; you can also use any other [supported protocol](#protocol-support). @@ -125,9 +158,9 @@ Each port definition can have the same `protocol`, or a different one. ### Services without selectors -Services most commonly abstract access to Kubernetes Pods, but they can also -abstract other kinds of backends. -For example: +Services most commonly abstract access to Kubernetes Pods thanks to the selector, +but when used with a corresponding Endpoints object and without a selector, the Service can abstract other kinds of backends, +including ones that run outside the cluster. For example: * You want to have an external database cluster in production, but in your test environment you use your own databases. @@ -159,6 +192,7 @@ where it's running, by adding an Endpoints object manually: apiVersion: v1 kind: Endpoints metadata: + # the name here should match the name of the Service name: my-service subsets: - addresses: @@ -170,6 +204,10 @@ subsets: The name of the Endpoints object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). +When you create an [Endpoints](/docs/reference/kubernetes-api/service-resources/endpoints-v1/) +object for a Service, you set the name of the new object to be the same as that +of the Service. + {{< note >}} The endpoint IPs _must not_ be: loopback (127.0.0.0/8 for IPv4, ::1/128 for IPv6), or link-local (169.254.0.0/16 and 224.0.0.0/24 for IPv4, fe80::/64 for IPv6). @@ -183,6 +221,13 @@ Accessing a Service without a selector works the same as if it had a selector. In the example above, traffic is routed to the single endpoint defined in the YAML: `192.0.2.42:9376` (TCP). +{{< note >}} +The Kubernetes API server does not allow proxying to endpoints that are not mapped to +pods. Actions such as `kubectl proxy <service-name>` where the service has no +selector will fail due to this constraint. This prevents the Kubernetes API server +from being used as a proxy to endpoints the caller may not be authorized to access. +{{< /note >}} + An ExternalName Service is a special case of Service that does not have selectors and uses DNS names instead. For more information, see the [ExternalName](#externalname) section later in this document. @@ -244,7 +289,7 @@ There are a few reasons for using proxying for Services: Later in this page you can read about various kube-proxy implementations work. Overall, you should note that, when running `kube-proxy`, kernel level rules may be -modified (for example, iptables rules might get created), which won't get cleaned up, +modified (for example, iptables rules might get created), which won't get cleaned up, in some cases until you reboot. Thus, running kube-proxy is something that should only be done by an administrator which understands the consequences of having a low level, privileged network proxying service on a computer. Although the `kube-proxy` @@ -254,9 +299,14 @@ thus is only available to use as-is. ### Configuration Note that the kube-proxy starts up in different modes, which are determined by its configuration. -- The kube-proxy's configuration is done via a ConfigMap, and the ConfigMap for kube-proxy effectively deprecates the behaviour for almost all of the flags for the kube-proxy. +- The kube-proxy's configuration is done via a ConfigMap, and the ConfigMap for kube-proxy + effectively deprecates the behaviour for almost all of the flags for the kube-proxy. - The ConfigMap for the kube-proxy does not support live reloading of configuration. -- The ConfigMap parameters for the kube-proxy cannot all be validated and verified on startup. For example, if your operating system doesn't allow you to run iptables commands, the standard kernel kube-proxy implementation will not work. Likewise, if you have an operating system which doesn't support `netsh`, it will not run in Windows userspace mode. +- The ConfigMap parameters for the kube-proxy cannot all be validated and verified on startup. + For example, if your operating system doesn't allow you to run iptables commands, + the standard kernel kube-proxy implementation will not work. + Likewise, if you have an operating system which doesn't support `netsh`, + it will not run in Windows userspace mode. ### User space proxy mode {#proxy-mode-userspace} @@ -354,6 +404,10 @@ You can also set the maximum session sticky time by setting `service.spec.sessionAffinityConfig.clientIP.timeoutSeconds` appropriately. (the default value is 10800, which works out to be 3 hours). +{{< note >}} +On Windows, setting the maximum session sticky time for Services is not supported. +{{< /note >}} + ## Multi-Port Services For some Services, you need to expose more than one port. @@ -369,7 +423,7 @@ metadata: name: my-service spec: selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: - name: http protocol: TCP @@ -407,14 +461,14 @@ server will return a 422 HTTP status code to indicate that there's a problem. You can set the `spec.externalTrafficPolicy` field to control how traffic from external sources is routed. Valid values are `Cluster` and `Local`. Set the field to `Cluster` to route external traffic to all ready endpoints -and `Local` to only route to ready node-local endpoints. If the traffic policy is `Local` and there are are no node-local +and `Local` to only route to ready node-local endpoints. If the traffic policy is `Local` and there are no node-local endpoints, the kube-proxy does not forward any traffic for the relevant Service. {{< note >}} {{< feature-state for_k8s_version="v1.22" state="alpha" >}} If you enable the `ProxyTerminatingEndpoints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`ProxyTerminatingEndpoints` for the kube-proxy, the kube-proxy checks if the node +for the kube-proxy, the kube-proxy checks if the node has local endpoints and whether or not all the local endpoints are marked as terminating. If there are local endpoints and **all** of those are terminating, then the kube-proxy ignores any external traffic policy of `Local`. Instead, whilst the node-local endpoints remain as all @@ -443,11 +497,11 @@ variables and DNS. ### Environment variables When a Pod is run on a Node, the kubelet adds a set of environment variables -for each active Service. It supports both [Docker links -compatible](https://docs.docker.com/userguide/dockerlinks/) variables (see -[makeLinkVariables](https://releases.k8s.io/{{< param "fullversion" >}}/pkg/kubelet/envvars/envvars.go#L49)) -and simpler `{SVCNAME}_SERVICE_HOST` and `{SVCNAME}_SERVICE_PORT` variables, +for each active Service. It adds `{SVCNAME}_SERVICE_HOST` and `{SVCNAME}_SERVICE_PORT` variables, where the Service name is upper-cased and dashes are converted to underscores. +It also supports variables (see [makeLinkVariables](https://github.com/kubernetes/kubernetes/blob/dd2d12f6dc0e654c15d5db57a5f9f6ba61192726/pkg/kubelet/envvars/envvars.go#L72)) +that are compatible with Docker Engine's +"_[legacy container links](https://docs.docker.com/network/links/)_" feature. For example, the Service `redis-master` which exposes TCP port 6379 and has been allocated cluster IP address 10.0.0.11, produces the following environment @@ -544,7 +598,7 @@ The default is `ClusterIP`. * `ClusterIP`: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default `ServiceType`. -* [`NodePort`](#nodeport): Exposes the Service on each Node's IP at a static port +* [`NodePort`](#type-nodeport): Exposes the Service on each Node's IP at a static port (the `NodePort`). A `ClusterIP` Service, to which the `NodePort` Service routes, is automatically created. You'll be able to contact the `NodePort` Service, from outside the cluster, @@ -559,10 +613,12 @@ The default is `ClusterIP`. to use the `ExternalName` type. {{< /note >}} -You can also use [Ingress](/docs/concepts/services-networking/ingress/) to expose your Service. Ingress is not a Service type, but it acts as the entry point for your cluster. It lets you consolidate your routing rules -into a single resource as it can expose multiple services under the same IP address. +You can also use [Ingress](/docs/concepts/services-networking/ingress/) to expose your Service. +Ingress is not a Service type, but it acts as the entry point for your cluster. +It lets you consolidate your routing rules into a single resource as it can expose multiple +services under the same IP address. -### Type NodePort {#nodeport} +### Type NodePort {#type-nodeport} If you set the `type` field to `NodePort`, the Kubernetes control plane allocates a port from a range specified by `--service-node-port-range` flag (default: 30000-32767). @@ -575,9 +631,14 @@ field of the [kube-proxy configuration file](/docs/reference/config-api/kube-proxy-config.v1alpha1/) to particular IP block(s). -This flag takes a comma-delimited list of IP blocks (e.g. `10.0.0.0/8`, `192.0.2.0/25`) to specify IP address ranges that kube-proxy should consider as local to this node. +This flag takes a comma-delimited list of IP blocks (e.g. `10.0.0.0/8`, `192.0.2.0/25`) +to specify IP address ranges that kube-proxy should consider as local to this node. -For example, if you start kube-proxy with the `--nodeport-addresses=127.0.0.0/8` flag, kube-proxy only selects the loopback interface for NodePort Services. The default for `--nodeport-addresses` is an empty list. This means that kube-proxy should consider all available network interfaces for NodePort. (That's also compatible with earlier Kubernetes releases). +For example, if you start kube-proxy with the `--nodeport-addresses=127.0.0.0/8` flag, +kube-proxy only selects the loopback interface for NodePort Services. +The default for `--nodeport-addresses` is an empty list. +his means that kube-proxy should consider all available network interfaces for NodePort. +(That's also compatible with earlier Kubernetes releases). If you want a specific port number, you can specify a value in the `nodePort` field. The control plane will either allocate you that port or report that @@ -605,7 +666,7 @@ metadata: spec: type: NodePort selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: # By default and for convenience, the `targetPort` is set to the same value as the `port` field. - port: 80 @@ -631,7 +692,7 @@ metadata: name: my-service spec: selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: - protocol: TCP port: 80 @@ -644,7 +705,8 @@ status: - ip: 192.0.2.127 ``` -Traffic from the external load balancer is directed at the backend Pods. The cloud provider decides how it is load balanced. +Traffic from the external load balancer is directed at the backend Pods. +The cloud provider decides how it is load balanced. Some cloud providers allow you to specify the `loadBalancerIP`. In those cases, the load-balancer is created with the user-specified `loadBalancerIP`. If the `loadBalancerIP` field is not specified, @@ -659,52 +721,57 @@ to create a static type public IP address resource. This public IP address resou be in the same resource group of the other automatically created resources of the cluster. For example, `MC_myResourceGroup_myAKSCluster_eastus`. -Specify the assigned IP address as loadBalancerIP. Ensure that you have updated the securityGroupName in the cloud provider configuration file. For information about troubleshooting `CreatingLoadBalancerFailed` permission issues see, [Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](https://docs.microsoft.com/en-us/azure/aks/static-ip) or [CreatingLoadBalancerFailed on AKS cluster with advanced networking](https://github.com/Azure/AKS/issues/357). +Specify the assigned IP address as loadBalancerIP. Ensure that you have updated the +`securityGroupName` in the cloud provider configuration file. +For information about troubleshooting `CreatingLoadBalancerFailed` permission issues see, +[Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](https://docs.microsoft.com/en-us/azure/aks/static-ip) +or [CreatingLoadBalancerFailed on AKS cluster with advanced networking](https://github.com/Azure/AKS/issues/357). {{< /note >}} #### Load balancers with mixed protocol types -{{< feature-state for_k8s_version="v1.20" state="alpha" >}} +{{< feature-state for_k8s_version="v1.24" state="beta" >}} By default, for LoadBalancer type of Services, when there is more than one port defined, all ports must have the same protocol, and the protocol must be one which is supported by the cloud provider. -If the feature gate `MixedProtocolLBService` is enabled for the kube-apiserver it is allowed to use different protocols when there is more than one port defined. +The feature gate `MixedProtocolLBService` (enabled by default for the kube-apiserver as of v1.24) allows the use of +different protocols for LoadBalancer type of Services, when there is more than one port defined. {{< note >}} -The set of protocols that can be used for LoadBalancer type of Services is still defined by the cloud provider. +The set of protocols that can be used for LoadBalancer type of Services is still defined by the cloud provider. If a +cloud provider does not support mixed protocols they will provide only a single protocol. {{< /note >}} #### Disabling load balancer NodePort allocation {#load-balancer-nodeport-allocation} -{{< feature-state for_k8s_version="v1.20" state="alpha" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -Starting in v1.20, you can optionally disable node port allocation for a Service Type=LoadBalancer by setting +You can optionally disable node port allocation for a Service of `type=LoadBalancer`, by setting the field `spec.allocateLoadBalancerNodePorts` to `false`. This should only be used for load balancer implementations that route traffic directly to pods as opposed to using node ports. By default, `spec.allocateLoadBalancerNodePorts` is `true` and type LoadBalancer Services will continue to allocate node ports. If `spec.allocateLoadBalancerNodePorts` -is set to `false` on an existing Service with allocated node ports, those node ports will NOT be de-allocated automatically. +is set to `false` on an existing Service with allocated node ports, those node ports will **not** be de-allocated automatically. You must explicitly remove the `nodePorts` entry in every Service port to de-allocate those node ports. -You must enable the `ServiceLBNodePortControl` feature gate to use this field. #### Specifying class of load balancer implementation {#load-balancer-class} -{{< feature-state for_k8s_version="v1.22" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -`spec.loadBalancerClass` enables you to use a load balancer implementation other than the cloud provider default. This feature is available from v1.21, you must enable the `ServiceLoadBalancerClass` feature gate to use this field in v1.21, and the feature gate is enabled by default from v1.22 onwards. +`spec.loadBalancerClass` enables you to use a load balancer implementation other than the cloud provider default. By default, `spec.loadBalancerClass` is `nil` and a `LoadBalancer` type of Service uses the cloud provider's default load balancer implementation if the cluster is configured with -a cloud provider using the `--cloud-provider` component flag. +a cloud provider using the `--cloud-provider` component flag. If `spec.loadBalancerClass` is specified, it is assumed that a load balancer implementation that matches the specified class is watching for Services. Any default load balancer implementation (for example, the one provided by the cloud provider) will ignore Services that have this field set. `spec.loadBalancerClass` can be set on a Service of type `LoadBalancer` only. -Once set, it cannot be changed. +Once set, it cannot be changed. The value of `spec.loadBalancerClass` must be a label-style identifier, with an optional prefix such as "`internal-vip`" or "`example.com/internal-vip`". Unprefixed names are reserved for end-users. @@ -714,7 +781,8 @@ Unprefixed names are reserved for end-users. In a mixed environment it is sometimes necessary to route traffic from Services inside the same (virtual) network address block. -In a split-horizon DNS environment you would need two Services to be able to route both external and internal traffic to your endpoints. +In a split-horizon DNS environment you would need two Services to be able to route both external +and internal traffic to your endpoints. To set an internal load balancer, add one of the following annotations to your Service depending on the cloud Service provider you're using. @@ -816,6 +884,17 @@ metadata: [...] ``` +{{% /tab %}} +{{% tab name="OCI" %}} + +```yaml +[...] +metadata: + name: my-service + annotations: + service.beta.kubernetes.io/oci-load-balancer-internal: true +[...] +``` {{% /tab %}} {{< /tabs >}} @@ -868,7 +947,9 @@ you can use the following annotations: In the above example, if the Service contained three ports, `80`, `443`, and `8443`, then `443` and `8443` would use the SSL certificate, but `80` would be proxied HTTP. -From Kubernetes v1.9 onwards you can use [predefined AWS SSL policies](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) with HTTPS or SSL listeners for your Services. +From Kubernetes v1.9 onwards you can use +[predefined AWS SSL policies](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) +with HTTPS or SSL listeners for your Services. To see which policies are available for use, you can use the `aws` command line tool: ```bash @@ -924,14 +1005,17 @@ specifies the logical hierarchy you created for your Amazon S3 bucket. metadata: name: my-service annotations: - service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true" # Specifies whether access logs are enabled for the load balancer - service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "60" + service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true" + # The interval for publishing the access logs. You can specify an interval of either 5 or 60 (minutes). - service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "my-bucket" + service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "60" + # The name of the Amazon S3 bucket where the access logs are stored - service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "my-bucket-prefix/prod" + service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "my-bucket" + # The logical hierarchy you created for your Amazon S3 bucket, for example `my-bucket-prefix/prod` + service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "my-bucket-prefix/prod" ``` #### Connection Draining on AWS @@ -940,7 +1024,8 @@ Connection draining for Classic ELBs can be managed with the annotation `service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled` set to the value of `"true"`. The annotation `service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout` can -also be used to set maximum time, in seconds, to keep the existing connections open before deregistering the instances. +also be used to set maximum time, in seconds, to keep the existing connections open before +deregistering the instances. ```yaml metadata: @@ -958,50 +1043,56 @@ There are other annotations to manage Classic Elastic Load Balancers that are de metadata: name: my-service annotations: + # The time, in seconds, that the connection is allowed to be idle (no data has been sent + # over the connection) before it is closed by the load balancer service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60" - # The time, in seconds, that the connection is allowed to be idle (no data has been sent over the connection) before it is closed by the load balancer - service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" # Specifies whether cross-zone load balancing is enabled for the load balancer + service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" - service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "environment=prod,owner=devops" # A comma-separated list of key-value pairs which will be recorded as # additional tags in the ELB. + service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "environment=prod,owner=devops" - service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "" # The number of successive successful health checks required for a backend to # be considered healthy for traffic. Defaults to 2, must be between 2 and 10 + service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "" - service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3" # The number of unsuccessful health checks required for a backend to be # considered unhealthy for traffic. Defaults to 6, must be between 2 and 10 + service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3" - service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "20" # The approximate interval, in seconds, between health checks of an # individual instance. Defaults to 10, must be between 5 and 300 + service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "20" - service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "5" # The amount of time, in seconds, during which no response means a failed # health check. This value must be less than the service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval # value. Defaults to 5, must be between 2 and 60 + service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "5" - service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-53fae93f" # A list of existing security groups to be configured on the ELB created. Unlike the annotation - # service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other security groups previously assigned to the ELB and also overrides the creation + # service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other + # security groups previously assigned to the ELB and also overrides the creation # of a uniquely generated security group for this ELB. - # The first security group ID on this list is used as a source to permit incoming traffic to target worker nodes (service traffic and health checks). - # If multiple ELBs are configured with the same security group ID, only a single permit line will be added to the worker node security groups, that means if you delete any + # The first security group ID on this list is used as a source to permit incoming traffic to + # target worker nodes (service traffic and health checks). + # If multiple ELBs are configured with the same security group ID, only a single permit line + # will be added to the worker node security groups, that means if you delete any # of those ELBs it will remove the single permit line and block access for all ELBs that shared the same security group ID. # This can cause a cross-service outage if not used properly + service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-53fae93f" + # A list of additional security groups to be added to the created ELB, this leaves the uniquely + # generated security group in place, this ensures that every ELB + # has a unique security group ID and a matching permit line to allow traffic to the target worker nodes + # (service traffic and health checks). + # Security groups defined here can be shared between services. service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-53fae93f,sg-42efd82e" - # A list of additional security groups to be added to the created ELB, this leaves the uniquely generated security group in place, this ensures that every ELB - # has a unique security group ID and a matching permit line to allow traffic to the target worker nodes (service traffic and health checks). - # Security groups defined here can be shared between services. - service.beta.kubernetes.io/aws-load-balancer-target-node-labels: "ingress-gw,gw-name=public-api" # A comma separated list of key-value pairs which are used # to select the target nodes for the load balancer + service.beta.kubernetes.io/aws-load-balancer-target-node-labels: "ingress-gw,gw-name=public-api" ``` #### Network Load Balancer support on AWS {#aws-nlb-support} @@ -1018,7 +1109,8 @@ To use a Network Load Balancer on AWS, use the annotation `service.beta.kubernet ``` {{< note >}} -NLB only works with certain instance classes; see the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html#register-deregister-targets) +NLB only works with certain instance classes; see the +[AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html#register-deregister-targets) on Elastic Load Balancing for a list of supported instance types. {{< /note >}} @@ -1066,6 +1158,9 @@ in those modified security groups. {{< /note >}} +Further documentation on annotations for Elastic IPs and other common use-cases may be found +in the [AWS Load Balancer Controller documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/). + #### Other CLB annotations on Tencent Kubernetes Engine (TKE) There are other annotations for managing Cloud Load Balancers on TKE as shown below. @@ -1122,7 +1217,8 @@ spec: ``` {{< note >}} -ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName +ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. +ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using [headless Services](#headless-services). {{< /note >}} @@ -1136,9 +1232,13 @@ can start its Pods, add appropriate selectors or endpoints, and change the Service's `type`. {{< warning >}} -You may have trouble using ExternalName for some common protocols, including HTTP and HTTPS. If you use ExternalName then the hostname used by clients inside your cluster is different from the name that the ExternalName references. +You may have trouble using ExternalName for some common protocols, including HTTP and HTTPS. +If you use ExternalName then the hostname used by clients inside your cluster is different from +the name that the ExternalName references. -For protocols that use hostnames this difference may lead to errors or unexpected responses. HTTP requests will have a `Host:` header that the origin server does not recognize; TLS servers will not be able to provide a certificate matching the hostname that the client connected to. +For protocols that use hostnames this difference may lead to errors or unexpected responses. +HTTP requests will have a `Host:` header that the origin server does not recognize; +TLS servers will not be able to provide a certificate matching the hostname that the client connected to. {{< /warning >}} {{< note >}} @@ -1163,7 +1263,7 @@ metadata: name: my-service spec: selector: - app: MyApp + app.kubernetes.io/name: MyApp ports: - name: http protocol: TCP @@ -1208,7 +1308,8 @@ someone else's choice. That is an isolation failure. In order to allow you to choose a port number for your Services, we must ensure that no two Services can collide. Kubernetes does that by allocating each -Service its own IP address. +Service its own IP address from within the `service-cluster-ip-range` +CIDR range that is configured for the API server. To ensure each Service receives a unique IP, an internal allocator atomically updates a global allocation map in {{< glossary_tooltip term_id="etcd" >}} @@ -1222,6 +1323,25 @@ in-memory locking). Kubernetes also uses controllers to check for invalid assignments (eg due to administrator intervention) and for cleaning up allocated IP addresses that are no longer used by any Services. +#### IP address ranges for `type: ClusterIP` Services {#service-ip-static-sub-range} + +{{< feature-state for_k8s_version="v1.24" state="alpha" >}} +However, there is a problem with this `ClusterIP` allocation strategy, because a user +can also [choose their own address for the service](#choosing-your-own-ip-address). +This could result in a conflict if the internal allocator selects the same IP address +for another Service. + +If you enable the `ServiceIPStaticSubrange` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/), +the allocation strategy divides the `ClusterIP` range into two bands, based on +the size of the configured `service-cluster-ip-range` by using the following formula +`min(max(16, cidrSize / 16), 256)`, described as _never less than 16 or more than 256, +with a graduated step function between them_. Dynamic IP allocations will be preferentially +chosen from the upper band, reducing risks of conflicts with the IPs +assigned from the lower band. +This allows users to use the lower band of the `service-cluster-ip-range` for their +Services with static IPs assigned with a very low risk of running into conflicts. + ### Service IP addresses {#ips-and-vips} Unlike Pod IP addresses, which actually route to a fixed destination, @@ -1277,12 +1397,15 @@ through a load-balancer, though in those cases the client IP does get altered. #### IPVS iptables operations slow down dramatically in large scale cluster e.g 10,000 Services. -IPVS is designed for load balancing and based on in-kernel hash tables. So you can achieve performance consistency in large number of Services from IPVS-based kube-proxy. Meanwhile, IPVS-based kube-proxy has more sophisticated load balancing algorithms (least conns, locality, weighted, persistence). +IPVS is designed for load balancing and based on in-kernel hash tables. +So you can achieve performance consistency in large number of Services from IPVS-based kube-proxy. +Meanwhile, IPVS-based kube-proxy has more sophisticated load balancing algorithms +(least conns, locality, weighted, persistence). ## API Object Service is a top-level resource in the Kubernetes REST API. You can find more details -about the API object at: [Service API object](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core). +about the [Service API object](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core). ## Supported protocols {#protocol-support} @@ -1308,7 +1431,8 @@ provider offering this facility. (Most do not). ##### Support for multihomed SCTP associations {#caveat-sctp-multihomed} {{< warning >}} -The support of multihomed SCTP associations requires that the CNI plugin can support the assignment of multiple interfaces and IP addresses to a Pod. +The support of multihomed SCTP associations requires that the CNI plugin can support the +assignment of multiple interfaces and IP addresses to a Pod. NAT for multihomed SCTP associations requires special logic in the corresponding kernel modules. {{< /warning >}} diff --git a/content/en/docs/concepts/services-networking/topology-aware-hints.md b/content/en/docs/concepts/services-networking/topology-aware-hints.md index f471caff6b..cd02b4015c 100644 --- a/content/en/docs/concepts/services-networking/topology-aware-hints.md +++ b/content/en/docs/concepts/services-networking/topology-aware-hints.md @@ -9,7 +9,7 @@ weight: 45 <!-- overview --> -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} _Topology Aware Hints_ enable topology aware routing by including suggestions for how clients should consume endpoints. This approach adds metadata to enable @@ -19,6 +19,12 @@ those network endpoints can be routed closer to where it originated. For example, you can route traffic within a locality to reduce costs, or to improve network performance. +{{< note >}} +The "topology-aware hints" feature is at Beta stage and it is **NOT** enabled +by default. To try out this feature, you have to enable the `TopologyAwareHints` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +{{< /note >}} + <!-- body --> ## Motivation @@ -30,13 +36,12 @@ Routing". When calculating the endpoints for a {{< glossary_tooltip term_id="Ser the EndpointSlice controller considers the topology (region and zone) of each endpoint and populates the hints field to allocate it to a zone. Cluster components such as the {{< glossary_tooltip term_id="kube-proxy" text="kube-proxy" >}} -can then consume those hints, and use them to influence how traffic to is routed +can then consume those hints, and use them to influence how the traffic is routed (favoring topologically closer endpoints). ## Using Topology Aware Hints -If you have [enabled](/docs/tasks/administer-cluster/enabling-topology-aware-hints) the -overall feature, you can activate Topology Aware Hints for a Service by setting the +You can activate Topology Aware Hints for a Service by setting the `service.kubernetes.io/topology-aware-hints` annotation to `auto`. This tells the EndpointSlice controller to set topology hints if it is deemed safe. Importantly, this does not guarantee that hints will always be set. @@ -156,5 +161,4 @@ zone. ## {{% heading "whatsnext" %}} -* Read about [enabling Topology Aware Hints](/docs/tasks/administer-cluster/enabling-topology-aware-hints/) * Read [Connecting Applications with Services](/docs/concepts/services-networking/connect-applications-service/) diff --git a/content/en/docs/concepts/services-networking/windows-networking.md b/content/en/docs/concepts/services-networking/windows-networking.md new file mode 100644 index 0000000000..6aa79f0a03 --- /dev/null +++ b/content/en/docs/concepts/services-networking/windows-networking.md @@ -0,0 +1,164 @@ +--- +reviewers: +- aravindhp +- jayunit100 +- jsturtevant +- marosset +title: Networking on Windows +content_type: concept +weight: 75 +--- + +<!-- overview --> + +Kubernetes supports running nodes on either Linux or Windows. You can mix both kinds of node +within a single cluster. +This page provides an overview to networking specific to the Windows operating system. + +<!-- body --> +## Container networking on Windows {#networking} + +Networking for Windows containers is exposed through +[CNI plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). +Windows containers function similarly to virtual machines in regards to +networking. Each container has a virtual network adapter (vNIC) which is connected +to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the +Host Compute Service (HCS) work together to create containers and attach container +vNICs to networks. HCS is responsible for the management of containers whereas HNS +is responsible for the management of networking resources such as: + +* Virtual networks (including creation of vSwitches) +* Endpoints / vNICs +* Namespaces +* Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules. + +The Windows HNS and vSwitch implement namespacing and can +create virtual NICs as needed for a pod or container. However, many configurations such +as DNS, routes, and metrics are stored in the Windows registry database rather than as +files inside `/etc`, which is how Linux stores those configurations. The Windows registry for the container +is separate from that of the host, so concepts like mapping `/etc/resolv.conf` from +the host into a container don't have the same effect they would on Linux. These must +be configured using Windows APIs run in the context of that container. Therefore +CNI implementations need to call the HNS instead of relying on file mappings to pass +network details into the pod or container. + +## Network modes + +Windows supports five different networking drivers/modes: L2bridge, L2tunnel, +Overlay (Beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux +worker nodes, you need to select a networking solution that is compatible on both +Windows and Linux. The following table lists the out-of-tree plugins are supported on Windows, +with recommendations on when to use each CNI: + +| Network Driver | Description | Container Packet Modifications | Network Plugins | Network Plugin Characteristics | +| -------------- | ----------- | ------------------------------ | --------------- | ------------------------------ | +| L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. | MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity. | +| L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/azure/aks/concepts-network#azure-cni-advanced-networking) | +| Overlay | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header. | [win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires [KB4489899](https://support.microsoft.com/help/4489899) on Windows Server 2019. | +| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host. <br/> Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller. <br/> NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | +| NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://techcommunity.microsoft.com/t5/virtualization/windows-nat-winnat-capabilities-and-limitations/ba-p/382303) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness | + +As outlined above, the [Flannel](https://github.com/coreos/flannel) +[CNI plugin](https://github.com/flannel-io/cni-plugin) +is also [supported](https://github.com/flannel-io/cni-plugin#windows-support-experimental) on Windows via the +[VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**Beta support** ; delegates to win-overlay) +and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge). + +This plugin supports delegating to one of the reference CNI plugins (win-overlay, +win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for +automatic node subnet lease assignment and HNS network creation. This plugin reads +in its own configuration file (cni.conf), and aggregates it with the environment +variables from the FlannelD generated subnet.env file. It then delegates to one of +the reference CNI plugins for network plumbing, and sends the correct configuration +containing the node-assigned subnet to the IPAM plugin (for example: `host-local`). + +For Node, Pod, and Service objects, the following network flows are supported for +TCP/UDP traffic: + +* Pod → Pod (IP) +* Pod → Pod (Name) +* Pod → Service (Cluster IP) +* Pod → Service (PQDN, but only if there are no ".") +* Pod → Service (FQDN) +* Pod → external (IP) +* Pod → external (DNS) +* Node → Pod +* Pod → Node + +## IP address management (IPAM) {#ipam} + +The following IPAM options are supported on Windows: + +* [host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local) +* [azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only) +* [Windows Server IPAM](https://docs.microsoft.com/windows-server/networking/technologies/ipam/ipam-top) (fallback option if no IPAM is set) + +## Load balancing and Services + +A Kubernetes {{< glossary_tooltip text="Service" term_id="service" >}} is an abstraction +that defines a logical set of Pods and a means to access them over a network. +In a cluster that includes Windows nodes, you can use the following types of Service: + +* `NodePort` +* `ClusterIP` +* `LoadBalancer` +* `ExternalName` + +Windows container networking differs in some important ways from Linux networking. +The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) +provides additional details and background. + +On Windows, you can use the following settings to configure Services and load +balancing behavior: + +{{< table caption="Windows Service Settings" >}} +| Feature | Description | Minimum Supported Windows OS build | How to enable | +| ------- | ----------- | -------------------------- | ------------- | +| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | Windows Server 2022 | Set `service.spec.sessionAffinity` to "ClientIP" | +| Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` | +| Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | Windows Server, version 1903 | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. | +| IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | Windows Server 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) | +| Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. | Windows Server 2019 | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy | +{{< /table >}} + +{{< warning >}} +There are known issue with NodePort Services on overlay networking, if the destination node is running Windows Server 2022. +To avoid the issue entirely, you can configure the service with `externalTrafficPolicy: Local`. + +There are known issues with Pod to Pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed. +To workaround the issue and restore Pod to Pod connectivity, you can disable the WinDSR feature in kube-proxy. + +These issues require OS fixes. +Please follow https://github.com/microsoft/Windows-Containers/issues/204 for updates. +{{< /warning >}} + +## Limitations + +The following networking functionality is _not_ supported on Windows nodes: + +* Host networking mode +* Local NodePort access from the node itself (works for other nodes or external clients) +* More than 64 backend pods (or unique destination addresses) for a single Service +* IPv6 communication between Windows pods connected to overlay networks +* Local Traffic Policy in non-DSR mode +* Outbound communication using the ICMP protocol via the `win-overlay`, `win-bridge`, or using the Azure-CNI plugin.\ + Specifically, the Windows data plane ([VFP](https://www.microsoft.com/research/project/azure-virtual-filtering-platform/)) + doesn't support ICMP packet transpositions, and this means: + * ICMP packets directed to destinations within the same network (such as pod to pod communication via ping) + work as expected; + * TCP/UDP packets work as expected; + * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) + cannot be transposed and thus will not be routed back to their source; + * Since TCP/UDP packets can still be transposed, you can substitute `ping <destination>` with + `curl <destination>` when debugging connectivity with the outside world. + +Other limitations: + +* Windows reference network plugins win-bridge and win-overlay do not implement + [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) v0.4.0, + due to a missing `CHECK` implementation. +* The Flannel VXLAN CNI plugin has the following limitations on Windows: + * Node-pod connectivity is only possible for local pods with Flannel v0.12.0 (or higher). + * Flannel is restricted to using VNI 4096 and UDP port 4789. See the official + [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) + backend docs for more details on these parameters. diff --git a/content/en/docs/concepts/storage/dynamic-provisioning.md b/content/en/docs/concepts/storage/dynamic-provisioning.md index 63263fb370..6da1850ada 100644 --- a/content/en/docs/concepts/storage/dynamic-provisioning.md +++ b/content/en/docs/concepts/storage/dynamic-provisioning.md @@ -116,7 +116,7 @@ can enable this behavior by: is enabled on the API server. An administrator can mark a specific `StorageClass` as default by adding the -`storageclass.kubernetes.io/is-default-class` annotation to it. +`storageclass.kubernetes.io/is-default-class` [annotation](/docs/reference/labels-annotations-taints/#storageclass-kubernetes-io-is-default-class) to it. When a default `StorageClass` exists in a cluster and a user creates a `PersistentVolumeClaim` with `storageClassName` unspecified, the `DefaultStorageClass` admission controller automatically adds the diff --git a/content/en/docs/concepts/storage/ephemeral-volumes.md b/content/en/docs/concepts/storage/ephemeral-volumes.md index e76f76f492..045bcafe76 100644 --- a/content/en/docs/concepts/storage/ephemeral-volumes.md +++ b/content/en/docs/concepts/storage/ephemeral-volumes.md @@ -7,7 +7,7 @@ reviewers: - pohly title: Ephemeral Volumes content_type: concept -weight: 50 +weight: 30 --- <!-- overview --> @@ -76,8 +76,8 @@ is managed by kubelet, or injecting different data. {{< feature-state for_k8s_version="v1.16" state="beta" >}} -This feature requires the `CSIInlineVolume` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled. It -is enabled by default starting with Kubernetes 1.16. +This feature requires the `CSIInlineVolume` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +to be enabled. It is enabled by default starting with Kubernetes 1.16. {{< note >}} CSI ephemeral volumes are only supported by a subset of CSI drivers. @@ -107,7 +107,7 @@ metadata: spec: containers: - name: my-frontend - image: busybox + image: busybox:1.28 volumeMounts: - mountPath: "/data" name: my-csi-inline-vol @@ -125,15 +125,26 @@ driver. These attributes are specific to each driver and not standardized. See the documentation of each CSI driver for further instructions. -As a cluster administrator, you can use a [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) to control which CSI drivers can be used in a Pod, specified with the -[`allowedCSIDrivers` field](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicyspec-v1beta1-policy). +### CSI driver restrictions + +CSI ephemeral volumes allow users to provide `volumeAttributes` +directly to the CSI driver as part of the Pod spec. A CSI driver +allowing `volumeAttributes` that are typically restricted to +administrators is NOT suitable for use in an inline ephemeral volume. +For example, parameters that are normally defined in the StorageClass +should not be exposed to users through the use of inline ephemeral volumes. + +Cluster administrators who need to restrict the CSI drivers that are +allowed to be used as inline volumes within a Pod spec may do so by: + +- Removing `Ephemeral` from `volumeLifecycleModes` in the CSIDriver spec, which prevents the + driver from being used as an inline ephemeral volume. +- Using an [admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/) + to restrict how this driver is used. ### Generic ephemeral volumes -{{< feature-state for_k8s_version="v1.21" state="beta" >}} - -This feature requires the `GenericEphemeralVolume` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be -enabled. Because this is a beta feature, it is enabled by default. +{{< feature-state for_k8s_version="v1.23" state="stable" >}} Generic ephemeral volumes are similar to `emptyDir` volumes in the sense that they provide a per-pod directory for scratch data that is @@ -146,7 +157,7 @@ features: parameters. - Typical operations on volumes are supported assuming that the driver supports them, including - ([snapshotting](/docs/concepts/storage/volume-snapshots/), + [snapshotting](/docs/concepts/storage/volume-snapshots/), [cloning](/docs/concepts/storage/volume-pvc-datasource/), [resizing](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims), and [storage capacity tracking](/docs/concepts/storage/storage-capacity/). @@ -161,7 +172,7 @@ metadata: spec: containers: - name: my-frontend - image: busybox + image: busybox:1.28 volumeMounts: - mountPath: "/scratch" name: scratch-volume @@ -199,7 +210,7 @@ because then the scheduler is free to choose a suitable node for the Pod. With immediate binding, the scheduler is forced to select a node that has access to the volume once it is available. -In terms of [resource ownership](/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents), +In terms of [resource ownership](/docs/concepts/architecture/garbage-collection/#owners-dependents), a Pod that has generic ephemeral storage is the owner of the PersistentVolumeClaim(s) that provide that ephemeral storage. When the Pod is deleted, the Kubernetes garbage collector deletes the PVC, which then usually @@ -243,19 +254,12 @@ same namespace, so that these conflicts can't occur. Enabling the GenericEphemeralVolume feature allows users to create PVCs indirectly if they can create Pods, even if they do not have permission to create PVCs directly. Cluster administrators must be -aware of this. If this does not fit their security model, they have -two choices: -- Explicitly disable the feature through the feature gate. -- Use a [Pod Security - Policy](/docs/concepts/policy/pod-security-policy/) where the - `volumes` list does not contain the `ephemeral` volume type - (deprecated in Kubernetes 1.21). -- Use an [admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/) - which rejects objects like Pods that have a generic ephemeral - volume. +aware of this. If this does not fit their security model, they should +use an [admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/) +that rejects objects like Pods that have a generic ephemeral volume. -The normal [namespace quota for PVCs](/docs/concepts/policy/resource-quotas/#storage-resource-quota) still applies, so -even if users are allowed to use this new mechanism, they cannot use +The normal [namespace quota for PVCs](/docs/concepts/policy/resource-quotas/#storage-resource-quota) +still applies, so even if users are allowed to use this new mechanism, they cannot use it to circumvent other policies. ## {{% heading "whatsnext" %}} @@ -266,12 +270,13 @@ See [local ephemeral storage](/docs/concepts/configuration/manage-resources-cont ### CSI ephemeral volumes -- For more information on the design, see the [Ephemeral Inline CSI - volumes KEP](https://github.com/kubernetes/enhancements/blob/ad6021b3d61a49040a3f835e12c8bb5424db2bbb/keps/sig-storage/20190122-csi-inline-volumes.md). -- For more information on further development of this feature, see the [enhancement tracking issue #596](https://github.com/kubernetes/enhancements/issues/596). +- For more information on the design, see the + [Ephemeral Inline CSI volumes KEP](https://github.com/kubernetes/enhancements/blob/ad6021b3d61a49040a3f835e12c8bb5424db2bbb/keps/sig-storage/20190122-csi-inline-volumes.md). +- For more information on further development of this feature, see the + [enhancement tracking issue #596](https://github.com/kubernetes/enhancements/issues/596). ### Generic ephemeral volumes - For more information on the design, see the -[Generic ephemeral inline volumes KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1698-generic-ephemeral-volumes/README.md). -- For more information on further development of this feature, see the [enhancement tracking issue #1698](https://github.com/kubernetes/enhancements/issues/1698). + [Generic ephemeral inline volumes KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1698-generic-ephemeral-volumes/README.md). + diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index a029ceaeda..07521f42ee 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -10,14 +10,13 @@ feature: title: Storage orchestration description: > Automatically mount the storage system of your choice, whether from local storage, a public cloud provider such as <a href="https://cloud.google.com/storage/">GCP</a> or <a href="https://aws.amazon.com/products/storage/">AWS</a>, or a network storage system such as NFS, iSCSI, Gluster, Ceph, Cinder, or Flocker. - content_type: concept weight: 20 --- <!-- overview --> -This document describes the current state of _persistent volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested. +This document describes _persistent volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested. <!-- body --> @@ -131,7 +130,9 @@ The `Retain` reclaim policy allows for manual reclamation of the resource. When 1. Delete the PersistentVolume. The associated storage asset in external infrastructure (such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume) still exists after the PV is deleted. 1. Manually clean up the data on the associated storage asset accordingly. -1. Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the storage asset definition. +1. Manually delete the associated storage asset. + +If you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition. #### Delete @@ -174,6 +175,74 @@ spec: However, the particular path specified in the custom recycler Pod template in the `volumes` part is replaced with the particular path of the volume that is being recycled. +### PersistentVolume deletion protection finalizer +{{< feature-state for_k8s_version="v1.23" state="alpha" >}} + +Finalizers can be added on a PersistentVolume to ensure that PersistentVolumes +having `Delete` reclaim policy are deleted only after the backing storage are deleted. + +The newly introduced finalizers `kubernetes.io/pv-controller` and `external-provisioner.volume.kubernetes.io/finalizer` +are only added to dynamically provisioned volumes. + +The finalizer `kubernetes.io/pv-controller` is added to in-tree plugin volumes. The following is an example + +```shell +kubectl describe pv pvc-74a498d6-3929-47e8-8c02-078c1ece4d78 +Name: pvc-74a498d6-3929-47e8-8c02-078c1ece4d78 +Labels: <none> +Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner + pv.kubernetes.io/bound-by-controller: yes + pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume +Finalizers: [kubernetes.io/pv-protection kubernetes.io/pv-controller] +StorageClass: vcp-sc +Status: Bound +Claim: default/vcp-pvc-1 +Reclaim Policy: Delete +Access Modes: RWO +VolumeMode: Filesystem +Capacity: 1Gi +Node Affinity: <none> +Message: +Source: + Type: vSphereVolume (a Persistent Disk resource in vSphere) + VolumePath: [vsanDatastore] d49c4a62-166f-ce12-c464-020077ba5d46/kubernetes-dynamic-pvc-74a498d6-3929-47e8-8c02-078c1ece4d78.vmdk + FSType: ext4 + StoragePolicyName: vSAN Default Storage Policy +Events: <none> +``` + +The finalizer `external-provisioner.volume.kubernetes.io/finalizer` is added for CSI volumes. +The following is an example: +```shell +Name: pvc-2f0bab97-85a8-4552-8044-eb8be45cf48d +Labels: <none> +Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com +Finalizers: [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer] +StorageClass: fast +Status: Bound +Claim: demo-app/nginx-logs +Reclaim Policy: Delete +Access Modes: RWO +VolumeMode: Filesystem +Capacity: 200Mi +Node Affinity: <none> +Message: +Source: + Type: CSI (a Container Storage Interface (CSI) volume source) + Driver: csi.vsphere.vmware.com + FSType: ext4 + VolumeHandle: 44830fa8-79b4-406b-8b58-621ba25353fd + ReadOnly: false + VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1648442357185-8081-csi.vsphere.vmware.com + type=vSphere CNS Block Volume +Events: <none> +``` + +Enabling the `CSIMigration` feature for a specific in-tree volume plugin will remove +the `kubernetes.io/pv-controller` finalizer, while adding the `external-provisioner.volume.kubernetes.io/finalizer` +finalizer. Similarly, disabling `CSIMigration` will remove the `external-provisioner.volume.kubernetes.io/finalizer` +finalizer, while adding the `kubernetes.io/pv-controller` finalizer. + ### Reserving a PersistentVolume The control plane can [bind PersistentVolumeClaims to matching PersistentVolumes](#binding) in the @@ -219,19 +288,19 @@ to `Retain`, including cases where you are reusing an existing PV. {{< feature-state for_k8s_version="v1.11" state="beta" >}} -Support for expanding PersistentVolumeClaims (PVCs) is now enabled by default. You can expand +Support for expanding PersistentVolumeClaims (PVCs) is enabled by default. You can expand the following types of volumes: -* gcePersistentDisk +* azureDisk +* azureFile * awsElasticBlockStore -* Cinder +* cinder (deprecated) +* {{< glossary_tooltip text="csi" term_id="csi" >}} +* flexVolume (deprecated) +* gcePersistentDisk * glusterfs * rbd -* Azure File -* Azure Disk -* Portworx -* FlexVolumes -* {{< glossary_tooltip text="CSI" term_id="csi" >}} +* portworxVolume You can only expand a PVC if its storage class's `allowVolumeExpansion` field is set to true. @@ -253,6 +322,16 @@ To request a larger volume for a PVC, edit the PVC object and specify a larger size. This triggers expansion of the volume that backs the underlying PersistentVolume. A new PersistentVolume is never created to satisfy the claim. Instead, an existing volume is resized. +{{< warning >}} +Directly editing the size of a PersistentVolume can prevent an automatic resize of that volume. +If you edit the capacity of a PersistentVolume, and then edit the `.spec` of a matching +PersistentVolumeClaim to make the size of the PersistentVolumeClaim match the PersistentVolume, +then no storage resize happens. +The Kubernetes control plane will see that the desired state of both resources matches, +conclude that the backing volume size has been manually +increased and that no resize is necessary. +{{< /warning >}} + #### CSI Volume expansion {{< feature-state for_k8s_version="v1.16" state="beta" >}} @@ -268,23 +347,18 @@ When a volume contains a file system, the file system is only resized when a new the PersistentVolumeClaim in `ReadWrite` mode. File system expansion is either done when a Pod is starting up or when a Pod is running and the underlying file system supports online expansion. -FlexVolumes allow resize if the driver is set with the `RequiresFSResize` capability to `true`. -The FlexVolume can be resized on Pod restart. +FlexVolumes (deprecated since Kubernetes v1.23) allow resize if the driver is configured with the +`RequiresFSResize` capability to `true`. The FlexVolume can be resized on Pod restart. #### Resizing an in-use PersistentVolumeClaim -{{< feature-state for_k8s_version="v1.15" state="beta" >}} - -{{< note >}} -Expanding in-use PVCs is available as beta since Kubernetes 1.15, and as alpha since 1.11. The `ExpandInUsePersistentVolumes` feature must be enabled, which is the case automatically for many clusters for beta features. Refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. -{{< /note >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} In this case, you don't need to delete and recreate a Pod or deployment that is using an existing PVC. Any in-use PVC automatically becomes available to its Pod as soon as its file system has been expanded. This feature has no effect on PVCs that are not in use by a Pod or deployment. You must create a Pod that uses the PVC before the expansion can complete. - Similar to other volume types - FlexVolume volumes can also be expanded when in-use by a Pod. {{< note >}} @@ -297,6 +371,11 @@ Expanding EBS volumes is a time-consuming operation. Also, there is a per-volume #### Recovering from Failure when Expanding Volumes +If a user specifies a new size that is too big to be satisfied by underlying storage system, expansion of PVC will be continuously retried until user or cluster administrator takes some action. This can be undesirable and hence Kubernetes provides following methods of recovering from such failures. + +{{< tabs name="recovery_methods" >}} +{{% tab name="Manually with Cluster Administrator access" %}} + If expanding underlying storage fails, the cluster administrator can manually recover the Persistent Volume Claim (PVC) state and cancel the resize requests. Otherwise, the resize requests are continuously retried by the controller without administrator intervention. 1. Mark the PersistentVolume(PV) that is bound to the PersistentVolumeClaim(PVC) with `Retain` reclaim policy. @@ -305,6 +384,30 @@ If expanding underlying storage fails, the cluster administrator can manually re 4. Re-create the PVC with smaller size than PV and set `volumeName` field of the PVC to the name of the PV. This should bind new PVC to existing PV. 5. Don't forget to restore the reclaim policy of the PV. +{{% /tab %}} +{{% tab name="By requesting expansion to smaller size" %}} +{{% feature-state for_k8s_version="v1.23" state="alpha" %}} + +{{< note >}} +Recovery from failing PVC expansion by users is available as an alpha feature since Kubernetes 1.23. The `RecoverVolumeExpansionFailure` feature must be enabled for this feature to work. Refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. +{{< /note >}} + +If the feature gates `RecoverVolumeExpansionFailure` is +enabled in your cluster, and expansion has failed for a PVC, you can retry expansion with a +smaller size than the previously requested value. To request a new expansion attempt with a +smaller proposed size, edit `.spec.resources` for that PVC and choose a value that is less than the +value you previously tried. +This is useful if expansion to a higher value did not succeed because of capacity constraint. +If that has happened, or you suspect that it might have, you can retry expansion by specifying a +size that is within the capacity limits of underlying storage provider. You can monitor status of resize operation by watching `.status.resizeStatus` and events on the PVC. + +Note that, +although you can specify a lower amount of storage than what was requested previously, +the new value must still be higher than `.status.capacity`. +Kubernetes does not support shrinking a PVC to less than its current size. +{{% /tab %}} +{{% /tabs %}} + ## Types of Persistent Volumes @@ -316,7 +419,6 @@ PersistentVolume types are implemented as plugins. Kubernetes currently supports * [`cephfs`](/docs/concepts/storage/volumes/#cephfs) - CephFS volume * [`csi`](/docs/concepts/storage/volumes/#csi) - Container Storage Interface (CSI) * [`fc`](/docs/concepts/storage/volumes/#fc) - Fibre Channel (FC) storage -* [`flexVolume`](/docs/concepts/storage/volumes/#flexVolume) - FlexVolume * [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk) - GCE Persistent Disk * [`glusterfs`](/docs/concepts/storage/volumes/#glusterfs) - Glusterfs volume * [`hostPath`](/docs/concepts/storage/volumes/#hostpath) - HostPath volume @@ -334,6 +436,8 @@ The following types of PersistentVolume are deprecated. This means that support * [`cinder`](/docs/concepts/storage/volumes/#cinder) - Cinder (OpenStack block storage) (**deprecated** in v1.18) +* [`flexVolume`](/docs/concepts/storage/volumes/#flexvolume) - FlexVolume + (**deprecated** in v1.23) * [`flocker`](/docs/concepts/storage/volumes/#flocker) - Flocker storage (**deprecated** in v1.22) * [`quobyte`](/docs/concepts/storage/volumes/#quobyte) - Quobyte volume @@ -381,7 +485,7 @@ Helper programs relating to the volume type may be required for consumption of a ### Capacity -Generally, a PV will have a specific storage capacity. This is set using the PV's `capacity` attribute. See the Kubernetes [Resource Model](https://git.k8s.io/community/contributors/design-proposals/scheduling/resources.md) to understand the units expected by `capacity`. +Generally, a PV will have a specific storage capacity. This is set using the PV's `capacity` attribute. Read the glossary term [Quantity](/docs/reference/glossary/?all=true#term-quantity) to understand the units expected by `capacity`. Currently, storage size is the only resource that can be set or requested. Future attributes may include IOPS, throughput, etc. @@ -412,11 +516,22 @@ A PersistentVolume can be mounted on a host in any way supported by the resource The access modes are: -* ReadWriteOnce -- the volume can be mounted as read-write by a single node -* ReadOnlyMany -- the volume can be mounted read-only by many nodes -* ReadWriteMany -- the volume can be mounted as read-write by many nodes -* ReadWriteOncePod -- the volume can be mounted as read-write by a single Pod. - This is only supported for CSI volumes and Kubernetes version 1.22+. +`ReadWriteOnce` +: the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node. + +`ReadOnlyMany` +: the volume can be mounted as read-only by many nodes. + +`ReadWriteMany` +: the volume can be mounted as read-write by many nodes. + + `ReadWriteOncePod` +: the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+. + + + +The blog article [Introducing Single Pod Access Mode for PersistentVolumes](/blog/2021/09/13/read-write-once-pod-access-mode-alpha/) covers this in more detail. + In the CLI, the access modes are abbreviated to: @@ -425,6 +540,15 @@ In the CLI, the access modes are abbreviated to: * RWX - ReadWriteMany * RWOP - ReadWriteOncePod +{{< note >}} +Kubernetes uses volume access modes to match PersistentVolumeClaims and PersistentVolumes. +In some cases, the volume access modes also constrain where the PersistentVolume can be mounted. +Volume access modes do **not** enforce write protection once the storage has been mounted. +Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don't set any constraints on the volume. +For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only. +If the access modes are specified as ReadWriteOncePod, the volume is constrained and can be mounted on only a single Pod. +{{< /note >}} + > __Important!__ A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time. @@ -434,7 +558,7 @@ In the CLI, the access modes are abbreviated to: | AzureFile | ✓ | ✓ | ✓ | - | | AzureDisk | ✓ | - | - | - | | CephFS | ✓ | ✓ | ✓ | - | -| Cinder | ✓ | - | - | - | +| Cinder | ✓ | - | ([if multi-attach volumes are available](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/features.md#multi-attach-volumes)) | - | | CSI | depends on the driver | depends on the driver | depends on the driver | depends on the driver | | FC | ✓ | ✓ | - | - | | FlexVolume | ✓ | ✓ | depends on the driver | - | @@ -483,19 +607,19 @@ Not all Persistent Volume types support mount options. The following volume types support mount options: -* AWSElasticBlockStore -* AzureDisk -* AzureFile -* CephFS -* Cinder (OpenStack block storage) -* GCEPersistentDisk -* Glusterfs -* NFS -* Quobyte Volumes -* RBD (Ceph Block Device) -* StorageOS -* VsphereVolume -* iSCSI +* `awsElasticBlockStore` +* `azureDisk` +* `azureFile` +* `cephfs` +* `cinder` (**deprecated** in v1.18) +* `gcePersistentDisk` +* `glusterfs` +* `iscsi` +* `nfs` +* `quobyte` (**deprecated** in v1.22) +* `rbd` +* `storageos` (**deprecated** in v1.22) +* `vsphereVolume` Mount options are not validated. If a mount option is invalid, the mount fails. @@ -558,7 +682,7 @@ Claims use [the same convention as volumes](#volume-mode) to indicate the consum ### Resources -Claims, like Pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://git.k8s.io/community/contributors/design-proposals/scheduling/resources.md) applies to both volumes and claims. +Claims, like Pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://git.k8s.io/design-proposals-archive/scheduling/resources.md) applies to both volumes and claims. ### Selector @@ -797,17 +921,12 @@ spec: ## Volume populators and data sources -{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +{{< feature-state for_k8s_version="v1.24" state="beta" >}} -{{< note >}} -Kubernetes supports custom volume populators; this alpha feature was introduced -in Kubernetes 1.18. Kubernetes 1.22 reimplemented the mechanism with a redesigned API. -Check that you are reading the version of the Kubernetes documentation that matches your -cluster. {{% version-check %}} +Kubernetes supports custom volume populators. To use custom volume populators, you must enable the `AnyVolumeDataSource` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the kube-apiserver and kube-controller-manager. -{{< /note >}} Volume populators take advantage of a PVC spec field called `dataSourceRef`. Unlike the `dataSource` field, which can only contain either a reference to another PersistentVolumeClaim @@ -825,6 +944,7 @@ contents. There are two differences between the `dataSourceRef` field and the `dataSource` field that users should be aware of: + * The `dataSource` field ignores invalid values (as if the field was blank) while the `dataSourceRef` field never ignores values and will cause an error if an invalid value is used. Invalid values are any core object (objects with no apiGroup) except for PVCs. @@ -901,7 +1021,7 @@ and need persistent storage, it is recommended that you use the following patter * Learn more about [Creating a PersistentVolume](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolume). * Learn more about [Creating a PersistentVolumeClaim](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolumeclaim). -* Read the [Persistent Storage design document](https://git.k8s.io/community/contributors/design-proposals/storage/persistent-storage.md). +* Read the [Persistent Storage design document](https://git.k8s.io/design-proposals-archive/storage/persistent-storage.md). ### API references {#reference} diff --git a/content/en/docs/concepts/storage/projected-volumes.md b/content/en/docs/concepts/storage/projected-volumes.md new file mode 100644 index 0000000000..a404a26e5e --- /dev/null +++ b/content/en/docs/concepts/storage/projected-volumes.md @@ -0,0 +1,127 @@ +--- +reviewers: +- marosset +- jsturtevant +- zshihang +title: Projected Volumes +content_type: concept +weight: 21 # just after persistent volumes +--- + +<!-- overview --> + +This document describes _projected volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested. + +<!-- body --> + +## Introduction + +A `projected` volume maps several existing volume sources into the same directory. + +Currently, the following types of volume sources can be projected: + +* [`secret`](/docs/concepts/storage/volumes/#secret) +* [`downwardAPI`](/docs/concepts/storage/volumes/#downwardapi) +* [`configMap`](/docs/concepts/storage/volumes/#configmap) +* [`serviceAccountToken`](#serviceaccounttoken) + +All sources are required to be in the same namespace as the Pod. For more details, +see the [all-in-one volume](https://git.k8s.io/design-proposals-archive/node/all-in-one-volume.md) design document. + +### Example configuration with a secret, a downwardAPI, and a configMap {#example-configuration-secret-downwardapi-configmap} + +{{< codenew file="pods/storage/projected-secret-downwardapi-configmap.yaml" >}} + +### Example configuration: secrets with a non-default permission mode set {#example-configuration-secrets-nondefault-permission-mode} + +{{< codenew file="pods/storage/projected-secrets-nondefault-permission-mode.yaml" >}} + +Each projected volume source is listed in the spec under `sources`. The +parameters are nearly the same with two exceptions: + +* For secrets, the `secretName` field has been changed to `name` to be consistent + with ConfigMap naming. +* The `defaultMode` can only be specified at the projected level and not for each + volume source. However, as illustrated above, you can explicitly set the `mode` + for each individual projection. + +## serviceAccountToken projected volumes {#serviceaccounttoken} +When the `TokenRequestProjection` feature is enabled, you can inject the token +for the current [service account](/docs/reference/access-authn-authz/authentication/#service-account-tokens) +into a Pod at a specified path. For example: + +{{< codenew file="pods/storage/projected-service-account-token.yaml" >}} + +The example Pod has a projected volume containing the injected service account +token. Containers in this Pod can use that token to access the Kubernetes API +server, authenticating with the identity of [the pod's ServiceAccount](/docs/tasks/configure-pod-container/configure-service-account/). +The `audience` field contains the intended audience of the +token. A recipient of the token must identify itself with an identifier specified +in the audience of the token, and otherwise should reject the token. This field +is optional and it defaults to the identifier of the API server. + +The `expirationSeconds` is the expected duration of validity of the service account +token. It defaults to 1 hour and must be at least 10 minutes (600 seconds). An administrator +can also limit its maximum value by specifying the `--service-account-max-token-expiration` +option for the API server. The `path` field specifies a relative path to the mount point +of the projected volume. + +{{< note >}} +A container using a projected volume source as a [`subPath`](/docs/concepts/storage/volumes/#using-subpath) +volume mount will not receive updates for those volume sources. +{{< /note >}} + +## SecurityContext interactions + +The [proposal](https://git.k8s.io/enhancements/keps/sig-storage/2451-service-account-token-volumes#proposal) for file permission handling in projected service account volume enhancement introduced the projected files having the correct owner permissions set. + +### Linux + +In Linux pods that have a projected volume and `RunAsUser` set in the Pod +[`SecurityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context), +the projected files have the correct ownership set including container user +ownership. + +### Windows + +In Windows pods that have a projected volume and `RunAsUsername` set in the +Pod `SecurityContext`, the ownership is not enforced due to the way user +accounts are managed in Windows. Windows stores and manages local user and group +accounts in a database file called Security Account Manager (SAM). Each +container maintains its own instance of the SAM database, to which the host has +no visibility into while the container is running. Windows containers are +designed to run the user mode portion of the OS in isolation from the host, +hence the maintenance of a virtual SAM database. As a result, the kubelet running +on the host does not have the ability to dynamically configure host file +ownership for virtualized container accounts. It is recommended that if files on +the host machine are to be shared with the container then they should be placed +into their own volume mount outside of `C:\`. + +By default, the projected files will have the following ownership as shown for +an example projected volume file: + +```powershell +PS C:\> Get-Acl C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt | Format-List + +Path : Microsoft.PowerShell.Core\FileSystem::C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt +Owner : BUILTIN\Administrators +Group : NT AUTHORITY\SYSTEM +Access : NT AUTHORITY\SYSTEM Allow FullControl + BUILTIN\Administrators Allow FullControl + BUILTIN\Users Allow ReadAndExecute, Synchronize +Audit : +Sddl : O:BAG:SYD:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;0x1200a9;;;BU) +``` + +This implies all administrator users like `ContainerAdministrator` will have +read, write and execute access while, non-administrator users will have read and +execute access. + +{{< note >}} +In general, granting the container access to the host is discouraged as it can +open the door for potential security exploits. + +Creating a Windows Pod with `RunAsUser` in it's `SecurityContext` will result in +the Pod being stuck at `ContainerCreating` forever. So it is advised to not use +the Linux only `RunAsUser` option with Windows Pods. +{{< /note >}} diff --git a/content/en/docs/concepts/storage/storage-capacity.md b/content/en/docs/concepts/storage/storage-capacity.md index 13ae8ab722..38c1a56c7e 100644 --- a/content/en/docs/concepts/storage/storage-capacity.md +++ b/content/en/docs/concepts/storage/storage-capacity.md @@ -7,7 +7,7 @@ reviewers: - pohly title: Storage Capacity content_type: concept -weight: 45 +weight: 70 --- <!-- overview --> @@ -16,38 +16,41 @@ Storage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with. -{{< feature-state for_k8s_version="v1.19" state="alpha" >}} -{{< feature-state for_k8s_version="v1.21" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} This page describes how Kubernetes keeps track of storage capacity and -how the scheduler uses that information to schedule Pods onto nodes +how the scheduler uses that information to [schedule Pods](/docs/concepts/scheduling-eviction/) onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed. -Tracking storage capacity is supported for {{< glossary_tooltip -text="Container Storage Interface" term_id="csi" >}} (CSI) drivers and -[needs to be enabled](#enabling-storage-capacity-tracking) when installing a CSI driver. +## {{% heading "prerequisites" %}} + +Kubernetes v{{< skew currentVersion >}} includes cluster-level API support for +storage capacity tracking. To use this you must also be using a CSI driver that +supports capacity tracking. Consult the documentation for the CSI drivers that +you use to find out whether this support is available and, if so, how to use +it. If you are not running Kubernetes v{{< skew currentVersion >}}, check the +documentation for that version of Kubernetes. <!-- body --> ## API There are two API extensions for this feature: -- CSIStorageCapacity objects: +- [CSIStorageCapacity](/docs/reference/kubernetes-api/config-and-storage-resources/csi-storage-capacity-v1/) objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage. -- [The `CSIDriverSpec.StorageCapacity` field](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#csidriverspec-v1-storage-k8s-io): +- [The `CSIDriverSpec.StorageCapacity` field](/docs/reference/kubernetes-api/config-and-storage-resources/csi-driver-v1/#CSIDriverSpec): when set to `true`, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver. ## Scheduling Storage capacity information is used by the Kubernetes scheduler if: -- the `CSIStorageCapacity` feature gate is true, - a Pod uses a volume that has not been created yet, - that volume uses a {{< glossary_tooltip text="StorageClass" term_id="storage-class" >}} which references a CSI driver and uses `WaitForFirstConsumer` [volume binding @@ -98,20 +101,9 @@ multiple volumes: one volume might have been created already in a topology segment which then does not have enough capacity left for another volume. Manual intervention is necessary to recover from this, for example by increasing capacity or deleting the volume that was -already created. [Further -work](https://github.com/kubernetes/enhancements/pull/1703) is needed -to handle this automatically. - -## Enabling storage capacity tracking - -Storage capacity tracking is a beta feature and enabled by default in -a Kubernetes cluster since Kubernetes 1.21. In addition to having the -feature enabled in the cluster, a CSI driver also has to support -it. Please refer to the driver's documentation for details. +already created. ## {{% heading "whatsnext" %}} - For more information on the design, see the [Storage Capacity Constraints for Pod Scheduling KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1472-storage-capacity-tracking/README.md). -- For more information on further development of this feature, see the [enhancement tracking issue #1472](https://github.com/kubernetes/enhancements/issues/1472). -- Learn about [Kubernetes Scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/) diff --git a/content/en/docs/concepts/storage/storage-classes.md b/content/en/docs/concepts/storage/storage-classes.md index 36615098c6..8fda0b2ff3 100644 --- a/content/en/docs/concepts/storage/storage-classes.md +++ b/content/en/docs/concepts/storage/storage-classes.md @@ -87,7 +87,7 @@ for provisioning PVs. This field must be specified. You are not restricted to specifying the "internal" provisioners listed here (whose names are prefixed with "kubernetes.io" and shipped alongside Kubernetes). You can also run and specify external provisioners, -which are independent programs that follow a [specification](https://git.k8s.io/community/contributors/design-proposals/storage/volume-provisioning.md) +which are independent programs that follow a [specification](https://git.k8s.io/design-proposals-archive/storage/volume-provisioning.md) defined by Kubernetes. Authors of external provisioners have full discretion over where their code lives, how the provisioner is shipped, how it needs to be run, what volume plugin it uses (including Flex), etc. The repository @@ -241,8 +241,8 @@ allowedTopologies: - matchLabelExpressions: - key: failure-domain.beta.kubernetes.io/zone values: - - us-central1-a - - us-central1-b + - us-central-1a + - us-central-1b ``` ## Parameters @@ -434,7 +434,7 @@ provisioner: example.com/external-nfs parameters: server: nfs-server.example.com path: /share - readOnly: false + readOnly: "false" ``` * `server`: Server is the hostname or IP address of the NFS server. @@ -470,14 +470,14 @@ This internal provisioner of OpenStack is deprecated. Please use [the external c There are two types of provisioners for vSphere storage classes: -- [CSI provisioner](#csi-provisioner): `csi.vsphere.vmware.com` +- [CSI provisioner](#vsphere-provisioner-csi): `csi.vsphere.vmware.com` - [vCP provisioner](#vcp-provisioner): `kubernetes.io/vsphere-volume` In-tree provisioners are [deprecated](/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/#why-are-we-migrating-in-tree-plugins-to-csi). For more information on the CSI provisioner, see [Kubernetes vSphere CSI Driver](https://vsphere-csi-driver.sigs.k8s.io/) and [vSphereVolume CSI migration](/docs/concepts/storage/volumes/#csi-migration-5). #### CSI Provisioner {#vsphere-provisioner-csi} -The vSphere CSI StorageClass provisioner works with Tanzu Kubernetes clusters. For an example, refer to the [vSphere CSI repository](https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/master/example/vanilla-k8s-file-driver/example-sc.yaml). +The vSphere CSI StorageClass provisioner works with Tanzu Kubernetes clusters. For an example, refer to the [vSphere CSI repository](https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/example/vanilla-k8s-RWM-filesystem-volumes/example-sc.yaml). #### vCP Provisioner @@ -797,7 +797,7 @@ parameters: storagePool: sp1 storageMode: ThinProvisioned secretRef: sio-secret - readOnly: false + readOnly: "false" fsType: xfs ``` diff --git a/content/en/docs/concepts/storage/volume-health-monitoring.md b/content/en/docs/concepts/storage/volume-health-monitoring.md index c5fb9c1929..0e39011a0f 100644 --- a/content/en/docs/concepts/storage/volume-health-monitoring.md +++ b/content/en/docs/concepts/storage/volume-health-monitoring.md @@ -24,7 +24,7 @@ If a CSI Driver supports Volume Health Monitoring feature from the controller si The External Health Monitor {{< glossary_tooltip text="controller" term_id="controller" >}} also watches for node failure events. You can enable node failure monitoring by setting the `enable-node-watcher` flag to true. When the external health monitor detects a node failure event, the controller reports an Event will be reported on the PVC to indicate that pods using this PVC are on a failed node. -If a CSI Driver supports Volume Health Monitoring feature from the node side, an Event will be reported on every Pod using the PVC when an abnormal volume condition is detected on a CSI volume. +If a CSI Driver supports Volume Health Monitoring feature from the node side, an Event will be reported on every Pod using the PVC when an abnormal volume condition is detected on a CSI volume. In addition, Volume Health information is exposed as Kubelet VolumeStats metrics. A new metric kubelet_volume_stats_health_status_abnormal is added. This metric includes two labels: `namespace` and `persistentvolumeclaim`. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy. For more information, please check [KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1432-volume-health-monitor#kubelet-metrics-changes). {{< note >}} You need to enable the `CSIVolumeHealth` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to use this feature from the node side. diff --git a/content/en/docs/concepts/storage/volume-pvc-datasource.md b/content/en/docs/concepts/storage/volume-pvc-datasource.md index 9e59560d1d..f800d9107a 100644 --- a/content/en/docs/concepts/storage/volume-pvc-datasource.md +++ b/content/en/docs/concepts/storage/volume-pvc-datasource.md @@ -6,7 +6,7 @@ reviewers: - msau42 title: CSI Volume Cloning content_type: concept -weight: 30 +weight: 60 --- <!-- overview --> diff --git a/content/en/docs/concepts/storage/volume-snapshot-classes.md b/content/en/docs/concepts/storage/volume-snapshot-classes.md index ee781d665f..485ca0a4a4 100644 --- a/content/en/docs/concepts/storage/volume-snapshot-classes.md +++ b/content/en/docs/concepts/storage/volume-snapshot-classes.md @@ -8,7 +8,7 @@ reviewers: - yuxiangqian title: Volume Snapshot Classes content_type: concept -weight: 30 +weight: 41 # just after volume snapshots --- <!-- overview --> @@ -39,6 +39,10 @@ request a particular class. Administrators set the name and other parameters of a class when first creating VolumeSnapshotClass objects, and the objects cannot be updated once they are created. +{{< note >}} +Installation of the CRDs is the responsibility of the Kubernetes distribution. Without the required CRDs present, the creation of a VolumeSnapshotClass fails. +{{< /note >}} + ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass diff --git a/content/en/docs/concepts/storage/volume-snapshots.md b/content/en/docs/concepts/storage/volume-snapshots.md index 7673ecf2b4..0b4d2c2f2b 100644 --- a/content/en/docs/concepts/storage/volume-snapshots.md +++ b/content/en/docs/concepts/storage/volume-snapshots.md @@ -8,7 +8,7 @@ reviewers: - yuxiangqian title: Volume Snapshots content_type: concept -weight: 20 +weight: 40 --- <!-- overview --> @@ -120,6 +120,7 @@ spec: driver: hostpath.csi.k8s.io source: volumeHandle: ee0cfb94-f8d4-11e9-b2d8-0242ac110002 + sourceVolumeMode: Filesystem volumeSnapshotClassName: csi-hostpath-snapclass volumeSnapshotRef: name: new-snapshot-test @@ -141,6 +142,7 @@ spec: driver: hostpath.csi.k8s.io source: snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002 + sourceVolumeMode: Filesystem volumeSnapshotRef: name: new-snapshot-test namespace: default @@ -148,6 +150,51 @@ spec: `snapshotHandle` is the unique identifier of the volume snapshot created on the storage backend. This field is required for the pre-provisioned snapshots. It specifies the CSI snapshot id on the storage system that this `VolumeSnapshotContent` represents. +`sourceVolumeMode` is the mode of the volume whose snapshot is taken. The value +of the `sourceVolumeMode` field can be either `Filesystem` or `Block`. If the +source volume mode is not specified, Kubernetes treats the snapshot as if the +source volume's mode is unknown. + +## Converting the volume mode of a Snapshot {#convert-volume-mode} + +If the `VolumeSnapshots` API installed on your cluster supports the `sourceVolumeMode` +field, then the API has the capability to prevent unauthorized users from converting +the mode of a volume. + +To check if your cluster has capability for this feature, run the following command: + +```yaml +$ kubectl get crd volumesnapshotcontent -o yaml +``` + +If you want to allow users to create a `PersistentVolumeClaim` from an existing +`VolumeSnapshot`, but with a different volume mode than the source, the annotation +`snapshot.storage.kubernetes.io/allowVolumeModeChange: "true"`needs to be added to +the `VolumeSnapshotContent` that corresponds to the `VolumeSnapshot`. + +For pre-provisioned snapshots, `Spec.SourceVolumeMode` needs to be populated +by the cluster administrator. + +An example `VolumeSnapshotContent` resource with this feature enabled would look like: + +```yaml +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshotContent +metadata: + name: new-snapshot-content-test + annotations: + - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true" +spec: + deletionPolicy: Delete + driver: hostpath.csi.k8s.io + source: + snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002 + sourceVolumeMode: Filesystem + volumeSnapshotRef: + name: new-snapshot-test + namespace: default +``` + ## Provisioning Volumes from Snapshots You can provision a new volume, pre-populated with data from a snapshot, by using diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 56694dee66..fd994c97fa 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -33,8 +33,8 @@ drivers, but the functionality is somewhat limited. Kubernetes supports many types of volumes. A {{< glossary_tooltip term_id="pod" text="Pod" >}} can use any number of volume types simultaneously. Ephemeral volume types have a lifetime of a pod, but persistent volumes exist beyond -the lifetime of a pod. When a pod ceases to exist, Kubernetes destroys ephemeral volumes; -however, Kubernetes does not destroy persistent volumes. +the lifetime of a pod. When a pod ceases to exist, Kubernetes destroys ephemeral volumes; +however, Kubernetes does not destroy persistent volumes. For any kind of volume in a given pod, data is preserved across container restarts. At its core, a volume is a directory, possibly with some data in it, which @@ -44,18 +44,29 @@ volume type used. To use a volume, specify the volumes to provide for the Pod in `.spec.volumes` and declare where to mount those volumes into containers in `.spec.containers[*].volumeMounts`. -A process in a container sees a filesystem view composed from their Docker -image and volumes. The [Docker image](https://docs.docker.com/userguide/dockerimages/) -is at the root of the filesystem hierarchy. Volumes mount at the specified paths within -the image. Volumes can not mount onto other volumes or have hard links to -other volumes. Each Container in the Pod's configuration must independently specify where to -mount each volume. +A process in a container sees a filesystem view composed from the initial contents of +the {{< glossary_tooltip text="container image" term_id="image" >}}, plus volumes +(if defined) mounted inside the container. +The process sees a root filesystem that initially matches the contents of the container +image. +Any writes to within that filesystem hierarchy, if allowed, affect what that process views +when it performs a subsequent filesystem access. +Volumes mount at the [specified paths](#using-subpath) within +the image. +For each container defined within a Pod, you must independently specify where +to mount each volume that the container uses. + +Volumes cannot mount within other volumes (but see [Using subPath](#using-subpath) +for a related mechanism). Also, a volume cannot contain a hard link to anything in +a different volume. ## Types of Volumes {#volume-types} Kubernetes supports several types of volumes. -### awsElasticBlockStore {#awselasticblockstore} +### awsElasticBlockStore (deprecated) {#awselasticblockstore} + +{{< feature-state for_k8s_version="v1.17" state="deprecated" >}} An `awsElasticBlockStore` volume mounts an Amazon Web Services (AWS) [EBS volume](https://aws.amazon.com/ebs/) into your pod. Unlike @@ -106,7 +117,7 @@ spec: fsType: ext4 ``` -If the EBS volume is partitioned, you can supply the optional field `partition: "<partition number>"` to specify which parition to mount on. +If the EBS volume is partitioned, you can supply the optional field `partition: "<partition number>"` to specify which partition to mount on. #### AWS EBS CSI migration @@ -126,7 +137,9 @@ beta features must be enabled. To disable the `awsElasticBlockStore` storage plugin from being loaded by the controller manager and the kubelet, set the `InTreePluginAWSUnregister` flag to `true`. -### azureDisk {#azuredisk} +### azureDisk (deprecated) {#azuredisk} + +{{< feature-state for_k8s_version="v1.19" state="deprecated" >}} The `azureDisk` volume type mounts a Microsoft Azure [Data Disk](https://docs.microsoft.com/en-us/azure/aks/csi-storage-drivers) into a pod. @@ -134,16 +147,24 @@ For more details, see the [`azureDisk` volume plugin](https://github.com/kuberne #### azureDisk CSI migration -{{< feature-state for_k8s_version="v1.19" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} The `CSIMigration` feature for `azureDisk`, when enabled, redirects all plugin operations from the existing in-tree plugin to the `disk.csi.azure.com` Container -Storage Interface (CSI) Driver. In order to use this feature, the [Azure Disk CSI -Driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver) -must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureDisk` -features must be enabled. +Storage Interface (CSI) Driver. In order to use this feature, the +[Azure Disk CSI Driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver) +must be installed on the cluster and the `CSIMigration` feature must be enabled. -### azureFile {#azurefile} +#### azureDisk CSI migration complete + +{{< feature-state for_k8s_version="v1.21" state="alpha" >}} + +To disable the `azureDisk` storage plugin from being loaded by the controller manager +and the kubelet, set the `InTreePluginAzureDiskUnregister` flag to `true`. + +### azureFile (deprecated) {#azurefile} + +{{< feature-state for_k8s_version="v1.21" state="deprecated" >}} The `azureFile` volume type mounts a Microsoft Azure File volume (SMB 2.1 and 3.0) into a pod. @@ -161,7 +182,15 @@ Driver](https://github.com/kubernetes-sigs/azurefile-csi-driver) must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureFile` [feature gates](/docs/reference/command-line-tools-reference/feature-gates/) must be enabled. -Azure File CSI driver does not support using same volume with different fsgroups, if Azurefile CSI migration is enabled, using same volume with different fsgroups won't be supported at all. +Azure File CSI driver does not support using same volume with different fsgroups. If +`CSIMigrationAzureFile` is enabled, using same volume with different fsgroups won't be supported at all. + +#### azureFile CSI migration complete + +{{< feature-state for_k8s_version="v1.21" state="alpha" >}} + +To disable the `azureFile` storage plugin from being loaded by the controller manager +and the kubelet, set the `InTreePluginAzureFileUnregister` flag to `true`. ### cephfs @@ -178,7 +207,9 @@ You must have your own Ceph server running with the share exported before you ca See the [CephFS example](https://github.com/kubernetes/examples/tree/master/volumes/cephfs/) for more details. -### cinder +### cinder (deprecated) {#cinder} + +{{< feature-state for_k8s_version="v1.18" state="deprecated" >}} {{< note >}} Kubernetes must be configured with the OpenStack cloud provider. @@ -210,17 +241,17 @@ spec: #### OpenStack CSI migration -{{< feature-state for_k8s_version="v1.21" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -The `CSIMigration` feature for Cinder is enabled by default in Kubernetes 1.21. +The `CSIMigration` feature for Cinder is enabled by default since Kubernetes 1.21. It redirects all plugin operations from the existing in-tree plugin to the `cinder.csi.openstack.org` Container Storage Interface (CSI) Driver. [OpenStack Cinder CSI Driver](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) must be installed on the cluster. -You can disable Cinder CSI migration for your cluster by setting the `CSIMigrationOpenStack` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to `false`. -If you disable the `CSIMigrationOpenStack` feature, the in-tree Cinder volume plugin takes responsibility -for all aspects of Cinder volume storage management. + +To disable the in-tree Cinder plugin from being loaded by the controller manager +and the kubelet, you can enable the `InTreePluginOpenStackUnregister` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/). ### configMap @@ -242,7 +273,7 @@ metadata: spec: containers: - name: test - image: busybox + image: busybox:1.28 volumeMounts: - name: config-vol mountPath: /etc/config @@ -272,15 +303,17 @@ keyed with `log_level`. ### downwardAPI {#downwardapi} -A `downwardAPI` volume makes downward API data available to applications. -It mounts a directory and writes the requested data in plain text files. +A `downwardAPI` volume makes {{< glossary_tooltip term_id="downward-api" text="downward API" >}} +data available to applications. Within the volume, you can find the exposed +data as read-only files in plain text format. {{< note >}} -A container using the downward API as a [`subPath`](#using-subpath) volume mount will not -receive downward API updates. +A container using the downward API as a [`subPath`](#using-subpath) volume mount does not +receive updates when field values change. {{< /note >}} -See the [downward API example](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) for more details. +See [Expose Pod Information to Containers Through Files](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) +to learn more. ### emptyDir {#emptydir} @@ -367,7 +400,9 @@ You must have your own Flocker installation running before you can use it. See the [Flocker example](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker) for more details. -### gcePersistentDisk +### gcePersistentDisk (deprecated) {#gcepersistentdisk} + +{{< feature-state for_k8s_version="v1.17" state="deprecated" >}} A `gcePersistentDisk` volume mounts a Google Compute Engine (GCE) [persistent disk](https://cloud.google.com/compute/docs/disks) (PD) into your Pod. @@ -801,143 +836,8 @@ For more details, see the [Portworx volume](https://github.com/kubernetes/exampl ### projected -A `projected` volume maps several existing volume sources into the same directory. - -Currently, the following types of volume sources can be projected: - -* [`secret`](#secret) -* [`downwardAPI`](#downwardapi) -* [`configMap`](#configmap) -* `serviceAccountToken` - -All sources are required to be in the same namespace as the Pod. For more details, -see the [all-in-one volume design document](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/all-in-one-volume.md). - -#### Example configuration with a secret, a downwardAPI, and a configMap {#example-configuration-secret-downwardapi-configmap} - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: volume-test -spec: - containers: - - name: container-test - image: busybox - volumeMounts: - - name: all-in-one - mountPath: "/projected-volume" - readOnly: true - volumes: - - name: all-in-one - projected: - sources: - - secret: - name: mysecret - items: - - key: username - path: my-group/my-username - - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - path: "cpu_limit" - resourceFieldRef: - containerName: container-test - resource: limits.cpu - - configMap: - name: myconfigmap - items: - - key: config - path: my-group/my-config -``` - -#### Example configuration: secrets with a non-default permission mode set {#example-configuration-secrets-nondefault-permission-mode} - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: volume-test -spec: - containers: - - name: container-test - image: busybox - volumeMounts: - - name: all-in-one - mountPath: "/projected-volume" - readOnly: true - volumes: - - name: all-in-one - projected: - sources: - - secret: - name: mysecret - items: - - key: username - path: my-group/my-username - - secret: - name: mysecret2 - items: - - key: password - path: my-group/my-password - mode: 511 -``` - -Each projected volume source is listed in the spec under `sources`. The -parameters are nearly the same with two exceptions: - -* For secrets, the `secretName` field has been changed to `name` to be consistent - with ConfigMap naming. -* The `defaultMode` can only be specified at the projected level and not for each - volume source. However, as illustrated above, you can explicitly set the `mode` - for each individual projection. - -When the `TokenRequestProjection` feature is enabled, you can inject the token -for the current [service account](/docs/reference/access-authn-authz/authentication/#service-account-tokens) -into a Pod at a specified path. For example: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: sa-token-test -spec: - containers: - - name: container-test - image: busybox - volumeMounts: - - name: token-vol - mountPath: "/service-account" - readOnly: true - volumes: - - name: token-vol - projected: - sources: - - serviceAccountToken: - audience: api - expirationSeconds: 3600 - path: token -``` - -The example Pod has a projected volume containing the injected service account -token. This token can be used by a Pod's containers to access the Kubernetes API -server. The `audience` field contains the intended audience of the -token. A recipient of the token must identify itself with an identifier specified -in the audience of the token, and otherwise should reject the token. This field -is optional and it defaults to the identifier of the API server. - -The `expirationSeconds` is the expected duration of validity of the service account -token. It defaults to 1 hour and must be at least 10 minutes (600 seconds). An administrator -can also limit its maximum value by specifying the `--service-account-max-token-expiration` -option for the API server. The `path` field specifies a relative path to the mount point -of the projected volume. - -{{< note >}} -A container using a projected volume source as a [`subPath`](#using-subpath) volume mount will not -receive updates for those volume sources. -{{< /note >}} +A projected volume maps several existing volume sources into the same +directory. For more details, see [projected volumes](/docs/concepts/storage/projected-volumes/). ### quobyte (deprecated) {#quobyte} @@ -956,11 +856,11 @@ GitHub project has [instructions](https://github.com/quobyte/quobyte-csi#quobyte ### rbd An `rbd` volume allows a -[Rados Block Device](https://docs.ceph.com/en/latest/rbd/) (RBD) volume to mount into your -Pod. Unlike `emptyDir`, which is erased when a pod is removed, the contents of -an `rbd` volume are preserved and the volume is unmounted. This -means that a RBD volume can be pre-populated with data, and that data can -be shared between pods. +[Rados Block Device](https://docs.ceph.com/en/latest/rbd/) (RBD) volume to mount +into your Pod. Unlike `emptyDir`, which is erased when a pod is removed, the +contents of an `rbd` volume are preserved and the volume is unmounted. This +means that a RBD volume can be pre-populated with data, and that data can be +shared between pods. {{< note >}} You must have a Ceph installation running before you can use RBD. @@ -975,6 +875,38 @@ Simultaneous writers are not allowed. See the [RBD example](https://github.com/kubernetes/examples/tree/master/volumes/rbd) for more details. +#### RBD CSI migration {#rbd-csi-migration} + +{{< feature-state for_k8s_version="v1.23" state="alpha" >}} + +The `CSIMigration` feature for `RBD`, when enabled, redirects all plugin +operations from the existing in-tree plugin to the `rbd.csi.ceph.com` {{< +glossary_tooltip text="CSI" term_id="csi" >}} driver. In order to use this +feature, the +[Ceph CSI driver](https://github.com/ceph/ceph-csi) +must be installed on the cluster and the `CSIMigration` and `csiMigrationRBD` +[feature gates](/docs/reference/command-line-tools-reference/feature-gates/) +must be enabled. + +{{< note >}} + +As a Kubernetes cluster operator that administers storage, here are the +prerequisites that you must complete before you attempt migration to the +RBD CSI driver: + +* You must install the Ceph CSI driver (`rbd.csi.ceph.com`), v3.5.0 or above, + into your Kubernetes cluster. +* considering the `clusterID` field is a required parameter for CSI driver for + its operations, but in-tree StorageClass has `monitors` field as a required + parameter, a Kubernetes storage admin has to create a clusterID based on the + monitors hash ( ex:`#echo -n + '<monitors_string>' | md5sum`) in the CSI config map and keep the monitors + under this clusterID configuration. +* Also, if the value of `adminId` in the in-tree Storageclass is different from + `admin`, the `adminSecretName` mentioned in the in-tree Storageclass has to be + patched with the base64 value of the `adminId` parameter value, otherwise this + step can be skipped. {{< /note >}} + ### secret A `secret` volume is used to pass sensitive information, such as passwords, to @@ -1049,66 +981,15 @@ spec: For more information about StorageOS, dynamic provisioning, and PersistentVolumeClaims, see the [StorageOS examples](https://github.com/kubernetes/examples/blob/master/volumes/storageos). -### vsphereVolume {#vspherevolume} +### vsphereVolume (deprecated) {#vspherevolume} {{< note >}} -You must configure the Kubernetes vSphere Cloud Provider. For cloudprovider -configuration, refer to the [vSphere Getting Started guide](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/). +We recommend to use vSphere CSI out-of-tree driver instead. {{< /note >}} A `vsphereVolume` is used to mount a vSphere VMDK volume into your Pod. The contents of a volume are preserved when it is unmounted. It supports both VMFS and VSAN datastore. -{{< note >}} -You must create vSphere VMDK volume using one of the following methods before using with a Pod. -{{< /note >}} - -#### Creating a VMDK volume {#creating-vmdk-volume} - -Choose one of the following methods to create a VMDK. - -{{< tabs name="tabs_volumes" >}} -{{% tab name="Create using vmkfstools" %}} -First ssh into ESX, then use the following command to create a VMDK: - -```shell -vmkfstools -c 2G /vmfs/volumes/DatastoreName/volumes/myDisk.vmdk -``` - -{{% /tab %}} -{{% tab name="Create using vmware-vdiskmanager" %}} -Use the following command to create a VMDK: - -```shell -vmware-vdiskmanager -c -t 0 -s 40GB -a lsilogic myDisk.vmdk -``` - -{{% /tab %}} - -{{< /tabs >}} - -#### vSphere VMDK configuration example {#vsphere-vmdk-configuration} - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: test-vmdk -spec: - containers: - - image: k8s.gcr.io/test-webserver - name: test-container - volumeMounts: - - mountPath: /test-vmdk - name: test-volume - volumes: - - name: test-volume - # This VMDK volume must already exist. - vsphereVolume: - volumePath: "[DatastoreName] volumes/myDisk" - fsType: ext4 -``` - For more information, see the [vSphere volume](https://github.com/kubernetes/examples/tree/master/staging/volumes/vsphere) examples. #### vSphere CSI migration {#vsphere-csi-migration} @@ -1120,8 +1001,15 @@ from the existing in-tree plugin to the `csi.vsphere.vmware.com` {{< glossary_to [vSphere CSI driver](https://github.com/kubernetes-sigs/vsphere-csi-driver) must be installed on the cluster and the `CSIMigration` and `CSIMigrationvSphere` [feature gates](/docs/reference/command-line-tools-reference/feature-gates/) must be enabled. +You can find additional advice on how to migrate in VMware's +documentation page [Migrating In-Tree vSphere Volumes to vSphere Container Storage Plug-in](https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/2.0/vmware-vsphere-csp-getting-started/GUID-968D421F-D464-4E22-8127-6CB9FF54423F.html). -This also requires minimum vSphere vCenter/ESXi Version to be 7.0u1 and minimum HW Version to be VM version 15. +Kubernetes v{{< skew currentVersion >}} requires that you are using vSphere 7.0u2 or later +in order to migrate to the out-of-tree CSI driver. +If you are running a version of Kubernetes other than v{{< skew currentVersion >}}, consult +the documentation for that version of Kubernetes. +If you are running Kubernetes v{{< skew currentVersion >}} and an older version of vSphere, +consider upgrading to at least vSphere 7.0u2. {{< note >}} The following StorageClass parameters from the built-in `vsphereVolume` plugin are not supported by the vSphere CSI driver: @@ -1144,6 +1032,16 @@ but new volumes created by the vSphere CSI driver will not be honoring these par To turn off the `vsphereVolume` plugin from being loaded by the controller manager and the kubelet, you need to set `InTreePluginvSphereUnregister` feature flag to `true`. You must install a `csi.vsphere.vmware.com` {{< glossary_tooltip text="CSI" term_id="csi" >}} driver on all worker nodes. +#### Portworx CSI migration +{{< feature-state for_k8s_version="v1.23" state="alpha" >}} + +The `CSIMigration` feature for Portworx has been added but disabled by default in Kubernetes 1.23 since it's in alpha state. +It redirects all plugin operations from the existing in-tree plugin to the +`pxd.portworx.com` Container Storage Interface (CSI) Driver. +[Portworx CSI Driver](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/csi/) +must be installed on the cluster. +To enable the feature, set `CSIMigrationPortworx=true` in kube-controller-manager and kubelet. + ## Using subPath {#using-subpath} Sometimes, it is useful to share one volume for multiple uses in a single pod. @@ -1212,11 +1110,12 @@ spec: fieldRef: apiVersion: v1 fieldPath: metadata.name - image: busybox + image: busybox:1.28 command: [ "sh", "-c", "while [ true ]; do echo 'Hello'; sleep 10; done | tee -a /logs/hello.txt" ] volumeMounts: - name: workdir1 mountPath: /logs + # The variable expansion uses round brackets (not curly brackets). subPathExpr: $(POD_NAME) restartPolicy: Never volumes: @@ -1239,8 +1138,7 @@ To learn about requesting space using a resource specification, see ## Out-of-tree volume plugins The out-of-tree volume plugins include -{{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} (CSI) -and FlexVolume. These plugins enable storage vendors to create custom storage plugins +{{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} (CSI), and also FlexVolume (which is deprecated). These plugins enable storage vendors to create custom storage plugins without adding their plugin source code to the Kubernetes repository. Previously, all volume plugins were "in-tree". The "in-tree" plugins were built, linked, compiled, @@ -1260,7 +1158,7 @@ to the [volume plugin FAQ](https://github.com/kubernetes/community/blob/master/s (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads. -Please read the [CSI design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) for more information. +Please read the [CSI design proposal](https://git.k8s.io/design-proposals-archive/storage/container-storage-interface.md) for more information. {{< note >}} Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes @@ -1280,9 +1178,8 @@ CSI driver. A `csi` volume can be used in a Pod in three different ways: * through a reference to a [PersistentVolumeClaim](#persistentvolumeclaim) -* with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volume) -(alpha feature) -* with a [CSI ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) +* with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) +* with a [CSI ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes) if the driver supports that (beta feature) The following fields are available to storage administrators to configure a CSI @@ -1349,12 +1246,26 @@ You can set up your You can directly configure CSI volumes within the Pod specification. Volumes specified in this way are ephemeral and do not persist across pod restarts. See [Ephemeral -Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) +Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes) for more information. For more information on how to develop a CSI driver, refer to the [kubernetes-csi documentation](https://kubernetes-csi.github.io/docs/) +#### Windows CSI proxy + +{{< feature-state for_k8s_version="v1.22" state="stable" >}} + +CSI node plugins need to perform various privileged +operations like scanning of disk devices and mounting of file systems. These operations +differ for each host operating system. For Linux worker nodes, containerized CSI node +node plugins are typically deployed as privileged containers. For Windows worker nodes, +privileged operations for containerized CSI node plugins is supported using +[csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed, +stand-alone binary that needs to be pre-installed on each Windows node. + +For more details, refer to the deployment guide of the CSI plugin you wish to deploy. + #### Migrating to CSI drivers from in-tree plugins {{< feature-state for_k8s_version="v1.17" state="beta" >}} @@ -1371,15 +1282,37 @@ provisioning/delete, attach/detach, mount/unmount and resizing of volumes. In-tree plugins that support `CSIMigration` and have a corresponding CSI driver implemented are listed in [Types of Volumes](#volume-types). +The following in-tree plugins support persistent storage on Windows nodes: + +* [`awsElasticBlockStore`](#awselasticblockstore) +* [`azureDisk`](#azuredisk) +* [`azureFile`](#azurefile) +* [`gcePersistentDisk`](#gcepersistentdisk) +* [`vsphereVolume`](#vspherevolume) + ### flexVolume -FlexVolume is an out-of-tree plugin interface that has existed in Kubernetes -since version 1.2 (before CSI). It uses an exec-based model to interface with -drivers. The FlexVolume driver binaries must be installed in a pre-defined volume -plugin path on each node and in some cases the control plane nodes as well. +{{< feature-state for_k8s_version="v1.23" state="deprecated" >}} -Pods interact with FlexVolume drivers through the `flexvolume` in-tree volume plugin. -For more details, see the [FlexVolume](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-storage/flexvolume.md) examples. +FlexVolume is an out-of-tree plugin interface that uses an exec-based model to interface +with storage drivers. The FlexVolume driver binaries must be installed in a pre-defined +volume plugin path on each node and in some cases the control plane nodes as well. + +Pods interact with FlexVolume drivers through the `flexVolume` in-tree volume plugin. +For more details, see the FlexVolume [README](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-storage/flexvolume.md#readme) document. + +The following FlexVolume [plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), +deployed as PowerShell scripts on the host, support Windows nodes: + +* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd) +* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd) + +{{< note >}} +FlexVolume is deprecated. Using an out-of-tree CSI driver is the recommended way to integrate external storage with Kubernetes. + +Maintainers of FlexVolume driver should implement a CSI Driver and help to migrate users of FlexVolume drivers to CSI. +Users of FlexVolume should move their workloads to use the equivalent CSI Driver. +{{< /note >}} ## Mount propagation diff --git a/content/en/docs/concepts/storage/windows-storage.md b/content/en/docs/concepts/storage/windows-storage.md new file mode 100644 index 0000000000..b8f40177ca --- /dev/null +++ b/content/en/docs/concepts/storage/windows-storage.md @@ -0,0 +1,71 @@ +--- +reviewers: +- jingxu97 +- mauriciopoppe +- jayunit100 +- jsturtevant +- marosset +- aravindhp +title: Windows Storage +content_type: concept +--- + +<!-- overview --> + +This page provides an storage overview specific to the Windows operating system. + +<!-- body --> + +## Persistent storage {#storage} + +Windows has a layered filesystem driver to mount container layers and create a copy +filesystem based on NTFS. All file paths in the container are resolved only within +the context of that container. + +* With Docker, volume mounts can only target a directory in the container, and not + an individual file. This limitation does not apply to containerd. +* Volume mounts cannot project files or directories back to the host filesystem. +* Read-only filesystems are not supported because write access is always required + for the Windows registry and SAM database. However, read-only volumes are supported. +* Volume user-masks and permissions are not available. Because the SAM is not shared + between the host & container, there's no mapping between them. All permissions are + resolved within the context of the container. + +As a result, the following storage functionality is not supported on Windows nodes: + +* Volume subpath mounts: only the entire volume can be mounted in a Windows container +* Subpath volume mounting for Secrets +* Host mount projection +* Read-only root filesystem (mapped volumes still support `readOnly`) +* Block device mapping +* Memory as the storage medium (for example, `emptyDir.medium` set to `Memory`) +* File system features like uid/gid; per-user Linux filesystem permissions +* Setting [secret permissions with DefaultMode](/docs/concepts/configuration/secret/#secret-files-permissions) (due to UID/GID dependency) +* NFS based storage/volume support +* Expanding the mounted volume (resizefs) + +Kubernetes {{< glossary_tooltip text="volumes" term_id="volume" >}} enable complex +applications, with data persistence and Pod volume sharing requirements, to be deployed +on Kubernetes. Management of persistent volumes associated with a specific storage +back-end or protocol includes actions such as provisioning/de-provisioning/resizing +of volumes, attaching/detaching a volume to/from a Kubernetes node and +mounting/dismounting a volume to/from individual containers in a pod that needs to +persist data. + +Volume management components are shipped as Kubernetes volume +[plugin](/docs/concepts/storage/volumes/#types-of-volumes). +The following broad classes of Kubernetes volume plugins are supported on Windows: + +* [`FlexVolume plugins`](/docs/concepts/storage/volumes/#flexVolume) + * Please note that FlexVolumes have been deprecated as of 1.23 +* [`CSI Plugins`](/docs/concepts/storage/volumes/#csi) + +##### In-tree volume plugins + +The following in-tree plugins support persistent storage on Windows nodes: + +* [`awsElasticBlockStore`](/docs/concepts/storage/volumes/#awselasticblockstore) +* [`azureDisk`](/docs/concepts/storage/volumes/#azuredisk) +* [`azureFile`](/docs/concepts/storage/volumes/#azurefile) +* [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk) +* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume) diff --git a/content/en/docs/setup/production-environment/windows/_index.md b/content/en/docs/concepts/windows/_index.md similarity index 100% rename from content/en/docs/setup/production-environment/windows/_index.md rename to content/en/docs/concepts/windows/_index.md diff --git a/content/en/docs/concepts/windows/intro.md b/content/en/docs/concepts/windows/intro.md new file mode 100644 index 0000000000..f3867ea86c --- /dev/null +++ b/content/en/docs/concepts/windows/intro.md @@ -0,0 +1,387 @@ +--- +reviewers: +- jayunit100 +- jsturtevant +- marosset +- perithompson +title: Windows containers in Kubernetes +content_type: concept +weight: 65 +--- + +<!-- overview --> + +Windows applications constitute a large portion of the services and applications that +run in many organizations. [Windows containers](https://aka.ms/windowscontainers) +provide a way to encapsulate processes and package dependencies, making it easier +to use DevOps practices and follow cloud native patterns for Windows applications. + +Organizations with investments in Windows-based applications and Linux-based +applications don't have to look for separate orchestrators to manage their workloads, +leading to increased operational efficiencies across their deployments, regardless +of operating system. + +<!-- body --> + +## Windows nodes in Kubernetes + +To enable the orchestration of Windows containers in Kubernetes, include Windows nodes +in your existing Linux cluster. Scheduling Windows containers in +{{< glossary_tooltip text="Pods" term_id="pod" >}} on Kubernetes is similar to +scheduling Linux-based containers. + +In order to run Windows containers, your Kubernetes cluster must include +multiple operating systems. +While you can only run the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} on Linux, +you can deploy worker nodes running either Windows or Linux. + +Windows {{< glossary_tooltip text="nodes" term_id="node" >}} are +[supported](#windows-os-version-support) provided that the operating system is +Windows Server 2019. + +This document uses the term *Windows containers* to mean Windows containers with +process isolation. Kubernetes does not support running Windows containers with +[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container). + +## Compatibility and limitations {#limitations} + +Some node features are only available if you use a specific +[container runtime](#container-runtime); others are not available on Windows nodes, +including: + +* HugePages: not supported for Windows containers +* Privileged containers: not supported for Windows containers. + [HostProcess Containers](/docs/tasks/configure-pod-container/create-hostprocess-pod/) offer similar functionality. +* TerminationGracePeriod: requires containerD + +Not all features of shared namespaces are supported. See [API compatibility](#api) +for more details. + +See [Windows OS version compatibility](#windows-os-version-support) for details on +the Windows versions that Kubernetes is tested against. + +From an API and kubectl perspective, Windows containers behave in much the same +way as Linux-based containers. However, there are some notable differences in key +functionality which are outlined in this section. + +### Comparison with Linux {#compatibility-linux-similarities} + +Key Kubernetes elements work the same way in Windows as they do in Linux. This +section refers to several key workload abstractions and how they map to Windows. + +* [Pods](/docs/concepts/workloads/pods/) + + A Pod is the basic building block of Kubernetes–the smallest and simplest unit in + the Kubernetes object model that you create or deploy. You may not deploy Windows and + Linux containers in the same Pod. All containers in a Pod are scheduled onto a single + Node where each Node represents a specific platform and architecture. The following + Pod capabilities, properties and events are supported with Windows containers: + + * Single or multiple containers per Pod with process isolation and volume sharing + * Pod `status` fields + * Readiness, liveness, and startup probes + * postStart & preStop container lifecycle hooks + * ConfigMap, Secrets: as environment variables or volumes + * `emptyDir` volumes + * Named pipe host mounts + * Resource limits + * OS field: + + The `.spec.os.name` field should be set to `windows` to indicate that the current Pod uses Windows containers. + The `IdentifyPodOS` feature gate needs to be enabled for this field to be recognized. + + {{< note >}} + Starting from 1.24, the `IdentifyPodOS` feature gate is in Beta stage and defaults to be enabled. + {{< /note >}} + + If the `IdentifyPodOS` feature gate is enabled and you set the `.spec.os.name` field to `windows`, + you must not set the following fields in the `.spec` of that Pod: + + * `spec.hostPID` + * `spec.hostIPC` + * `spec.securityContext.seLinuxOptions` + * `spec.securityContext.seccompProfile` + * `spec.securityContext.fsGroup` + * `spec.securityContext.fsGroupChangePolicy` + * `spec.securityContext.sysctls` + * `spec.shareProcessNamespace` + * `spec.securityContext.runAsUser` + * `spec.securityContext.runAsGroup` + * `spec.securityContext.supplementalGroups` + * `spec.containers[*].securityContext.seLinuxOptions` + * `spec.containers[*].securityContext.seccompProfile` + * `spec.containers[*].securityContext.capabilities` + * `spec.containers[*].securityContext.readOnlyRootFilesystem` + * `spec.containers[*].securityContext.privileged` + * `spec.containers[*].securityContext.allowPrivilegeEscalation` + * `spec.containers[*].securityContext.procMount` + * `spec.containers[*].securityContext.runAsUser` + * `spec.containers[*].securityContext.runAsGroup` + + In the above list, wildcards (`*`) indicate all elements in a list. + For example, `spec.containers[*].securityContext` refers to the SecurityContext object + for all containers. If any of these fields is specified, the Pod will + not be admitted by the API server. + +* [Workload resources](/docs/concepts/workloads/controllers/) including: + * ReplicaSet + * Deployment + * StatefulSet + * DaemonSet + * Job + * CronJob + * ReplicationController +* {{< glossary_tooltip text="Services" term_id="service" >}} + See [Load balancing and Services](/docs/concepts/services-networking/windows-networking/#load-balancing-and-services) for more details. + +Pods, workload resources, and Services are critical elements to managing Windows +workloads on Kubernetes. However, on their own they are not enough to enable +the proper lifecycle management of Windows workloads in a dynamic cloud native +environment. + +* `kubectl exec` +* Pod and container metrics +* {{< glossary_tooltip text="Horizontal pod autoscaling" term_id="horizontal-pod-autoscaler" >}} +* {{< glossary_tooltip text="Resource quotas" term_id="resource-quota" >}} +* Scheduler preemption + +### Command line options for the kubelet {#kubelet-compatibility} + +Some kubelet command line options behave differently on Windows, as described below: + +* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process + (see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu)) +* The `--kube-reserved`, `--system-reserved` , and `--eviction-hard` flags update + [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +* Eviction by using `--enforce-node-allocable` is not implemented +* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented +* When running on a Windows node the kubelet does not have memory or CPU + restrictions. `--kube-reserved` and `--system-reserved` only subtract from `NodeAllocatable` + and do not guarantee resource provided for workloads. + See [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/#resource-reservation) + for more information. +* The `MemoryPressure` Condition is not implemented +* The kubelet does not take OOM eviction actions + +### API compatibility {#api} + +There are subtle differences in the way the Kubernetes APIs work for Windows due to the OS +and container runtime. Some workload properties were designed for Linux, and fail to run on Windows. + +At a high level, these OS concepts are different: + +* Identity - Linux uses userID (UID) and groupID (GID) which + are represented as integer types. User and group names + are not canonical - they are just an alias in `/etc/groups` + or `/etc/passwd` back to UID+GID. Windows uses a larger binary + [security identifier](https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-identifiers) (SID) + which is stored in the Windows Security Access Manager (SAM) database. This + database is not shared between the host and containers, or between containers. +* File permissions - Windows uses an access control list based on (SIDs), whereas + POSIX systems such as Linux use a bitmask based on object permissions and UID+GID, + plus _optional_ access control lists. +* File paths - the convention on Windows is to use `\` instead of `/`. The Go IO + libraries typically accept both and just make it work, but when you're setting a + path or command line that's interpreted inside a container, `\` may be needed. +* Signals - Windows interactive apps handle termination differently, and can + implement one or more of these: + * A UI thread handles well-defined messages including `WM_CLOSE`. + * Console apps handle Ctrl-C or Ctrl-break using a Control Handler. + * Services register a Service Control Handler function that can accept + `SERVICE_CONTROL_STOP` control codes. + +Container exit codes follow the same convention where 0 is success, and nonzero is failure. +The specific error codes may differ across Windows and Linux. However, exit codes +passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. + +#### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} + +The following list documents differences between how Pod container specifications +work between Windows and Linux: + +* Huge pages are not implemented in the Windows container + runtime, and are not available. They require [asserting a user + privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support) + that's not configurable for containers. +* `requests.cpu` and `requests.memory` - requests are subtracted + from node available resources, so they can be used to avoid overprovisioning a + node. However, they cannot be used to guarantee resources in an overprovisioned + node. They should be applied to all containers as a best practice if the operator + wants to avoid overprovisioning entirely. +* `securityContext.allowPrivilegeEscalation` - + not possible on Windows; none of the capabilities are hooked up +* `securityContext.capabilities` - + POSIX capabilities are not implemented on Windows +* `securityContext.privileged` - + Windows doesn't support privileged containers +* `securityContext.procMount` - + Windows doesn't have a `/proc` filesystem +* `securityContext.readOnlyRootFilesystem` - + not possible on Windows; write access is required for registry & system + processes to run inside the container +* `securityContext.runAsGroup` - + not possible on Windows as there is no GID support +* `securityContext.runAsNonRoot` - + this setting will prevent containers from running as `ContainerAdministrator` + which is the closest equivalent to a root user on Windows. +* `securityContext.runAsUser` - + use [`runAsUserName`](/docs/tasks/configure-pod-container/configure-runasusername) + instead +* `securityContext.seLinuxOptions` - + not possible on Windows as SELinux is Linux-specific +* `terminationMessagePath` - + this has some limitations in that Windows doesn't support mapping single files. The + default value is `/dev/termination-log`, which does work because it does not + exist on Windows by default. + +#### Field compatibility for Pod specifications {#compatibility-v1-pod} + +The following list documents differences between how Pod specifications work between Windows and Linux: + +* `hostIPC` and `hostpid` - host namespace sharing is not possible on Windows +* `hostNetwork` - There is no Windows OS support to share the host network +* `dnsPolicy` - setting the Pod `dnsPolicy` to `ClusterFirstWithHostNet` is + not supported on Windows because host networking is not provided. Pods always + run with a container network. +* `podSecurityContext` (see below) +* `shareProcessNamespace` - this is a beta feature, and depends on Linux namespaces + which are not implemented on Windows. Windows cannot share process namespaces or + the container's root filesystem. Only the network can be shared. +* `terminationGracePeriodSeconds` - this is not fully implemented in Docker on Windows, + see the [GitHub issue](https://github.com/moby/moby/issues/25982). + The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, + then Windows waits 5 seconds by default, and finally shuts down + all processes using the normal Windows shutdown behavior. The 5 + second default is actually in the Windows registry + [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), + so it can be overridden when the container is built. +* `volumeDevices` - this is a beta feature, and is not implemented on Windows. + Windows cannot attach raw block devices to pods. +* `volumes` + * If you define an `emptyDir` volume, you cannot set its volume source to `memory`. +* You cannot enable `mountPropagation` for volume mounts as this is not + supported on Windows. + +#### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} + +None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows. + +## Node problem detector + +The node problem detector (see +[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/)) +has preliminary support for Windows. +For more information, visit the project's [GitHub page](https://github.com/kubernetes/node-problem-detector#windows). + +## Pause container + +In a Kubernetes Pod, an infrastructure or “pause” container is first created +to host the container. In Linux, the cgroups and namespaces that make up a pod +need a process to maintain their continued existence; the pause process provides +this. Containers that belong to the same pod, including infrastructure and worker +containers, share a common network endpoint (same IPv4 and / or IPv6 address, same +network port spaces). Kubernetes uses pause containers to allow for worker containers +crashing or restarting without losing any of the networking configuration. + +Kubernetes maintains a multi-architecture image that includes support for Windows. +For Kubernetes v{{< skew currentVersion >}} the recommended pause image is `k8s.gcr.io/pause:3.6`. +The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause) +is available on GitHub. + +Microsoft maintains a different multi-architecture image, with Linux and Windows +amd64 support, that you can find as `mcr.microsoft.com/oss/kubernetes/pause:3.6`. +This image is built from the same source as the Kubernetes maintained image but +all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft. +The Kubernetes project recommends using the Microsoft maintained image if you are +deploying to a production or production-like environment that requires signed +binaries. + +## Container runtimes {#container-runtime} + +You need to install a +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} +into each node in the cluster so that Pods can run there. + +The following container runtimes work with Windows: + +{{% thirdparty-content %}} + +### ContainerD + +{{< feature-state for_k8s_version="v1.20" state="stable" >}} + +You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ +as the container runtime for Kubernetes nodes that run Windows. + +Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd). + +{{< note >}} +There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations) +when using GMSA with containerd to access Windows network shares, which requires a +kernel patch. +{{< /note >}} + +### Mirantis Container Runtime {#mcr} + +[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) +is available as a container runtime for all Windows Server 2019 and later versions. + +See [Install MCR on Windows Servers](https://docs.mirantis.com/mcr/20.10/install/mcr-windows.html) for more information. + +## Windows OS version compatibility {#windows-os-version-support} + +On Windows nodes, strict compatibility rules apply where the host OS version must +match the container base image OS version. Only Windows containers with a container +operating system of Windows Server 2019 are fully supported. + +For Kubernetes v{{< skew currentVersion >}}, operating system compatibility for Windows nodes (and Pods) +is as follows: + +Windows Server LTSC release +: Windows Server 2019 +: Windows Server 2022 + +Windows Server SAC release +: Windows Server version 20H2 + +The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) also applies. + +## Getting help and troubleshooting {#troubleshooting} + +Your main source of help for troubleshooting your Kubernetes cluster should start +with the [Troubleshooting](/docs/tasks/debug/) +page. + +Some additional, Windows-specific troubleshooting help is included +in this section. Logs are an important element of troubleshooting +issues in Kubernetes. Make sure to include them any time you seek +troubleshooting assistance from other contributors. Follow the +instructions in the +SIG Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs). + +### Reporting issues and feature requests + +If you have what looks like a bug, or you would like to +make a feature request, please follow the [SIG Windows contributing guide](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#reporting-issues-and-feature-requests) to create a new issue. +You should first search the list of issues in case it was +reported previously and comment with your experience on the issue and add additional +logs. SIG Windows channel on the Kubernetes Slack is also a great avenue to get some initial support and +troubleshooting ideas prior to creating a ticket. + +## Deployment tools + +The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control +plane to manage the cluster it, and nodes to run your workloads. +[Adding Windows nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) +explains how to deploy Windows nodes to your cluster using kubeadm. + +The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes. + +## Windows distribution channels + +For a detailed explanation of Windows distribution channels see the +[Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). + +Information on the different Windows Server servicing channels +including their support models can be found at +[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison). diff --git a/content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md b/content/en/docs/concepts/windows/user-guide.md similarity index 72% rename from content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md rename to content/en/docs/concepts/windows/user-guide.md index ec47f5637a..da5b8a6fec 100644 --- a/content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md +++ b/content/en/docs/concepts/windows/user-guide.md @@ -3,7 +3,6 @@ reviewers: - jayunit100 - jsturtevant - marosset -- perithompson title: Guide for scheduling Windows containers in Kubernetes content_type: concept weight: 75 @@ -11,31 +10,29 @@ weight: 75 <!-- overview --> -Windows applications constitute a large portion of the services and applications that run in many organizations. -This guide walks you through the steps to configure and deploy a Windows container in Kubernetes. - - +Windows applications constitute a large portion of the services and applications that run in many organizations. +This guide walks you through the steps to configure and deploy Windows containers in Kubernetes. <!-- body --> ## Objectives * Configure an example deployment to run Windows containers on the Windows node -* (Optional) Configure an Active Directory Identity for your Pod using Group Managed Service Accounts (GMSA) +* Highlight Windows specific funcationality in Kubernetes ## Before you begin -* Create a Kubernetes cluster that includes a +* Create a Kubernetes cluster that includes a control plane and a [worker node running Windows Server](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) -* It is important to note that creating and deploying services and workloads on Kubernetes -behaves in much the same way for Linux and Windows containers. -[Kubectl commands](/docs/reference/kubectl/overview/) to interface with the cluster are identical. +* It is important to note that creating and deploying services and workloads on Kubernetes +behaves in much the same way for Linux and Windows containers. +[Kubectl commands](/docs/reference/kubectl/) to interface with the cluster are identical. The example in the section below is provided to jumpstart your experience with Windows containers. ## Getting Started: Deploying a Windows container -To deploy a Windows container on Kubernetes, you must first create an example application. -The example YAML file below creates a simple webserver application. +The example YAML file below deploys a simple webserver application running inside a Windows container. + Create a service spec named `win-webserver.yaml` with the contents below: ```yaml @@ -83,8 +80,8 @@ spec: ``` {{< note >}} -Port mapping is also supported, but for simplicity in this example -the container port 80 is exposed directly to the service. +Port mapping is also supported, but for simplicity this example exposes +port 80 of the container directly to the Service. {{< /note >}} 1. Check that all nodes are healthy: @@ -104,20 +101,19 @@ the container port 80 is exposed directly to the service. 1. Check that the deployment succeeded. To verify: - * Two containers per pod on the Windows node, use `docker ps` - * Two pods listed from the Linux control plane node, use `kubectl get pods` - * Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux control plane node + * Two pods listed from the Linux control plane node, use `kubectl get pods` + * Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux control plane node to check for a web server response - * Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) - using docker exec or kubectl exec - * Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`) + * Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) + using `docker exec` or `kubectl exec` + * Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`) from the Linux control plane node and from individual pods * Service discovery, `curl` the service name with the Kubernetes [default DNS suffix](/docs/concepts/services-networking/dns-pod-service/#services) * Inbound connectivity, `curl` the NodePort from the Linux control plane node or machines outside of the cluster - * Outbound connectivity, `curl` external IPs from inside the pod using kubectl exec + * Outbound connectivity, `curl` external IPs from inside the pod using `kubectl exec` {{< note >}} -Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. +Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs. {{< /note >}} @@ -125,64 +121,85 @@ Only Windows pods are able to access service IPs. ### Capturing logs from workloads -Logs are an important element of observability; they enable users to gain insights -into the operational aspect of workloads and are a key ingredient to troubleshooting issues. -Because Windows containers and workloads inside Windows containers behave differently from Linux containers, -users had a hard time collecting logs, limiting operational visibility. -Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows) -or push entries to the application event log. -[LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor), an open source tool by Microsoft, -is the recommended way to monitor configured log sources inside a Windows container. -LogMonitor supports monitoring event logs, ETW providers, and custom application logs, +Logs are an important element of observability; they enable users to gain insights +into the operational aspect of workloads and are a key ingredient to troubleshooting issues. +Because Windows containers and workloads inside Windows containers behave differently from Linux containers, +users had a hard time collecting logs, limiting operational visibility. +Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows) +or push entries to the application event log. +[LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor), an open source tool by Microsoft, +is the recommended way to monitor configured log sources inside a Windows container. +LogMonitor supports monitoring event logs, ETW providers, and custom application logs, piping them to STDOUT for consumption by `kubectl logs <pod>`. -Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files +Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT. -## Using configurable Container usernames +## Configuring container user -Starting with Kubernetes v1.16, Windows containers can be configured to run their entrypoints and processes -with different usernames than the image defaults. -The way this is achieved is a bit different from the way it is done for Linux containers. +### Using configurable Container usernames + +Windows containers can be configured to run their entrypoints and processes +with different usernames than the image defaults. Learn more about it [here](/docs/tasks/configure-pod-container/configure-runasusername/). -## Managing Workload Identity with Group Managed Service Accounts +### Managing Workload Identity with Group Managed Service Accounts -Starting with Kubernetes v1.14, Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). -Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, -simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. -Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. +Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). +Group Managed Service Accounts are a specific type of Active Directory account that provide automatic password management, +simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. +Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers [here](/docs/tasks/configure-pod-container/configure-gmsa/). ## Taints and Tolerations -Users today need to use some combination of taints and node selectors in order to -keep Linux and Windows workloads on their respective OS-specific nodes. -This likely imposes a burden only on Windows users. The recommended approach is outlined below, +Users need to use some combination of taints and node selectors in order to +schedule Linux and Windows workloads to their respective OS-specific nodes. +The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads. +If the `IdentifyPodOS` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is +enabled, you can (and should) set `.spec.os.name` for a Pod to indicate the operating system +that the containers in that Pod are designed for. For Pods that run Linux containers, set +`.spec.os.name` to `linux`. For Pods that run Windows containers, set `.spec.os.name` +to Windows. + +{{< note >}} +Starting from 1.24, the `IdentifyPodOS` feature is in Beta stage and defaults to be enabled. +{{< /note >}} + +The scheduler does not use the value of `.spec.os.name` when assigning Pods to nodes. You should +use normal Kubernetes mechanisms for +[assigning pods to nodes](/docs/concepts/scheduling-eviction/assign-pod-node/) +to ensure that the control plane for your cluster places pods onto nodes that are running the +appropriate operating system. + +The `.spec.os.name` value has no effect on the scheduling of the Windows pods, +so taints and tolerations and node selectors are still required + to ensure that the Windows pods land onto appropriate Windows nodes. + ### Ensuring OS-specific workloads land on the appropriate container host -Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. +Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All Kubernetes nodes today have the following default labels: * kubernetes.io/os = [windows|linux] * kubernetes.io/arch = [amd64|arm64|...] -If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`, -it is possible the Pod can be scheduled on any host, Windows or Linux. -This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. +If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`, +it is possible the Pod can be scheduled on any host, Windows or Linux. +This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector. -However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, -as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. -In those situations, you may be hesitant to make the configuration change to add nodeSelectors. -The alternative is to use Taints. Because the kubelet can set Taints during registration, +However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, +as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. +In those situations, you may be hesitant to make the configuration change to add nodeSelectors. +The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only. For example: `--register-with-taints='os=windows:NoSchedule'` -By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). +By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector and the appropriate matching toleration to choose Windows. @@ -202,26 +219,24 @@ tolerations: The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelectors. -Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this. +Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this. If you're running an older version, then it's recommended to add this label manually to Windows nodes. -This label reflects the Windows major, minor, and build number that need to match for compatibility. +This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used today for each Windows Server version. | Product Name | Build Number(s) | |--------------------------------------|------------------------| | Windows Server 2019 | 10.0.17763 | -| Windows Server version 1809 | 10.0.17763 | -| Windows Server version 1903 | 10.0.18362 | - +| Windows Server, Version 20H2 | 10.0.19042 | +| Windows Server 2022 | 10.0.20348 | ### Simplifying with RuntimeClass -[RuntimeClass] can be used to simplify the process of using taints and tolerations. +[RuntimeClass] can be used to simplify the process of using taints and tolerations. A cluster administrator can create a `RuntimeClass` object which is used to encapsulate these taints and tolerations. - -1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector` +1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector` for the Windows OS, architecture, and version. ```yaml @@ -292,7 +307,4 @@ spec: app: iis-2019 ``` - - - -[RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/ +[RuntimeClass]: /docs/concepts/containers/runtime-class/ diff --git a/content/en/docs/concepts/workloads/_index.md b/content/en/docs/concepts/workloads/_index.md index 2c9dd8aa8e..dffd727505 100644 --- a/content/en/docs/concepts/workloads/_index.md +++ b/content/en/docs/concepts/workloads/_index.md @@ -70,7 +70,7 @@ visit [Configuration](/docs/concepts/configuration/). There are two supporting concepts that provide backgrounds about how Kubernetes manages pods for applications: -* [Garbage collection](/docs/concepts/workloads/controllers/garbage-collection/) tidies up objects +* [Garbage collection](/docs/concepts/architecture/garbage-collection/) tidies up objects from your cluster after their _owning resource_ has been removed. * The [_time-to-live after finished_ controller](/docs/concepts/workloads/controllers/ttlafterfinished/) removes Jobs once a defined time has passed since they completed. diff --git a/content/en/docs/concepts/workloads/controllers/cron-jobs.md b/content/en/docs/concepts/workloads/controllers/cron-jobs.md index c6cd4d1336..2d2416226a 100644 --- a/content/en/docs/concepts/workloads/controllers/cron-jobs.md +++ b/content/en/docs/concepts/workloads/controllers/cron-jobs.md @@ -17,8 +17,6 @@ A _CronJob_ creates {{< glossary_tooltip term_id="job" text="Jobs" >}} on a repe One CronJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format. -In addition, the CronJob schedule supports timezone handling, you can specify the timezone by adding "CRON_TZ=<time zone>" at the beginning of the CronJob schedule, and it is recommended to always set `CRON_TZ`. - {{< caution >}} All **CronJob** `schedule:` times are based on the timezone of the {{< glossary_tooltip term_id="kube-controller-manager" text="kube-controller-manager" >}}. @@ -28,6 +26,16 @@ containers, the timezone set for the kube-controller-manager container determine that the cron job controller uses. {{< /caution >}} +{{< caution >}} +The [v1 CronJob API](/docs/reference/kubernetes-api/workload-resources/cron-job-v1/) +does not officially support setting timezone as explained above. + +Setting variables such as `CRON_TZ` or `TZ` is not officially supported by the Kubernetes project. +`CRON_TZ` or `TZ` is an implementation detail of the internal library being used +for parsing and calculating the next Job creation time. Any usage of it is not +recommended in a production cluster. +{{< /caution >}} + When creating the manifest for a CronJob resource, make sure the name you provide is a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). The name must be no longer than 52 characters. This is because the CronJob controller will automatically @@ -55,16 +63,15 @@ takes you through this example in more detail). ### Cron schedule syntax ``` -# ┌────────────────── timezone (optional) -# | ┌───────────── minute (0 - 59) -# | │ ┌───────────── hour (0 - 23) -# | │ │ ┌───────────── day of the month (1 - 31) -# | │ │ │ ┌───────────── month (1 - 12) -# | │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday; -# | │ │ │ │ │ 7 is also Sunday on some systems) -# | │ │ │ │ │ -# | │ │ │ │ │ -# CRON_TZ=UTC * * * * * +# ┌───────────── minute (0 - 59) +# │ ┌───────────── hour (0 - 23) +# │ │ ┌───────────── day of the month (1 - 31) +# │ │ │ ┌───────────── month (1 - 12) +# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday; +# │ │ │ │ │ 7 is also Sunday on some systems) +# │ │ │ │ │ OR sun, mon, tue, wed, thu, fri, sat +# │ │ │ │ │ +# * * * * * ``` @@ -78,12 +85,27 @@ takes you through this example in more detail). -For example, the line below states that the task must be started every Friday at midnight, as well as on the 13th of each month at midnight(in UTC): +For example, the line below states that the task must be started every Friday at midnight, as well as on the 13th of each month at midnight: -`CRON_TZ=UTC 0 0 13 * 5` +`0 0 13 * 5` To generate CronJob schedule expressions, you can also use web tools like [crontab.guru](https://crontab.guru/). +## Time zones +For CronJobs with no time zone specified, the kube-controller-manager interprets schedules relative to its local time zone. + +{{< feature-state for_k8s_version="v1.24" state="alpha" >}} + +If you enable the `CronJobTimeZone` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), +you can specify a time zone for a CronJob (if you don't enable that feature gate, or if you are using a version of +Kubernetes that does not have experimental time zone support, all CronJobs in your cluster have an unspecified +timezone). + +When you have the feature enabled, you can set `spec.timeZone` to the name of a valid [time zone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) name. For example, setting +`spec.timeZone: "Etc/UTC"` instructs Kubernetes to interpret the schedule relative to Coordinated Universal Time. + +A time zone database from the Go standard library is included in the binaries and used as a fallback in case an external database is not available on the system. + ## CronJob limitations {#cron-job-limitations} A cron job creates a job object _about_ once per execution time of its schedule. We say "about" because there @@ -136,9 +158,16 @@ and set this flag to `false`. For example: ## {{% heading "whatsnext" %}} -[Cron expression format](https://en.wikipedia.org/wiki/Cron) -documents the format of CronJob `schedule` fields. - -For instructions on creating and working with cron jobs, and for an example of CronJob -manifest, see [Running automated tasks with cron jobs](/docs/tasks/job/automated-tasks-with-cron-jobs). - +* Learn about [Pods](/docs/concepts/workloads/pods/) and + [Jobs](/docs/concepts/workloads/controllers/job/), two concepts + that CronJobs rely upon. +* Read about the [format](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-CRON_Expression_Format) + of CronJob `.spec.schedule` fields. +* For instructions on creating and working with CronJobs, and for an example + of a CronJob manifest, + see [Running automated tasks with CronJobs](/docs/tasks/job/automated-tasks-with-cron-jobs/). +* For instructions to clean up failed or completed jobs automatically, + see [Clean up Jobs automatically](/docs/concepts/workloads/controllers/job/#clean-up-finished-jobs-automatically) +* `CronJob` is part of the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/cron-job-v1" >}} + object definition to understand the API for Kubernetes cron jobs. diff --git a/content/en/docs/concepts/workloads/controllers/daemonset.md b/content/en/docs/concepts/workloads/controllers/daemonset.md index c5d3c4c6c8..d74a4a0c62 100644 --- a/content/en/docs/concepts/workloads/controllers/daemonset.md +++ b/content/en/docs/concepts/workloads/controllers/daemonset.md @@ -76,9 +76,9 @@ A Pod Template in a DaemonSet must have a [`RestartPolicy`](/docs/concepts/workl The `.spec.selector` field is a pod selector. It works the same as the `.spec.selector` of a [Job](/docs/concepts/workloads/controllers/job/). -As of Kubernetes 1.8, you must specify a pod selector that matches the labels of the -`.spec.template`. The pod selector will no longer be defaulted when left empty. Selector -defaulting was not compatible with `kubectl apply`. Also, once a DaemonSet is created, +You must specify a pod selector that matches the labels of the +`.spec.template`. +Also, once a DaemonSet is created, its `.spec.selector` can not be mutated. Mutating the pod selector can lead to the unintentional orphaning of Pods, and it was found to be confusing to users. @@ -91,8 +91,8 @@ The `.spec.selector` is an object consisting of two fields: When the two are specified the result is ANDed. -If the `.spec.selector` is specified, it must match the `.spec.template.metadata.labels`. -Config with these not matching will be rejected by the API. +The `.spec.selector` must match the `.spec.template.metadata.labels`. +Config with these two not matching will be rejected by the API. ### Running Pods on select Nodes @@ -107,7 +107,7 @@ If you do not specify either, then the DaemonSet controller will create Pods on ### Scheduled by default scheduler -{{< feature-state state="stable" for-kubernetes-version="1.17" >}} +{{< feature-state for_k8s_version="1.17" state="stable" >}} A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the node that a Pod runs on is selected by the Kubernetes scheduler. However, @@ -185,7 +185,7 @@ You can modify the Pods that a DaemonSet creates. However, Pods do not allow al fields to be updated. Also, the DaemonSet controller will use the original template the next time a node (even with the same name) is created. -You can delete a DaemonSet. If you specify `--cascade=false` with `kubectl`, then the Pods +You can delete a DaemonSet. If you specify `--cascade=orphan` with `kubectl`, then the Pods will be left on the nodes. If you subsequently create a new DaemonSet with the same selector, the new DaemonSet adopts the existing Pods. If any Pods need replacing the DaemonSet replaces them according to its `updateStrategy`. @@ -235,3 +235,18 @@ all or certain hosts, if the DaemonSet provides node-level functionality that al For example, [network plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) often include a component that runs as a DaemonSet. The DaemonSet component makes sure that the node where it's running has working cluster networking. +## {{% heading "whatsnext" %}} + +* Learn about [Pods](/docs/concepts/workloads/pods). + * Learn about [static Pods](#static-pods), which are useful for running Kubernetes + {{< glossary_tooltip text="control plane" term_id="control-plane" >}} components. +* Find out how to use DaemonSets + * [Perform a rolling update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/) + * [Perform a rollback on a DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/) + (for example, if a roll out didn't work how you expected). +* Understand [how Kubernetes assigns Pods to Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/). +* Learn about [device plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) and + [add ons](/docs/concepts/cluster-administration/addons/), which often run as DaemonSets. +* `DaemonSet` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/daemon-set-v1" >}} + object definition to understand the API for daemon sets. diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index 22b95255c5..30a7457d4c 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -32,7 +32,7 @@ The following are typical use cases for Deployments: * [Declare the new state of the Pods](#updating-a-deployment) by updating the PodTemplateSpec of the Deployment. A new ReplicaSet is created and the Deployment manages moving the Pods from the old ReplicaSet to the new one at a controlled rate. Each new ReplicaSet updates the revision of the Deployment. * [Rollback to an earlier Deployment revision](#rolling-back-a-deployment) if the current state of the Deployment is not stable. Each rollback updates the revision of the Deployment. * [Scale up the Deployment to facilitate more load](#scaling-a-deployment). -* [Pause the Deployment](#pausing-and-resuming-a-deployment) to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout. +* [Pause the rollout of a Deployment](#pausing-and-resuming-a-deployment) to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout. * [Use the status of the Deployment](#deployment-status) as an indicator that a rollout has stuck. * [Clean up older ReplicaSets](#clean-up-policy) that you don't need anymore. @@ -75,11 +75,6 @@ Follow the steps given below to create the above Deployment: kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml ``` - {{< note >}} - You can specify the `--record` flag to write the command executed in the resource annotation `kubernetes.io/change-cause`. - The recorded change is useful for future introspection. For example, to see the commands executed in each Deployment revision. - {{< /note >}} - 2. Run `kubectl get deployments` to check if the Deployment was created. @@ -169,13 +164,13 @@ Follow the steps given below to update your Deployment: 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image. ```shell - kubectl --record deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 ``` or use the following command: ```shell - kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 --record + kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` The output is similar to: @@ -187,7 +182,7 @@ Follow the steps given below to update your Deployment: Alternatively, you can `edit` the Deployment and change `.spec.template.spec.containers[0].image` from `nginx:1.14.2` to `nginx:1.16.1`: ```shell - kubectl edit deployment.v1.apps/nginx-deployment + kubectl edit deployment/nginx-deployment ``` The output is similar to: @@ -260,10 +255,11 @@ up to 3 replicas, as well as scaling down the old ReplicaSet to 0 replicas. Deployment also ensures that only a certain number of Pods are created above the desired number of Pods. By default, it ensures that at most 125% of the desired number of Pods are up (25% max surge). - For example, if you look at the above Deployment closely, you will see that it first created a new Pod, - then deleted some old Pods, and created new ones. It does not kill old Pods until a sufficient number of + For example, if you look at the above Deployment closely, you will see that it first creates a new Pod, + then deletes an old Pod, and creates another new one. It does not kill old Pods until a sufficient number of new Pods have come up, and does not create new Pods until a sufficient number of old Pods have been killed. - It makes sure that at least 2 Pods are available and that at max 4 Pods in total are available. + It makes sure that at least 3 Pods are available and that at max 4 Pods in total are available. In case of + a Deployment with 4 replicas, the number of Pods would be between 3 and 5. * Get details of your Deployment: ```shell @@ -310,10 +306,17 @@ up to 3 replicas, as well as scaling down the old ReplicaSet to 0 replicas. ``` Here you see that when you first created the Deployment, it created a ReplicaSet (nginx-deployment-2035384211) and scaled it up to 3 replicas directly. When you updated the Deployment, it created a new ReplicaSet - (nginx-deployment-1564180365) and scaled it up to 1 and then scaled down the old ReplicaSet to 2, so that at - least 2 Pods were available and at most 4 Pods were created at all times. It then continued scaling up and down - the new and the old ReplicaSet, with the same rolling update strategy. Finally, you'll have 3 available replicas - in the new ReplicaSet, and the old ReplicaSet is scaled down to 0. + (nginx-deployment-1564180365) and scaled it up to 1 and waited for it to come up. Then it scaled down the old ReplicaSet + to 2 and scaled up the new ReplicaSet to 2 so that at least 3 Pods were available and at most 4 Pods were created at all times. + It then continued scaling up and down the new and the old ReplicaSet, with the same rolling update strategy. + Finally, you'll have 3 available replicas in the new ReplicaSet, and the old ReplicaSet is scaled down to 0. + +{{< note >}} +Kubernetes doesn't count terminating Pods when calculating the number of `availableReplicas`, which must be between +`replicas - maxUnavailable` and `replicas + maxSurge`. As a result, you might notice that there are more Pods than +expected during a rollout, and that the total resources consumed by the Deployment is more than `replicas + maxSurge` +until the `terminationGracePeriodSeconds` of the terminating Pods expires. +{{< /note >}} ### Rollover (aka multiple updates in-flight) @@ -370,7 +373,7 @@ rolled back. * Suppose that you made a typo while updating the Deployment, by putting the image name as `nginx:1.161` instead of `nginx:1.16.1`: ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true + kubectl set image deployment/nginx-deployment nginx=nginx:1.161 ``` The output is similar to this: @@ -479,26 +482,25 @@ Follow the steps given below to check the rollout history: 1. First, check the revisions of this Deployment: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment + kubectl rollout history deployment/nginx-deployment ``` The output is similar to this: ``` deployments "nginx-deployment" REVISION CHANGE-CAUSE - 1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml --record=true - 2 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true - 3 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true + 1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml + 2 kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 + 3 kubectl set image deployment/nginx-deployment nginx=nginx:1.161 ``` `CHANGE-CAUSE` is copied from the Deployment annotation `kubernetes.io/change-cause` to its revisions upon creation. You can specify the`CHANGE-CAUSE` message by: - * Annotating the Deployment with `kubectl annotate deployment.v1.apps/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"` - * Append the `--record` flag to save the `kubectl` command that is making changes to the resource. + * Annotating the Deployment with `kubectl annotate deployment/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"` * Manually editing the manifest of the resource. 2. To see the details of each revision, run: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2 + kubectl rollout history deployment/nginx-deployment --revision=2 ``` The output is similar to this: @@ -506,7 +508,7 @@ Follow the steps given below to check the rollout history: deployments "nginx-deployment" revision 2 Labels: app=nginx pod-template-hash=1159050644 - Annotations: kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true + Annotations: kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 Containers: nginx: Image: nginx:1.16.1 @@ -523,7 +525,7 @@ Follow the steps given below to rollback the Deployment from the current version 1. Now you've decided to undo the current rollout and rollback to the previous revision: ```shell - kubectl rollout undo deployment.v1.apps/nginx-deployment + kubectl rollout undo deployment/nginx-deployment ``` The output is similar to this: @@ -533,7 +535,7 @@ Follow the steps given below to rollback the Deployment from the current version Alternatively, you can rollback to a specific revision by specifying it with `--to-revision`: ```shell - kubectl rollout undo deployment.v1.apps/nginx-deployment --to-revision=2 + kubectl rollout undo deployment/nginx-deployment --to-revision=2 ``` The output is similar to this: @@ -567,7 +569,7 @@ Follow the steps given below to rollback the Deployment from the current version CreationTimestamp: Sun, 02 Sep 2018 18:17:55 -0500 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=4 - kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true + kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 Selector: app=nginx Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate @@ -610,7 +612,7 @@ Follow the steps given below to rollback the Deployment from the current version You can scale a Deployment by using the following command: ```shell -kubectl scale deployment.v1.apps/nginx-deployment --replicas=10 +kubectl scale deployment/nginx-deployment --replicas=10 ``` The output is similar to this: ``` @@ -622,7 +624,7 @@ in your cluster, you can setup an autoscaler for your Deployment and choose the Pods you want to run based on the CPU utilization of your existing Pods. ```shell -kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80 +kubectl autoscale deployment/nginx-deployment --min=10 --max=15 --cpu-percent=80 ``` The output is similar to this: ``` @@ -651,7 +653,7 @@ For example, you are running a Deployment with 10 replicas, [maxSurge](#max-surg * You update to a new image which happens to be unresolvable from inside the cluster. ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:sometag + kubectl set image deployment/nginx-deployment nginx=nginx:sometag ``` The output is similar to this: @@ -703,12 +705,16 @@ nginx-deployment-1989198191 7 7 0 7m nginx-deployment-618515232 11 11 11 7m ``` -## Pausing and Resuming a Deployment +## Pausing and Resuming a rollout of a Deployment {#pausing-and-resuming-a-deployment} -You can pause a Deployment before triggering one or more updates and then resume it. This allows you to +When you update a Deployment, or plan to, you can pause rollouts +for that Deployment before you trigger one or more updates. When +you're ready to apply those changes, you resume rollouts for the +Deployment. This approach allows you to apply multiple fixes in between pausing and resuming without triggering unnecessary rollouts. * For example, with a Deployment that was created: + Get the Deployment details: ```shell kubectl get deploy @@ -730,7 +736,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Pause by running the following command: ```shell - kubectl rollout pause deployment.v1.apps/nginx-deployment + kubectl rollout pause deployment/nginx-deployment ``` The output is similar to this: @@ -740,7 +746,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Then update the image of the Deployment: ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` The output is similar to this: @@ -750,7 +756,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Notice that no new rollout started: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment + kubectl rollout history deployment/nginx-deployment ``` The output is similar to this: @@ -759,7 +765,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess REVISION CHANGE-CAUSE 1 <none> ``` -* Get the rollout status to ensure that the Deployment is updated successfully: +* Get the rollout status to verify that the existing ReplicaSet has not changed: ```shell kubectl get rs ``` @@ -772,7 +778,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * You can make as many updates as you wish, for example, update the resources that will be used: ```shell - kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi + kubectl set resources deployment/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi ``` The output is similar to this: @@ -780,12 +786,12 @@ apply multiple fixes in between pausing and resuming without triggering unnecess deployment.apps/nginx-deployment resource requirements updated ``` - The initial state of the Deployment prior to pausing it will continue its function, but new updates to - the Deployment will not have any effect as long as the Deployment is paused. + The initial state of the Deployment prior to pausing its rollout will continue its function, but new updates to + the Deployment will not have any effect as long as the Deployment rollout is paused. -* Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates: +* Eventually, resume the Deployment rollout and observe a new ReplicaSet coming up with all the new updates: ```shell - kubectl rollout resume deployment.v1.apps/nginx-deployment + kubectl rollout resume deployment/nginx-deployment ``` The output is similar to this: @@ -844,6 +850,13 @@ Kubernetes marks a Deployment as _progressing_ when one of the following tasks i * The Deployment is scaling down its older ReplicaSet(s). * New Pods become ready or available (ready for at least [MinReadySeconds](#min-ready-seconds)). +When the rollout becomes “progressing”, the Deployment controller adds a condition with the following +attributes to the Deployment's `.status.conditions`: + +* `type: Progressing` +* `status: "True"` +* `reason: NewReplicaSetCreated` | `reason: FoundNewReplicaSet` | `reason: ReplicaSetUpdated` + You can monitor the progress for a Deployment by using `kubectl rollout status`. ### Complete Deployment @@ -855,6 +868,17 @@ updates you've requested have been completed. * All of the replicas associated with the Deployment are available. * No old replicas for the Deployment are running. +When the rollout becomes “complete”, the Deployment controller sets a condition with the following +attributes to the Deployment's `.status.conditions`: + +* `type: Progressing` +* `status: "True"` +* `reason: NewReplicaSetAvailable` + +This `Progressing` condition will retain a status value of `"True"` until a new rollout +is initiated. The condition holds even when availability of replicas changes (which +does instead affect the `Available` condition). + You can check if a Deployment has completed by using `kubectl rollout status`. If the rollout completed successfully, `kubectl rollout status` returns a zero exit code. @@ -892,10 +916,10 @@ number of seconds the Deployment controller waits before indicating (in the Depl Deployment progress has stalled. The following `kubectl` command sets the spec with `progressDeadlineSeconds` to make the controller report -lack of progress for a Deployment after 10 minutes: +lack of progress of a rollout for a Deployment after 10 minutes: ```shell -kubectl patch deployment.v1.apps/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}' +kubectl patch deployment/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}' ``` The output is similar to this: ``` @@ -904,21 +928,24 @@ deployment.apps/nginx-deployment patched Once the deadline has been exceeded, the Deployment controller adds a DeploymentCondition with the following attributes to the Deployment's `.status.conditions`: -* Type=Progressing -* Status=False -* Reason=ProgressDeadlineExceeded +* `type: Progressing` +* `status: "False"` +* `reason: ProgressDeadlineExceeded` + +This condition can also fail early and is then set to status value of `"False"` due to reasons as `ReplicaSetCreateError`. +Also, the deadline is not taken into account anymore once the Deployment rollout completes. See the [Kubernetes API conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties) for more information on status conditions. {{< note >}} Kubernetes takes no action on a stalled Deployment other than to report a status condition with -`Reason=ProgressDeadlineExceeded`. Higher level orchestrators can take advantage of it and act accordingly, for +`reason: ProgressDeadlineExceeded`. Higher level orchestrators can take advantage of it and act accordingly, for example, rollback the Deployment to its previous version. {{< /note >}} {{< note >}} -If you pause a Deployment, Kubernetes does not check progress against your specified deadline. -You can safely pause a Deployment in the middle of a rollout and resume without triggering +If you pause a Deployment rollout, Kubernetes does not check progress against your specified deadline. +You can safely pause a Deployment rollout in the middle of a rollout and resume without triggering the condition for exceeding the deadline. {{< /note >}} @@ -986,7 +1013,7 @@ Conditions: You can address an issue of insufficient quota by scaling down your Deployment, by scaling down other controllers you may be running, or by increasing quota in your namespace. If you satisfy the quota conditions and the Deployment controller then completes the Deployment rollout, you'll see the -Deployment's status update with a successful condition (`Status=True` and `Reason=NewReplicaSetAvailable`). +Deployment's status update with a successful condition (`status: "True"` and `reason: NewReplicaSetAvailable`). ``` Conditions: @@ -996,11 +1023,11 @@ Conditions: Progressing True NewReplicaSetAvailable ``` -`Type=Available` with `Status=True` means that your Deployment has minimum availability. Minimum availability is dictated -by the parameters specified in the deployment strategy. `Type=Progressing` with `Status=True` means that your Deployment +`type: Available` with `status: "True"` means that your Deployment has minimum availability. Minimum availability is dictated +by the parameters specified in the deployment strategy. `type: Progressing` with `status: "True"` means that your Deployment is either in the middle of a rollout and it is progressing or that it has successfully completed its progress and the minimum required new replicas are available (see the Reason of the condition for the particulars - in our case -`Reason=NewReplicaSetAvailable` means that the Deployment is complete). +`reason: NewReplicaSetAvailable` means that the Deployment is complete). You can check if a Deployment has failed to progress by using `kubectl rollout status`. `kubectl rollout status` returns a non-zero exit code if the Deployment has exceeded the progression deadline. @@ -1056,7 +1083,7 @@ A Deployment also needs a [`.spec` section](https://git.k8s.io/community/contrib ### Pod Template -The `.spec.template` and `.spec.selector` are the only required field of the `.spec`. +The `.spec.template` and `.spec.selector` are the only required fields of the `.spec`. The `.spec.template` is a [Pod template](/docs/concepts/workloads/pods/#pod-templates). It has exactly the same schema as a {{< glossary_tooltip text="Pod" term_id="pod" >}}, except it is nested and does not have an `apiVersion` or `kind`. @@ -1070,6 +1097,18 @@ allowed, which is the default if not specified. `.spec.replicas` is an optional field that specifies the number of desired Pods. It defaults to 1. +Should you manually scale a Deployment, example via `kubectl scale deployment +deployment --replicas=X`, and then you update that Deployment based on a manifest +(for example: by running `kubectl apply -f deployment.yaml`), +then applying that manifest overwrites the manual scaling that you previously did. + +If a [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) (or any +similar API for horizontal scaling) is managing scaling for a Deployment, don't set `.spec.replicas`. + +Instead, allow the Kubernetes +{{< glossary_tooltip text="control plane" term_id="control-plane" >}} to manage the +`.spec.replicas` field automatically. + ### Selector `.spec.selector` is a required field that specifies a [label selector](/docs/concepts/overview/working-with-objects/labels/) @@ -1145,8 +1184,8 @@ total number of Pods running at any time during the update is at most 130% of de `.spec.progressDeadlineSeconds` is an optional field that specifies the number of seconds you want to wait for your Deployment to progress before the system reports back that the Deployment has -[failed progressing](#failed-deployment) - surfaced as a condition with `Type=Progressing`, `Status=False`. -and `Reason=ProgressDeadlineExceeded` in the status of the resource. The Deployment controller will keep +[failed progressing](#failed-deployment) - surfaced as a condition with `type: Progressing`, `status: "False"`. +and `reason: ProgressDeadlineExceeded` in the status of the resource. The Deployment controller will keep retrying the Deployment. This defaults to 600. In the future, once automatic rollback will be implemented, the Deployment controller will roll back a Deployment as soon as it observes such a condition. @@ -1176,4 +1215,12 @@ a paused Deployment and one that is not paused, is that any changes into the Pod Deployment will not trigger new rollouts as long as it is paused. A Deployment is not paused by default when it is created. +## {{% heading "whatsnext" %}} +* Learn about [Pods](/docs/concepts/workloads/pods). +* [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/). +* `Deployment` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/deployment-v1" >}} + object definition to understand the API for deployments. +* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how + you can use it to manage application availability during disruptions. diff --git a/content/en/docs/concepts/workloads/controllers/job.md b/content/en/docs/concepts/workloads/controllers/job.md index 1635caf7b3..cb1b72fd03 100644 --- a/content/en/docs/concepts/workloads/controllers/job.md +++ b/content/en/docs/concepts/workloads/controllers/job.md @@ -25,6 +25,9 @@ due to a node hardware failure or a node reboot). You can also use a Job to run multiple Pods in parallel. +If you want to run a Job (either a single task, or several in parallel) on a schedule, +see [CronJob](/docs/concepts/workloads/controllers/cron-jobs/). + <!-- body --> ## Running an example Job @@ -39,18 +42,17 @@ You can run the example with this command: ```shell kubectl apply -f https://kubernetes.io/examples/controllers/job.yaml ``` + The output is similar to this: + ``` job.batch/pi created ``` Check on the status of the Job with `kubectl`: -```shell -kubectl describe jobs/pi -``` -The output is similar to this: -``` +{{< tabs name="Check status of Job" >}} +{{< tab name="kubectl describe job pi" codelang="bash" >}} Name: pi Namespace: default Selector: controller-uid=c9948307-e56d-4b5d-8302-ae2d7b7da67c @@ -69,7 +71,7 @@ Pod Template: job-name=pi Containers: pi: - Image: perl + Image: perl:5.34.0 Port: <none> Host Port: <none> Command: @@ -84,7 +86,62 @@ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 14m job-controller Created pod: pi-5rwd7 -``` +{{< /tab >}} +{{< tab name="kubectl get job pi -o yaml" codelang="bash" >}} +apiVersion: batch/v1 +kind: Job +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"batch/v1","kind":"Job","metadata":{"annotations":{},"name":"pi","namespace":"default"},"spec":{"backoffLimit":4,"template":{"spec":{"containers":[{"command":["perl","-Mbignum=bpi","-wle","print bpi(2000)"],"image":"perl","name":"pi"}],"restartPolicy":"Never"}}}} + creationTimestamp: "2022-06-15T08:40:15Z" + generation: 1 + labels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + job-name: pi + name: pi + namespace: default + resourceVersion: "987" + uid: 863452e6-270d-420e-9b94-53a54146c223 +spec: + backoffLimit: 4 + completionMode: NonIndexed + completions: 1 + parallelism: 1 + selector: + matchLabels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + suspend: false + template: + metadata: + creationTimestamp: null + labels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + job-name: pi + spec: + containers: + - command: + - perl + - -Mbignum=bpi + - -wle + - print bpi(2000) + image: perl:5.34.0 + imagePullPolicy: Always + name: pi + resources: {} + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Never + schedulerName: default-scheduler + securityContext: {} + terminationGracePeriodSeconds: 30 +status: + active: 1 + ready: 1 + startTime: "2022-06-15T08:40:15Z" +{{< /tab >}} +{{< /tabs >}} To view completed Pods of a Job, use `kubectl get pods`. @@ -94,7 +151,9 @@ To list all the Pods that belong to a Job in a machine readable form, you can us pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath='{.items[*].metadata.name}') echo $pods ``` + The output is similar to this: + ``` pi-5rwd7 ``` @@ -107,8 +166,10 @@ View the standard output of one of the pods: ```shell kubectl logs $pods ``` + The output is similar to this: -```shell + +``` 3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901 ``` @@ -187,7 +248,7 @@ parallelism, for a variety of reasons: ### Completion mode -{{< feature-state for_k8s_version="v1.22" state="beta" >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} Jobs with _fixed completion count_ - that is, jobs that have non null `.spec.completions` - can have a completion mode that is specified in `.spec.completionMode`: @@ -203,7 +264,8 @@ Jobs with _fixed completion count_ - that is, jobs that have non null When you use an Indexed Job in combination with a {{< glossary_tooltip term_id="Service" >}}, Pods within the Job can use the deterministic hostnames to address each other via DNS. - - From the containarized task, in the environment variable `JOB_COMPLETION_INDEX`. + - From the containerized task, in the environment variable `JOB_COMPLETION_INDEX`. + The Job is considered complete when there is one successfully completed Pod for each index. For more information about how to use this mode, see [Indexed Job for Parallel Processing with Static Work Assignment](/docs/tasks/job/indexed-parallel-processing-static/). @@ -241,12 +303,22 @@ due to a logical error in configuration etc. To do so, set `.spec.backoffLimit` to specify the number of retries before considering a Job as failed. The back-off limit is set by default to 6. Failed Pods associated with the Job are recreated by the Job controller with an -exponential back-off delay (10s, 20s, 40s ...) capped at six minutes. The -back-off count is reset when a Job's Pod is deleted or successful without any -other Pods for the Job failing around that time. +exponential back-off delay (10s, 20s, 40s ...) capped at six minutes. + +The number of retries is calculated in two ways: +- The number of Pods with `.status.phase = "Failed"`. +- When using `restartPolicy = "OnFailure"`, the number of retries in all the + containers of Pods with `.status.phase` equal to `Pending` or `Running`. + +If either of the calculations reaches the `.spec.backoffLimit`, the Job is +considered failed. + +When the [`JobTrackingWithFinalizers`](#job-tracking-with-finalizers) feature is +disabled, the number of failed Pods is only based on Pods that are still present +in the API. {{< note >}} -If your job has `restartPolicy = "OnFailure"`, keep in mind that your container running the Job +If your job has `restartPolicy = "OnFailure"`, keep in mind that your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting `restartPolicy = "Never"` when debugging the Job or using a logging system to ensure output from failed Jobs is not lost inadvertently. @@ -284,7 +356,7 @@ spec: spec: containers: - name: pi - image: perl + image: perl:5.34.0 command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] restartPolicy: Never ``` @@ -304,7 +376,7 @@ cleaned up by CronJobs based on the specified capacity-based cleanup policy. ### TTL mechanism for finished Jobs -{{< feature-state for_k8s_version="v1.21" state="beta" >}} +{{< feature-state for_k8s_version="v1.23" state="stable" >}} Another way to clean up finished Jobs (either `Complete` or `Failed`) automatically is to use a TTL mechanism provided by a @@ -330,7 +402,7 @@ spec: spec: containers: - name: pi - image: perl + image: perl:5.34.0 command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] restartPolicy: Never ``` @@ -342,6 +414,25 @@ If the field is set to `0`, the Job will be eligible to be automatically deleted immediately after it finishes. If the field is unset, this Job won't be cleaned up by the TTL controller after it finishes. +{{< note >}} +It is recommended to set `ttlSecondsAfterFinished` field because unmanaged jobs +(Jobs that you created directly, and not indirectly through other workload APIs +such as CronJob) have a default deletion +policy of `orphanDependents` causing Pods created by an unmanaged Job to be left around +after that Job is fully deleted. +Even though the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} eventually +[garbage collects](/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection) +the Pods from a deleted Job after they either fail or complete, sometimes those +lingering pods may cause cluster performance degradation or in worst case cause the +cluster to go offline due to this degradation. + +You can use [LimitRanges](/docs/concepts/policy/limit-range/) and +[ResourceQuotas](/docs/concepts/policy/resource-quotas/) to place a +cap on the amount of resources that a particular namespace can +consume. +{{< /note >}} + + ## Job patterns The Job object can be used to support reliable parallel execution of Pods. The Job object is not @@ -374,7 +465,7 @@ The pattern names are also links to examples and more detailed description. | ----------------------------------------- |:-----------------:|:---------------------------:|:-------------------:| | [Queue with Pod Per Work Item] | ✓ | | sometimes | | [Queue with Variable Pod Count] | ✓ | ✓ | | -| [Indexed Job with Static Work Assignment] | ✓ | | ✓ | +| [Indexed Job with Static Work Assignment] | ✓ | | ✓ | | [Job Template Expansion] | | | ✓ | When you specify completions with `.spec.completions`, each Pod created by the Job controller @@ -402,18 +493,15 @@ Here, `W` is the number of work items. ### Suspending a Job -{{< feature-state for_k8s_version="v1.22" state="beta" >}} - -{{< note >}} -In Kubernetes version 1.21, this feature was in alpha, which required additional -steps to enable this feature; make sure to read the [right documentation for the -version of Kubernetes you're using](/docs/home/supported-doc-versions/). -{{< /note >}} +{{< feature-state for_k8s_version="v1.24" state="stable" >}} When a Job is created, the Job controller will immediately begin creating Pods to satisfy the Job's requirements and will continue to do so until the Job is complete. However, you may want to temporarily suspend a Job's execution and -resume it later. To suspend a Job, you can update the `.spec.suspend` field of +resume it later, or start Jobs in suspended state and have a custom controller +decide later when to start them. + +To suspend a Job, you can update the `.spec.suspend` field of the Job to true; later, when you want to resume it again, update it to false. Creating a Job with `.spec.suspend` set to true will create it in the suspended state. @@ -422,8 +510,7 @@ When a Job is resumed from suspension, its `.status.startTime` field will be reset to the current time. This means that the `.spec.activeDeadlineSeconds` timer will be stopped and reset when a Job is suspended and resumed. -Remember that suspending a Job will delete all active Pods. When the Job is -suspended, your [Pods will be terminated](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) +When you suspend a Job, any running Pods that don't have a status of `Completed` will be [terminated](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). with a SIGTERM signal. The Pod's graceful termination period will be honored and your Pod must handle this signal in this period. This may involve saving progress for later or undoing changes. Pods terminated this way will not count @@ -449,6 +536,20 @@ spec: ... ``` +You can also toggle Job suspension by patching the Job using the command line. + +Suspend an active Job: + +```shell +kubectl patch job/myjob --type=strategic --patch '{"spec":{"suspend":true}}' +``` + +Resume a suspended Job: + +```shell +kubectl patch job/myjob --type=strategic --patch '{"spec":{"suspend":false}}' +``` + The Job's status can be used to determine if a Job is suspended or has been suspended in the past: @@ -456,7 +557,7 @@ suspended in the past: kubectl get jobs/myjob -o yaml ``` -```json +```yaml apiVersion: batch/v1 kind: Job # .metadata and .spec omitted @@ -499,6 +600,32 @@ directly a result of toggling the `.spec.suspend` field. In the time between these two events, we see that no Pods were created, but Pod creation restarted as soon as the Job was resumed. +### Mutable Scheduling Directives + +{{< feature-state for_k8s_version="v1.23" state="beta" >}} + +{{< note >}} +In order to use this behavior, you must enable the `JobMutableNodeSchedulingDirectives` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/). +It is enabled by default. +{{< /note >}} + +In most cases a parallel job will want the pods to run with constraints, +like all in the same zone, or all either on GPU model x or y but not a mix of both. + +The [suspend](#suspending-a-job) field is the first step towards achieving those semantics. Suspend allows a +custom queue controller to decide when a job should start; However, once a job is unsuspended, +a custom queue controller has no influence on where the pods of a job will actually land. + +This feature allows updating a Job's scheduling directives before it starts, which gives custom queue +controllers the ability to influence pod placement while at the same time offloading actual +pod-to-node assignment to kube-scheduler. This is allowed only for suspended Jobs that have never +been unsuspended before. + +The fields in a Job's pod template that can be updated are node affinity, node selector, +tolerations, labels and annotations. + ### Specifying your own Pod selector Normally, when you create a Job object, you do not specify `.spec.selector`. @@ -523,13 +650,15 @@ to keep running, but you want the rest of the Pods it creates to use a different pod template and for the Job to have a new name. You cannot update the Job because these fields are not updatable. Therefore, you delete Job `old` but _leave its pods -running_, using `kubectl delete jobs/old --cascade=false`. +running_, using `kubectl delete jobs/old --cascade=orphan`. Before deleting it, you make a note of what selector it uses: ```shell kubectl get job old -o yaml ``` + The output is similar to this: + ```yaml kind: Job metadata: @@ -563,23 +692,23 @@ spec: ``` The new Job itself will have a different uid from `a8f3d00d-c6d2-11e5-9f87-42010af00002`. Setting -`manualSelector: true` tells the system to that you know what you are doing and to allow this +`manualSelector: true` tells the system that you know what you are doing and to allow this mismatch. ### Job tracking with finalizers -{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} {{< note >}} In order to use this behavior, you must enable the `JobTrackingWithFinalizers` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/) and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/). -It is disabled by default. When enabled, the control plane tracks new Jobs using the behavior described -below. Existing Jobs are unaffected. As a user, the only difference you would -see is that the control plane tracking of Job completion is more accurate. +below. Jobs created before the feature was enabled are unaffected. As a user, +the only difference you would see is that the control plane tracking of Job +completion is more accurate. {{< /note >}} When this feature isn't enabled, the Job {{< glossary_tooltip term_id="controller" >}} @@ -638,6 +767,19 @@ driver, and then cleans up. An advantage of this approach is that the overall process gets the completion guarantee of a Job object, but maintains complete control over what Pods are created and how work is assigned to them. -## Cron Jobs {#cron-jobs} +## {{% heading "whatsnext" %}} -You can use a [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/) to create a Job that will run at specified times/dates, similar to the Unix tool `cron`. +* Learn about [Pods](/docs/concepts/workloads/pods). +* Read about different ways of running Jobs: + * [Coarse Parallel Processing Using a Work Queue](/docs/tasks/job/coarse-parallel-processing-work-queue/) + * [Fine Parallel Processing Using a Work Queue](/docs/tasks/job/fine-parallel-processing-work-queue/) + * Use an [indexed Job for parallel processing with static work assignment](/docs/tasks/job/indexed-parallel-processing-static/) (beta) + * Create multiple Jobs based on a template: [Parallel Processing using Expansions](/docs/tasks/job/parallel-processing-expansion/) +* Follow the links within [Clean up finished jobs automatically](#clean-up-finished-jobs-automatically) + to learn more about how your cluster can clean up completed and / or failed tasks. +* `Job` is part of the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/job-v1" >}} + object definition to understand the API for jobs. +* Read about [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/), which you + can use to define a series of Jobs that will run based on a schedule, similar to + the UNIX tool `cron`. diff --git a/content/en/docs/concepts/workloads/controllers/replicaset.md b/content/en/docs/concepts/workloads/controllers/replicaset.md index e7b36d1b4a..a282b8455a 100644 --- a/content/en/docs/concepts/workloads/controllers/replicaset.md +++ b/content/en/docs/concepts/workloads/controllers/replicaset.md @@ -13,9 +13,6 @@ weight: 20 A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods. - - - <!-- body --> ## How a ReplicaSet works @@ -26,14 +23,14 @@ it should create to meet the number of replicas criteria. A ReplicaSet then fulf and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod template. -A ReplicaSet is linked to its Pods via the Pods' [metadata.ownerReferences](/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents) +A ReplicaSet is linked to its Pods via the Pods' [metadata.ownerReferences](/docs/concepts/architecture/garbage-collection/#owners-and-dependents) field, which specifies what resource the current object is owned by. All Pods acquired by a ReplicaSet have their owning ReplicaSet's identifying information within their ownerReferences field. It's through this link that the ReplicaSet knows of the state of the Pods it is maintaining and plans accordingly. -A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no OwnerReference or the -OwnerReference is not a {{< glossary_tooltip term_id="controller" >}} and it matches a ReplicaSet's selector, it will be immediately acquired by said -ReplicaSet. +A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no +OwnerReference or the OwnerReference is not a {{< glossary_tooltip term_id="controller" >}} and it +matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet. ## When to use a ReplicaSet @@ -78,7 +75,7 @@ kubectl describe rs/frontend And you will see output similar to: -```shell +``` Name: frontend Namespace: default Selector: tier=frontend @@ -130,7 +127,7 @@ kubectl get pods frontend-b2zdv -o yaml The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field: -```shell +```yaml apiVersion: v1 kind: Pod metadata: @@ -181,7 +178,7 @@ kubectl get pods The output shows that the new Pods are either already terminated, or in the process of being terminated: -```shell +``` NAME READY STATUS RESTARTS AGE frontend-b2zdv 1/1 Running 0 10m frontend-vcmts 1/1 Running 0 10m @@ -210,7 +207,7 @@ kubectl get pods ``` Will reveal in its output: -```shell +``` NAME READY STATUS RESTARTS AGE frontend-hmmj2 1/1 Running 0 9s pod1 1/1 Running 0 36s @@ -223,8 +220,6 @@ In this manner, a ReplicaSet can own a non-homogenous set of Pods As with all other Kubernetes API objects, a ReplicaSet needs the `apiVersion`, `kind`, and `metadata` fields. For ReplicaSets, the `kind` is always a ReplicaSet. -In Kubernetes 1.9 the API version `apps/v1` on the ReplicaSet kind is the current version and is enabled by default. The API version `apps/v1beta2` is deprecated. -Refer to the first lines of the `frontend.yaml` example for guidance. The name of a ReplicaSet object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). @@ -255,7 +250,9 @@ In the ReplicaSet, `.spec.template.metadata.labels` must match `spec.selector`, be rejected by the API. {{< note >}} -For 2 ReplicaSets specifying the same `.spec.selector` but different `.spec.template.metadata.labels` and `.spec.template.spec` fields, each ReplicaSet ignores the Pods created by the other ReplicaSet. +For 2 ReplicaSets specifying the same `.spec.selector` but different +`.spec.template.metadata.labels` and `.spec.template.spec` fields, each ReplicaSet ignores the +Pods created by the other ReplicaSet. {{< /note >}} ### Replicas @@ -269,11 +266,14 @@ If you do not specify `.spec.replicas`, then it defaults to 1. ### Deleting a ReplicaSet and its Pods -To delete a ReplicaSet and all of its Pods, use [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). The [Garbage collector](/docs/concepts/workloads/controllers/garbage-collection/) automatically deletes all of the dependent Pods by default. +To delete a ReplicaSet and all of its Pods, use +[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). The +[Garbage collector](/docs/concepts/architecture/garbage-collection/) automatically deletes all of +the dependent Pods by default. + +When using the REST API or the `client-go` library, you must set `propagationPolicy` to +`Background` or `Foreground` in the `-d` option. For example: -When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Background` or `Foreground` in -the -d option. -For example: ```shell kubectl proxy --port=8080 curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/frontend' \ @@ -283,9 +283,12 @@ curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/fron ### Deleting just a ReplicaSet -You can delete a ReplicaSet without affecting any of its Pods using [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete) with the `--cascade=orphan` option. +You can delete a ReplicaSet without affecting any of its Pods using +[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete) +with the `--cascade=orphan` option. When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Orphan`. For example: + ```shell kubectl proxy --port=8080 curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/frontend' \ @@ -297,7 +300,8 @@ Once the original is deleted, you can create a new ReplicaSet to replace it. As as the old and new `.spec.selector` are the same, then the new one will adopt the old Pods. However, it will not make any effort to make existing Pods match a new, different pod template. To update Pods to a new spec in a controlled way, use a -[Deployment](/docs/concepts/workloads/controllers/deployment/#creating-a-deployment), as ReplicaSets do not support a rolling update directly. +[Deployment](/docs/concepts/workloads/controllers/deployment/#creating-a-deployment), as +ReplicaSets do not support a rolling update directly. ### Isolating Pods from a ReplicaSet @@ -312,17 +316,19 @@ ensures that a desired number of Pods with a matching label selector are availab When scaling down, the ReplicaSet controller chooses which pods to delete by sorting the available pods to prioritize scaling down pods based on the following general algorithm: - 1. Pending (and unschedulable) pods are scaled down first - 2. If controller.kubernetes.io/pod-deletion-cost annotation is set, then - the pod with the lower value will come first. - 3. Pods on nodes with more replicas come before pods on nodes with fewer replicas. - 4. If the pods' creation times differ, the pod that was created more recently - comes before the older pod (the creation times are bucketed on an integer log scale - when the `LogarithmicScaleDown` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) + +1. Pending (and unschedulable) pods are scaled down first +1. If `controller.kubernetes.io/pod-deletion-cost` annotation is set, then + the pod with the lower value will come first. +1. Pods on nodes with more replicas come before pods on nodes with fewer replicas. +1. If the pods' creation times differ, the pod that was created more recently + comes before the older pod (the creation times are bucketed on an integer log scale + when the `LogarithmicScaleDown` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) If all of the above match, then selection is random. ### Pod deletion cost + {{< feature-state for_k8s_version="v1.22" state="beta" >}} Using the [`controller.kubernetes.io/pod-deletion-cost`](/docs/reference/labels-annotations-taints/#pod-deletion-cost) @@ -346,6 +352,7 @@ This feature is beta and enabled by default. You can disable it using the {{< /note >}} #### Example Use Case + The different pods of an application could have different utilization levels. On scale down, the application may prefer to remove the pods with lower utilization. To avoid frequently updating the pods, the application should update `controller.kubernetes.io/pod-deletion-cost` once before issuing a scale down (setting the @@ -389,12 +396,17 @@ As such, it is recommended to use Deployments when you want ReplicaSets. ### Bare Pods -Unlike the case where a user directly created Pods, a ReplicaSet replaces Pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single Pod. Think of it similarly to a process supervisor, only it supervises multiple Pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node (for example, Kubelet or Docker). +Unlike the case where a user directly created Pods, a ReplicaSet replaces Pods that are deleted or +terminated for any reason, such as in the case of node failure or disruptive node maintenance, +such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your +application requires only a single Pod. Think of it similarly to a process supervisor, only it +supervises multiple Pods across multiple nodes instead of individual processes on a single node. A +ReplicaSet delegates local container restarts to some agent on the node such as Kubelet. ### Job -Use a [`Job`](/docs/concepts/workloads/controllers/job/) instead of a ReplicaSet for Pods that are expected to terminate on their own -(that is, batch jobs). +Use a [`Job`](/docs/concepts/workloads/controllers/job/) instead of a ReplicaSet for Pods that are +expected to terminate on their own (that is, batch jobs). ### DaemonSet @@ -404,9 +416,21 @@ to a machine lifetime: the Pod needs to be running on the machine before other P safe to terminate when the machine is otherwise ready to be rebooted/shutdown. ### ReplicationController -ReplicaSets are the successors to [_ReplicationControllers_](/docs/concepts/workloads/controllers/replicationcontroller/). + +ReplicaSets are the successors to [ReplicationControllers](/docs/concepts/workloads/controllers/replicationcontroller/). The two serve the same purpose, and behave similarly, except that a ReplicationController does not support set-based selector requirements as described in the [labels user guide](/docs/concepts/overview/working-with-objects/labels/#label-selectors). As such, ReplicaSets are preferred over ReplicationControllers +## {{% heading "whatsnext" %}} + +* Learn about [Pods](/docs/concepts/workloads/pods). +* Learn about [Deployments](/docs/concepts/workloads/controllers/deployment/). +* [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/), + which relies on ReplicaSets to work. +* `ReplicaSet` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/replica-set-v1" >}} + object definition to understand the API for replica sets. +* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how + you can use it to manage application availability during disruptions. diff --git a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md index 2b06539fd6..b55db484e0 100644 --- a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md +++ b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md @@ -185,16 +185,16 @@ delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). Kubectl wi for it to delete each pod before deleting the ReplicationController itself. If this kubectl command is interrupted, it can be restarted. -When using the REST API or Go client library, you need to do the steps explicitly (scale replicas to +When using the REST API or [client library](/docs/reference/using-api/client-libraries), you need to do the steps explicitly (scale replicas to 0, wait for pod deletions, then delete the ReplicationController). ### Deleting only a ReplicationController You can delete a ReplicationController without affecting any of its pods. -Using kubectl, specify the `--cascade=false` option to [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). +Using kubectl, specify the `--cascade=orphan` option to [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). -When using the REST API or Go client library, you can delete the ReplicationController object. +When using the REST API or [client library](/docs/reference/using-api/client-libraries), you can delete the ReplicationController object. Once the original is deleted, you can create a new ReplicationController to replace it. As long as the old and new `.spec.selector` are the same, then the new one will adopt the old pods. @@ -248,7 +248,7 @@ The ReplicationController ensures that the desired number of pods matches its la The ReplicationController is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://issue.k8s.io/492)), which would change its `replicas` field. We will not add scheduling policies (for example, [spreading](https://issue.k8s.io/367#issuecomment-48428019)) to the ReplicationController. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsewhere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://issue.k8s.io/170)). -The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc. +The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://netflixtechblog.com/asgard-web-based-cloud-management-and-deployment-2c9fc4e4d3a1) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc. ## API Object @@ -266,7 +266,7 @@ Note that we recommend using Deployments instead of directly using Replica Sets, ### Deployment (Recommended) -[`Deployment`](/docs/concepts/workloads/controllers/deployment/) is a higher-level API object that updates its underlying Replica Sets and their Pods. Deployments are recommended if you want this rolling update functionality because, they are declarative, server-side, and have additional features. +[`Deployment`](/docs/concepts/workloads/controllers/deployment/) is a higher-level API object that updates its underlying Replica Sets and their Pods. Deployments are recommended if you want the rolling update functionality, because they are declarative, server-side, and have additional features. ### Bare Pods @@ -284,6 +284,11 @@ machine-level function, such as machine monitoring or machine logging. These po to a machine lifetime: the pod needs to be running on the machine before other pods start, and are safe to terminate when the machine is otherwise ready to be rebooted/shutdown. -## For more information +## {{% heading "whatsnext" %}} -Read [Run Stateless Application Deployment](/docs/tasks/run-application/run-stateless-application-deployment/). +* Learn about [Pods](/docs/concepts/workloads/pods). +* Learn about [Deployment](/docs/concepts/workloads/controllers/deployment/), the replacement + for ReplicationController. +* `ReplicationController` is part of the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/replication-controller-v1" >}} + object definition to understand the API for replication controllers. diff --git a/content/en/docs/concepts/workloads/controllers/statefulset.md b/content/en/docs/concepts/workloads/controllers/statefulset.md index 289722fb07..1687399abd 100644 --- a/content/en/docs/concepts/workloads/controllers/statefulset.md +++ b/content/en/docs/concepts/workloads/controllers/statefulset.md @@ -39,10 +39,18 @@ that provides a set of stateless replicas. ## Limitations -* The storage for a given Pod must either be provisioned by a [PersistentVolume Provisioner](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/README.md) based on the requested `storage class`, or pre-provisioned by an admin. -* Deleting and/or scaling a StatefulSet down will *not* delete the volumes associated with the StatefulSet. This is done to ensure data safety, which is generally more valuable than an automatic purge of all related StatefulSet resources. -* StatefulSets currently require a [Headless Service](/docs/concepts/services-networking/service/#headless-services) to be responsible for the network identity of the Pods. You are responsible for creating this Service. -* StatefulSets do not provide any guarantees on the termination of pods when a StatefulSet is deleted. To achieve ordered and graceful termination of the pods in the StatefulSet, it is possible to scale the StatefulSet down to 0 prior to deletion. +* The storage for a given Pod must either be provisioned by a + [PersistentVolume Provisioner](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/README.md) + based on the requested `storage class`, or pre-provisioned by an admin. +* Deleting and/or scaling a StatefulSet down will *not* delete the volumes associated with the + StatefulSet. This is done to ensure data safety, which is generally more valuable than an + automatic purge of all related StatefulSet resources. +* StatefulSets currently require a [Headless Service](/docs/concepts/services-networking/service/#headless-services) + to be responsible for the network identity of the Pods. You are responsible for creating this + Service. +* StatefulSets do not provide any guarantees on the termination of pods when a StatefulSet is + deleted. To achieve ordered and graceful termination of the pods in the StatefulSet, it is + possible to scale the StatefulSet down to 0 prior to deletion. * When using [Rolling Updates](#rolling-updates) with the default [Pod Management Policy](#pod-management-policies) (`OrderedReady`), it's possible to get into a broken state that requires @@ -77,6 +85,7 @@ spec: app: nginx # has to match .spec.template.metadata.labels serviceName: "nginx" replicas: 3 # by default is 1 + minReadySeconds: 10 # by default is 0 template: metadata: labels: @@ -107,14 +116,37 @@ In the above example: * A Headless Service, named `nginx`, is used to control the network domain. * The StatefulSet, named `web`, has a Spec that indicates that 3 replicas of the nginx container will be launched in unique Pods. -* The `volumeClaimTemplates` will provide stable storage using [PersistentVolumes](/docs/concepts/storage/persistent-volumes/) provisioned by a PersistentVolume Provisioner. +* The `volumeClaimTemplates` will provide stable storage using + [PersistentVolumes](/docs/concepts/storage/persistent-volumes/) provisioned by a + PersistentVolume Provisioner. The name of a StatefulSet object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). -## Pod Selector +### Pod Selector -You must set the `.spec.selector` field of a StatefulSet to match the labels of its `.spec.template.metadata.labels`. Prior to Kubernetes 1.8, the `.spec.selector` field was defaulted when omitted. In 1.8 and later versions, failing to specify a matching Pod Selector will result in a validation error during StatefulSet creation. +You must set the `.spec.selector` field of a StatefulSet to match the labels of its +`.spec.template.metadata.labels`. Failing to specify a matching Pod Selector will result in a +validation error during StatefulSet creation. + +### Volume Claim Templates + +You can set the `.spec.volumeClaimTemplates` which can provide stable storage using +[PersistentVolumes](/docs/concepts/storage/persistent-volumes/) provisioned by a PersistentVolume +Provisioner. + + +### Minimum ready seconds + +{{< feature-state for_k8s_version="v1.23" state="beta" >}} + +`.spec.minReadySeconds` is an optional field that specifies the minimum number of seconds for which a newly +created Pod should be ready without any of its containers crashing, for it to be considered available. +Please note that this feature is beta and enabled by default. Please opt out by unsetting the +StatefulSetMinReadySeconds flag, if you don't +want this feature to be enabled. This field defaults to 0 (the Pod will be considered +available as soon as it is ready). To learn more about when a Pod is considered ready, see +[Container Probes](/docs/concepts/workloads/pods/pod-lifecycle/#container-probes). ## Pod Identity @@ -150,8 +182,8 @@ remembered and reused, even after the Pod is running, for at least a few seconds If you need to discover Pods promptly after they are created, you have a few options: - Query the Kubernetes API directly (for example, using a watch) rather than relying on DNS lookups. -- Decrease the time of caching in your Kubernetes DNS provider (typically this means editing the config map for CoreDNS, which currently caches for 30 seconds). - +- Decrease the time of caching in your Kubernetes DNS provider (typically this means editing the + config map for CoreDNS, which currently caches for 30 seconds). As mentioned in the [limitations](#limitations) section, you are responsible for creating the [Headless Service](/docs/concepts/services-networking/service/#headless-services) @@ -173,7 +205,9 @@ Cluster Domain will be set to `cluster.local` unless ### Stable Storage -For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one PersistentVolumeClaim. In the nginx example above, each Podreceives a single PersistentVolume with a StorageClass of `my-storage-class` and 1 Gib of provisioned storage. If no StorageClass +For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one +PersistentVolumeClaim. In the nginx example above, each Pod receives a single PersistentVolume +with a StorageClass of `my-storage-class` and 1 Gib of provisioned storage. If no StorageClass is specified, then the default StorageClass will be used. When a Pod is (re)scheduled onto a node, its `volumeMounts` mount the PersistentVolumes associated with its PersistentVolume Claims. Note that, the PersistentVolumes associated with the @@ -194,7 +228,9 @@ the StatefulSet. * Before a scaling operation is applied to a Pod, all of its predecessors must be Running and Ready. * Before a Pod is terminated, all of its successors must be completely shutdown. -The StatefulSet should not specify a `pod.Spec.TerminationGracePeriodSeconds` of 0. This practice is unsafe and strongly discouraged. For further explanation, please refer to [force deleting StatefulSet Pods](/docs/tasks/run-application/force-delete-stateful-set-pod/). +The StatefulSet should not specify a `pod.Spec.TerminationGracePeriodSeconds` of 0. This practice +is unsafe and strongly discouraged. For further explanation, please refer to +[force deleting StatefulSet Pods](/docs/tasks/run-application/force-delete-stateful-set-pod/). When the nginx example above is created, three Pods will be deployed in the order web-0, web-1, web-2. web-1 will not be deployed before web-0 is @@ -210,7 +246,7 @@ is completely shutdown, but prior to web-1's termination, web-1 would not be ter until web-0 is Running and Ready. ### Pod Management Policies -In Kubernetes 1.7 and later, StatefulSet allows you to relax its ordering guarantees while +StatefulSet allows you to relax its ordering guarantees while preserving its uniqueness and identity guarantees via its `.spec.podManagementPolicy` field. #### OrderedReady Pod Management @@ -240,7 +276,8 @@ annotations for the Pods in a StatefulSet. There are two possible values: create new Pods that reflect modifications made to a StatefulSet's `.spec.template`. `RollingUpdate` -: The `RollingUpdate` update strategy implements automated, rolling update for the Pods in a StatefulSet. This is the default update strategy. +: The `RollingUpdate` update strategy implements automated, rolling update for the Pods in a + StatefulSet. This is the default update strategy. ## Rolling Updates @@ -250,7 +287,9 @@ in the same order as Pod termination (from the largest ordinal to the smallest), each Pod one at a time. The Kubernetes control plane waits until an updated Pod is Running and Ready prior -to updating its predecessor. If you have set `.spec.minReadySeconds` (see [Minimum Ready Seconds](#minimum-ready-seconds)), the control plane additionally waits that amount of time after the Pod turns ready, before moving on. +to updating its predecessor. If you have set `.spec.minReadySeconds` (see +[Minimum Ready Seconds](#minimum-ready-seconds)), the control plane additionally waits that +amount of time after the Pod turns ready, before moving on. ### Partitioned rolling updates {#partitions} @@ -264,6 +303,27 @@ updates to its `.spec.template` will not be propagated to its Pods. In most cases you will not need to use a partition, but they are useful if you want to stage an update, roll out a canary, or perform a phased roll out. +### Maximum unavailable Pods + +{{< feature-state for_k8s_version="v1.24" state="alpha" >}} + +You can control the maximum number of Pods that can be unavailable during an update +by specifying the `.spec.updateStrategy.rollingUpdate.maxUnavailable` field. +The value can be an absolute number (for example, `5`) or a percentage of desired +Pods (for example, `10%`). Absolute number is calculated from the percentage value +by rounding it up. This field cannot be 0. The default setting is 1. + +This field applies to all Pods in the range `0` to `replicas - 1`. If there is any +unavailable Pod in the range `0` to `replicas - 1`, it will be counted towards +`maxUnavailable`. + +{{< note >}} +The `maxUnavailable` field is in Alpha stage and it is honored only by API servers +that are running with the `MaxUnavailableStatefulSet` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +enabled. +{{< /note >}} + ### Forced rollback When using [Rolling Updates](#rolling-updates) with the default @@ -284,21 +344,115 @@ After reverting the template, you must also delete any Pods that StatefulSet had already attempted to run with the bad configuration. StatefulSet will then begin to recreate the Pods using the reverted template. -### Minimum ready seconds -{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +## PersistentVolumeClaim retention -`.spec.minReadySeconds` is an optional field that specifies the minimum number of seconds for which a newly -created Pod should be ready without any of its containers crashing, for it to be considered available. -This defaults to 0 (the Pod will be considered available as soon as it is ready). To learn more about when -a Pod is considered ready, see [Container Probes](/docs/concepts/workloads/pods/pod-lifecycle/#container-probes). +{{< feature-state for_k8s_version="v1.23" state="alpha" >}} -Please note that this field only works if you enable the `StatefulSetMinReadySeconds` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +The optional `.spec.persistentVolumeClaimRetentionPolicy` field controls if +and how PVCs are deleted during the lifecycle of a StatefulSet. You must enable the +`StatefulSetAutoDeletePVC` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +to use this field. Once enabled, there are two policies you can configure for each +StatefulSet: + +`whenDeleted` +: configures the volume retention behavior that applies when the StatefulSet is deleted + +`whenScaled` +: configures the volume retention behavior that applies when the replica count of + the StatefulSet is reduced; for example, when scaling down the set. + +For each policy that you can configure, you can set the value to either `Delete` or `Retain`. + +`Delete` +: The PVCs created from the StatefulSet `volumeClaimTemplate` are deleted for each Pod + affected by the policy. With the `whenDeleted` policy all PVCs from the + `volumeClaimTemplate` are deleted after their Pods have been deleted. With the + `whenScaled` policy, only PVCs corresponding to Pod replicas being scaled down are + deleted, after their Pods have been deleted. + +`Retain` (default) +: PVCs from the `volumeClaimTemplate` are not affected when their Pod is + deleted. This is the behavior before this new feature. + +Bear in mind that these policies **only** apply when Pods are being removed due to the +StatefulSet being deleted or scaled down. For example, if a Pod associated with a StatefulSet +fails due to node failure, and the control plane creates a replacement Pod, the StatefulSet +retains the existing PVC. The existing volume is unaffected, and the cluster will attach it to +the node where the new Pod is about to launch. + +The default for policies is `Retain`, matching the StatefulSet behavior before this new feature. + +Here is an example policy. + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +... +spec: + persistentVolumeClaimRetentionPolicy: + whenDeleted: Retain + whenScaled: Delete +... +``` + +The StatefulSet {{<glossary_tooltip text="controller" term_id="controller">}} adds +[owner references](/docs/concepts/overview/working-with-objects/owners-dependents/#owner-references-in-object-specifications) +to its PVCs, which are then deleted by the {{<glossary_tooltip text="garbage collector" +term_id="garbage-collection">}} after the Pod is terminated. This enables the Pod to +cleanly unmount all volumes before the PVCs are deleted (and before the backing PV and +volume are deleted, depending on the retain policy). When you set the `whenDeleted` +policy to `Delete`, an owner reference to the StatefulSet instance is placed on all PVCs +associated with that StatefulSet. + +The `whenScaled` policy must delete PVCs only when a Pod is scaled down, and not when a +Pod is deleted for another reason. When reconciling, the StatefulSet controller compares +its desired replica count to the actual Pods present on the cluster. Any StatefulSet Pod +whose id greater than the replica count is condemned and marked for deletion. If the +`whenScaled` policy is `Delete`, the condemned Pods are first set as owners to the +associated StatefulSet template PVCs, before the Pod is deleted. This causes the PVCs +to be garbage collected after only the condemned Pods have terminated. + +This means that if the controller crashes and restarts, no Pod will be deleted before its +owner reference has been updated appropriate to the policy. If a condemned Pod is +force-deleted while the controller is down, the owner reference may or may not have been +set up, depending on when the controller crashed. It may take several reconcile loops to +update the owner references, so some condemned Pods may have set up owner references and +other may not. For this reason we recommend waiting for the controller to come back up, +which will verify owner references before terminating Pods. If that is not possible, the +operator should verify the owner references on PVCs to ensure the expected objects are +deleted when Pods are force-deleted. + +### Replicas + +`.spec.replicas` is an optional field that specifies the number of desired Pods. It defaults to 1. + +Should you manually scale a deployment, example via `kubectl scale +statefulset statefulset --replicas=X`, and then you update that StatefulSet +based on a manifest (for example: by running `kubectl apply -f +statefulset.yaml`), then applying that manifest overwrites the manual scaling +that you previously did. + +If a [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) +(or any similar API for horizontal scaling) is managing scaling for a +Statefulset, don't set `.spec.replicas`. Instead, allow the Kubernetes +{{<glossary_tooltip text="control plane" term_id="control-plane" >}} to manage +the `.spec.replicas` field automatically. ## {{% heading "whatsnext" %}} - -* Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/). -* Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/). -* Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/). +* Learn about [Pods](/docs/concepts/workloads/pods). +* Find out how to use StatefulSets + * Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/). + * Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/). + * Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/). + * Learn how to [scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/). + * Learn what's involved when you [delete a StatefulSet](/docs/tasks/run-application/delete-stateful-set/). + * Learn how to [configure a Pod to use a volume for storage](/docs/tasks/configure-pod-container/configure-volume-storage/). + * Learn how to [configure a Pod to use a PersistentVolume for storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/). +* `StatefulSet` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/stateful-set-v1" >}} + object definition to understand the API for stateful sets. +* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how + you can use it to manage application availability during disruptions. diff --git a/content/en/docs/concepts/workloads/controllers/ttlafterfinished.md b/content/en/docs/concepts/workloads/controllers/ttlafterfinished.md index 266e72a79f..a51c88602f 100644 --- a/content/en/docs/concepts/workloads/controllers/ttlafterfinished.md +++ b/content/en/docs/concepts/workloads/controllers/ttlafterfinished.md @@ -1,75 +1,68 @@ --- reviewers: - janetkuo -title: TTL Controller for Finished Resources +title: Automatic Clean-up for Finished Jobs content_type: concept weight: 70 --- <!-- overview --> -{{< feature-state for_k8s_version="v1.21" state="beta" >}} +{{< feature-state for_k8s_version="v1.23" state="stable" >}} -The TTL controller provides a TTL (time to live) mechanism to limit the lifetime of resource -objects that have finished execution. TTL controller only handles -{{< glossary_tooltip text="Jobs" term_id="job" >}} for now, -and may be expanded to handle other resources that will finish execution, -such as Pods and custom resources. - -This feature is currently beta and enabled by default, and can be disabled via -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`TTLAfterFinished` in both kube-apiserver and kube-controller-manager. +TTL-after-finished {{<glossary_tooltip text="controller" term_id="controller">}} provides a +TTL (time to live) mechanism to limit the lifetime of resource objects that +have finished execution. TTL controller only handles +{{< glossary_tooltip text="Jobs" term_id="job" >}}. <!-- body --> -## TTL Controller +## TTL-after-finished Controller -The TTL controller only supports Jobs for now. A cluster operator can use this feature to clean +The TTL-after-finished controller is only supported for Jobs. A cluster operator can use this feature to clean up finished Jobs (either `Complete` or `Failed`) automatically by specifying the `.spec.ttlSecondsAfterFinished` field of a Job, as in this [example](/docs/concepts/workloads/controllers/job/#clean-up-finished-jobs-automatically). -The TTL controller will assume that a resource is eligible to be cleaned up -TTL seconds after the resource has finished, in other words, when the TTL has expired. When the -TTL controller cleans up a resource, it will delete it cascadingly, that is to say it will delete -its dependent objects together with it. Note that when the resource is deleted, +The TTL-after-finished controller will assume that a job is eligible to be cleaned up +TTL seconds after the job has finished, in other words, when the TTL has expired. When the +TTL-after-finished controller cleans up a job, it will delete it cascadingly, that is to say it will delete +its dependent objects together with it. Note that when the job is deleted, its lifecycle guarantees, such as finalizers, will be honored. The TTL seconds can be set at any time. Here are some examples for setting the `.spec.ttlSecondsAfterFinished` field of a Job: -* Specify this field in the resource manifest, so that a Job can be cleaned up +* Specify this field in the job manifest, so that a Job can be cleaned up automatically some time after it finishes. -* Set this field of existing, already finished resources, to adopt this new +* Set this field of existing, already finished jobs, to adopt this new feature. * Use a [mutating admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) - to set this field dynamically at resource creation time. Cluster administrators can - use this to enforce a TTL policy for finished resources. + to set this field dynamically at job creation time. Cluster administrators can + use this to enforce a TTL policy for finished jobs. * Use a [mutating admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) - to set this field dynamically after the resource has finished, and choose - different TTL values based on resource status, labels, etc. + to set this field dynamically after the job has finished, and choose + different TTL values based on job status, labels, etc. ## Caveat ### Updating TTL Seconds Note that the TTL period, e.g. `.spec.ttlSecondsAfterFinished` field of Jobs, -can be modified after the resource is created or has finished. However, once the +can be modified after the job is created or has finished. However, once the Job becomes eligible to be deleted (when the TTL has expired), the system won't guarantee that the Jobs will be kept, even if an update to extend the TTL returns a successful API response. ### Time Skew -Because TTL controller uses timestamps stored in the Kubernetes resources to +Because TTL-after-finished controller uses timestamps stored in the Kubernetes jobs to determine whether the TTL has expired or not, this feature is sensitive to time -skew in the cluster, which may cause TTL controller to clean up resource objects +skew in the cluster, which may cause TTL-after-finish controller to clean up job objects at the wrong time. -In Kubernetes, it's required to run NTP on all nodes -(see [#6159](https://github.com/kubernetes/kubernetes/issues/6159#issuecomment-93844058)) -to avoid time skew. Clocks aren't always correct, but the difference should be +Clocks aren't always correct, but the difference should be very small. Please be aware of this risk when setting a non-zero TTL. diff --git a/content/en/docs/concepts/workloads/pods/_index.md b/content/en/docs/concepts/workloads/pods/_index.md index 7132d19529..77994f754f 100644 --- a/content/en/docs/concepts/workloads/pods/_index.md +++ b/content/en/docs/concepts/workloads/pods/_index.md @@ -48,6 +48,21 @@ with shared namespaces and shared filesystem volumes. ## Using Pods +The following is an example of a Pod which consists of a container running the image `nginx:1.14.2`. + +{{< codenew file="pods/simple-pod.yaml" >}} + +To create the Pod shown above, run the following command: +```shell +kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml +``` + +Pods are generally not created directly and are created using workload resources. +See [Working with Pods](#working-with-pods) for more information on how Pods are used +with workload resources. + +### Workload resources for managing pods + Usually you don't need to create Pods directly, even singleton Pods. Instead, create them using workload resources such as {{< glossary_tooltip text="Deployment" term_id="deployment" >}} or {{< glossary_tooltip text="Job" term_id="job" >}}. If your Pods need to track state, consider the @@ -97,7 +112,7 @@ For example, you might have a container that acts as a web server for files in a shared volume, and a separate "sidecar" container that updates those files from a remote source, as in the following diagram: -{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}} +{{< figure src="/images/docs/pod.svg" alt="Pod creation diagram" class="diagram-medium" >}} Some Pods have {{< glossary_tooltip text="init containers" term_id="init-container" >}} as well as {{< glossary_tooltip text="app containers" term_id="app-container" >}}. Init containers run and complete before the app containers are started. @@ -165,7 +180,7 @@ spec: spec: containers: - name: hello - image: busybox + image: busybox:1.28 command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] restartPolicy: OnFailure # The pod template ends here @@ -246,8 +261,7 @@ Within a Pod, containers share an IP address and port space, and can find each other via `localhost`. The containers in a Pod can also communicate with each other using standard inter-process communications like SystemV semaphores or POSIX shared memory. Containers in different Pods have distinct IP addresses -and can not communicate by IPC without -[special configuration](/docs/concepts/policy/pod-security-policy/). +and can not communicate by OS-level IPC without special configuration. Containers that want to interact with a container running in a different Pod can use IP networking to communicate. @@ -306,12 +320,12 @@ in the Pod Lifecycle documentation. * Learn about the [lifecycle of a Pod](/docs/concepts/workloads/pods/pod-lifecycle/). * Learn about [RuntimeClass](/docs/concepts/containers/runtime-class/) and how you can use it to configure different Pods with different container runtime configurations. -* Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). * Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how you can use it to manage application availability during disruptions. * Pod is a top-level resource in the Kubernetes REST API. - The [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) + The {{< api-reference page="workload-resources/pod-v1" >}} object definition describes the object in detail. -* [The Distributed System Toolkit: Patterns for Composite Containers](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns) explains common layouts for Pods with more than one container. +* [The Distributed System Toolkit: Patterns for Composite Containers](/blog/2015/06/the-distributed-system-toolkit-patterns/) explains common layouts for Pods with more than one container. +* Read about [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) To understand the context for why Kubernetes wraps a common Pod API in other resources (such as {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} or {{< glossary_tooltip text="Deployments" term_id="deployment" >}}), you can read about the prior art, including: diff --git a/content/en/docs/concepts/workloads/pods/disruptions.md b/content/en/docs/concepts/workloads/pods/disruptions.md index 430d4b7155..055fc0a65d 100644 --- a/content/en/docs/concepts/workloads/pods/disruptions.md +++ b/content/en/docs/concepts/workloads/pods/disruptions.md @@ -128,8 +128,8 @@ The "intended" number of pods is computed from the `.spec.replicas` of the workl that is managing those pods. The control plane discovers the owning workload resource by examining the `.metadata.ownerReferences` of the Pod. -PDBs cannot prevent [involuntary disruptions](#voluntary-and-involuntary-disruptions) from -occurring, but they do count against the budget. +[Involuntary disruptions](#voluntary-and-involuntary-disruptions) cannot be prevented by PDBs; however they +do count against the budget. Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) diff --git a/content/en/docs/concepts/workloads/pods/downward-api.md b/content/en/docs/concepts/workloads/pods/downward-api.md new file mode 100644 index 0000000000..fcee383c45 --- /dev/null +++ b/content/en/docs/concepts/workloads/pods/downward-api.md @@ -0,0 +1,131 @@ +--- +title: Downward API +content_type: concept +description: > + There are two ways to expose Pod and container fields to a running container: + environment variables, and as files that are populated by a special volume type. + Together, these two ways of exposing Pod and container fields are called the downward API. +--- + +<!-- overview --> + +It is sometimes useful for a container to have information about itself, without +being overly coupled to Kubernetes. The _downward API_ allows containers to consume +information about themselves or the cluster without using the Kubernetes client +or API server. + +An example is an existing application that assumes a particular well-known +environment variable holds a unique identifier. One possibility is to wrap the +application, but that is tedious and error-prone, and it violates the goal of low +coupling. A better option would be to use the Pod's name as an identifier, and +inject the Pod's name into the well-known environment variable. + +In Kubernetes, there are two ways to expose Pod and container fields to a running container: + +* as [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) +* as [files in a `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) + +Together, these two ways of exposing Pod and container fields are called the +_downward API_. + +<!-- body --> + +## Available fields + +Only some Kubernetes API fields are available through the downward API. This +section lists which fields you can make available. + +You can pass information from available Pod-level fields using `fieldRef`. +At the API level, the `spec` for a Pod always defines at least one +[Container](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container). +You can pass information from available Container-level fields using +`resourceFieldRef`. + +### Information available via `fieldRef` {#downwardapi-fieldRef} + +For most Pod-level fields, you can provide them to a container either as +an environment variable or using a `downwardAPI` volume. The fields available +via either mechanism are: + +`metadata.name` +: the pod's name + +`metadata.namespace` +: the pod's {{< glossary_tooltip text="namespace" term_id="namespace" >}} + +`metadata.uid` +: the pod's unique ID + +`metadata.annotations['<KEY>']` +: the value of the pod's {{< glossary_tooltip text="annotation" term_id="annotation" >}} named `<KEY>` (for example, `metadata.annotations['myannotation']`) + +`metadata.labels['<KEY>']` +: the text value of the pod's {{< glossary_tooltip text="label" term_id="label" >}} named `<KEY>` (for example, `metadata.labels['mylabel']`) + +`spec.serviceAccountName` +: the name of the pod's {{< glossary_tooltip text="service account" term_id="service-account" >}} + +`spec.nodeName` +: the name of the {{< glossary_tooltip term_id="node" text="node">}} where the Pod is executing + +`status.hostIP` +: the primary IP address of the node to which the Pod is assigned + +`status.podIP` +: the pod's primary IP address (usually, its IPv4 address) + +In addition, the following information is available through +a `downwardAPI` volume `fieldRef`, but **not as environment variables**: + +`metadata.labels` +: all of the pod's labels, formatted as `label-key="escaped-label-value"` with one label per line + +`metadata.annotations` +: all of the pod's annotations, formatted as `annotation-key="escaped-annotation-value"` with one annotation per line + +### Information available via `resourceFieldRef` {#downwardapi-resourceFieldRef} + +These container-level fields allow you to provide information about +[requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) +for resources such as CPU and memory. + + +`resource: limits.cpu` +: A container's CPU limit + +`resource: requests.cpu` +: A container's CPU request + +`resource: limits.memory` +: A container's memory limit + +`resource: requests.memory` +: A container's memory request + +`resource: limits.hugepages-*` +: A container's hugepages limit (provided that the `DownwardAPIHugePages` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) + +`resource: requests.hugepages-*` +: A container's hugepages request (provided that the `DownwardAPIHugePages` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) + +`resource: limits.ephemeral-storage` +: A container's ephemeral-storage limit + +`resource: requests.ephemeral-storage` +: A container's ephemeral-storage request + +#### Fallback information for resource limits + +If CPU and memory limits are not specified for a container, and you use the +downward API to try to expose that information, then the +kubelet defaults to exposing the maximum allocatable value for CPU and memory +based on the [node allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +calculation. + +## {{% heading "whatsnext" %}} + +You can read about [`downwardAPI` volumes](/docs/concepts/storage/volumes/#downwardapi). + +You can try using the downward API to expose container- or Pod-level information: +* as [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) +* as [files in `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) diff --git a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md index c26a63b183..0a70fedd6f 100644 --- a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md +++ b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md @@ -9,22 +9,13 @@ weight: 80 <!-- overview --> -{{< feature-state state="alpha" for_k8s_version="v1.22" >}} +{{< feature-state state="beta" for_k8s_version="v1.23" >}} This page provides an overview of ephemeral containers: a special type of container that runs temporarily in an existing {{< glossary_tooltip term_id="pod" >}} to accomplish user-initiated actions such as troubleshooting. You use ephemeral containers to inspect services rather than to build applications. -{{< warning >}} -Ephemeral containers are in alpha state and are not suitable for production -clusters. In accordance with the [Kubernetes Deprecation Policy]( -/docs/reference/using-api/deprecation-policy/), this alpha feature could change -significantly in the future or be removed entirely. -{{< /warning >}} - - - <!-- body --> ## Understanding ephemeral containers @@ -79,5 +70,5 @@ you can view processes in other containers. ## {{% heading "whatsnext" %}} -* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container). +* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container). diff --git a/content/en/docs/concepts/workloads/pods/init-containers.md b/content/en/docs/concepts/workloads/pods/init-containers.md index edf9407845..85c5ec413b 100644 --- a/content/en/docs/concepts/workloads/pods/init-containers.md +++ b/content/en/docs/concepts/workloads/pods/init-containers.md @@ -28,7 +28,7 @@ Init containers are exactly like regular containers, except: * Init containers always run to completion. * Each init container must complete successfully before the next one starts. -If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds. +If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds. However, if the Pod has a `restartPolicy` of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed. To specify an init container for a Pod, add the `initContainers` field into @@ -115,7 +115,7 @@ kind: Pod metadata: name: myapp-pod labels: - app: myapp + app.kubernetes.io/name: MyApp spec: containers: - name: myapp-container @@ -159,7 +159,7 @@ The output is similar to this: Name: myapp-pod Namespace: default [...] -Labels: app=myapp +Labels: app.kubernetes.io/name=MyApp Status: Pending [...] Init Containers: @@ -282,7 +282,7 @@ define readiness distinct from completion. This is enforced during validation. Use `activeDeadlineSeconds` on the Pod to prevent init containers from failing forever. The active deadline includes init containers. -However it is recommended to use `activeDeadlineSeconds` if user deploy their application +However it is recommended to use `activeDeadlineSeconds` only if teams deploy their application as a Job, because `activeDeadlineSeconds` has an effect even after initContainer finished. The Pod which is already running correctly would be killed by `activeDeadlineSeconds` if you set. @@ -332,5 +332,5 @@ Kubernetes, consult the documentation for the version you are using. ## {{% heading "whatsnext" %}} * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container) -* Learn how to [debug init containers](/docs/tasks/debug-application-cluster/debug-init-containers/) +* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/) diff --git a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md index 291c8e6af3..596d835d73 100644 --- a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md +++ b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md @@ -55,7 +55,7 @@ exists. If that Pod is deleted for any reason, and even if an identical replacem is created, the related thing (a volume, in this example) is also destroyed and created anew. -{{< figure src="/images/docs/pod.svg" title="Pod diagram" width="50%" >}} +{{< figure src="/images/docs/pod.svg" title="Pod diagram" class="diagram-medium" >}} *A multi-container Pod that contains a file puller and a web server that uses a persistent volume for shared storage between the containers.* @@ -136,7 +136,7 @@ completion or failed for some reason. When you use `kubectl` to query a Pod with a container that is `Terminated`, you see a reason, an exit code, and the start and finish time for that container's period of execution. -If a container has a `preStop` hook configured, that runs before the container enters +If a container has a `preStop` hook configured, this hook runs before the container enters the `Terminated` state. ## Container restart policy {#restart-policy} @@ -159,7 +159,7 @@ through which the Pod has or has not passed: * `PodScheduled`: the Pod has been scheduled to a node. * `ContainersReady`: all containers in the Pod are ready. * `Initialized`: all [init containers](/docs/concepts/workloads/pods/init-containers/) - have started successfully. + have completed successfully. * `Ready`: the Pod is able to serve requests and should be added to the load balancing pools of all matching Services. @@ -229,61 +229,91 @@ when both the following statements apply: * All conditions specified in `readinessGates` are `True`. When a Pod's containers are Ready but at least one custom condition is missing or -`False`, the kubelet sets the Pod's [condition](#pod-condition) to `ContainersReady`. +`False`, the kubelet sets the Pod's [condition](#pod-conditions) to `ContainersReady`. ## Container probes -A [Probe](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#probe-v1-core) is a diagnostic +A _probe_ is a diagnostic performed periodically by the [kubelet](/docs/reference/command-line-tools-reference/kubelet/) -on a Container. To perform a diagnostic, -the kubelet calls a -[Handler](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#handler-v1-core) implemented by -the container. There are three types of handlers: +on a container. To perform a diagnostic, +the kubelet either executes code within the container, or makes +a network request. -* [ExecAction](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#execaction-v1-core): - Executes a specified command inside the container. The diagnostic +### Check mechanisms {#probe-check-methods} + +There are four different ways to check a container using a probe. +Each probe must define exactly one of these four mechanisms: + +`exec` +: Executes a specified command inside the container. The diagnostic is considered successful if the command exits with a status code of 0. -* [TCPSocketAction](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#tcpsocketaction-v1-core): - Performs a TCP check against the Pod's IP address on - a specified port. The diagnostic is considered successful if the port is open. +`grpc` +: Performs a remote procedure call using [gRPC](https://grpc.io/). + The target should implement + [gRPC health checks](https://grpc.io/grpc/core/md_doc_health-checking.html). + The diagnostic is considered successful if the `status` + of the response is `SERVING`. + gRPC probes are an alpha feature and are only available if you + enable the `GRPCContainerProbe` + [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). -* [HTTPGetAction](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#httpgetaction-v1-core): - Performs an HTTP `GET` request against the Pod's IP - address on a specified port and path. The diagnostic is considered successful - if the response has a status code greater than or equal to 200 and less than 400. +`httpGet` +: Performs an HTTP `GET` request against the Pod's IP + address on a specified port and path. The diagnostic is + considered successful if the response has a status code + greater than or equal to 200 and less than 400. + +`tcpSocket` +: Performs a TCP check against the Pod's IP address on + a specified port. The diagnostic is considered successful if + the port is open. If the remote system (the container) closes + the connection immediately after it opens, this counts as healthy. + +### Probe outcome Each probe has one of three results: -* `Success`: The container passed the diagnostic. -* `Failure`: The container failed the diagnostic. -* `Unknown`: The diagnostic failed, so no action should be taken. +`Success` +: The container passed the diagnostic. + +`Failure` +: The container failed the diagnostic. + +`Unknown` +: The diagnostic failed (no action should be taken, and the kubelet + will make further checks). + +### Types of probe The kubelet can optionally perform and react to three kinds of probes on running containers: -* `livenessProbe`: Indicates whether the container is running. If - the liveness probe fails, the kubelet kills the container, and the container - is subjected to its [restart policy](#restart-policy). If a Container does not - provide a liveness probe, the default state is `Success`. +`livenessProbe` +: Indicates whether the container is running. If + the liveness probe fails, the kubelet kills the container, and the container + is subjected to its [restart policy](#restart-policy). If a container does not + provide a liveness probe, the default state is `Success`. -* `readinessProbe`: Indicates whether the container is ready to respond to requests. - If the readiness probe fails, the endpoints controller removes the Pod's IP - address from the endpoints of all Services that match the Pod. The default - state of readiness before the initial delay is `Failure`. If a Container does - not provide a readiness probe, the default state is `Success`. +`readinessProbe` +: Indicates whether the container is ready to respond to requests. + If the readiness probe fails, the endpoints controller removes the Pod's IP + address from the endpoints of all Services that match the Pod. The default + state of readiness before the initial delay is `Failure`. If a container does + not provide a readiness probe, the default state is `Success`. -* `startupProbe`: Indicates whether the application within the container is started. - All other probes are disabled if a startup probe is provided, until it succeeds. - If the startup probe fails, the kubelet kills the container, and the container - is subjected to its [restart policy](#restart-policy). If a Container does not - provide a startup probe, the default state is `Success`. +`startupProbe` +: Indicates whether the application within the container is started. + All other probes are disabled if a startup probe is provided, until it succeeds. + If the startup probe fails, the kubelet kills the container, and the container + is subjected to its [restart policy](#restart-policy). If a container does not + provide a startup probe, the default state is `Success`. For more information about how to set up a liveness, readiness, or startup probe, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). -### When should you use a liveness probe? +#### When should you use a liveness probe? {{< feature-state for_k8s_version="v1.0" state="stable" >}} @@ -295,7 +325,7 @@ with the Pod's `restartPolicy`. If you'd like your container to be killed and restarted if a probe fails, then specify a liveness probe, and specify a `restartPolicy` of Always or OnFailure. -### When should you use a readiness probe? +#### When should you use a readiness probe? {{< feature-state for_k8s_version="v1.0" state="stable" >}} @@ -329,7 +359,7 @@ The Pod remains in the unready state while it waits for the containers in the Po to stop. {{< /note >}} -### When should you use a startup probe? +#### When should you use a startup probe? {{< feature-state for_k8s_version="v1.20" state="stable" >}} @@ -451,13 +481,13 @@ This avoids a resource leak as Pods are created and terminated over time. ## {{% heading "whatsnext" %}} * Get hands-on experience - [attaching handlers to Container lifecycle events](/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/). + [attaching handlers to container lifecycle events](/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/). * Get hands-on experience [configuring Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). * Learn more about [container lifecycle hooks](/docs/concepts/containers/container-lifecycle-hooks/). -* For detailed information about Pod / Container status in the API, see [PodStatus](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podstatus-v1-core) -and -[ContainerStatus](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#containerstatus-v1-core). +* For detailed information about Pod and container status in the API, see + the API reference documentation covering + [`.status`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus) for Pod. diff --git a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md deleted file mode 100644 index 719fcee457..0000000000 --- a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md +++ /dev/null @@ -1,402 +0,0 @@ ---- -title: Pod Topology Spread Constraints -content_type: concept -weight: 40 ---- - -{{< feature-state for_k8s_version="v1.19" state="stable" >}} -<!-- leave this shortcode in place until the note about EvenPodsSpread is -obsolete --> - -<!-- overview --> - -You can use _topology spread constraints_ to control how {{< glossary_tooltip text="Pods" term_id="Pod" >}} are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. - -{{< note >}} -In versions of Kubernetes before v1.18, you must enable the `EvenPodsSpread` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on -the [API server](/docs/concepts/overview/components/#kube-apiserver) and the -[scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/) in order to use Pod -topology spread constraints. -{{< /note >}} - -<!-- body --> - -## Prerequisites - -### Node Labels - -Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. For example, a Node might have labels: `node=node1,zone=us-east-1a,region=us-east-1` - -Suppose you have a 4-node cluster with the following labels: - -``` -NAME STATUS ROLES AGE VERSION LABELS -node1 Ready <none> 4m26s v1.16.0 node=node1,zone=zoneA -node2 Ready <none> 3m58s v1.16.0 node=node2,zone=zoneA -node3 Ready <none> 3m17s v1.16.0 node=node3,zone=zoneB -node4 Ready <none> 2m43s v1.16.0 node=node4,zone=zoneB -``` - -Then the cluster is logically viewed as below: - -{{<mermaid>}} -graph TB - subgraph "zoneB" - n3(Node3) - n4(Node4) - end - subgraph "zoneA" - n1(Node1) - n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4 k8s; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -Instead of manually applying labels, you can also reuse the [well-known labels](/docs/reference/labels-annotations-taints/) that are created and populated automatically on most clusters. - -## Spread Constraints for Pods - -### API - -The API field `pod.spec.topologySpreadConstraints` is defined as below: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - topologySpreadConstraints: - - maxSkew: <integer> - topologyKey: <string> - whenUnsatisfiable: <string> - labelSelector: <object> -``` - -You can define one or multiple `topologySpreadConstraint` to instruct the kube-scheduler how to place each incoming Pod in relation to the existing Pods across your cluster. The fields are: - -- **maxSkew** describes the degree to which Pods may be unevenly distributed. - It must be greater than zero. Its semantics differs according to the value of `whenUnsatisfiable`: - - when `whenUnsatisfiable` equals to "DoNotSchedule", `maxSkew` is the maximum - permitted difference between the number of matching pods in the target - topology and the global minimum - (the minimum number of pods that match the label selector in a topology domain. For example, if you have 3 zones with 0, 2 and 3 matching pods respectively, The global minimum is 0). - - when `whenUnsatisfiable` equals to "ScheduleAnyway", scheduler gives higher - precedence to topologies that would help reduce the skew. -- **topologyKey** is the key of node labels. If two Nodes are labelled with this key and have identical values for that label, the scheduler treats both Nodes as being in the same topology. The scheduler tries to place a balanced number of Pods into each topology domain. -- **whenUnsatisfiable** indicates how to deal with a Pod if it doesn't satisfy the spread constraint: - - `DoNotSchedule` (default) tells the scheduler not to schedule it. - - `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew. -- **labelSelector** is used to find matching Pods. Pods that match this label selector are counted to determine the number of Pods in their corresponding topology domain. See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) for more details. - -When a Pod defines more than one `topologySpreadConstraint`, those constraints are ANDed: The kube-scheduler looks for a node for the incoming Pod that satisfies all the constraints. - -You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`. - -### Example: One TopologySpreadConstraint - -Suppose you have a 4-node cluster where 3 Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: - -{{<mermaid>}} -graph BT - subgraph "zoneB" - p3(Pod) --> n3(Node3) - n4(Node4) - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3 k8s; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -If we want an incoming Pod to be evenly spread with existing Pods across zones, the spec can be given as: - -{{< codenew file="pods/topology-spread-constraints/one-constraint.yaml" >}} - -`topologyKey: zone` implies the even distribution will only be applied to the nodes which have label pair "zone:<any value>" present. `whenUnsatisfiable: DoNotSchedule` tells the scheduler to let it stay pending if the incoming Pod can't satisfy the constraint. - -If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In this example, the incoming Pod can only be placed onto "zoneB": - -{{<mermaid>}} -graph BT - subgraph "zoneB" - p3(Pod) --> n3(Node3) - p4(mypod) --> n4(Node4) - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3 k8s; - class p4 plain; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -OR - -{{<mermaid>}} -graph BT - subgraph "zoneB" - p3(Pod) --> n3(Node3) - p4(mypod) --> n3 - n4(Node4) - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3 k8s; - class p4 plain; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -You can tweak the Pod spec to meet various kinds of requirements: - -- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed onto "zoneA" as well. -- Change `topologyKey` to "node" so as to distribute the Pods evenly across nodes instead of zones. In the above example, if `maxSkew` remains "1", the incoming Pod can only be placed onto "node4". -- Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs are satisfied). However, it's preferred to be placed onto the topology domain which has fewer matching Pods. (Be aware that this preferability is jointly normalized with other internal scheduling priorities like resource usage ratio, etc.) - -### Example: Multiple TopologySpreadConstraints - -This builds upon the previous example. Suppose you have a 4-node cluster where 3 Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: - -{{<mermaid>}} -graph BT - subgraph "zoneB" - p3(Pod) --> n3(Node3) - n4(Node4) - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3 k8s; - class p4 plain; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -You can use 2 TopologySpreadConstraints to control the Pods spreading on both zone and node: - -{{< codenew file="pods/topology-spread-constraints/two-constraints.yaml" >}} - -In this case, to match the first constraint, the incoming Pod can only be placed onto "zoneB"; while in terms of the second constraint, the incoming Pod can only be placed onto "node4". Then the results of 2 constraints are ANDed, so the only viable option is to place on "node4". - -Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones: - -{{<mermaid>}} -graph BT - subgraph "zoneB" - p4(Pod) --> n3(Node3) - p5(Pod) --> n3 - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n1 - p3(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3,p4,p5 k8s; - class zoneA,zoneB cluster; -{{< /mermaid >}} - -If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in `Pending` state. This is because: to satisfy the first constraint, "mypod" can only be put to "zoneB"; while in terms of the second constraint, "mypod" can only put to "node2". Then a joint result of "zoneB" and "node2" returns nothing. - -To overcome this situation, you can either increase the `maxSkew` or modify one of the constraints to use `whenUnsatisfiable: ScheduleAnyway`. - -### Interaction With Node Affinity and Node Selectors - -The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. - - Suppose you have a 5-node cluster ranging from zoneA to zoneC: - - {{<mermaid>}} - graph BT - subgraph "zoneB" - p3(Pod) --> n3(Node3) - n4(Node4) - end - subgraph "zoneA" - p1(Pod) --> n1(Node1) - p2(Pod) --> n2(Node2) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n1,n2,n3,n4,p1,p2,p3 k8s; - class p4 plain; - class zoneA,zoneB cluster; - {{< /mermaid >}} - - {{<mermaid>}} - graph BT - subgraph "zoneC" - n5(Node5) - end - - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class n5 k8s; - class zoneC cluster; - {{< /mermaid >}} - - and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. - - {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} - -The scheduler doesn't have prior knowledge of all the zones or other topology domains that a cluster has. They are determined from the existing nodes in the cluster. This could lead to a problem in autoscaled clusters, when a node pool (or node group) is scaled to zero nodes and the user is expecting them to scale up, because, in this case, those topology domains won't be considered until there is at least one node in them. - -### Other Noticeable Semantics - -There are some implicit conventions worth noting here: - -- Only the Pods holding the same namespace as the incoming Pod can be matching candidates. - -- The scheduler will bypass the nodes without `topologySpreadConstraints[*].topologyKey` present. This implies that: - - 1. the Pods located on those nodes do not impact `maxSkew` calculation - in the above example, suppose "node1" does not have label "zone", then the 2 Pods will be disregarded, hence the incoming Pod will be scheduled into "zoneA". - 2. the incoming Pod has no chances to be scheduled onto this kind of nodes - in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` joins the cluster, it will be bypassed due to the absence of label key "zone". - -- Be aware of what will happen if the incomingPod's `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the above example, if we remove the incoming Pod's labels, it can still be placed onto "zoneB" since the constraints are still satisfied. However, after the placement, the degree of imbalance of the cluster remains unchanged - it's still zoneA having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds label {foo:bar}. So if this is not what you expect, we recommend the workload's `topologySpreadConstraints[*].labelSelector` to match its own labels. - -### Cluster-level default constraints - -It is possible to set default topology spread constraints for a cluster. Default -topology spread constraints are applied to a Pod if, and only if: - -- It doesn't define any constraints in its `.spec.topologySpreadConstraints`. -- It belongs to a service, replication controller, replica set or stateful set. - -Default constraints can be set as part of the `PodTopologySpread` plugin args -in a [scheduling profile](/docs/reference/scheduling/config/#profiles). -The constraints are specified with the same [API above](#api), except that -`labelSelector` must be empty. The selectors are calculated from the services, -replication controllers, replica sets or stateful sets that the Pod belongs to. - -An example configuration might look like follows: - -```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 -kind: KubeSchedulerConfiguration - -profiles: - - pluginConfig: - - name: PodTopologySpread - args: - defaultConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: ScheduleAnyway - defaultingType: List -``` - -{{< note >}} -The score produced by default scheduling constraints might conflict with the -score produced by the -[`SelectorSpread` plugin](/docs/reference/scheduling/config/#scheduling-plugins). -It is recommended that you disable this plugin in the scheduling profile when -using default constraints for `PodTopologySpread`. -{{< /note >}} - -#### Internal default constraints - -{{< feature-state for_k8s_version="v1.20" state="beta" >}} - -With the `DefaultPodTopologySpread` feature gate, enabled by default, the -legacy `SelectorSpread` plugin is disabled. -kube-scheduler uses the following default topology constraints for the -`PodTopologySpread` plugin configuration: - -```yaml -defaultConstraints: - - maxSkew: 3 - topologyKey: "kubernetes.io/hostname" - whenUnsatisfiable: ScheduleAnyway - - maxSkew: 5 - topologyKey: "topology.kubernetes.io/zone" - whenUnsatisfiable: ScheduleAnyway -``` - -Also, the legacy `SelectorSpread` plugin, which provides an equivalent behavior, -is disabled. - -{{< note >}} -If your nodes are not expected to have **both** `kubernetes.io/hostname` and -`topology.kubernetes.io/zone` labels set, define your own constraints -instead of using the Kubernetes defaults. - -The `PodTopologySpread` plugin does not score the nodes that don't have -the topology keys specified in the spreading constraints. -{{< /note >}} - -If you don't want to use the default Pod spreading constraints for your cluster, -you can disable those defaults by setting `defaultingType` to `List` and leaving -empty `defaultConstraints` in the `PodTopologySpread` plugin configuration: - -```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 -kind: KubeSchedulerConfiguration - -profiles: - - pluginConfig: - - name: PodTopologySpread - args: - defaultConstraints: [] - defaultingType: List -``` - -## Comparison with PodAffinity/PodAntiAffinity - -In Kubernetes, directives related to "Affinity" control how Pods are -scheduled - more packed or more scattered. - -- For `PodAffinity`, you can try to pack any number of Pods into qualifying - topology domain(s) -- For `PodAntiAffinity`, only one Pod can be scheduled into a - single topology domain. - -For finer control, you can specify topology spread constraints to distribute -Pods across different topology domains - to achieve either high availability or -cost-saving. This can also help on rolling update workloads and scaling out -replicas smoothly. See -[Motivation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/895-pod-topology-spread#motivation) -for more details. - -## Known Limitations - -- There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution. -You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution. -- Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921) - -## {{% heading "whatsnext" %}} - -- [Blog: Introducing PodTopologySpread](https://kubernetes.io/blog/2020/05/introducing-podtopologyspread/) - explains `maxSkew` in details, as well as bringing up some advanced usage examples. diff --git a/content/en/docs/contribute/_index.md b/content/en/docs/contribute/_index.md index 1db65ec56d..b4c7d838fa 100644 --- a/content/en/docs/contribute/_index.md +++ b/content/en/docs/contribute/_index.md @@ -18,8 +18,15 @@ card: {{< note >}} To learn more about contributing to Kubernetes in general, see the [contributor documentation](https://www.kubernetes.dev/docs/). + +You can also read the +{{< glossary_tooltip text="CNCF" term_id="cncf" >}} +[page](https://contribute.cncf.io/contributors/projects/#kubernetes) +about contributing to Kubernetes. {{< /note >}} +--- + This website is maintained by [Kubernetes SIG Docs](/docs/contribute/#get-involved-with-sig-docs). Kubernetes documentation contributors: @@ -29,6 +36,8 @@ Kubernetes documentation contributors: - Translate the documentation - Manage and publish the documentation parts of the Kubernetes release cycle + + <!-- body --> ## Getting started @@ -44,19 +53,99 @@ to work effectively in the Kubernetes community. To get involved with documentation: 1. Sign the CNCF [Contributor License Agreement](https://github.com/kubernetes/community/blob/master/CLA.md). -1. Familiarize yourself with the [documentation repository](https://github.com/kubernetes/website) +2. Familiarize yourself with the [documentation repository](https://github.com/kubernetes/website) and the website's [static site generator](https://gohugo.io). -1. Make sure you understand the basic processes for +3. Make sure you understand the basic processes for [opening a pull request](/docs/contribute/new-content/open-a-pr/) and [reviewing changes](/docs/contribute/review/reviewing-prs/). +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> + +{{< mermaid >}} +flowchart TB +subgraph third[Open PR] +direction TB +U[ ] -.- +Q[Improve content] --- N[Create content] +N --- O[Translate docs] +O --- P[Manage/publish docs parts<br>of K8s release cycle] + +end + +subgraph second[Review] +direction TB + T[ ] -.- + D[Look over the<br>K8s/website<br>repository] --- E[Check out the<br>Hugo static site<br>generator] + E --- F[Understand basic<br>GitHub commands] + F --- G[Review open PR<br>and change review <br>processes] +end + +subgraph first[Sign up] + direction TB + S[ ] -.- + B[Sign the CNCF<br>Contributor<br>License Agreement] --- C[Join sig-docs<br>Slack channel] + C --- V[Join kubernetes-sig-docs<br>mailing list] + V --- M[Attend weekly<br>sig-docs calls<br>or slack meetings] +end + +A([fa:fa-user New<br>Contributor]) --> first +A --> second +A --> third +A --> H[Ask Questions!!!] + + +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class A,B,C,D,E,F,G,H,M,Q,N,O,P,V grey +class S,T,U spacewhite +class first,second,third white +{{</ mermaid >}} +Figure 1. Getting started for a new contributor. + +Figure 1 outlines a roadmap for new contributors. You can follow some or all of the steps for `Sign up` and `Review`. Now you are ready to open PRs that achieve your contribution objectives with some listed under `Open PR`. Again, questions are always welcome! + Some tasks require more trust and more access in the Kubernetes organization. See [Participating in SIG Docs](/docs/contribute/participate/) for more details about roles and permissions. ## Your first contribution -- Read the [Contribution overview](/docs/contribute/new-content/overview/) to +You can prepare for your first contribution by reviewing several steps beforehand. Figure 2 outlines the steps and the details follow. + +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> + +{{< mermaid >}} +flowchart LR + subgraph second[First Contribution] + direction TB + S[ ] -.- + G[Review PRs from other<br>K8s members] --> + A[Check K8s/website<br>issues list for<br>good first PRs] --> B[Open a PR!!] + end + subgraph first[Suggested Prep] + direction TB + T[ ] -.- + D[Read contribution overview] -->E[Read K8s content<br>and style guides] + E --> F[Learn about Hugo page<br>content types<br>and shortcodes] + end + + + first ----> second + + +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class A,B,D,E,F,G grey +class S,T spacewhite +class first,second white +{{</ mermaid >}} +Figure 2. Preparation for your first contribution. + +- Read the [Contribution overview](/docs/contribute/new-content/) to learn about the different ways you can contribute. - Check [`kubernetes/website` issues list](https://github.com/kubernetes/website/issues/) for issues that make good entry points. @@ -92,10 +181,13 @@ SIG Docs communicates with different methods: introduce yourself! - [Join the `kubernetes-sig-docs` mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs), where broader discussions take place and official decisions are recorded. -- Join the [weekly SIG Docs video meeting](https://github.com/kubernetes/community/tree/master/sig-docs). Meetings are always announced on `#sig-docs` and added to the [Kubernetes community meetings calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles). You'll need to download the [Zoom client](https://zoom.us/download) or dial in using a phone. +- Join the [SIG Docs video meeting](https://github.com/kubernetes/community/tree/master/sig-docs) held every two weeks. Meetings are always announced on `#sig-docs` and added to the [Kubernetes community meetings calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles). You'll need to download the [Zoom client](https://zoom.us/download) or dial in using a phone. +- Join the SIG Docs async Slack standup meeting on those weeks when the in-person Zoom video meeting does not take place. Meetings are always announced on `#sig-docs`. You can contribute to any one of the threads up to 24 hours after meeting announcement. ## Other ways to contribute - Visit the [Kubernetes community site](/community/). Participate on Twitter or Stack Overflow, learn about local Kubernetes meetups and events, and more. -- Read the [contributor cheatsheet](https://github.com/kubernetes/community/tree/master/contributors/guide/contributor-cheatsheet) to get involved with Kubernetes feature development. +- Read the [contributor cheatsheet](https://www.kubernetes.dev/docs/contributor-cheatsheet/) to get involved with Kubernetes feature development. +- Visit the contributor site to learn more about [Kubernetes Contributors](https://www.kubernetes.dev/) and [additional contributor resources](https://www.kubernetes.dev/resources/). - Submit a [blog post or case study](/docs/contribute/new-content/blogs-case-studies/). + diff --git a/content/en/docs/contribute/advanced.md b/content/en/docs/contribute/advanced.md index 2bddacbc3f..b825ad5e2c 100644 --- a/content/en/docs/contribute/advanced.md +++ b/content/en/docs/contribute/advanced.md @@ -8,7 +8,7 @@ weight: 98 <!-- overview --> This page assumes that you understand how to -[contribute to new content](/docs/contribute/new-content/overview) and +[contribute to new content](/docs/contribute/new-content/) and [review others' work](/docs/contribute/review/reviewing-prs/), and are ready to learn about more ways to contribute. You need to use the Git command line client and other tools for some of these tasks. @@ -27,7 +27,7 @@ the documentation, the website style, the processes for reviewing and merging pull requests, or other aspects of the documentation. For maximum transparency, these types of proposals need to be discussed in a SIG Docs meeting or on the [kubernetes-sig-docs mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs). -In addition, it can really help to have some context about the way things +In addition, it can help to have some context about the way things currently work and why past decisions have been made before proposing sweeping changes. The quickest way to get answers to questions about how the documentation currently works is to ask in the `#sig-docs` Slack channel on @@ -54,7 +54,7 @@ refer to The SIG Docs representative for a given release coordinates the following tasks: - Monitor the feature-tracking spreadsheet for new or changed features with an - impact on documentation. If documentation for a given feature won't be ready + impact on documentation. If the documentation for a given feature won't be ready for the release, the feature may not be allowed to go into the release. - Attend sig-release meetings regularly and give updates on the status of the docs for the release. @@ -82,12 +82,13 @@ few PR submissions. Responsibilities for New Contributor Ambassadors include: - Monitoring the [#sig-docs Slack channel](https://kubernetes.slack.com) for questions from new contributors. -- Working with PR wranglers to identify good first issues for new contributors. +- Working with PR wranglers to identify [good first issues](https://kubernetes.dev/docs/guide/help-wanted/#good-first-issue) for new contributors. - Mentoring new contributors through their first few PRs to the docs repo. - Helping new contributors create the more complex PRs they need to become Kubernetes members. - [Sponsoring contributors](/docs/contribute/advanced/#sponsor-a-new-contributor) on their path to becoming Kubernetes members. +- Hosting a monthly meeting to help and mentor new contributors. -Current New Contributor Ambassadors are announced at each SIG-Docs meeting, and in the [Kubernetes #sig-docs channel](https://kubernetes.slack.com). +Current New Contributor Ambassadors are announced at each SIG-Docs meeting and in the [Kubernetes #sig-docs channel](https://kubernetes.slack.com). ## Sponsor a new contributor @@ -110,22 +111,22 @@ membership in the Kubernetes organization. ## Serve as a SIG Co-chair -SIG Docs [approvers](/docs/contribute/participate/roles-and-responsibilities/#approvers) +SIG Docs [members](/docs/contribute/participate/roles-and-responsibilities/#members) can serve a term as a co-chair of SIG Docs. ### Prerequisites -Approvers must meet the following requirements to be a co-chair: +A Kubernetes member must meet the following requirements to be a co-chair: -- Have been a SIG Docs approver for at least 6 months -- Have [led a Kubernetes docs release](/docs/contribute/advanced/#coordinate-docs-for-a-kubernetes-release) or shadowed two releases - Understand SIG Docs workflows and tooling: git, Hugo, localization, blog subproject - Understand how other Kubernetes SIGs and repositories affect the SIG Docs workflow, including: - [teams in k/org](https://github.com/kubernetes/org/blob/master/config/kubernetes/sig-docs/teams.yaml), + [teams in k/org](https://github.com/kubernetes/org/blob/master/config/kubernetes/sig-docs/teams.yaml), the [process in k/community](https://github.com/kubernetes/community/tree/master/sig-docs), plugins in [k/test-infra](https://github.com/kubernetes/test-infra/), and the role of [SIG Architecture](https://github.com/kubernetes/community/tree/master/sig-architecture). + In addition, understand how the [Kubernetes docs release process](/docs/contribute/advanced/#coordinate-docs-for-a-kubernetes-release) works. +- Approved by the SIG Docs community either directly or via lazy consensus. - Commit at least 5 hours per week (and often more) to the role for a minimum of 6 months ### Responsibilities @@ -135,7 +136,7 @@ The role of co-chair is one of service: co-chairs build contributor capacity, ha Responsibilities include: - Keep SIG Docs focused on maximizing developer happiness through excellent documentation -- Exemplify the [community code of conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and hold SIG members accountable to it +- Exemplify the [community code of conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md) and hold SIG members accountable to it - Learn and set best practices for the SIG by updating contribution guidelines - Schedule and run SIG meetings: weekly status updates, quarterly retro/planning sessions, and others as needed - Schedule and run doc sprints at KubeCon events and other conferences @@ -146,7 +147,7 @@ Responsibilities include: To schedule and run effective meetings, these guidelines show what to do, how to do it, and why. -**Uphold the [community code of conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md)**: +**Uphold the [community code of conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md)**: - Hold respectful, inclusive discussions with respectful, inclusive language. diff --git a/content/en/docs/contribute/generate-ref-docs/contribute-upstream.md b/content/en/docs/contribute/generate-ref-docs/contribute-upstream.md index 4abf2de68b..ffce805718 100644 --- a/content/en/docs/contribute/generate-ref-docs/contribute-upstream.md +++ b/content/en/docs/contribute/generate-ref-docs/contribute-upstream.md @@ -27,6 +27,8 @@ API or the `kube-*` components from the upstream code, see the following instruc - [Golang](https://golang.org/doc/install) version 1.13+ - [Docker](https://docs.docker.com/engine/installation/) - [etcd](https://github.com/coreos/etcd/) + - [make](https://www.gnu.org/software/make/) + - [gcc compiler/linker](https://gcc.gnu.org/) - Your `GOPATH` environment variable must be set, and the location of `etcd` must be in your `PATH` environment variable. @@ -142,8 +144,9 @@ Run `git status` to see what was generated. On branch master ... modified: api/openapi-spec/swagger.json + modified: api/openapi-spec/v3/apis__apps__v1_openapi.json + modified: pkg/generated/openapi/zz_generated.openapi.go modified: staging/src/k8s.io/api/apps/v1/generated.proto - modified: staging/src/k8s.io/api/apps/v1/types.go modified: staging/src/k8s.io/api/apps/v1/types_swagger_doc_generated.go ``` diff --git a/content/en/docs/contribute/localization.md b/content/en/docs/contribute/localization.md index c9cd5a5385..1c741b32e5 100644 --- a/content/en/docs/contribute/localization.md +++ b/content/en/docs/contribute/localization.md @@ -13,14 +13,19 @@ card: <!-- overview --> -This page shows you how to [localize](https://blog.mozilla.org/l10n/2011/12/14/i18n-vs-l10n-whats-the-diff/) the docs for a different language. +This page shows you how to +[localize](https://blog.mozilla.org/l10n/2011/12/14/i18n-vs-l10n-whats-the-diff/) the docs for a +different language. <!-- body --> ## Contribute to an existing localization -You can help add or improve content to an existing localization. In [Kubernetes Slack](https://slack.k8s.io/) you'll find a channel for each localization. There is also a general [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations) where you can say hello. +You can help add or improve content to an existing localization. In [Kubernetes +Slack](https://slack.k8s.io/) you'll find a channel for each localization. There is also a general +[SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations) +where you can say hello. {{< note >}} If you want to work on a localization that already exists, check @@ -30,11 +35,14 @@ English original. You might see extra details there. ### Find your two-letter language code -First, consult the [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) to find your localization's two-letter language code. For example, the two-letter code for Korean is `ko`. +First, consult the [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) +to find your localization's two-letter language code. For example, the two-letter code for Korean +is `ko`. ### Fork and clone the repo -First, [create your own fork](/docs/contribute/new-content/open-a-pr/#fork-the-repo) of the [kubernetes/website](https://github.com/kubernetes/website) repository. +First, [create your own fork](/docs/contribute/new-content/open-a-pr/#fork-the-repo) of the +[kubernetes/website](https://github.com/kubernetes/website) repository. Then, clone your fork and `cd` into it: @@ -43,7 +51,8 @@ git clone https://github.com/<username>/website cd website ``` -The website content directory includes sub-directories for each language. The localization you want to help out with is inside `content/<two-letter-code>`. +The website content directory includes sub-directories for each language. The localization you +want to help out with is inside `content/<two-letter-code>`. ### Suggest changes @@ -57,8 +66,9 @@ equivalent fix by updating the localization you're working on. Please limit pull requests to a single localization, since pull requests that change content in multiple localizations could be difficult to review. -Follow [Suggesting Content Improvements](/docs/contribute/suggest-improvements/) to propose changes to -that localization. The process is very similar to proposing changes to the upstream (English) content. +Follow [Suggesting Content Improvements](/docs/contribute/suggesting-improvements/) +to propose changes to that localization. The process is very similar to proposing changes to the +upstream (English) content. ## Start a new localization @@ -86,57 +96,92 @@ can incrementally work towards that goal. ### Find community -Let Kubernetes SIG Docs know you're interested in creating a localization! Join the [SIG Docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) and the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). Other localization teams are happy to help you get started and answer any questions you have. +Let Kubernetes SIG Docs know you're interested in creating a localization! Join the +[SIG Docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) and the +[SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). +Other localization teams are happy to help you get started and answer any questions you have. -Please also consider participating in the [SIG Docs Localization Subgroup meeting](https://github.com/kubernetes/community/tree/master/sig-docs). The mission of the SIG Docs localization subgroup is to work across the SIG Docs localization teams to collaborate on defining and documenting the processes for creating localized contribution guides. In addition, the SIG Docs localization subgroup will look for opportunities for the creation and sharing of common tools across localization teams and also serve to identify new requirements to the SIG Docs Leadership team. If you have questions about this meeting, please inquire on the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). +Please also consider participating in the +[SIG Docs Localization Subgroup meeting](https://github.com/kubernetes/community/tree/master/sig-docs). +The mission of the SIG Docs localization subgroup is to work across the SIG Docs localization +teams to collaborate on defining and documenting the processes for creating localized contribution +guides. In addition, the SIG Docs localization subgroup will look for opportunities for the +creation and sharing of common tools across localization teams and also serve to identify new +requirements to the SIG Docs Leadership team. If you have questions about this meeting, please +inquire on the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). -You can also create a Slack channel for your localization in the `kubernetes/community` repository. For an example of adding a Slack channel, see the PR for [adding a channel for Persian](https://github.com/kubernetes/community/pull/4980). +You can also create a Slack channel for your localization in the `kubernetes/community` +repository. For an example of adding a Slack channel, see the PR for +[adding a channel for Persian](https://github.com/kubernetes/community/pull/4980). ### Join the Kubernetes GitHub organization -Once you've opened a localization PR, you can become members of the Kubernetes GitHub organization. Each person on the team needs to create their own [Organization Membership Request](https://github.com/kubernetes/org/issues/new/choose) in the `kubernetes/org` repository. +Once you've opened a localization PR, you can become members of the Kubernetes GitHub +organization. Each person on the team needs to create their own +[Organization Membership Request](https://github.com/kubernetes/org/issues/new/choose) +in the `kubernetes/org` repository. ### Add your localization team in GitHub -Next, add your Kubernetes localization team to [`sig-docs/teams.yaml`](https://github.com/kubernetes/org/blob/main/config/kubernetes/sig-docs/teams.yaml). For an example of adding a localization team, see the PR to add the [Spanish localization team](https://github.com/kubernetes/org/pull/685). - -Members of `@kubernetes/sig-docs-**-owners` can approve PRs that change content within (and only within) your localization directory: `/content/**/`. - -For each localization, The `@kubernetes/sig-docs-**-reviews` team automates review assignment for new PRs. +Next, add your Kubernetes localization team to +[`sig-docs/teams.yaml`](https://github.com/kubernetes/org/blob/main/config/kubernetes/sig-docs/teams.yaml). +For an example of adding a localization team, see the PR to add the +[Spanish localization team](https://github.com/kubernetes/org/pull/685). +Members of `@kubernetes/sig-docs-**-owners` can approve PRs that change content within (and only +within) your localization directory: `/content/**/`. +For each localization, The `@kubernetes/sig-docs-**-reviews` team automates review assignment for +new PRs. Members of `@kubernetes/website-maintainers` can create new localization branches to coordinate translation efforts. - -Members of `@kubernetes/website-milestone-maintainers` can use the `/milestone` [Prow command](https://prow.k8s.io/command-help) to assign a milestone to issues or PRs. +Members of `@kubernetes/website-milestone-maintainers` can use the `/milestone` +[Prow command](https://prow.k8s.io/command-help) to assign a milestone to issues or PRs. ### Configure the workflow -Next, add a GitHub label for your localization in the `kubernetes/test-infra` repository. A label lets you filter issues and pull requests for your specific language. - -For an example of adding a label, see the PR for adding the [Italian language label](https://github.com/kubernetes/test-infra/pull/11316). +Next, add a GitHub label for your localization in the `kubernetes/test-infra` repository. A label +lets you filter issues and pull requests for your specific language. +For an example of adding a label, see the PR for adding the +[Italian language label](https://github.com/kubernetes/test-infra/pull/11316). ### Modify the site configuration -The Kubernetes website uses Hugo as its web framework. The website's Hugo configuration resides in the [`config.toml`](https://github.com/kubernetes/website/tree/main/config.toml) file. To support a new localization, you'll need to modify `config.toml`. +The Kubernetes website uses Hugo as its web framework. The website's Hugo configuration resides in +the [`config.toml`](https://github.com/kubernetes/website/tree/main/config.toml) file. +To support a new localization, you'll need to modify `config.toml`. -Add a configuration block for the new language to `config.toml`, under the existing `[languages]` block. The German block, for example, looks like: +Add a configuration block for the new language to `config.toml`, under the existing `[languages]` block. +The German block, for example, looks like: ```toml [languages.de] title = "Kubernetes" description = "Produktionsreife Container-Verwaltung" -languageName = "Deutsch" +languageName = "Deutsch (German)" +languageNameLatinScript = "Deutsch" contentDir = "content/de" -weight = 3 +weight = 8 ``` -When assigning a `weight` parameter for your block, find the language block with the highest weight and add 1 to that value. +The value for `languageName` will be listed in language selection bar. Assign "language name in +native script and language (English language name in latin script)" to `languageName`. +For example, `languageName = "한국어 (Korean)"` or `languageName = "Deutsch (German)"`. -For more information about Hugo's multilingual support, see "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)". +`languageNameLatinScript` can be used to access the language name in latin script and use it in +the theme. Assign "language name in latin script" to `languageNameLatinScript`. For example, +`languageNameLatinScript ="Korean"` or `languageNameLatinScript = "Deutsch"`. + +When assigning a `weight` parameter for your block, find the language block with the highest +weight and add 1 to that value. + +For more information about Hugo's multilingual support, see +"[Multilingual Mode](https://gohugo.io/content-management/multilingual/)". ### Add a new localization directory -Add a language-specific subdirectory to the [`content`](https://github.com/kubernetes/website/tree/main/content) folder in the repository. For example, the two-letter code for German is `de`: +Add a language-specific subdirectory to the +[`content`](https://github.com/kubernetes/website/tree/main/content) folder in the repository. +For example, the two-letter code for German is `de`: ```shell mkdir content/de @@ -146,28 +191,34 @@ You also need to create a directory inside `data/i18n` for [localized strings](#site-strings-in-i18n); look at existing localizations for an example. To use these new strings, you must also create a symbolic link from `i18n/<localization>.toml` to the actual string configuration in -`data/i18n/<localization>/<localization>.toml` (remember to commit the symbolic -link). +`data/i18n/<localization>/<localization>.toml` (remember to commit the symbolic link). For example, for German the strings live in `data/i18n/de/de.toml`, and `i18n/de.toml` is a symbolic link to `data/i18n/de/de.toml`. ### Localize the community code of conduct -Open a PR against the [`cncf/foundation`](https://github.com/cncf/foundation/tree/master/code-of-conduct-languages) repository to add the code of conduct in your language. - +Open a PR against the [`cncf/foundation`](https://github.com/cncf/foundation/tree/main/code-of-conduct-languages) +repository to add the code of conduct in your language. ### Setting up the OWNERS files -To set the roles of each user contributing to the localization, create an `OWNERS` file inside the language-specific subdirectory with: +To set the roles of each user contributing to the localization, create an `OWNERS` file inside the +language-specific subdirectory with: -- **reviewers**: A list of kubernetes teams with reviewer roles, in this case, the `sig-docs-**-reviews` team created in [Add your localization team in GitHub](#add-your-localization-team-in-github). -- **approvers**: A list of kubernetes teams with approvers roles, in this case, the `sig-docs-**-owners` team created in [Add your localization team in GitHub](#add-your-localization-team-in-github). -- **labels**: A list of GitHub labels to automatically apply to a PR, in this case, the language label created in [Configure the workflow](#configure-the-workflow). +- **reviewers**: A list of kubernetes teams with reviewer roles, in this case, the + `sig-docs-**-reviews` team created in + [Add your localization team in GitHub](#add-your-localization-team-in-github). +- **approvers**: A list of kubernetes teams with approvers roles, in this case, the + `sig-docs-**-owners` team created in + [Add your localization team in GitHub](#add-your-localization-team-in-github). +- **labels**: A list of GitHub labels to automatically apply to a PR, in this case, the language + label created in [Configure the workflow](#configure-the-workflow). More information about the `OWNERS` file can be found at [go.k8s.io/owners](https://go.k8s.io/owners). -The [Spanish OWNERS file](https://git.k8s.io/website/content/es/OWNERS), with language code `es`, looks like: +The [Spanish OWNERS file](https://git.k8s.io/website/content/es/OWNERS), +with language code `es`, looks like: ```yaml # See the OWNERS docs at https://go.k8s.io/owners @@ -185,9 +236,13 @@ labels: - language/es ``` -After adding the language-specific `OWNERS` file, update the [root `OWNERS_ALIASES`](https://git.k8s.io/website/OWNERS_ALIASES) file with the new Kubernetes teams for the localization, `sig-docs-**-owners` and `sig-docs-**-reviews`. +After adding the language-specific `OWNERS` file, update the [root +`OWNERS_ALIASES`](https://git.k8s.io/website/OWNERS_ALIASES) file with the new Kubernetes teams +for the localization, `sig-docs-**-owners` and `sig-docs-**-reviews`. -For each team, add the list of GitHub users requested in [Add your localization team in GitHub](#add-your-localization-team-in-github), in alphabetical order. +For each team, add the list of GitHub users requested in +[Add your localization team in GitHub](#add-your-localization-team-in-github), +in alphabetical order. ```diff --- a/OWNERS_ALIASES @@ -211,33 +266,45 @@ For each team, add the list of GitHub users requested in [Add your localization ### Open a pull request -Next, [open a pull request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) to add a localization to the `kubernetes/website` repository. +Next, [open a pull request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) to add a +localization to the `kubernetes/website` repository. +The PR must include all of the [minimum required content](#minimum-required-content) before it can +be approved. -The PR must include all of the [minimum required content](#minimum-required-content) before it can be approved. - -For an example of adding a new localization, see the PR to enable [docs in French](https://github.com/kubernetes/website/pull/12548). +For an example of adding a new localization, see the PR to enable +[docs in French](https://github.com/kubernetes/website/pull/12548). ### Add a localized README file -To guide other localization contributors, add a new [`README-**.md`](https://help.github.com/articles/about-readmes/) to the top level of [k/website](https://github.com/kubernetes/website/), where `**` is the two-letter language code. For example, a German README file would be `README-de.md`. +To guide other localization contributors, add a new +[`README-**.md`](https://help.github.com/articles/about-readmes/) to the top level of +[kubernetes/website](https://github.com/kubernetes/website/), where `**` is the two-letter language code. +For example, a German README file would be `README-de.md`. -Provide guidance to localization contributors in the localized `README-**.md` file. Include the same information contained in `README.md` as well as: +Provide guidance to localization contributors in the localized `README-**.md` file. +Include the same information contained in `README.md` as well as: - A point of contact for the localization project - Any information specific to the localization -After you create the localized README, add a link to the file from the main English `README.md`, and include contact information in English. You can provide a GitHub ID, email address, [Slack channel](https://slack.com/), or other method of contact. You must also provide a link to your localized Community Code of Conduct. +After you create the localized README, add a link to the file from the main English `README.md`, +and include contact information in English. You can provide a GitHub ID, email address, +[Slack channel](https://slack.com/), or other method of contact. You must also provide a link to your +localized Community Code of Conduct. ### Launching your new localization Once a localization meets requirements for workflow and minimum output, SIG Docs will: - Enable language selection on the website -- Publicize the localization's availability through [Cloud Native Computing Foundation](https://www.cncf.io/about/) (CNCF) channels, including the [Kubernetes blog](https://kubernetes.io/blog/). +- Publicize the localization's availability through + [Cloud Native Computing Foundation](https://www.cncf.io/about/)(CNCF) channels, including the + [Kubernetes blog](/blog/). ## Translating content -Localizing *all* of the Kubernetes documentation is an enormous task. It's okay to start small and expand over time. +Localizing *all* of the Kubernetes documentation is an enormous task. It's okay to start small and +expand over time. ### Minimum required content @@ -248,22 +315,29 @@ Description | URLs Home | [All heading and subheading URLs](/docs/home/) Setup | [All heading and subheading URLs](/docs/setup/) Tutorials | [Kubernetes Basics](/docs/tutorials/kubernetes-basics/), [Hello Minikube](/docs/tutorials/hello-minikube/) -Site strings | [All site strings](#Site-strings-in-i18n) in a new localized TOML file +Site strings | [All site strings](#site-strings-in-i18n) in a new localized TOML file +Releases | [All heading and subheading URLs](/releases) -Translated documents must reside in their own `content/**/` subdirectory, but otherwise follow the same URL path as the English source. For example, to prepare the [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) tutorial for translation into German, create a subfolder under the `content/de/` folder and copy the English source: +Translated documents must reside in their own `content/**/` subdirectory, but otherwise follow the +same URL path as the English source. For example, to prepare the +[Kubernetes Basics](/docs/tutorials/kubernetes-basics/) tutorial for translation into German, +create a subfolder under the `content/de/` folder and copy the English source: ```shell mkdir -p content/de/docs/tutorials cp content/en/docs/tutorials/kubernetes-basics.md content/de/docs/tutorials/kubernetes-basics.md ``` -Translation tools can speed up the translation process. For example, some editors offers plugins to quickly translate text. +Translation tools can speed up the translation process. For example, some editors offers plugins +to quickly translate text. {{< caution >}} -Machine-generated translation is insufficient on its own. Localization requires extensive human review to meet minimum standards of quality. +Machine-generated translation is insufficient on its own. Localization requires extensive human +review to meet minimum standards of quality. {{< /caution >}} -To ensure accuracy in grammar and meaning, members of your localization team should carefully review all machine-generated translations before publishing. +To ensure accuracy in grammar and meaning, members of your localization team should carefully +review all machine-generated translations before publishing. ### Source files @@ -274,17 +348,21 @@ To find source files for your target version: 1. Navigate to the Kubernetes website repository at https://github.com/kubernetes/website. 2. Select a branch for your target version from the following table: - Target version | Branch - -----|----- - Latest version | [`main`](https://github.com/kubernetes/website/tree/main) - Previous version | [`release-{{< skew prevMinorVersion >}}`](https://github.com/kubernetes/website/tree/release-{{< skew prevMinorVersion >}}) - Next version | [`dev-{{< skew nextMinorVersion >}}`](https://github.com/kubernetes/website/tree/dev-{{< skew nextMinorVersion >}}) -The `main` branch holds content for the current release `{{< latest-version >}}`. The release team will create a `{{< release-branch >}}` branch before the next release: v{{< skew nextMinorVersion >}}. + Target version | Branch + -----|----- + Latest version | [`main`](https://github.com/kubernetes/website/tree/main) + Previous version | [`release-{{< skew prevMinorVersion >}}`](https://github.com/kubernetes/website/tree/release-{{< skew prevMinorVersion >}}) + Next version | [`dev-{{< skew nextMinorVersion >}}`](https://github.com/kubernetes/website/tree/dev-{{< skew nextMinorVersion >}}) + +The `main` branch holds content for the current release `{{< latest-version >}}`. The release team +will create a `{{< release-branch >}}` branch before the next release: v{{< skew nextMinorVersion >}}. ### Site strings in i18n -Localizations must include the contents of [`data/i18n/en/en.toml`](https://github.com/kubernetes/website/blob/main/data/i18n/en/en.toml) in a new language-specific file. Using German as an example: `data/i18n/de/de.toml`. +Localizations must include the contents of +[`data/i18n/en/en.toml`](https://github.com/kubernetes/website/blob/main/data/i18n/en/en.toml) +in a new language-specific file. Using German as an example: `data/i18n/de/de.toml`. Add a new localization directory and file to `data/i18n/`. For example, with German (`de`): @@ -302,17 +380,22 @@ placeholder text for the search form: other = "Suchen" ``` -Localizing site strings lets you customize site-wide text and features: for example, the legal copyright text in the footer on each page. +Localizing site strings lets you customize site-wide text and features: for example, the legal +copyright text in the footer on each page. ### Language specific style guide and glossary -Some language teams have their own language-specific style guide and glossary. For example, see the [Korean Localization Guide](/ko/docs/contribute/localization_ko/). +Some language teams have their own language-specific style guide and glossary. +For example, see the [Korean Localization Guide](/ko/docs/contribute/localization_ko/). ### Language specific Zoom meetings -If the localization project needs a separate meeting time, contact a SIG Docs Co-Chair or Tech Lead to create a new reoccurring Zoom meeting and calendar invite. This is only needed when the the team is large enough to sustain and require a separate meeting. +If the localization project needs a separate meeting time, contact a SIG Docs Co-Chair or Tech +Lead to create a new reoccurring Zoom meeting and calendar invite. This is only needed when +the team is large enough to sustain and require a separate meeting. -Per CNCF policy, the localization teams must upload their meetings to the SIG Docs YouTube playlist. A SIG Docs Co-Chair or Tech Lead can help with the process until SIG Docs automates it. +Per CNCF policy, the localization teams must upload their meetings to the SIG Docs YouTube +playlist. A SIG Docs Co-Chair or Tech Lead can help with the process until SIG Docs automates it. ## Branching strategy @@ -322,44 +405,66 @@ when starting out and the localization is not yet live. To collaborate on a localization branch: -1. A team member of [@kubernetes/website-maintainers](https://github.com/orgs/kubernetes/teams/website-maintainers) opens a localization branch from a source branch on https://github.com/kubernetes/website. +1. A team member of + [@kubernetes/website-maintainers](https://github.com/orgs/kubernetes/teams/website-maintainers) + opens a localization branch from a source branch on https://github.com/kubernetes/website. - Your team approvers joined the `@kubernetes/website-maintainers` team when you [added your localization team](#add-your-localization-team-in-github) to the [`kubernetes/org`](https://github.com/kubernetes/org) repository. + Your team approvers joined the `@kubernetes/website-maintainers` team when you + [added your localization team](#add-your-localization-team-in-github) to the + [`kubernetes/org`](https://github.com/kubernetes/org) repository. - We recommend the following branch naming scheme: + We recommend the following branch naming scheme: - `dev-<source version>-<language code>.<team milestone>` + `dev-<source version>-<language code>.<team milestone>` - For example, an approver on a German localization team opens the localization branch `dev-1.12-de.1` directly against the k/website repository, based on the source branch for Kubernetes v1.12. + For example, an approver on a German localization team opens the localization branch + `dev-1.12-de.1` directly against the `kubernetes/website` repository, based on the source branch for + Kubernetes v1.12. 2. Individual contributors open feature branches based on the localization branch. - For example, a German contributor opens a pull request with changes to `kubernetes:dev-1.12-de.1` from `username:local-branch-name`. + For example, a German contributor opens a pull request with changes to + `kubernetes:dev-1.12-de.1` from `username:local-branch-name`. 3. Approvers review and merge feature branches into the localization branch. -4. Periodically, an approver merges the localization branch to its source branch by opening and approving a new pull request. Be sure to squash the commits before approving the pull request. +4. Periodically, an approver merges the localization branch to its source branch by opening and + approving a new pull request. Be sure to squash the commits before approving the pull request. -Repeat steps 1-4 as needed until the localization is complete. For example, subsequent German localization branches would be: `dev-1.12-de.2`, `dev-1.12-de.3`, etc. +Repeat steps 1-4 as needed until the localization is complete. For example, subsequent German +localization branches would be: `dev-1.12-de.2`, `dev-1.12-de.3`, etc. Teams must merge localized content into the same branch from which the content was sourced. - For example: + - a localization branch sourced from `main` must be merged into `main`. -- a localization branch sourced from `release-{{ skew "prevMinorVersion" }}` must be merged into `release-{{ skew "prevMinorVersion" }}`. +- a localization branch sourced from `release-{{% skew "prevMinorVersion" %}}` must be merged into + `release-{{% skew "prevMinorVersion" %}}`. {{< note >}} -If your localization branch was created from `main` branch but it is not merged into `main` before new release branch `{{< release-branch >}}` created, merge it into both `main` and new release branch `{{< release-branch >}}`. To merge your localization branch into new release branch `{{< release-branch >}}`, you need to switch upstream branch of your localization branch to `{{< release-branch >}}`. +If your localization branch was created from `main` branch but it is not merged into `main` before +new release branch `{{< release-branch >}}` created, merge it into both `main` and new release +branch `{{< release-branch >}}`. To merge your localization branch into new release branch +`{{< release-branch >}}`, you need to switch upstream branch of your localization branch to +`{{< release-branch >}}`. {{< /note >}} -At the beginning of every team milestone, it's helpful to open an issue comparing upstream changes between the previous localization branch and the current localization branch. There are two scripts for comparing upstream changes. [`upstream_changes.py`](https://github.com/kubernetes/website/tree/main/scripts#upstream_changespy) is useful for checking the changes made to a specific file. And [`diff_l10n_branches.py`](https://github.com/kubernetes/website/tree/main/scripts#diff_l10n_branchespy) is useful for creating a list of outdated files for a specific localization branch. +At the beginning of every team milestone, it's helpful to open an issue comparing upstream changes +between the previous localization branch and the current localization branch. +There are two scripts for comparing upstream changes. -While only approvers can open a new localization branch and merge pull requests, anyone can open a pull request for a new localization branch. No special permissions are required. +- [`upstream_changes.py`](https://github.com/kubernetes/website/tree/main/scripts#upstream_changespy) + is useful for checking the changes made to a specific file. And +- [`diff_l10n_branches.py`](https://github.com/kubernetes/website/tree/main/scripts#diff_l10n_branchespy) + is useful for creating a list of outdated files for a specific localization branch. -For more information about working from forks or directly from the repository, see ["fork and clone the repo"](#fork-and-clone-the-repo). +While only approvers can open a new localization branch and merge pull requests, anyone can open a +pull request for a new localization branch. No special permissions are required. + +For more information about working from forks or directly from the repository, see +["fork and clone the repo"](#fork-and-clone-the-repo). ## Upstream contributions SIG Docs welcomes upstream contributions and corrections to the English source. - diff --git a/content/en/docs/contribute/new-content/_index.md b/content/en/docs/contribute/new-content/_index.md index 4992a37654..93bbeb750b 100644 --- a/content/en/docs/contribute/new-content/_index.md +++ b/content/en/docs/contribute/new-content/_index.md @@ -1,4 +1,120 @@ --- title: Contributing new content +content_type: concept +main_menu: true weight: 20 --- + + + +<!-- overview --> + +This section contains information you should know before contributing new +content. +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> + +{{< mermaid >}} +flowchart LR + subgraph second[Before you begin] + direction TB + S[ ] -.- + A[Sign the CNCF CLA] --> B[Choose Git branch] + B --> C[One language per PR] + C --> F[Check out<br>contributor tools] + end + subgraph first[Contributing Basics] + direction TB + T[ ] -.- + D[Write docs in markdown<br>and build site with Hugo] --- E[source in GitHub] + E --- G['/content/../docs' folder contains docs<br>for multiple languages] + G --- H[Review Hugo page content<br>types and shortcodes] + end + + + first ----> second + + +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class A,B,C,D,E,F,G,H grey +class S,T spacewhite +class first,second white +{{</ mermaid >}} + +***Figure - Contributing new content preparation*** + +The figure above depicts the information you should know +prior to submitting new content. The information details follow. + + + +<!-- body --> + +## Contributing basics + +- Write Kubernetes documentation in Markdown and build the Kubernetes site + using [Hugo](https://gohugo.io/). +- Kubernetes documentation uses [CommonMark](https://commonmark.org/) as its flavor of Markdown. +- The source is in [GitHub](https://github.com/kubernetes/website). You can find + Kubernetes documentation at `/content/en/docs/`. Some of the reference + documentation is automatically generated from scripts in + the `update-imported-docs/` directory. +- [Page content types](/docs/contribute/style/page-content-types/) describe the + presentation of documentation content in Hugo. +- You can use [Docsy shortcodes](https://www.docsy.dev/docs/adding-content/shortcodes/) or [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) to contribute to Kubernetes documentation. +- In addition to the standard Hugo shortcodes, we use a number of + [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our + documentation to control the presentation of content. +- Documentation source is available in multiple languages in `/content/`. Each + language has its own folder with a two-letter code determined by the + [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) + . For example, English documentation source is stored in `/content/en/docs/`. +- For more information about contributing to documentation in multiple languages + or starting a new translation, + see [localization](/docs/contribute/localization). + +## Before you begin {#before-you-begin} + +### Sign the CNCF CLA {#sign-the-cla} + +All Kubernetes contributors **must** read +the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md) +and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md) +. + +Pull requests from contributors who haven't signed the CLA fail the automated +tests. The name and email you provide must match those found in +your `git config`, and your git name and email must match those used for the +CNCF CLA. + +### Choose which Git branch to use + +When opening a pull request, you need to know in advance which branch to base +your work on. + +Scenario | Branch +:---------|:------------ +Existing or new English language content for the current release | `main` +Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-<version>`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch. +Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information. + +If you're still not sure which branch to choose, ask in `#sig-docs` on Slack. + +{{< note >}} If you already submitted your pull request and you know that the +base branch was wrong, you (and only you, the submitter) can change it. {{< +/note >}} + +### Languages per PR + +Limit pull requests to one language per PR. If you need to make an identical +change to the same code sample in multiple languages, open a separate PR for +each language. + +## Tools for contributors + +The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools) +directory in the `kubernetes/website` repository contains tools to help your +contribution journey go more smoothly. + diff --git a/content/en/docs/contribute/new-content/blogs-case-studies.md b/content/en/docs/contribute/new-content/blogs-case-studies.md index ee06f204cb..3034f66ce8 100644 --- a/content/en/docs/contribute/new-content/blogs-case-studies.md +++ b/content/en/docs/contribute/new-content/blogs-case-studies.md @@ -16,80 +16,184 @@ Case studies require extensive review before they're approved. ## The Kubernetes Blog -The Kubernetes blog is used by the project to communicate new features, community reports, and any news that might be relevant to the Kubernetes community. -This includes end users and developers. -Most of the blog's content is about things happening in the core project, but we encourage you to submit about things happening elsewhere in the ecosystem too! +The Kubernetes blog is used by the project to communicate new features, community reports, and any +news that might be relevant to the Kubernetes community. This includes end users and developers. +Most of the blog's content is about things happening in the core project, but we encourage you to +submit about things happening elsewhere in the ecosystem too! Anyone can write a blog post and submit it for review. +### Submit a Post + +Blog posts should not be commercial in nature and should consist of original content that applies +broadly to the Kubernetes community. Appropriate blog content includes: + +- New Kubernetes capabilities +- Kubernetes projects updates +- Updates from Special Interest Groups +- Tutorials and walkthroughs +- Thought leadership around Kubernetes +- Kubernetes Partner OSS integration +- **Original content only** + +Unsuitable content includes: + +- Vendor product pitches +- Partner updates without an integration and customer story +- Syndicated posts (language translations ok) + +To submit a blog post, follow these steps: + +1. [Sign the CLA](https://github.com/kubernetes/community/blob/master/CLA.md) + if you have not yet done so. + +1. Have a look at the Markdown format for existing blog posts in the + [website repository](https://github.com/kubernetes/website/tree/master/content/en/blog/_posts). + +1. Write out your blog post in a text editor of your choice. + +1. On the same link from step 2, click the Create new file button. Paste your content into the editor. + Name the file to match the proposed title of the blog post, but don’t put the date in the file name. + The blog reviewers will work with you on the final file name and the date the blog will be published. + +1. When you save the file, GitHub will walk you through the pull request process. + +1. A blog post reviewer will review your submission and work with you on feedback and final details. + When the blog post is approved, the blog will be scheduled for publication. + ### Guidelines and expectations - Blog posts should not be vendor pitches. - - Articles must contain content that applies broadly to the Kubernetes community. For example, a submission should focus on upstream Kubernetes as opposed to vendor-specific configurations. Check the [Documentation style guide](/docs/contribute/style/content-guide/#what-s-allowed) for what is typically allowed on Kubernetes properties. - - Links should primarily be to the official Kubernetes documentation. When using external references, links should be diverse - For example a submission shouldn't contain only links back to a single company's blog. - - Sometimes this is a delicate balance. The [blog team](https://kubernetes.slack.com/messages/sig-docs-blog/) is there to give guidance on whether a post is appropriate for the Kubernetes blog, so don't hesitate to reach out. + + - Articles must contain content that applies broadly to the Kubernetes community. For example, a + submission should focus on upstream Kubernetes as opposed to vendor-specific configurations. + Check the [Documentation style guide](/docs/contribute/style/content-guide/#what-s-allowed) for + what is typically allowed on Kubernetes properties. + - Links should primarily be to the official Kubernetes documentation. When using external + references, links should be diverse - For example a submission shouldn't contain only links + back to a single company's blog. + - Sometimes this is a delicate balance. The [blog team](https://kubernetes.slack.com/messages/sig-docs-blog/) + is there to give guidance on whether a post is appropriate for the Kubernetes blog, so don't + hesitate to reach out. + - Blog posts are not published on specific dates. - - Articles are reviewed by community volunteers. We'll try our best to accommodate specific timing, but we make no guarantees. - - Many core parts of the Kubernetes projects submit blog posts during release windows, delaying publication times. Consider submitting during a quieter period of the release cycle. - - If you are looking for greater coordination on post release dates, coordinating with [CNCF marketing](https://www.cncf.io/about/contact/) is a more appropriate choice than submitting a blog post. - - Sometimes reviews can get backed up. If you feel your review isn't getting the attention it needs, you can reach out to the blog team via [this slack channel](https://kubernetes.slack.com/messages/sig-docs-blog/) to ask in real time. + + - Articles are reviewed by community volunteers. We'll try our best to accommodate specific + timing, but we make no guarantees. + - Many core parts of the Kubernetes projects submit blog posts during release windows, delaying + publication times. Consider submitting during a quieter period of the release cycle. + - If you are looking for greater coordination on post release dates, coordinating with + [CNCF marketing](https://www.cncf.io/about/contact/) is a more appropriate choice than submitting a blog post. + - Sometimes reviews can get backed up. If you feel your review isn't getting the attention it needs, + you can reach out to the blog team via [this slack channel](https://kubernetes.slack.com/messages/sig-docs-blog/) + to ask in real time. + - Blog posts should be relevant to Kubernetes users. - - Topics related to participation in or results of Kubernetes SIGs activities are always on topic (see the work in the [Upstream Marketing Team](https://github.com/kubernetes/community/blob/master/communication/marketing-team/blog-guidelines.md#upstream-marketing-blog-guidelines) for support on these posts). - - The components of Kubernetes are purposely modular, so tools that use existing integration points like CNI and CSI are on topic. - - Posts about other CNCF projects may or may not be on topic. We recommend asking the blog team before submitting a draft. - - Many CNCF projects have their own blog. These are often a better choice for posts. There are times of major feature or milestone for a CNCF project that users would be interested in reading on the Kubernetes blog. + + - Topics related to participation in or results of Kubernetes SIGs activities are always on + topic (see the work in the [Upstream Marketing Team](https://github.com/kubernetes/community/blob/master/communication/marketing-team/blog-guidelines.md#upstream-marketing-blog-guidelines) + for support on these posts). + - The components of Kubernetes are purposely modular, so tools that use existing integration + points like CNI and CSI are on topic. + - Posts about other CNCF projects may or may not be on topic. We recommend asking the blog team + before submitting a draft. + - Many CNCF projects have their own blog. These are often a better choice for posts. There are + times of major feature or milestone for a CNCF project that users would be interested in + reading on the Kubernetes blog. + - Blog posts about contributing to the Kubernetes project should be in the + [Kubernetes Contributors site](https://kubernetes.dev) + - Blog posts should be original content + - The official blog is not for repurposing existing content from a third party as new content. - - The [license](https://github.com/kubernetes/website/blob/main/LICENSE) for the blog allows commercial use of the content for commercial purposes, but not the other way around. + - The [license](https://github.com/kubernetes/website/blob/main/LICENSE) for the blog allows + commercial use of the content for commercial purposes, but not the other way around. + - Blog posts should aim to be future proof - - Given the development velocity of the project, we want evergreen content that won't require updates to stay accurate for the reader. - - It can be a better choice to add a tutorial or update official documentation than to write a high level overview as a blog post. - - Consider concentrating the long technical content as a call to action of the blog post, and focus on the problem space or why readers should care. + + - Given the development velocity of the project, we want evergreen content that won't require + updates to stay accurate for the reader. + - It can be a better choice to add a tutorial or update official documentation than to write a + high level overview as a blog post. + - Consider concentrating the long technical content as a call to action of the blog post, and + focus on the problem space or why readers should care. ### Technical Considerations for submitting a blog post -Submissions need to be in Markdown format to be used by the [Hugo](https://gohugo.io/) generator for the blog. There are [many resources available](https://gohugo.io/documentation/) on how to use this technology stack. +Submissions need to be in Markdown format to be used by the [Hugo](https://gohugo.io/) generator +for the blog. There are [many resources available](https://gohugo.io/documentation/) on how to use +this technology stack. -We recognize that this requirement makes the process more difficult for less-familiar folks to submit, and we're constantly looking at solutions to lower this bar. If you have ideas on how to lower the barrier, please volunteer to help out. +We recognize that this requirement makes the process more difficult for less-familiar folks to +submit, and we're constantly looking at solutions to lower this bar. If you have ideas on how to +lower the barrier, please volunteer to help out. -The SIG Docs [blog subproject](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject) manages the review process for blog posts. For more information, see [Submit a post](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject#submit-a-post). +The SIG Docs [blog subproject](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject) +manages the review process for blog posts. For more information, see +[Submit a post](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject#submit-a-post). To submit a blog post follow these directions: -- [Open a pull request](/docs/contribute/new-content/open-a-pr/#fork-the-repo) with a new blog post. New blog posts go under the [`content/en/blog/_posts`](https://github.com/kubernetes/website/tree/main/content/en/blog/_posts) directory. +- [Open a pull request](/docs/contribute/new-content/open-a-pr/#fork-the-repo) with a new blog post. + New blog posts go under the [`content/en/blog/_posts`](https://github.com/kubernetes/website/tree/main/content/en/blog/_posts) + directory. -- Ensure that your blog post follows the correct naming conventions and the following frontmatter (metadata) information: +- Ensure that your blog post follows the correct naming conventions and the following frontmatter + (metadata) information: - - The Markdown file name must follow the format `YYYY-MM-DD-Your-Title-Here.md`. For example, `2020-02-07-Deploying-External-OpenStack-Cloud-Provider-With-Kubeadm.md`. - - Do **not** include dots in the filename. A name like `2020-01-01-whats-new-in-1.19.md` causes failures during a build. + - The Markdown file name must follow the format `YYYY-MM-DD-Your-Title-Here.md`. For example, + `2020-02-07-Deploying-External-OpenStack-Cloud-Provider-With-Kubeadm.md`. + - Do **not** include dots in the filename. A name like `2020-01-01-whats-new-in-1.19.md` causes + failures during a build. - The front matter must include the following: - ```yaml - --- - layout: blog - title: "Your Title Here" - date: YYYY-MM-DD - slug: text-for-URL-link-here-no-spaces - --- - ``` - - The first or initial commit message should be a short summary of the work being done and should stand alone as a description of the blog post. Please note that subsequent edits to your blog will be squashed into this main commit, so it should be as useful as possible. + ```yaml + --- + layout: blog + title: "Your Title Here" + date: YYYY-MM-DD + slug: text-for-URL-link-here-no-spaces + --- + ``` + + - The first or initial commit message should be a short summary of the work being done and + should stand alone as a description of the blog post. Please note that subsequent edits to + your blog will be squashed into this main commit, so it should be as useful as possible. + - Examples of a good commit message: - - _Add blog post on the foo kubernetes feature_ - - _blog: foobar announcement_ + - _Add blog post on the foo kubernetes feature_ + - _blog: foobar announcement_ - Examples of bad commit message: - _Add blog post_ - _._ - _initial commit_ - _draft post_ - - The blog team will then review your PR and give you comments on things you might need to fix. After that the bot will merge your PR and your blog post will be published. + - The blog team will then review your PR and give you comments on things you might need to fix. + After that the bot will merge your PR and your blog post will be published. + + - If the content of the blog post contains only content that is not expected to require updates + to stay accurate for the reader, it can be marked as evergreen and exempted from the automatic + warning about outdated content added to blog posts older than one year. + + - To mark a blog post as evergreen, add this to the front matter: + + ```yaml + evergreen: true + ``` + - Examples of content that should not be marked evergreen: + - **Tutorials** that only apply to specific releases or versions and not all future versions + - References to pre-GA APIs or features ## Submit a case study -Case studies highlight how organizations are using Kubernetes to solve -real-world problems. The Kubernetes marketing team and members of the {{< glossary_tooltip text="CNCF" term_id="cncf" >}} collaborate with you on all case studies. +Case studies highlight how organizations are using Kubernetes to solve real-world problems. The +Kubernetes marketing team and members of the {{< glossary_tooltip text="CNCF" term_id="cncf" >}} +collaborate with you on all case studies. Have a look at the source for the [existing case studies](https://github.com/kubernetes/website/tree/main/content/en/case-studies). -Refer to the [case study guidelines](https://github.com/cncf/foundation/blob/master/case-study-guidelines.md) and submit your request as outlined in the guidelines. +Refer to the [case study guidelines](https://github.com/cncf/foundation/blob/master/case-study-guidelines.md) +and submit your request as outlined in the guidelines. + diff --git a/content/en/docs/contribute/new-content/new-features.md b/content/en/docs/contribute/new-content/new-features.md index 268c447402..d40195c32d 100644 --- a/content/en/docs/contribute/new-content/new-features.md +++ b/content/en/docs/contribute/new-content/new-features.md @@ -37,7 +37,7 @@ the techniques described in ### Find out about upcoming features To find out about upcoming features, attend the weekly SIG Release meeting (see -the [community](https://kubernetes.io/community/) page for upcoming meetings) +the [community](/community/) page for upcoming meetings) and monitor the release-specific documentation in the [kubernetes/sig-release](https://github.com/kubernetes/sig-release/) repository. Each release has a sub-directory in the [/sig-release/tree/master/releases/](https://github.com/kubernetes/sig-release/tree/master/releases) diff --git a/content/en/docs/contribute/new-content/open-a-pr.md b/content/en/docs/contribute/new-content/open-a-pr.md index 973a29167d..50b5a00608 100644 --- a/content/en/docs/contribute/new-content/open-a-pr.md +++ b/content/en/docs/contribute/new-content/open-a-pr.md @@ -15,53 +15,89 @@ upcoming Kubernetes release, see [Document a new feature](/docs/contribute/new-content/new-features/). {{< /note >}} -To contribute new content pages or improve existing content pages, open a pull request (PR). Make sure you follow all the requirements in the [Before you begin](/docs/contribute/new-content/overview/#before-you-begin) section. - -If your change is small, or you're unfamiliar with git, read [Changes using GitHub](#changes-using-github) to learn how to edit a page. - -If your changes are large, read [Work from a local fork](#fork-the-repo) to learn how to make changes locally on your computer. +To contribute new content pages or improve existing content pages, open a pull request (PR). +Make sure you follow all the requirements in the +[Before you begin](/docs/contribute/new-content/) section. +If your change is small, or you're unfamiliar with git, read +[Changes using GitHub](#changes-using-github) to learn how to edit a page. +If your changes are large, read [Work from a local fork](#fork-the-repo) to learn how to make +changes locally on your computer. <!-- body --> ## Changes using GitHub If you're less experienced with git workflows, here's an easier method of -opening a pull request. +opening a pull request. Figure 1 outlines the steps and the details follow. -1. On the page where you see the issue, select the pencil icon at the top right. - You can also scroll to the bottom of the page and select **Edit this page**. +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> -2. Make your changes in the GitHub markdown editor. +{{< mermaid >}} +flowchart LR +A([fa:fa-user New<br>Contributor]) --- id1[(K8s/Website<br>GitHub)] +subgraph tasks[Changes using GitHub] +direction TB + 0[ ] -.- + 1[1. Edit this page] --> 2[2. Use GitHub markdown<br>editor to make changes] + 2 --> 3[3. fill in Propose file change] -3. Below the editor, fill in the **Propose file change** - form. In the first field, give your commit message a title. In - the second field, provide a description. +end +subgraph tasks2[ ] +direction TB +4[4. select Propose file change] --> 5[5. select Create pull request] --> 6[6. fill in Open a pull request] +6 --> 7[7. select Create pull request] +end - {{< note >}} - Do not use any [GitHub Keywords](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in your commit message. You can add those to the pull request - description later. - {{< /note >}} +id1 --> tasks --> tasks2 -4. Select **Propose file change**. +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class A,1,2,3,4,5,6,7 grey +class 0 spacewhite +class tasks,tasks2 white +class id1 k8s +{{</ mermaid >}} -5. Select **Create pull request**. +Figure 1. Steps for opening a PR using GitHub. -6. The **Open a pull request** screen appears. Fill in the form: +1. On the page where you see the issue, select the pencil icon at the top right. + You can also scroll to the bottom of the page and select **Edit this page**. - - The **Subject** field of the pull request defaults to the commit summary. - You can change it if needed. - - The **Body** contains your extended commit message, if you have one, - and some template text. Add the - details the template text asks for, then delete the extra template text. - - Leave the **Allow edits from maintainers** checkbox selected. +1. Make your changes in the GitHub markdown editor. - {{< note >}} - PR descriptions are a great way to help reviewers understand your change. For more information, see [Opening a PR](#open-a-pr). - {{</ note >}} +1. Below the editor, fill in the **Propose file change** form. + In the first field, give your commit message a title. + In the second field, provide a description. -7. Select **Create pull request**. + {{< note >}} + Do not use any [GitHub Keywords](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) + in your commit message. You can add those to the pull request description later. + {{< /note >}} + +1. Select **Propose file change**. + +1. Select **Create pull request**. + +1. The **Open a pull request** screen appears. Fill in the form: + + - The **Subject** field of the pull request defaults to the commit summary. + You can change it if needed. + - The **Body** contains your extended commit message, if you have one, + and some template text. Add the + details the template text asks for, then delete the extra template text. + - Leave the **Allow edits from maintainers** checkbox selected. + + {{< note >}} + PR descriptions are a great way to help reviewers understand your change. + For more information, see [Opening a PR](#open-a-pr). + {{</ note >}} + +1. Select **Create pull request**. ### Addressing feedback in GitHub @@ -73,12 +109,12 @@ leave a comment with their GitHub username in it. If a reviewer asks you to make changes: 1. Go to the **Files changed** tab. -2. Select the pencil (edit) icon on any files changed by the -pull request. -3. Make the changes requested. -4. Commit the changes. +1. Select the pencil (edit) icon on any files changed by the pull request. +1. Make the changes requested. +1. Commit the changes. -If you are waiting on a reviewer, reach out once every 7 days. You can also post a message in the `#sig-docs` Slack channel. +If you are waiting on a reviewer, reach out once every 7 days. You can also post a message in the +`#sig-docs` Slack channel. When your review is complete, a reviewer merges your PR and your changes go live a few minutes later. @@ -87,80 +123,118 @@ When your review is complete, a reviewer merges your PR and your changes go live If you're more experienced with git, or if your changes are larger than a few lines, work from a local fork. -Make sure you have [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed on your computer. You can also use a git UI application. +Make sure you have [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed +on your computer. You can also use a git UI application. + +Figure 2 shows the steps to follow when you work from a local fork. The details for each step follow. + +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> + +{{< mermaid >}} +flowchart LR +1[Fork the K8s/website<br>repository] --> 2[Create local clone<br>and set upstream] +subgraph changes[Your changes] +direction TB +S[ ] -.- +3[Create a branch<br>example: my_new_branch] --> 3a[Make changes using<br>text editor] --> 4["Preview your changes<br>locally using Hugo<br>(localhost:1313)<br>or build container image"] +end +subgraph changes2[Commit / Push] +direction TB +T[ ] -.- +5[Commit your changes] --> 6[Push commit to<br>origin/my_new_branch] +end + +2 --> changes --> changes2 + +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class 1,2,3,3a,4,5,6 grey +class S,T spacewhite +class changes,changes2 white +{{</ mermaid >}} + +Figure 2. Working from a local fork to make your changes. ### Fork the kubernetes/website repository 1. Navigate to the [`kubernetes/website`](https://github.com/kubernetes/website/) repository. -2. Select **Fork**. +1. Select **Fork**. ### Create a local clone and set the upstream -3. In a terminal window, clone your fork and update the [Docsy Hugo theme](https://github.com/google/docsy#readme): +1. In a terminal window, clone your fork and update the [Docsy Hugo theme](https://github.com/google/docsy#readme): - ```bash - git clone git@github.com/<github_username>/website - cd website - git submodule update --init --recursive --depth 1 - ``` + ```shell + git clone git@github.com/<github_username>/website + cd website + git submodule update --init --recursive --depth 1 + ``` -4. Navigate to the new `website` directory. Set the `kubernetes/website` repository as the `upstream` remote: +1. Navigate to the new `website` directory. Set the `kubernetes/website` repository as the `upstream` remote: - ```bash - cd website + ```shell + cd website - git remote add upstream https://github.com/kubernetes/website.git - ``` + git remote add upstream https://github.com/kubernetes/website.git + ``` -5. Confirm your `origin` and `upstream` repositories: +1. Confirm your `origin` and `upstream` repositories: - ```bash - git remote -v - ``` + ```shell + git remote -v + ``` - Output is similar to: + Output is similar to: - ```bash - origin git@github.com:<github_username>/website.git (fetch) - origin git@github.com:<github_username>/website.git (push) - upstream https://github.com/kubernetes/website.git (fetch) - upstream https://github.com/kubernetes/website.git (push) - ``` + ```none + origin git@github.com:<github_username>/website.git (fetch) + origin git@github.com:<github_username>/website.git (push) + upstream https://github.com/kubernetes/website.git (fetch) + upstream https://github.com/kubernetes/website.git (push) + ``` -6. Fetch commits from your fork's `origin/main` and `kubernetes/website`'s `upstream/main`: +1. Fetch commits from your fork's `origin/main` and `kubernetes/website`'s `upstream/main`: - ```bash - git fetch origin - git fetch upstream - ``` + ```shell + git fetch origin + git fetch upstream + ``` - This makes sure your local repository is up to date before you start making changes. + This makes sure your local repository is up to date before you start making changes. - {{< note >}} - This workflow is different than the [Kubernetes Community GitHub Workflow](https://github.com/kubernetes/community/blob/master/contributors/guide/github-workflow.md). You do not need to merge your local copy of `main` with `upstream/main` before pushing updates to your fork. - {{< /note >}} + {{< note >}} + This workflow is different than the + [Kubernetes Community GitHub Workflow](https://github.com/kubernetes/community/blob/master/contributors/guide/github-workflow.md). + You do not need to merge your local copy of `main` with `upstream/main` before pushing updates + to your fork. + {{< /note >}} ### Create a branch 1. Decide which branch base to your work on: - - For improvements to existing content, use `upstream/main`. - - For new content about existing features, use `upstream/main`. - - For localized content, use the localization's conventions. For more information, see [localizing Kubernetes documentation](/docs/contribute/localization/). - - For new features in an upcoming Kubernetes release, use the feature branch. For more information, see [documenting for a release](/docs/contribute/new-content/new-features/). - - For long-running efforts that multiple SIG Docs contributors collaborate on, - like content reorganization, use a specific feature branch created for that - effort. + - For improvements to existing content, use `upstream/main`. + - For new content about existing features, use `upstream/main`. + - For localized content, use the localization's conventions. For more information, see + [localizing Kubernetes documentation](/docs/contribute/localization/). + - For new features in an upcoming Kubernetes release, use the feature branch. For more + information, see [documenting for a release](/docs/contribute/new-content/new-features/). + - For long-running efforts that multiple SIG Docs contributors collaborate on, + like content reorganization, use a specific feature branch created for that effort. - If you need help choosing a branch, ask in the `#sig-docs` Slack channel. + If you need help choosing a branch, ask in the `#sig-docs` Slack channel. -2. Create a new branch based on the branch identified in step 1. This example assumes the base branch is `upstream/main`: +1. Create a new branch based on the branch identified in step 1. This example assumes the base + branch is `upstream/main`: - ```bash - git checkout -b <my_new_branch> upstream/main - ``` + ```shell + git checkout -b <my_new_branch> upstream/main + ``` -3. Make your changes using a text editor. +1. Make your changes using a text editor. At any time, use the `git status` command to see what files you've changed. @@ -170,167 +244,202 @@ When you are ready to submit a pull request, commit your changes. 1. In your local repository, check which files you need to commit: - ```bash - git status - ``` + ```shell + git status + ``` - Output is similar to: + Output is similar to: - ```bash - On branch <my_new_branch> - Your branch is up to date with 'origin/<my_new_branch>'. + ```none + On branch <my_new_branch> + Your branch is up to date with 'origin/<my_new_branch>'. - Changes not staged for commit: - (use "git add <file>..." to update what will be committed) - (use "git checkout -- <file>..." to discard changes in working directory) + Changes not staged for commit: + (use "git add <file>..." to update what will be committed) + (use "git checkout -- <file>..." to discard changes in working directory) - modified: content/en/docs/contribute/new-content/contributing-content.md + modified: content/en/docs/contribute/new-content/contributing-content.md - no changes added to commit (use "git add" and/or "git commit -a") - ``` + no changes added to commit (use "git add" and/or "git commit -a") + ``` -2. Add the files listed under **Changes not staged for commit** to the commit: +1. Add the files listed under **Changes not staged for commit** to the commit: - ```bash - git add <your_file_name> - ``` + ```shell + git add <your_file_name> + ``` - Repeat this for each file. + Repeat this for each file. -3. After adding all the files, create a commit: +1. After adding all the files, create a commit: - ```bash - git commit -m "Your commit message" - ``` + ```shell + git commit -m "Your commit message" + ``` - {{< note >}} - Do not use any [GitHub Keywords](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in your commit message. You can add those to the pull request - description later. - {{< /note >}} + {{< note >}} + Do not use any [GitHub Keywords](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) + in your commit message. You can add those to the pull request + description later. + {{< /note >}} -4. Push your local branch and its new commit to your remote fork: +1. Push your local branch and its new commit to your remote fork: - ```bash - git push origin <my_new_branch> - ``` + ```shell + git push origin <my_new_branch> + ``` ### Preview your changes locally {#preview-locally} -It's a good idea to preview your changes locally before pushing them or opening a pull request. A preview lets you catch build errors or markdown formatting problems. +It's a good idea to preview your changes locally before pushing them or opening a pull request. +A preview lets you catch build errors or markdown formatting problems. -You can either build the website's container image or run Hugo locally. Building the container image is slower but displays [Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/), which can be useful for debugging. +You can either build the website's container image or run Hugo locally. Building the container +image is slower but displays [Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/), which can +be useful for debugging. {{< tabs name="tab_with_hugo" >}} {{% tab name="Hugo in a container" %}} {{< note >}} -The commands below use Docker as default container engine. Set the `CONTAINER_ENGINE` environment variable to override this behaviour. +The commands below use Docker as default container engine. Set the `CONTAINER_ENGINE` environment +variable to override this behaviour. {{< /note >}} -1. Build the image locally: +1. Build the container image locally + _You only need this step if you are testing a change to the Hugo tool itself_ - ```bash - # Use docker (default) - make container-image + ```shell + # Run this in a terminal (if required) + make container-image + ``` - ### OR ### +1. Start Hugo in a container: - # Use podman - CONTAINER_ENGINE=podman make container-image - ``` + ```shell + # Run this in a terminal + make container-serve + ``` -2. After building the `kubernetes-hugo` image locally, build and serve the site: +1. In a web browser, navigate to `https://localhost:1313`. Hugo watches the + changes and rebuilds the site as needed. - ```bash - # Use docker (default) - make container-serve - - ### OR ### - - # Use podman - CONTAINER_ENGINE=podman make container-serve - ``` - -3. In a web browser, navigate to `https://localhost:1313`. Hugo watches the - changes and rebuilds the site as needed. - -4. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, - or close the terminal window. +1. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, + or close the terminal window. {{% /tab %}} {{% tab name="Hugo on the command line" %}} Alternately, install and use the `hugo` command on your computer: -1. Install the [Hugo](https://gohugo.io/getting-started/installing/) version specified in [`website/netlify.toml`](https://raw.githubusercontent.com/kubernetes/website/main/netlify.toml). +1. Install the [Hugo](https://gohugo.io/getting-started/installing/) version specified in + [`website/netlify.toml`](https://raw.githubusercontent.com/kubernetes/website/main/netlify.toml). -2. If you have not updated your website repository, the `website/themes/docsy` directory is empty. - The site cannot build without a local copy of the theme. To update the website theme, run: +1. If you have not updated your website repository, the `website/themes/docsy` directory is empty. + The site cannot build without a local copy of the theme. To update the website theme, run: - ```bash - git submodule update --init --recursive --depth 1 - ``` + ```shell + git submodule update --init --recursive --depth 1 + ``` -3. In a terminal, go to your Kubernetes website repository and start the Hugo server: +1. In a terminal, go to your Kubernetes website repository and start the Hugo server: - ```bash - cd <path_to_your_repo>/website - hugo server --buildFuture - ``` + ```shell + cd <path_to_your_repo>/website + hugo server --buildFuture + ``` -4. In a web browser, navigate to `https://localhost:1313`. Hugo watches the - changes and rebuilds the site as needed. +1. In a web browser, navigate to `https://localhost:1313`. Hugo watches the + changes and rebuilds the site as needed. -5. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, - or close the terminal window. +1. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, + or close the terminal window. {{% /tab %}} {{< /tabs >}} ### Open a pull request from your fork to kubernetes/website {#open-a-pr} +Figure 3 shows the steps to open a PR from your fork to the K8s/website. The details follow. + +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> + +{{< mermaid >}} +flowchart LR +subgraph first[ ] +direction TB +1[1. Go to K8s/website repository] --> 2[2. Select New Pull Request] +2 --> 3[3. Select compare across forks] +3 --> 4[4. Select your fork from<br>head repository drop-down menu] +end +subgraph second [ ] +direction TB +5[5. Select your branch from<br>the compare drop-down menu] --> 6[6. Select Create Pull Request] +6 --> 7[7. Add a description<br>to your PR] +7 --> 8[8. Select Create pull request] +end + +first --> second + +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +class 1,2,3,4,5,6,7,8 grey +class first,second white +{{</ mermaid >}} + +Figure 3. Steps to open a PR from your fork to the K8s/website. + 1. In a web browser, go to the [`kubernetes/website`](https://github.com/kubernetes/website/) repository. -2. Select **New Pull Request**. -3. Select **compare across forks**. -4. From the **head repository** drop-down menu, select your fork. -5. From the **compare** drop-down menu, select your branch. -6. Select **Create Pull Request**. -7. Add a description for your pull request: +1. Select **New Pull Request**. +1. Select **compare across forks**. +1. From the **head repository** drop-down menu, select your fork. +1. From the **compare** drop-down menu, select your branch. +1. Select **Create Pull Request**. +1. Add a description for your pull request: + - **Title** (50 characters or less): Summarize the intent of the change. - **Description**: Describe the change in more detail. - - If there is a related GitHub issue, include `Fixes #12345` or `Closes #12345` in the description. GitHub's automation closes the mentioned issue after merging the PR if used. If there are other related PRs, link those as well. - - If you want advice on something specific, include any questions you'd like reviewers to think about in your description. -8. Select the **Create pull request** button. + - If there is a related GitHub issue, include `Fixes #12345` or `Closes #12345` in the + description. GitHub's automation closes the mentioned issue after merging the PR if used. + If there are other related PRs, link those as well. + - If you want advice on something specific, include any questions you'd like reviewers to + think about in your description. - Congratulations! Your pull request is available in [Pull requests](https://github.com/kubernetes/website/pulls). +1. Select the **Create pull request** button. +Congratulations! Your pull request is available in [Pull requests](https://github.com/kubernetes/website/pulls). -After opening a PR, GitHub runs automated tests and tries to deploy a preview using [Netlify](https://www.netlify.com/). +After opening a PR, GitHub runs automated tests and tries to deploy a preview using +[Netlify](https://www.netlify.com/). - - If the Netlify build fails, select **Details** for more information. - - If the Netlify build succeeds, select **Details** opens a staged version of the Kubernetes website with your changes applied. This is how reviewers check your changes. +- If the Netlify build fails, select **Details** for more information. +- If the Netlify build succeeds, select **Details** opens a staged version of the Kubernetes + website with your changes applied. This is how reviewers check your changes. -GitHub also automatically assigns labels to a PR, to help reviewers. You can add them too, if needed. For more information, see [Adding and removing issue labels](/docs/contribute/review/for-approvers/#adding-and-removing-issue-labels). +GitHub also automatically assigns labels to a PR, to help reviewers. You can add them too, if +needed. For more information, see [Adding and removing issue labels](/docs/contribute/review/for-approvers/#adding-and-removing-issue-labels). ### Addressing feedback locally 1. After making your changes, amend your previous commit: - ```bash - git commit -a --amend - ``` + ```shell + git commit -a --amend + ``` - - `-a`: commits all changes - - `--amend`: amends the previous commit, rather than creating a new one + - `-a`: commits all changes + - `--amend`: amends the previous commit, rather than creating a new one -2. Update your commit message if needed. +1. Update your commit message if needed. -3. Use `git push origin <my_new_branch>` to push your changes and re-run the Netlify tests. +1. Use `git push origin <my_new_branch>` to push your changes and re-run the Netlify tests. - {{< note >}} - If you use `git commit -m` instead of amending, you must [squash your commits](#squashing-commits) before merging. - {{< /note >}} + {{< note >}} + If you use `git commit -m` instead of amending, you must [squash your commits](#squashing-commits) + before merging. + {{< /note >}} #### Changes from reviewers @@ -338,90 +447,97 @@ Sometimes reviewers commit to your pull request. Before making any other changes 1. Fetch commits from your remote fork and rebase your working branch: - ```bash - git fetch origin - git rebase origin/<your-branch-name> - ``` + ```shell + git fetch origin + git rebase origin/<your-branch-name> + ``` -2. After rebasing, force-push new changes to your fork: +1. After rebasing, force-push new changes to your fork: - ```bash - git push --force-with-lease origin <your-branch-name> - ``` + ```shell + git push --force-with-lease origin <your-branch-name> + ``` #### Merge conflicts and rebasing {{< note >}} -For more information, see [Git Branching - Basic Branching and Merging](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging#_basic_merge_conflicts), [Advanced Merging](https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging), or ask in the `#sig-docs` Slack channel for help. +For more information, see [Git Branching - Basic Branching and Merging](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging#_basic_merge_conflicts), +[Advanced Merging](https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging), or ask in the +`#sig-docs` Slack channel for help. {{< /note >}} -If another contributor commits changes to the same file in another PR, it can create a merge conflict. You must resolve all merge conflicts in your PR. +If another contributor commits changes to the same file in another PR, it can create a merge +conflict. You must resolve all merge conflicts in your PR. 1. Update your fork and rebase your local branch: - ```bash - git fetch origin - git rebase origin/<your-branch-name> - ``` + ```shell + git fetch origin + git rebase origin/<your-branch-name> + ``` - Then force-push the changes to your fork: + Then force-push the changes to your fork: - ```bash - git push --force-with-lease origin <your-branch-name> - ``` + ```shell + git push --force-with-lease origin <your-branch-name> + ``` -2. Fetch changes from `kubernetes/website`'s `upstream/main` and rebase your branch: +1. Fetch changes from `kubernetes/website`'s `upstream/main` and rebase your branch: - ```bash - git fetch upstream - git rebase upstream/main - ``` + ```shell + git fetch upstream + git rebase upstream/main + ``` -3. Inspect the results of the rebase: +1. Inspect the results of the rebase: - ```bash - git status - ``` + ```shell + git status + ``` - This results in a number of files marked as conflicted. + This results in a number of files marked as conflicted. -4. Open each conflicted file and look for the conflict markers: `>>>`, `<<<`, and `===`. Resolve the conflict and delete the conflict marker. +1. Open each conflicted file and look for the conflict markers: `>>>`, `<<<`, and `===`. + Resolve the conflict and delete the conflict marker. - {{< note >}} - For more information, see [How conflicts are presented](https://git-scm.com/docs/git-merge#_how_conflicts_are_presented). - {{< /note >}} + {{< note >}} + For more information, see [How conflicts are presented](https://git-scm.com/docs/git-merge#_how_conflicts_are_presented). + {{< /note >}} -5. Add the files to the changeset: +1. Add the files to the changeset: - ```bash - git add <filename> - ``` -6. Continue the rebase: + ```shell + git add <filename> + ``` - ```bash - git rebase --continue - ``` +1. Continue the rebase: -7. Repeat steps 2 to 5 as needed. + ```shell + git rebase --continue + ``` - After applying all commits, the `git status` command shows that the rebase is complete. +1. Repeat steps 2 to 5 as needed. -8. Force-push the branch to your fork: + After applying all commits, the `git status` command shows that the rebase is complete. - ```bash - git push --force-with-lease origin <your-branch-name> - ``` +1. Force-push the branch to your fork: - The pull request no longer shows any conflicts. + ```shell + git push --force-with-lease origin <your-branch-name> + ``` + The pull request no longer shows any conflicts. ### Squashing commits {{< note >}} -For more information, see [Git Tools - Rewriting History](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History), or ask in the `#sig-docs` Slack channel for help. +For more information, see [Git Tools - Rewriting History](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History), +or ask in the `#sig-docs` Slack channel for help. {{< /note >}} -If your PR has multiple commits, you must squash them into a single commit before merging your PR. You can check the number of commits on your PR's **Commits** tab or by running the `git log` command locally. +If your PR has multiple commits, you must squash them into a single commit before merging your PR. +You can check the number of commits on your PR's **Commits** tab or by running the `git log` +command locally. {{< note >}} This topic assumes `vim` as the command line text editor. @@ -429,82 +545,83 @@ This topic assumes `vim` as the command line text editor. 1. Start an interactive rebase: - ```bash - git rebase -i HEAD~<number_of_commits_in_branch> - ``` + ```shell + git rebase -i HEAD~<number_of_commits_in_branch> + ``` - Squashing commits is a form of rebasing. The `-i` switch tells git you want to rebase interactively. `HEAD~<number_of_commits_in_branch` indicates how many commits to look at for the rebase. + Squashing commits is a form of rebasing. The `-i` switch tells git you want to rebase interactively. + `HEAD~<number_of_commits_in_branch` indicates how many commits to look at for the rebase. - Output is similar to: + Output is similar to: - ```bash - pick d875112ca Original commit - pick 4fa167b80 Address feedback 1 - pick 7d54e15ee Address feedback 2 + ```none + pick d875112ca Original commit + pick 4fa167b80 Address feedback 1 + pick 7d54e15ee Address feedback 2 - # Rebase 3d18sf680..7d54e15ee onto 3d183f680 (3 commands) + # Rebase 3d18sf680..7d54e15ee onto 3d183f680 (3 commands) - ... + ... - # These lines can be re-ordered; they are executed from top to bottom. - ``` + # These lines can be re-ordered; they are executed from top to bottom. + ``` - The first section of the output lists the commits in the rebase. The second section lists the options for each commit. Changing the word `pick` changes the status of the commit once the rebase is complete. + The first section of the output lists the commits in the rebase. The second section lists the + options for each commit. Changing the word `pick` changes the status of the commit once the rebase + is complete. - For the purposes of rebasing, focus on `squash` and `pick`. + For the purposes of rebasing, focus on `squash` and `pick`. - {{< note >}} - For more information, see [Interactive Mode](https://git-scm.com/docs/git-rebase#_interactive_mode). - {{< /note >}} + {{< note >}} + For more information, see [Interactive Mode](https://git-scm.com/docs/git-rebase#_interactive_mode). + {{< /note >}} -2. Start editing the file. +1. Start editing the file. - Change the original text: + Change the original text: - ```bash - pick d875112ca Original commit - pick 4fa167b80 Address feedback 1 - pick 7d54e15ee Address feedback 2 - ``` + ```none + pick d875112ca Original commit + pick 4fa167b80 Address feedback 1 + pick 7d54e15ee Address feedback 2 + ``` - To: + To: - ```bash - pick d875112ca Original commit - squash 4fa167b80 Address feedback 1 - squash 7d54e15ee Address feedback 2 - ``` + ```none + pick d875112ca Original commit + squash 4fa167b80 Address feedback 1 + squash 7d54e15ee Address feedback 2 + ``` - This squashes commits `4fa167b80 Address feedback 1` and `7d54e15ee Address feedback 2` into `d875112ca Original commit`, leaving only `d875112ca Original commit` as a part of the timeline. + This squashes commits `4fa167b80 Address feedback 1` and `7d54e15ee Address feedback 2` into + `d875112ca Original commit`, leaving only `d875112ca Original commit` as a part of the timeline. -3. Save and exit your file. +1. Save and exit your file. -4. Push your squashed commit: +1. Push your squashed commit: - ```bash - git push --force-with-lease origin <branch_name> - ``` + ```shell + git push --force-with-lease origin <branch_name> + ``` ## Contribute to other repos -The [Kubernetes project](https://github.com/kubernetes) contains 50+ repositories. Many of these repositories contain documentation: user-facing help text, error messages, API references or code comments. +The [Kubernetes project](https://github.com/kubernetes) contains 50+ repositories. Many of these +repositories contain documentation: user-facing help text, error messages, API references or code +comments. -If you see text you'd like to improve, use GitHub to search all repositories in the Kubernetes organization. -This can help you figure out where to submit your issue or PR. - -Each repository has its own processes and procedures. Before you file an -issue or submit a PR, read that repository's `README.md`, `CONTRIBUTING.md`, and -`code-of-conduct.md`, if they exist. - -Most repositories use issue and PR templates. Have a look through some open -issues and PRs to get a feel for that team's processes. Make sure to fill out -the templates with as much detail as possible when you file issues or PRs. +If you see text you'd like to improve, use GitHub to search all repositories in the Kubernetes +organization. This can help you figure out where to submit your issue or PR. +Each repository has its own processes and procedures. Before you file an issue or submit a PR, +read that repository's `README.md`, `CONTRIBUTING.md`, and `code-of-conduct.md`, if they exist. +Most repositories use issue and PR templates. Have a look through some open issues and PRs to get +a feel for that team's processes. Make sure to fill out the templates with as much detail as +possible when you file issues or PRs. ## {{% heading "whatsnext" %}} - - Read [Reviewing](/docs/contribute/review/reviewing-prs) to learn more about the review process. - diff --git a/content/en/docs/contribute/new-content/overview.md b/content/en/docs/contribute/new-content/overview.md deleted file mode 100644 index 8b4da4970a..0000000000 --- a/content/en/docs/contribute/new-content/overview.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: Contributing new content overview -linktitle: Overview -content_type: concept -main_menu: true -weight: 5 ---- - -<!-- overview --> - -This section contains information you should know before contributing new content. - - - - -<!-- body --> - -## Contributing basics - -- Write Kubernetes documentation in Markdown and build the Kubernetes site using [Hugo](https://gohugo.io/). -- The source is in [GitHub](https://github.com/kubernetes/website). You can find Kubernetes documentation at `/content/en/docs/`. Some of the reference documentation is automatically generated from scripts in the `update-imported-docs/` directory. -- [Page content types](/docs/contribute/style/page-content-types/) describe the presentation of documentation content in Hugo. -- In addition to the standard Hugo shortcodes, we use a number of - [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our documentation to control the presentation of content. -- Documentation source is available in multiple languages in `/content/`. Each - language has its own folder with a two-letter code determined by the - [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php). For - example, English documentation source is stored in `/content/en/docs/`. -- For more information about contributing to documentation in multiple languages or starting a new translation, see [localization](/docs/contribute/localization). - -## Before you begin {#before-you-begin} - -### Sign the CNCF CLA {#sign-the-cla} - -All Kubernetes contributors **must** read the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md) and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md). - -Pull requests from contributors who haven't signed the CLA fail the automated tests. The name and email you provide must match those found in your `git config`, and your git name and email must match those used for the CNCF CLA. - -### Choose which Git branch to use - -When opening a pull request, you need to know in advance which branch to base your work on. - -Scenario | Branch -:---------|:------------ -Existing or new English language content for the current release | `main` -Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-<version>`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch. -Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information. - - -If you're still not sure which branch to choose, ask in `#sig-docs` on Slack. - -{{< note >}} -If you already submitted your pull request and you know that the base branch -was wrong, you (and only you, the submitter) can change it. -{{< /note >}} - -### Languages per PR - -Limit pull requests to one language per PR. If you need to make an identical change to the same code sample in multiple languages, open a separate PR for each language. - -## Tools for contributors - -The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools) directory in the `kubernetes/website` repository contains tools to help your contribution journey go more smoothly. - - diff --git a/content/en/docs/contribute/participate/_index.md b/content/en/docs/contribute/participate/_index.md index ff4714f803..b76423eded 100644 --- a/content/en/docs/contribute/participate/_index.md +++ b/content/en/docs/contribute/participate/_index.md @@ -115,6 +115,6 @@ SIG Docs approvers. Here's how it works. For more information about contributing to the Kubernetes documentation, see: -- [Contributing new content](/docs/contribute/new-content/overview/) +- [Contributing new content](/docs/contribute/new-content/) - [Reviewing content](/docs/contribute/review/reviewing-prs) - [Documentation style guide](/docs/contribute/style/) diff --git a/content/en/docs/contribute/participate/pr-wranglers.md b/content/en/docs/contribute/participate/pr-wranglers.md index 727a8a81d5..ee553c64d6 100644 --- a/content/en/docs/contribute/participate/pr-wranglers.md +++ b/content/en/docs/contribute/participate/pr-wranglers.md @@ -26,9 +26,16 @@ Each day in a week-long shift as PR Wrangler: - If you need to verify content, comment on the PR and request more details. - Assign relevant `sig/` label(s). - If needed, assign reviewers from the `reviewers:` block in the file's front matter. + - You can also tag a [SIG](https://github.com/kubernetes/community/blob/master/sig-list.md) for a review by commenting `@kubernetes/<sig>-pr-reviews` on the PR. - Use the `/approve` comment to approve a PR for merging. Merge the PR when ready. - PRs should have a `/lgtm` comment from another member before merging. - - Consider accepting technically accurate content that doesn't meet the [style guidelines](/docs/contribute/style/style-guide/). Open a new issue with the label `good first issue` to address style concerns. + - Consider accepting technically accurate content that doesn't meet the + [style guidelines](/docs/contribute/style/style-guide/). As you approve the change, + open a new issue to address the style concern. You can usually write these style fix + issues as [good first issues](https://kubernetes.dev/docs/guide/help-wanted/#good-first-issue). + - Using style fixups as good first issues is a good way to ensure a supply of easier tasks + to help onboard new contributors. + ### Helpful GitHub queries for wranglers @@ -77,6 +84,20 @@ To close a pull request, leave a `/close` comment on the PR. {{< note >}} -The [`fejta-bot`](https://github.com/fejta-bot) bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them. PR wranglers should close issues after 14-30 days of inactivity. +The [`k8s-triage-robot`](https://github.com/k8s-triage-robot) bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them. PR wranglers should close issues after 14-30 days of inactivity. {{< /note >}} + +## PR Wrangler shadow program + +In late 2021, SIG Docs introduced the PR Wrangler Shadow Program. The program was introduced to help new contributors understand the PR wrangling process. + +### Become a shadow + +- If you are interested in shadowing as a PR wrangler, please visit the [PR Wranglers Wiki page](https://github.com/kubernetes/website/wiki/PR-Wranglers) to see the PR wrangling schedule for this year and sign up. + +- Kubernetes org members can edit the [PR Wranglers Wiki page](https://github.com/kubernetes/website/wiki/PR-Wranglers) and sign up to shadow an existing PR Wrangler for a week. + +- Others can reach out on the [#sig-docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) for requesting to shadow an assigned PR Wrangler for a specific week. Feel free to reach out to Brad Topol (`@bradtopol`) or one of the [SIG Docs co-chairs/leads](https://github.com/kubernetes/community/tree/master/sig-docs#leadership). + +- Once you've signed up to shadow a PR Wrangler, introduce yourself to the PR Wrangler on the [Kubernetes Slack](https://slack.k8s.io). diff --git a/content/en/docs/contribute/participate/roles-and-responsibilities.md b/content/en/docs/contribute/participate/roles-and-responsibilities.md index e59ffe4fe3..10d6072ce3 100644 --- a/content/en/docs/contribute/participate/roles-and-responsibilities.md +++ b/content/en/docs/contribute/participate/roles-and-responsibilities.md @@ -32,7 +32,7 @@ Anyone can: - Suggest improvements on [Slack](https://slack.k8s.io/) or the [SIG docs mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs). -After [signing the CLA](/docs/contribute/new-content/overview/#sign-the-cla), anyone can also: +After [signing the CLA](https://github.com/kubernetes/community/blob/master/CLA.md), anyone can also: - Open a pull request to improve existing content, add new content, or write a blog post or case study - Create diagrams, graphics assets, and embeddable screencasts and videos @@ -147,7 +147,7 @@ separately for reviewer status in SIG Docs. To apply: 1. Open a pull request that adds your GitHub user name to a section of the - [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS) file + [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS_ALIASES) file in the `kubernetes/website` repository. {{< note >}} @@ -219,7 +219,7 @@ separately for approver status in SIG Docs. To apply: 1. Open a pull request adding yourself to a section of the - [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS) + [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS_ALIASES) file in the `kubernetes/website` repository. {{< note >}} diff --git a/content/en/docs/contribute/review/for-approvers.md b/content/en/docs/contribute/review/for-approvers.md index 5c781ec53f..5d0606d60d 100644 --- a/content/en/docs/contribute/review/for-approvers.md +++ b/content/en/docs/contribute/review/for-approvers.md @@ -79,7 +79,7 @@ The most common prow commands reviewers and approvers use are: {{< table caption="Prow commands for reviewing" >}} Prow Command | Role Restrictions | Description :------------|:------------------|:----------- -`/lgtm` | Anyone, but triggers automation if a Reviewer or Approver uses it | Signals that you've finished reviewing a PR and are satisfied with the changes. +`/lgtm` | Organization members | Signals that you've finished reviewing a PR and are satisfied with the changes. `/approve` | Approvers | Approves a PR for merging. `/assign` | Reviewers or Approvers | Assigns a person to review or approve a PR `/close` | Reviewers or Approvers | Closes an issue or PR. @@ -121,7 +121,7 @@ finds issues that might need triage. `priority/important-longterm` | Do this within 6 months. `priority/backlog` | Deferrable indefinitely. Do when resources are available. `priority/awaiting-more-evidence` | Placeholder for a potentially good issue so it doesn't get lost. - `help` or `good first issue` | Suitable for someone with very little Kubernetes or SIG Docs experience. See [Help Wanted and Good First Issue Labels](https://github.com/kubernetes/community/blob/master/contributors/guide/help-wanted.md) for more information. + `help` or `good first issue` | Suitable for someone with very little Kubernetes or SIG Docs experience. See [Help Wanted and Good First Issue Labels](https://kubernetes.dev/docs/guide/help-wanted/) for more information. {{< /table >}} @@ -141,7 +141,7 @@ To add a label, leave a comment in one of the following formats: To remove a label, leave a comment in one of the following formats: - `/remove-<label-to-remove>` (for example, `/remove-help`) -- `/remove-<label-category> <label-to-remove>` (for example, `/remove-triage needs-information`)` +- `/remove-<label-category> <label-to-remove>` (for example, `/remove-triage needs-information`) In both cases, the label must already exist. If you try to add a label that does not exist, the command is silently ignored. @@ -181,7 +181,7 @@ If the dead link issue is in the API or `kubectl` documentation, assign them `/p ### Blog issues -We expect [Kubernetes Blog](https://kubernetes.io/blog/) entries to become +We expect [Kubernetes Blog](/blog/) entries to become outdated over time. Therefore, we only maintain blog entries less than a year old. If an issue is related to a blog entry that is more than one year old, close the issue without fixing. diff --git a/content/en/docs/contribute/review/reviewing-prs.md b/content/en/docs/contribute/review/reviewing-prs.md index ff6ef9d709..41aaecea17 100644 --- a/content/en/docs/contribute/review/reviewing-prs.md +++ b/content/en/docs/contribute/review/reviewing-prs.md @@ -7,11 +7,11 @@ weight: 10 <!-- overview --> -Anyone can review a documentation pull request. Visit the [pull requests](https://github.com/kubernetes/website/pulls) section in the Kubernetes website repository to see open pull requests. +Anyone can review a documentation pull request. Visit the [pull requests](https://github.com/kubernetes/website/pulls) +section in the Kubernetes website repository to see open pull requests. -Reviewing documentation pull requests is a -great way to introduce yourself to the Kubernetes community. -It helps you learn the code base and build trust with other contributors. +Reviewing documentation pull requests is a great way to introduce yourself to the Kubernetes +community. It helps you learn the code base and build trust with other contributors. Before reviewing, it's a good idea to: @@ -27,7 +27,8 @@ Before reviewing, it's a good idea to: Before you start a review: -- Read the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and ensure that you abide by it at all times. +- Read the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md) + and ensure that you abide by it at all times. - Be polite, considerate, and helpful. - Comment on positive aspects of PRs as well as changes. - Be empathetic and mindful of how your review may be received. @@ -36,33 +37,85 @@ Before you start a review: ## Review process -In general, review pull requests for content and style in English. +In general, review pull requests for content and style in English. Figure 1 outlines the steps for +the review process. The details for each step follow. -1. Go to - [https://github.com/kubernetes/website/pulls](https://github.com/kubernetes/website/pulls). - You see a list of every open pull request against the Kubernetes website and - docs. +<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure --> +<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it --> -2. Filter the open PRs using one or all of the following labels: - - `cncf-cla: yes` (Recommended): PRs submitted by contributors who have not signed the CLA cannot be merged. See [Sign the CLA](/docs/contribute/new-content/overview/#sign-the-cla) for more information. - - `language/en` (Recommended): Filters for english language PRs only. - - `size/<size>`: filters for PRs of a certain size. If you're new, start with smaller PRs. +{{< mermaid >}} +flowchart LR + subgraph fourth[Start review] + direction TB + S[ ] -.- + M[add comments] --> N[review changes] + N --> O[new contributors should<br>choose Comment] + end + subgraph third[Select PR] + direction TB + T[ ] -.- + J[read description<br>and comments]--> K[preview changes in<br>Netlify preview build] + end + + A[Review open PR list]--> B[Filter open PRs<br>by label] + B --> third --> fourth + - Additionally, ensure the PR isn't marked as a work in progress. PRs using the `work in progress` label are not ready for review yet. +classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; +classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class A,B,J,K,M,N,O grey +class S,T spacewhite +class third,fourth white +{{</ mermaid >}} -3. Once you've selected a PR to review, understand the change by: - - Reading the PR description to understand the changes made, and read any linked issues - - Reading any comments by other reviewers - - Clicking the **Files changed** tab to see the files and lines changed - - Previewing the changes in the Netlify preview build by scrolling to the PR's build check section at the bottom of the **Conversation** tab and clicking the **deploy/netlify** line's **Details** link. +Figure 1. Review process steps. -4. Go to the **Files changed** tab to start your review. - 1. Click on the `+` symbol beside the line you want to comment on. - 2. Fill in any comments you have about the line and click either **Add single comment** (if you have only one comment to make) or **Start a review** (if you have multiple comments to make). - 3. When finished, click **Review changes** at the top of the page. Here, you can add - add a summary of your review (and leave some positive comments for the contributor!), - approve the PR, comment or request changes as needed. New contributors should always - choose **Comment**. + +1. Go to [https://github.com/kubernetes/website/pulls](https://github.com/kubernetes/website/pulls). + You see a list of every open pull request against the Kubernetes website and docs. + +2. Filter the open PRs using one or all of the following labels: + + - `cncf-cla: yes` (Recommended): PRs submitted by contributors who have not signed the CLA + cannot be merged. See [Sign the CLA](/docs/contribute/new-content/#sign-the-cla) + for more information. + - `language/en` (Recommended): Filters for english language PRs only. + - `size/<size>`: filters for PRs of a certain size. If you're new, start with smaller PRs. + + Additionally, ensure the PR isn't marked as a work in progress. PRs using the `work in + progress` label are not ready for review yet. + +3. Once you've selected a PR to review, understand the change by: + + - Reading the PR description to understand the changes made, and read any linked issues + - Reading any comments by other reviewers + - Clicking the **Files changed** tab to see the files and lines changed + - Previewing the changes in the Netlify preview build by scrolling to the PR's build check + section at the bottom of the **Conversation** tab. + Here's a screenshot (this shows GitHub's desktop site; if you're reviewing + on a tablet or smartphone device, the GitHub web UI is slightly different): + {{< figure src="/images/docs/github_netlify_deploy_preview.png" alt="GitHub pull request details including link to Netlify preview" >}} + To open the preview, click on the **Details** link of the **deploy/netlify** line in the + list of checks. + +4. Go to the **Files changed** tab to start your review. + + 1. Click on the `+` symbol beside the line you want to comment on. + 1. Fill in any comments you have about the line and click either **Add single comment** + (if you have only one comment to make) or **Start a review** (if you have multiple comments to make). + 1. When finished, click **Review changes** at the top of the page. Here, you can add + a summary of your review (and leave some positive comments for the contributor!). + Please always use the "Comment" + + - Avoid clicking the "Request changes" button when finishing your review. + If you want to block a PR from being merged before some further changes are made, + you can leave a "/hold" comment. + Mention why you are setting a hold, and optionally specify the conditions under + which the hold can be removed by you or other reviewers. + + - Avoid clicking the "Approve" button when finishing your review. + Leaving a "/approve" comment is recommended most of the time. ## Reviewing checklist @@ -84,14 +137,22 @@ When reviewing, use the following as a starting point. ### Website -- Did this PR change or remove a page title, slug/alias or anchor link? If so, are there broken links as a result of this PR? Is there another option, like changing the page title without changing the slug? +- Did this PR change or remove a page title, slug/alias or anchor link? If so, are there broken + links as a result of this PR? Is there another option, like changing the page title without + changing the slug? + - Does the PR introduce a new page? If so: - - Is the page using the right [page content type](/docs/contribute/style/page-content-types/) and associated Hugo shortcodes? + + - Is the page using the right [page content type](/docs/contribute/style/page-content-types/) + and associated Hugo shortcodes? - Does the page appear correctly in the section's side navigation (or at all)? - Should the page appear on the [Docs Home](/docs/home/) listing? -- Do the changes show up in the Netlify preview? Be particularly vigilant about lists, code blocks, tables, notes and images. + +- Do the changes show up in the Netlify preview? Be particularly vigilant about lists, code + blocks, tables, notes and images. ### Other -For small issues with a PR, like typos or whitespace, prefix your comments with `nit:`. This lets the author know the issue is non-critical. +For small issues with a PR, like typos or whitespace, prefix your comments with `nit:`. +This lets the author know the issue is non-critical. diff --git a/content/en/docs/contribute/style/diagram-guide.md b/content/en/docs/contribute/style/diagram-guide.md new file mode 100644 index 0000000000..ac3a4fd529 --- /dev/null +++ b/content/en/docs/contribute/style/diagram-guide.md @@ -0,0 +1,779 @@ +--- +title: Diagram Guide +linktitle: Diagram guide +content_type: concept +weight: 15 +--- + +<!--Overview--> + +This guide shows you how to create, edit and share diagrams using the Mermaid +JavaScript library. Mermaid.js allows you to generate diagrams using a simple +markdown-like syntax inside Markdown files. You can also use Mermaid to +generate `.svg` or `.png` image files that you can add to your documentation. + +The target audience for this guide is anybody wishing to learn about Mermaid +and/or how to create and add diagrams to Kubernetes documentation. + +Figure 1 outlines the topics covered in this section. + +{{< mermaid >}} +flowchart LR +subgraph m[Mermaid.js] +direction TB +S[ ]-.- +C[build<br>diagrams<br>with markdown] --> +D[on-line<br>live editor] +end +A[Why are diagrams<br>useful?] --> m +m --> N[3 x methods<br>for creating<br>diagrams] +N --> T[Examples] +T --> X[Styling<br>and<br>captions] +X --> V[Tips] + + + + classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; + classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 + class A,C,D,N,X,m,T,V box + class S spacewhite + +%% you can hyperlink Mermaid diagram nodes to a URL using click statements + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click N "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click T "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click X "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank +click V "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank + +{{< /mermaid >}} + +Figure 1. Topics covered in this section. + +All you need to begin working with Mermaid is the following: + +* Basic understanding of markdown. +* Using the Mermaid live editor. +* Using [Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/). +* Using the [Hugo {{</* figure */>}} shortcode](https://gohugo.io/content-management/shortcodes/#figure). +* Performing [Hugo local previews](/docs/contribute/new-content/open-a-pr/#preview-locally). +* Familiar with the [Contributing new content](/docs/contribute/new-content/) process. + +{{< note >}} +You can click on each diagram in this section to view the code and rendered +diagram in the Mermaid live editor. +{{< /note >}} + +<!--body--> + +## Why you should use diagrams in documentation + +Diagrams improve documentation clarity and comprehension. There are advantages for both the user and the contributor. + +The user benefits include: + +* __Friendly landing spot__. A detailed text-only greeting page could + intimidate users, in particular, first-time Kubernetes users. +* __Faster grasp of concepts__. A diagram can help users understand the key + points of a complex topic. Your diagram can serve as a visual learning guide + to dive into the topic details. +* __Better retention__. For some, it is easier to recall pictures rather than text. + +The contributor benefits include: + +* __Assist in developing the structure and content__ of your contribution. For + example, you can start with a simple diagram covering the high-level points + and then dive into details. +* __Expand and grow the user community__. Easily consumed documentation + augmented with diagrams attracts new users who might previously have been + reluctant to engage due to perceived complexities. + +You should consider your target audience. In addition to experienced K8s +users, you will have many who are new to Kubernetes. Even a simple diagram can +assist new users in absorbing Kubernetes concepts. They become emboldened and +more confident to further explore Kubernetes and the documentation. + +## Mermaid + +[Mermaid](https://mermaid-js.github.io/mermaid/#/) is an open source +JavaScript library that allows you to create, edit and easily share diagrams +using a simple, markdown-like syntax configured inline in Markdown files. + +The following lists features of Mermaid: + +* Simple code syntax. +* Includes a web-based tool allowing you to code and preview your diagrams. +* Supports multiple formats including flowchart, state and sequence. +* Easy collaboration with colleagues by sharing a per-diagram URL. +* Broad selection of shapes, lines, themes and styling. + +The following lists advantages of using Mermaid: + +* No need for separate, non-Mermaid diagram tools. +* Adheres to existing PR workflow. You can think of Mermaid code as just + Markdown text included in your PR. +* Simple tool builds simple diagrams. You don't want to get bogged down + (re)crafting an overly complex and detailed picture. Keep it simple! + +Mermaid provides a simple, open and transparent method for the SIG communities +to add, edit and collaborate on diagrams for new or existing documentation. + +{{< note >}} +You can still use Mermaid to create/edit diagrams even if it's not supported +in your environment. This method is called __Mermaid+SVG__ and is explained +below. +{{< /note >}} + +### Live editor + +The [Mermaid live editor](https://mermaid-js.github.io/mermaid-live-editor) is +a web-based tool that enables you to create, edit and review diagrams. + +The following lists live editor functions: + +* Displays Mermaid code and rendered diagram. +* Generates a URL for each saved diagram. The URL is displayed in the URL + field of your browser. You can share the URL with colleagues who can access + and modify the diagram. +* Option to download `.svg` or `.png` files. + +{{< note >}} +The live editor is the easiest and fastest way to create and edit Mermaid diagrams. +{{< /note >}} + +## Methods for creating diagrams + +Figure 2 outlines the three methods to generate and add diagrams. + +{{< mermaid >}} +graph TB +A[Contributor] +B[Inline<br><br>Mermaid code<br>added to .md file] +C[Mermaid+SVG<br><br>Add mermaid-generated<br>svg file to .md file] +D[External tool<br><br>Add external-tool-<br>generated svg file<br>to .md file] + + A --> B + A --> C + A --> D + + classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; + class A,B,C,D box + +%% you can hyperlink Mermaid diagram nodes to a URL using click statements + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +{{< /mermaid >}} + +Figure 2. Methods to create diagrams. + + +### Inline + +Figure 3 outlines the steps to follow for adding a diagram using the Inline +method. + +{{< mermaid >}} +graph LR +A[1. Use live editor<br> to create/edit<br>diagram] --> +B[2. Store diagram<br>URL somewhere] --> +C[3. Copy Mermaid code<br>to page markdown file] --> +D[4. Add caption] + + + classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; + class A,B,C,D box + +%% you can hyperlink Mermaid diagram nodes to a URL using click statements + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + +click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank + + + +{{< /mermaid >}} + +Figure 3. Inline Method steps. + + +The following lists the steps you should follow for adding a diagram using the Inline method: + +1. Create your diagram using the live editor. +2. Store the diagram URL somewhere for later access. +3. Copy the mermaid code to the location in your `.md` file where you want the diagram to appear. +4. Add a caption below the diagram using Markdown text. + +A Hugo build runs the Mermaid code and turns it into a diagram. + +{{< note >}} +You may find keeping track of diagram URLs is cumbersome. If so, make a note +in the `.md` file that the Mermaid code is self-documenting. Contributors can +copy the Mermaid code to and from the live editor for diagram edits. +{{< /note >}} + +Here is a sample code snippet contained in an `.md` file: + +``` +--- +title: My PR +--- +Figure 17 shows a simple A to B process. +some markdown text +... +{{</* mermaid */>}} + graph TB + A --> B +{{</* /mermaid */>}} + +Figure 17. A to B +more text +``` +{{< note >}} +You must include the Hugo Mermaid shortcode +tags at the start and end of the Mermaid code block. You should add a diagram +caption below the diagram. +{{< /note >}} + +For more details on diagram captions, see [How to use captions](#how-to-use-captions). + +The following lists advantages of the Inline method: + +* Live editor tool. +* Easy to copy Mermaid code to and from the live editor and your `.md` file. +* No need for separate `.svg` image file handling. +* Content text, diagram code and diagram caption contained in the same `.md` file. + +You should use the [local](/docs/contribute/new-content/open-a-pr/#preview-locally) +and Netlify previews to verify the diagram is properly rendered. + +{{< caution >}} +The Mermaid live editor feature set may not support the K8s/website Mermaid feature set. +You might see a syntax error or a blank screen after the Hugo build. +If that is the case, consider using the Mermaid+SVG method. +{{< /caution >}} + +### Mermaid+SVG + +Figure 4 outlines the steps to follow for adding a diagram using the Mermaid+SVG method. + +{{< mermaid >}} +flowchart LR +A[1. Use live editor<br> to create/edit<br>diagram] +B[2. Store diagram<br>URL somewhere] +C[3. Generate .svg file<br>and download to<br>images/ folder] +subgraph w[ ] +direction TB +D[4. Use figure shortcode<br>to reference .svg<br>file in page<br>.md file] --> +E[5. Add caption] +end +A --> B +B --> C +C --> w + + classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; + class A,B,C,D,E,w box + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click E "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + + + +{{< /mermaid >}} + +Figure 4. Mermaid+SVG method steps. + +The following lists the steps you should follow for adding a diagram using the Mermaid+SVG method: + +1. Create your diagram using the live editor. +2. Store the diagram URL somewhere for later access. +3. Generate an `.svg` image file for the diagram and download it to the appropriate `images/` folder. +4. Use the `{{</* figure */>}}` shortcode to reference the diagram in the `.md` file. +5. Add a caption using the `{{</* figure */>}}` shortcode's `caption` parameter. + +For example, use the live editor to create a diagram called `boxnet`. +Store the diagram URL somewhere for later access. Generate and download a +`boxnet.svg` file to the appropriate `../images/` folder. + +Use the `{{</* figure */>}}` shortcode in your PR's `.md` file to reference +the `.svg` image file and add a caption. + +```json +{{</* figure src="/static/images/boxnet.svg" alt="Boxnet figure" class="diagram-large" caption="Figure 14. Boxnet caption" */>}} +``` + +For more details on diagram captions, see [How to use captions](#how-to-use-captions). + +{{< note >}} +The `{{</* figure */>}}` shortcode is the preferred method for adding `.svg` image files +to your documentation. You can also use the standard markdown image syntax like so: +`![my boxnet diagram](static/images/boxnet.svg)`. +And you will need to add a caption below the diagram. +{{< /note >}} + +You should add the live editor URL as a comment block in the `.svg` image file using a text editor. +For example, you would include the following at the beginning of the `.svg` image file: + +``` +<!-- To view or edit the mermaid code, use the following URL: --> +<!-- https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb ... <remainder of the URL> --> +``` + +The following lists advantages of the Mermaid+SVG method: + +* Live editor tool. +* Live editor tool supports the most current Mermaid feature set. +* Employ existing K8s/website methods for handling `.svg` image files. +* Environment doesn't require Mermaid support. + +Be sure to check that your diagram renders properly using the +[local](/docs/contribute/new-content/open-a-pr/#preview-locally) +and Netlify previews. + +### External tool + +Figure 5 outlines the steps to follow for adding a diagram using the External Tool method. + +First, use your external tool to create the diagram and save it as an `.svg` +or `.png` image file. After that, use the same steps as the __Mermaid+SVG__ +method for adding `.svg` image files. + +{{< mermaid >}} +flowchart LR +A[1. Use external<br>tool to create/edit<br>diagram] +B[2. If possible, save<br>diagram coordinates<br>for contributor<br>access] +C[3. Generate .svg <br>or.png file<br>and download to<br>appropriate<br>images/ folder] +subgraph w[ ] +direction TB +D[4. Use figure shortcode<br>to reference svg or<br>png file in<br>page .md file] --> +E[5. Add caption] +end +A --> B +B --> C +C --> w +classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; +class A,B,C,D,E,w box + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" + +click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" + +click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" + +click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" + +click E "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" + +{{< /mermaid >}} + +Figure 5. External Tool method steps + + +The following lists the steps you should follow for adding a diagram using the External Tool method: + +1. Use your external tool to create a diagram. +2. Save the diagram coordinates for contributor access. For example, your tool + may offer a link to the diagram image, or you could place the source code + file, such as an `.xml` file, in a public repository for later contributor access. +3. Generate and save the diagram as an `.svg` or `.png` image file. + Download this file to the appropriate `../images/` folder. +4. Use the `{{</* figure */>}}` shortcode to reference the diagram in the `.md` file. +5. Add a caption using the `{{</* figure */>}}` shortcode's `caption` parameter. + +Here is the `{{</* figure */>}}` shortcode for the `images/apple.svg` diagram: +```text +{{</* figure src="/static/images/apple.svg" alt="red-apple-figure" class="diagram-large" caption="Figure 9. A Big Red Apple" */>}} +``` + +If your external drawing tool permits: + +* You can incorporate multiple `.svg` or `.png` logos, icons and images into your diagram. + However, make sure you observe copyright and follow the Kubernetes documentation + [guidelines](/docs/contribute/style/content-guide/) on the use of third party content. +* You should save the diagram source coordinates for later contributor access. + For example, your tool may offer a link to the diagram image, or you could + place the source code file, such as an `.xml` file, somewhere for contributor access. + +For more information on K8s and CNCF logos and images, check out +[CNCF Artwork](https://github.com/cncf/artwork). + +The following lists advantages of the External Tool method: + +* Contributor familiarity with external tool. +* Diagrams require more detail than what Mermaid can offer. + +Don't forget to check that your diagram renders correctly using the +[local](/docs/contribute/new-content/open-a-pr/#preview-locally) and Netlify previews. + +## Examples + +This section shows several examples of Mermaid diagrams. + +{{< note >}} +The code block examples omit the Hugo Mermaid +shortcode tags. This allows you to copy the code block into the live editor +to experiment on your own. +Note that the live editor doesn't recognize Hugo shortcodes. +{{< /note >}} + +### Example 1 - Pod topology spread constraints + +Figure 6 shows the diagram appearing in the +[Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/#node-labels) +page. + +{{< mermaid >}} + graph TB + subgraph "zoneB" + n3(Node3) + n4(Node4) + end + subgraph "zoneA" + n1(Node1) + n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4 k8s; + class zoneA,zoneB cluster; + +click n3 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click n4 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click n1 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +click n2 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank + +{{< /mermaid >}} + +Figure 6. Pod Topology Spread Constraints. + +Code block: + +``` +graph TB + subgraph "zoneB" + n3(Node3) + n4(Node4) + end + subgraph "zoneA" + n1(Node1) + n2(Node2) + end + + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class n1,n2,n3,n4 k8s; + class zoneA,zoneB cluster; +``` + +### Example 2 - Ingress + +Figure 7 shows the diagram appearing in the [What is Ingress](/docs/concepts/services-networking/ingress/#what-is-ingress) page. + +{{< mermaid >}} +graph LR; +client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress]; +ingress-->|routing rule|service[Service]; +subgraph cluster +ingress; +service-->pod1[Pod]; +service-->pod2[Pod]; +end +classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; +classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; +class ingress,service,pod1,pod2 k8s; +class client plain; +class cluster cluster; + +click client "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank + +click ingress "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank + +click service "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank + +click pod1 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank + +click pod2 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank + + + +{{< /mermaid >}} +Figure 7. Ingress + +Code block: + +```mermaid +graph LR; + client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress]; + ingress-->|routing rule|service[Service]; + subgraph cluster + ingress; + service-->pod1[Pod]; + service-->pod2[Pod]; + end + classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; + classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; + classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; + class ingress,service,pod1,pod2 k8s; + class client plain; + class cluster cluster; +``` + +### Example 3 - K8s system flow + +Figure 8 depicts a Mermaid sequence diagram showing the system flow between +K8s components to start a container. + +{{< figure src="/docs/images/diagram-guide-example-3.svg" alt="K8s system flow diagram" class="diagram-large" caption="Figure 8. K8s system flow diagram" link="https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiJSV7aW5pdDp7XCJ0aGVtZVwiOlwibmV1dHJhbFwifX0lJVxuc2VxdWVuY2VEaWFncmFtXG4gICAgYWN0b3IgbWVcbiAgICBwYXJ0aWNpcGFudCBhcGlTcnYgYXMgY29udHJvbCBwbGFuZTxicj48YnI-YXBpLXNlcnZlclxuICAgIHBhcnRpY2lwYW50IGV0Y2QgYXMgY29udHJvbCBwbGFuZTxicj48YnI-ZXRjZCBkYXRhc3RvcmVcbiAgICBwYXJ0aWNpcGFudCBjbnRybE1nciBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5jb250cm9sbGVyPGJyPm1hbmFnZXJcbiAgICBwYXJ0aWNpcGFudCBzY2hlZCBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5zY2hlZHVsZXJcbiAgICBwYXJ0aWNpcGFudCBrdWJlbGV0IGFzIG5vZGU8YnI-PGJyPmt1YmVsZXRcbiAgICBwYXJ0aWNpcGFudCBjb250YWluZXIgYXMgbm9kZTxicj48YnI-Y29udGFpbmVyPGJyPnJ1bnRpbWVcbiAgICBtZS0-PmFwaVNydjogMS4ga3ViZWN0bCBjcmVhdGUgLWYgcG9kLnlhbWxcbiAgICBhcGlTcnYtLT4-ZXRjZDogMi4gc2F2ZSBuZXcgc3RhdGVcbiAgICBjbnRybE1nci0-PmFwaVNydjogMy4gY2hlY2sgZm9yIGNoYW5nZXNcbiAgICBzY2hlZC0-PmFwaVNydjogNC4gd2F0Y2ggZm9yIHVuYXNzaWduZWQgcG9kcyhzKVxuICAgIGFwaVNydi0-PnNjaGVkOiA1LiBub3RpZnkgYWJvdXQgcG9kIHcgbm9kZW5hbWU9XCIgXCJcbiAgICBzY2hlZC0-PmFwaVNydjogNi4gYXNzaWduIHBvZCB0byBub2RlXG4gICAgYXBpU3J2LS0-PmV0Y2Q6IDcuIHNhdmUgbmV3IHN0YXRlXG4gICAga3ViZWxldC0-PmFwaVNydjogOC4gbG9vayBmb3IgbmV3bHkgYXNzaWduZWQgcG9kKHMpXG4gICAgYXBpU3J2LT4-a3ViZWxldDogOS4gYmluZCBwb2QgdG8gbm9kZVxuICAgIGt1YmVsZXQtPj5jb250YWluZXI6IDEwLiBzdGFydCBjb250YWluZXJcbiAgICBrdWJlbGV0LT4-YXBpU3J2OiAxMS4gdXBkYXRlIHBvZCBzdGF0dXNcbiAgICBhcGlTcnYtLT4-ZXRjZDogMTIuIHNhdmUgbmV3IHN0YXRlIiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjp0cnVlfQ" >}} + + + +Code block: + +``` +%%{init:{"theme":"neutral"}}%% +sequenceDiagram + actor me + participant apiSrv as control plane<br><br>api-server + participant etcd as control plane<br><br>etcd datastore + participant cntrlMgr as control plane<br><br>controller<br>manager + participant sched as control plane<br><br>scheduler + participant kubelet as node<br><br>kubelet + participant container as node<br><br>container<br>runtime + me->>apiSrv: 1. kubectl create -f pod.yaml + apiSrv-->>etcd: 2. save new state + cntrlMgr->>apiSrv: 3. check for changes + sched->>apiSrv: 4. watch for unassigned pods(s) + apiSrv->>sched: 5. notify about pod w nodename=" " + sched->>apiSrv: 6. assign pod to node + apiSrv-->>etcd: 7. save new state + kubelet->>apiSrv: 8. look for newly assigned pod(s) + apiSrv->>kubelet: 9. bind pod to node + kubelet->>container: 10. start container + kubelet->>apiSrv: 11. update pod status + apiSrv-->>etcd: 12. save new state +``` + +## How to style diagrams + +You can style one or more diagram elements using well-known CSS nomenclature. +You accomplish this using two types of statements in the Mermaid code. + +* `classDef` defines a class of style attributes. +* `class` defines one or more elements to apply the class to. + +In the code for +[figure 7](https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0), +you can see examples of both. + +``` +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; // defines style for the k8s class +class ingress,service,pod1,pod2 k8s; // k8s class is applied to elements ingress, service, pod1 and pod2. +``` + +You can include one or multiple `classDef` and `class` statements in your diagram. +You can also use the official K8s `#326ce5` hex color code for K8s components in your diagram. + +For more information on styling and classes, see +[Mermaid Styling and classes docs](https://mermaid-js.github.io/mermaid/#/flowchart?id=styling-and-classes). + +## How to use captions + +A caption is a brief description of a diagram. A title or a short description +of the diagram are examples of captions. Captions aren't meant to replace +explanatory text you have in your documentation. Rather, they serve as a +"context link" between that text and your diagram. + +The combination of some text and a diagram tied together with a caption help +provide a concise representation of the information you wish to convey to the +user. + +Without captions, you are asking the user to scan the text above or below the +diagram to figure out a meaning. This can be frustrating for the user. + +Figure 9 lays out the three components for proper captioning: diagram, diagram +caption and the diagram referral. + +{{< mermaid >}} +flowchart +A[Diagram<br><br>Inline Mermaid or<br>SVG image files] +B[Diagram Caption<br><br>Add Figure Number. and<br>Caption Text] +C[Diagram Referral<br><br>Referenence Figure Number<br>in text] + + classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; + class A,B,C box + +click A "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank + +click B "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank + +click C "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank + +{{< /mermaid >}} +Figure 9. Caption Components. + +{{< note >}} +You should always add a caption to each diagram in your documentation. +{{< /note >}} + +**Diagram** + +The `Mermaid+SVG` and `External Tool` methods generate `.svg` image files. + +Here is the `{{</* figure */>}}` shortcode for the diagram defined in an +`.svg` image file saved to `/images/docs/components-of-kubernetes.svg`: + +```none +{{</* figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 4. Kubernetes Architecture Components */>}} +``` + +You should pass the `src`, `alt`, `class` and `caption` values into the +`{{</* figure */>}}` shortcode. You can adjust the size of the diagram using +`diagram-large`, `diagram-medium` and `diagram-small` classes. + +{{< note >}} +Diagrams created using the `Inline` method don't use the `{{</* figure */>}}` +shortcode. The Mermaid code defines how the diagram will render on your page. +{{< /note >}} + +See [Methods for creating diagrams](#methods-for-creating-diagrams) +for more information on the different methods for creating diagrams. + +**Diagram Caption** + +Next, add a diagram caption. + +If you define your diagram in an `.svg` image file, then you should use the +`{{</* figure */>}}` shortcode's `caption` parameter. + +```none +{{</* figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 4. Kubernetes Architecture Components" */>}} +``` + +If you define your diagram using inline Mermaid code, then you should use Markdown text. + +```none +Figure 4. Kubernetes Architecture Components +``` + +The following lists several items to consider when adding diagram captions: + +* Use the `{{</* figure */>}}` shortcode to add a diagram caption for `Mermaid+SVG` + and `External Tool` diagrams. +* Use simple Markdown text to add a diagram caption for the `Inline` method. +* Prepend your diagram caption with `Figure NUMBER.`. You must use `Figure` + and the number must be unique for each diagram in your documentation page. + Add a period after the number. +* Add your diagram caption text after the `Figure NUMBER.` on the same line. + You must puncuate the caption with a period. Keep the caption text short. +* Position your diagram caption __BELOW__ your diagram. + +**Diagram Referral** + +Finally, you can add a diagram referral. This is used inside your text and +should precede the diagram itself. It allows a user to connect your text with +the associated diagram. The `Figure NUMBER` in your referral and caption must +match. + +You should avoid using spatial references such as `..the image below..` or +`..the following figure ..` + +Here is an example of a diagram referral: + +```text +Figure 10 depicts the components of the Kubernetes architecture. +The control plane ... +``` +Diagram referrals are optional and there are cases where they might not be +suitable. If you are not sure, add a diagram referral to your text to see if +it looks and sounds okay. When in doubt, use a diagram referral. + +**Complete picture** + +Figure 10 shows the Kubernetes Architecture diagram that includes the diagram, +diagram caption and diagram referral. The `{{</* figure */>}}` shortcode +renders the diagram, adds the caption and includes the optional `link` +parameter so you can hyperlink the diagram. The diagram referral is contained +in this paragraph. + +Here is the `{{</* figure */>}}` shortcode for this diagram: + +``` +{{</* figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 10. Kubernetes Architecture." link="https://kubernetes.io/docs/concepts/overview/components/" */>}} +``` + +{{< figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 10. Kubernetes Architecture." link="https://kubernetes.io/docs/concepts/overview/components/" >}} + +## Tips + +* Always use the live editor to create/edit your diagram. + +* Always use Hugo local and Netlify previews to check out how the diagram + appears in the documentation. + +* Include diagram source pointers such as a URL, source code location, or + indicate the code is self-documenting. + +* Always use diagram captions. + +* Very helpful to include the diagram `.svg` or `.png` image and/or Mermaid + source code in issues and PRs. + +* With the `Mermaid+SVG` and `External Tool` methods, use `.svg` image files + because they stay sharp when you zoom in on the diagram. + +* Best practice for `.svg` files is to load it into an SVG editing tool and use the + "Convert text to paths" function. + This ensures that the diagram renders the same on all systems, regardless of font + availability and font rendering support. + +* No Mermaid support for additional icons or artwork. + +* Hugo Mermaid shortcodes don't work in the live editor. + +* Any time you modify a diagram in the live editor, you __must__ save it + to generate a new URL for the diagram. + +* Click on the diagrams in this section to view the code and diagram rendering + in the live editor. + +* Look over the source code of this page, `diagram-guide.md`, for more examples. + +* Check out the [Mermaid docs](https://mermaid-js.github.io/mermaid/#/) + for explanations and examples. + +Most important, __Keep Diagrams Simple__. +This will save time for you and fellow contributors, and allow for easier reading +by new and experienced users. + + + + + + + diff --git a/content/en/docs/contribute/style/hugo-shortcodes/index.md b/content/en/docs/contribute/style/hugo-shortcodes/index.md index 2a836ffb91..5463019ac0 100644 --- a/content/en/docs/contribute/style/hugo-shortcodes/index.md +++ b/content/en/docs/contribute/style/hugo-shortcodes/index.md @@ -12,11 +12,13 @@ Read more about shortcodes in the [Hugo documentation](https://gohugo.io/content ## Feature state -In a Markdown page (`.md` file) on this site, you can add a shortcode to display version and state of the documented feature. +In a Markdown page (`.md` file) on this site, you can add a shortcode to +display version and state of the documented feature. ### Feature state demo -Below is a demo of the feature state snippet, which displays the feature as stable in the latest Kubernetes version. +Below is a demo of the feature state snippet, which displays the feature as +stable in the latest Kubernetes version. ``` {{</* feature-state state="stable" */>}} @@ -50,16 +52,22 @@ Renders to: There are two glossary shortcodes: `glossary_tooltip` and `glossary_definition`. -You can reference glossary terms with an inclusion that automatically updates and replaces content with the relevant links from [our glossary](/docs/reference/glossary/). When the glossary term is moused-over, the glossary entry displays a tooltip. The glossary term also displays as a link. +You can reference glossary terms with an inclusion that automatically updates +and replaces content with the relevant links from [our glossary](/docs/reference/glossary/). +When the glossary term is moused-over, the glossary entry displays a tooltip. +The glossary term also displays as a link. As well as inclusions with tooltips, you can reuse the definitions from the glossary in page content. -The raw data for glossary terms is stored at [https://github.com/kubernetes/website/tree/main/content/en/docs/reference/glossary](https://github.com/kubernetes/website/tree/main/content/en/docs/reference/glossary), with a content file for each glossary term. +The raw data for glossary terms is stored at +[the glossary directory](https://github.com/kubernetes/website/tree/main/content/en/docs/reference/glossary), +with a content file for each glossary term. ### Glossary demo -For example, the following include within the Markdown renders to {{< glossary_tooltip text="cluster" term_id="cluster" >}} with a tooltip: +For example, the following include within the Markdown renders to +{{< glossary_tooltip text="cluster" term_id="cluster" >}} with a tooltip: ``` {{</* glossary_tooltip text="cluster" term_id="cluster" */>}} @@ -83,9 +91,44 @@ You can also include a full definition: which renders as: {{< glossary_definition term_id="cluster" length="all" >}} +## Links to API Reference + +You can link to a page of the Kubernetes API reference using the +`api-reference` shortcode, for example to the +{{< api-reference page="workload-resources/pod-v1" >}} reference: + +``` +{{</* api-reference page="workload-resources/pod-v1" */>}} +``` + +The content of the `page` parameter is the suffix of the URL of the API reference page. + + +You can link to a specific place into a page by specifying an `anchor` +parameter, for example to the {{< api-reference page="workload-resources/pod-v1" anchor="PodSpec" >}} +reference or the {{< api-reference page="workload-resources/pod-v1" anchor="environment-variables" >}} +section of the page: + +``` +{{</* api-reference page="workload-resources/pod-v1" anchor="PodSpec" */>}} +{{</* api-reference page="workload-resources/pod-v1" anchor="environment-variables" */>}} +``` + + +You can change the text of the link by specifying a `text` parameter, for +example by linking to the +{{< api-reference page="workload-resources/pod-v1" anchor="environment-variables" text="Environment Variables">}} +section of the page: + +``` +{{</* api-reference page="workload-resources/pod-v1" anchor="environment-variables" text="Environment Variable" */>}} +``` + ## Table captions -You can make tables more accessible to screen readers by adding a table caption. To add a [caption](https://www.w3schools.com/tags/tag_caption.asp) to a table, enclose the table with a `table` shortcode and specify the caption with the `caption` parameter. +You can make tables more accessible to screen readers by adding a table caption. To add a +[caption](https://www.w3schools.com/tags/tag_caption.asp) to a table, +enclose the table with a `table` shortcode and specify the caption with the `caption` parameter. {{< note >}} Table captions are visible to screen readers but invisible when viewed in standard HTML. @@ -111,7 +154,8 @@ Parameter | Description | Default `logLevel` | The log level for log output | `INFO` {{< /table >}} -If you inspect the HTML for the table, you should see this element immediately after the opening `<table>` element: +If you inspect the HTML for the table, you should see this element immediately +after the opening `<table>` element: ```html <caption style="display: none;">Configuration parameters</caption> @@ -119,14 +163,25 @@ If you inspect the HTML for the table, you should see this element immediately a ## Tabs -In a markdown page (`.md` file) on this site, you can add a tab set to display multiple flavors of a given solution. +In a markdown page (`.md` file) on this site, you can add a tab set to display +multiple flavors of a given solution. The `tabs` shortcode takes these parameters: * `name`: The name as shown on the tab. -* `codelang`: If you provide inner content to the `tab` shortcode, you can tell Hugo what code language to use for highlighting. -* `include`: The file to include in the tab. If the tab lives in a Hugo [leaf bundle](https://gohugo.io/content-management/page-bundles/#leaf-bundles), the file -- which can be any MIME type supported by Hugo -- is looked up in the bundle itself. If not, the content page that needs to be included is looked up relative to the current page. Note that with the `include`, you do not have any shortcode inner content and must use the self-closing syntax. For example, <code>{{</* tab name="Content File #1" include="example1" /*/>}}</code>. The language needs to be specified under `codelang` or the language is taken based on the file name. Non-content files are code-highlighted by default. -* If your inner content is markdown, you must use the `%`-delimiter to surround the tab. For example, `{{%/* tab name="Tab 1" %}}This is **markdown**{{% /tab */%}}` +* `codelang`: If you provide inner content to the `tab` shortcode, you can tell Hugo + what code language to use for highlighting. +* `include`: The file to include in the tab. If the tab lives in a Hugo + [leaf bundle](https://gohugo.io/content-management/page-bundles/#leaf-bundles), + the file -- which can be any MIME type supported by Hugo -- is looked up in the bundle itself. + If not, the content page that needs to be included is looked up relative to the current page. + Note that with the `include`, you do not have any shortcode inner content and must use the + self-closing syntax. For example, + `{{</* tab name="Content File #1" include="example1" /*/>}}`. The language needs to be specified + under `codelang` or the language is taken based on the file name. + Non-content files are code-highlighted by default. +* If your inner content is markdown, you must use the `%`-delimiter to surround the tab. + For example, `{{%/* tab name="Tab 1" %}}This is **markdown**{{% /tab */%}}` * You can combine the variations mentioned above inside a tab set. Below is a demo of the tabs shortcode. @@ -215,6 +270,43 @@ Renders to: {{< tab name="JSON File" include="podtemplate.json" />}} {{< /tabs >}} +## Third party content marker + +Running Kubernetes requires third-party software. For example: you +usually need to add a +[DNS server](/docs/tasks/administer-cluster/dns-custom-nameservers/#introduction) +to your cluster so that name resolution works. + +When we link to third-party software, or otherwise mention it, +we follow the [content guide](/docs/contribute/style/content-guide/) +and we also mark those third party items. + +Using these shortcodes adds a disclaimer to any documentation page +that uses them. + +### Lists {#third-party-content-list} + +For a list of several third-party items, add: +``` +{{%/* thirdparty-content */%}} +``` +just below the heading for the section that includes all items. + +### Items {#third-party-content-item} + +If you have a list where most of the items refer to in-project +software (for example: Kubernetes itself, and the separate +[Descheduler](https://github.com/kubernetes-sigs/descheduler) +component), then there is a different form to use. + +Add the shortcode: +``` +{{%/* thirdparty-content single="true" */%}} +``` + +before the item, or just below the heading for the specific item. + + ## Version strings To generate a version string for inclusion in the documentation, you can choose from @@ -224,13 +316,17 @@ The two most commonly used version parameters are `latest` and `version`. ### `{{</* param "version" */>}}` -The `{{</* param "version" */>}}` shortcode generates the value of the current version of -the Kubernetes documentation from the `version` site parameter. The `param` shortcode accepts the name of one site parameter, in this case: `version`. +The `{{</* param "version" */>}}` shortcode generates the value of the current +version of the Kubernetes documentation from the `version` site parameter. The +`param` shortcode accepts the name of one site parameter, in this case: +`version`. {{< note >}} -In previously released documentation, `latest` and `version` parameter values are not equivalent. -After a new version is released, `latest` is incremented and the value of `version` for the documentation set remains unchanged. For example, a previously released version of the documentation displays `version` as -`v1.19` and `latest` as `v1.20`. +In previously released documentation, `latest` and `version` parameter values +are not equivalent. After a new version is released, `latest` is incremented +and the value of `version` for the documentation set remains unchanged. For +example, a previously released version of the documentation displays `version` +as `v1.19` and `latest` as `v1.20`. {{< /note >}} Renders to: @@ -249,7 +345,8 @@ Renders to: ### `{{</* latest-semver */>}}` -The `{{</* latest-semver */>}}` shortcode generates the value of `latest` without the "v" prefix. +The `{{</* latest-semver */>}}` shortcode generates the value of `latest` +without the "v" prefix. Renders to: @@ -266,8 +363,9 @@ Renders to: ### `{{</* latest-release-notes */>}}` -The `{{</* latest-release-notes */>}}` shortcode generates a version string from `latest` and removes -the "v" prefix. The shortcode prints a new URL for the release note CHANGELOG page with the modified version string. +The `{{</* latest-release-notes */>}}` shortcode generates a version string +from `latest` and removes the "v" prefix. The shortcode prints a new URL for +the release note CHANGELOG page with the modified version string. Renders to: @@ -280,3 +378,4 @@ Renders to: * Learn about [page content types](/docs/contribute/style/page-content-types/). * Learn about [opening a pull request](/docs/contribute/new-content/open-a-pr/). * Learn about [advanced contributing](/docs/contribute/advanced/). + diff --git a/content/en/docs/contribute/style/style-guide.md b/content/en/docs/contribute/style/style-guide.md index 74379e4276..960efc01a7 100644 --- a/content/en/docs/contribute/style/style-guide.md +++ b/content/en/docs/contribute/style/style-guide.md @@ -14,7 +14,7 @@ For additional information on creating new content for the Kubernetes documentation, read the [Documentation Content Guide](/docs/contribute/style/content-guide/). Changes to the style guide are made by SIG Docs as a group. To propose a change -or addition, [add it to the agenda](https://docs.google.com/document/d/1ddHwLK3kUMX1wVFIwlksjTk0MsqitBnWPe1LRa1Rx5A/edit) for an upcoming SIG Docs meeting, and attend the meeting to participate in the +or addition, [add it to the agenda](https://bit.ly/sig-docs-agenda) for an upcoming SIG Docs meeting, and attend the meeting to participate in the discussion. <!-- body --> @@ -46,10 +46,6 @@ When you refer specifically to interacting with an API object, use [UpperCamelCa When you are generally discussing an API object, use [sentence-style capitalization](https://docs.microsoft.com/en-us/style-guide/text-formatting/using-type/use-sentence-style-capitalization). -You may use the word "resource", "API", or "object" to clarify a Kubernetes resource type in a sentence. - -Don't split an API object name into separate words. For example, use PodTemplateList, not Pod Template List. - The following examples focus on capitalization. For more information about formatting API object names, review the related guidance on [Code Style](#code-style-inline-code). {{< table caption = "Do and Don't - Use Pascal case for API objects" >}} @@ -187,6 +183,36 @@ Set the value of `image` to nginx:1.16. | Set the value of `image` to `nginx:1.1 Set the value of the `replicas` field to 2. | Set the value of the `replicas` field to `2`. {{< /table >}} +## Referring to Kubernetes API resources + +This section talks about how we reference API resources in the documentation. + +### Clarification about "resource" + +Kubernetes uses the word "resource" to refer to API resources, such as `pod`, `deployment`, and so on. We also use "resource" to talk about CPU and memory requests and limits. Always refer to API resources as "API resources" to avoid confusion with CPU and memory resources. + +### When to use Kubernetes API terminologies + +The different Kubernetes API terminologies are: + +- Resource type: the name used in the API URL (such as `pods`, `namespaces`) +- Resource: a single instance of a resource type (such as `pod`, `secret`) +- Object: a resource that serves as a "record of intent". An object is a desired state for a specific part of your cluster, which the Kubernetes control plane tries to maintain. + +Always use "resource" or "object" when referring to an API resource in docs. For example, use "a `Secret` object" over just "a `Secret`". + +### API resource names + +Always format API resource names using [UpperCamelCase](https://en.wikipedia.org/wiki/Camel_case), also known as PascalCase, and code formatting. + +For inline code in an HTML document, use the `<code>` tag. In a Markdown document, use the backtick (`` ` ``). + +Don't split an API object name into separate words. For example, use `PodTemplateList`, not Pod Template List. + +For more information about PascalCase and code formatting, please review the related guidance on [Use upper camel case for API objects](/docs/contribute/style/style-guide/#use-upper-camel-case-for-api-objects) and [Use code style for inline code, commands, and API objects](/docs/contribute/style/style-guide/#code-style-inline-code). + +For more information about Kubernetes API terminologies, please review the related guidance on [Kubernetes API terminology](/docs/reference/using-api/api-concepts/#standard-api-terminology). + ## Code snippet formatting ### Don't include the command prompt @@ -361,7 +387,7 @@ Beware. ### Katacoda Embedded Live Environment -This button lets users run Minikube in their browser using the [Katacoda Terminal](https://www.katacoda.com/embed/panel). +This button lets users run Minikube in their browser using the Katacoda Terminal. It lowers the barrier of entry by allowing users to use Minikube with one click instead of going through the complete Minikube and Kubectl installation process locally. diff --git a/content/en/docs/contribute/suggesting-improvements.md b/content/en/docs/contribute/suggesting-improvements.md index 9cab3f7a72..d79df11476 100644 --- a/content/en/docs/contribute/suggesting-improvements.md +++ b/content/en/docs/contribute/suggesting-improvements.md @@ -1,6 +1,5 @@ --- title: Suggesting content improvements -slug: suggest-improvements content_type: concept weight: 10 card: diff --git a/content/en/docs/doc-contributor-tools/linkchecker/README.md b/content/en/docs/doc-contributor-tools/linkchecker/README.md index a575c4d1fe..6d4b714655 100644 --- a/content/en/docs/doc-contributor-tools/linkchecker/README.md +++ b/content/en/docs/doc-contributor-tools/linkchecker/README.md @@ -46,7 +46,7 @@ To run the link checker: 2. Run the following command: ``` - make docker-internal-linkcheck + make container-internal-linkcheck ``` ## Understanding the output diff --git a/content/en/docs/home/_index.md b/content/en/docs/home/_index.md index b2ebb004a7..7297da2806 100644 --- a/content/en/docs/home/_index.md +++ b/content/en/docs/home/_index.md @@ -60,7 +60,7 @@ cards: title: K8s Release Notes description: If you are installing Kubernetes or upgrading to the newest version, refer to the current release notes. button: "Download Kubernetes" - button_path: "/docs/setup/release/notes" + button_path: "/releases/download" - name: about title: About the documentation description: This website contains documentation for the current and previous 4 versions of Kubernetes. diff --git a/content/en/docs/home/supported-doc-versions.md b/content/en/docs/home/supported-doc-versions.md index b955f95f56..fd3559a4d3 100644 --- a/content/en/docs/home/supported-doc-versions.md +++ b/content/en/docs/home/supported-doc-versions.md @@ -10,3 +10,8 @@ card: This website contains documentation for the current version of Kubernetes and the four previous versions of Kubernetes. + +The availability of documentation for a Kubernetes version is separate from whether +that release is currently supported. +Read [Support period](/releases/patch-releases/#support-period) to learn about +which versions of Kubernetes are officially supported, and for how long. \ No newline at end of file diff --git a/content/en/docs/images/diagram-guide-example-3.svg b/content/en/docs/images/diagram-guide-example-3.svg new file mode 100644 index 0000000000..85fef13910 --- /dev/null +++ b/content/en/docs/images/diagram-guide-example-3.svg @@ -0,0 +1 @@ +<svg id="graph-div" width="100%" xmlns="http://www.w3.org/2000/svg" height="747" style="max-width: 1540px;" viewBox="-50 -10 1540 747"><style>#graph-div {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#graph-div .error-icon{fill:#552222;}#graph-div .error-text{fill:#552222;stroke:#552222;}#graph-div .edge-thickness-normal{stroke-width:2px;}#graph-div .edge-thickness-thick{stroke-width:3.5px;}#graph-div .edge-pattern-solid{stroke-dasharray:0;}#graph-div .edge-pattern-dashed{stroke-dasharray:3;}#graph-div .edge-pattern-dotted{stroke-dasharray:2;}#graph-div .marker{fill:#666;stroke:#666;}#graph-div .marker.cross{stroke:#666;}#graph-div svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#graph-div .actor{stroke:hsl(0, 0%, 83%);fill:#eee;}#graph-div text.actor>tspan{fill:#333;stroke:none;}#graph-div .actor-line{stroke:#666;}#graph-div .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#graph-div .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#graph-div #arrowhead path{fill:#333;stroke:#333;}#graph-div .sequenceNumber{fill:white;}#graph-div #sequencenumber{fill:#333;}#graph-div #crosshead path{fill:#333;stroke:#333;}#graph-div .messageText{fill:#333;stroke:#333;}#graph-div .labelBox{stroke:hsl(0, 0%, 83%);fill:#eee;}#graph-div .labelText,#graph-div .labelText>tspan{fill:#333;stroke:none;}#graph-div .loopText,#graph-div .loopText>tspan{fill:#333;stroke:none;}#graph-div .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(0, 0%, 83%);fill:hsl(0, 0%, 83%);}#graph-div .note{stroke:#999;fill:#666;}#graph-div .noteText,#graph-div .noteText>tspan{fill:#fff;stroke:none;}#graph-div .activation0{fill:#f4f4f4;stroke:#666;}#graph-div .activation1{fill:#f4f4f4;stroke:#666;}#graph-div .activation2{fill:#f4f4f4;stroke:#666;}#graph-div .actorPopupMenu{position:absolute;}#graph-div .actorPopupMenuPanel{position:absolute;fill:#eee;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#graph-div .actor-man line{stroke:hsl(0, 0%, 83%);fill:#eee;}#graph-div .actor-man circle,#graph-div line{stroke:hsl(0, 0%, 83%);fill:#eee;stroke-width:2px;}#graph-div :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}</style><g></g><defs><symbol id="computer" width="24" height="24"><path transform="scale(.5)" d="M2 2v13h20v-13h-20zm18 11h-16v-9h16v9zm-10.228 6l.466-1h3.524l.467 1h-4.457zm14.228 3h-24l2-6h2.104l-1.33 4h18.45l-1.297-4h2.073l2 6zm-5-10h-14v-7h14v7z"></path></symbol></defs><defs><symbol id="database" fill-rule="evenodd" clip-rule="evenodd"><path transform="scale(.5)" d="M12.258.001l.256.004.255.005.253.008.251.01.249.012.247.015.246.016.242.019.241.02.239.023.236.024.233.027.231.028.229.031.225.032.223.034.22.036.217.038.214.04.211.041.208.043.205.045.201.046.198.048.194.05.191.051.187.053.183.054.18.056.175.057.172.059.168.06.163.061.16.063.155.064.15.066.074.033.073.033.071.034.07.034.069.035.068.035.067.035.066.035.064.036.064.036.062.036.06.036.06.037.058.037.058.037.055.038.055.038.053.038.052.038.051.039.05.039.048.039.047.039.045.04.044.04.043.04.041.04.04.041.039.041.037.041.036.041.034.041.033.042.032.042.03.042.029.042.027.042.026.043.024.043.023.043.021.043.02.043.018.044.017.043.015.044.013.044.012.044.011.045.009.044.007.045.006.045.004.045.002.045.001.045v17l-.001.045-.002.045-.004.045-.006.045-.007.045-.009.044-.011.045-.012.044-.013.044-.015.044-.017.043-.018.044-.02.043-.021.043-.023.043-.024.043-.026.043-.027.042-.029.042-.03.042-.032.042-.033.042-.034.041-.036.041-.037.041-.039.041-.04.041-.041.04-.043.04-.044.04-.045.04-.047.039-.048.039-.05.039-.051.039-.052.038-.053.038-.055.038-.055.038-.058.037-.058.037-.06.037-.06.036-.062.036-.064.036-.064.036-.066.035-.067.035-.068.035-.069.035-.07.034-.071.034-.073.033-.074.033-.15.066-.155.064-.16.063-.163.061-.168.06-.172.059-.175.057-.18.056-.183.054-.187.053-.191.051-.194.05-.198.048-.201.046-.205.045-.208.043-.211.041-.214.04-.217.038-.22.036-.223.034-.225.032-.229.031-.231.028-.233.027-.236.024-.239.023-.241.02-.242.019-.246.016-.247.015-.249.012-.251.01-.253.008-.255.005-.256.004-.258.001-.258-.001-.256-.004-.255-.005-.253-.008-.251-.01-.249-.012-.247-.015-.245-.016-.243-.019-.241-.02-.238-.023-.236-.024-.234-.027-.231-.028-.228-.031-.226-.032-.223-.034-.22-.036-.217-.038-.214-.04-.211-.041-.208-.043-.204-.045-.201-.046-.198-.048-.195-.05-.19-.051-.187-.053-.184-.054-.179-.056-.176-.057-.172-.059-.167-.06-.164-.061-.159-.063-.155-.064-.151-.066-.074-.033-.072-.033-.072-.034-.07-.034-.069-.035-.068-.035-.067-.035-.066-.035-.064-.036-.063-.036-.062-.036-.061-.036-.06-.037-.058-.037-.057-.037-.056-.038-.055-.038-.053-.038-.052-.038-.051-.039-.049-.039-.049-.039-.046-.039-.046-.04-.044-.04-.043-.04-.041-.04-.04-.041-.039-.041-.037-.041-.036-.041-.034-.041-.033-.042-.032-.042-.03-.042-.029-.042-.027-.042-.026-.043-.024-.043-.023-.043-.021-.043-.02-.043-.018-.044-.017-.043-.015-.044-.013-.044-.012-.044-.011-.045-.009-.044-.007-.045-.006-.045-.004-.045-.002-.045-.001-.045v-17l.001-.045.002-.045.004-.045.006-.045.007-.045.009-.044.011-.045.012-.044.013-.044.015-.044.017-.043.018-.044.02-.043.021-.043.023-.043.024-.043.026-.043.027-.042.029-.042.03-.042.032-.042.033-.042.034-.041.036-.041.037-.041.039-.041.04-.041.041-.04.043-.04.044-.04.046-.04.046-.039.049-.039.049-.039.051-.039.052-.038.053-.038.055-.038.056-.038.057-.037.058-.037.06-.037.061-.036.062-.036.063-.036.064-.036.066-.035.067-.035.068-.035.069-.035.07-.034.072-.034.072-.033.074-.033.151-.066.155-.064.159-.063.164-.061.167-.06.172-.059.176-.057.179-.056.184-.054.187-.053.19-.051.195-.05.198-.048.201-.046.204-.045.208-.043.211-.041.214-.04.217-.038.22-.036.223-.034.226-.032.228-.031.231-.028.234-.027.236-.024.238-.023.241-.02.243-.019.245-.016.247-.015.249-.012.251-.01.253-.008.255-.005.256-.004.258-.001.258.001zm-9.258 20.499v.01l.001.021.003.021.004.022.005.021.006.022.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.023.018.024.019.024.021.024.022.025.023.024.024.025.052.049.056.05.061.051.066.051.07.051.075.051.079.052.084.052.088.052.092.052.097.052.102.051.105.052.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.048.144.049.147.047.152.047.155.047.16.045.163.045.167.043.171.043.176.041.178.041.183.039.187.039.19.037.194.035.197.035.202.033.204.031.209.03.212.029.216.027.219.025.222.024.226.021.23.02.233.018.236.016.24.015.243.012.246.01.249.008.253.005.256.004.259.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.021.224-.024.22-.026.216-.027.212-.028.21-.031.205-.031.202-.034.198-.034.194-.036.191-.037.187-.039.183-.04.179-.04.175-.042.172-.043.168-.044.163-.045.16-.046.155-.046.152-.047.148-.048.143-.049.139-.049.136-.05.131-.05.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.053.083-.051.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.05.023-.024.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.023.01-.022.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.127l-.077.055-.08.053-.083.054-.085.053-.087.052-.09.052-.093.051-.095.05-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.045-.118.044-.12.043-.122.042-.124.042-.126.041-.128.04-.13.04-.132.038-.134.038-.135.037-.138.037-.139.035-.142.035-.143.034-.144.033-.147.032-.148.031-.15.03-.151.03-.153.029-.154.027-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.01-.179.008-.179.008-.181.006-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.006-.179-.008-.179-.008-.178-.01-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.027-.153-.029-.151-.03-.15-.03-.148-.031-.146-.032-.145-.033-.143-.034-.141-.035-.14-.035-.137-.037-.136-.037-.134-.038-.132-.038-.13-.04-.128-.04-.126-.041-.124-.042-.122-.042-.12-.044-.117-.043-.116-.045-.113-.045-.112-.046-.109-.047-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.05-.093-.052-.09-.051-.087-.052-.085-.053-.083-.054-.08-.054-.077-.054v4.127zm0-5.654v.011l.001.021.003.021.004.021.005.022.006.022.007.022.009.022.01.022.011.023.012.023.013.023.015.024.016.023.017.024.018.024.019.024.021.024.022.024.023.025.024.024.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.052.11.051.114.051.119.052.123.05.127.051.131.05.135.049.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.044.171.042.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.022.23.02.233.018.236.016.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.012.241-.015.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.048.139-.05.136-.049.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.051.051-.049.023-.025.023-.024.021-.025.02-.024.019-.024.018-.024.017-.024.015-.023.014-.023.013-.024.012-.022.01-.023.01-.023.008-.022.006-.022.006-.022.004-.021.004-.022.001-.021.001-.021v-4.139l-.077.054-.08.054-.083.054-.085.052-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.044-.118.044-.12.044-.122.042-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.035-.143.033-.144.033-.147.033-.148.031-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.009-.179.009-.179.007-.181.007-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.007-.179-.007-.179-.009-.178-.009-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.031-.146-.033-.145-.033-.143-.033-.141-.035-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.04-.126-.041-.124-.042-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.051-.093-.051-.09-.051-.087-.053-.085-.052-.083-.054-.08-.054-.077-.054v4.139zm0-5.666v.011l.001.02.003.022.004.021.005.022.006.021.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.024.018.023.019.024.021.025.022.024.023.024.024.025.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.051.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.043.171.043.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.021.23.02.233.018.236.017.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.013.241-.014.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.049.139-.049.136-.049.131-.051.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.049.023-.025.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.022.01-.023.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.153l-.077.054-.08.054-.083.053-.085.053-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.048-.105.048-.106.048-.109.046-.111.046-.114.046-.115.044-.118.044-.12.043-.122.043-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.034-.143.034-.144.033-.147.032-.148.032-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.024-.161.024-.162.023-.163.023-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.01-.178.01-.179.009-.179.007-.181.006-.182.006-.182.004-.184.003-.184.001-.185.001-.185-.001-.184-.001-.184-.003-.182-.004-.182-.006-.181-.006-.179-.007-.179-.009-.178-.01-.176-.01-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.023-.162-.023-.161-.024-.159-.024-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.032-.146-.032-.145-.033-.143-.034-.141-.034-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.041-.126-.041-.124-.041-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.048-.105-.048-.102-.048-.1-.05-.097-.049-.095-.051-.093-.051-.09-.052-.087-.052-.085-.053-.083-.053-.08-.054-.077-.054v4.153zm8.74-8.179l-.257.004-.254.005-.25.008-.247.011-.244.012-.241.014-.237.016-.233.018-.231.021-.226.022-.224.023-.22.026-.216.027-.212.028-.21.031-.205.032-.202.033-.198.034-.194.036-.191.038-.187.038-.183.04-.179.041-.175.042-.172.043-.168.043-.163.045-.16.046-.155.046-.152.048-.148.048-.143.048-.139.049-.136.05-.131.05-.126.051-.123.051-.118.051-.114.052-.11.052-.106.052-.101.052-.096.052-.092.052-.088.052-.083.052-.079.052-.074.051-.07.052-.065.051-.06.05-.056.05-.051.05-.023.025-.023.024-.021.024-.02.025-.019.024-.018.024-.017.023-.015.024-.014.023-.013.023-.012.023-.01.023-.01.022-.008.022-.006.023-.006.021-.004.022-.004.021-.001.021-.001.021.001.021.001.021.004.021.004.022.006.021.006.023.008.022.01.022.01.023.012.023.013.023.014.023.015.024.017.023.018.024.019.024.02.025.021.024.023.024.023.025.051.05.056.05.06.05.065.051.07.052.074.051.079.052.083.052.088.052.092.052.096.052.101.052.106.052.11.052.114.052.118.051.123.051.126.051.131.05.136.05.139.049.143.048.148.048.152.048.155.046.16.046.163.045.168.043.172.043.175.042.179.041.183.04.187.038.191.038.194.036.198.034.202.033.205.032.21.031.212.028.216.027.22.026.224.023.226.022.231.021.233.018.237.016.241.014.244.012.247.011.25.008.254.005.257.004.26.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.022.224-.023.22-.026.216-.027.212-.028.21-.031.205-.032.202-.033.198-.034.194-.036.191-.038.187-.038.183-.04.179-.041.175-.042.172-.043.168-.043.163-.045.16-.046.155-.046.152-.048.148-.048.143-.048.139-.049.136-.05.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.05.051-.05.023-.025.023-.024.021-.024.02-.025.019-.024.018-.024.017-.023.015-.024.014-.023.013-.023.012-.023.01-.023.01-.022.008-.022.006-.023.006-.021.004-.022.004-.021.001-.021.001-.021-.001-.021-.001-.021-.004-.021-.004-.022-.006-.021-.006-.023-.008-.022-.01-.022-.01-.023-.012-.023-.013-.023-.014-.023-.015-.024-.017-.023-.018-.024-.019-.024-.02-.025-.021-.024-.023-.024-.023-.025-.051-.05-.056-.05-.06-.05-.065-.051-.07-.052-.074-.051-.079-.052-.083-.052-.088-.052-.092-.052-.096-.052-.101-.052-.106-.052-.11-.052-.114-.052-.118-.051-.123-.051-.126-.051-.131-.05-.136-.05-.139-.049-.143-.048-.148-.048-.152-.048-.155-.046-.16-.046-.163-.045-.168-.043-.172-.043-.175-.042-.179-.041-.183-.04-.187-.038-.191-.038-.194-.036-.198-.034-.202-.033-.205-.032-.21-.031-.212-.028-.216-.027-.22-.026-.224-.023-.226-.022-.231-.021-.233-.018-.237-.016-.241-.014-.244-.012-.247-.011-.25-.008-.254-.005-.257-.004-.26-.001-.26.001z"></path></symbol></defs><defs><symbol id="clock" width="24" height="24"><path transform="scale(.5)" d="M12 2c5.514 0 10 4.486 10 10s-4.486 10-10 10-10-4.486-10-10 4.486-10 10-10zm0-2c-6.627 0-12 5.373-12 12s5.373 12 12 12 12-5.373 12-12-5.373-12-12-12zm5.848 12.459c.202.038.202.333.001.372-1.907.361-6.045 1.111-6.547 1.111-.719 0-1.301-.582-1.301-1.301 0-.512.77-5.447 1.125-7.445.034-.192.312-.181.343.014l.985 6.238 5.394 1.011z"></path></symbol></defs><line id="actor14" x1="75" y1="80" x2="75" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g class="actor-man"><line id="actor-man-torso14" x1="75" y1="25" x2="75" y2="45"></line><line id="actor-man-arms14" x1="57" y1="33" x2="93" y2="33"></line><line x1="57" y1="60" x2="75" y2="45"></line><line x1="75" y1="45" x2="91" y2="60"></line><circle cx="75" cy="10" r="15" width="150" height="65"></circle><text x="75" y="67.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="75" dy="0">me</tspan></text></g><g><line id="actor15" x1="357" y1="5" x2="357" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-15"><rect x="282" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="357" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="-14">control plane</tspan></text><text x="357" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="0"></tspan></text><text x="357" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="14">api-server</tspan></text></g></g><g><line id="actor16" x1="561" y1="5" x2="561" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-16"><rect x="486" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="561" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="-14">control plane</tspan></text><text x="561" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="0"></tspan></text><text x="561" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="14">etcd datastore</tspan></text></g></g><g><line id="actor17" x1="761" y1="5" x2="761" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-17"><rect x="686" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="761" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="-21">control plane</tspan></text><text x="761" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="-7"></tspan></text><text x="761" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="7">controller</tspan></text><text x="761" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="21">manager</tspan></text></g></g><g><line id="actor18" x1="961" y1="5" x2="961" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-18"><rect x="886" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="961" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="-14">control plane</tspan></text><text x="961" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="0"></tspan></text><text x="961" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="14">scheduler</tspan></text></g></g><g><line id="actor19" x1="1161" y1="5" x2="1161" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-19"><rect x="1086" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="1161" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="-14">node</tspan></text><text x="1161" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="0"></tspan></text><text x="1161" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="14">kubelet</tspan></text></g></g><g><line id="actor20" x1="1365" y1="5" x2="1365" y2="681" class="200" stroke-width="0.5px" stroke="#999"></line><g id="root-20"><rect x="1290" y="0" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="1365" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="-21">node</tspan></text><text x="1365" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="-7"></tspan></text><text x="1365" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="7">container</tspan></text><text x="1365" y="32.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="21">runtime</tspan></text></g></g><defs><marker id="arrowhead" refX="9" refY="5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z"></path></marker></defs><defs><marker id="crosshead" markerWidth="15" markerHeight="8" orient="auto" refX="16" refY="4"><path fill="black" stroke="#000000" stroke-width="1px" d="M 9,2 V 6 L16,4 Z" style="stroke-dasharray: 0, 0;"></path><path fill="none" stroke="#000000" stroke-width="1px" d="M 0,1 L 6,7 M 6,1 L 0,7" style="stroke-dasharray: 0, 0;"></path></marker></defs><defs><marker id="filled-head" refX="18" refY="7" markerWidth="20" markerHeight="28" orient="auto"><path d="M 18,7 L9,13 L14,7 L9,1 Z"></path></marker></defs><defs><marker id="sequencenumber" refX="15" refY="15" markerWidth="60" markerHeight="40" orient="auto"><circle cx="15" cy="15" r="6"></circle></marker></defs><text x="216" y="80" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">1. kubectl create -f pod.yaml</text><line x1="75" y1="113" x2="357" y2="113" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="459" y="128" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">2. save new state</text><line x1="357" y1="161" x2="561" y2="161" class="messageLine1" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="stroke-dasharray: 3, 3; fill: none;"></line><text x="559" y="176" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">3. check for changes</text><line x1="761" y1="209" x2="357" y2="209" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="659" y="224" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">4. watch for unassigned pods(s)</text><line x1="961" y1="257" x2="357" y2="257" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="659" y="272" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">5. notify about pod w nodename=" "</text><line x1="357" y1="305" x2="961" y2="305" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="659" y="320" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">6. assign pod to node</text><line x1="961" y1="353" x2="357" y2="353" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="459" y="368" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">7. save new state</text><line x1="357" y1="401" x2="561" y2="401" class="messageLine1" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="stroke-dasharray: 3, 3; fill: none;"></line><text x="759" y="416" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">8. look for newly assigned pod(s)</text><line x1="1161" y1="449" x2="357" y2="449" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="759" y="464" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">9. bind pod to node</text><line x1="357" y1="497" x2="1161" y2="497" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="1263" y="512" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">10. start container</text><line x1="1161" y1="545" x2="1365" y2="545" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="759" y="560" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">11. update pod status</text><line x1="1161" y1="593" x2="357" y2="593" class="messageLine0" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="fill: none;"></line><text x="459" y="608" text-anchor="middle" dominant-baseline="middle" alignment-baseline="middle" class="messageText" dy="1em" style="font-family: "trebuchet ms", verdana, arial, sans-serif; font-size: 16px; font-weight: 400;">12. save new state</text><line x1="357" y1="641" x2="561" y2="641" class="messageLine1" stroke-width="2" stroke="none" marker-end="url(#arrowhead)" style="stroke-dasharray: 3, 3; fill: none;"></line><g class="actor-man"><line id="actor-man-torso20" x1="75" y1="686" x2="75" y2="706"></line><line id="actor-man-arms20" x1="57" y1="694" x2="93" y2="694"></line><line x1="57" y1="721" x2="75" y2="706"></line><line x1="75" y1="706" x2="91" y2="721"></line><circle cx="75" cy="671" r="15" width="150" height="65"></circle><text x="75" y="728.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="75" dy="0">me</tspan></text></g><g><rect x="282" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="357" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="-14">control plane</tspan></text><text x="357" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="0"></tspan></text><text x="357" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="357" dy="14">api-server</tspan></text></g><g><rect x="486" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="561" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="-14">control plane</tspan></text><text x="561" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="0"></tspan></text><text x="561" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="561" dy="14">etcd datastore</tspan></text></g><g><rect x="686" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="761" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="-21">control plane</tspan></text><text x="761" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="-7"></tspan></text><text x="761" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="7">controller</tspan></text><text x="761" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="761" dy="21">manager</tspan></text></g><g><rect x="886" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="961" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="-14">control plane</tspan></text><text x="961" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="0"></tspan></text><text x="961" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="961" dy="14">scheduler</tspan></text></g><g><rect x="1086" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="1161" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="-14">node</tspan></text><text x="1161" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="0"></tspan></text><text x="1161" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1161" dy="14">kubelet</tspan></text></g><g><rect x="1290" y="661" fill="#eaeaea" stroke="#666" width="150" height="65" rx="3" ry="3" class="actor"></rect><text x="1365" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="-21">node</tspan></text><text x="1365" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="-7"></tspan></text><text x="1365" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="7">container</tspan></text><text x="1365" y="693.5" dominant-baseline="central" alignment-baseline="central" class="actor" style="text-anchor: middle; font-size: 14px; font-weight: 400; font-family: Open-Sans, "sans-serif";"><tspan x="1365" dy="21">runtime</tspan></text></g></svg> \ No newline at end of file diff --git a/content/en/docs/images/ingress.svg b/content/en/docs/images/ingress.svg new file mode 100644 index 0000000000..450a0aae9b --- /dev/null +++ b/content/en/docs/images/ingress.svg @@ -0,0 +1 @@ +<svg viewBox="0 0 683.28125 224" style="max-width: 683.28125px;" height="224" aria-labelledby="chart-title-graph-div chart-desc-graph-div" role="img" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" width="100%" id="graph-div"><title id="chart-title-graph-div">
cluster
Ingress-managed
load balancer
routing rule
Ingress
Pod
Service
Pod
client
 \ No newline at end of file diff --git a/content/en/docs/images/ingressFanOut.svg b/content/en/docs/images/ingressFanOut.svg new file mode 100644 index 0000000000..a6bf202635 --- /dev/null +++ b/content/en/docs/images/ingressFanOut.svg @@ -0,0 +1 @@ +
cluster
Ingress-managed
load balancer
/foo
/bar
Ingress, 178.91.123.132
Pod
Service service1:4200
Pod
Pod
Service service2:8080
Pod
client
 \ No newline at end of file diff --git a/content/en/docs/images/ingressNameBased.svg b/content/en/docs/images/ingressNameBased.svg new file mode 100644 index 0000000000..7e1d7be98c --- /dev/null +++ b/content/en/docs/images/ingressNameBased.svg @@ -0,0 +1 @@ +
cluster
Ingress-managed
load balancer
Host: foo.bar.com
Host: bar.foo.com
Ingress, 178.91.123.132
Pod
Service service1:80
Pod
Pod
Service service2:80
Pod
client
 \ No newline at end of file diff --git a/content/en/docs/images/tutor-service-nodePort-fig01.svg b/content/en/docs/images/tutor-service-nodePort-fig01.svg new file mode 100644 index 0000000000..bb4d866f85 --- /dev/null +++ b/content/en/docs/images/tutor-service-nodePort-fig01.svg @@ -0,0 +1 @@ +
SNAT
SNAT
client
Node 2
Node 1
Endpoint
 \ No newline at end of file diff --git a/content/en/docs/images/tutor-service-nodePort-fig02.svg b/content/en/docs/images/tutor-service-nodePort-fig02.svg new file mode 100644 index 0000000000..1a891575e5 --- /dev/null +++ b/content/en/docs/images/tutor-service-nodePort-fig02.svg @@ -0,0 +1 @@ +
client
Node 1
Node 2
endpoint
 \ No newline at end of file diff --git a/content/en/docs/reference/_index.md b/content/en/docs/reference/_index.md index c22e0ca836..2ed1396c8f 100644 --- a/content/en/docs/reference/_index.md +++ b/content/en/docs/reference/_index.md @@ -43,7 +43,7 @@ client libraries: ## CLI -* [kubectl](/docs/reference/kubectl/overview/) - Main CLI tool for running commands and managing Kubernetes clusters. +* [kubectl](/docs/reference/kubectl/) - Main CLI tool for running commands and managing Kubernetes clusters. * [JSONPath](/docs/reference/kubectl/jsonpath/) - Syntax guide for using [JSONPath expressions](https://goessner.net/articles/JsonPath/) with kubectl. * [kubeadm](/docs/reference/setup-tools/kubeadm/) - CLI tool to easily provision a secure Kubernetes cluster. @@ -64,6 +64,9 @@ client libraries: * [Scheduler Policies](/docs/reference/scheduling/policies) * [Scheduler Profiles](/docs/reference/scheduling/config#profiles) +* List of [ports and protocols](/docs/reference/ports-and-protocols/) that + should be open on control plane and worker nodes + ## Config APIs This section hosts the documentation for "unpublished" APIs which are used to @@ -72,14 +75,21 @@ by the API server in a RESTful way though they are essential for a user or an operator to use or manage a cluster. * [kube-apiserver configuration (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/) -* [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) -* [kube-scheduler configuration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) -* [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) -* [kube-scheduler policy reference (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) +* [kube-apiserver configuration (v1)](/docs/reference/config-api/apiserver-config.v1/) +* [kube-apiserver encryption (v1)](/docs/reference/config-api/apiserver-encryption.v1/) +* [kube-apiserver event rate limit (v1alpha1)](/docs/reference/config-api/apiserver-eventratelimit.v1alpha1/) +* [kubelet configuration (v1alpha1)](/docs/reference/config-api/kubelet-config.v1alpha1/) and + [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) +* [kubelet credential providers (v1alpha1)](/docs/reference/config-api/kubelet-credentialprovider.v1alpha1/) +* [kubelet credential providers (v1beta1)](/docs/reference/config-api/kubelet-credentialprovider.v1beta1/) +* [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) and + [kube-scheduler configuration (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) * [kube-proxy configuration (v1alpha1)](/docs/reference/config-api/kube-proxy-config.v1alpha1/) * [`audit.k8s.io/v1` API](/docs/reference/config-api/apiserver-audit.v1/) -* [Client authentication API (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) +* [Client authentication API (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) and + [Client authentication API (v1)](/docs/reference/config-api/client-authentication.v1/) * [WebhookAdmission configuration (v1)](/docs/reference/config-api/apiserver-webhookadmission.v1/) +* [ImagePolicy API (v1alpha1)](/docs/reference/config-api/imagepolicy.v1alpha1/) ## Config API for kubeadm @@ -89,6 +99,6 @@ operator to use or manage a cluster. ## Design Docs An archive of the design docs for Kubernetes functionality. Good starting points are -[Kubernetes Architecture](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md) and -[Kubernetes Design Overview](https://git.k8s.io/community/contributors/design-proposals). +[Kubernetes Architecture](https://git.k8s.io/design-proposals-archive/architecture/architecture.md) and +[Kubernetes Design Overview](https://git.k8s.io/design-proposals-archive). diff --git a/content/en/docs/reference/access-authn-authz/_index.md b/content/en/docs/reference/access-authn-authz/_index.md index 86d06488a8..3677f79c57 100644 --- a/content/en/docs/reference/access-authn-authz/_index.md +++ b/content/en/docs/reference/access-authn-authz/_index.md @@ -24,3 +24,5 @@ Reference documentation: - Service accounts - [Developer guide](/docs/tasks/configure-pod-container/configure-service-account/) - [Administration](/docs/reference/access-authn-authz/service-accounts-admin/) +- [Kubelet Authentication & Authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/) + - including kubelet [TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) diff --git a/content/en/docs/reference/access-authn-authz/abac.md b/content/en/docs/reference/access-authn-authz/abac.md index 197901a170..4587bf1f83 100644 --- a/content/en/docs/reference/access-authn-authz/abac.md +++ b/content/en/docs/reference/access-authn-authz/abac.md @@ -33,13 +33,13 @@ properties: - `group`, type string; if you specify `group`, it must match one of the groups of the authenticated user. `system:authenticated` matches all authenticated requests. `system:unauthenticated` matches all unauthenticated requests. - Resource-matching properties: - `apiGroup`, type string; an API group. - - Ex: `extensions` + - Ex: `apps`, `networking.k8s.io` - Wildcard: `*` matches all API groups. - `namespace`, type string; a namespace. - Ex: `kube-system` - Wildcard: `*` matches all resource requests. - `resource`, type string; a resource type - - Ex: `pods` + - Ex: `pods`, `deployments` - Wildcard: `*` matches all resource requests. - Non-resource-matching properties: - `nonResourcePath`, type string; non-resource request paths. diff --git a/content/en/docs/reference/access-authn-authz/admission-controllers.md b/content/en/docs/reference/access-authn-authz/admission-controllers.md index 0461c09f53..f03b04f8e3 100644 --- a/content/en/docs/reference/access-authn-authz/admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/admission-controllers.md @@ -30,9 +30,9 @@ mutating and validating (respectively) which are configured in the API. Admission controllers may be "validating", "mutating", or both. Mutating -controllers may modify the objects they admit; validating controllers may not. +controllers may modify related objects to the requests they admit; validating controllers may not. -Admission controllers limit requests to create, delete, modify or connect to (proxy). They do not support read requests. +Admission controllers limit requests to create, delete, modify objects or connect to proxy. They do not limit requests to read objects. The admission control process proceeds in two phases. In the first phase, mutating admission controllers are run. In the second phase, validating @@ -94,7 +94,7 @@ kube-apiserver -h | grep enable-admission-plugins In the current version, the default ones are: ```shell -CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook +CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook ``` ## What does each admission controller do? @@ -139,7 +139,7 @@ requests with the `spec.signerName` requested on the CertificateSigningRequest r See [Certificate Signing Requests](/docs/reference/access-authn-authz/certificate-signing-requests/) for more information on the permissions required to perform different actions on CertificateSigningRequest resources. -### CertificateSubjectRestrictions {#certificatesubjectrestrictions} +### CertificateSubjectRestriction {#certificatesubjectrestriction} This admission controller observes creation of CertificateSigningRequest resources that have a `spec.signerName` of `kubernetes.io/kube-apiserver-client`. It rejects any request that specifies a 'group' (or 'organization attribute') @@ -232,12 +232,10 @@ of it. This admission controller mitigates the problem where the API server gets flooded by event requests. The cluster admin can specify event rate limits by: - * Enabling the `EventRateLimit` admission controller; - * Referencing an `EventRateLimit` configuration file from the file provided to the API - server's command line flag `--admission-control-config-file`: +* Enabling the `EventRateLimit` admission controller; +* Referencing an `EventRateLimit` configuration file from the file provided to the API + server's command line flag `--admission-control-config-file`: -{{< tabs name="eventratelimit_example" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -246,19 +244,6 @@ plugins: path: eventconfig.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: EventRateLimit - path: eventconfig.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} There are four types of limits that can be specified in the configuration: @@ -283,7 +268,7 @@ limits: burst: 50 ``` -See the [EventRateLimit proposal](https://git.k8s.io/community/contributors/design-proposals/api-machinery/admission_control_event_rate_limit.md) +See the [EventRateLimit Config API (v1alpha1)](/docs/reference/config-api/apiserver-eventratelimit.v1alpha1/) for more details. ### ExtendedResourceToleration {#extendedresourcetoleration} @@ -319,8 +304,6 @@ imagePolicy: Reference the ImagePolicyWebhook configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: -{{< tabs name="imagepolicywebhook_example1" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -329,24 +312,9 @@ plugins: path: imagepolicyconfig.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: ImagePolicyWebhook - path: imagepolicyconfig.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} Alternatively, you can embed the configuration directly in the file: -{{< tabs name="imagepolicywebhook_example2" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -360,31 +328,14 @@ plugins: retryBackoff: 500 defaultAllow: true ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: ImagePolicyWebhook - configuration: - imagePolicy: - kubeConfigFile: - allowTTL: 50 - denyTTL: 50 - retryBackoff: 500 - defaultAllow: true -``` -{{% /tab %}} -{{< /tabs >}} The ImagePolicyWebhook config file must reference a [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS. -The kubeconfig file's cluster field must point to the remote service, and the user field must contain the returned authorizer. +The kubeconfig file's `cluster` field must point to the remote service, and the `user` field +must contain the returned authorizer. ```yaml # clusters refers to the remote service. @@ -405,11 +356,21 @@ users: For additional HTTP configuration, refer to the [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) documentation. -#### Request Payloads +#### Request payloads -When faced with an admission decision, the API Server POSTs a JSON serialized `imagepolicy.k8s.io/v1alpha1` `ImageReview` object describing the action. This object contains fields describing the containers being admitted, as well as any pod annotations that match `*.image-policy.k8s.io/*`. +When faced with an admission decision, the API Server POSTs a JSON serialized +`imagepolicy.k8s.io/v1alpha1` `ImageReview` object describing the action. +This object contains fields describing the containers being admitted, as well as +any pod annotations that match `*.image-policy.k8s.io/*`. -Note that webhook API objects are subject to the same versioning compatibility rules as other Kubernetes API objects. Implementers should be aware of looser compatibility promises for alpha objects and check the "apiVersion" field of the request to ensure correct deserialization. Additionally, the API Server must enable the imagepolicy.k8s.io/v1alpha1 API extensions group (`--runtime-config=imagepolicy.k8s.io/v1alpha1=true`). +{{ note }} +The webhook API objects are subject to the same versioning compatibility rules +as other Kubernetes API objects. Implementers should be aware of looser compatibility +promises for alpha objects and check the `apiVersion` field of the request to +ensure correct deserialization. +Additionally, the API Server must enable the `imagepolicy.k8s.io/v1alpha1` API extensions +group (`--runtime-config=imagepolicy.k8s.io/v1alpha1=true`). +{{ /note }} An example request body: @@ -434,7 +395,9 @@ An example request body: } ``` -The remote service is expected to fill the `ImageReviewStatus` field of the request and respond to either allow or disallow access. The response body's "spec" field is ignored and may be omitted. A permissive response would return: +The remote service is expected to fill the `ImageReviewStatus` field of the request and +respond to either allow or disallow access. The response body's `spec` field is ignored and +may be omitted. A permissive response would return: ```json { @@ -459,19 +422,23 @@ To disallow access, the service would return: } ``` -For further documentation refer to the `imagepolicy.v1alpha1` API objects and `plugin/pkg/admission/imagepolicy/admission.go`. +For further documentation refer to the +[`imagepolicy.v1alpha1` API](/docs/reference/config-api/imagepolicy.v1alpha1/). #### Extending with Annotations -All annotations on a Pod that match `*.image-policy.k8s.io/*` are sent to the webhook. Sending annotations allows users who are aware of the image policy backend to send extra information to it, and for different backends implementations to accept different information. +All annotations on a Pod that match `*.image-policy.k8s.io/*` are sent to the webhook. +Sending annotations allows users who are aware of the image policy backend to +send extra information to it, and for different backends implementations to +accept different information. Examples of information you might put here are: - * request to "break glass" to override a policy, in case of emergency. - * a ticket number from a ticket system that documents the break-glass request - * provide a hint to the policy server as to the imageID of the image being provided, to save it a lookup +* request to "break glass" to override a policy, in case of emergency. +* a ticket number from a ticket system that documents the break-glass request +* provide a hint to the policy server as to the imageID of the image being provided, to save it a lookup -In any case, the annotations are provided by the user and are not validated by Kubernetes in any way. In the future, if an annotation is determined to be widely useful, it may be promoted to a named field of `ImageReviewSpec`. +In any case, the annotations are provided by the user and are not validated by Kubernetes in any way. ### LimitPodHardAntiAffinityTopology {#limitpodhardantiaffinitytopology} @@ -480,14 +447,16 @@ This admission controller denies any pod that defines `AntiAffinity` topology ke ### LimitRanger {#limitranger} -This admission controller will observe the incoming request and ensure that it does not violate any of the constraints -enumerated in the `LimitRange` object in a `Namespace`. If you are using `LimitRange` objects in -your Kubernetes deployment, you MUST use this admission controller to enforce those constraints. LimitRanger can also -be used to apply default resource requests to Pods that don't specify any; currently, the default LimitRanger -applies a 0.1 CPU requirement to all Pods in the `default` namespace. +This admission controller will observe the incoming request and ensure that it does not violate +any of the constraints enumerated in the `LimitRange` object in a `Namespace`. If you are using +`LimitRange` objects in your Kubernetes deployment, you MUST use this admission controller to +enforce those constraints. LimitRanger can also be used to apply default resource requests to Pods +that don't specify any; currently, the default LimitRanger applies a 0.1 CPU requirement to all +Pods in the `default` namespace. -See the [limitRange design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) -and the [example of Limit Range](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/) for more details. +See the [LimitRange API reference](/docs/reference/kubernetes-api/policy-resources/limit-range-v1/) +and the [example of LimitRange](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/) +for more details. ### MutatingAdmissionWebhook {#mutatingadmissionwebhook} @@ -502,21 +471,20 @@ webhooks or validating admission controllers will permit the request to finish. If you disable the MutatingAdmissionWebhook, you must also disable the `MutatingWebhookConfiguration` object in the `admissionregistration.k8s.io/v1` -group/version via the `--runtime-config` flag (both are on by default in -versions >= 1.9). +group/version via the `--runtime-config` flag, both are on by default. #### Use caution when authoring and installing mutating webhooks - * Users may be confused when the objects they try to create are different from - what they get back. - * Built in control loops may break when the objects they try to create are - different when read back. - * Setting originally unset fields is less likely to cause problems than - overwriting fields set in the original request. Avoid doing the latter. - * Future changes to control loops for built-in resources or third-party resources - may break webhooks that work well today. Even when the webhook installation API - is finalized, not all possible webhook behaviors will be guaranteed to be supported - indefinitely. +* Users may be confused when the objects they try to create are different from + what they get back. +* Built in control loops may break when the objects they try to create are + different when read back. + * Setting originally unset fields is less likely to cause problems than + overwriting fields set in the original request. Avoid doing the latter. +* Future changes to control loops for built-in resources or third-party resources + may break webhooks that work well today. Even when the webhook installation API + is finalized, not all possible webhook behaviors will be guaranteed to be supported + indefinitely. ### NamespaceAutoProvision {#namespaceautoprovision} @@ -533,26 +501,28 @@ If the namespace referenced from a request doesn't exist, the request is rejecte ### NamespaceLifecycle {#namespacelifecycle} -This admission controller enforces that a `Namespace` that is undergoing termination cannot have new objects created in it, -and ensures that requests in a non-existent `Namespace` are rejected. This admission controller also prevents deletion of -three system reserved namespaces `default`, `kube-system`, `kube-public`. +This admission controller enforces that a `Namespace` that is undergoing termination cannot have +new objects created in it, and ensures that requests in a non-existent `Namespace` are rejected. +This admission controller also prevents deletion of three system reserved namespaces `default`, +`kube-system`, `kube-public`. -A `Namespace` deletion kicks off a sequence of operations that remove all objects (pods, services, etc.) in that -namespace. In order to enforce integrity of that process, we strongly recommend running this admission controller. +A `Namespace` deletion kicks off a sequence of operations that remove all objects (pods, services, +etc.) in that namespace. In order to enforce integrity of that process, we strongly recommend +running this admission controller. ### NodeRestriction {#noderestriction} This admission controller limits the `Node` and `Pod` objects a kubelet can modify. In order to be limited by this admission controller, kubelets must use credentials in the `system:nodes` group, with a username in the form `system:node:`. Such kubelets will only be allowed to modify their own `Node` API object, and only modify `Pod` API objects that are bound to their node. -In Kubernetes 1.11+, kubelets are not allowed to update or remove taints from their `Node` API object. +kubelets are not allowed to update or remove taints from their `Node` API object. -In Kubernetes 1.13+, the `NodeRestriction` admission plugin prevents kubelets from deleting their `Node` API object, +The `NodeRestriction` admission plugin prevents kubelets from deleting their `Node` API object, and enforces kubelet modification of labels under the `kubernetes.io/` or `k8s.io/` prefixes as follows: * **Prevents** kubelets from adding/removing/updating labels with a `node-restriction.kubernetes.io/` prefix. -This label prefix is reserved for administrators to label their `Node` objects for workload isolation purposes, -and kubelets will not be allowed to modify labels with that prefix. + This label prefix is reserved for administrators to label their `Node` objects for workload isolation purposes, + and kubelets will not be allowed to modify labels with that prefix. * **Allows** kubelets to add/remove/update these labels and label prefixes: * `kubernetes.io/hostname` * `kubernetes.io/arch` @@ -566,9 +536,11 @@ and kubelets will not be allowed to modify labels with that prefix. * `kubelet.kubernetes.io/`-prefixed labels * `node.kubernetes.io/`-prefixed labels -Use of any other labels under the `kubernetes.io` or `k8s.io` prefixes by kubelets is reserved, and may be disallowed or allowed by the `NodeRestriction` admission plugin in the future. +Use of any other labels under the `kubernetes.io` or `k8s.io` prefixes by kubelets is reserved, +and may be disallowed or allowed by the `NodeRestriction` admission plugin in the future. -Future versions may add additional restrictions to ensure kubelets have the minimal set of permissions required to operate correctly. +Future versions may add additional restrictions to ensure kubelets have the minimal set of +permissions required to operate correctly. ### OwnerReferencesPermissionEnforcement {#ownerreferencespermissionenforcement} @@ -580,15 +552,13 @@ subresource of the referenced *owner* can change it. ### PersistentVolumeClaimResize {#persistentvolumeclaimresize} -This admission controller implements additional validations for checking incoming `PersistentVolumeClaim` resize requests. +{{< feature-state for_k8s_version="v1.24" state="stable" >}} -{{< note >}} -Support for volume resizing is available as an alpha feature. Admins must set the feature gate `ExpandPersistentVolumes` -to `true` to enable resizing. -{{< /note >}} +This admission controller implements additional validations for checking incoming +`PersistentVolumeClaim` resize requests. -After enabling the `ExpandPersistentVolumes` feature gate, enabling the `PersistentVolumeClaimResize` admission -controller is recommended, too. This admission controller prevents resizing of all claims by default unless a claim's `StorageClass` +Enabling the `PersistentVolumeClaimResize` admission controller is recommended. +This admission controller prevents resizing of all claims by default unless a claim's `StorageClass` explicitly enables resizing by setting `allowVolumeExpansion` to `true`. For example: all `PersistentVolumeClaim`s created from the following `StorageClass` support volume expansion: @@ -627,9 +597,10 @@ Starting from 1.11, this admission controller is disabled by default. {{< feature-state for_k8s_version="v1.5" state="alpha" >}} -This admission controller defaults and limits what node selectors may be used within a namespace by reading a namespace annotation and a global configuration. +This admission controller defaults and limits what node selectors may be used within a namespace +by reading a namespace annotation and a global configuration. -#### Configuration File Format +#### Configuration file format `PodNodeSelector` uses a configuration file to set options for the behavior of the backend. Note that the configuration file format will move to a versioned file in a future release. @@ -642,10 +613,9 @@ podNodeSelectorPluginConfig: namespace2: name-of-node-selector ``` -Reference the `PodNodeSelector` configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: +Reference the `PodNodeSelector` configuration file from the file provided to the API server's +command line flag `--admission-control-config-file`: -{{< tabs name="podnodeselector_example1" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -654,22 +624,11 @@ plugins: path: podnodeselector.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: PodNodeSelector - path: podnodeselector.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} #### Configuration Annotation Format -`PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces. + +`PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign +node selectors to namespaces. ```yaml apiVersion: v1 @@ -681,15 +640,17 @@ metadata: ``` #### Internal Behavior + This admission controller has the following behavior: -1. If the `Namespace` has an annotation with a key `scheduler.alpha.kubernetes.io/node-selector`, use its value as the -node selector. -2. If the namespace lacks such an annotation, use the `clusterDefaultNodeSelector` defined in the `PodNodeSelector` -plugin configuration file as the node selector. -3. Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts result in rejection. -4. Evaluate the pod's node selector against the namespace-specific allowed selector defined the plugin configuration file. -Conflicts result in rejection. +1. If the `Namespace` has an annotation with a key `scheduler.alpha.kubernetes.io/node-selector`, + use its value as the node selector. +2. If the namespace lacks such an annotation, use the `clusterDefaultNodeSelector` defined in the + `PodNodeSelector` plugin configuration file as the node selector. +3. Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts + result in rejection. +4. Evaluate the pod's node selector against the namespace-specific allowed selector defined the + plugin configuration file. Conflicts result in rejection. {{< note >}} PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction @@ -698,7 +659,7 @@ admission plugin, which allows preventing pods from running on specifically tain ### PodSecurity {#podsecurity} -{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} This is the replacement for the deprecated [PodSecurityPolicy](#podsecuritypolicy) admission controller defined in the next section. This admission controller acts on creation and modification of the pod and @@ -715,14 +676,15 @@ for more information. This admission controller acts on creation and modification of the pod and determines if it should be admitted based on the requested security context and the available Pod Security Policies. -See also [Pod Security Policy documentation](/docs/concepts/policy/pod-security-policy/) +See also the [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) documentation for more information. ### PodTolerationRestriction {#podtolerationrestriction} {{< feature-state for_k8s_version="v1.7" state="alpha" >}} -The PodTolerationRestriction admission controller verifies any conflict between tolerations of a pod and the tolerations of its namespace. +The PodTolerationRestriction admission controller verifies any conflict between tolerations of a +pod and the tolerations of its namespace. It rejects the pod request if there is a conflict. It then merges the tolerations annotated on the namespace into the tolerations of the pod. The resulting tolerations are checked against a list of allowed tolerations annotated to the namespace. @@ -749,54 +711,62 @@ metadata: ### Priority {#priority} -The priority admission controller uses the `priorityClassName` field and populates the integer value of the priority. If the priority class is not found, the Pod is rejected. +The priority admission controller uses the `priorityClassName` field and populates the integer +value of the priority. +If the priority class is not found, the Pod is rejected. ### ResourceQuota {#resourcequota} -This admission controller will observe the incoming request and ensure that it does not violate any of the constraints -enumerated in the `ResourceQuota` object in a `Namespace`. If you are using `ResourceQuota` -objects in your Kubernetes deployment, you MUST use this admission controller to enforce quota constraints. +This admission controller will observe the incoming request and ensure that it does not violate +any of the constraints enumerated in the `ResourceQuota` object in a `Namespace`. If you are +using `ResourceQuota` objects in your Kubernetes deployment, you MUST use this admission +controller to enforce quota constraints. -See the [resourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) and the [example of Resource Quota](/docs/concepts/policy/resource-quotas/) for more details. +See the [ResourceQuota API reference](/docs/reference/kubernetes-api/policy-resources/resource-quota-v1/) +and the [example of Resource Quota](/docs/concepts/policy/resource-quotas/) for more details. ### RuntimeClass {#runtimeclass} {{< feature-state for_k8s_version="v1.20" state="stable" >}} -If you enable the `PodOverhead` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), and define a RuntimeClass with [Pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/) configured, this admission controller checks incoming -Pods. When enabled, this admission controller rejects any Pod create requests that have the overhead already set. -For Pods that have a RuntimeClass is configured and selected in their `.spec`, this admission controller sets `.spec.overhead` in the Pod based on the value defined in the corresponding RuntimeClass. - -{{< note >}} -The `.spec.overhead` field for Pod and the `.overhead` field for RuntimeClass are both in beta. If you do not enable the `PodOverhead` feature gate, all Pods are treated as if `.spec.overhead` is unset. -{{< /note >}} +If you define a RuntimeClass with [Pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/) +configured, this admission controller checks incoming Pods. +When enabled, this admission controller rejects any Pod create requests +that have the overhead already set. +For Pods that have a RuntimeClass configured and selected in their `.spec`, +this admission controller sets `.spec.overhead` in the Pod based on the value +defined in the corresponding RuntimeClass. See also [Pod Overhead](/docs/concepts/scheduling-eviction/pod-overhead/) for more information. ### SecurityContextDeny {#securitycontextdeny} -This admission controller will deny any pod that attempts to set certain escalating +This admission controller will deny any Pod that attempts to set certain escalating [SecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#securitycontext-v1-core) fields, as shown in the [Configure a Security Context for a Pod or Container](/docs/tasks/configure-pod-container/security-context/) task. -This should be enabled if a cluster doesn't utilize -[pod security policies](/docs/concepts/policy/pod-security-policy/) -to restrict the set of values a security context can take. +If you don't use [Pod Security admission](/docs/concepts/security/pod-security-admission/), +[PodSecurityPolicies](/docs/concepts/security/pod-security-policy/), nor any external enforcement mechanism, +then you could use this admission controller to restrict the set of values a security context can take. + +See [Pod Security Standards](/docs/concepts/security/pod-security-standards/) for more context on restricting +pod privileges. ### ServiceAccount {#serviceaccount} This admission controller implements automation for [serviceAccounts](/docs/tasks/configure-pod-container/configure-service-account/). -We strongly recommend using this admission controller if you intend to make use of Kubernetes `ServiceAccount` objects. +We strongly recommend using this admission controller if you intend to make use of Kubernetes +`ServiceAccount` objects. ### StorageObjectInUseProtection The `StorageObjectInUseProtection` plugin adds the `kubernetes.io/pvc-protection` or `kubernetes.io/pv-protection` finalizers to newly created Persistent Volume Claims (PVCs) or Persistent Volumes (PV). In case a user deletes a PVC or PV the PVC or PV is not removed until the finalizer is removed -from the PVC or PV by PVC or PV Protection Controller. +from the PVC or PV by PVC or PV Protection Controller. Refer to the [Storage Object in Use Protection](/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection) for more detailed information. @@ -805,7 +775,10 @@ for more detailed information. {{< feature-state for_k8s_version="v1.17" state="stable" >}} -This admission controller {{< glossary_tooltip text="taints" term_id="taint" >}} newly created Nodes as `NotReady` and `NoSchedule`. That tainting avoids a race condition that could cause Pods to be scheduled on new Nodes before their taints were updated to accurately reflect their reported conditions. +This admission controller {{< glossary_tooltip text="taints" term_id="taint" >}} newly created +Nodes as `NotReady` and `NoSchedule`. That tainting avoids a race condition that could cause Pods +to be scheduled on new Nodes before their taints were updated to accurately reflect their reported +conditions. ### ValidatingAdmissionWebhook {#validatingadmissionwebhook} @@ -823,11 +796,11 @@ If you disable the ValidatingAdmissionWebhook, you must also disable the group/version via the `--runtime-config` flag (both are on by default in versions 1.9 and later). - ## Is there a recommended set of admission controllers to use? -Yes. The recommended admission controllers are enabled by default (shown [here](/docs/reference/command-line-tools-reference/kube-apiserver/#options)), so you do not need to explicitly specify them. You can enable additional admission controllers beyond the default set using the `--enable-admission-plugins` flag (**order doesn't matter**). +Yes. The recommended admission controllers are enabled by default +(shown [here](/docs/reference/command-line-tools-reference/kube-apiserver/#options)), +so you do not need to explicitly specify them. +You can enable additional admission controllers beyond the default set using the +`--enable-admission-plugins` flag (**order doesn't matter**). -{{< note >}} -`--admission-control` was deprecated in 1.10 and replaced with `--enable-admission-plugins`. -{{< /note >}} diff --git a/content/en/docs/reference/access-authn-authz/authentication.md b/content/en/docs/reference/access-authn-authz/authentication.md index 70d416af60..1641cb8e54 100644 --- a/content/en/docs/reference/access-authn-authz/authentication.md +++ b/content/en/docs/reference/access-authn-authz/authentication.md @@ -53,7 +53,7 @@ or be treated as an anonymous user. ## Authentication strategies -Kubernetes uses client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth to +Kubernetes uses client certificates, bearer tokens, or an authenticating proxy to authenticate API requests through authentication plugins. As HTTP requests are made to the API server, plugins attempt to associate the following attributes with the request: @@ -104,7 +104,7 @@ See [Managing Certificates](/docs/tasks/administer-cluster/certificates/) for ho ### Static Token File The API server reads bearer tokens from a file when given the `--token-auth-file=SOMEFILE` option on the command line. Currently, tokens last indefinitely, and the token list cannot be -changed without restarting API server. +changed without restarting the API server. The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names. @@ -121,7 +121,7 @@ token,user,uid,"group1,group2,group3" When using bearer token authentication from an http client, the API server expects an `Authorization` header with a value of `Bearer -THETOKEN`. The bearer token must be a character sequence that can be +`. The bearer token must be a character sequence that can be put in an HTTP header value using no more than the encoding and quoting facilities of HTTP. For example: if the bearer token is `31ada4fd-adec-460c-809a-9e56ceb75269` then it would appear in an HTTP @@ -356,7 +356,7 @@ You can use an existing public OpenID Connect Identity Provider (such as Google, Or, you can run your own Identity Provider, such as [dex](https://dexidp.io/), [Keycloak](https://github.com/keycloak/keycloak), CloudFoundry [UAA](https://github.com/cloudfoundry/uaa), or -Tremolo Security's [OpenUnison](https://github.com/tremolosecurity/openunison). +Tremolo Security's [OpenUnison](https://openunison.github.io/). For an identity provider to work with Kubernetes it must: @@ -733,7 +733,7 @@ The following HTTP headers can be used to performing an impersonation request: * `Impersonate-User`: The username to act as. * `Impersonate-Group`: A group name to act as. Can be provided multiple times to set multiple groups. Optional. Requires "Impersonate-User". -* `Impersonate-Extra-( extra name )`: A dynamic header used to associate extra fields with the user. Optional. Requires "Impersonate-User". In order to be preserved consistently, `( extra name )` should be lower-case, and any characters which aren't [legal in HTTP header labels](https://tools.ietf.org/html/rfc7230#section-3.2.6) MUST be utf8 and [percent-encoded](https://tools.ietf.org/html/rfc3986#section-2.1). +* `Impersonate-Extra-( extra name )`: A dynamic header used to associate extra fields with the user. Optional. Requires "Impersonate-User". In order to be preserved consistently, `( extra name )` must be lower-case, and any characters which aren't [legal in HTTP header labels](https://tools.ietf.org/html/rfc7230#section-3.2.6) MUST be utf8 and [percent-encoded](https://tools.ietf.org/html/rfc3986#section-2.1). * `Impersonate-Uid`: A unique identifier that represents the user being impersonated. Optional. Requires "Impersonate-User". Kubernetes does not impose any format requirements on this string. {{< note >}} @@ -856,6 +856,14 @@ rules: resourceNames: ["06f6ce97-e2c5-4ab8-7ba5-7654dd08d52b"] ``` +{{< note >}} +Impersonating a user or group allows you to perform any action as if you were that user or group; +for that reason, impersonation is not namespace scoped. +If you want to allow impersonation using Kubernetes RBAC, +this requires using a `ClusterRole` and a `ClusterRoleBinding`, +not a `Role` and `RoleBinding`. +{{< /note >}} + ## client-go credential plugins {{< feature-state for_k8s_version="v1.22" state="stable" >}} @@ -905,7 +913,7 @@ users: # # The API version returned by the plugin MUST match the version listed here. # - # To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1alpha1), + # To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1beta1), # set an environment variable, pass an argument to the tool that indicates which version the exec plugin expects, # or read the version from the ExecCredential object in the KUBERNETES_EXEC_INFO environment variable. apiVersion: "client.authentication.k8s.io/v1" @@ -978,7 +986,7 @@ users: # # The API version returned by the plugin MUST match the version listed here. # - # To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1alpha1), + # To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1), # set an environment variable, pass an argument to the tool that indicates which version the exec plugin expects, # or read the version from the ExecCredential object in the KUBERNETES_EXEC_INFO environment variable. apiVersion: "client.authentication.k8s.io/v1beta1" @@ -1144,7 +1152,9 @@ If specified, `clientKeyData` and `clientCertificateData` must both must be pres {{< /tabs >}} Optionally, the response can include the expiry of the credential formatted as a -RFC3339 timestamp. Presence or absence of an expiry has the following impact: +[RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) timestamp. + +Presence or absence of an expiry has the following impact: - If an expiry is included, the bearer token and TLS credentials are cached until the expiry time is reached, or if the server responds with a 401 HTTP status code, @@ -1232,3 +1242,5 @@ The following `ExecCredential` manifest describes a cluster information sample. ## {{% heading "whatsnext" %}} * Read the [client authentication reference (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) +* Read the [client authentication reference (v1)](/docs/reference/config-api/client-authentication.v1/) + diff --git a/content/en/docs/reference/access-authn-authz/authorization.md b/content/en/docs/reference/access-authn-authz/authorization.md index af73a23350..ea6147fcba 100644 --- a/content/en/docs/reference/access-authn-authz/authorization.md +++ b/content/en/docs/reference/access-authn-authz/authorization.md @@ -74,9 +74,13 @@ PUT | update PATCH | patch DELETE | delete (for individual resources), deletecollection (for collections) +{{< caution >}} +The `get`, `list` and `watch` verbs can all return the full details of a resource. In terms of the returned data they are equivalent. For example, `list` on `secrets` will still reveal the `data` attributes of any returned resources. +{{< /caution >}} + Kubernetes sometimes checks authorization for additional permissions using specialized verbs. For example: -* [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) +* [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) * `use` verb on `podsecuritypolicies` resources in the `policy` API group. * [RBAC](/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping) * `bind` and `escalate` verbs on `roles` and `clusterroles` resources in the `rbac.authorization.k8s.io` API group. @@ -134,6 +138,21 @@ The output is similar to this: no ``` +Similarly, to check whether a ServiceAccount named `dev-sa` in Namespace `dev` +can list Pods in the Namespace `target`: + +```bash +kubectl auth can-i list pods \ + --namespace target \ + --as system:serviceaccount:dev:dev-sa +``` + +The output is similar to this: + +``` +yes +``` + `SelfSubjectAccessReview` is part of the `authorization.k8s.io` API group, which exposes the API server authorization to external services. Other resources in this group include: @@ -192,22 +211,33 @@ The following flags can be used: You can choose more than one authorization module. Modules are checked in order so an earlier module has higher priority to allow or deny a request. -## Privilege escalation via pod creation +## Privilege escalation via workload creation or edits {#privilege-escalation-via-pod-creation} -Users who have the ability to create pods in a namespace can potentially -escalate their privileges within that namespace. They can create pods that -access their privileges within that namespace. They can create pods that access -secrets the user cannot themselves read, or that run under a service account -with different/greater permissions. +Users who can create/edit pods in a namespace, either directly or through a [controller](/docs/concepts/architecture/controller/) +such as an operator, could escalate their privileges in that namespace. {{< caution >}} -System administrators, use care when granting access to pod creation. A user -granted permission to create pods (or controllers that create pods) in the -namespace can: read all secrets in the namespace; read all config maps in the -namespace; and impersonate any service account in the namespace and take any -action the account could take. This applies regardless of authorization mode. +System administrators, use care when granting access to create or edit workloads. +Details of how these can be misused are documented in [escalation paths](/docs/reference/access-authn-authz/authorization/#escalation-paths) {{< /caution >}} +### Escalation paths {#escalation-paths} +- Mounting arbitrary secrets in that namespace + - Can be used to access secrets meant for other workloads + - Can be used to obtain a more privileged service account's service account token +- Using arbitrary Service Accounts in that namespace + - Can perform Kubernetes API actions as another workload (impersonation) + - Can perform any privileged actions that Service Account has +- Mounting configmaps meant for other workloads in that namespace + - Can be used to obtain information meant for other workloads, such as DB host names. +- Mounting volumes meant for other workloads in that namespace + - Can be used to obtain information meant for other workloads, and change it. + +{{< caution >}} +System administrators should be cautious when deploying CRDs that +change the above areas. These may open privilege escalations paths. +This should be considered when deciding on your RBAC controls. +{{< /caution >}} ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md b/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md index 7e743be63d..74367d50c9 100644 --- a/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md +++ b/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md @@ -15,7 +15,7 @@ creating new clusters or joining new nodes to an existing cluster. It was built to support [kubeadm](/docs/reference/setup-tools/kubeadm/), but can be used in other contexts for users that wish to start clusters without `kubeadm`. It is also built to work, via RBAC policy, with the -[Kubelet TLS Bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) system. +[Kubelet TLS Bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) system. @@ -70,7 +70,7 @@ controller on the controller manager. Each valid token is backed by a secret in the `kube-system` namespace. You can find the full design doc -[here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md). +[here](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/bootstrap-discovery.md). Here is what the secret looks like. diff --git a/content/en/docs/reference/access-authn-authz/certificate-signing-requests.md b/content/en/docs/reference/access-authn-authz/certificate-signing-requests.md index 73b43f38bf..e9c033ef3a 100644 --- a/content/en/docs/reference/access-authn-authz/certificate-signing-requests.md +++ b/content/en/docs/reference/access-authn-authz/certificate-signing-requests.md @@ -199,7 +199,7 @@ To allow signing a CertificateSigningRequest: ## Normal user A few steps are required in order to get a normal user to be able to -authenticate and invoke an API. First, this user must have certificate issued +authenticate and invoke an API. First, this user must have a certificate issued by the Kubernetes cluster, and then present that certificate to the Kubernetes API. ### Create private key @@ -274,10 +274,10 @@ kubectl get csr myuser -o jsonpath='{.status.certificate}'| base64 -d > myuser.c ### Create Role and RoleBinding -With the certificate created. it is time to define the Role and RoleBinding for +With the certificate created it is time to define the Role and RoleBinding for this user to access Kubernetes cluster resources. -This is a sample script to create a Role for this new user: +This is a sample command to create a Role for this new user: ```shell kubectl create role developer --verb=create --verb=get --verb=list --verb=update --verb=delete --resource=pods diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index aa714bdaa2..05f9b8369c 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -16,8 +16,8 @@ In addition to [compiled-in admission plugins](/docs/reference/access-authn-auth admission plugins can be developed as extensions and run as webhooks configured at runtime. This page describes how to build, configure, use, and monitor admission webhooks. - + ## What are admission webhooks? Admission webhooks are HTTP callbacks that receive admission requests and do @@ -37,29 +37,25 @@ should use a validating admission webhook, since objects can be modified after b ## Experimenting with admission webhooks Admission webhooks are essentially part of the cluster control-plane. You should -write and deploy them with great caution. Please read the [user -guides](/docs/reference/access-authn-authz/extensible-admission-controllers/#write-an-admission-webhook-server) for -instructions if you intend to write/deploy production-grade admission webhooks. +write and deploy them with great caution. Please read the +[user guides](/docs/reference/access-authn-authz/extensible-admission-controllers/#write-an-admission-webhook-server) +for instructions if you intend to write/deploy production-grade admission webhooks. In the following, we describe how to quickly experiment with admission webhooks. ### Prerequisites -* Ensure that the Kubernetes cluster is at least as new as v1.16 (to use `admissionregistration.k8s.io/v1`), - or v1.9 (to use `admissionregistration.k8s.io/v1beta1`). - * Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers are enabled. [Here](/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use) is a recommended set of admission controllers to enable in general. -* Ensure that the `admissionregistration.k8s.io/v1` or `admissionregistration.k8s.io/v1beta1` API is enabled. +* Ensure that the `admissionregistration.k8s.io/v1` API is enabled. ### Write an admission webhook server -Please refer to the implementation of the [admission webhook -server](https://github.com/kubernetes/kubernetes/blob/release-1.21/test/images/agnhost/webhook/main.go) +Please refer to the implementation of the [admission webhook server](https://github.com/kubernetes/kubernetes/blob/release-1.21/test/images/agnhost/webhook/main.go) that is validated in a Kubernetes e2e test. The webhook handles the -`AdmissionReview` request sent by the apiservers, and sends back its decision +`AdmissionReview` request sent by the API servers, and sends back its decision as an `AdmissionReview` object in the same version it received. See the [webhook request](#request) section for details on the data sent to webhooks. @@ -67,11 +63,11 @@ See the [webhook request](#request) section for details on the data sent to webh See the [webhook response](#response) section for the data expected from webhooks. The example admission webhook server leaves the `ClientAuth` field -[empty](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48), +[empty](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/images/agnhost/webhook/config.go#L38-L39), which defaults to `NoClientCert`. This means that the webhook server does not -authenticate the identity of the clients, supposedly apiservers. If you need +authenticate the identity of the clients, supposedly API servers. If you need mutual TLS or other ways to authenticate the clients, see -how to [authenticate apiservers](#authenticate-apiservers). +how to [authenticate API servers](#authenticate-apiservers). ### Deploy the admission webhook service @@ -79,7 +75,7 @@ The webhook server in the e2e test is deployed in the Kubernetes cluster, via the [deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps). The test also creates a [service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core) as the front-end of the webhook server. See -[code](https://github.com/kubernetes/kubernetes/blob/v1.15.0/test/e2e/apimachinery/webhook.go#L301). +[code](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/e2e/apimachinery/webhook.go#L748). You may also deploy your webhooks outside of the cluster. You will need to update your webhook configurations accordingly. @@ -95,8 +91,6 @@ or The following is an example `ValidatingWebhookConfiguration`, a mutating webhook configuration is similar. See the [webhook configuration](#webhook-configuration) section for details about each config field. -{{< tabs name="ValidatingWebhookConfiguration_example_1" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration @@ -114,39 +108,18 @@ webhooks: service: namespace: "example-namespace" name: "example-service" - caBundle: "Ci0tLS0tQk...<`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.>...tLS0K" - admissionReviewVersions: ["v1", "v1beta1"] + caBundle: + admissionReviewVersions: ["v1"] sideEffects: None timeoutSeconds: 5 ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -metadata: - name: "pod-policy.example.com" -webhooks: -- name: "pod-policy.example.com" - rules: - - apiGroups: [""] - apiVersions: ["v1"] - operations: ["CREATE"] - resources: ["pods"] - scope: "Namespaced" - clientConfig: - service: - namespace: "example-namespace" - name: "example-service" - caBundle: "Ci0tLS0tQk...<`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate>...tLS0K" - admissionReviewVersions: ["v1beta1"] - timeoutSeconds: 5 -``` -{{% /tab %}} -{{< /tabs >}} -The scope field specifies if only cluster-scoped resources ("Cluster") or namespace-scoped +{{< note >}} +You must replace the `` in the above example by a valid CA bundle +which is a PEM-encoded CA bundle for validating the webhook's server certificate. +{{< /note >}} + +The `scope` field specifies if only cluster-scoped resources ("Cluster") or namespace-scoped resources ("Namespaced") will match this rule. "∗" means that there are no scope restrictions. {{< note >}} @@ -155,27 +128,26 @@ When using `clientConfig.service`, the server cert must be valid for {{< /note >}} {{< note >}} -Default timeout for a webhook call is 10 seconds for webhooks registered created using `admissionregistration.k8s.io/v1`, -and 30 seconds for webhooks created using `admissionregistration.k8s.io/v1beta1`. Starting in kubernetes 1.14 you -can set the timeout and it is encouraged to use a small timeout for webhooks. +Default timeout for a webhook call is 10 seconds, +You can set the `timeout` and it is encouraged to use a short timeout for webhooks. If the webhook call times out, the request is handled according to the webhook's failure policy. {{< /note >}} -When an apiserver receives a request that matches one of the `rules`, the -apiserver sends an `admissionReview` request to webhook as specified in the +When an API server receives a request that matches one of the `rules`, the +API server sends an `admissionReview` request to webhook as specified in the `clientConfig`. After you create the webhook configuration, the system will take a few seconds to honor the new configuration. -### Authenticate apiservers +### Authenticate API servers {#authenticate-apiservers} If your admission webhooks require authentication, you can configure the -apiservers to use basic auth, bearer token, or a cert to authenticate itself to +API servers to use basic auth, bearer token, or a cert to authenticate itself to the webhooks. There are three steps to complete the configuration. -* When starting the apiserver, specify the location of the admission control +* When starting the API server, specify the location of the admission control configuration file via the `--admission-control-config-file` flag. * In the admission control configuration file, specify where the @@ -228,55 +200,55 @@ For more information about `AdmissionConfiguration`, see the [AdmissionConfiguration (v1) reference](/docs/reference/config-api/apiserver-webhookadmission.v1/). See the [webhook configuration](#webhook-configuration) section for details about each config field. -* In the kubeConfig file, provide the credentials: +In the kubeConfig file, provide the credentials: - ```yaml - apiVersion: v1 - kind: Config - users: - # name should be set to the DNS name of the service or the host (including port) of the URL the webhook is configured to speak to. - # If a non-443 port is used for services, it must be included in the name when configuring 1.16+ API servers. - # - # For a webhook configured to speak to a service on the default port (443), specify the DNS name of the service: - # - name: webhook1.ns1.svc - # user: ... - # - # For a webhook configured to speak to a service on non-default port (e.g. 8443), specify the DNS name and port of the service in 1.16+: - # - name: webhook1.ns1.svc:8443 - # user: ... - # and optionally create a second stanza using only the DNS name of the service for compatibility with 1.15 API servers: - # - name: webhook1.ns1.svc - # user: ... - # - # For webhooks configured to speak to a URL, match the host (and port) specified in the webhook's URL. Examples: - # A webhook with `url: https://www.example.com`: - # - name: www.example.com - # user: ... - # - # A webhook with `url: https://www.example.com:443`: - # - name: www.example.com:443 - # user: ... - # - # A webhook with `url: https://www.example.com:8443`: - # - name: www.example.com:8443 - # user: ... - # - - name: 'webhook1.ns1.svc' - user: - client-certificate-data: "" - client-key-data: "" - # The `name` supports using * to wildcard-match prefixing segments. - - name: '*.webhook-company.org' - user: - password: "" - username: "" - # '*' is the default match. - - name: '*' - user: - token: "" - ``` +```yaml +apiVersion: v1 +kind: Config +users: +# name should be set to the DNS name of the service or the host (including port) of the URL the webhook is configured to speak to. +# If a non-443 port is used for services, it must be included in the name when configuring 1.16+ API servers. +# +# For a webhook configured to speak to a service on the default port (443), specify the DNS name of the service: +# - name: webhook1.ns1.svc +# user: ... +# +# For a webhook configured to speak to a service on non-default port (e.g. 8443), specify the DNS name and port of the service in 1.16+: +# - name: webhook1.ns1.svc:8443 +# user: ... +# and optionally create a second stanza using only the DNS name of the service for compatibility with 1.15 API servers: +# - name: webhook1.ns1.svc +# user: ... +# +# For webhooks configured to speak to a URL, match the host (and port) specified in the webhook's URL. Examples: +# A webhook with `url: https://www.example.com`: +# - name: www.example.com +# user: ... +# +# A webhook with `url: https://www.example.com:443`: +# - name: www.example.com:443 +# user: ... +# +# A webhook with `url: https://www.example.com:8443`: +# - name: www.example.com:8443 +# user: ... +# +- name: 'webhook1.ns1.svc' + user: + client-certificate-data: "" + client-key-data: "" +# The `name` supports using * to wildcard-match prefixing segments. +- name: '*.webhook-company.org' + user: + password: "" + username: "" +# '*' is the default match. +- name: '*' + user: + token: "" +``` -Of course you need to set up the webhook server to handle these authentications. +Of course you need to set up the webhook server to handle these authentication requests. ## Webhook request and response @@ -289,39 +261,17 @@ serialized to JSON as the body. Webhooks can specify what versions of `AdmissionReview` objects they accept with the `admissionReviewVersions` field in their configuration: -{{< tabs name="ValidatingWebhookConfiguration_admissionReviewVersions" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com admissionReviewVersions: ["v1", "v1beta1"] - ... ``` -`admissionReviewVersions` is a required field when creating -`admissionregistration.k8s.io/v1` webhook configurations. +`admissionReviewVersions` is a required field when creating webhook configurations. Webhooks are required to support at least one `AdmissionReview` version understood by the current and previous API server. -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - admissionReviewVersions: ["v1beta1"] - ... -``` - -If no `admissionReviewVersions` are specified, the default when creating -`admissionregistration.k8s.io/v1beta1` webhook configurations is `v1beta1`. -{{% /tab %}} -{{< /tabs >}} API servers send the first `AdmissionReview` version in the `admissionReviewVersions` list they support. If none of the versions in the list are supported by the API server, the configuration will not be allowed to be created. @@ -331,154 +281,100 @@ versions the API server knows how to send, attempts to call to the webhook will This example shows the data contained in an `AdmissionReview` object for a request to update the `scale` subresource of an `apps/v1` `Deployment`: - -{{< tabs name="AdmissionReview_request" >}} -{{% tab name="admission.k8s.io/v1" %}} ```yaml -{ - "apiVersion": "admission.k8s.io/v1", - "kind": "AdmissionReview", - "request": { - # Random uid uniquely identifying this admission call - "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", +apiVersion: admission.k8s.io/v1 +kind: AdmissionReview +request: + # Random uid uniquely identifying this admission call + uid: 705ab4f5-6393-11e8-b7cc-42010a800002 - # Fully-qualified group/version/kind of the incoming object - "kind": {"group":"autoscaling","version":"v1","kind":"Scale"}, - # Fully-qualified group/version/kind of the resource being modified - "resource": {"group":"apps","version":"v1","resource":"deployments"}, - # subresource, if the request is to a subresource - "subResource": "scale", + # Fully-qualified group/version/kind of the incoming object + kind: + group: autoscaling + version: v1 + kind: Scale - # Fully-qualified group/version/kind of the incoming object in the original request to the API server. - # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"}, - # Fully-qualified group/version/kind of the resource being modified in the original request to the API server. - # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - "requestResource": {"group":"apps","version":"v1","resource":"deployments"}, - # subresource, if the request is to a subresource - # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - "requestSubResource": "scale", + # Fully-qualified group/version/kind of the resource being modified + resource: + group: apps + version: v1 + resource: deployments - # Name of the resource being modified - "name": "my-deployment", - # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object) - "namespace": "my-namespace", + # subresource, if the request is to a subresource + subResource: scale - # operation can be CREATE, UPDATE, DELETE, or CONNECT - "operation": "UPDATE", + # Fully-qualified group/version/kind of the incoming object in the original request to the API server. + # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the + # original request to the API server was converted to a version the webhook registered for. + requestKind: + group: autoscaling + version: v1 + kind: Scale - "userInfo": { - # Username of the authenticated user making the request to the API server - "username": "admin", - # UID of the authenticated user making the request to the API server - "uid": "014fbff9a07c", - # Group memberships of the authenticated user making the request to the API server - "groups": ["system:authenticated","my-admin-group"], - # Arbitrary extra info associated with the user making the request to the API server. - # This is populated by the API server authentication layer and should be included - # if any SubjectAccessReview checks are performed by the webhook. - "extra": { - "some-key":["some-value1", "some-value2"] - } - }, + # Fully-qualified group/version/kind of the resource being modified in the original request to the API server. + # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the + # original request to the API server was converted to a version the webhook registered for. + requestResource: + group: apps + version: v1 + resource: deployments - # object is the new object being admitted. - # It is null for DELETE operations. - "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, - # oldObject is the existing object. - # It is null for CREATE and CONNECT operations. - "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, - # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions. - # It is null for CONNECT operations. - "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...}, + # subresource, if the request is to a subresource + # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the + # original request to the API server was converted to a version the webhook registered for. + requestSubResource: scale - # dryRun indicates the API request is running in dry run mode and will not be persisted. - # Webhooks with side effects should avoid actuating those side effects when dryRun is true. - # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details. - "dryRun": false - } -} + # Name of the resource being modified + name: my-deployment + + # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object) + namespace: my-namespace + + # operation can be CREATE, UPDATE, DELETE, or CONNECT + operation: UPDATE + + userInfo: + # Username of the authenticated user making the request to the API server + username: admin + + # UID of the authenticated user making the request to the API server + uid: 014fbff9a07c + + # Group memberships of the authenticated user making the request to the API server + groups: + - system:authenticated + - my-admin-group + # Arbitrary extra info associated with the user making the request to the API server. + # This is populated by the API server authentication layer and should be included + # if any SubjectAccessReview checks are performed by the webhook. + extra: + some-key: + - some-value1 + - some-value2 + + # object is the new object being admitted. + # It is null for DELETE operations. + object: + apiVersion: autoscaling/v1 + kind: Scale + + # oldObject is the existing object. + # It is null for CREATE and CONNECT operations. + oldObject: + apiVersion: autoscaling/v1 + kind: Scale + + # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions. + # It is null for CONNECT operations. + options: + apiVersion: meta.k8s.io/v1 + kind: UpdateOptions + + # dryRun indicates the API request is running in dry run mode and will not be persisted. + # Webhooks with side effects should avoid actuating those side effects when dryRun is true. + # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details. + dryRun: False ``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```yaml -{ - # Deprecated in v1.16 in favor of admission.k8s.io/v1 - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "request": { - # Random uid uniquely identifying this admission call - "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", - - # Fully-qualified group/version/kind of the incoming object - "kind": {"group":"autoscaling","version":"v1","kind":"Scale"}, - # Fully-qualified group/version/kind of the resource being modified - "resource": {"group":"apps","version":"v1","resource":"deployments"}, - # subresource, if the request is to a subresource - "subResource": "scale", - - # Fully-qualified group/version/kind of the incoming object in the original request to the API server. - # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - # Only sent by v1.15+ API servers. - "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"}, - # Fully-qualified group/version/kind of the resource being modified in the original request to the API server. - # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - # Only sent by v1.15+ API servers. - "requestResource": {"group":"apps","version":"v1","resource":"deployments"}, - # subresource, if the request is to a subresource - # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the - # original request to the API server was converted to a version the webhook registered for. - # Only sent by v1.15+ API servers. - "requestSubResource": "scale", - - # Name of the resource being modified - "name": "my-deployment", - # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object) - "namespace": "my-namespace", - - # operation can be CREATE, UPDATE, DELETE, or CONNECT - "operation": "UPDATE", - - "userInfo": { - # Username of the authenticated user making the request to the API server - "username": "admin", - # UID of the authenticated user making the request to the API server - "uid": "014fbff9a07c", - # Group memberships of the authenticated user making the request to the API server - "groups": ["system:authenticated","my-admin-group"], - # Arbitrary extra info associated with the user making the request to the API server. - # This is populated by the API server authentication layer and should be included - # if any SubjectAccessReview checks are performed by the webhook. - "extra": { - "some-key":["some-value1", "some-value2"] - } - }, - - # object is the new object being admitted. - # It is null for DELETE operations. - "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, - # oldObject is the existing object. - # It is null for CREATE and CONNECT operations (and for DELETE operations in API servers prior to v1.15.0) - "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, - # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions. - # It is null for CONNECT operations. - # Only sent by v1.15+ API servers. - "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...}, - - # dryRun indicates the API request is running in dry run mode and will not be persisted. - # Webhooks with side effects should avoid actuating those side effects when dryRun is true. - # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details. - "dryRun": false - } -} -``` -{{% /tab %}} -{{< /tabs >}} ### Response @@ -492,8 +388,7 @@ At a minimum, the `response` stanza must contain the following fields: * `allowed`, either set to `true` or `false` Example of a minimal response from a webhook to allow a request: -{{< tabs name="AdmissionReview_response_allow" >}} -{{% tab name="admission.k8s.io/v1" %}} + ```json { "apiVersion": "admission.k8s.io/v1", @@ -504,55 +399,26 @@ Example of a minimal response from a webhook to allow a request: } } ``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": true - } -} -``` -{{% /tab %}} -{{< /tabs >}} Example of a minimal response from a webhook to forbid a request: -{{< tabs name="AdmissionReview_response_forbid_minimal" >}} -{{% tab name="admission.k8s.io/v1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": false - } -} -``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": false - } -} -``` -{{% /tab %}} -{{< /tabs >}} -When rejecting a request, the webhook can customize the http code and message returned to the user using the `status` field. -The specified status object is returned to the user. -See the [API documentation](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#status-v1-meta) for details about the status type. +```json +{ + "apiVersion": "admission.k8s.io/v1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": false + } +} +``` + +When rejecting a request, the webhook can customize the http code and message returned to the user +using the `status` field. The specified status object is returned to the user. +See the [API documentation](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#status-v1-meta) +for details about the `status` type. Example of a response to forbid a request, customizing the HTTP status code and message presented to the user: -{{< tabs name="AdmissionReview_response_forbid_details" >}} -{{% tab name="admission.k8s.io/v1" %}} + ```json { "apiVersion": "admission.k8s.io/v1", @@ -567,24 +433,6 @@ Example of a response to forbid a request, customizing the HTTP status code and } } ``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": false, - "status": { - "code": 403, - "message": "You cannot do this because it is Tuesday and your name starts with A" - } - } -} -``` -{{% /tab %}} -{{< /tabs >}} When allowing a request, a mutating admission webhook may optionally modify the incoming object as well. This is done using the `patch` and `patchType` fields in the response. @@ -592,13 +440,13 @@ The only currently supported `patchType` is `JSONPatch`. See [JSON patch](https://jsonpatch.com/) documentation for more details. For `patchType: JSONPatch`, the `patch` field contains a base64-encoded array of JSON patch operations. -As an example, a single patch operation that would set `spec.replicas` would be `[{"op": "add", "path": "/spec/replicas", "value": 3}]` +As an example, a single patch operation that would set `spec.replicas` would be +`[{"op": "add", "path": "/spec/replicas", "value": 3}]` Base64-encoded, this would be `W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=` So a webhook response to add that label would be: -{{< tabs name="AdmissionReview_response_modify" >}} -{{% tab name="admission.k8s.io/v1" %}} + ```json { "apiVersion": "admission.k8s.io/v1", @@ -611,27 +459,12 @@ So a webhook response to add that label would be: } } ``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": true, - "patchType": "JSONPatch", - "patch": "W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=" - } -} -``` -{{% /tab %}} -{{< /tabs >}} -Starting in v1.19, admission webhooks can optionally return warning messages that are returned to the requesting client +Admission webhooks can optionally return warning messages that are returned to the requesting client in HTTP `Warning` headers with a warning code of 299. Warnings can be sent with allowed or rejected admission responses. If you're implementing a webhook that returns a warning: + * Don't include a "Warning:" prefix in the message * Use warning messages to describe problems the client making the API request should correct or be aware of * Limit warnings to 120 characters if possible @@ -641,8 +474,6 @@ Individual warning messages over 256 characters may be truncated by the API serv If more than 4096 characters of warning messages are added (from all sources), additional warning messages are ignored. {{< /caution >}} -{{< tabs name="AdmissionReview_response_warning" >}} -{{% tab name="admission.k8s.io/v1" %}} ```json { "apiVersion": "admission.k8s.io/v1", @@ -657,24 +488,6 @@ If more than 4096 characters of warning messages are added (from all sources), a } } ``` -{{% /tab %}} -{{% tab name="admission.k8s.io/v1beta1" %}} -```json -{ - "apiVersion": "admission.k8s.io/v1beta1", - "kind": "AdmissionReview", - "response": { - "uid": "", - "allowed": true, - "warnings": [ - "duplicate envvar entries specified with name MY_ENV", - "memory request less than 4MB specified for container mycontainer, which will not start successfully" - ] - } -} -``` -{{% /tab %}} -{{< /tabs >}} ## Webhook configuration @@ -683,9 +496,9 @@ The name of a `MutatingWebhookConfiguration` or a `ValidatingWebhookConfiguratio [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). Each configuration can contain one or more webhooks. -If multiple webhooks are specified in a single configuration, each should be given a unique name. -This is required in `admissionregistration.k8s.io/v1`, but strongly recommended when using `admissionregistration.k8s.io/v1beta1`, -in order to make resulting audit logs and metrics easier to match up to active configurations. +If multiple webhooks are specified in a single configuration, each must be given a unique name. +This is required in order to make resulting audit logs and metrics easier to match up to active +configurations. Each webhook defines the following things. @@ -694,27 +507,31 @@ Each webhook defines the following things. Each webhook must specify a list of rules used to determine if a request to the API server should be sent to the webhook. Each rule specifies one or more operations, apiGroups, apiVersions, and resources, and a resource scope: -* `operations` lists one or more operations to match. Can be `"CREATE"`, `"UPDATE"`, `"DELETE"`, `"CONNECT"`, or `"*"` to match all. +* `operations` lists one or more operations to match. Can be `"CREATE"`, `"UPDATE"`, `"DELETE"`, `"CONNECT"`, + or `"*"` to match all. * `apiGroups` lists one or more API groups to match. `""` is the core API group. `"*"` matches all API groups. * `apiVersions` lists one or more API versions to match. `"*"` matches all API versions. * `resources` lists one or more resources to match. - * `"*"` matches all resources, but not subresources. - * `"*/*"` matches all resources and subresources. - * `"pods/*"` matches all subresources of pods. - * `"*/status"` matches all status subresources. -* `scope` specifies a scope to match. Valid values are `"Cluster"`, `"Namespaced"`, and `"*"`. Subresources match the scope of their parent resource. Supported in v1.14+. Default is `"*"`, matching pre-1.14 behavior. - * `"Cluster"` means that only cluster-scoped resources will match this rule (Namespace API objects are cluster-scoped). - * `"Namespaced"` means that only namespaced resources will match this rule. - * `"*"` means that there are no scope restrictions. -If an incoming request matches one of the specified operations, groups, versions, resources, and scope for any of a webhook's rules, the request is sent to the webhook. + * `"*"` matches all resources, but not subresources. + * `"*/*"` matches all resources and subresources. + * `"pods/*"` matches all subresources of pods. + * `"*/status"` matches all status subresources. + +* `scope` specifies a scope to match. Valid values are `"Cluster"`, `"Namespaced"`, and `"*"`. + Subresources match the scope of their parent resource. Default is `"*"`. + + * `"Cluster"` means that only cluster-scoped resources will match this rule (Namespace API objects are cluster-scoped). + * `"Namespaced"` means that only namespaced resources will match this rule. + * `"*"` means that there are no scope restrictions. + +If an incoming request matches one of the specified `operations`, `groups`, `versions`, +`resources`, and `scope` for any of a webhook's `rules`, the request is sent to the webhook. Here are other examples of rules that could be used to specify which resources should be intercepted. Match `CREATE` or `UPDATE` requests to `apps/v1` and `apps/v1beta1` `deployments` and `replicasets`: -{{< tabs name="ValidatingWebhookConfiguration_rules_1" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration @@ -729,123 +546,56 @@ webhooks: scope: "Namespaced" ... ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - rules: - - operations: ["CREATE", "UPDATE"] - apiGroups: ["apps"] - apiVersions: ["v1", "v1beta1"] - resources: ["deployments", "replicasets"] - scope: "Namespaced" - ... -``` -{{% /tab %}} -{{< /tabs >}} Match create requests for all resources (but not subresources) in all API groups and versions: -{{< tabs name="ValidatingWebhookConfiguration_rules_2" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "*" - ... + - name: my-webhook.example.com + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "*" ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "*" - ... -``` -{{% /tab %}} -{{< /tabs >}} Match update requests for all `status` subresources in all API groups and versions: -{{< tabs name="ValidatingWebhookConfiguration_rules_3" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - rules: - - operations: ["UPDATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*/status"] - scope: "*" - ... + - name: my-webhook.example.com + rules: + - operations: ["UPDATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*/status"] + scope: "*" ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - rules: - - operations: ["UPDATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*/status"] - scope: "*" - ... -``` -{{% /tab %}} -{{< /tabs >}} ### Matching requests: objectSelector -In v1.15+, webhooks may optionally limit which requests are intercepted based on the labels of the +Webhooks may optionally limit which requests are intercepted based on the labels of the objects they would be sent, by specifying an `objectSelector`. If specified, the objectSelector is evaluated against both the object and oldObject that would be sent to the webhook, and is considered to match if either object matches the selector. -A null object (oldObject in the case of create, or newObject in the case of delete), +A null object (`oldObject` in the case of create, or `newObject` in the case of delete), or an object that cannot have labels (like a `DeploymentRollback` or a `PodProxyOptions` object) is not considered to match. -Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. +Use the object selector only if the webhook is opt-in, because end users may skip +the admission webhook by setting the labels. This example shows a mutating webhook that would match a `CREATE` of any resource with the label `foo: bar`: -{{< tabs name="objectSelector_example" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com objectSelector: @@ -857,32 +607,10 @@ webhooks: apiVersions: ["*"] resources: ["*"] scope: "*" - ... ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - objectSelector: - matchLabels: - foo: bar - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "*" - ... -``` -{{% /tab %}} -{{< /tabs >}} -See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors. +See [labels concept](/docs/concepts/overview/working-with-objects/labels) +for more examples of label selectors. ### Matching requests: namespaceSelector @@ -897,128 +625,75 @@ If the object is a cluster scoped resource other than a Namespace, `namespaceSel This example shows a mutating webhook that matches a `CREATE` of any namespaced resource inside a namespace that does not have a "runlevel" label of "0" or "1": -{{< tabs name="MutatingWebhookConfiguration_namespaceSelector_1" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - namespaceSelector: - matchExpressions: - - key: runlevel - operator: NotIn - values: ["0","1"] - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "Namespaced" - ... + - name: my-webhook.example.com + namespaceSelector: + matchExpressions: + - key: runlevel + operator: NotIn + values: ["0","1"] + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "Namespaced" ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - namespaceSelector: - matchExpressions: - - key: runlevel - operator: NotIn - values: ["0","1"] - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "Namespaced" - ... -``` -{{% /tab %}} -{{< /tabs >}} -This example shows a validating webhook that matches a `CREATE` of any namespaced resource inside a namespace -that is associated with the "environment" of "prod" or "staging": +This example shows a validating webhook that matches a `CREATE` of any namespaced resource inside +a namespace that is associated with the "environment" of "prod" or "staging": -{{< tabs name="ValidatingWebhookConfiguration_namespaceSelector_2" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - namespaceSelector: - matchExpressions: - - key: environment - operator: In - values: ["prod","staging"] - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "Namespaced" - ... + - name: my-webhook.example.com + namespaceSelector: + matchExpressions: + - key: environment + operator: In + values: ["prod","staging"] + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "Namespaced" ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - namespaceSelector: - matchExpressions: - - key: environment - operator: In - values: ["prod","staging"] - rules: - - operations: ["CREATE"] - apiGroups: ["*"] - apiVersions: ["*"] - resources: ["*"] - scope: "Namespaced" - ... -``` -{{% /tab %}} -{{< /tabs >}} -See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors. +See [labels concept](/docs/concepts/overview/working-with-objects/labels) +for more examples of label selectors. ### Matching requests: matchPolicy API servers can make objects available via multiple API groups or versions. -For example, the Kubernetes API server may allow creating and modifying `Deployment` objects -via `extensions/v1beta1`, `apps/v1beta1`, `apps/v1beta2`, and `apps/v1` APIs. -For example, if a webhook only specified a rule for some API groups/versions (like `apiGroups:["apps"], apiVersions:["v1","v1beta1"]`), +For example, if a webhook only specified a rule for some API groups/versions +(like `apiGroups:["apps"], apiVersions:["v1","v1beta1"]`), and a request was made to modify the resource via another API group/version (like `extensions/v1beta1`), the request would not be sent to the webhook. -In v1.15+, `matchPolicy` lets a webhook define how its `rules` are used to match incoming requests. +The `matchPolicy` lets a webhook define how its `rules` are used to match incoming requests. Allowed values are `Exact` or `Equivalent`. * `Exact` means a request should be intercepted only if it exactly matches a specified rule. -* `Equivalent` means a request should be intercepted if modifies a resource listed in `rules`, even via another API group or version. +* `Equivalent` means a request should be intercepted if modifies a resource listed in `rules`, + even via another API group or version. In the example given above, the webhook that only registered for `apps/v1` could use `matchPolicy`: * `matchPolicy: Exact` would mean the `extensions/v1beta1` request would not be sent to the webhook -* `matchPolicy: Equivalent` means the `extensions/v1beta1` request would be sent to the webhook (with the objects converted to a version the webhook had specified: `apps/v1`) +* `matchPolicy: Equivalent` means the `extensions/v1beta1` request would be sent to the webhook + (with the objects converted to a version the webhook had specified: `apps/v1`) Specifying `Equivalent` is recommended, and ensures that webhooks continue to intercept the resources they expect when upgrades enable new versions of the resource in the API server. -When a resource stops being served by the API server, it is no longer considered equivalent to other versions of that resource that are still served. +When a resource stops being served by the API server, it is no longer considered equivalent to +other versions of that resource that are still served. For example, `extensions/v1beta1` deployments were first deprecated and then removed (in Kubernetes v1.16). Since that removal, a webhook with a `apiGroups:["extensions"], apiVersions:["v1beta1"], resources:["deployments"]` rule @@ -1028,12 +703,9 @@ for stable versions of resources. This example shows a validating webhook that intercepts modifications to deployments (no matter the API group or version), and is always sent an `apps/v1` `Deployment` object: -{{< tabs name="ValidatingWebhookConfiguration_matchPolicy" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com matchPolicy: Equivalent @@ -1043,32 +715,9 @@ webhooks: apiVersions: ["v1"] resources: ["deployments"] scope: "Namespaced" - ... ``` -Admission webhooks created using `admissionregistration.k8s.io/v1` default to `Equivalent`. -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - matchPolicy: Equivalent - rules: - - operations: ["CREATE","UPDATE","DELETE"] - apiGroups: ["apps"] - apiVersions: ["v1"] - resources: ["deployments"] - scope: "Namespaced" - ... -``` - -Admission webhooks created using `admissionregistration.k8s.io/v1beta1` default to `Exact`. -{{% /tab %}} -{{< /tabs >}} +The `matchPolicy` for an admission webhooks defaults to `Equivalent`. ### Contacting the webhook @@ -1086,51 +735,32 @@ and can optionally include a custom CA bundle to use to verify the TLS connectio The `host` should not refer to a service running in the cluster; use a service reference by specifying the `service` field instead. -The host might be resolved via external DNS in some apiservers +The host might be resolved via external DNS in some API servers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address. Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts -which run an apiserver which might need to make calls to this +which run an API server which might need to make calls to this webhook. Such installations are likely to be non-portable or not readily run in a new cluster. The scheme must be "https"; the URL must begin with "https://". -Attempting to use a user or basic auth (for example "user:password@") is not allowed. -Fragments ("#...") and query parameters ("?...") are also not allowed. +Attempting to use a user or basic auth (for example `user:password@`) is not allowed. +Fragments (`#...`) and query parameters (`?...`) are also not allowed. Here is an example of a mutating webhook configured to call a URL (and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle): -{{< tabs name="MutatingWebhookConfiguration_url" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com clientConfig: url: "https://my-webhook.example.com:9443/my-webhook-path" - ... ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - clientConfig: - url: "https://my-webhook.example.com:9443/my-webhook-path" - ... -``` -{{% /tab %}} -{{< /tabs >}} #### Service reference @@ -1143,43 +773,24 @@ Here is an example of a mutating webhook configured to call a service on port "1 at the subpath "/my-path", and to verify the TLS connection against the ServerName `my-service-name.my-service-namespace.svc` using a custom CA bundle: -{{< tabs name="MutatingWebhookConfiguration_service" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com clientConfig: - caBundle: "Ci0tLS0tQk......tLS0K" + caBundle: service: namespace: my-service-namespace name: my-service-name path: /my-path port: 1234 - ... ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - clientConfig: - caBundle: "Ci0tLS0tQk...<`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate>...tLS0K" - service: - namespace: my-service-namespace - name: my-service-name - path: /my-path - port: 1234 - ... -``` -{{% /tab %}} -{{< /tabs >}} + +{{< note >}} +You must replace the `` in the above example by a valid CA bundle +which is a PEM-encoded CA bundle for validating the webhook's server certificate. +{{< /note >}} ### Side effects @@ -1199,46 +810,20 @@ or the dry-run request will not be sent to the webhook and the API request will Webhooks indicate whether they have side effects using the `sideEffects` field in the webhook configuration: -* `Unknown`: no information is known about the side effects of calling the webhook. -If a request with `dryRun: true` would trigger a call to this webhook, the request will instead fail, and the webhook will not be called. * `None`: calling the webhook will have no side effects. -* `Some`: calling the webhook will possibly have side effects. -If a request with the dry-run attribute would trigger a call to this webhook, the request will instead fail, and the webhook will not be called. -* `NoneOnDryRun`: calling the webhook will possibly have side effects, -but if a request with `dryRun: true` is sent to the webhook, the webhook will suppress the side effects (the webhook is `dryRun`-aware). - -Allowed values: - -* In `admissionregistration.k8s.io/v1beta1`, `sideEffects` may be set to `Unknown`, `None`, `Some`, or `NoneOnDryRun`, and defaults to `Unknown`. -* In `admissionregistration.k8s.io/v1`, `sideEffects` must be set to `None` or `NoneOnDryRun`. +* `NoneOnDryRun`: calling the webhook will possibly have side effects, but if a request with + `dryRun: true` is sent to the webhook, the webhook will suppress the side effects (the webhook + is `dryRun`-aware). Here is an example of a validating webhook indicating it has no side effects on `dryRun: true` requests: -{{< tabs name="ValidatingWebhookConfiguration_sideEffects" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - sideEffects: NoneOnDryRun - ... + - name: my-webhook.example.com + sideEffects: NoneOnDryRun ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - sideEffects: NoneOnDryRun - ... -``` -{{% /tab %}} -{{< /tabs >}} ### Timeouts @@ -1253,35 +838,15 @@ The timeout value must be between 1 and 30 seconds. Here is an example of a validating webhook with a custom timeout of 2 seconds: -{{< tabs name="ValidatingWebhookConfiguration_timeoutSeconds" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration -... webhooks: -- name: my-webhook.example.com - timeoutSeconds: 2 - ... + - name: my-webhook.example.com + timeoutSeconds: 2 ``` -Admission webhooks created using `admissionregistration.k8s.io/v1` default timeouts to 10 seconds. -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - timeoutSeconds: 2 - ... -``` - -Admission webhooks created using `admissionregistration.k8s.io/v1` default timeouts to 30 seconds. -{{% /tab %}} -{{< /tabs >}} +The timeout for an admission webhook defaults to 10 seconds. ### Reinvocation policy @@ -1290,50 +855,35 @@ A single ordering of mutating admissions plugins (including webhooks) does not w to the object (like adding a `container` to a `pod`), and other mutating plugins which have already run may have opinions on those new structures (like setting an `imagePullPolicy` on all containers). -In v1.15+, to allow mutating admission plugins to observe changes made by other plugins, +To allow mutating admission plugins to observe changes made by other plugins, built-in mutating admission plugins are re-run if a mutating webhook modifies an object, and mutating webhooks can specify a `reinvocationPolicy` to control whether they are reinvoked as well. `reinvocationPolicy` may be set to `Never` or `IfNeeded`. It defaults to `Never`. -* `Never`: the webhook must not be called more than once in a single admission evaluation +* `Never`: the webhook must not be called more than once in a single admission evaluation. * `IfNeeded`: the webhook may be called again as part of the admission evaluation if the object -being admitted is modified by other admission plugins after the initial webhook call. + being admitted is modified by other admission plugins after the initial webhook call. The important elements to note are: * The number of additional invocations is not guaranteed to be exactly one. -* If additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. +* If additional invocations result in further modifications to the object, webhooks are not + guaranteed to be invoked again. * Webhooks that use this option may be reordered to minimize the number of additional invocations. -* To validate an object after all mutations are guaranteed complete, use a validating admission webhook instead (recommended for webhooks with side-effects). +* To validate an object after all mutations are guaranteed complete, use a validating admission + webhook instead (recommended for webhooks with side-effects). -Here is an example of a mutating webhook opting into being re-invoked if later admission plugins modify the object: +Here is an example of a mutating webhook opting into being re-invoked if later admission plugins +modify the object: -{{< tabs name="MutatingWebhookConfiguration_reinvocationPolicy" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com reinvocationPolicy: IfNeeded - ... ``` -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - reinvocationPolicy: IfNeeded - ... -``` -{{% /tab %}} -{{< /tabs >}} Mutating webhooks must be [idempotent](#idempotence), able to successfully process an object they have already admitted and potentially modified. This is true for all mutating admission webhooks, since any change they can make @@ -1349,35 +899,15 @@ are handled. Allowed values are `Ignore` or `Fail`. Here is a mutating webhook configured to reject an API request if errors are encountered calling the admission webhook: -{{< tabs name="MutatingWebhookConfiguration_failurePolicy" >}} -{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration -... webhooks: - name: my-webhook.example.com failurePolicy: Fail - ... ``` -Admission webhooks created using `admissionregistration.k8s.io/v1` default `failurePolicy` to `Fail`. -{{% /tab %}} -{{% tab name="admissionregistration.k8s.io/v1beta1" %}} -```yaml -# Deprecated in v1.16 in favor of admissionregistration.k8s.io/v1 -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -... -webhooks: -- name: my-webhook.example.com - failurePolicy: Fail - ... -``` - -Admission webhooks created using `admissionregistration.k8s.io/v1beta1` default `failurePolicy` to `Ignore`. -{{% /tab %}} -{{< /tabs >}} +The default `failurePolicy` for an admission webhooks is `Fail`. ## Monitoring admission webhooks @@ -1388,121 +918,123 @@ monitoring mechanisms help cluster admins to answer questions like: 2. What change did the mutating webhook applied to the object? -3. Which webhooks are frequently rejecting API requests? What's the reason for a - rejection? +3. Which webhooks are frequently rejecting API requests? What's the reason for a rejection? ### Mutating webhook auditing annotations Sometimes it's useful to know which mutating webhook mutated the object in a API request, and what change did the webhook apply. -In v1.16+, kube-apiserver performs [auditing](/docs/tasks/debug-application-cluster/audit/) on each mutating webhook -invocation. Each invocation generates an auditing annotation -capturing if a request object is mutated by the invocation, and optionally generates an annotation capturing the applied -patch from the webhook admission response. The annotations are set in the audit event for given request on given stage of -its execution, which is then pre-processed according to a certain policy and written to a backend. +The Kubernetes API server performs [auditing](/docs/tasks/debug/debug-cluster/audit/) on each +mutating webhook invocation. Each invocation generates an auditing annotation +capturing if a request object is mutated by the invocation, and optionally generates an annotation +capturing the applied patch from the webhook admission response. The annotations are set in the +audit event for given request on given stage of its execution, which is then pre-processed +according to a certain policy and written to a backend. The audit level of a event determines which annotations get recorded: - At `Metadata` audit level or higher, an annotation with key -`mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}` gets logged with JSON payload indicating -a webhook gets invoked for given request and whether it mutated the object or not. + `mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}` gets logged with JSON + payload indicating a webhook gets invoked for given request and whether it mutated the object or not. -For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the third in the -mutating webhook chain, and didn't mutated the request object during the invocation. + For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is + ordered the third in the mutating webhook chain, and didn't mutated the request object during the + invocation. -```yaml -# the audit event recorded -{ - "kind": "Event", - "apiVersion": "audit.k8s.io/v1", - "annotations": { - "mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook.example.com\",\"mutated\": false}" - # other annotations - ... - } - # other fields - ... -} -``` + ```yaml + # the audit event recorded + { + "kind": "Event", + "apiVersion": "audit.k8s.io/v1", + "annotations": { + "mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook.example.com\",\"mutated\": false}" + # other annotations + ... + } + # other fields + ... + } + ``` + + ```yaml + # the annotation value deserialized + { + "configuration": "my-mutating-webhook-configuration.example.com", + "webhook": "my-webhook.example.com", + "mutated": false + } + ``` + + The following annotation gets recorded for a webhook being invoked in the first round. The webhook + is ordered the first in the mutating webhook chain, and mutated the request object during the + invocation. -```yaml -# the annotation value deserialized -{ - "configuration": "my-mutating-webhook-configuration.example.com", - "webhook": "my-webhook.example.com", - "mutated": false -} -``` - -The following annotation gets recorded for a webhook being invoked in the first round. The webhook is ordered the first in\ -the mutating webhook chain, and mutated the request object during the invocation. - -```yaml -# the audit event recorded -{ - "kind": "Event", - "apiVersion": "audit.k8s.io/v1", - "annotations": { - "mutation.webhook.admission.k8s.io/round_0_index_0": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"mutated\": true}" - # other annotations - ... - } - # other fields - ... -} -``` - -```yaml -# the annotation value deserialized -{ - "configuration": "my-mutating-webhook-configuration.example.com", - "webhook": "my-webhook-always-mutate.example.com", - "mutated": true -} -``` + ```yaml + # the audit event recorded + { + "kind": "Event", + "apiVersion": "audit.k8s.io/v1", + "annotations": { + "mutation.webhook.admission.k8s.io/round_0_index_0": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"mutated\": true}" + # other annotations + ... + } + # other fields + ... + } + ``` + + ```yaml + # the annotation value deserialized + { + "configuration": "my-mutating-webhook-configuration.example.com", + "webhook": "my-webhook-always-mutate.example.com", + "mutated": true + } + ``` - At `Request` audit level or higher, an annotation with key -`patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}` gets logged with JSON payload indicating -a webhook gets invoked for given request and what patch gets applied to the request object. + `patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}` gets logged with JSON payload indicating + a webhook gets invoked for given request and what patch gets applied to the request object. -For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the fourth in the -mutating webhook chain, and responded with a JSON patch which got applied to the request object. - -```yaml -# the audit event recorded -{ - "kind": "Event", - "apiVersion": "audit.k8s.io/v1", - "annotations": { - "patch.webhook.admission.k8s.io/round_1_index_3": "{\"configuration\":\"my-other-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"patch\":[{\"op\":\"add\",\"path\":\"/data/mutation-stage\",\"value\":\"yes\"}],\"patchType\":\"JSONPatch\"}" - # other annotations - ... - } - # other fields - ... -} -``` - -```yaml -# the annotation value deserialized -{ - "configuration": "my-other-mutating-webhook-configuration.example.com", - "webhook": "my-webhook-always-mutate.example.com", - "patchType": "JSONPatch", - "patch": [ - { - "op": "add", - "path": "/data/mutation-stage", - "value": "yes" - } - ] -} -``` + For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the fourth in the + mutating webhook chain, and responded with a JSON patch which got applied to the request object. + + ```yaml + # the audit event recorded + { + "kind": "Event", + "apiVersion": "audit.k8s.io/v1", + "annotations": { + "patch.webhook.admission.k8s.io/round_1_index_3": "{\"configuration\":\"my-other-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"patch\":[{\"op\":\"add\",\"path\":\"/data/mutation-stage\",\"value\":\"yes\"}],\"patchType\":\"JSONPatch\"}" + # other annotations + ... + } + # other fields + ... + } + ``` + + ```yaml + # the annotation value deserialized + { + "configuration": "my-other-mutating-webhook-configuration.example.com", + "webhook": "my-webhook-always-mutate.example.com", + "patchType": "JSONPatch", + "patch": [ + { + "op": "add", + "path": "/data/mutation-stage", + "value": "yes" + } + ] + } + ``` ### Admission webhook metrics -Kube-apiserver exposes Prometheus metrics from the `/metrics` endpoint, which can be used for monitoring and +The API server exposes Prometheus metrics from the `/metrics` endpoint, which can be used for monitoring and diagnosing API server status. The following metrics record status related to admission webhooks. #### API server admission webhook rejection count @@ -1510,7 +1042,7 @@ diagnosing API server status. The following metrics record status related to adm Sometimes it's useful to know which admission webhooks are frequently rejecting API requests, and the reason for a rejection. -In v1.16+, kube-apiserver exposes a Prometheus counter metric recording admission webhook rejections. The +The API server exposes a Prometheus counter metric recording admission webhook rejections. The metrics are labelled to identify the causes of webhook rejection(s): - `name`: the name of the webhook that rejected a request. @@ -1519,11 +1051,13 @@ metrics are labelled to identify the causes of webhook rejection(s): - `type`: the admission webhook type, can be one of `admit` and `validating`. - `error_type`: identifies if an error occurred during the webhook invocation that caused the rejection. Its value can be one of: - - `calling_webhook_error`: unrecognized errors or timeout errors from the admission webhook happened and the - webhook's [Failure policy](#failure-policy) is set to `Fail`. - - `no_error`: no error occurred. The webhook rejected the request with `allowed: false` in the admission - response. The metrics label `rejection_code` records the `.status.code` set in the admission response. - - `apiserver_internal_error`: an API server internal error happened. + + - `calling_webhook_error`: unrecognized errors or timeout errors from the admission webhook happened and the + webhook's [Failure policy](#failure-policy) is set to `Fail`. + - `no_error`: no error occurred. The webhook rejected the request with `allowed: false` in the admission + response. The metrics label `rejection_code` records the `.status.code` set in the admission response. + - `apiserver_internal_error`: an API server internal error happened. + - `rejection_code`: the HTTP status code set in the admission response when a webhook rejected a request. @@ -1553,7 +1087,8 @@ the initial application. 2. For a `CREATE` pod request, if the field `.spec.containers[].resources.limits` of a container is not set, set default resource limits. -3. For a `CREATE` pod request, inject a sidecar container with name `foo-sidecar` if no container with the name `foo-sidecar` already exists. +3. For a `CREATE` pod request, inject a sidecar container with name `foo-sidecar` if no container + with the name `foo-sidecar` already exists. In the cases above, the webhook can be safely reinvoked, or admit an object that already has the fields set. @@ -1587,21 +1122,25 @@ versions. See [Matching requests: matchPolicy](#matching-requests-matchpolicy) f ### Availability -It is recommended that admission webhooks should evaluate as quickly as possible (typically in milliseconds), since they add to API request latency. +It is recommended that admission webhooks should evaluate as quickly as possible (typically in +milliseconds), since they add to API request latency. It is encouraged to use a small timeout for webhooks. See [Timeouts](#timeouts) for more detail. -It is recommended that admission webhooks should leverage some format of load-balancing, to provide high availability and -performance benefits. If a webhook is running within the cluster, you can run multiple webhook backends behind a service -to leverage the load-balancing that service supports. +It is recommended that admission webhooks should leverage some format of load-balancing, to +provide high availability and performance benefits. If a webhook is running within the cluster, +you can run multiple webhook backends behind a service to leverage the load-balancing that service +supports. ### Guaranteeing the final state of the object is seen Admission webhooks that need to guarantee they see the final state of the object in order to enforce policy should use a validating admission webhook, since objects can be modified after being seen by mutating webhooks. -For example, a mutating admission webhook is configured to inject a sidecar container with name "foo-sidecar" on every -`CREATE` pod request. If the sidecar *must* be present, a validating admisson webhook should also be configured to intercept `CREATE` pod requests, and validate -that a container with name "foo-sidecar" with the expected configuration exists in the to-be-created object. +For example, a mutating admission webhook is configured to inject a sidecar container with name +"foo-sidecar" on every `CREATE` pod request. If the sidecar *must* be present, a validating +admisson webhook should also be configured to intercept `CREATE` pod requests, and validate that a +container with name "foo-sidecar" with the expected configuration exists in the to-be-created +object. ### Avoiding deadlocks in self-hosted webhooks @@ -1614,7 +1153,8 @@ When a node that runs the webhook server pods becomes unhealthy, the webhook deployment will try to reschedule the pods to another node. However the requests will get rejected by the existing webhook server since the `"env"` label is unset, and the migration cannot happen. -It is recommended to exclude the namespace where your webhook is running with a [namespaceSelector](#matching-requests-namespaceselector). +It is recommended to exclude the namespace where your webhook is running with a +[namespaceSelector](#matching-requests-namespaceselector). ### Side effects @@ -1636,4 +1176,3 @@ If your admission webhooks don't intend to modify the behavior of the Kubernetes plane, exclude the `kube-system` namespace from being intercepted using a [`namespaceSelector`](#matching-requests-namespaceselector). - diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet-authentication-authorization.md b/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md similarity index 100% rename from content/en/docs/reference/command-line-tools-reference/kubelet-authentication-authorization.md rename to content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md b/content/en/docs/reference/access-authn-authz/kubelet-tls-bootstrapping.md similarity index 99% rename from content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md rename to content/en/docs/reference/access-authn-authz/kubelet-tls-bootstrapping.md index 5d2458079e..78aee8cf06 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md +++ b/content/en/docs/reference/access-authn-authz/kubelet-tls-bootstrapping.md @@ -126,7 +126,6 @@ of provisioning. 2. [Token authentication file](#token-authentication-file) Bootstrap tokens are a simpler and more easily managed method to authenticate kubelets, and do not require any additional flags when starting kube-apiserver. -Using bootstrap tokens is currently __beta__ as of Kubernetes version 1.12. Whichever method you choose, the requirement is that the kubelet be able to authenticate as a user with the rights to: diff --git a/content/en/docs/reference/access-authn-authz/node.md b/content/en/docs/reference/access-authn-authz/node.md index 141dc8bb61..bc9863219f 100644 --- a/content/en/docs/reference/access-authn-authz/node.md +++ b/content/en/docs/reference/access-authn-authz/node.md @@ -34,8 +34,8 @@ Write operations: Auth-related operations: -* read/write access to the certificationsigningrequests API for TLS bootstrapping -* the ability to create tokenreviews and subjectaccessreviews for delegated authentication/authorization checks +* read/write access to the [CertificateSigningRequests API](/docs/reference/access-authn-authz/certificate-signing-requests/) for TLS bootstrapping +* the ability to create TokenReviews and SubjectAccessReviews for delegated authentication/authorization checks In future releases, the node authorizer may add or remove permissions to ensure kubelets have the minimal set of permissions required to operate correctly. @@ -43,7 +43,7 @@ have the minimal set of permissions required to operate correctly. In order to be authorized by the Node authorizer, kubelets must use a credential that identifies them as being in the `system:nodes` group, with a username of `system:node:`. This group and user name format match the identity created for each kubelet as part of -[kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/). +[kubelet TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/). The value of `` **must** match precisely the name of the node as registered by the kubelet. By default, this is the host name as provided by `hostname`, or overridden via the [kubelet option](/docs/reference/command-line-tools-reference/kubelet/) `--hostname-override`. However, when using the `--cloud-provider` kubelet option, the specific hostname may be determined by the cloud provider, ignoring the local `hostname` and the `--hostname-override` option. For specifics about how the kubelet determines the hostname, see the [kubelet options reference](/docs/reference/command-line-tools-reference/kubelet/). diff --git a/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md b/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md new file mode 100644 index 0000000000..82394b363a --- /dev/null +++ b/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md @@ -0,0 +1,275 @@ +--- +reviewers: +- tallclair +- liggitt +title: Mapping PodSecurityPolicies to Pod Security Standards +content_type: concept +weight: 95 +--- + + +The tables below enumerate the configuration parameters on +[PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) objects, whether the field mutates +and/or validates pods, and how the configuration values map to the +[Pod Security Standards](/docs/concepts/security/pod-security-standards/). + +For each applicable parameter, the allowed values for the +[Baseline](/docs/concepts/security/pod-security-standards/#baseline) and +[Restricted](/docs/concepts/security/pod-security-standards/#restricted) profiles are listed. +Anything outside the allowed values for those profiles would fall under the +[Privileged](/docs/concepts/security/pod-security-standards/#privileged) profile. "No opinion" +means all values are allowed under all Pod Security Standards. + +For a step-by-step migration guide, see +[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/). + + + +## PodSecurityPolicy Spec + +The fields enumerated in this table are part of the `PodSecurityPolicySpec`, which is specified +under the `.spec` field path. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Mapping PodSecurityPolicySpec fields to Pod Security Standards
PodSecurityPolicySpecTypePod Security Standards Equivalent
privilegedValidatingBaseline & Restricted: false / undefined / nil
defaultAddCapabilitiesMutating & ValidatingRequirements match allowedCapabilities below.
allowedCapabilitiesValidating +

Baseline: subset of

+
    +
  • AUDIT_WRITE
    • +
  • CHOWN
    • +
  • DAC_OVERRIDE
    • +
  • FOWNER
    • +
  • FSETID
    • +
  • KILL
    • +
  • MKNOD
    • +
  • NET_BIND_SERVICE
    • +
  • SETFCAP
    • +
  • SETGID
    • +
  • SETPCAP
    • +
  • SETUID
    • +
  • SYS_CHROOT
    • +
+

Restricted: empty / undefined / nil OR a list containing only NET_BIND_SERVICE +

requiredDropCapabilitiesMutating & Validating +

Baseline: no opinion

+

Restricted: must include ALL

+
volumesValidating +

Baseline: anything except

+
    +
  • hostPath
    • +
  • *
    • +
+

Restricted: subset of

+
    +
  • configMap
    • +
  • csi
    • +
  • downwardAPI
    • +
  • emptyDir
    • +
  • ephemeral
    • +
  • persistentVolumeClaim
    • +
  • projected
    • +
  • secret
    • +
+
hostNetworkValidatingBaseline & Restricted: false / undefined / nil
hostPortsValidatingBaseline & Restricted: undefined / nil / empty
hostPIDValidatingBaseline & Restricted: false / undefined / nil
hostIPCValidatingBaseline & Restricted: false / undefined / nil
seLinuxMutating & Validating +

Baseline & Restricted: + seLinux.rule is MustRunAs, with the following options

+
    +
  • user is unset ("" / undefined / nil)
    • +
  • role is unset ("" / undefined / nil)
    • +
  • type is unset or one of: container_t, container_init_t, container_kvm_t
    • +
  • level is anything
    • +
+
runAsUserMutating & Validating +

Baseline: Anything

+

Restricted: rule is MustRunAsNonRoot

+
runAsGroupMutating (MustRunAs) & Validating + No opinion +
supplementalGroupsMutating & Validating + No opinion +
fsGroupMutating & Validating + No opinion +
readOnlyRootFilesystemMutating & Validating + No opinion +
defaultAllowPrivilegeEscalationMutating + No opinion (non-validating) +
allowPrivilegeEscalationMutating & Validating +

Only mutating if set to false

+

Baseline: No opinion

+

Restricted: false

+
allowedHostPathsValidatingNo opinion (volumes takes precedence)
allowedFlexVolumesValidatingNo opinion (volumes takes precedence)
allowedCSIDriversValidatingNo opinion (volumes takes precedence)
allowedUnsafeSysctlsValidatingBaseline & Restricted: undefined / nil / empty
forbiddenSysctlsValidatingNo opinion
allowedProcMountTypes
(alpha feature)		ValidatingBaseline & Restricted: ["Default"] OR undefined / nil / empty
runtimeClass
 .defaultRuntimeClassName		MutatingNo opinion
runtimeClass
 .allowedRuntimeClassNames		ValidatingNo opinion
+ +## PodSecurityPolicy annotations + +The [annotations](/docs/concepts/overview/working-with-objects/annotations/) enumerated in this +table can be specified under `.metadata.annotations` on the PodSecurityPolicy object. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Mapping PodSecurityPolicy annotations to Pod Security Standards
PSP AnnotationTypePod Security Standards Equivalent
seccomp.security.alpha.kubernetes.io
/defaultProfileName		MutatingNo opinion
seccomp.security.alpha.kubernetes.io
/allowedProfileNames		Validating +

Baseline: "runtime/default," (Trailing comma to allow unset)

+

Restricted: "runtime/default" (No trailing comma)

+

localhost/* values are also permitted for both Baseline & Restricted.

+
apparmor.security.beta.kubernetes.io
/defaultProfileName		MutatingNo opinion
apparmor.security.beta.kubernetes.io
/allowedProfileNames		Validating +

Baseline: "runtime/default," (Trailing comma to allow unset)

+

Restricted: "runtime/default" (No trailing comma)

+

localhost/* values are also permitted for both Baseline & Restricted.

+
diff --git a/content/en/docs/reference/access-authn-authz/rbac.md b/content/en/docs/reference/access-authn-authz/rbac.md index a75897d04a..a681164fe9 100644 --- a/content/en/docs/reference/access-authn-authz/rbac.md +++ b/content/en/docs/reference/access-authn-authz/rbac.md @@ -31,7 +31,7 @@ kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options The RBAC API declares four kinds of Kubernetes object: _Role_, _ClusterRole_, _RoleBinding_ and _ClusterRoleBinding_. You can [describe objects](/docs/concepts/overview/working-with-objects/kubernetes-objects/#understanding-kubernetes-objects), -or amend them, using tools such as `kubectl,` just like any other Kubernetes object. +or amend them, using tools such as `kubectl`, just like any other Kubernetes object. {{< caution >}} These objects, by design, impose access restrictions. If you are making changes @@ -54,8 +54,8 @@ it can't be both. ClusterRoles have several uses. You can use a ClusterRole to: -1. define permissions on namespaced resources and be granted within individual namespace(s) -1. define permissions on namespaced resources and be granted across all namespaces +1. define permissions on namespaced resources and be granted access within individual namespace(s) +1. define permissions on namespaced resources and be granted access across all namespaces 1. define permissions on cluster-scoped resources If you want to define a role within a namespace, use a Role; if you want to define @@ -384,11 +384,11 @@ rules: ``` Allow reading/writing Deployments (at the HTTP level: objects with `"deployments"` -in the resource part of their URL) in both the `"extensions"` and `"apps"` API groups: +in the resource part of their URL) in the `"apps"` API groups: ```yaml rules: -- apiGroups: ["extensions", "apps"] +- apiGroups: ["apps"] # # at the HTTP level, the name of the resource for accessing Deployment # objects is "deployments" @@ -397,7 +397,7 @@ rules: ``` Allow reading Pods in the core API group, as well as reading or writing Job -resources in the `"batch"` or `"extensions"` API groups: +resources in the `"batch"` API group: ```yaml rules: @@ -407,7 +407,7 @@ rules: # objects is "pods" resources: ["pods"] verbs: ["get", "list", "watch"] -- apiGroups: ["batch", "extensions"] +- apiGroups: ["batch"] # # at the HTTP level, the name of the resource for accessing Job # objects is "jobs" @@ -517,7 +517,7 @@ subjects: namespace: kube-system ``` -For all service accounts in the "qa" group in any namespace: +For all service accounts in the "qa" namespace: ```yaml subjects: @@ -525,15 +525,6 @@ subjects: name: system:serviceaccounts:qa apiGroup: rbac.authorization.k8s.io ``` -For all service accounts in the "dev" group in the "development" namespace: - -```yaml -subjects: -- kind: Group - name: system:serviceaccounts:dev - apiGroup: rbac.authorization.k8s.io - namespace: development -``` For all service accounts in any namespace: @@ -807,7 +798,7 @@ This is commonly used by add-on API servers for unified authentication and autho system:node-bootstrapper None Allows access to the resources required to perform -kubelet TLS bootstrapping. +kubelet TLS bootstrapping. system:node-problem-detector @@ -817,7 +808,7 @@ This is commonly used by add-on API servers for unified authentication and autho system:persistent-volume-provisioner None -Allows access to the resources required by most dynamic volume provisioners. +Allows access to the resources required by most dynamic volume provisioners. system:monitoring diff --git a/content/en/docs/reference/access-authn-authz/service-accounts-admin.md b/content/en/docs/reference/access-authn-authz/service-accounts-admin.md index f40cc7d00c..820b9b6cca 100644 --- a/content/en/docs/reference/access-authn-authz/service-accounts-admin.md +++ b/content/en/docs/reference/access-authn-authz/service-accounts-admin.md @@ -72,7 +72,7 @@ The ServiceAccount admission controller will add the following projected volume defaultMode: 420 # 0644 sources: - serviceAccountToken: - expirationSeconds: 3600 + expirationSeconds: 3607 path: token - configMap: items: diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index d639029659..d2d740c193 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -25,10 +25,11 @@ on each Kubernetes component. Each Kubernetes component lets you enable or disable a set of feature gates that are relevant to that component. Use `-h` flag to see a full set of feature gates for all components. -To set feature gates for a component, such as kubelet, use the `--feature-gates` flag assigned to a list of feature pairs: +To set feature gates for a component, such as kubelet, use the `--feature-gates` +flag assigned to a list of feature pairs: ```shell ---feature-gates="...,GracefulNodeShutdown=true" +--feature-gates=...,GracefulNodeShutdown=true ``` The following tables are a summary of the feature gates that you can set on @@ -53,153 +54,154 @@ different Kubernetes components. |---------|---------|-------|-------|-------| | `APIListChunking` | `false` | Alpha | 1.8 | 1.8 | | `APIListChunking` | `true` | Beta | 1.9 | | -| `APIPriorityAndFairness` | `false` | Alpha | 1.17 | 1.19 | +| `APIPriorityAndFairness` | `false` | Alpha | 1.18 | 1.19 | | `APIPriorityAndFairness` | `true` | Beta | 1.20 | | | `APIResponseCompression` | `false` | Alpha | 1.7 | 1.15 | | `APIResponseCompression` | `true` | Beta | 1.16 | | | `APIServerIdentity` | `false` | Alpha | 1.20 | | | `APIServerTracing` | `false` | Alpha | 1.22 | | | `AllowInsecureBackendProxy` | `true` | Beta | 1.17 | | -| `AnyVolumeDataSource` | `false` | Alpha | 1.18 | | +| `AnyVolumeDataSource` | `false` | Alpha | 1.18 | 1.23 | +| `AnyVolumeDataSource` | `true` | Beta | 1.24 | | | `AppArmor` | `true` | Beta | 1.4 | | -| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | | | `CPUManager` | `false` | Alpha | 1.8 | 1.9 | | `CPUManager` | `true` | Beta | 1.10 | | -| `CPUManagerPolicyOptions` | `false` | Alpha | 1.22 | | +| `CPUManagerPolicyAlphaOptions` | `false` | Alpha | 1.23 | | +| `CPUManagerPolicyBetaOptions` | `true` | Beta | 1.23 | | +| `CPUManagerPolicyOptions` | `false` | Alpha | 1.22 | 1.22 | +| `CPUManagerPolicyOptions` | `true` | Beta | 1.23 | | | `CSIInlineVolume` | `false` | Alpha | 1.15 | 1.15 | | `CSIInlineVolume` | `true` | Beta | 1.16 | - | | `CSIMigration` | `false` | Alpha | 1.14 | 1.16 | | `CSIMigration` | `true` | Beta | 1.17 | | -| `CSIMigrationAWS` | `false` | Alpha | 1.14 | | -| `CSIMigrationAWS` | `false` | Beta | 1.17 | | -| `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | 1.18 | -| `CSIMigrationAzureDisk` | `false` | Beta | 1.19 | | +| `CSIMigrationAWS` | `false` | Alpha | 1.14 | 1.16 | +| `CSIMigrationAWS` | `false` | Beta | 1.17 | 1.22 | +| `CSIMigrationAWS` | `true` | Beta | 1.23 | | | `CSIMigrationAzureFile` | `false` | Alpha | 1.15 | 1.19 | -| `CSIMigrationAzureFile` | `false` | Beta | 1.21 | | +| `CSIMigrationAzureFile` | `false` | Beta | 1.21 | 1.23 | +| `CSIMigrationAzureFile` | `true` | Beta | 1.24 | | | `CSIMigrationGCE` | `false` | Alpha | 1.14 | 1.16 | -| `CSIMigrationGCE` | `false` | Beta | 1.17 | | -| `CSIMigrationOpenStack` | `false` | Alpha | 1.14 | 1.17 | -| `CSIMigrationOpenStack` | `true` | Beta | 1.18 | | +| `CSIMigrationGCE` | `false` | Beta | 1.17 | 1.22 | +| `CSIMigrationGCE` | `true` | Beta | 1.23 | | | `CSIMigrationvSphere` | `false` | Beta | 1.19 | | -| `CSIStorageCapacity` | `false` | Alpha | 1.19 | 1.20 | -| `CSIStorageCapacity` | `true` | Beta | 1.21 | | -| `CSIVolumeFSGroupPolicy` | `false` | Alpha | 1.19 | 1.19 | -| `CSIVolumeFSGroupPolicy` | `true` | Beta | 1.20 | | +| `CSIMigrationPortworx` | `false` | Alpha | 1.23 | | +| `csiMigrationRBD` | `false` | Alpha | 1.23 | | | `CSIVolumeHealth` | `false` | Alpha | 1.21 | | -| `CSRDuration` | `true` | Beta | 1.22 | | -| `ConfigurableFSGroupPolicy` | `false` | Alpha | 1.18 | 1.19 | -| `ConfigurableFSGroupPolicy` | `true` | Beta | 1.20 | | -| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | 1.21 | -| `ControllerManagerLeaderMigration` | `true` | Beta | 1.22 | | +| `ContextualLogging` | `false` | Alpha | 1.24 | | | `CustomCPUCFSQuotaPeriod` | `false` | Alpha | 1.12 | | +| `CustomResourceValidationExpressions` | `false` | Alpha | 1.23 | | | `DaemonSetUpdateSurge` | `false` | Alpha | 1.21 | 1.21 | | `DaemonSetUpdateSurge` | `true` | Beta | 1.22 | | -| `DefaultPodTopologySpread` | `false` | Alpha | 1.19 | 1.19 | -| `DefaultPodTopologySpread` | `true` | Beta | 1.20 | | -| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | | +| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | 1.22 | +| `DelegateFSGroupToCSIDriver` | `true` | Beta | 1.23 | | | `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 | | `DevicePlugins` | `true` | Beta | 1.10 | | | `DisableAcceleratorUsageMetrics` | `false` | Alpha | 1.19 | 1.19 | | `DisableAcceleratorUsageMetrics` | `true` | Beta | 1.20 | | | `DisableCloudProviders` | `false` | Alpha | 1.22 | | +| `DisableKubeletCloudCredentialProviders` | `false` | Alpha | 1.23 | | | `DownwardAPIHugePages` | `false` | Alpha | 1.20 | 1.20 | -| `DownwardAPIHugePages` | `false` | Beta | 1.21 | | -| `EfficientWatchResumption` | `false` | Alpha | 1.20 | 1.20 | -| `EfficientWatchResumption` | `true` | Beta | 1.21 | | +| `DownwardAPIHugePages` | `false` | Beta | 1.21 | 1.21 | +| `DownwardAPIHugePages` | `true` | Beta | 1.22 | | | `EndpointSliceTerminatingCondition` | `false` | Alpha | 1.20 | 1.21 | | `EndpointSliceTerminatingCondition` | `true` | Beta | 1.22 | | -| `EphemeralContainers` | `false` | Alpha | 1.16 | | -| `ExpandCSIVolumes` | `false` | Alpha | 1.14 | 1.15 | -| `ExpandCSIVolumes` | `true` | Beta | 1.16 | | +| `EphemeralContainers` | `false` | Alpha | 1.16 | 1.22 | +| `EphemeralContainers` | `true` | Beta | 1.23 | | | `ExpandedDNSConfig` | `false` | Alpha | 1.22 | | -| `ExpandInUsePersistentVolumes` | `false` | Alpha | 1.11 | 1.14 | -| `ExpandInUsePersistentVolumes` | `true` | Beta | 1.15 | | -| `ExpandPersistentVolumes` | `false` | Alpha | 1.8 | 1.10 | -| `ExpandPersistentVolumes` | `true` | Beta | 1.11 | | | `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | | -| `GenericEphemeralVolume` | `false` | Alpha | 1.19 | 1.20 | -| `GenericEphemeralVolume` | `true` | Beta | 1.21 | | | `GracefulNodeShutdown` | `false` | Alpha | 1.20 | 1.20 | | `GracefulNodeShutdown` | `true` | Beta | 1.21 | | +| `GracefulNodeShutdownBasedOnPodPriority` | `false` | Alpha | 1.23 | 1.23 | +| `GracefulNodeShutdownBasedOnPodPriority` | `true` | Beta | 1.24 | | +| `GRPCContainerProbe` | `false` | Alpha | 1.23 | 1.23 | +| `GRPCContainerProbe` | `true` | Beta | 1.24 | | +| `HonorPVReclaimPolicy` | `false` | Alpha | 1.23 | | | `HPAContainerMetrics` | `false` | Alpha | 1.20 | | | `HPAScaleToZero` | `false` | Alpha | 1.16 | | -| `IndexedJob` | `false` | Alpha | 1.21 | 1.21 | -| `IndexedJob` | `true` | Beta | 1.22 | | -| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | | -| `IngressClassNamespacedParams` | `false` | Alpha | 1.21 | 1.21 | -| `IngressClassNamespacedParams` | `true` | Beta | 1.22 | | +| `IdentifyPodOS` | `false` | Alpha | 1.23 | 1.23 | +| `IdentifyPodOS` | `true` | Beta | 1.24 | | | `InTreePluginAWSUnregister` | `false` | Alpha | 1.21 | | | `InTreePluginAzureDiskUnregister` | `false` | Alpha | 1.21 | | | `InTreePluginAzureFileUnregister` | `false` | Alpha | 1.21 | | | `InTreePluginGCEUnregister` | `false` | Alpha | 1.21 | | | `InTreePluginOpenStackUnregister` | `false` | Alpha | 1.21 | | +| `InTreePluginPortworxUnregister` | `false` | Alpha | 1.23 | | +| `InTreePluginRBDUnregister` | `false` | Alpha | 1.23 | | | `InTreePluginvSphereUnregister` | `false` | Alpha | 1.21 | | -| `IPv6DualStack` | `false` | Alpha | 1.15 | 1.20 | -| `IPv6DualStack` | `true` | Beta | 1.21 | | -| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | | -| `KubeletCredentialProviders` | `false` | Alpha | 1.20 | | +| `JobMutableNodeSchedulingDirectives` | `true` | Beta | 1.23 | | +| `JobReadyPods` | `false` | Alpha | 1.23 | 1.23 | +| `JobReadyPods` | `true` | Beta | 1.24 | | +| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | 1.22 | +| `JobTrackingWithFinalizers` | `true` | Beta | 1.23 | 1.23 | +| `JobTrackingWithFinalizers` | `false` | Beta | 1.24 | | +| `KubeletCredentialProviders` | `false` | Alpha | 1.20 | 1.23 | +| `KubeletCredentialProviders` | `true` | Beta | 1.24 | | +| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | | +| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 | +| `KubeletPodResources` | `true` | Beta | 1.15 | | +| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | 1.22 | +| `KubeletPodResourcesGetAllocatable` | `true` | Beta | 1.23 | | | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 | | `LocalStorageCapacityIsolation` | `true` | Beta | 1.10 | | | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | | | `LogarithmicScaleDown` | `false` | Alpha | 1.21 | 1.21 | | `LogarithmicScaleDown` | `true` | Beta | 1.22 | | -| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | | -| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | | +| `MaxUnavailableStatefulSet` | `false` | Alpha | 1.24 | | | `MemoryManager` | `false` | Alpha | 1.21 | 1.21 | | `MemoryManager` | `true` | Beta | 1.22 | | | `MemoryQoS` | `false` | Alpha | 1.22 | | -| `MixedProtocolLBService` | `false` | Alpha | 1.20 | | +| `MinDomainsInPodTopologySpread` | `false` | Alpha | 1.24 | | +| `MixedProtocolLBService` | `false` | Alpha | 1.20 | 1.23 | +| `MixedProtocolLBService` | `true` | Beta | 1.24 | | | `NetworkPolicyEndPort` | `false` | Alpha | 1.21 | 1.21 | | `NetworkPolicyEndPort` | `true` | Beta | 1.22 | | +| `NetworkPolicyStatus` | `false` | Alpha | 1.24 | | | `NodeSwap` | `false` | Alpha | 1.22 | | -| `NonPreemptingPriority` | `false` | Alpha | 1.15 | 1.18 | -| `NonPreemptingPriority` | `true` | Beta | 1.19 | | +| `NodeOutOfServiceVolumeDetach` | `false` | Alpha | 1.24 | | +| `OpenAPIEnums` | `false` | Alpha | 1.23 | 1.23 | +| `OpenAPIEnums` | `true` | Beta | 1.24 | | +| `OpenAPIV3` | `false` | Alpha | 1.23 | 1.23 | +| `OpenAPIV3` | `true` | Beta | 1.24 | | +| `PodAndContainerStatsFromCRI` | `false` | Alpha | 1.23 | | | `PodDeletionCost` | `false` | Alpha | 1.21 | 1.21 | | `PodDeletionCost` | `true` | Beta | 1.22 | | -| `PodAffinityNamespaceSelector` | `false` | Alpha | 1.21 | 1.21 | -| `PodAffinityNamespaceSelector` | `true` | Beta | 1.22 | | -| `PodOverhead` | `false` | Alpha | 1.16 | 1.17 | -| `PodOverhead` | `true` | Beta | 1.18 | | -| `PodSecurity` | `false` | Alpha | 1.22 | | -| `PreferNominatedNode` | `false` | Alpha | 1.21 | 1.21 | -| `PreferNominatedNode` | `true` | Beta | 1.22 | | +| `PodSecurity` | `false` | Alpha | 1.22 | 1.22 | +| `PodSecurity` | `true` | Beta | 1.23 | | | `ProbeTerminationGracePeriod` | `false` | Alpha | 1.21 | 1.21 | | `ProbeTerminationGracePeriod` | `false` | Beta | 1.22 | | | `ProcMountType` | `false` | Alpha | 1.12 | | | `ProxyTerminatingEndpoints` | `false` | Alpha | 1.22 | | | `QOSReserved` | `false` | Alpha | 1.11 | | | `ReadWriteOncePod` | `false` | Alpha | 1.22 | | +| `RecoverVolumeExpansionFailure` | `false` | Alpha | 1.23 | | | `RemainingItemCount` | `false` | Alpha | 1.15 | 1.15 | | `RemainingItemCount` | `true` | Beta | 1.16 | | -| `RemoveSelfLink` | `false` | Alpha | 1.16 | 1.19 | -| `RemoveSelfLink` | `true` | Beta | 1.20 | | | `RotateKubeletServerCertificate` | `false` | Alpha | 1.7 | 1.11 | | `RotateKubeletServerCertificate` | `true` | Beta | 1.12 | | | `SeccompDefault` | `false` | Alpha | 1.22 | | +| `ServerSideFieldValidation` | `false` | Alpha | 1.23 | - | | `ServiceInternalTrafficPolicy` | `false` | Alpha | 1.21 | 1.21 | | `ServiceInternalTrafficPolicy` | `true` | Beta | 1.22 | | -| `ServiceLBNodePortControl` | `false` | Alpha | 1.20 | 1.21 | -| `ServiceLBNodePortControl` | `true` | Beta | 1.22 | | -| `ServiceLoadBalancerClass` | `false` | Alpha | 1.21 | 1.21 | -| `ServiceLoadBalancerClass` | `true` | Beta | 1.22 | | +| `ServiceIPStaticSubrange` | `false` | Alpha | 1.24 | | | `SizeMemoryBackedVolumes` | `false` | Alpha | 1.20 | 1.21 | | `SizeMemoryBackedVolumes` | `true` | Beta | 1.22 | | -| `StatefulSetMinReadySeconds` | `false` | Alpha | 1.22 | | +| `StatefulSetAutoDeletePVC` | `false` | Alpha | 1.22 | | +| `StatefulSetMinReadySeconds` | `false` | Alpha | 1.22 | 1.22 | +| `StatefulSetMinReadySeconds` | `true` | Beta | 1.23 | | | `StorageVersionAPI` | `false` | Alpha | 1.20 | | | `StorageVersionHash` | `false` | Alpha | 1.14 | 1.14 | | `StorageVersionHash` | `true` | Beta | 1.15 | | -| `SuspendJob` | `false` | Alpha | 1.21 | 1.21 | -| `SuspendJob` | `true` | Beta | 1.22 | | -| `TTLAfterFinished` | `false` | Alpha | 1.12 | 1.20 | -| `TTLAfterFinished` | `true` | Beta | 1.21 | | -| `TopologyAwareHints` | `false` | Alpha | 1.21 | | +| `TopologyAwareHints` | `false` | Alpha | 1.21 | 1.22 | +| `TopologyAwareHints` | `false` | Beta | 1.23 | 1.23 | +| `TopologyAwareHints` | `true` | Beta | 1.24 | | | `TopologyManager` | `false` | Alpha | 1.16 | 1.17 | | `TopologyManager` | `true` | Beta | 1.18 | | | `VolumeCapacityPriority` | `false` | Alpha | 1.21 | - | | `WinDSR` | `false` | Alpha | 1.14 | | | `WinOverlay` | `false` | Alpha | 1.14 | 1.19 | | `WinOverlay` | `true` | Beta | 1.20 | | -| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | | +| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | 1.22 | +| `WindowsHostProcessContainers` | `true` | Beta | 1.23 | | {{< /table >}} ### Feature gates for graduated or deprecated features @@ -228,6 +230,12 @@ different Kubernetes components. | `BoundServiceAccountTokenVolume` | `false` | Alpha | 1.13 | 1.20 | | `BoundServiceAccountTokenVolume` | `true` | Beta | 1.21 | 1.21 | | `BoundServiceAccountTokenVolume` | `true` | GA | 1.22 | - | +| `ConfigurableFSGroupPolicy` | `false` | Alpha | 1.18 | 1.19 | +| `ConfigurableFSGroupPolicy` | `true` | Beta | 1.20 | 1.22 | +| `ConfigurableFSGroupPolicy` | `true` | GA | 1.23 | - | +| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | 1.21 | +| `ControllerManagerLeaderMigration` | `true` | Beta | 1.22 | 1.23 | +| `ControllerManagerLeaderMigration` | `true` | GA | 1.24 | - | | `CRIContainerLogRotation` | `false` | Alpha | 1.10 | 1.10 | | `CRIContainerLogRotation` | `true` | Beta | 1.11 | 1.20 | | `CRIContainerLogRotation` | `true` | GA | 1.21 | - | @@ -236,31 +244,47 @@ different Kubernetes components. | `CSIBlockVolume` | `true` | GA | 1.18 | - | | `CSIDriverRegistry` | `false` | Alpha | 1.12 | 1.13 | | `CSIDriverRegistry` | `true` | Beta | 1.14 | 1.17 | -| `CSIDriverRegistry` | `true` | GA | 1.18 | | +| `CSIDriverRegistry` | `true` | GA | 1.18 | - | | `CSIMigrationAWSComplete` | `false` | Alpha | 1.17 | 1.20 | | `CSIMigrationAWSComplete` | - | Deprecated | 1.21 | - | +| `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | 1.18 | +| `CSIMigrationAzureDisk` | `false` | Beta | 1.19 | 1.22 | +| `CSIMigrationAzureDisk` | `true` | Beta | 1.23 | 1.23 | +| `CSIMigrationAzureDisk` | `true` | GA | 1.24 | | | `CSIMigrationAzureDiskComplete` | `false` | Alpha | 1.17 | 1.20 | | `CSIMigrationAzureDiskComplete` | - | Deprecated | 1.21 | - | | `CSIMigrationAzureFileComplete` | `false` | Alpha | 1.17 | 1.20 | | `CSIMigrationAzureFileComplete` | - | Deprecated | 1.21 | - | | `CSIMigrationGCEComplete` | `false` | Alpha | 1.17 | 1.20 | | `CSIMigrationGCEComplete` | - | Deprecated | 1.21 | - | +| `CSIMigrationOpenStack` | `false` | Alpha | 1.14 | 1.17 | +| `CSIMigrationOpenStack` | `true` | Beta | 1.18 | 1.23 | +| `CSIMigrationOpenStack` | `true` | GA | 1.24 | | | `CSIMigrationOpenStackComplete` | `false` | Alpha | 1.17 | 1.20 | | `CSIMigrationOpenStackComplete` | - | Deprecated | 1.21 | - | | `CSIMigrationvSphereComplete` | `false` | Beta | 1.19 | 1.21 | | `CSIMigrationvSphereComplete` | - | Deprecated | 1.22 | - | | `CSINodeInfo` | `false` | Alpha | 1.12 | 1.13 | | `CSINodeInfo` | `true` | Beta | 1.14 | 1.16 | -| `CSINodeInfo` | `true` | GA | 1.17 | | +| `CSINodeInfo` | `true` | GA | 1.17 | - | | `CSIPersistentVolume` | `false` | Alpha | 1.9 | 1.9 | | `CSIPersistentVolume` | `true` | Beta | 1.10 | 1.12 | | `CSIPersistentVolume` | `true` | GA | 1.13 | - | | `CSIServiceAccountToken` | `false` | Alpha | 1.20 | 1.20 | | `CSIServiceAccountToken` | `true` | Beta | 1.21 | 1.21 | -| `CSIServiceAccountToken` | `true` | GA | 1.22 | | +| `CSIServiceAccountToken` | `true` | GA | 1.22 | - | +| `CSIStorageCapacity` | `false` | Alpha | 1.19 | 1.20 | +| `CSIStorageCapacity` | `true` | Beta | 1.21 | 1.23 | +| `CSIStorageCapacity` | `true` | GA | 1.24 | - | +| `CSIVolumeFSGroupPolicy` | `false` | Alpha | 1.19 | 1.19 | +| `CSIVolumeFSGroupPolicy` | `true` | Beta | 1.20 | 1.22 | +| `CSIVolumeFSGroupPolicy` | `true` | GA | 1.23 | | +| `CSRDuration` | `true` | Beta | 1.22 | 1.23 | +| `CSRDuration` | `true` | GA | 1.24 | - | | `CronJobControllerV2` | `false` | Alpha | 1.20 | 1.20 | | `CronJobControllerV2` | `true` | Beta | 1.21 | 1.21 | | `CronJobControllerV2` | `true` | GA | 1.22 | - | +| `CronJobTimeZone` | `false` | Alpha | 1.24 | | | `CustomPodDNS` | `false` | Alpha | 1.9 | 1.9 | | `CustomPodDNS` | `true` | Beta| 1.10 | 1.13 | | `CustomPodDNS` | `true` | GA | 1.14 | - | @@ -279,6 +303,9 @@ different Kubernetes components. | `CustomResourceWebhookConversion` | `false` | Alpha | 1.13 | 1.14 | | `CustomResourceWebhookConversion` | `true` | Beta | 1.15 | 1.15 | | `CustomResourceWebhookConversion` | `true` | GA | 1.16 | - | +| `DefaultPodTopologySpread` | `false` | Alpha | 1.19 | 1.19 | +| `DefaultPodTopologySpread` | `true` | Beta | 1.20 | 1.23 | +| `DefaultPodTopologySpread` | `true` | GA | 1.24 | - | | `DryRun` | `false` | Alpha | 1.12 | 1.12 | | `DryRun` | `true` | Beta | 1.13 | 1.18 | | `DryRun` | `true` | GA | 1.19 | - | @@ -289,46 +316,64 @@ different Kubernetes components. | `DynamicKubeletConfig` | `false` | Deprecated | 1.22 | - | | `DynamicProvisioningScheduling` | `false` | Alpha | 1.11 | 1.11 | | `DynamicProvisioningScheduling` | - | Deprecated| 1.12 | - | -| `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 | -| `DynamicKubeletConfig` | `true` | Beta | 1.11 | 1.21 | -| `DynamicKubeletConfig` | `false` | Deprecated | 1.22 | - | | `DynamicVolumeProvisioning` | `true` | Alpha | 1.3 | 1.7 | | `DynamicVolumeProvisioning` | `true` | GA | 1.8 | - | +| `EfficientWatchResumption` | `false` | Alpha | 1.20 | 1.20 | +| `EfficientWatchResumption` | `true` | Beta | 1.21 | 1.23 | +| `EfficientWatchResumption` | `true` | GA | 1.24 | - | | `EnableAggregatedDiscoveryTimeout` | `true` | Deprecated | 1.16 | - | | `EnableEquivalenceClassCache` | `false` | Alpha | 1.8 | 1.14 | | `EnableEquivalenceClassCache` | - | Deprecated | 1.15 | - | | `EndpointSlice` | `false` | Alpha | 1.16 | 1.16 | | `EndpointSlice` | `false` | Beta | 1.17 | 1.17 | | `EndpointSlice` | `true` | Beta | 1.18 | 1.20 | -| `EndpointSlice` | `true` | GA | 1.21 | - | +| `EndpointSlice` | `true` | GA | 1.21 | - | | `EndpointSliceNodeName` | `false` | Alpha | 1.20 | 1.20 | | `EndpointSliceNodeName` | `true` | GA | 1.21 | - | | `EndpointSliceProxying` | `false` | Alpha | 1.18 | 1.18 | | `EndpointSliceProxying` | `true` | Beta | 1.19 | 1.21 | | `EndpointSliceProxying` | `true` | GA | 1.22 | - | -| `ExperimentalCriticalPodAnnotation` | `false` | Alpha | 1.5 | 1.12 | -| `ExperimentalCriticalPodAnnotation` | `false` | Deprecated | 1.13 | - | | `EvenPodsSpread` | `false` | Alpha | 1.16 | 1.17 | | `EvenPodsSpread` | `true` | Beta | 1.18 | 1.18 | | `EvenPodsSpread` | `true` | GA | 1.19 | - | | `ExecProbeTimeout` | `true` | GA | 1.20 | - | +| `ExpandCSIVolumes` | `false` | Alpha | 1.14 | 1.15 | +| `ExpandCSIVolumes` | `true` | Beta | 1.16 | 1.23 | +| `ExpandCSIVolumes` | `true` | GA | 1.24 | - | +| `ExpandInUsePersistentVolumes` | `false` | Alpha | 1.11 | 1.14 | +| `ExpandInUsePersistentVolumes` | `true` | Beta | 1.15 | 1.23 | +| `ExpandInUsePersistentVolumes` | `true` | GA | 1.24 | - | +| `ExpandPersistentVolumes` | `false` | Alpha | 1.8 | 1.10 | +| `ExpandPersistentVolumes` | `true` | Beta | 1.11 | 1.23 | +| `ExpandPersistentVolumes` | `true` | GA | 1.24 |- | +| `ExperimentalCriticalPodAnnotation` | `false` | Alpha | 1.5 | 1.12 | +| `ExperimentalCriticalPodAnnotation` | `false` | Deprecated | 1.13 | - | | `ExternalPolicyForExternalIP` | `true` | GA | 1.18 | - | | `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | 1.12 | | `GCERegionalPersistentDisk` | `true` | GA | 1.13 | - | +| `GenericEphemeralVolume` | `false` | Alpha | 1.19 | 1.20 | +| `GenericEphemeralVolume` | `true` | Beta | 1.21 | 1.22 | +| `GenericEphemeralVolume` | `true` | GA | 1.23 | - | | `HugePageStorageMediumSize` | `false` | Alpha | 1.18 | 1.18 | | `HugePageStorageMediumSize` | `true` | Beta | 1.19 | 1.21 | | `HugePageStorageMediumSize` | `true` | GA | 1.22 | - | | `HugePages` | `false` | Alpha | 1.8 | 1.9 | | `HugePages` | `true` | Beta| 1.10 | 1.13 | | `HugePages` | `true` | GA | 1.14 | - | -| `HugePageStorageMediumSize` | `false` | Alpha | 1.18 | 1.18 | -| `HugePageStorageMediumSize` | `true` | Beta | 1.19 | 1.21 | -| `HugePageStorageMediumSize` | `true` | GA | 1.22 | - | | `HyperVContainer` | `false` | Alpha | 1.10 | 1.19 | | `HyperVContainer` | `false` | Deprecated | 1.20 | - | +| `IPv6DualStack` | `false` | Alpha | 1.15 | 1.20 | +| `IPv6DualStack` | `true` | Beta | 1.21 | 1.22 | +| `IPv6DualStack` | `true` | GA | 1.23 | - | | `ImmutableEphemeralVolumes` | `false` | Alpha | 1.18 | 1.18 | | `ImmutableEphemeralVolumes` | `true` | Beta | 1.19 | 1.20 | | `ImmutableEphemeralVolumes` | `true` | GA | 1.21 | | +| `IndexedJob` | `false` | Alpha | 1.21 | 1.21 | +| `IndexedJob` | `true` | Beta | 1.22 | 1.23 | +| `IndexedJob` | `true` | GA | 1.24 | - | +| `IngressClassNamespacedParams` | `false` | Alpha | 1.21 | 1.21 | +| `IngressClassNamespacedParams` | `true` | Beta | 1.22 | 1.22 | +| `IngressClassNamespacedParams` | `true` | GA | 1.23 | - | | `Initializers` | `false` | Alpha | 1.7 | 1.13 | | `Initializers` | - | Deprecated | 1.14 | - | | `KubeletConfigFile` | `false` | Alpha | 1.8 | 1.9 | @@ -336,33 +381,40 @@ different Kubernetes components. | `KubeletPluginsWatcher` | `false` | Alpha | 1.11 | 1.11 | | `KubeletPluginsWatcher` | `true` | Beta | 1.12 | 1.12 | | `KubeletPluginsWatcher` | `true` | GA | 1.13 | - | -| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 | -| `KubeletPodResources` | `true` | Beta | 1.15 | | -| `KubeletPodResources` | `true` | GA | 1.20 | | | `LegacyNodeRoleBehavior` | `false` | Alpha | 1.16 | 1.18 | | `LegacyNodeRoleBehavior` | `true` | Beta | 1.19 | 1.20 | | `LegacyNodeRoleBehavior` | `false` | GA | 1.21 | - | +| `LegacyServiceAccountTokenNoAutoGeneration` | `true` | Beta | 1.24 | | | `MountContainers` | `false` | Alpha | 1.9 | 1.16 | | `MountContainers` | `false` | Deprecated | 1.17 | - | | `MountPropagation` | `false` | Alpha | 1.8 | 1.9 | | `MountPropagation` | `true` | Beta | 1.10 | 1.11 | | `MountPropagation` | `true` | GA | 1.12 | - | +| `NamespaceDefaultLabelName` | `true` | Beta | 1.21 | 1.21 | +| `NamespaceDefaultLabelName` | `true` | GA | 1.22 | - | | `NodeDisruptionExclusion` | `false` | Alpha | 1.16 | 1.18 | | `NodeDisruptionExclusion` | `true` | Beta | 1.19 | 1.20 | | `NodeDisruptionExclusion` | `true` | GA | 1.21 | - | | `NodeLease` | `false` | Alpha | 1.12 | 1.13 | | `NodeLease` | `true` | Beta | 1.14 | 1.16 | | `NodeLease` | `true` | GA | 1.17 | - | -| `NamespaceDefaultLabelName` | `true` | Beta | 1.21 | 1.21 | -| `NamespaceDefaultLabelName` | `true` | GA | 1.22 | - | +| `NonPreemptingPriority` | `false` | Alpha | 1.15 | 1.18 | +| `NonPreemptingPriority` | `true` | Beta | 1.19 | 1.23 | +| `NonPreemptingPriority` | `true` | GA | 1.24 | - | | `PVCProtection` | `false` | Alpha | 1.9 | 1.9 | | `PVCProtection` | - | Deprecated | 1.10 | - | | `PersistentLocalVolumes` | `false` | Alpha | 1.7 | 1.9 | | `PersistentLocalVolumes` | `true` | Beta | 1.10 | 1.13 | | `PersistentLocalVolumes` | `true` | GA | 1.14 | - | +| `PodAffinityNamespaceSelector` | `false` | Alpha | 1.21 | 1.21 | +| `PodAffinityNamespaceSelector` | `true` | Beta | 1.22 | 1.23 | +| `PodAffinityNamespaceSelector` | `true` | GA | 1.24 | - | | `PodDisruptionBudget` | `false` | Alpha | 1.3 | 1.4 | | `PodDisruptionBudget` | `true` | Beta | 1.5 | 1.20 | | `PodDisruptionBudget` | `true` | GA | 1.21 | - | +| `PodOverhead` | `false` | Alpha | 1.16 | 1.17 | +| `PodOverhead` | `true` | Beta | 1.18 | 1.23 | +| `PodOverhead` | `true` | GA | 1.24 | - | | `PodPriority` | `false` | Alpha | 1.8 | 1.10 | | `PodPriority` | `true` | Beta | 1.11 | 1.13 | | `PodPriority` | `true` | GA | 1.14 | - | @@ -372,8 +424,14 @@ different Kubernetes components. | `PodShareProcessNamespace` | `false` | Alpha | 1.10 | 1.11 | | `PodShareProcessNamespace` | `true` | Beta | 1.12 | 1.16 | | `PodShareProcessNamespace` | `true` | GA | 1.17 | - | +| `PreferNominatedNode` | `false` | Alpha | 1.21 | 1.21 | +| `PreferNominatedNode` | `true` | Beta | 1.22 | 1.23 | +| `PreferNominatedNode` | `true` | GA | 1.24 | - | +| `RemoveSelfLink` | `false` | Alpha | 1.16 | 1.19 | +| `RemoveSelfLink` | `true` | Beta | 1.20 | 1.23 | +| `RemoveSelfLink` | `true` | GA | 1.24 | - | | `RequestManagement` | `false` | Alpha | 1.15 | 1.16 | -| `RequestManagement` | - | Derecated | 1.17 | - | +| `RequestManagement` | - | Deprecated | 1.17 | - | | `ResourceLimitsPriorityFunction` | `false` | Alpha | 1.9 | 1.18 | | `ResourceLimitsPriorityFunction` | - | Deprecated | 1.19 | - | | `ResourceQuotaScopeSelectors` | `false` | Alpha | 1.11 | 1.11 | @@ -407,6 +465,12 @@ different Kubernetes components. | `ServiceAppProtocol` | `false` | Alpha | 1.18 | 1.18 | | `ServiceAppProtocol` | `true` | Beta | 1.19 | 1.19 | | `ServiceAppProtocol` | `true` | GA | 1.20 | - | +| `ServiceLBNodePortControl` | `false` | Alpha | 1.20 | 1.21 | +| `ServiceLBNodePortControl` | `true` | Beta | 1.22 | 1.23 | +| `ServiceLBNodePortControl` | `true` | GA | 1.24 | - | +| `ServiceLoadBalancerClass` | `false` | Alpha | 1.21 | 1.21 | +| `ServiceLoadBalancerClass` | `true` | Beta | 1.22 | 1.23 | +| `ServiceLoadBalancerClass` | `true` | GA | 1.24 | - | | `ServiceLoadBalancerFinalizer` | `false` | Alpha | 1.15 | 1.15 | | `ServiceLoadBalancerFinalizer` | `true` | Beta | 1.16 | 1.16 | | `ServiceLoadBalancerFinalizer` | `true` | GA | 1.17 | - | @@ -437,8 +501,14 @@ different Kubernetes components. | `SupportPodPidsLimit` | `false` | Alpha | 1.10 | 1.13 | | `SupportPodPidsLimit` | `true` | Beta | 1.14 | 1.19 | | `SupportPodPidsLimit` | `true` | GA | 1.20 | - | +| `SuspendJob` | `false` | Alpha | 1.21 | 1.21 | +| `SuspendJob` | `true` | Beta | 1.22 | 1.23 | +| `SuspendJob` | `true` | GA | 1.24 | - | | `Sysctls` | `true` | Beta | 1.11 | 1.20 | -| `Sysctls` | `true` | GA | 1.21 | | +| `Sysctls` | `true` | GA | 1.21 | - | +| `TTLAfterFinished` | `false` | Alpha | 1.12 | 1.20 | +| `TTLAfterFinished` | `true` | Beta | 1.21 | 1.22 | +| `TTLAfterFinished` | `true` | GA | 1.23 | - | | `TaintBasedEvictions` | `false` | Alpha | 1.6 | 1.12 | | `TaintBasedEvictions` | `true` | Beta | 1.13 | 1.17 | | `TaintBasedEvictions` | `true` | GA | 1.18 | - | @@ -478,7 +548,6 @@ different Kubernetes components. | `WindowsGMSA` | `false` | Alpha | 1.14 | 1.15 | | `WindowsGMSA` | `true` | Beta | 1.16 | 1.17 | | `WindowsGMSA` | `true` | GA | 1.18 | - | -| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | | `WindowsRunAsUserName` | `false` | Alpha | 1.16 | 1.16 | | `WindowsRunAsUserName` | `true` | Beta | 1.17 | 1.17 | | `WindowsRunAsUserName` | `true` | GA | 1.18 | - | @@ -534,8 +603,11 @@ Each feature gate is designed for enabling/disabling a specific feature: - `APIResponseCompression`: Compress the API responses for `LIST` or `GET` requests. - `APIServerIdentity`: Assign each API server an ID in a cluster. - `APIServerTracing`: Add support for distributed tracing in the API server. -- `Accelerators`: Enable Nvidia GPU support when using Docker -- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug-application-cluster/audit/#advanced-audit) +- `Accelerators`: Provided an early form of plugin to enable Nvidia GPU support when using + Docker Engine; no longer available. See + [Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for + an alternative. +- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit) - `AffinityInAnnotations`: Enable setting [Pod affinity or anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity). - `AllowExtTrafficLocalEndpoints`: Enable a service to route external requests to node local endpoints. @@ -543,37 +615,51 @@ Each feature gate is designed for enabling/disabling a specific feature: kubelets on Pod log requests. - `AnyVolumeDataSource`: Enable use of any custom resource as the `DataSource` of a {{< glossary_tooltip text="PVC" term_id="persistent-volume-claim" >}}. -- `AppArmor`: Enable AppArmor based mandatory access control on Linux nodes when using Docker. - See [AppArmor Tutorial](/docs/tutorials/clusters/apparmor/) for more details. +- `AppArmor`: Enable use of AppArmor mandatory access control for Pods running on Linux nodes. + See [AppArmor Tutorial](/docs/tutorials/security/apparmor/) for more details. - `AttachVolumeLimit`: Enable volume plugins to report limits on number of volumes that can be attached to a node. - See [dynamic volume limits](/docs/concepts/storage/storage-limits/#dynamic-volume-limits) for more details. -- `BalanceAttachedNodeVolumes`: Include volume count on node to be considered for balanced resource allocation - while scheduling. A node which has closer CPU, memory utilization, and volume count is favored by the scheduler - while making decisions. + See [dynamic volume limits](/docs/concepts/storage/storage-limits/#dynamic-volume-limits) + for more details. +- `BalanceAttachedNodeVolumes`: Include volume count on node to be considered for + balanced resource allocation while scheduling. A node which has closer CPU, + memory utilization, and volume count is favored by the scheduler while making decisions. - `BlockVolume`: Enable the definition and consumption of raw block devices in Pods. See [Raw Block Volume Support](/docs/concepts/storage/persistent-volumes/#raw-block-volume-support) for more details. -- `BoundServiceAccountTokenVolume`: Migrate ServiceAccount volumes to use a projected volume consisting of a - ServiceAccountTokenVolumeProjection. Cluster admins can use metric `serviceaccount_stale_tokens_total` to - monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off - extended tokens by starting `kube-apiserver` with flag `--service-account-extend-token-expiration=false`. +- `BoundServiceAccountTokenVolume`: Migrate ServiceAccount volumes to use a projected volume + consisting of a ServiceAccountTokenVolumeProjection. Cluster admins can use metric + `serviceaccount_stale_tokens_total` to monitor workloads that are depending on the extended + tokens. If there are no such workloads, turn off extended tokens by starting `kube-apiserver` with + flag `--service-account-extend-token-expiration=false`. Check [Bound Service Account Tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md) - for more details. -- `ControllerManagerLeaderMigration`: Enables Leader Migration for - [kube-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#initial-leader-migration-configuration) and - [cloud-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#deploy-cloud-controller-manager) which allows a cluster operator to live migrate - controllers from the kube-controller-manager into an external controller-manager + for more details. +- `ControllerManagerLeaderMigration`: Enables Leader Migration for + [kube-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#initial-leader-migration-configuration) and + [cloud-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#deploy-cloud-controller-manager) + which allows a cluster operator to live migrate + controllers from the kube-controller-manager into an external controller-manager (e.g. the cloud-controller-manager) in an HA cluster without downtime. - `CPUManager`: Enable container level CPU affinity support, see [CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/). +- `CPUManagerPolicyAlphaOptions`: This allows fine-tuning of CPUManager policies, + experimental, Alpha-quality options + This feature gate guards *a group* of CPUManager options whose quality level is alpha. + This feature gate will never graduate to beta or stable. +- `CPUManagerPolicyBetaOptions`: This allows fine-tuning of CPUManager policies, + experimental, Beta-quality options + This feature gate guards *a group* of CPUManager options whose quality level is beta. + This feature gate will never graduate to stable. - `CPUManagerPolicyOptions`: Allow fine-tuning of CPUManager policies. -- `CRIContainerLogRotation`: Enable container log rotation for CRI container runtime. The default max size of a log file is 10MB and the - default max number of log files allowed for a container is 5. These values can be configured in the kubelet config. - See the [logging at node level](/docs/concepts/cluster-administration/logging/#logging-at-the-node-level) documentation for more details. +- `CRIContainerLogRotation`: Enable container log rotation for CRI container runtime. + The default max size of a log file is 10MB and the default max number of + log files allowed for a container is 5. + These values can be configured in the kubelet config. + See [logging at node level](/docs/concepts/cluster-administration/logging/#logging-at-the-node-level) + for more details. - `CSIBlockVolume`: Enable external CSI volume drivers to support block storage. - See the [`csi` raw block volume support](/docs/concepts/storage/volumes/#csi-raw-block-volume-support) - documentation for more details. + See [`csi` raw block volume support](/docs/concepts/storage/volumes/#csi-raw-block-volume-support) + for more details. - `CSIDriverRegistry`: Enable all logic related to the CSIDriver API object in csi.storage.k8s.io. - `CSIInlineVolume`: Enable CSI Inline volumes support for pods. @@ -581,8 +667,10 @@ Each feature gate is designed for enabling/disabling a specific feature: operations from in-tree plugins to corresponding pre-installed CSI plugins - `CSIMigrationAWS`: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Supports - falling back to in-tree EBS plugin if a node does not have EBS CSI plugin - installed and configured. Requires CSIMigration feature flag enabled. + falling back to in-tree EBS plugin for mount operations to nodes that have + the feature disabled or that do not have EBS CSI plugin installed and + configured. Does not support falling back for provision operations, for those + the CSI plugin must be installed and configured. - `CSIMigrationAWSComplete`: Stops registering the EBS in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. @@ -592,21 +680,26 @@ Each feature gate is designed for enabling/disabling a specific feature: which prevents the registration of in-tree EBS plugin. - `CSIMigrationAzureDisk`: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. - Supports falling back to in-tree AzureDisk plugin if a node does not have - AzureDisk CSI plugin installed and configured. Requires CSIMigration feature - flag enabled. + Supports falling back to in-tree AzureDisk plugin for mount operations to + nodes that have the feature disabled or that do not have AzureDisk CSI plugin + installed and configured. Does not support falling back for provision + operations, for those the CSI plugin must be installed and configured. + Requires CSIMigration feature flag enabled. - `CSIMigrationAzureDiskComplete`: Stops registering the Azure-Disk in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Requires CSIMigration and CSIMigrationAzureDisk feature flags enabled and AzureDisk CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the - `InTreePluginAzureDiskUnregister` feature flag which prevents the registration of in-tree AzureDisk plugin. + `InTreePluginAzureDiskUnregister` feature flag which prevents the registration + of in-tree AzureDisk plugin. - `CSIMigrationAzureFile`: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. - Supports falling back to in-tree AzureFile plugin if a node does not have - AzureFile CSI plugin installed and configured. Requires CSIMigration feature - flag enabled. + Supports falling back to in-tree AzureFile plugin for mount operations to + nodes that have the feature disabled or that do not have AzureFile CSI plugin + installed and configured. Does not support falling back for provision + operations, for those the CSI plugin must be installed and configured. + Requires CSIMigration feature flag enabled. - `CSIMigrationAzureFileComplete`: Stops registering the Azure-File in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to @@ -617,37 +710,58 @@ Each feature gate is designed for enabling/disabling a specific feature: of in-tree AzureFile plugin. - `CSIMigrationGCE`: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling - back to in-tree GCE plugin if a node does not have PD CSI plugin installed and - configured. Requires CSIMigration feature flag enabled. + back to in-tree GCE plugin for mount operations to nodes that have the + feature disabled or that do not have PD CSI plugin installed and configured. + Does not support falling back for provision operations, for those the CSI + plugin must be installed and configured. Requires CSIMigration feature flag + enabled. - `CSIMigrationGCEComplete`: Stops registering the GCE-PD in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Requires CSIMigration and CSIMigrationGCE feature flags enabled and PD CSI plugin installed and configured on all nodes in the cluster. This flag has - been deprecated in favor of the `InTreePluginGCEUnregister` feature flag which prevents the registration of in-tree GCE PD plugin. + been deprecated in favor of the `InTreePluginGCEUnregister` feature flag which + prevents the registration of in-tree GCE PD plugin. - `CSIMigrationOpenStack`: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Supports - falling back to in-tree Cinder plugin if a node does not have Cinder CSI - plugin installed and configured. Requires CSIMigration feature flag enabled. + falling back to in-tree Cinder plugin for mount operations to nodes that have + the feature disabled or that do not have Cinder CSI plugin installed and + configured. Does not support falling back for provision operations, for those + the CSI plugin must be installed and configured. Requires CSIMigration + feature flag enabled. - `CSIMigrationOpenStackComplete`: Stops registering the Cinder in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Requires CSIMigration and CSIMigrationOpenStack feature flags enabled and Cinder CSI plugin installed and configured on all nodes in the cluster. This flag has - been deprecated in favor of the `InTreePluginOpenStackUnregister` feature flag which prevents the registration of in-tree openstack cinder plugin. + been deprecated in favor of the `InTreePluginOpenStackUnregister` feature flag + which prevents the registration of in-tree openstack cinder plugin. +- `csiMigrationRBD`: Enables shims and translation logic to route volume + operations from the RBD in-tree plugin to Ceph RBD CSI plugin. Requires + CSIMigration and csiMigrationRBD feature flags enabled and Ceph CSI plugin + installed and configured in the cluster. This flag has been deprecated in + favor of the `InTreePluginRBDUnregister` feature flag which prevents the registration of + in-tree RBD plugin. - `CSIMigrationvSphere`: Enables shims and translation logic to route volume operations - from the vSphere in-tree plugin to vSphere CSI plugin. - Supports falling back to in-tree vSphere plugin if a node does not have vSphere - CSI plugin installed and configured. Requires CSIMigration feature flag enabled. + from the vSphere in-tree plugin to vSphere CSI plugin. Supports falling back + to in-tree vSphere plugin for mount operations to nodes that have the feature + disabled or that do not have vSphere CSI plugin installed and configured. + Does not support falling back for provision operations, for those the CSI + plugin must be installed and configured. Requires CSIMigration feature flag + enabled. - `CSIMigrationvSphereComplete`: Stops registering the vSphere in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Requires CSIMigration and CSIMigrationvSphere feature flags enabled and vSphere CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor - of the `InTreePluginvSphereUnregister` feature flag which prevents the registration of in-tree vsphere plugin. -- `CSINodeInfo`: Enable all logic related to the CSINodeInfo API object in csi.storage.k8s.io. + of the `InTreePluginvSphereUnregister` feature flag which prevents the + registration of in-tree vsphere plugin. +- `CSIMigrationPortworx`: Enables shims and translation logic to route volume operations + from the Portworx in-tree plugin to Portworx CSI plugin. + Requires Portworx CSI driver to be installed and configured in the cluster. +- `CSINodeInfo`: Enable all logic related to the CSINodeInfo API object in `csi.storage.k8s.io`. - `CSIPersistentVolume`: Enable discovering and mounting volumes provisioned through a - [CSI (Container Storage Interface)](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) + [CSI (Container Storage Interface)](https://git.k8s.io/design-proposals-archive/storage/container-storage-interface.md) compatible volume plugin. - `CSIServiceAccountToken`: Enable CSI drivers to receive the pods' service account token that they mount volumes for. See @@ -666,13 +780,19 @@ Each feature gate is designed for enabling/disabling a specific feature: for fsGroups when mounting a volume in a Pod. See [Configure volume permission and ownership change policy for Pods](/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods) for more details. +- `ContextualLogging`: When you enable this feature gate, Kubernetes components that support + contextual logging add extra detail to log output. - `ControllerManagerLeaderMigration`: Enables leader migration for `kube-controller-manager` and `cloud-controller-manager`. - `CronJobControllerV2`: Use an alternative implementation of the {{< glossary_tooltip text="CronJob" term_id="cronjob" >}} controller. Otherwise, version 1 of the same controller is selected. +- `CronJobTimeZone`: Allow the use of the `timeZone` optional field in [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/) - `CustomCPUCFSQuotaPeriod`: Enable nodes to change `cpuCFSQuotaPeriod` in [kubelet config](/docs/tasks/administer-cluster/kubelet-config-file/). +- `CustomResourceValidationExpressions`: Enable expression language validation in CRD + which will validate customer resource based on validation rules written in + the `x-kubernetes-validations` extension. - `CustomPodDNS`: Enable customizing the DNS settings for a Pod using its `dnsConfig` property. Check [Pod's DNS Config](/docs/concepts/services-networking/dns-pod-service/#pods-dns-config) for more details. @@ -686,8 +806,9 @@ Each feature gate is designed for enabling/disabling a specific feature: on resources created from [CustomResourceDefinition](/docs/concepts/extend-kubernetes/api-extension/custom-resources/). - `DaemonSetUpdateSurge`: Enables the DaemonSet workloads to maintain availability during update per node. + See [Perform a Rolling Update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/). - `DefaultPodTopologySpread`: Enables the use of `PodTopologySpread` scheduling plugin to do - [default spreading](/docs/concepts/workloads/pods/pod-topology-spread-constraints/#internal-default-constraints). + [default spreading](/docs/concepts/scheduling-eviction/topology-spread-constraints/#internal-default-constraints). - `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the role of applying `fsGroup` from a Pod's `securityContext` to the driver by passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls. @@ -698,13 +819,16 @@ Each feature gate is designed for enabling/disabling a specific feature: - `DisableCloudProviders`: Disables any functionality in `kube-apiserver`, `kube-controller-manager` and `kubelet` related to the `--cloud-provider` component flag. +- `DisableKubeletCloudCredentialProviders`: Disable the in-tree functionality in kubelet + to authenticate to a cloud provider container registry for image pull credentials. - `DownwardAPIHugePages`: Enables usage of hugepages in [downward API](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information). - `DryRun`: Enable server-side [dry run](/docs/reference/using-api/api-concepts/#dry-run) requests so that validation, merging, and mutation can be tested without committing. - `DynamicAuditing`: Used to enable dynamic auditing before v1.19. -- `DynamicKubeletConfig`: Enable the dynamic configuration of kubelet. See - [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). +- `DynamicKubeletConfig`: Enable the dynamic configuration of kubelet. The + feature is no longer supported outside of supported skew policy. The feature + gate was removed from kubelet in 1.24. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). - `DynamicProvisioningScheduling`: Extend the default scheduler to be aware of volume topology and handle PV provisioning. This feature is superseded by the `VolumeScheduling` feature completely in v1.12. @@ -718,26 +842,27 @@ Each feature gate is designed for enabling/disabling a specific feature: - `EnableEquivalenceClassCache`: Enable the scheduler to cache equivalence of nodes when scheduling Pods. - `EndpointSlice`: Enables EndpointSlices for more scalable and extensible - network endpoints. See [Enabling EndpointSlices](/docs/tasks/administer-cluster/enabling-endpointslices/). + network endpoints. See [Enabling EndpointSlices](/docs/concepts/services-networking/endpoint-slices/). - `EndpointSliceNodeName`: Enables EndpointSlice `nodeName` field. - `EndpointSliceProxying`: When enabled, kube-proxy running on Linux will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See - [Enabling Endpoint Slices](/docs/tasks/administer-cluster/enabling-endpointslices/). + [Enabling Endpoint Slices](/docs/concepts/services-networking/endpoint-slices/). - `EndpointSliceTerminatingCondition`: Enables EndpointSlice `terminating` and `serving` condition fields. - `EphemeralContainers`: Enable the ability to add {{< glossary_tooltip text="ephemeral containers" term_id="ephemeral-container" >}} to running pods. - `EvenPodsSpread`: Enable pods to be scheduled evenly across topology domains. See - [Pod Topology Spread Constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). + [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). - `ExecProbeTimeout`: Ensure kubelet respects exec probe timeouts. This feature gate exists in case any of your existing workloads depend on a now-corrected fault where Kubernetes ignored exec probe timeouts. See [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes). - `ExpandCSIVolumes`: Enable the expanding of CSI volumes. - `ExpandedDNSConfig`: Enable kubelet and kube-apiserver to allow more DNS - search paths and longer list of DNS search paths. See + search paths and longer list of DNS search paths. This feature requires container + runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See [Expanded DNS Configuration](/docs/concepts/services-networking/dns-pod-service/#expanded-dns-configuration). - `ExpandInUsePersistentVolumes`: Enable expanding in-use PVCs. See [Resizing an in-use PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim). @@ -751,7 +876,8 @@ Each feature gate is designed for enabling/disabling a specific feature: host mounts, or containers that are privileged or using specific non-namespaced capabilities (e.g. `MKNODE`, `SYS_MODULE` etc.). This should only be enabled if user namespace remapping is enabled in the Docker daemon. -- `ExternalPolicyForExternalIP`: Fix a bug where ExternalTrafficPolicy is not applied to Service ExternalIPs. +- `ExternalPolicyForExternalIP`: Fix a bug where ExternalTrafficPolicy is not + applied to Service ExternalIPs. - `GCERegionalPersistentDisk`: Enable the regional PD feature on GCE. - `GenericEphemeralVolume`: Enables ephemeral, inline volumes that support all features of normal volumes (can be provided by third-party storage vendors, storage capacity tracking, @@ -762,6 +888,14 @@ Each feature gate is designed for enabling/disabling a specific feature: and gracefully terminate pods running on the node. See [Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) for more details. +- `GracefulNodeShutdownBasedOnPodPriority`: Enables the kubelet to check Pod priorities + when shutting down a node gracefully. +- `GRPCContainerProbe`: Enables the gRPC probe method for {Liveness,Readiness,Startup}Probe. + See [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe). +- `HonorPVReclaimPolicy`: Honor persistent volume reclaim policy when it is `Delete` irrespective of PV-PVC deletion ordering. + For more details, check the + [PersistentVolume deletion protection finalizer](/docs/concepts/storage/persistent-volumes/#persistentvolume-deletion-protection-finalizer) + documentation. - `HPAContainerMetrics`: Enable the `HorizontalPodAutoscaler` to scale based on metrics from individual containers in target pods. - `HPAScaleToZero`: Enables setting `minReplicas` to 0 for `HorizontalPodAutoscaler` @@ -773,8 +907,19 @@ Each feature gate is designed for enabling/disabling a specific feature: - `HyperVContainer`: Enable [Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container) for Windows containers. +- `IdentifyPodOS`: Allows the Pod OS field to be specified. This helps in identifying + the OS of the pod authoritatively during the API server admission time. + In Kubernetes {{< skew currentVersion >}}, the allowed values for the `pod.spec.os.name` + are `windows` and `linux`. - `ImmutableEphemeralVolumes`: Allows for marking individual Secrets and ConfigMaps as immutable for better safety and performance. +- `IndexedJob`: Allows the [Job](/docs/concepts/workloads/controllers/job/) + controller to manage Pod completions per completion index. +- `IngressClassNamespacedParams`: Allow namespace-scoped parameters reference in + `IngressClass` resource. This feature adds two fields - `Scope` and `Namespace` + to `IngressClass.spec.parameters`. +- `Initializers`: Allow asynchronous coordination of object creation using the + Initializers admission plugin. - `InTreePluginAWSUnregister`: Stops registering the aws-ebs in-tree plugin in kubelet and volume controllers. - `InTreePluginAzureDiskUnregister`: Stops registering the azuredisk in-tree plugin in kubelet @@ -785,17 +930,21 @@ Each feature gate is designed for enabling/disabling a specific feature: and volume controllers. - `InTreePluginOpenStackUnregister`: Stops registering the OpenStack cinder in-tree plugin in kubelet and volume controllers. +- `InTreePluginPortworxUnregister`: Stops registering the Portworx in-tree plugin in kubelet + and volume controllers. +- `InTreePluginRBDUnregister`: Stops registering the RBD in-tree plugin in kubelet + and volume controllers. - `InTreePluginvSphereUnregister`: Stops registering the vSphere in-tree plugin in kubelet and volume controllers. -- `IndexedJob`: Allows the [Job](/docs/concepts/workloads/controllers/job/) - controller to manage Pod completions per completion index. -- `IngressClassNamespacedParams`: Allow namespace-scoped parameters reference in - `IngressClass` resource. This feature adds two fields - `Scope` and `Namespace` - to `IngressClass.spec.parameters`. -- `Initializers`: Allow asynchronous coordination of object creation using the - Initializers admission plugin. - `IPv6DualStack`: Enable [dual stack](/docs/concepts/services-networking/dual-stack/) support for IPv6. +- `JobMutableNodeSchedulingDirectives`: Allows updating node scheduling directives in + the pod template of [Job](/docs/concepts/workloads/controllers/job). +- `JobReadyPods`: Enables tracking the number of Pods that have a `Ready` + [condition](/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions). + The count of `Ready` pods is recorded in the + [status](/docs/reference/kubernetes-api/workload-resources/job-v1/#JobStatus) + of a [Job](/docs/concepts/workloads/controllers/job) status. - `JobTrackingWithFinalizers`: Enables tracking [Job](/docs/concepts/workloads/controllers/job) completions without relying on Pods remaining in the cluster indefinitely. The Job controller uses Pod finalizers and a field in the Job status to keep @@ -804,20 +953,26 @@ Each feature gate is designed for enabling/disabling a specific feature: a file specified using a config file. See [setting kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/) for more details. -- `KubeletCredentialProviders`: Enable kubelet exec credential providers for image pull credentials. -- `KubeletInUserNamespace`: Enables support for running kubelet in a {{}}. +- `KubeletCredentialProviders`: Enable kubelet exec credential providers for + image pull credentials. +- `KubeletInUserNamespace`: Enables support for running kubelet in a + {{}}. See [Running Kubernetes Node Components as a Non-root User](/docs/tasks/administer-cluster/kubelet-in-userns/). - `KubeletPluginsWatcher`: Enable probe-based plugin watcher utility to enable kubelet to discover plugins such as [CSI volume drivers](/docs/concepts/storage/volumes/#csi). - `KubeletPodResources`: Enable the kubelet's pod resources gRPC endpoint. See [Support Device Monitoring](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/606-compute-device-assignment/README.md) for more details. -- `KubeletPodResourcesGetAllocatable`: Enable the kubelet's pod resources `GetAllocatableResources` functionality. - This API augments the [resource allocation reporting](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources) - with informations about the allocatable resources, enabling clients to properly track the free compute resources on a node. +- `KubeletPodResourcesGetAllocatable`: Enable the kubelet's pod resources + `GetAllocatableResources` functionality. This API augments the + [resource allocation reporting](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources) + with informations about the allocatable resources, enabling clients to properly + track the free compute resources on a node. - `LegacyNodeRoleBehavior`: When disabled, legacy behavior in service load balancers and node disruption will ignore the `node-role.kubernetes.io/master` label in favor of the feature-specific labels provided by `NodeDisruptionExclusion` and `ServiceNodeExclusion`. +- `LegacyServiceAccountTokenNoAutoGeneration`: Stop auto-generation of Secret-based + [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens). - `LocalStorageCapacityIsolation`: Enable the consumption of [local ephemeral storage](/docs/concepts/configuration/manage-resources-containers/) and also the `sizeLimit` property of an @@ -831,39 +986,58 @@ Each feature gate is designed for enabling/disabling a specific feature: filesystem walk for better performance and accuracy. - `LogarithmicScaleDown`: Enable semi-random selection of pods to evict on controller scaledown based on logarithmic bucketing of pod timestamps. +- `MaxUnavailableStatefulSet`: Enables setting the `maxUnavailable` field for the + [rolling update strategy](/docs/concepts/workloads/controllers/statefulset/#rolling-updates) + of a StatefulSet. The field specifies the maximum number of Pods + that can be unavailable during the update. - `MemoryManager`: Allows setting memory affinity for a container based on NUMA topology. -- `MemoryQoS`: Enable memory protection and usage throttle on pod / container using cgroup v2 memory controller. +- `MemoryQoS`: Enable memory protection and usage throttle on pod / container using + cgroup v2 memory controller. +- `MinDomainsInPodTopologySpread`: Enable `minDomains` in Pod + [topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). - `MixedProtocolLBService`: Enable using different protocols in the same `LoadBalancer` type Service instance. - `MountContainers`: Enable using utility containers on host as the volume mounter. - `MountPropagation`: Enable sharing volume mounted by one container to other containers or pods. For more details, please see [mount propagation](/docs/concepts/storage/volumes/#mount-propagation). -- `NamespaceDefaultLabelName`: Configure the API Server to set an immutable {{< glossary_tooltip text="label" term_id="label" >}} - `kubernetes.io/metadata.name` on all namespaces, containing the namespace name. -- `NetworkPolicyEndPort`: Enable use of the field `endPort` in NetworkPolicy objects, allowing the selection of a port range instead of a single port. +- `NamespaceDefaultLabelName`: Configure the API Server to set an immutable + {{< glossary_tooltip text="label" term_id="label" >}} `kubernetes.io/metadata.name` + on all namespaces, containing the namespace name. +- `NetworkPolicyEndPort`: Enable use of the field `endPort` in NetworkPolicy objects, + allowing the selection of a port range instead of a single port. +- `NetworkPolicyStatus`: Enable the `status` subresource for NetworkPolicy objects. - `NodeDisruptionExclusion`: Enable use of the Node label `node.kubernetes.io/exclude-disruption` which prevents nodes from being evacuated during zone failures. - `NodeLease`: Enable the new Lease API to report node heartbeats, which could be used as a node health signal. +- `NodeOutOfServiceVolumeDetach`: When a Node is marked out-of-service using the + `node.kubernetes.io/out-of-service` taint, Pods on the node will be forcefully deleted + if they can not tolerate this taint, and the volume detach operations for Pods terminating + on the node will happen immediately. The deleted Pods can recover quickly on different nodes. - `NodeSwap`: Enable the kubelet to allocate swap memory for Kubernetes workloads on a node. Must be used with `KubeletConfiguration.failSwapOn` set to false. For more details, please see [swap memory](/docs/concepts/architecture/nodes/#swap-memory) - `NonPreemptingPriority`: Enable `preemptionPolicy` field for PriorityClass and Pod. -- `PVCProtection`: Enable the prevention of a PersistentVolumeClaim (PVC) from - being deleted when it is still used by any Pod. +- `OpenAPIEnums`: Enables populating "enum" fields of OpenAPI schemas in the + spec returned from the API server. +- `OpenAPIV3`: Enables the API server to publish OpenAPI v3. - `PodDeletionCost`: Enable the [Pod Deletion Cost](/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost) feature which allows users to influence ReplicaSet downscaling order. - `PersistentLocalVolumes`: Enable the usage of `local` volume type in Pods. Pod affinity has to be specified if requesting a `local` volume. +- `PodAndContainerStatsFromCRI`: Configure the kubelet to gather container and + pod stats from the CRI container runtime rather than gathering them from cAdvisor. - `PodDisruptionBudget`: Enable the [PodDisruptionBudget](/docs/tasks/run-application/configure-pdb/) feature. -- `PodAffinityNamespaceSelector`: Enable the [Pod Affinity Namespace Selector](/docs/concepts/scheduling-eviction/assign-pod-node/#namespace-selector) - and [CrossNamespacePodAffinity](/docs/concepts/policy/resource-quotas/#cross-namespace-pod-affinity-quota) quota scope features. +- `PodAffinityNamespaceSelector`: Enable the + [Pod Affinity Namespace Selector](/docs/concepts/scheduling-eviction/assign-pod-node/#namespace-selector) + and [CrossNamespacePodAffinity](/docs/concepts/policy/resource-quotas/#cross-namespace-pod-affinity-quota) + quota scope features. - `PodOverhead`: Enable the [PodOverhead](/docs/concepts/scheduling-eviction/pod-overhead/) feature to account for pod overheads. - `PodPriority`: Enable the descheduling and preemption of Pods based on their [priorities](/docs/concepts/scheduling-eviction/pod-priority-preemption/). - `PodReadinessGates`: Enable the setting of `PodReadinessGate` field for extending - Pod readiness evaluation. See [Pod readiness gate](/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate) + Pod readiness evaluation. See [Pod readiness gate](/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate) for more details. - `PodSecurity`: Enables the `PodSecurity` admission plugin. - `PodShareProcessNamespace`: Enable the setting of `shareProcessNamespace` in a Pod for sharing @@ -874,21 +1048,30 @@ Each feature gate is designed for enabling/disabling a specific feature: the cluster. - `ProbeTerminationGracePeriod`: Enable [setting probe-level `terminationGracePeriodSeconds`](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#probe-level-terminationgraceperiodseconds) - on pods. See the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2238-liveness-probe-grace-period) for more details. + on pods. See the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2238-liveness-probe-grace-period) + for more details. - `ProcMountType`: Enables control over the type proc mounts for containers by setting the `procMount` field of a SecurityContext. - `ProxyTerminatingEndpoints`: Enable the kube-proxy to handle terminating endpoints when `ExternalTrafficPolicy=Local`. +- `PVCProtection`: Enable the prevention of a PersistentVolumeClaim (PVC) from + being deleted when it is still used by any Pod. - `QOSReserved`: Allows resource reservations at the QoS level preventing pods at lower QoS levels from bursting into resources requested at higher QoS levels (memory only for now). - `ReadWriteOncePod`: Enables the usage of `ReadWriteOncePod` PersistentVolume access mode. +- `RecoverVolumeExpansionFailure`: Enables users to edit their PVCs to smaller + sizes so as they can recover from previously issued volume expansion failures. + See [Recovering from Failure when Expanding Volumes](/docs/concepts/storage/persistent-volumes/#recovering-from-failure-when-expanding-volumes) + for more details. - `RemainingItemCount`: Allow the API servers to show a count of remaining items in the response to a [chunking list request](/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks). -- `RemoveSelfLink`: Deprecates and removes `selfLink` from ObjectMeta and - ListMeta. +- `RemoveSelfLink`: Sets the `.metadata.selfLink` field to blank (empty string) for all + objects and collections. This field has been deprecated since the Kubernetes v1.16 + release. When this feature is enabled, the `.metadata.selfLink` field remains part of + the Kubernetes API, but is always unset. - `RequestManagement`: Enables managing request concurrency with prioritization and fairness at each API server. Deprecated by `APIPriorityAndFairness` since 1.17. - `ResourceLimitsPriorityFunction`: Enable a scheduler priority function that @@ -903,9 +1086,10 @@ Each feature gate is designed for enabling/disabling a specific feature: [Bound Service Account Tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md) for more details. - `RotateKubeletClientCertificate`: Enable the rotation of the client TLS certificate on the kubelet. - See [kubelet configuration](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#kubelet-configuration) for more details. + See [kubelet configuration](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#kubelet-configuration) + for more details. - `RotateKubeletServerCertificate`: Enable the rotation of the server TLS certificate on the kubelet. - See [kubelet configuration](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#kubelet-configuration) + See [kubelet configuration](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#kubelet-configuration) for more details. - `RunAsGroup`: Enable control over the primary group ID set on the init processes of containers. @@ -915,12 +1099,16 @@ Each feature gate is designed for enabling/disabling a specific feature: instead of the DaemonSet controller. - `SCTPSupport`: Enables the _SCTP_ `protocol` value in Pod, Service, Endpoints, EndpointSlice, and NetworkPolicy definitions. -- `SeccompDefault`: Enables the use of `RuntimeDefault` as the default seccomp profile for all workloads. +- `SeccompDefault`: Enables the use of `RuntimeDefault` as the default seccomp profile + for all workloads. The seccomp profile is specified in the `securityContext` of a Pod and/or a Container. - `SelectorIndex`: Allows label and field based indexes in API server watch cache to accelerate list operations. - `ServerSideApply`: Enables the [Sever Side Apply (SSA)](/docs/reference/using-api/server-side-apply/) feature on the API Server. +- `ServerSideFieldValidation`: Enables server-side field validation. This means the validation + of resource schema is performed at the API server side rather than the client side + (for example, the `kubectl create` or `kubectl apply` command line). - `ServiceAccountIssuerDiscovery`: Enable OIDC discovery endpoints (issuer and JWKS URLs) for the service account issuer in the API server. See [Configure Service Accounts for Pods](/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery) @@ -929,7 +1117,8 @@ Each feature gate is designed for enabling/disabling a specific feature: - `ServiceInternalTrafficPolicy`: Enables the `internalTrafficPolicy` field on Services - `ServiceLBNodePortControl`: Enables the `allocateLoadBalancerNodePorts` field on Services. - `ServiceLoadBalancerClass`: Enables the `loadBalancerClass` field on Services. See - [Specifying class of load balancer implementation](/docs/concepts/services-networking/service/#load-balancer-class) for more details. + [Specifying class of load balancer implementation](/docs/concepts/services-networking/service/#load-balancer-class) + for more details. - `ServiceLoadBalancerFinalizer`: Enable finalizer protection for Service load balancers. - `ServiceNodeExclusion`: Enable the exclusion of nodes from load balancers created by a cloud provider. A node is eligible for exclusion if labelled with @@ -938,6 +1127,12 @@ Each feature gate is designed for enabling/disabling a specific feature: topology of the cluster. See [ServiceTopology](/docs/concepts/services-networking/service-topology/) for more details. +- `ServiceIPStaticSubrange`: Enables a strategy for Services ClusterIP allocations, whereby the + ClusterIP range is subdivided. Dynamic allocated ClusterIP addresses will be allocated preferently + from the upper range allowing users to assign static ClusterIPs from the lower range with a low + risk of collision. See + [Avoiding collisions](/docs/concepts/services-networking/service/#avoiding-collisions) + for more details. - `SetHostnameAsFQDN`: Enable the ability of setting Fully Qualified Domain Name(FQDN) as the hostname of a pod. See [Pod's `setHostnameAsFQDN` field](/docs/concepts/services-networking/dns-pod-service/#pod-sethostnameasfqdn-field). @@ -1011,7 +1206,7 @@ Each feature gate is designed for enabling/disabling a specific feature: - `WindowsEndpointSliceProxying`: When enabled, kube-proxy running on Windows will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See - [Enabling Endpoint Slices](/docs/tasks/administer-cluster/enabling-endpointslices/). + [Enabling Endpoint Slices](/docs/concepts/services-networking/endpoint-slices/). - `WindowsGMSA`: Enables passing of GMSA credential specs from pods to container runtimes. - `WindowsHostProcessContainers`: Enables support for Windows HostProcess containers. - `WindowsRunAsUserName` : Enable support for running applications in Windows containers @@ -1024,3 +1219,8 @@ Each feature gate is designed for enabling/disabling a specific feature: * The [deprecation policy](/docs/reference/using-api/deprecation-policy/) for Kubernetes explains the project's approach to removing features and components. +* Since Kubernetes 1.24, new beta APIs are not enabled by default. When enabling a beta + feature, you will also need to enable any associated API resources. + For example, to enable a particular resource like + `storage.k8s.io/v1beta1/csistoragecapacities`, set `--runtime-config=storage.k8s.io/v1beta1/csistoragecapacities`. + See [API Versioning](/docs/reference/using-api/#api-versioning) for more details on the command line flags. diff --git a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md index 77b354dc70..ad2dc545b7 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md @@ -39,13 +39,6 @@ kube-apiserver [flags] - ---add-dir-header - - -

If true, adds the file directory to the header of the log messages

- - --admission-control-config-file string @@ -74,13 +67,6 @@ kube-apiserver [flags]

If true, allow privileged containers. [default=false]

- ---alsologtostderr - - -

log to standard error as well as files

- - --anonymous-auth     Default: true @@ -95,13 +81,6 @@ kube-apiserver [flags]

Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

- ---apiserver-count int     Default: 1 - - -

The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.)

- - --audit-log-batch-buffer-size int     Default: 10000 @@ -169,7 +148,7 @@ kube-apiserver [flags] --audit-log-maxbackup int -

The maximum number of old audit log files to retain.

+

The maximum number of old audit log files to retain. Setting a value of 0 will mean there's no restriction on the number of files.

@@ -540,7 +519,7 @@ kube-apiserver [flags] --endpoint-reconciler-type string     Default: "lease" -

Use an endpoint reconciler (master-count, lease, none)

+

Use an endpoint reconciler (master-count, lease, none) master-count is deprecated, and will be removed in a future version.

@@ -620,13 +599,6 @@ kube-apiserver [flags]

Amount of time to retain events.

- ---experimental-logging-sanitization - - -

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

- - --external-hostname string @@ -638,7 +610,7 @@ kube-apiserver [flags] --feature-gates <comma-separated 'key=True|False' pairs> -

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (ALPHA - default=false)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CSRDuration=true|false (BETA - default=true)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DefaultPodTopologySpread=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=false)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (BETA - default=true)
IngressClassNamespacedParams=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NetworkPolicyEndPort=true|false (BETA - default=true)
NodeSwap=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (BETA - default=true)
PodDeletionCost=true|false (BETA - default=true)
PodOverhead=true|false (BETA - default=true)
PodSecurity=true|false (ALPHA - default=false)
PreferNominatedNode=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
ServiceLBNodePortControl=true|false (BETA - default=true)
ServiceLoadBalancerClass=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetMinReadySeconds=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (BETA - default=true)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (ALPHA - default=false)

+

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=true)
CSIMigrationAzureFile=true|false (BETA - default=true)
CSIMigrationGCE=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (ALPHA - default=false)
CSIMigrationRBD=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIVolumeHealth=true|false (ALPHA - default=false)
ContextualLogging=true|false (ALPHA - default=false)
CronJobTimeZone=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (ALPHA - default=false)
IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
JobReadyPods=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (BETA - default=false)
KubeletCredentialProviders=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
NetworkPolicyStatus=true|false (ALPHA - default=false)
NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
NodeSwap=true|false (ALPHA - default=false)
OpenAPIEnums=true|false (BETA - default=true)
OpenAPIV3=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServerSideFieldValidation=true|false (ALPHA - default=false)
ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
StatefulSetMinReadySeconds=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (BETA - default=true)

@@ -732,34 +704,6 @@ kube-apiserver [flags]

This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.

- ---log-backtrace-at <a string in the form 'file:N'>     Default: :0 - - -

when logging hits line file:N, emit a stack trace

- - - ---log-dir string - - -

If non-empty, write log files in this directory

- - - ---log-file string - - -

If non-empty, use this log file

- - - ---log-file-max-size uint     Default: 1800 - - -

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

- - --log-flush-frequency duration     Default: 5s @@ -771,14 +715,7 @@ kube-apiserver [flags] --logging-format string     Default: "text" -

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

- - - ---logtostderr     Default: true - - -

log to standard error instead of files

+

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule.
Non-default choices are currently alpha and subject to change without warning.

@@ -862,7 +799,7 @@ kube-apiserver [flags] --oidc-signing-algs strings     Default: "RS256" -

Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a 'alg' header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.

+

Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a supported 'alg' header values are: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, PS512. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.

@@ -879,13 +816,6 @@ kube-apiserver [flags]

If provided, all usernames will be prefixed with this value. If not provided, username claims other than 'email' are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value '-'.

- ---one-output - - -

If true, only write logs to their native severity level (vs also writing to each lower severity level)

- - --permit-address-sharing @@ -995,14 +925,14 @@ kube-apiserver [flags] --service-account-jwks-uri string -

Overrides the URI for the JSON Web Key Set in the discovery doc served at /.well-known/openid-configuration. This flag is useful if the discovery docand key set are served to relying parties from a URL other than the API server's external (as auto-detected or overridden with external-hostname). Only valid if the ServiceAccountIssuerDiscovery feature gate is enabled.

+

Overrides the URI for the JSON Web Key Set in the discovery doc served at /.well-known/openid-configuration. This flag is useful if the discovery docand key set are served to relying parties from a URL other than the API server's external (as auto-detected or overridden with external-hostname).

--service-account-key-file strings -

File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key is provided

+

File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key-file is provided

@@ -1055,24 +985,10 @@ kube-apiserver [flags] ---skip-headers +--shutdown-send-retry-after -

If true, avoid header prefixes in the log messages

- - - ---skip-log-headers - - -

If true, avoid headers when opening log files

- - - ---stderrthreshold int     Default: 2 - - -

logs at or above this threshold go to stderr

+

If true the HTTP Server will continue listening until all non long running request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header, in addition 'Connection: close' response header is set in order to tear down the TCP connection when idle.

@@ -1107,7 +1023,7 @@ kube-apiserver [flags] --tls-cipher-suites strings -

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

+

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

@@ -1160,10 +1076,10 @@ kube-apiserver [flags] ---vmodule <comma-separated 'pattern=N' settings> +--vmodule pattern=N,... -

comma-separated list of pattern=N settings for file-filtered logging

+

comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md index a8389c69f0..f0d0ee4e98 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md @@ -43,13 +43,6 @@ kube-controller-manager [flags] - ---add-dir-header - - -

If true, adds the file directory to the header of the log messages

- - --allocate-node-cidrs @@ -64,13 +57,6 @@ kube-controller-manager [flags]

The map from metric-label to value allow-list of this label. The key's format is <MetricName>,<LabelName>. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.

- ---alsologtostderr - - -

log to standard error as well as files

- - --attach-detach-reconcile-sync-period duration     Default: 1m0s @@ -288,6 +274,13 @@ kube-controller-manager [flags]

The number of endpoint syncing operations that will be done concurrently. Larger number = faster endpoint updating, but more CPU (and network) load

+ +--concurrent-ephemeralvolume-syncs int32     Default: 5 + + +

The number of ephemeral volume syncing operations that will be done concurrently. Larger number = faster ephemeral volume updating, but more CPU (and network) load

+ + --concurrent-gc-syncs int32     Default: 20 @@ -386,13 +379,6 @@ kube-controller-manager [flags]

A list of controllers to enable. '*' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.
All controllers: attachdetach, bootstrapsigner, cloud-node-lifecycle, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, endpointslice, endpointslicemirroring, ephemeral-volume, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-publisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
Disabled-by-default controllers: bootstrapsigner, tokencleaner

- ---deployment-controller-sync-period duration     Default: 30s - - -

Period for syncing the deployments.

- - --disable-attach-detach-reconcile-sync @@ -456,13 +442,6 @@ kube-controller-manager [flags]

The length of endpoint slice updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

- ---experimental-logging-sanitization - - -

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

- - --external-cloud-volume-plugin string @@ -474,7 +453,7 @@ kube-controller-manager [flags] --feature-gates <comma-separated 'key=True|False' pairs> -

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (ALPHA - default=false)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CSRDuration=true|false (BETA - default=true)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DefaultPodTopologySpread=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=false)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (BETA - default=true)
IngressClassNamespacedParams=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NetworkPolicyEndPort=true|false (BETA - default=true)
NodeSwap=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (BETA - default=true)
PodDeletionCost=true|false (BETA - default=true)
PodOverhead=true|false (BETA - default=true)
PodSecurity=true|false (ALPHA - default=false)
PreferNominatedNode=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
ServiceLBNodePortControl=true|false (BETA - default=true)
ServiceLoadBalancerClass=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetMinReadySeconds=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (BETA - default=true)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (ALPHA - default=false)

+

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=true)
CSIMigrationAzureFile=true|false (BETA - default=true)
CSIMigrationGCE=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (ALPHA - default=false)
CSIMigrationRBD=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIVolumeHealth=true|false (ALPHA - default=false)
ContextualLogging=true|false (ALPHA - default=false)
CronJobTimeZone=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (ALPHA - default=false)
IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
JobReadyPods=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (BETA - default=false)
KubeletCredentialProviders=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
NetworkPolicyStatus=true|false (ALPHA - default=false)
NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
NodeSwap=true|false (ALPHA - default=false)
OpenAPIEnums=true|false (BETA - default=true)
OpenAPIV3=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServerSideFieldValidation=true|false (ALPHA - default=false)
ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
StatefulSetMinReadySeconds=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (BETA - default=true)

@@ -593,7 +572,7 @@ kube-controller-manager [flags] --leader-elect-resource-lock string     Default: "leases" -

The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

+

The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'.

@@ -624,34 +603,6 @@ kube-controller-manager [flags]

Path to the config file for controller leader migration, or empty to use the value that reflects default configuration of the controller manager. The config file should be of type LeaderMigrationConfiguration, group controllermanager.config.k8s.io, version v1alpha1.

- ---log-backtrace-at <a string in the form 'file:N'>     Default: :0 - - -

when logging hits line file:N, emit a stack trace

- - - ---log-dir string - - -

If non-empty, write log files in this directory

- - - ---log-file string - - -

If non-empty, use this log file

- - - ---log-file-max-size uint     Default: 1800 - - -

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

- - --log-flush-frequency duration     Default: 5s @@ -663,14 +614,7 @@ kube-controller-manager [flags] --logging-format string     Default: "text" -

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

- - - ---logtostderr     Default: true - - -

log to standard error instead of files

+

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule.
Non-default choices are currently alpha and subject to change without warning.

@@ -771,13 +715,6 @@ kube-controller-manager [flags]

Amount of time which we allow starting Node to be unresponsive before marking it unhealthy.

- ---one-output - - -

If true, only write logs to their native severity level (vs also writing to each lower severity level)

- - --permit-address-sharing @@ -946,27 +883,6 @@ kube-controller-manager [flags]

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

- ---skip-headers - - -

If true, avoid header prefixes in the log messages

- - - ---skip-log-headers - - -

If true, avoid headers when opening log files

- - - ---stderrthreshold int     Default: 2 - - -

logs at or above this threshold go to stderr

- - --terminated-pod-gc-threshold int32     Default: 12500 @@ -985,7 +901,7 @@ kube-controller-manager [flags] --tls-cipher-suites strings -

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

+

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

@@ -1038,10 +954,10 @@ kube-controller-manager [flags] ---vmodule <comma-separated 'pattern=N' settings> +--vmodule pattern=N,... -

comma-separated list of pattern=N settings for file-filtered logging

+

comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md index 3306668093..34ced450e0 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md @@ -43,7 +43,7 @@ kube-proxy [flags] ---add-dir-header +--add_dir_header

If true, adds the file directory to the header of the log messages

@@ -56,18 +56,11 @@ kube-proxy [flags]

log to standard error as well as files

- ---azure-container-registry-config string - - -

Path to the file containing Azure container registry configuration information.

- - --bind-address string     Default: 0.0.0.0 -

The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)

+

The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces). This parameter is ignored if a config file is specified by --config.

@@ -78,7 +71,7 @@ kube-proxy [flags] ---boot-id-file string     Default: "/proc/sys/kernel/random/boot_id" +--boot_id_file string     Default: "/proc/sys/kernel/random/boot_id"

Comma-separated list of files to check for boot-id. Use the first one that exists.

@@ -91,25 +84,11 @@ kube-proxy [flags]

If true cleanup iptables and ipvs rules and exit.

- ---cloud-provider-gce-l7lb-src-cidrs cidrs     Default: 130.211.0.0/22,35.191.0.0/16 - - -

CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks

- - - ---cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 - - -

CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks

- - --cluster-cidr string -

The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead

+

The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead. For dual-stack clusters, a comma-separated list is accepted with at least one CIDR per IP family (IPv4 and IPv6). This parameter is ignored if a config file is specified by --config.

@@ -154,39 +133,25 @@ kube-proxy [flags]

Idle timeout for established TCP connections (0 to leave as-is)

- ---default-not-ready-toleration-seconds int     Default: 300 - - -

Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.

- - - ---default-unreachable-toleration-seconds int     Default: 300 - - -

Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.

- - --detect-local-mode LocalMode -

Mode to use to detect local traffic

+

Mode to use to detect local traffic. This parameter is ignored if a config file is specified by --config.

--feature-gates <comma-separated 'key=True|False' pairs> -

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (ALPHA - default=false)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CSRDuration=true|false (BETA - default=true)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DefaultPodTopologySpread=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=false)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (BETA - default=true)
IngressClassNamespacedParams=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NetworkPolicyEndPort=true|false (BETA - default=true)
NodeSwap=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (BETA - default=true)
PodDeletionCost=true|false (BETA - default=true)
PodOverhead=true|false (BETA - default=true)
PodSecurity=true|false (ALPHA - default=false)
PreferNominatedNode=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
ServiceLBNodePortControl=true|false (BETA - default=true)
ServiceLoadBalancerClass=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetMinReadySeconds=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (BETA - default=true)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (ALPHA - default=false)

+

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=true)
CSIMigrationAzureFile=true|false (BETA - default=true)
CSIMigrationGCE=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (ALPHA - default=false)
CSIMigrationRBD=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIVolumeHealth=true|false (ALPHA - default=false)
CronJobTimeZone=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (ALPHA - default=false)
IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
JobReadyPods=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (BETA - default=false)
KubeletCredentialProviders=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
NetworkPolicyStatus=true|false (ALPHA - default=false)
NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
NodeSwap=true|false (ALPHA - default=false)
OpenAPIEnums=true|false (BETA - default=true)
OpenAPIV3=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServerSideFieldValidation=true|false (ALPHA - default=false)
ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
StatefulSetMinReadySeconds=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (BETA - default=true)This parameter is ignored if a config file is specified by --config.

--healthz-bind-address ipport     Default: 0.0.0.0:10256 -

The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable.

+

The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.

@@ -309,40 +274,33 @@ kube-proxy [flags] ---log-backtrace-at <a string in the form 'file:N'>     Default: :0 +--log_backtrace_at <a string in the form 'file:N'>     Default: :0

when logging hits line file:N, emit a stack trace

---log-dir string +--log_dir string

If non-empty, write log files in this directory

---log-file string +--log_file string

If non-empty, use this log file

---log-file-max-size uint     Default: 1800 +--log_file_max_size uint     Default: 1800

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

- ---log-flush-frequency duration     Default: 5s - - -

Maximum number of seconds between log flushes

- - --logtostderr     Default: true @@ -351,7 +309,7 @@ kube-proxy [flags] ---machine-id-file string     Default: "/etc/machine-id,/var/lib/dbus/machine-id" +--machine_id_file string     Default: "/etc/machine-id,/var/lib/dbus/machine-id"

Comma-separated list of files to check for machine-id. Use the first one that exists.

@@ -375,18 +333,18 @@ kube-proxy [flags] --metrics-bind-address ipport     Default: 127.0.0.1:10249 -

The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable.

+

The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.

--nodeport-addresses strings -

A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses.

+

A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses. This parameter is ignored if a config file is specified by --config.

---one-output +--one_output

If true, only write logs to their native severity level (vs also writing to each lower severity level)

@@ -396,21 +354,35 @@ kube-proxy [flags] --oom-score-adj int32     Default: -999 -

The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]

+

The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]. This parameter is ignored if a config file is specified by --config.

+ + + +--pod-bridge-interface string + + +

A bridge interface name in the cluster. Kube-proxy considers traffic as local if originating from an interface which matches the value. This argument should be set if DetectLocalMode is set to BridgeInterface.

+ + + +--pod-interface-name-prefix string + + +

An interface prefix in the cluster. Kube-proxy considers traffic as local if originating from interfaces that match the given prefix. This argument should be set if DetectLocalMode is set to InterfaceNamePrefix.

--profiling -

If true enables profiling via web interface on /debug/pprof handler.

+

If true enables profiling via web interface on /debug/pprof handler. This parameter is ignored if a config file is specified by --config.

--proxy-mode ProxyMode -

Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' or 'kernelspace' (windows). If blank, use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.

+

Which proxy mode to use: 'iptables' (Linux-only), 'ipvs' (Linux-only), 'kernelspace' (Windows-only), or 'userspace' (Linux/Windows, deprecated). The default value is 'iptables' on Linux and 'userspace' on Windows.This parameter is ignored if a config file is specified by --config.

@@ -424,18 +396,18 @@ kube-proxy [flags] --show-hidden-metrics-for-version string -

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

+

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.This parameter is ignored if a config file is specified by --config.

---skip-headers +--skip_headers

If true, avoid header prefixes in the log messages

---skip-log-headers +--skip_log_headers

If true, avoid headers when opening log files

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md index 621bac8aa2..84d1d84b15 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md @@ -27,7 +27,7 @@ each Pod in the scheduling queue according to constraints and available resources. The scheduler then ranks each valid Node and binds the Pod to a suitable Node. Multiple different schedulers may be used within a cluster; kube-scheduler is the reference implementation. -See [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/) +See [scheduling](/docs/concepts/scheduling-eviction/) for more information about scheduling and the kube-scheduler component. ``` @@ -43,20 +43,6 @@ kube-scheduler [flags] - ---add-dir-header - - -

If true, adds the file directory to the header of the log messages

- - - ---address string - - -

DEPRECATED: the IP address on which to listen for the --port port (set to 0.0.0.0 or :: for listening in all interfaces and IP families). See --bind-address instead. This parameter is ignored if a config file is specified in --config.

- - --allow-metric-labels stringToString     Default: [] @@ -64,13 +50,6 @@ kube-scheduler [flags]

The map from metric-label to value allow-list of this label. The key's format is <MetricName>,<LabelName>. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.

- ---alsologtostderr - - -

log to standard error as well as files

- - --authentication-kubeconfig string @@ -159,11 +138,11 @@ kube-scheduler [flags] --config string -

The path to the configuration file. The following flags can overwrite fields in this file:
--policy-config-file
--policy-configmap
--policy-configmap-namespace

+

The path to the configuration file.

---contention-profiling +--contention-profiling     Default: true

DEPRECATED: enable lock contention profiling, if profiling is enabled. This parameter is ignored if a config file is specified in --config.

@@ -176,18 +155,11 @@ kube-scheduler [flags]

This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.

- ---experimental-logging-sanitization - - -

[Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

- - --feature-gates <comma-separated 'key=True|False' pairs> -

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (ALPHA - default=false)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=false)
CSIMigrationAzureDisk=true|false (BETA - default=false)
CSIMigrationAzureFile=true|false (BETA - default=false)
CSIMigrationGCE=true|false (BETA - default=false)
CSIMigrationOpenStack=true|false (BETA - default=true)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIStorageCapacity=true|false (BETA - default=true)
CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CSRDuration=true|false (BETA - default=true)
ConfigurableFSGroupPolicy=true|false (BETA - default=true)
ControllerManagerLeaderMigration=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DefaultPodTopologySpread=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=false)
EfficientWatchResumption=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (ALPHA - default=false)
ExpandCSIVolumes=true|false (BETA - default=true)
ExpandInUsePersistentVolumes=true|false (BETA - default=true)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GenericEphemeralVolume=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
IPv6DualStack=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
IndexedJob=true|false (BETA - default=true)
IngressClassNamespacedParams=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (ALPHA - default=false)
KubeletCredentialProviders=true|false (ALPHA - default=false)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (ALPHA - default=false)
NetworkPolicyEndPort=true|false (BETA - default=true)
NodeSwap=true|false (ALPHA - default=false)
NonPreemptingPriority=true|false (BETA - default=true)
PodAffinityNamespaceSelector=true|false (BETA - default=true)
PodDeletionCost=true|false (BETA - default=true)
PodOverhead=true|false (BETA - default=true)
PodSecurity=true|false (ALPHA - default=false)
PreferNominatedNode=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
ServiceLBNodePortControl=true|false (BETA - default=true)
ServiceLoadBalancerClass=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetMinReadySeconds=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
SuspendJob=true|false (BETA - default=true)
TTLAfterFinished=true|false (BETA - default=true)
TopologyAwareHints=true|false (ALPHA - default=false)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (ALPHA - default=false)

+

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=true)
CSIMigrationAzureFile=true|false (BETA - default=true)
CSIMigrationGCE=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (ALPHA - default=false)
CSIMigrationRBD=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CSIVolumeHealth=true|false (ALPHA - default=false)
ContextualLogging=true|false (ALPHA - default=false)
CronJobTimeZone=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (ALPHA - default=false)
IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
JobReadyPods=true|false (BETA - default=true)
JobTrackingWithFinalizers=true|false (BETA - default=false)
KubeletCredentialProviders=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
NetworkPolicyStatus=true|false (ALPHA - default=false)
NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
NodeSwap=true|false (ALPHA - default=false)
OpenAPIEnums=true|false (BETA - default=true)
OpenAPIV3=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
ServerSideFieldValidation=true|false (ALPHA - default=false)
ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
StatefulSetMinReadySeconds=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (BETA - default=true)

@@ -205,21 +177,21 @@ kube-scheduler [flags] ---kube-api-burst int32 +--kube-api-burst int32     Default: 100

DEPRECATED: burst to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.

---kube-api-content-type string +--kube-api-content-type string     Default: "application/vnd.kubernetes.protobuf"

DEPRECATED: content type of requests sent to apiserver. This parameter is ignored if a config file is specified in --config.

---kube-api-qps float +--kube-api-qps float     Default: 50

DEPRECATED: QPS to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.

@@ -233,96 +205,68 @@ kube-scheduler [flags] ---leader-elect +--leader-elect     Default: true

Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability.

---leader-elect-lease-duration duration +--leader-elect-lease-duration duration     Default: 15s

The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.

---leader-elect-renew-deadline duration +--leader-elect-renew-deadline duration     Default: 10s

The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.

---leader-elect-resource-lock string +--leader-elect-resource-lock string     Default: "leases" -

The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

+

The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'.

---leader-elect-resource-name string +--leader-elect-resource-name string     Default: "kube-scheduler"

The name of resource object that is used for locking during leader election.

---leader-elect-resource-namespace string +--leader-elect-resource-namespace string     Default: "kube-system"

The namespace of resource object that is used for locking during leader election.

---leader-elect-retry-period duration +--leader-elect-retry-period duration     Default: 2s

The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.

---lock-object-name string +--lock-object-name string     Default: "kube-scheduler"

DEPRECATED: define the name of the lock object. Will be removed in favor of leader-elect-resource-name. This parameter is ignored if a config file is specified in --config.

---lock-object-namespace string +--lock-object-namespace string     Default: "kube-system"

DEPRECATED: define the namespace of the lock object. Will be removed in favor of leader-elect-resource-namespace. This parameter is ignored if a config file is specified in --config.

- ---log-backtrace-at <a string in the form 'file:N'>     Default: :0 - - -

when logging hits line file:N, emit a stack trace

- - - ---log-dir string - - -

If non-empty, write log files in this directory

- - - ---log-file string - - -

If non-empty, use this log file

- - - ---log-file-max-size uint     Default: 1800 - - -

Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

- - --log-flush-frequency duration     Default: 5s @@ -334,14 +278,7 @@ kube-scheduler [flags] --logging-format string     Default: "text" -

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning.

- - - ---logtostderr     Default: true - - -

log to standard error instead of files

+

Sets the log format. Permitted formats: "text".
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule.
Non-default choices are currently alpha and subject to change without warning.

@@ -351,13 +288,6 @@ kube-scheduler [flags]

The address of the Kubernetes API server (overrides any value in kubeconfig)

- ---one-output - - -

If true, only write logs to their native severity level (vs also writing to each lower severity level)

- - --permit-address-sharing @@ -373,35 +303,14 @@ kube-scheduler [flags] ---policy-config-file string +--pod-max-in-unschedulable-pods-duration duration     Default: 5m0s -

DEPRECATED: file with scheduler policy configuration. This file is used if policy ConfigMap is not provided or --use-legacy-policy-config=true. Note: The predicates/priorities defined in this file will take precedence over any profiles define in ComponentConfig.

+

DEPRECATED: the maximum time a pod can stay in unschedulablePods. If a pod stays in unschedulablePods for longer than this value, the pod will be moved from unschedulablePods to backoffQ or activeQ. This flag is deprecated and will be removed in 1.26

---policy-configmap string - - -

DEPRECATED: name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization if --use-legacy-policy-config=false. The config must be provided as the value of an element in 'Data' map with the key='policy.cfg'. Note: The predicates/priorities defined in this file will take precedence over any profiles define in ComponentConfig.

- - - ---policy-configmap-namespace string     Default: "kube-system" - - -

DEPRECATED: the namespace where policy ConfigMap is located. The kube-system namespace will be used if this is not provided or is empty. Note: The predicates/priorities defined in this file will take precedence over any profiles define in ComponentConfig.

- - - ---port int - - -

DEPRECATED: the port on which to serve HTTP insecurely without authentication and authorization. If 0, don't serve plain HTTP at all. See --secure-port instead. This parameter is ignored if a config file is specified in --config.

- - - ---profiling +--profiling     Default: true

DEPRECATED: enable profiling via web interface host:port/debug/pprof/. This parameter is ignored if a config file is specified in --config.

@@ -456,27 +365,6 @@ kube-scheduler [flags]

The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

- ---skip-headers - - -

If true, avoid header prefixes in the log messages

- - - ---skip-log-headers - - -

If true, avoid headers when opening log files

- - - ---stderrthreshold int     Default: 2 - - -

logs at or above this threshold go to stderr

- - --tls-cert-file string @@ -488,7 +376,7 @@ kube-scheduler [flags] --tls-cipher-suites strings -

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

+

Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.

@@ -512,13 +400,6 @@ kube-scheduler [flags]

A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".

- ---use-legacy-policy-config - - -

DEPRECATED: when set to true, scheduler will ignore policy ConfigMap and uses policy config file. Note: The scheduler will fail if this is combined with Plugin configs

- - -v, --v int @@ -534,10 +415,10 @@ kube-scheduler [flags] ---vmodule <comma-separated 'pattern=N' settings> +--vmodule pattern=N,... -

comma-separated list of pattern=N settings for file-filtered logging

+

comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)

diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index 0531f0847a..f2a8031502 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -44,70 +44,70 @@ kubelet [flags] --add-dir-header -If true, adds the file directory to the header of the log messages +If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see here.) - + --address string     Default: 0.0.0.0 -The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in gll interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) - + --allowed-unsafe-sysctls strings -Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --alsologtostderr -Log to standard error as well as files +Log to standard error as well as files (DEPRECATED: will be removed in a future release, see here.) --anonymous-auth     Default: true -Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authentication-token-webhook -Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authentication-token-webhook-cache-ttl duration     Default: 2m0s -The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) ---authorization-mode string +--authorization-mode string     Default: AlwaysAllow -Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. Default AlwaysAllow when --config flag is not provided; Webhook when --config flag presents. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authorization-webhook-cache-authorized-ttl duration     Default: 5m0s -The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authorization-webhook-cache-unauthorized-ttl duration     Default: 30s -The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -135,77 +135,56 @@ kubelet [flags] --cgroup-driver string     Default: cgroupfs -Driver that the kubelet uses to manipulate cgroups on the host. Possible values: cgroupfs, systemd. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)/td> +Driver that the kubelet uses to manipulate cgroups on the host. Possible values: cgroupfs, systemd. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cgroup-root string     Default: '' -Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cgroups-per-qos     Default: true -Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --client-ca-file string -If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cloud-config string -The path to the cloud provider configuration file. Empty string for no configuration file. (DEPRECATED: will be removed in 1.23, in favor of removing cloud providers code from Kubelet.) +The path to the cloud provider configuration file. Empty string for no configuration file. (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud providers code from kubelet.) --cloud-provider string -The provider for cloud services. Set to empty string for running with no cloud provider. If set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used). (DEPRECATED: will be removed in 1.23, in favor of removing cloud provider code from Kubelet.) +The provider for cloud services. Set to empty string for running with no cloud provider. If set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used). (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.) --cluster-dns strings -Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cluster-domain string -Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---cni-bin-dir string     Default: /opt/cni/bin - - -A comma-separated list of full paths of directories in which to search for CNI plugin binaries. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) - - - ---cni-cache-dir string     Default: /var/lib/cni/cache - - -The full path of the directory in which CNI should store cache files. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) - - - ---cni-conf-dir string     Default: /etc/cni/net.d - - -<Warning: Alpha feature> The full path of the directory in which to search for CNI config files. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) +Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -219,169 +198,154 @@ kubelet [flags] --container-log-max-files int32     Default: 5 -<Warning: Beta feature> Set the maximum number of container log files that can be present for a container. The number must be >= 2. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Set the maximum number of container log files that can be present for a container. The number must be >= 2. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --container-log-max-size string     Default: 10Mi -<Warning: Beta feature> Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) ---container-runtime string     Default: docker +--container-runtime string     Default: remote -The container runtime to use. Possible values: docker, remote. +The container runtime to use. Possible values: docker, remote. (DEPRECATED: will be removed in 1.27 as the only valid value is 'remote') ---container-runtime-endpoint string     Default: unix:///var/run/dockershim.sock +--container-runtime-endpoint string -[Experimental] The endpoint of remote runtime service. Currently unix socket endpoint is supported on Linux, while npipe and tcp endpoints are supported on windows. Examples: unix:///var/run/dockershim.sock, npipe:////./pipe/dockershim. +The endpoint of remote runtime service. Unix Domain SOckets are supported on Linux, while npipe and tcp endpoints are supported on windows. Examples: unix:///var/run/dockershim.sock, npipe:////./pipe/dockershim. - --contention-profiling -Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-cfs-quota     Default: true -Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-cfs-quota-period duration     Default: 100ms -Sets CPU CFS quota period value, cpu.cfs_period_us, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Sets CPU CFS quota period value, cpu.cfs_period_us, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-manager-policy string     Default: none -CPU Manager policy to use. Possible values: none, static. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +CPU Manager policy to use. Possible values: none, static. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) ---cpu-manager-policy-options strings +--cpu-manager-policy-options mapStringString -Comma-separated list of options to fine-tune the behavior of the selected CPU Manager policy. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of key=value CPU Manager policy options to use, to fine tune their behaviour. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-manager-reconcile-period duration     Default: 10s -<Warning: Alpha feature> CPU Manager reconciliation period. Examples: 10s, or 1m. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---docker-endpoint string     Default: unix:///var/run/docker.sock - - -Use this for the docker endpoint to communicate with. This docker-specific flag only works when container-runtime is set to docker. - - - ---dynamic-config-dir string - - -The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. The DynamicKubeletConfig feature gate must be enabled to pass this flag. (DEPRECATED: Feature DynamicKubeletConfig is deprecated in 1.22 and will not move to GA. It is planned to be removed from Kubernetes in the version 1.23. Please use alternative ways to update kubelet configuration.) +<Warning: Alpha feature> CPU Manager reconciliation period. Examples: 10s, or 1m. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enable-controller-attach-detach     Default: true -Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enable-debugging-handlers     Default: true -Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enable-server     Default: true -Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enforce-node-allocatable strings     Default: pods -A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are none, pods, system-reserved, and kube-reserved. If the latter two options are specified, --system-reserved-cgroup and --kube-reserved-cgroup must also be set, respectively. If none is specified, no additional options should be set. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are none, pods, system-reserved, and kube-reserved. If the latter two options are specified, --system-reserved-cgroup and --kube-reserved-cgroup must also be set, respectively. If none is specified, no additional options should be set. See here for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --event-burst int32     Default: 10 -Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding --event-qps. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding --event-qps. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --event-qps int32     Default: 5 -QPS to limit event creations. The number must be >= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +QPS to limit event creations. The number must be >= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-hard mapStringString     Default: imagefs.available<15%,memory.available<100Mi,nodefs.available<10% -A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. On a Linux node, the default value also includes nodefs.inodesFree<5%. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. On a Linux node, the default value also includes nodefs.inodesFree<5%. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-max-pod-grace-period int32 - Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) + Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-minimum-reclaim mapStringString -A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-pressure-transition-period duration     Default: 5m0s -Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-soft mapStringString -A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-soft-grace-period mapStringString -A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -395,49 +359,28 @@ kubelet [flags] --experimental-allocatable-ignore-eviction     Default: false -When set to true, hard eviction thresholds will be ignored while calculating node allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: will be removed in 1.23) - - - ---experimental-bootstrap-kubeconfig string - - -DEPRECATED: Use --bootstrap-kubeconfig - - - ---experimental-check-node-capabilities-before-mount - - -[Experimental] if set to true, the kubelet will check the underlying node for required components (binaries, etc.) before performing the mount (DEPRECATED: will be removed in 1.23, in favor of using CSI.) +When set to true, hard eviction thresholds will be ignored while calculating node allocatable. See here for more details. (DEPRECATED: will be removed in 1.24 or later) --experimental-kernel-memcg-notification -If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. This flag will be removed in 1.23. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---experimental-log-sanitization bool - - -[Experimental] When enabled, prevents logging of fields tagged as sensitive (passwords, keys, tokens). Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Use kernelMemcgNotification configuration, this flag will be removed in 1.24 or later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --experimental-mounter-path string     Default: mount -[Experimental] Path of mounter binary. Leave empty to use the default mount. (DEPRECATED: will be removed in 1.23, in favor of using CSI.) +[Experimental] Path of mounter binary. Leave empty to use the default mount. (DEPRECATED: will be removed in 1.24 or later, in favor of using CSI.) --fail-swap-on     Default: true -Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -452,125 +395,128 @@ APIServerIdentity=true|false (ALPHA - default=false)
APIServerTracing=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
-AnyVolumeDataSource=true|false (ALPHA - default=false)
+AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManager=true|false (BETA - default=true)
-CPUManagerPolicyOptions=true|false (ALPHA - default=false)
+CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
+CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
+CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
-CSIMigrationAWS=true|false (BETA - default=false)
-CSIMigrationAzureDisk=true|false (BETA - default=false)
-CSIMigrationAzureFile=true|false (BETA - default=false)
-CSIMigrationGCE=true|false (BETA - default=false)
-CSIMigrationOpenStack=true|false (BETA - default=true)
+CSIMigrationAWS=true|false (BETA - default=true)
+CSIMigrationAzureFile=true|false (BETA - default=true)
+CSIMigrationGCE=true|false (BETA - default=true)
+CSIMigrationPortworx=true|false (ALPHA - default=false)
+CSIMigrationRBD=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
-CSIStorageCapacity=true|false (BETA - default=true)
-CSIVolumeFSGroupPolicy=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
-CSRDuration=true|false (BETA - default=true)
-ConfigurableFSGroupPolicy=true|false (BETA - default=true)
-ControllerManagerLeaderMigration=true|false (BETA - default=true)
+ContextualLogging=true|false (ALPHA - default=false)
+CronJobTimeZone=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
+CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
-DefaultPodTopologySpread=true|false (BETA - default=true)
-DelegateFSGroupToCSIDriver=true|false (ALPHA - default=false)
+DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
-DownwardAPIHugePages=true|false (BETA - default=false)
-EfficientWatchResumption=true|false (BETA - default=true)
+DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
+DownwardAPIHugePages=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
-EphemeralContainers=true|false (ALPHA - default=false)
-ExpandCSIVolumes=true|false (BETA - default=true)
-ExpandInUsePersistentVolumes=true|false (BETA - default=true)
-ExpandPersistentVolumes=true|false (BETA - default=true)
+EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
-GenericEphemeralVolume=true|false (BETA - default=true)
+GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
+GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
-IPv6DualStack=true|false (BETA - default=true)
+HonorPVReclaimPolicy=true|false (ALPHA - default=false)
+IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
+InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
+InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
-IndexedJob=true|false (BETA - default=true)
-IngressClassNamespacedParams=true|false (BETA - default=true)
-JobTrackingWithFinalizers=true|false (ALPHA - default=false)
-KubeletCredentialProviders=true|false (ALPHA - default=false)
+JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
+JobReadyPods=true|false (BETA - default=true)
+JobTrackingWithFinalizers=true|false (BETA - default=false)
+KubeletCredentialProviders=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
-KubeletPodResourcesGetAllocatable=true|false (ALPHA - default=false)
+KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
+LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
+MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
-MixedProtocolLBService=true|false (ALPHA - default=false)
+MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
+MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
+NetworkPolicyStatus=true|false (ALPHA - default=false)
+NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
NodeSwap=true|false (ALPHA - default=false)
-NonPreemptingPriority=true|false (BETA - default=true)
-PodAffinityNamespaceSelector=true|false (BETA - default=true)
+OpenAPIEnums=true|false (BETA - default=true)
+OpenAPIV3=true|false (BETA - default=true)
+PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
-PodOverhead=true|false (BETA - default=true)
-PodSecurity=true|false (ALPHA - default=false)
-PreferNominatedNode=true|false (BETA - default=true)
+PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
+RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
-RemoveSelfLink=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SeccompDefault=true|false (ALPHA - default=false)
+ServerSideFieldValidation=true|false (ALPHA - default=false)
+ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
-ServiceLBNodePortControl=true|false (BETA - default=true)
-ServiceLoadBalancerClass=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
-StatefulSetMinReadySeconds=true|false (ALPHA - default=false)
+StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
+StatefulSetMinReadySeconds=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
-SuspendJob=true|false (BETA - default=true)
-TTLAfterFinished=true|false (BETA - default=true)
-TopologyAwareHints=true|false (ALPHA - default=false)
+TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
-WindowsHostProcessContainers=true|false (ALPHA - default=false)
-(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +WindowsHostProcessContainers=true|false (BETA - default=true)
+(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --file-check-frequency duration     Default: 20s -Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --hairpin-mode string     Default: promiscuous-bridge -How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are promiscuous-bridge, hairpin-veth and none. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are promiscuous-bridge, hairpin-veth and none. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --healthz-bind-address string     Default: 127.0.0.1 -The IP address for the healthz server to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The IP address for the healthz server to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --healthz-port int32     Default: 10248 -The port of the localhost healthz endpoint (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The port of the localhost healthz endpoint (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -591,7 +537,7 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--http-check-frequency duration     Default: 20s -Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -612,44 +558,37 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--image-gc-high-threshold int32     Default: 85 -The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --image-gc-low-threshold int32     Default: 80 -The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---image-pull-progress-deadline duration     Default: 1m0s - - -If no pulling progress is made before this deadline, the image pulling will be cancelled. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) +The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --image-service-endpoint string -[Experimental] The endpoint of remote image service. If not specified, it will be the same with --container-runtime-endpoint by default. Currently UNIX socket endpoint is supported on Linux, while npipe and TCP endpoints are supported on Windows. Examples: unix:///var/run/dockershim.sock, npipe:////./pipe/dockershim +[Experimental] The endpoint of remote image service. If not specified, it will be the same with --container-runtime-endpoint by default. Unix Domain Socket are supported on Linux, while npipe and TCP endpoints are supported on Windows. Examples: unix:///var/run/dockershim.sock, npipe:////./pipe/dockershim - + --iptables-drop-bit int32     Default: 15 -The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) - + --iptables-masquerade-bit int32     Default: 14 -The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) - + --keep-terminated-pod-volumes @@ -661,42 +600,42 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--kernel-memcg-notification -If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-burst int32     Default: 10 -Burst to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Burst to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-content-type string     Default: application/vnd.kubernetes.protobuf -Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-qps int32     Default: 5 -QPS to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +QPS to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default QPS (5). Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-reserved mapStringString     Default: <None> -A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral-storage for root file system are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral-storage for root file system are supported. See here for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-reserved-cgroup string     Default: '' -Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via --kube-reserved flag. Ex. /kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via --kube-reserved flag. Ex. /kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -710,7 +649,7 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--kubelet-cgroups string -Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -724,28 +663,28 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--log-backtrace-at <A string of format 'file:line'>     Default: ":0" -When logging hits line :, emit a stack trace. +When logging hits line :, emit a stack trace. (DEPRECATED: will be removed in a future release, see here.) --log-dir string -If non-empty, write log files in this directory +If non-empty, write log files in this directory. (DEPRECATED: will be removed in a future release, see here.) --log-file string -If non-empty, use this log file +If non-empty, use this log file. (DEPRECATED: will be removed in a future release, see here.) --log-file-max-size uint     Default: 1800 -Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. +Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see here.) @@ -755,59 +694,74 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
Maximum number of seconds between log flushes. + +--log-json-info-buffer-size string     Default: '0' + + +[Experimental] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) + + + +--log-json-split-stream + + +[Experimental] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) + + --logging-format string     Default: text -Sets the log format. Permitted formats: text, json.
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --skip_headers, --skip_log_headers, --stderrthreshold, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Sets the log format. Permitted formats: text, json.
Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --skip_headers, --skip_log_headers, --stderrthreshold, --log-flush-frequency.
Non-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --logtostderr     Default: true -log to standard error instead of files. +log to standard error instead of files. (DEPRECATED: will be removed in a future release, see here.) --make-iptables-util-chains     Default: true -If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --manifest-url string -URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --manifest-url-header string -Comma-separated list of HTTP headers to use when accessing the URL provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated list of HTTP headers to use when accessing the URL provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) + --master-service-namespace string     Default: default The namespace from which the kubernetes master services should be injected into pods. (DEPRECATED: This flag will be removed in a future version.) - + --max-open-files int     Default: 1000000 -Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --max-pods int32     Default: 110 -Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -828,7 +782,7 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--memory-manager-policy string     Default: None -Memory Manager policy to use. Possible values: 'None', 'Static'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Memory Manager policy to use. Possible values: 'None', 'Static'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -842,21 +796,7 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--minimum-image-ttl-duration duration     Default: 2m0s -Minimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---network-plugin string - - -The name of the network plugin to be invoked for various events in kubelet/pod lifecycle. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) - - - ---network-plugin-mtu int32 - - -The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.) +Minimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -877,46 +817,39 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--node-status-max-images int32     Default: 50 -The maximum number of images to report in node.status.images. If -1 is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The maximum number of images to report in node.status.images. If -1 is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --node-status-update-frequency duration     Default: 10s -Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---non-masquerade-cidr string     Default: 10.0.0.0/8 - - -Traffic to IPs outside this range will use IP masquerade. Set to '0.0.0.0/0' to never masquerade. (DEPRECATED: will be removed in a future version) +Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --one-output -If true, only write logs to their native severity level (vs also writing to each lower severity level). +If true, only write logs to their native severity level (vs also writing to each lower severity level). (DEPRECATED: will be removed in a future release, see here.) --oom-score-adj int32     Default: -999 -The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pod-cidr string -The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) ---pod-infra-container-image string     Default: k8s.gcr.io/pause:3.5 +--pod-infra-container-image string     Default: k8s.gcr.io/pause:3.6 Specified image will not be pruned by the image garbage collector. When container-runtime is set to docker, all containers in each pod will use the network/IPC namespaces from this image. Other CRI implementations have their own configuration to set this image. @@ -926,70 +859,63 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--pod-manifest-path string -Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pod-max-pids int     Default: -1 -Set the maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Set the maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pods-per-core int32 -Number of Pods per core that can run on this kubelet. The total number of pods on this kubelet cannot exceed --max-pods, so --max-pods will be used if this calculation results in a larger number of pods allowed on the kubelet. A value of 0 disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of Pods per core that can run on this kubelet. The total number of pods on this kubelet cannot exceed --max-pods, so --max-pods will be used if this calculation results in a larger number of pods allowed on the kubelet. A value of 0 disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --port int32     Default: 10250 -The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's --config flag. See kubelet-config-file for more information.) --protect-kernel-defaults - Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) + Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --provider-id string -Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --qos-reserved mapStringString -<Warning: Alpha feature> A set of <resource name>=<percentage> (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the QOSReserved feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Alpha feature> A set of <resource name>=<percentage> (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the QOSReserved feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --read-only-port int32     Default: 10255 -The read-only port for the kubelet to serve on with no authentication/authorization (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) - - - ---really-crash-for-testing - - -If true, when panics occur crash. Intended for testing. (DEPRECATED: will be removed in a future version.) +The read-only port for the kubelet to serve on with no authentication/authorization (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --register-node     Default: true -Register the node with the API server. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. +Register the node with the API server. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1003,42 +929,42 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--register-with-taints mapStringString -Register the node with the given list of taints (comma separated <key>=<value>:<effect>). No-op if --register-node is false. +Register the node with the given list of taints (comma separated <key>=<value>:<effect>). No-op if --register-node is false. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --registry-burst int32     Default: 10 -Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding --registry-qps. Only used if --registry-qps is greater than 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding --registry-qps. Only used if --registry-qps is greater than 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --registry-qps int32     Default: 5 -If > 0, limit registry pull QPS to this value. If 0, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If > 0, limit registry pull QPS to this value. If 0, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --reserved-cpus string -A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --reserved-memory string -A comma-separated list of memory reservations for NUMA nodes. (e.g. --reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi). The total sum for each memory type should be equal to the sum of --kube-reserved, --system-reserved and --eviction-threshold. See https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/#reserved-memory-flag for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma-separated list of memory reservations for NUMA nodes. (e.g. --reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi). The total sum for each memory type should be equal to the sum of --kube-reserved, --system-reserved and --eviction-threshold. See here for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --resolv-conf string     Default: /etc/resolv.conf -Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1052,21 +978,21 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--rotate-certificates -<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --rotate-server-certificates -Auto-request and rotate the kubelet serving certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. Requires the RotateKubeletServerCertificate feature gate to be enabled, and approval of the submitted CertificateSigningRequest objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Auto-request and rotate the kubelet serving certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. Requires the RotateKubeletServerCertificate feature gate to be enabled, and approval of the submitted CertificateSigningRequest objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --runonce -If true, exit after spawning pods from local manifests or remote urls. Exclusive with --enable-server (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If true, exit after spawning pods from local manifests or remote urls. Exclusive with --enable-server (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1080,92 +1006,84 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
--runtime-request-timeout duration     Default: 2m0s -Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) ---seccomp-default RuntimeDefault +--seccomp-default string <Warning: Alpha feature> Enable the use of RuntimeDefault as the default seccomp profile for all workloads. The SeccompDefault feature gate must be enabled to allow this flag, which is disabled by default. - ---seccomp-profile-root string     Default: /var/lib/kubelet/seccomp - - -<Warning: Alpha feature> Directory path for seccomp profiles. (DEPRECATED: will be removed in 1.23, in favor of using the /seccomp directory) - - - --serialize-image-pulls     Default: true -Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an aufs storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an aufs storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --skip-headers -If true, avoid header prefixes in the log messages +If true, avoid header prefixes in the log messages. (DEPRECATED: will be removed in a future release, see here.) --skip-log-headers -If true, avoid headers when opening log files +If true, avoid headers when opening log files. (DEPRECATED: will be removed in a future release, see here.) --stderrthreshold int     Default: 2 -logs at or above this threshold go to stderr. +logs at or above this threshold go to stderr. (DEPRECATED: will be removed in a future release, see here.) --streaming-connection-idle-timeout duration     Default: 4h0m0s -Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: 5m. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: 5m. Note: All connections to the kubelet server have a maximum duration of 4 hours. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --sync-frequency duration     Default: 1m0s -Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-cgroups string -Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-reserved mapStringString     Default: <none> -A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See here for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-reserved-cgroup string     Default: '' -Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via --system-reserved flag. Ex. /system-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via --system-reserved flag. Ex. /system-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --tls-cert-file string -File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1174,24 +1092,24 @@ WindowsHostProcessContainers=true|false (ALPHA - default=false)
Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: -TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
-Insecure values: -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. -(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +`TLS_AES_128_GCM_SHA256`, `TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_RSA_WITH_AES_128_CBC_SHA`, `TLS_RSA_WITH_AES_128_GCM_SHA256`, `TLS_RSA_WITH_AES_256_CBC_SHA`, `TLS_RSA_WITH_AES_256_GCM_SHA384`
+Insecure values:
+`TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_ECDSA_WITH_RC4_128_SHA`, `TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_RSA_WITH_RC4_128_SHA`, `TLS_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_RSA_WITH_AES_128_CBC_SHA256`, `TLS_RSA_WITH_RC4_128_SHA`.
+(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See kubelet-config-file for more information.) --tls-min-version string -Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --tls-private-key-file string -File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1199,14 +1117,14 @@ TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_E --topology-manager-policy string     Default: 'none' -Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --topology-manager-scope string     Default: container -Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1224,7 +1142,7 @@ TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_E ---vmodule <A list of 'pattern=N' string> +--vmodule <A list of 'pattern=N' strings> Comma-separated list of pattern=N settings for file-filtered logging @@ -1234,14 +1152,14 @@ TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_E --volume-plugin-dir string     Default: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ -The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --volume-stats-agg-period duration     Default: 1m0s -Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) diff --git a/content/en/docs/reference/config-api/apiserver-audit.v1.md b/content/en/docs/reference/config-api/apiserver-audit.v1.md index 11df06bd8c..e48824c649 100644 --- a/content/en/docs/reference/config-api/apiserver-audit.v1.md +++ b/content/en/docs/reference/config-api/apiserver-audit.v1.md @@ -16,18 +16,16 @@ auto_generated: true - ## `Event` {#audit-k8s-io-v1-Event} - - **Appears in:** - [EventList](#audit-k8s-io-v1-EventList) -Event captures all the information that can be included in an API audit log. +

Event captures all the information that can be included in an API audit log.

+ @@ -36,165 +34,152 @@ Event captures all the information that can be included in an API audit log. - - +

AuditLevel at which event was generated

+ - - +

Unique audit ID, generated for each request.

+ - - +

Stage of the request handling when this event instance was generated.

+ - - +

RequestURI is the request URI as sent by the client to a server.

+ - - +

Verb is the kubernetes verb associated with the request. +For non-resource requests, this is the lower-cased HTTP method.

+ - - +

Authenticated user information.

+ - - +

Impersonated user information.

+ - - +

Source IPs, from where the request originated and intermediate proxies. +The source IPs are listed from (in order):

+
    +
  1. X-Forwarded-For request header IPs
    2. +
  2. X-Real-Ip header, if not present in the X-Forwarded-For list
    3. +
  3. The remote address for the connection, if it doesn't match the last +IP in the list up to here (X-Forwarded-For or X-Real-Ip). +Note: All but the last IP can be arbitrarily set by the client.
    4. +
+ - - +

UserAgent records the user agent string reported by the client. +Note that the UserAgent is provided by the client, and must not be trusted.

+ - - +

Object reference this request is targeted at. +Does not apply for List-type requests, or non-resource requests.

+ - - +For non-status type error responses, this will be auto-populated with the error Message.

+ - - +Omitted for non-resource requests. Only logged at Request Level and higher.

+ - - +at Response Level.

+ - - +

Time the request reached the apiserver.

+ - - +

Time the request reached current audit stage.

+ - - +should be short. Annotations are included in the Metadata level.

+ - -
FieldDescription
apiVersion
string		audit.k8s.io/v1
kind
string		Event
level [Required]
Level 		- AuditLevel at which event was generated
auditID [Required]
-k8s.io/apimachinery/pkg/types.UID +k8s.io/apimachinery/pkg/types.UID 		- Unique audit ID, generated for each request.
stage [Required]
Stage 		- Stage of the request handling when this event instance was generated.
requestURI [Required]
string 		- RequestURI is the request URI as sent by the client to a server.
verb [Required]
string 		- Verb is the kubernetes verb associated with the request. -For non-resource requests, this is the lower-cased HTTP method.
user [Required]
-authentication/v1.UserInfo +authentication/v1.UserInfo 		- Authenticated user information.
impersonatedUser
-authentication/v1.UserInfo +authentication/v1.UserInfo 		- Impersonated user information.
sourceIPs
[]string 		- Source IPs, from where the request originated and intermediate proxies.
userAgent
string 		- UserAgent records the user agent string reported by the client. -Note that the UserAgent is provided by the client, and must not be trusted.
objectRef
ObjectReference 		- Object reference this request is targeted at. -Does not apply for List-type requests, or non-resource requests.
responseStatus
-meta/v1.Status +meta/v1.Status 		- The response status, populated even when the ResponseObject is not a Status type. +

The response status, populated even when the ResponseObject is not a Status type. For successful responses, this will only include the Code and StatusSuccess. -For non-status type error responses, this will be auto-populated with the error Message.
requestObject
-k8s.io/apimachinery/pkg/runtime.Unknown +k8s.io/apimachinery/pkg/runtime.Unknown 		- API object from the request, in JSON format. The RequestObject is recorded as-is in the request +

API object from the request, in JSON format. The RequestObject is recorded as-is in the request (possibly re-encoded as JSON), prior to version conversion, defaulting, admission or merging. It is an external versioned object type, and may not be a valid object on its own. -Omitted for non-resource requests. Only logged at Request Level and higher.
responseObject
-k8s.io/apimachinery/pkg/runtime.Unknown +k8s.io/apimachinery/pkg/runtime.Unknown 		- API object returned in the response, in JSON. The ResponseObject is recorded after conversion +

API object returned in the response, in JSON. The ResponseObject is recorded after conversion to the external type, and serialized as JSON. Omitted for non-resource requests. Only logged -at Response Level.
requestReceivedTimestamp
-meta/v1.MicroTime +meta/v1.MicroTime 		- Time the request reached the apiserver.
stageTimestamp
-meta/v1.MicroTime +meta/v1.MicroTime 		- Time the request reached current audit stage.
annotations
map[string]string 		- Annotations is an unstructured key value map stored with an audit event that may be set by +

Annotations is an unstructured key value map stored with an audit event that may be set by plugins invoked in the request serving chain, including authentication, authorization and admission plugins. Note that these annotations are for the audit event, and do not correspond to the metadata.annotations of the submitted object. Keys should uniquely identify the informing component to avoid name collisions (e.g. podsecuritypolicy.admission.k8s.io/policy). Values -should be short. Annotations are included in the Metadata level.
- - ## `EventList` {#audit-k8s-io-v1-EventList} +

EventList is a list of audit Events.

-EventList is a list of audit Events. - @@ -202,44 +187,33 @@ EventList is a list of audit Events. - - + No description provided. - - + No description provided. - -
FieldDescription
apiVersion
string		audit.k8s.io/v1
kind
string		EventList
metadata
-meta/v1.ListMeta +meta/v1.ListMeta 		- No description provided. -
items [Required]
[]Event 		- No description provided. -
- - ## `Policy` {#audit-k8s-io-v1-Policy} - - **Appears in:** - [PolicyList](#audit-k8s-io-v1-PolicyList) -Policy defines the configuration of audit logging, and the rules for how different request -categories are logged. +

Policy defines the configuration of audit logging, and the rules for how different request +categories are logged.

+ @@ -248,50 +222,54 @@ categories are logged. - - +

ObjectMeta is included for interoperability with API infrastructure.

+Refer to the Kubernetes API documentation for the fields of the metadata field. - - +PolicyRules are strictly ordered.

+ - - +

OmitStages is a list of stages for which no events are created. Note that this can also +be specified per rule in which case the union of both are omitted.

+ + + + - -
FieldDescription
apiVersion
string		audit.k8s.io/v1
kind
string		Policy
metadata
-meta/v1.ObjectMeta +meta/v1.ObjectMeta 		- ObjectMeta is included for interoperability with API infrastructure.Refer to the Kubernetes API documentation for the fields of the metadata field.
rules [Required]
[]PolicyRule 		- Rules specify the audit Level a request should be recorded at. +

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. -PolicyRules are strictly ordered.
omitStages
[]Stage 		- OmitStages is a list of stages for which no events are created. Note that this can also -be specified per rule in which case the union of both are omitted.
omitManagedFields
+bool +		 +

OmitManagedFields indicates whether to omit the managed fields of the request +and response bodies from being written to the API audit log. +This is used as a global default - a value of 'true' will omit the managed fileds, +otherwise the managed fields will be included in the API audit log. +Note that this can also be specified per rule in which case the value specified +in a rule will override the global default.

+
- - ## `PolicyList` {#audit-k8s-io-v1-PolicyList} +

PolicyList is a list of audit Policies.

-PolicyList is a list of audit Policies. - @@ -299,99 +277,78 @@ PolicyList is a list of audit Policies. - - + No description provided. - - + No description provided. - -
FieldDescription
apiVersion
string		audit.k8s.io/v1
kind
string		PolicyList
metadata
-meta/v1.ListMeta +meta/v1.ListMeta 		- No description provided. -
items [Required]
[]Policy 		- No description provided. -
- - ## `GroupResources` {#audit-k8s-io-v1-GroupResources} - - **Appears in:** - [PolicyRule](#audit-k8s-io-v1-PolicyRule) -GroupResources represents resource kinds in an API group. +

GroupResources represents resource kinds in an API group.

+ - +

Group is the name of the API group that contains the resources. +The empty string represents the core API group.

+ - - +'' matches all resources and their subresources. +'pods/' matches all subresources of pods. +'*/scale' matches all scale subresources.

+

If wildcard is present, the validation rule will ensure resources do not +overlap with each other.

+

An empty list implies all resources and subresources in this API groups apply.

+ - - +An empty list implies that every instance of the resource is matched.

+ - -
FieldDescription
group
string 		- Group is the name of the API group that contains the resources. -The empty string represents the core API group.
resources
[]string 		- Resources is a list of resources this rule applies to. - -For example: +

Resources is a list of resources this rule applies to.

+

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. -'∗' matches all resources and their subresources. -'pods/∗' matches all subresources of pods. -'∗/scale' matches all scale subresources. - -If wildcard is present, the validation rule will ensure resources do not -overlap with each other. - -An empty list implies all resources and subresources in this API groups apply.
resourceNames
[]string 		- ResourceNames is a list of resource instance names that the policy matches. +

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. -An empty list implies that every instance of the resource is matched.
- - ## `Level` {#audit-k8s-io-v1-Level} (Alias of `string`) - **Appears in:** - [Event](#audit-k8s-io-v1-Event) @@ -399,211 +356,189 @@ An empty list implies that every instance of the resource is matched. - [PolicyRule](#audit-k8s-io-v1-PolicyRule) -Level defines the amount of information logged during auditing +

Level defines the amount of information logged during auditing

- ## `ObjectReference` {#audit-k8s-io-v1-ObjectReference} - - **Appears in:** - [Event](#audit-k8s-io-v1-Event) -ObjectReference contains enough information to let you inspect or modify the referred object. +

ObjectReference contains enough information to let you inspect or modify the referred object.

+ - + No description provided. - - + No description provided. - - + No description provided. - - + No description provided. - - +

APIGroup is the name of the API group that contains the referred object. +The empty string represents the core API group.

+ - - +

APIVersion is the version of the API group that contains the referred object.

+ - - + No description provided. - - + No description provided. - -
FieldDescription
resource
string 		- No description provided. -
namespace
string 		- No description provided. -
name
string 		- No description provided. -
uid
-k8s.io/apimachinery/pkg/types.UID +k8s.io/apimachinery/pkg/types.UID 		- No description provided. -
apiGroup
string 		- APIGroup is the name of the API group that contains the referred object. -The empty string represents the core API group.
apiVersion
string 		- APIVersion is the version of the API group that contains the referred object.
resourceVersion
string 		- No description provided. -
subresource
string 		- No description provided. -
- - ## `PolicyRule` {#audit-k8s-io-v1-PolicyRule} - - **Appears in:** - [Policy](#audit-k8s-io-v1-Policy) -PolicyRule maps requests based off metadata to an audit Level. -Requests must match the rules of every field (an intersection of rules). +

PolicyRule maps requests based off metadata to an audit Level. +Requests must match the rules of every field (an intersection of rules).

+ - +

The Level that requests matching this rule are recorded at.

+ - - +

The users (by authenticated user name) this rule applies to. +An empty list implies every user.

+ - - +An empty list implies every user group.

+ - - +

The verbs that match this rule. +An empty list implies every verb.

+ - - +

Resources that this rule matches. An empty list implies all kinds in all API groups.

+ - - +

Namespaces that this rule matches. +The empty string "" matches non-namespaced resources. +An empty list implies every namespace.

+ - - +"/metrics" - Log requests for apiserver metrics +"/healthz" - Log all health checks

+ - - +An empty list means no restrictions will apply.

+ + + + - -
FieldDescription
level [Required]
Level 		- The Level that requests matching this rule are recorded at.
users
[]string 		- The users (by authenticated user name) this rule applies to. -An empty list implies every user.
userGroups
[]string 		- The user groups this rule applies to. A user is considered matching +

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. -An empty list implies every user group.
verbs
[]string 		- The verbs that match this rule. -An empty list implies every verb.
resources
[]GroupResources 		- Resources that this rule matches. An empty list implies all kinds in all API groups.
namespaces
[]string 		- Namespaces that this rule matches. -The empty string "" matches non-namespaced resources. -An empty list implies every namespace.
nonResourceURLs
[]string 		- NonResourceURLs is a set of URL paths that should be audited. -∗s are allowed, but only as the full, final step in the path. +

NonResourceURLs is a set of URL paths that should be audited. +s are allowed, but only as the full, final step in the path. Examples: - "/metrics" - Log requests for apiserver metrics - "/healthz∗" - Log all health checks
omitStages
[]Stage 		- OmitStages is a list of stages for which no events are created. Note that this can also +

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. -An empty list means no restrictions will apply.

omitManagedFields
+bool +		 +

OmitManagedFields indicates whether to omit the managed fields of the request +and response bodies from being written to the API audit log.

+
    +
  • a value of 'true' will drop the managed fields from the API audit log
    • +
  • a value of 'false' indicates that the managed fileds should be included +in the API audit log +Note that the value, if specified, in this rule will override the global default +If a value is not specified then the global default specified in +Policy.OmitManagedFields will stand.
    • +
+
- - ## `Stage` {#audit-k8s-io-v1-Stage} (Alias of `string`) - **Appears in:** - [Event](#audit-k8s-io-v1-Event) @@ -613,8 +548,8 @@ An empty list means no restrictions will apply. - [PolicyRule](#audit-k8s-io-v1-PolicyRule) -Stage defines the stages in request handling that audit events may be generated. +

Stage defines the stages in request handling that audit events may be generated.

+ - diff --git a/content/en/docs/reference/config-api/apiserver-config.v1.md b/content/en/docs/reference/config-api/apiserver-config.v1.md new file mode 100644 index 0000000000..dcb903e848 --- /dev/null +++ b/content/en/docs/reference/config-api/apiserver-config.v1.md @@ -0,0 +1,84 @@ +--- +title: kube-apiserver Configuration (v1) +content_type: tool-reference +package: apiserver.config.k8s.io/v1 +auto_generated: true +--- +

Package v1 is the v1 version of the API.

+ + +## Resource Types + + +- [AdmissionConfiguration](#apiserver-config-k8s-io-v1-AdmissionConfiguration) + + + +## `AdmissionConfiguration` {#apiserver-config-k8s-io-v1-AdmissionConfiguration} + + + +

AdmissionConfiguration provides versioned configuration for admission controllers.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		apiserver.config.k8s.io/v1
kind
string		AdmissionConfiguration
plugins
+[]AdmissionPluginConfiguration +		 +

Plugins allows specifying a configuration per admission control plugin.

+
+ +## `AdmissionPluginConfiguration` {#apiserver-config-k8s-io-v1-AdmissionPluginConfiguration} + + +**Appears in:** + +- [AdmissionConfiguration](#apiserver-config-k8s-io-v1-AdmissionConfiguration) + + +

AdmissionPluginConfiguration provides the configuration for a single plug-in.

+ + + + + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name is the name of the admission controller. +It must match the registered admission plugin name.

+
path
+string +		 +

Path is the path to a configuration file that contains the plugin's +configuration

+
configuration
+k8s.io/apimachinery/pkg/runtime.Unknown +		 +

Configuration is an embedded configuration object to be used as the plugin's +configuration. If present, it will be used instead of the path to the configuration file.

+
+ diff --git a/content/en/docs/reference/config-api/apiserver-config.v1alpha1.md b/content/en/docs/reference/config-api/apiserver-config.v1alpha1.md index 81702355a5..8b9bf972ca 100644 --- a/content/en/docs/reference/config-api/apiserver-config.v1alpha1.md +++ b/content/en/docs/reference/config-api/apiserver-config.v1alpha1.md @@ -4,7 +4,8 @@ content_type: tool-reference package: apiserver.k8s.io/v1alpha1 auto_generated: true --- -Package v1alpha1 is the v1alpha1 version of the API. +

Package v1alpha1 is the v1alpha1 version of the API.

+ ## Resource Types @@ -15,15 +16,13 @@ Package v1alpha1 is the v1alpha1 version of the API. - ## `AdmissionConfiguration` {#apiserver-k8s-io-v1alpha1-AdmissionConfiguration} +

AdmissionConfiguration provides versioned configuration for admission controllers.

-AdmissionConfiguration provides versioned configuration for admission controllers. - @@ -31,30 +30,24 @@ AdmissionConfiguration provides versioned configuration for admission controller - - +

Plugins allows specifying a configuration per admission control plugin.

+ - -
FieldDescription
apiVersion
string		apiserver.k8s.io/v1alpha1
kind
string		AdmissionConfiguration
plugins
[]AdmissionPluginConfiguration 		- Plugins allows specifying a configuration per admission control plugin.
- - ## `EgressSelectorConfiguration` {#apiserver-k8s-io-v1alpha1-EgressSelectorConfiguration} +

EgressSelectorConfiguration provides versioned configuration for egress selector clients.

-EgressSelectorConfiguration provides versioned configuration for egress selector clients. - @@ -62,30 +55,24 @@ EgressSelectorConfiguration provides versioned configuration for egress selector - - +

connectionServices contains a list of egress selection client configurations

+ - -
FieldDescription
apiVersion
string		apiserver.k8s.io/v1alpha1
kind
string		EgressSelectorConfiguration
egressSelections [Required]
[]EgressSelection 		- connectionServices contains a list of egress selection client configurations
- - ## `TracingConfiguration` {#apiserver-k8s-io-v1alpha1-TracingConfiguration} +

TracingConfiguration provides versioned configuration for tracing clients.

-TracingConfiguration provides versioned configuration for tracing clients. - @@ -93,346 +80,296 @@ TracingConfiguration provides versioned configuration for tracing clients. - - +The connection is insecure, and does not support TLS.

+ - - +

SamplingRatePerMillion is the number of samples to collect per million spans. +Defaults to 0.

+ - -
FieldDescription
apiVersion
string		apiserver.k8s.io/v1alpha1
kind
string		TracingConfiguration
endpoint
string 		- Endpoint of the collector that's running on the control-plane node. +

Endpoint of the collector that's running on the control-plane node. The APIServer uses the egressType ControlPlane when sending data to the collector. The syntax is defined in https://github.com/grpc/grpc/blob/master/doc/naming.md. Defaults to the otlpgrpc default, localhost:4317 -The connection is insecure, and does not support TLS.
samplingRatePerMillion
int32 		- SamplingRatePerMillion is the number of samples to collect per million spans. -Defaults to 0.
- - ## `AdmissionPluginConfiguration` {#apiserver-k8s-io-v1alpha1-AdmissionPluginConfiguration} - - **Appears in:** - [AdmissionConfiguration](#apiserver-k8s-io-v1alpha1-AdmissionConfiguration) -AdmissionPluginConfiguration provides the configuration for a single plug-in. +

AdmissionPluginConfiguration provides the configuration for a single plug-in.

+ - +

Name is the name of the admission controller. +It must match the registered admission plugin name.

+ - - +

Path is the path to a configuration file that contains the plugin's +configuration

+ - - +

Configuration is an embedded configuration object to be used as the plugin's +configuration. If present, it will be used instead of the path to the configuration file.

+ - -
FieldDescription
name [Required]
string 		- Name is the name of the admission controller. -It must match the registered admission plugin name.
path
string 		- Path is the path to a configuration file that contains the plugin's -configuration
configuration
-k8s.io/apimachinery/pkg/runtime.Unknown +k8s.io/apimachinery/pkg/runtime.Unknown 		- Configuration is an embedded configuration object to be used as the plugin's -configuration. If present, it will be used instead of the path to the configuration file.
- - ## `Connection` {#apiserver-k8s-io-v1alpha1-Connection} - - **Appears in:** - [EgressSelection](#apiserver-k8s-io-v1alpha1-EgressSelection) -Connection provides the configuration for a single egress selection client. +

Connection provides the configuration for a single egress selection client.

+ - +

Protocol is the protocol used to connect from client to the konnectivity server.

+ - - +

Transport defines the transport configurations we use to dial to the konnectivity server. +This is required if ProxyProtocol is HTTPConnect or GRPC.

+ - -
FieldDescription
proxyProtocol [Required]
ProtocolType 		- Protocol is the protocol used to connect from client to the konnectivity server.
transport
Transport 		- Transport defines the transport configurations we use to dial to the konnectivity server. -This is required if ProxyProtocol is HTTPConnect or GRPC.
- - ## `EgressSelection` {#apiserver-k8s-io-v1alpha1-EgressSelection} - - **Appears in:** - [EgressSelectorConfiguration](#apiserver-k8s-io-v1alpha1-EgressSelectorConfiguration) -EgressSelection provides the configuration for a single egress selection client. +

EgressSelection provides the configuration for a single egress selection client.

+ - +

name is the name of the egress selection. +Currently supported values are "controlplane", "master", "etcd" and "cluster" +The "master" egress selector is deprecated in favor of "controlplane"

+ - - +

connection is the exact information used to configure the egress selection

+ - -
FieldDescription
name [Required]
string 		- name is the name of the egress selection. -Currently supported values are "controlplane", "master", "etcd" and "cluster" -The "master" egress selector is deprecated in favor of "controlplane"
connection [Required]
Connection 		- connection is the exact information used to configure the egress selection
- - ## `ProtocolType` {#apiserver-k8s-io-v1alpha1-ProtocolType} (Alias of `string`) - **Appears in:** - [Connection](#apiserver-k8s-io-v1alpha1-Connection) -ProtocolType is a set of valid values for Connection.ProtocolType +

ProtocolType is a set of valid values for Connection.ProtocolType

- ## `TCPTransport` {#apiserver-k8s-io-v1alpha1-TCPTransport} - - **Appears in:** - [Transport](#apiserver-k8s-io-v1alpha1-Transport) -TCPTransport provides the information to connect to konnectivity server via TCP +

TCPTransport provides the information to connect to konnectivity server via TCP

+ - +

URL is the location of the konnectivity server to connect to. +As an example it might be "https://127.0.0.1:8131"

+ - - +

TLSConfig is the config needed to use TLS when connecting to konnectivity server

+ - -
FieldDescription
url [Required]
string 		- URL is the location of the konnectivity server to connect to. -As an example it might be "https://127.0.0.1:8131"
tlsConfig
TLSConfig 		- TLSConfig is the config needed to use TLS when connecting to konnectivity server
- - ## `TLSConfig` {#apiserver-k8s-io-v1alpha1-TLSConfig} - - **Appears in:** - [TCPTransport](#apiserver-k8s-io-v1alpha1-TCPTransport) -TLSConfig provides the authentication information to connect to konnectivity server -Only used with TCPTransport +

TLSConfig provides the authentication information to connect to konnectivity server +Only used with TCPTransport

+ - +If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.

+ - - +Must be configured if TCPTransport.URL is prefixed with https://

+ - - +Must be configured if TCPTransport.URL is prefixed with https://

+ - -
FieldDescription
caBundle
string 		- caBundle is the file location of the CA to be used to determine trust with the konnectivity server. +

caBundle is the file location of the CA to be used to determine trust with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// -If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.
clientKey
string 		- clientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server. +

clientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// -Must be configured if TCPTransport.URL is prefixed with https://
clientCert
string 		- clientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server. +

clientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// -Must be configured if TCPTransport.URL is prefixed with https://
- - ## `Transport` {#apiserver-k8s-io-v1alpha1-Transport} - - **Appears in:** - [Connection](#apiserver-k8s-io-v1alpha1-Connection) -Transport defines the transport configurations we use to dial to the konnectivity server +

Transport defines the transport configurations we use to dial to the konnectivity server

+ - +Requires at least one of TCP or UDS to be set

+ - - +

UDS is the UDS configuration for communicating with the konnectivity server via UDS +Requires at least one of TCP or UDS to be set

+ - -
FieldDescription
tcp
TCPTransport 		- TCP is the TCP configuration for communicating with the konnectivity server via TCP +

TCP is the TCP configuration for communicating with the konnectivity server via TCP ProxyProtocol of GRPC is not supported with TCP transport at the moment -Requires at least one of TCP or UDS to be set
uds
UDSTransport 		- UDS is the UDS configuration for communicating with the konnectivity server via UDS -Requires at least one of TCP or UDS to be set
- - ## `UDSTransport` {#apiserver-k8s-io-v1alpha1-UDSTransport} - - **Appears in:** - [Transport](#apiserver-k8s-io-v1alpha1-Transport) -UDSTransport provides the information to connect to konnectivity server via UDS +

UDSTransport provides the information to connect to konnectivity server via UDS

+ - +

UDSName is the name of the unix domain socket to connect to konnectivity server +This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)

+ - -
FieldDescription
udsName [Required]
string 		- UDSName is the name of the unix domain socket to connect to konnectivity server -This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)
- diff --git a/content/en/docs/reference/config-api/apiserver-encryption.v1.md b/content/en/docs/reference/config-api/apiserver-encryption.v1.md new file mode 100644 index 0000000000..3b09e817a5 --- /dev/null +++ b/content/en/docs/reference/config-api/apiserver-encryption.v1.md @@ -0,0 +1,279 @@ +--- +title: kube-apiserver Encryption Configuration (v1) +content_type: tool-reference +package: apiserver.config.k8s.io/v1 +auto_generated: true +--- +

Package v1 is the v1 version of the API.

+ + +## Resource Types + + +- [EncryptionConfiguration](#apiserver-config-k8s-io-v1-EncryptionConfiguration) + + + +## `EncryptionConfiguration` {#apiserver-config-k8s-io-v1-EncryptionConfiguration} + + + +

EncryptionConfiguration stores the complete configuration for encryption providers.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		apiserver.config.k8s.io/v1
kind
string		EncryptionConfiguration
resources [Required]
+[]ResourceConfiguration +		 +

resources is a list containing resources, and their corresponding encryption providers.

+
+ +## `AESConfiguration` {#apiserver-config-k8s-io-v1-AESConfiguration} + + +**Appears in:** + +- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration) + + +

AESConfiguration contains the API configuration for an AES transformer.

+ + + + + + + + + + + +
FieldDescription
keys [Required]
+[]Key +		 +

keys is a list of keys to be used for creating the AES transformer. +Each key has to be 32 bytes long for AES-CBC and 16, 24 or 32 bytes for AES-GCM.

+
+ +## `IdentityConfiguration` {#apiserver-config-k8s-io-v1-IdentityConfiguration} + + +**Appears in:** + +- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration) + + +

IdentityConfiguration is an empty struct to allow identity transformer in provider configuration.

+ + + + +## `KMSConfiguration` {#apiserver-config-k8s-io-v1-KMSConfiguration} + + +**Appears in:** + +- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration) + + +

KMSConfiguration contains the name, cache size and path to configuration file for a KMS based envelope transformer.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

name is the name of the KMS plugin to be used.

+
cachesize
+int32 +		 +

cachesize is the maximum number of secrets which are cached in memory. The default value is 1000. +Set to a negative value to disable caching.

+
endpoint [Required]
+string +		 +

endpoint is the gRPC server listening address, for example "unix:///var/run/kms-provider.sock".

+
timeout
+meta/v1.Duration +		 +

timeout for gRPC calls to kms-plugin (ex. 5s). The default is 3 seconds.

+
+ +## `Key` {#apiserver-config-k8s-io-v1-Key} + + +**Appears in:** + +- [AESConfiguration](#apiserver-config-k8s-io-v1-AESConfiguration) + +- [SecretboxConfiguration](#apiserver-config-k8s-io-v1-SecretboxConfiguration) + + +

Key contains name and secret of the provided key for a transformer.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

name is the name of the key to be used while storing data to disk.

+
secret [Required]
+string +		 +

secret is the actual key, encoded in base64.

+
+ +## `ProviderConfiguration` {#apiserver-config-k8s-io-v1-ProviderConfiguration} + + +**Appears in:** + +- [ResourceConfiguration](#apiserver-config-k8s-io-v1-ResourceConfiguration) + + +

ProviderConfiguration stores the provided configuration for an encryption provider.

+ + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
aesgcm [Required]
+AESConfiguration +		 +

aesgcm is the configuration for the AES-GCM transformer.

+
aescbc [Required]
+AESConfiguration +		 +

aescbc is the configuration for the AES-CBC transformer.

+
secretbox [Required]
+SecretboxConfiguration +		 +

secretbox is the configuration for the Secretbox based transformer.

+
identity [Required]
+IdentityConfiguration +		 +

identity is the (empty) configuration for the identity transformer.

+
kms [Required]
+KMSConfiguration +		 +

kms contains the name, cache size and path to configuration file for a KMS based envelope transformer.

+
+ +## `ResourceConfiguration` {#apiserver-config-k8s-io-v1-ResourceConfiguration} + + +**Appears in:** + +- [EncryptionConfiguration](#apiserver-config-k8s-io-v1-EncryptionConfiguration) + + +

ResourceConfiguration stores per resource configuration.

+ + + + + + + + + + + + + + +
FieldDescription
resources [Required]
+[]string +		 +

resources is a list of kubernetes resources which have to be encrypted.

+
providers [Required]
+[]ProviderConfiguration +		 +

providers is a list of transformers to be used for reading and writing the resources to disk. +eg: aesgcm, aescbc, secretbox, identity.

+
+ +## `SecretboxConfiguration` {#apiserver-config-k8s-io-v1-SecretboxConfiguration} + + +**Appears in:** + +- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration) + + +

SecretboxConfiguration contains the API configuration for an Secretbox transformer.

+ + + + + + + + + + + +
FieldDescription
keys [Required]
+[]Key +		 +

keys is a list of keys to be used for creating the Secretbox transformer. +Each key has to be 32 bytes long.

+
+ diff --git a/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md b/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md new file mode 100644 index 0000000000..2189c4910d --- /dev/null +++ b/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md @@ -0,0 +1,121 @@ +--- +title: Event Rate Limit Configuration (v1alpha1) +content_type: tool-reference +package: eventratelimit.admission.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [Configuration](#eventratelimit-admission-k8s-io-v1alpha1-Configuration) + + + +## `Configuration` {#eventratelimit-admission-k8s-io-v1alpha1-Configuration} + + + +

Configuration provides configuration for the EventRateLimit admission +controller.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		eventratelimit.admission.k8s.io/v1alpha1
kind
string		Configuration
limits [Required]
+[]Limit +		 +

limits are the limits to place on event queries received. +Limits can be placed on events received server-wide, per namespace, +per user, and per source+object. +At least one limit is required.

+
+ +## `Limit` {#eventratelimit-admission-k8s-io-v1alpha1-Limit} + + +**Appears in:** + +- [Configuration](#eventratelimit-admission-k8s-io-v1alpha1-Configuration) + + +

Limit is the configuration for a particular limit type

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
type [Required]
+LimitType +		 +

type is the type of limit to which this configuration applies

+
qps [Required]
+int32 +		 +

qps is the number of event queries per second that are allowed for this +type of limit. The qps and burst fields are used together to determine if +a particular event query is accepted. The qps determines how many queries +are accepted once the burst amount of queries has been exhausted.

+
burst [Required]
+int32 +		 +

burst is the burst number of event queries that are allowed for this type +of limit. The qps and burst fields are used together to determine if a +particular event query is accepted. The burst determines the maximum size +of the allowance granted for a particular bucket. For example, if the burst +is 10 and the qps is 3, then the admission control will accept 10 queries +before blocking any queries. Every second, 3 more queries will be allowed. +If some of that allowance is not used, then it will roll over to the next +second, until the maximum allowance of 10 is reached.

+
cacheSize
+int32 +		 +

cacheSize is the size of the LRU cache for this type of limit. If a bucket +is evicted from the cache, then the allowance for that bucket is reset. If +more queries are later received for an evicted bucket, then that bucket +will re-enter the cache with a clean slate, giving that bucket a full +allowance of burst queries.

+

The default cache size is 4096.

+

If limitType is 'server', then cacheSize is ignored.

+
+ +## `LimitType` {#eventratelimit-admission-k8s-io-v1alpha1-LimitType} + +(Alias of `string`) + +**Appears in:** + +- [Limit](#eventratelimit-admission-k8s-io-v1alpha1-Limit) + + +

LimitType is the type of the limit (e.g., per-namespace)

+ + + + \ No newline at end of file diff --git a/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md b/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md index fb45ca7b1a..f0c6a5b953 100644 --- a/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md +++ b/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md @@ -4,7 +4,8 @@ content_type: tool-reference package: apiserver.config.k8s.io/v1 auto_generated: true --- -Package v1 is the v1 version of the API. +

Package v1 is the v1 version of the API.

+ ## Resource Types @@ -13,15 +14,13 @@ Package v1 is the v1 version of the API. - ## `WebhookAdmission` {#apiserver-config-k8s-io-v1-WebhookAdmission} +

WebhookAdmission provides configuration for the webhook admission controller.

-WebhookAdmission provides configuration for the webhook admission controller. - @@ -29,18 +28,14 @@ WebhookAdmission provides configuration for the webhook admission controller. - - +

KubeConfigFile is the path to the kubeconfig file.

+ - -
FieldDescription
apiVersion
string		apiserver.config.k8s.io/v1
kind
string		WebhookAdmission
kubeConfigFile [Required]
string 		- KubeConfigFile is the path to the kubeconfig file.
- diff --git a/content/en/docs/reference/config-api/client-authentication.v1.md b/content/en/docs/reference/config-api/client-authentication.v1.md new file mode 100644 index 0000000000..f06c337fab --- /dev/null +++ b/content/en/docs/reference/config-api/client-authentication.v1.md @@ -0,0 +1,229 @@ +--- +title: Client Authentication (v1) +content_type: tool-reference +package: client.authentication.k8s.io/v1 +auto_generated: true +--- + + +## Resource Types + + +- [ExecCredential](#client-authentication-k8s-io-v1-ExecCredential) + + + +## `ExecCredential` {#client-authentication-k8s-io-v1-ExecCredential} + + + +

ExecCredential is used by exec-based plugins to communicate credentials to +HTTP transports.

+ + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		client.authentication.k8s.io/v1
kind
string		ExecCredential
spec [Required]
+ExecCredentialSpec +		 +

Spec holds information passed to the plugin by the transport.

+
status
+ExecCredentialStatus +		 +

Status is filled in by the plugin and holds the credentials that the transport +should use to contact the API.

+
+ +## `Cluster` {#client-authentication-k8s-io-v1-Cluster} + + +**Appears in:** + +- [ExecCredentialSpec](#client-authentication-k8s-io-v1-ExecCredentialSpec) + + +

Cluster contains information to allow an exec plugin to communicate +with the kubernetes cluster being authenticated to.

+

To ensure that this struct contains everything someone would need to communicate +with a kubernetes cluster (just like they would via a kubeconfig), the fields +should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception +of CertificateAuthority, since CA data will always be passed to the plugin as bytes.

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
server [Required]
+string +		 +

Server is the address of the kubernetes cluster (https://hostname:port).

+
tls-server-name
+string +		 +

TLSServerName is passed to the server for SNI and is used in the client to +check server certificates against. If ServerName is empty, the hostname +used to contact the server is used.

+
insecure-skip-tls-verify
+bool +		 +

InsecureSkipTLSVerify skips the validity check for the server's certificate. +This will make your HTTPS connections insecure.

+
certificate-authority-data
+[]byte +		 +

CAData contains PEM-encoded certificate authority certificates. +If empty, system roots should be used.

+
proxy-url
+string +		 +

ProxyURL is the URL to the proxy to be used for all requests to this +cluster.

+
config
+k8s.io/apimachinery/pkg/runtime.RawExtension +		 +

Config holds additional config data that is specific to the exec +plugin with regards to the cluster being authenticated to.

+

This data is sourced from the clientcmd Cluster object's +extensions[client.authentication.k8s.io/exec] field:

+

clusters:

+
    +
  • name: my-cluster +cluster: +... +extensions: +
      +
    • name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config +extension: +audience: 06e3fbd18de8 # arbitrary config
      • +
    +
    • +
+

In some environments, the user config may be exactly the same across many clusters +(i.e. call this exec plugin) minus some details that are specific to each cluster +such as the audience. This field allows the per cluster config to be directly +specified with the cluster info. Using this field to store secret data is not +recommended as one of the prime benefits of exec plugins is that no secrets need +to be stored directly in the kubeconfig.

+
+ +## `ExecCredentialSpec` {#client-authentication-k8s-io-v1-ExecCredentialSpec} + + +**Appears in:** + +- [ExecCredential](#client-authentication-k8s-io-v1-ExecCredential) + + +

ExecCredentialSpec holds request and runtime specific information provided by +the transport.

+ + + + + + + + + + + + + + +
FieldDescription
cluster
+Cluster +		 +

Cluster contains information to allow an exec plugin to communicate with the +kubernetes cluster being authenticated to. Note that Cluster is non-nil only +when provideClusterInfo is set to true in the exec provider config (i.e., +ExecConfig.ProvideClusterInfo).

+
interactive [Required]
+bool +		 +

Interactive declares whether stdin has been passed to this exec plugin.

+
+ +## `ExecCredentialStatus` {#client-authentication-k8s-io-v1-ExecCredentialStatus} + + +**Appears in:** + +- [ExecCredential](#client-authentication-k8s-io-v1-ExecCredential) + + +

ExecCredentialStatus holds credentials for the transport to use.

+

Token and ClientKeyData are sensitive fields. This data should only be +transmitted in-memory between client and exec plugin process. Exec plugin +itself should at least be protected via file permissions.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
expirationTimestamp
+meta/v1.Time +		 +

ExpirationTimestamp indicates a time when the provided credentials expire.

+
token [Required]
+string +		 +

Token is a bearer token used by the client for request authentication.

+
clientCertificateData [Required]
+string +		 +

PEM-encoded client TLS certificates (including intermediates, if any).

+
clientKeyData [Required]
+string +		 +

PEM-encoded private key for the above certificate.

+
+ diff --git a/content/en/docs/reference/config-api/client-authentication.v1beta1.md b/content/en/docs/reference/config-api/client-authentication.v1beta1.md index d018fb208f..2e840b5805 100644 --- a/content/en/docs/reference/config-api/client-authentication.v1beta1.md +++ b/content/en/docs/reference/config-api/client-authentication.v1beta1.md @@ -13,16 +13,14 @@ auto_generated: true - ## `ExecCredential` {#client-authentication-k8s-io-v1beta1-ExecCredential} +

ExecCredential is used by exec-based plugins to communicate credentials to +HTTP transports.

-ExecCredential is used by exec-based plugins to communicate credentials to -HTTP transports. - @@ -30,231 +28,202 @@ HTTP transports. - - +

Spec holds information passed to the plugin by the transport.

+ - - +

Status is filled in by the plugin and holds the credentials that the transport +should use to contact the API.

+ - -
FieldDescription
apiVersion
string		client.authentication.k8s.io/v1beta1
kind
string		ExecCredential
spec [Required]
ExecCredentialSpec 		- Spec holds information passed to the plugin by the transport.
status
ExecCredentialStatus 		- Status is filled in by the plugin and holds the credentials that the transport -should use to contact the API.
- - ## `Cluster` {#client-authentication-k8s-io-v1beta1-Cluster} - - **Appears in:** - [ExecCredentialSpec](#client-authentication-k8s-io-v1beta1-ExecCredentialSpec) -Cluster contains information to allow an exec plugin to communicate -with the kubernetes cluster being authenticated to. - -To ensure that this struct contains everything someone would need to communicate +

Cluster contains information to allow an exec plugin to communicate +with the kubernetes cluster being authenticated to.

+

To ensure that this struct contains everything someone would need to communicate with a kubernetes cluster (just like they would via a kubeconfig), the fields -should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception -of CertificateAuthority, since CA data will always be passed to the plugin as bytes. +should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception +of CertificateAuthority, since CA data will always be passed to the plugin as bytes.

+ - +

Server is the address of the kubernetes cluster (https://hostname:port).

+ - - +used to contact the server is used.

+ - - +

InsecureSkipTLSVerify skips the validity check for the server's certificate. +This will make your HTTPS connections insecure.

+ - - +

CAData contains PEM-encoded certificate authority certificates. +If empty, system roots should be used.

+ - - +

ProxyURL is the URL to the proxy to be used for all requests to this +cluster.

+ - - +to be stored directly in the kubeconfig.

+ - -
FieldDescription
server [Required]
string 		- Server is the address of the kubernetes cluster (https://hostname:port).
tls-server-name
string 		- TLSServerName is passed to the server for SNI and is used in the client to +

TLSServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname -used to contact the server is used.
insecure-skip-tls-verify
bool 		- InsecureSkipTLSVerify skips the validity check for the server's certificate. -This will make your HTTPS connections insecure.
certificate-authority-data
[]byte 		- CAData contains PEM-encoded certificate authority certificates. -If empty, system roots should be used.
proxy-url
string 		- ProxyURL is the URL to the proxy to be used for all requests to this -cluster.
config
-k8s.io/apimachinery/pkg/runtime.RawExtension +k8s.io/apimachinery/pkg/runtime.RawExtension 		- Config holds additional config data that is specific to the exec -plugin with regards to the cluster being authenticated to. - -This data is sourced from the clientcmd Cluster object's -extensions[client.authentication.k8s.io/exec] field: - -clusters: -- name: my-cluster - cluster: - ... - extensions: - - name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config - extension: - audience: 06e3fbd18de8 # arbitrary config - -In some environments, the user config may be exactly the same across many clusters +

Config holds additional config data that is specific to the exec +plugin with regards to the cluster being authenticated to.

+

This data is sourced from the clientcmd Cluster object's +extensions[client.authentication.k8s.io/exec] field:

+

clusters:

+
    +
  • name: my-cluster +cluster: +... +extensions: +
      +
    • name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config +extension: +audience: 06e3fbd18de8 # arbitrary config
      • +
    +
    • +
+

In some environments, the user config may be exactly the same across many clusters (i.e. call this exec plugin) minus some details that are specific to each cluster such as the audience. This field allows the per cluster config to be directly specified with the cluster info. Using this field to store secret data is not recommended as one of the prime benefits of exec plugins is that no secrets need -to be stored directly in the kubeconfig.
- - ## `ExecCredentialSpec` {#client-authentication-k8s-io-v1beta1-ExecCredentialSpec} - - **Appears in:** - [ExecCredential](#client-authentication-k8s-io-v1beta1-ExecCredential) -ExecCredentialSpec holds request and runtime specific information provided by -the transport. +

ExecCredentialSpec holds request and runtime specific information provided by +the transport.

+ - +ExecConfig.ProvideClusterInfo).

+ - - +

Interactive declares whether stdin has been passed to this exec plugin.

+ - -
FieldDescription
cluster
Cluster 		- Cluster contains information to allow an exec plugin to communicate with the +

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to. Note that Cluster is non-nil only when provideClusterInfo is set to true in the exec provider config (i.e., -ExecConfig.ProvideClusterInfo).
interactive [Required]
bool 		- Interactive declares whether stdin has been passed to this exec plugin.
- - ## `ExecCredentialStatus` {#client-authentication-k8s-io-v1beta1-ExecCredentialStatus} - - **Appears in:** - [ExecCredential](#client-authentication-k8s-io-v1beta1-ExecCredential) -ExecCredentialStatus holds credentials for the transport to use. - -Token and ClientKeyData are sensitive fields. This data should only be +

ExecCredentialStatus holds credentials for the transport to use.

+

Token and ClientKeyData are sensitive fields. This data should only be transmitted in-memory between client and exec plugin process. Exec plugin -itself should at least be protected via file permissions. +itself should at least be protected via file permissions.

+ - +

ExpirationTimestamp indicates a time when the provided credentials expire.

+ - - +

Token is a bearer token used by the client for request authentication.

+ - - +

PEM-encoded client TLS certificates (including intermediates, if any).

+ - - +

PEM-encoded private key for the above certificate.

+ - -
FieldDescription
expirationTimestamp
-meta/v1.Time +meta/v1.Time 		- ExpirationTimestamp indicates a time when the provided credentials expire.
token [Required]
string 		- Token is a bearer token used by the client for request authentication.
clientCertificateData [Required]
string 		- PEM-encoded client TLS certificates (including intermediates, if any).
clientKeyData [Required]
string 		- PEM-encoded private key for the above certificate.
- diff --git a/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md b/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md new file mode 100644 index 0000000000..f420623559 --- /dev/null +++ b/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md @@ -0,0 +1,168 @@ +--- +title: Image Policy API (v1alpha1) +content_type: tool-reference +package: imagepolicy.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + + +## `ImageReview` {#imagepolicy-k8s-io-v1alpha1-ImageReview} + + + +

ImageReview checks if the set of images in a pod are allowed.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		imagepolicy.k8s.io/v1alpha1
kind
string		ImageReview
metadata
+meta/v1.ObjectMeta +		 +

Standard object's metadata. +More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

+Refer to the Kubernetes API documentation for the fields of the metadata field.
spec [Required]
+ImageReviewSpec +		 +

Spec holds information about the pod being evaluated

+
status
+ImageReviewStatus +		 +

Status is filled in by the backend and indicates whether the pod should be allowed.

+
+ +## `ImageReviewContainerSpec` {#imagepolicy-k8s-io-v1alpha1-ImageReviewContainerSpec} + + +**Appears in:** + +- [ImageReviewSpec](#imagepolicy-k8s-io-v1alpha1-ImageReviewSpec) + + +

ImageReviewContainerSpec is a description of a container within the pod creation request.

+ + + + + + + + + + + +
FieldDescription
image
+string +		 +

This can be in the form image:tag or image@SHA:012345679abcdef.

+
+ +## `ImageReviewSpec` {#imagepolicy-k8s-io-v1alpha1-ImageReviewSpec} + + +**Appears in:** + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + +

ImageReviewSpec is a description of the pod creation request.

+ + + + + + + + + + + + + + + + + +
FieldDescription
containers
+[]ImageReviewContainerSpec +		 +

Containers is a list of a subset of the information in each container of the Pod being created.

+
annotations
+map[string]string +		 +

Annotations is a list of key-value pairs extracted from the Pod's annotations. +It only includes keys which match the pattern *.image-policy.k8s.io/*. +It is up to each webhook backend to determine how to interpret these annotations, if at all.

+
namespace
+string +		 +

Namespace is the namespace the pod is being created in.

+
+ +## `ImageReviewStatus` {#imagepolicy-k8s-io-v1alpha1-ImageReviewStatus} + + +**Appears in:** + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + +

ImageReviewStatus is the result of the review for the pod creation request.

+ + + + + + + + + + + + + + + + + +
FieldDescription
allowed [Required]
+bool +		 +

Allowed indicates that all images were allowed to be run.

+
reason
+string +		 +

Reason should be empty unless Allowed is false in which case it +may contain a short description of what is wrong. Kubernetes +may truncate excessively long errors when displaying to the user.

+
auditAnnotations
+map[string]string +		 +

AuditAnnotations will be added to the attributes object of the +admission controller request using 'AddAnnotation'. The keys should +be prefix-less (i.e., the admission controller will add an +appropriate prefix).

+
+ \ No newline at end of file diff --git a/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md b/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md index 94209488fe..8c794f2755 100644 --- a/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md +++ b/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md @@ -13,16 +13,14 @@ auto_generated: true - ## `KubeProxyConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration} +

KubeProxyConfiguration contains everything necessary to configure the +Kubernetes proxy server.

-KubeProxyConfiguration contains everything necessary to configure the -Kubernetes proxy server. - @@ -30,509 +28,502 @@ Kubernetes proxy server. - - +

featureGates is a map of feature names to bools that enable or disable alpha/experimental features.

+ - - +

bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0 +for all interfaces)

+ - - +

healthzBindAddress is the IP address and port for the health check server to serve on, +defaulting to 0.0.0.0:10256

+ - - +

metricsBindAddress is the IP address and port for the metrics server to serve on, +defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)

+ - - +

bindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit

+ - - +

enableProfiling enables profiling via web interface on /debug/pprof handler. +Profiling handlers will be handled by metrics server.

+ - - +no off-cluster bridging will be performed.

+ - - +

hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.

+ - - +

clientConnection specifies the kubeconfig file and client connection settings for the proxy +server to use when communicating with the apiserver.

+ - - +

iptables contains iptables-related configuration options.

+ - - +

ipvs contains ipvs-related configuration options.

+ - - +

oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within +the range [-1000, 1000]

+ - - +

mode specifies which proxy mode to use.

+ - - +

portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed +in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.

+ - - +

udpIdleTimeout is how long an idle UDP connection will be kept open (e.g. '250ms', '2s'). +Must be greater than 0. Only applicable for proxyMode=userspace.

+ - - +

conntrack contains conntrack-related configuration options.

+ - - +

configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater +than 0.

+ - - +An empty string slice is meant to select all network interfaces.

+ - - +

winkernel contains winkernel-related configuration options.

+ - - +

ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.

+ - - +

DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR

+ + + + - -
FieldDescription
apiVersion
string		kubeproxy.config.k8s.io/v1alpha1
kind
string		KubeProxyConfiguration
featureGates [Required]
map[string]bool 		- featureGates is a map of feature names to bools that enable or disable alpha/experimental features.
bindAddress [Required]
string 		- bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0 -for all interfaces)
healthzBindAddress [Required]
string 		- healthzBindAddress is the IP address and port for the health check server to serve on, -defaulting to 0.0.0.0:10256
metricsBindAddress [Required]
string 		- metricsBindAddress is the IP address and port for the metrics server to serve on, -defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)
bindAddressHardFail [Required]
bool 		- bindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit
enableProfiling [Required]
bool 		- enableProfiling enables profiling via web interface on /debug/pprof handler. -Profiling handlers will be handled by metrics server.
clusterCIDR [Required]
string 		- clusterCIDR is the CIDR range of the pods in the cluster. It is used to +

clusterCIDR is the CIDR range of the pods in the cluster. It is used to bridge traffic coming from outside of the cluster. If not provided, -no off-cluster bridging will be performed.
hostnameOverride [Required]
string 		- hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.
clientConnection [Required]
ClientConnectionConfiguration 		- clientConnection specifies the kubeconfig file and client connection settings for the proxy -server to use when communicating with the apiserver.
iptables [Required]
KubeProxyIPTablesConfiguration 		- iptables contains iptables-related configuration options.
ipvs [Required]
KubeProxyIPVSConfiguration 		- ipvs contains ipvs-related configuration options.
oomScoreAdj [Required]
int32 		- oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within -the range [-1000, 1000]
mode [Required]
ProxyMode 		- mode specifies which proxy mode to use.
portRange [Required]
string 		- portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed -in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.
udpIdleTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- udpIdleTimeout is how long an idle UDP connection will be kept open (e.g. '250ms', '2s'). -Must be greater than 0. Only applicable for proxyMode=userspace.
conntrack [Required]
KubeProxyConntrackConfiguration 		- conntrack contains conntrack-related configuration options.
configSyncPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater -than 0.
nodePortAddresses [Required]
[]string 		- nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid +

nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid IP blocks. These values are as a parameter to select the interfaces where nodeport works. In case someone would like to expose a service on localhost for local visit and some other interfaces for particular purpose, a list of IP blocks would do that. -If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort. +If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort. If set it to a non-zero IP block, kube-proxy will filter that down to just the IPs that applied to the node. -An empty string slice is meant to select all network interfaces.
winkernel [Required]
KubeProxyWinkernelConfiguration 		- winkernel contains winkernel-related configuration options.
showHiddenMetricsForVersion [Required]
string 		- ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.
detectLocalMode [Required]
LocalMode 		- DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR
detectLocal [Required]
+DetectLocalConfiguration +		 +

DetectLocal contains optional configuration settings related to DetectLocalMode.

+
+ +## `DetectLocalConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-DetectLocalConfiguration} - -## `KubeProxyConntrackConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConntrackConfiguration} - - - - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -KubeProxyConntrackConfiguration contains conntrack settings for -the Kubernetes proxy server. +

DetectLocalConfiguration contains optional settings related to DetectLocalMode option

+ + + + + + + + + +
FieldDescription
bridgeInterface [Required]
+string +		 +

BridgeInterface is a string argument which represents a single bridge interface name. +Kube-proxy considers traffic as local if originating from this given bridge. +This argument should be set if DetectLocalMode is set to LocalModeBridgeInterface.

+
interfaceNamePrefix [Required]
+string +		 +

InterfaceNamePrefix is a string argument which represents a single interface prefix name. +Kube-proxy considers traffic as local if originating from one or more interfaces which match +the given prefix. This argument should be set if DetectLocalMode is set to LocalModeInterfaceNamePrefix.

+
+## `KubeProxyConntrackConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConntrackConfiguration} + + +**Appears in:** + +- [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) + + +

KubeProxyConntrackConfiguration contains conntrack settings for +the Kubernetes proxy server.

+ + + + + + +

maxPerCore is the maximum number of NAT connections to track +per CPU core (0 to leave the limit as-is and ignore min).

+ - - +

min is the minimum value of connect-tracking records to allocate, +regardless of conntrackMaxPerCore (set maxPerCore=0 to leave the limit as-is).

+ - - +

tcpEstablishedTimeout is how long an idle TCP connection will be kept open +(e.g. '2s'). Must be greater than 0 to set.

+ - - +table. (e.g. '60s'). Must be greater than 0 to set.

+ - -
FieldDescription
maxPerCore [Required]
int32 		- maxPerCore is the maximum number of NAT connections to track -per CPU core (0 to leave the limit as-is and ignore min).
min [Required]
int32 		- min is the minimum value of connect-tracking records to allocate, -regardless of conntrackMaxPerCore (set maxPerCore=0 to leave the limit as-is).
tcpEstablishedTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- tcpEstablishedTimeout is how long an idle TCP connection will be kept open -(e.g. '2s'). Must be greater than 0 to set.
tcpCloseWaitTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- tcpCloseWaitTimeout is how long an idle conntrack entry +

tcpCloseWaitTimeout is how long an idle conntrack entry in CLOSE_WAIT state will remain in the conntrack -table. (e.g. '60s'). Must be greater than 0 to set.
- - ## `KubeProxyIPTablesConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyIPTablesConfiguration} - - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -KubeProxyIPTablesConfiguration contains iptables-related configuration -details for the Kubernetes proxy server. +

KubeProxyIPTablesConfiguration contains iptables-related configuration +details for the Kubernetes proxy server.

+ - +

masqueradeBit is the bit of the iptables fwmark space to use for SNAT if using +the pure iptables proxy mode. Values must be within the range [0, 31].

+ - - +

masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode.

+ - - +

syncPeriod is the period that iptables rules are refreshed (e.g. '5s', '1m', +'2h22m'). Must be greater than 0.

+ - - +

minSyncPeriod is the minimum period that iptables rules are refreshed (e.g. '5s', '1m', +'2h22m').

+ - -
FieldDescription
masqueradeBit [Required]
int32 		- masqueradeBit is the bit of the iptables fwmark space to use for SNAT if using -the pure iptables proxy mode. Values must be within the range [0, 31].
masqueradeAll [Required]
bool 		- masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode.
syncPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- syncPeriod is the period that iptables rules are refreshed (e.g. '5s', '1m', -'2h22m'). Must be greater than 0.
minSyncPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- minSyncPeriod is the minimum period that iptables rules are refreshed (e.g. '5s', '1m', -'2h22m').
- - ## `KubeProxyIPVSConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyIPVSConfiguration} - - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -KubeProxyIPVSConfiguration contains ipvs-related configuration -details for the Kubernetes proxy server. +

KubeProxyIPVSConfiguration contains ipvs-related configuration +details for the Kubernetes proxy server.

+ - +

syncPeriod is the period that ipvs rules are refreshed (e.g. '5s', '1m', +'2h22m'). Must be greater than 0.

+ - - +

minSyncPeriod is the minimum period that ipvs rules are refreshed (e.g. '5s', '1m', +'2h22m').

+ - - +

ipvs scheduler

+ - - +

excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch +when cleaning up ipvs services.

+ - - +

strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries +from kube-ipvs0 interface

+ - - +

tcpTimeout is the timeout value used for idle IPVS TCP sessions. +The default value is 0, which preserves the current timeout value on the system.

+ - - +

tcpFinTimeout is the timeout value used for IPVS TCP sessions after receiving a FIN. +The default value is 0, which preserves the current timeout value on the system.

+ - - +

udpTimeout is the timeout value used for IPVS UDP packets. +The default value is 0, which preserves the current timeout value on the system.

+ - -
FieldDescription
syncPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- syncPeriod is the period that ipvs rules are refreshed (e.g. '5s', '1m', -'2h22m'). Must be greater than 0.
minSyncPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- minSyncPeriod is the minimum period that ipvs rules are refreshed (e.g. '5s', '1m', -'2h22m').
scheduler [Required]
string 		- ipvs scheduler
excludeCIDRs [Required]
[]string 		- excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch -when cleaning up ipvs services.
strictARP [Required]
bool 		- strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries -from kube-ipvs0 interface
tcpTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- tcpTimeout is the timeout value used for idle IPVS TCP sessions. -The default value is 0, which preserves the current timeout value on the system.
tcpFinTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- tcpFinTimeout is the timeout value used for IPVS TCP sessions after receiving a FIN. -The default value is 0, which preserves the current timeout value on the system.
udpTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- udpTimeout is the timeout value used for IPVS UDP packets. -The default value is 0, which preserves the current timeout value on the system.
- - ## `KubeProxyWinkernelConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-KubeProxyWinkernelConfiguration} - - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -KubeProxyWinkernelConfiguration contains Windows/HNS settings for -the Kubernetes proxy server. +

KubeProxyWinkernelConfiguration contains Windows/HNS settings for +the Kubernetes proxy server.

+ - +

networkName is the name of the network kube-proxy will use +to create endpoints and policies

+ - - +

sourceVip is the IP address of the source VIP endoint used for +NAT when loadbalancing

+ - - +

enableDSR tells kube-proxy whether HNS policies should be created +with DSR

+ + + + + + + - -
FieldDescription
networkName [Required]
string 		- networkName is the name of the network kube-proxy will use -to create endpoints and policies
sourceVip [Required]
string 		- sourceVip is the IP address of the source VIP endoint used for -NAT when loadbalancing
enableDSR [Required]
bool 		- enableDSR tells kube-proxy whether HNS policies should be created -with DSR
rootHnsEndpointName [Required]
+string +		 +

RootHnsEndpointName is the name of hnsendpoint that is attached to +l2bridge for root network namespace

+
forwardHealthCheckVip [Required]
+bool +		 +

ForwardHealthCheckVip forwards service VIP for health check port on +Windows

+
- - ## `LocalMode` {#kubeproxy-config-k8s-io-v1alpha1-LocalMode} (Alias of `string`) - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -LocalMode represents modes to detect local traffic from the node +

LocalMode represents modes to detect local traffic from the node

- ## `ProxyMode` {#kubeproxy-config-k8s-io-v1alpha1-ProxyMode} (Alias of `string`) - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) -ProxyMode represents modes used by the Kubernetes proxy server. - -Currently, three modes of proxy are available in Linux platform: 'userspace' (older, going to be EOL), 'iptables' -(newer, faster), 'ipvs'(newest, better in performance and scalability). - -Two modes of proxy are available in Windows platform: 'userspace'(older, stable) and 'kernelspace' (newer, faster). - -In Linux platform, if proxy mode is blank, use the best-available proxy (currently iptables, but may change in the +

ProxyMode represents modes used by the Kubernetes proxy server.

+

Currently, three modes of proxy are available in Linux platform: 'userspace' (older, going to be EOL), 'iptables' +(newer, faster), 'ipvs'(newest, better in performance and scalability).

+

Two modes of proxy are available in Windows platform: 'userspace'(older, stable) and 'kernelspace' (newer, faster).

+

In Linux platform, if proxy mode is blank, use the best-available proxy (currently iptables, but may change in the future). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to 'ipvs', -and the fall back path is firstly iptables and then userspace. - -In Windows platform, if proxy mode is blank, use the best-available proxy (currently userspace, but may change in the +and the fall back path is firstly iptables and then userspace.

+

In Windows platform, if proxy mode is blank, use the best-available proxy (currently userspace, but may change in the future). If winkernel proxy is selected, regardless of how, but the Windows kernel can't support this mode of proxy, -this always falls back to the userspace proxy. +this always falls back to the userspace proxy.

+ - @@ -540,241 +531,328 @@ this always falls back to the userspace proxy. ## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} - - **Appears in:** - [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + - [GenericControllerManagerConfiguration](#controllermanager-config-k8s-io-v1alpha1-GenericControllerManagerConfiguration) -ClientConnectionConfiguration contains details for constructing a client. +

ClientConnectionConfiguration contains details for constructing a client.

+ - +

kubeconfig is the path to a KubeConfig file.

+ - - +client.

+ - - +

contentType is the content type used when sending data to the server from this client.

+ - - +

qps controls the number of queries per second allowed for this connection.

+ - - +

burst allows extra queries to accumulate when a client is exceeding its rate.

+ - -
FieldDescription
kubeconfig [Required]
string 		- kubeconfig is the path to a KubeConfig file.
acceptContentTypes [Required]
string 		- acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the +

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular -client.
contentType [Required]
string 		- contentType is the content type used when sending data to the server from this client.
qps [Required]
float32 		- qps controls the number of queries per second allowed for this connection.
burst [Required]
int32 		- burst allows extra queries to accumulate when a client is exceeding its rate.
## `DebuggingConfiguration` {#DebuggingConfiguration} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + - [GenericControllerManagerConfiguration](#controllermanager-config-k8s-io-v1alpha1-GenericControllerManagerConfiguration) -DebuggingConfiguration holds configuration for Debugging related features. +

DebuggingConfiguration holds configuration for Debugging related features.

+ - +

enableProfiling enables profiling via web interface host:port/debug/pprof/

+ - - +

enableContentionProfiling enables lock contention profiling, if +enableProfiling is true.

+ + +
FieldDescription
enableProfiling [Required]
bool 		- enableProfiling enables profiling via web interface host:port/debug/pprof/
enableContentionProfiling [Required]
bool 		- enableContentionProfiling enables lock contention profiling, if -enableProfiling is true.
+ +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

FormatOptions contains options for the different logging formats.

+ + + + + + + + + +
FieldDescription
json [Required]
+JSONOptions +		 +

[Experimental] JSON contains options for logging format "json".

+
+ +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

JSONOptions contains options for logging format "json".

+ + + + + + + + + + + + +
FieldDescription
splitStream [Required]
+bool +		 +

[Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

+
infoBufferSize [Required]
+k8s.io/apimachinery/pkg/api/resource.QuantityValue +		 +

[Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

+
## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + - [GenericControllerManagerConfiguration](#controllermanager-config-k8s-io-v1alpha1-GenericControllerManagerConfiguration) -LeaderElectionConfiguration defines the configuration of leader election -clients for components that can run with leader election enabled. +

LeaderElectionConfiguration defines the configuration of leader election +clients for components that can run with leader election enabled.

+ - +components for high availability.

+ - - +enabled.

+ - - +election is enabled.

+ - - +leader election is enabled.

+ - - +

resourceLock indicates the resource object type that will be used to lock +during leader election cycles.

+ - - +

resourceName indicates the name of resource object that will be used to lock +during leader election cycles.

+ - - +

resourceName indicates the namespace of resource object that will be used to lock +during leader election cycles.

+ - -
FieldDescription
leaderElect [Required]
bool 		- leaderElect enables a leader election client to gain leadership +

leaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated -components for high availability.
leaseDuration [Required]
-meta/v1.Duration +meta/v1.Duration 		- leaseDuration is the duration that non-leader candidates will wait +

leaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is -enabled.
renewDeadline [Required]
-meta/v1.Duration +meta/v1.Duration 		- renewDeadline is the interval between attempts by the acting master to +

renewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader -election is enabled.
retryPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- retryPeriod is the duration the clients should wait between attempting +

retryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if -leader election is enabled.
resourceLock [Required]
string 		- resourceLock indicates the resource object type that will be used to lock -during leader election cycles.
resourceName [Required]
string 		- resourceName indicates the name of resource object that will be used to lock -during leader election cycles.
resourceNamespace [Required]
string 		- resourceName indicates the namespace of resource object that will be used to lock -during leader election cycles.
## `LoggingConfiguration` {#LoggingConfiguration} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) -LoggingConfiguration contains logging options -Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information. +

LoggingConfiguration contains logging options +Refer Logs Options for more information.

+ - +

Format Flag specifies the structure of log messages. +default value of format is text

+ - - - +

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

+ + + + + + + + + + - -
FieldDescription
format [Required]
string 		- Format Flag specifies the structure of log messages. -default value of format is `text`
sanitization [Required]
-bool +
flushFrequency [Required]
+time.Duration 		- [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)
verbosity [Required]
+uint32 +		 +

Verbosity is the threshold that determines which log messages are +logged. Default is zero which logs only the most important +messages. Higher values enable additional messages. Error messages +are always logged.

+
vmodule [Required]
+VModuleConfiguration +		 +

VModule overrides the verbosity threshold for individual files. +Only supported for "text" log format.

+
options [Required]
+FormatOptions +		 +

[Experimental] Options holds additional parameters that are specific +to the different logging formats. Only the options for the selected +format get used, but all of them get validated.

+
+ +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

+ + + diff --git a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta1.md b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta1.md deleted file mode 100644 index ac32e65674..0000000000 --- a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta1.md +++ /dev/null @@ -1,2156 +0,0 @@ ---- -title: kube-scheduler Configuration (v1beta1) -content_type: tool-reference -package: kubescheduler.config.k8s.io/v1 -auto_generated: true ---- - - -## Resource Types - - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) -- [DefaultPreemptionArgs](#kubescheduler-config-k8s-io-v1beta1-DefaultPreemptionArgs) -- [InterPodAffinityArgs](#kubescheduler-config-k8s-io-v1beta1-InterPodAffinityArgs) -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) -- [NodeAffinityArgs](#kubescheduler-config-k8s-io-v1beta1-NodeAffinityArgs) -- [NodeLabelArgs](#kubescheduler-config-k8s-io-v1beta1-NodeLabelArgs) -- [NodeResourcesFitArgs](#kubescheduler-config-k8s-io-v1beta1-NodeResourcesFitArgs) -- [NodeResourcesLeastAllocatedArgs](#kubescheduler-config-k8s-io-v1beta1-NodeResourcesLeastAllocatedArgs) -- [NodeResourcesMostAllocatedArgs](#kubescheduler-config-k8s-io-v1beta1-NodeResourcesMostAllocatedArgs) -- [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta1-PodTopologySpreadArgs) -- [RequestedToCapacityRatioArgs](#kubescheduler-config-k8s-io-v1beta1-RequestedToCapacityRatioArgs) -- [ServiceAffinityArgs](#kubescheduler-config-k8s-io-v1beta1-ServiceAffinityArgs) -- [VolumeBindingArgs](#kubescheduler-config-k8s-io-v1beta1-VolumeBindingArgs) - - - - -## `Policy` {#kubescheduler-config-k8s-io-v1-Policy} - - - - - -Policy describes a struct for a policy resource used in api. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1
kind
string		Policy
predicates [Required]
-[]PredicatePolicy -		 - Holds the information to configure the fit predicate functions
priorities [Required]
-[]PriorityPolicy -		 - Holds the information to configure the priority functions
extenders [Required]
-[]LegacyExtender -		 - Holds the information to communicate with the extender(s)
hardPodAffinitySymmetricWeight [Required]
-int32 -		 - RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule -corresponding to every RequiredDuringScheduling affinity rule. -HardPodAffinitySymmetricWeight represents the weight of implicit PreferredDuringScheduling affinity rule, in the range 1-100.
alwaysCheckAllPredicates [Required]
-bool -		 - When AlwaysCheckAllPredicates is set to true, scheduler checks all -the configured predicates even after one or more of them fails. -When the flag is set to false, scheduler skips checking the rest -of the predicates after it finds one predicate that failed.
- - - -## `ExtenderManagedResource` {#kubescheduler-config-k8s-io-v1-ExtenderManagedResource} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta1-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderManagedResource describes the arguments of extended resources -managed by an extender. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name is the extended resource name.
ignoredByScheduler [Required]
-bool -		 - IgnoredByScheduler indicates whether kube-scheduler should ignore this -resource when applying predicates.
- - - -## `ExtenderTLSConfig` {#kubescheduler-config-k8s-io-v1-ExtenderTLSConfig} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta1-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderTLSConfig contains settings to enable TLS with extender - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
insecure [Required]
-bool -		 - Server should be accessed without verifying the TLS certificate. For testing only.
serverName [Required]
-string -		 - ServerName is passed to the server for SNI and is used in the client to check server -certificates against. If ServerName is empty, the hostname used to contact the -server is used.
certFile [Required]
-string -		 - Server requires TLS client certificate authentication
keyFile [Required]
-string -		 - Server requires TLS client certificate authentication
caFile [Required]
-string -		 - Trusted root certificates for server
certData [Required]
-[]byte -		 - CertData holds PEM-encoded bytes (typically read from a client certificate file). -CertData takes precedence over CertFile
keyData [Required]
-[]byte -		 - KeyData holds PEM-encoded bytes (typically read from a client certificate key file). -KeyData takes precedence over KeyFile
caData [Required]
-[]byte -		 - CAData holds PEM-encoded bytes (typically read from a root certificates bundle). -CAData takes precedence over CAFile
- - - -## `LabelPreference` {#kubescheduler-config-k8s-io-v1-LabelPreference} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -LabelPreference holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
presence [Required]
-bool -		 - This is a boolean flag -If true, higher priority is given to nodes that have the label -If false, higher priority is given to nodes that do not have the label
- - - -## `LabelsPresence` {#kubescheduler-config-k8s-io-v1-LabelsPresence} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -LabelsPresence holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should be either present (or absent) for the node to be considered a fit for hosting the pod
presence [Required]
-bool -		 - The boolean flag that indicates whether the labels should be present or absent from the node
- - - -## `LegacyExtender` {#kubescheduler-config-k8s-io-v1-LegacyExtender} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -LegacyExtender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, -it is assumed that the extender chose not to provide that extension. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
urlPrefix [Required]
-string -		 - URLPrefix at which the extender is available
filterVerb [Required]
-string -		 - Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb [Required]
-string -		 - Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb [Required]
-string -		 - Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the prioritize call generates. -The weight should be a positive integer
bindVerb [Required]
-string -		 - Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. -If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender -can implement this function.
enableHttps [Required]
-bool -		 - EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig [Required]
-ExtenderTLSConfig -		 - TLSConfig specifies the transport layer security config
httpTimeout [Required]
-time.Duration -		 - HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize -timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable [Required]
-bool -		 - NodeCacheCapable specifies that the extender is capable of caching node information, -so the scheduler should only send minimal information about the eligible nodes -assuming that the extender already cached full details of all nodes in the cluster
managedResources
-[]ExtenderManagedResource -		 - ManagedResources is a list of extended resources that are managed by -this extender. -- A pod will be sent to the extender on the Filter, Prioritize and Bind - (if the extender is the binder) phases iff the pod requests at least - one of the extended resources in this list. If empty or unspecified, - all pods will be sent to this extender. -- If IgnoredByScheduler is set to true for a resource, kube-scheduler - will skip checking the resource in predicates.
ignorable [Required]
-bool -		 - Ignorable specifies if the extender is ignorable, i.e. scheduling should not -fail when the extender returns an error or is not reachable.
- - - -## `PredicateArgument` {#kubescheduler-config-k8s-io-v1-PredicateArgument} - - - - -**Appears in:** - -- [PredicatePolicy](#kubescheduler-config-k8s-io-v1-PredicatePolicy) - - -PredicateArgument represents the arguments to configure predicate functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAffinity [Required]
-ServiceAffinity -		 - The predicate that provides affinity for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelsPresence [Required]
-LabelsPresence -		 - The predicate that checks whether a particular node has a certain label -defined or not, regardless of value
- - - -## `PredicatePolicy` {#kubescheduler-config-k8s-io-v1-PredicatePolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PredicatePolicy describes a struct of a predicate policy. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the predicate policy -For a custom predicate, the name can be user-defined -For the Kubernetes provided predicates, the name is the identifier of the pre-defined predicate
argument [Required]
-PredicateArgument -		 - Holds the parameters to configure the given predicate
- - - -## `PriorityArgument` {#kubescheduler-config-k8s-io-v1-PriorityArgument} - - - - -**Appears in:** - -- [PriorityPolicy](#kubescheduler-config-k8s-io-v1-PriorityPolicy) - - -PriorityArgument represents the arguments to configure priority functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAntiAffinity [Required]
-ServiceAntiAffinity -		 - The priority function that ensures a good spread (anti-affinity) for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelPreference [Required]
-LabelPreference -		 - The priority function that checks whether a particular node has a certain label -defined or not, regardless of value
requestedToCapacityRatioArguments [Required]
-RequestedToCapacityRatioArguments -		 - The RequestedToCapacityRatio priority function is parametrized with function shape.
- - - -## `PriorityPolicy` {#kubescheduler-config-k8s-io-v1-PriorityPolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PriorityPolicy describes a struct of a priority policy. - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the priority policy -For a custom priority, the name can be user-defined -For the Kubernetes provided priority functions, the name is the identifier of the pre-defined priority function
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the priority function generates -The weight should be non-zero and can be a positive or a negative integer
argument [Required]
-PriorityArgument -		 - Holds the parameters to configure the given priority function
- - - -## `RequestedToCapacityRatioArguments` {#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -RequestedToCapacityRatioArguments holds arguments specific to RequestedToCapacityRatio priority function. - - - - - - - - - - - - - - - - - - -
FieldDescription
shape [Required]
-[]UtilizationShapePoint -		 - Array of point defining priority function shape.
resources [Required]
-[]ResourceSpec -		 - No description provided. -
- - - -## `ResourceSpec` {#kubescheduler-config-k8s-io-v1-ResourceSpec} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -ResourceSpec represents single resource and weight for bin packing of priority RequestedToCapacityRatioArguments. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name of the resource to be managed by RequestedToCapacityRatio function.
weight [Required]
-int64 -		 - Weight of the resource.
- - - -## `ServiceAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAffinity} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -ServiceAffinity holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should match for the node to be considered a fit for hosting the pod
- - - -## `ServiceAntiAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAntiAffinity} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -ServiceAntiAffinity holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
- - - -## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1-UtilizationShapePoint} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -UtilizationShapePoint represents single point of priority function shape. - - - - - - - - - - - - - - - - - - -
FieldDescription
utilization [Required]
-int32 -		 - Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score [Required]
-int32 -		 - Score assigned to given utilization (y axis). Valid values are 0 to 10.
- - - - - -## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} - - - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) - - -ClientConnectionConfiguration contains details for constructing a client. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
kubeconfig [Required]
-string -		 - kubeconfig is the path to a KubeConfig file.
acceptContentTypes [Required]
-string -		 - acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the -default value of 'application/json'. This field will control all connections to the server used by a particular -client.
contentType [Required]
-string -		 - contentType is the content type used when sending data to the server from this client.
qps [Required]
-float32 -		 - qps controls the number of queries per second allowed for this connection.
burst [Required]
-int32 -		 - burst allows extra queries to accumulate when a client is exceeding its rate.
- -## `DebuggingConfiguration` {#DebuggingConfiguration} - - - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) - - -DebuggingConfiguration holds configuration for Debugging related features. - - - - - - - - - - - - - - - - - - -
FieldDescription
enableProfiling [Required]
-bool -		 - enableProfiling enables profiling via web interface host:port/debug/pprof/
enableContentionProfiling [Required]
-bool -		 - enableContentionProfiling enables lock contention profiling, if -enableProfiling is true.
- -## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} - - - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) - - -LeaderElectionConfiguration defines the configuration of leader election -clients for components that can run with leader election enabled. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
leaderElect [Required]
-bool -		 - leaderElect enables a leader election client to gain leadership -before executing the main loop. Enable this when running replicated -components for high availability.
leaseDuration [Required]
-meta/v1.Duration -		 - leaseDuration is the duration that non-leader candidates will wait -after observing a leadership renewal until attempting to acquire -leadership of a led but unrenewed leader slot. This is effectively the -maximum duration that a leader can be stopped before it is replaced -by another candidate. This is only applicable if leader election is -enabled.
renewDeadline [Required]
-meta/v1.Duration -		 - renewDeadline is the interval between attempts by the acting master to -renew a leadership slot before it stops leading. This must be less -than or equal to the lease duration. This is only applicable if leader -election is enabled.
retryPeriod [Required]
-meta/v1.Duration -		 - retryPeriod is the duration the clients should wait between attempting -acquisition and renewal of a leadership. This is only applicable if -leader election is enabled.
resourceLock [Required]
-string -		 - resourceLock indicates the resource object type that will be used to lock -during leader election cycles.
resourceName [Required]
-string -		 - resourceName indicates the name of resource object that will be used to lock -during leader election cycles.
resourceNamespace [Required]
-string -		 - resourceName indicates the namespace of resource object that will be used to lock -during leader election cycles.
- -## `LoggingConfiguration` {#LoggingConfiguration} - - - - -**Appears in:** - -- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - -LoggingConfiguration contains logging options -Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information. - - - - - - - - - - - - - - - - - - -
FieldDescription
format [Required]
-string -		 - Format Flag specifies the structure of log messages. -default value of format is `text`
sanitization [Required]
-bool -		 - [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)
- - - - -## `DefaultPreemptionArgs` {#kubescheduler-config-k8s-io-v1beta1-DefaultPreemptionArgs} - - - - - -DefaultPreemptionArgs holds arguments used to configure the -DefaultPreemption plugin. - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		DefaultPreemptionArgs
minCandidateNodesPercentage [Required]
-int32 -		 - MinCandidateNodesPercentage is the minimum number of candidates to -shortlist when dry running preemption as a percentage of number of nodes. -Must be in the range [0, 100]. Defaults to 10% of the cluster size if -unspecified.
minCandidateNodesAbsolute [Required]
-int32 -		 - MinCandidateNodesAbsolute is the absolute minimum number of candidates to -shortlist. The likely number of candidates enumerated for dry running -preemption is given by the formula: -numCandidates = max(numNodes ∗ minCandidateNodesPercentage, minCandidateNodesAbsolute) -We say "likely" because there are other factors such as PDB violations -that play a role in the number of candidates shortlisted. Must be at least -0 nodes. Defaults to 100 nodes if unspecified.
- - - -## `InterPodAffinityArgs` {#kubescheduler-config-k8s-io-v1beta1-InterPodAffinityArgs} - - - - - -InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin. - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		InterPodAffinityArgs
hardPodAffinityWeight [Required]
-int32 -		 - HardPodAffinityWeight is the scoring weight for existing pods with a -matching hard affinity to the incoming pod.
- - - -## `KubeSchedulerConfiguration` {#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration} - - - - - -KubeSchedulerConfiguration configures a scheduler - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		KubeSchedulerConfiguration
parallelism [Required]
-int32 -		 - Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16
leaderElection [Required]
-LeaderElectionConfiguration -		 - LeaderElection defines the configuration of leader election client.
clientConnection [Required]
-ClientConnectionConfiguration -		 - ClientConnection specifies the kubeconfig file and client connection -settings for the proxy server to use when communicating with the apiserver.
healthzBindAddress [Required]
-string -		 - HealthzBindAddress is the IP address and port for the health check server to serve on, -defaulting to 0.0.0.0:10251
metricsBindAddress [Required]
-string -		 - MetricsBindAddress is the IP address and port for the metrics server to -serve on, defaulting to 0.0.0.0:10251.
DebuggingConfiguration [Required]
-DebuggingConfiguration -		(Members of DebuggingConfiguration are embedded into this type.) - DebuggingConfiguration holds configuration for Debugging related features -TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration
percentageOfNodesToScore [Required]
-int32 -		 - PercentageOfNodesToScore is the percentage of all nodes that once found feasible -for running a pod, the scheduler stops its search for more feasible nodes in -the cluster. This helps improve scheduler's performance. Scheduler always tries to find -at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. -Example: if the cluster size is 500 nodes and the value of this flag is 30, -then scheduler stops finding further feasible nodes once it finds 150 feasible ones. -When the value is 0, default percentage (5%--50% based on the size of the cluster) of the -nodes will be scored.
podInitialBackoffSeconds [Required]
-int64 -		 - PodInitialBackoffSeconds is the initial backoff for unschedulable pods. -If specified, it must be greater than 0. If this value is null, the default value (1s) -will be used.
podMaxBackoffSeconds [Required]
-int64 -		 - PodMaxBackoffSeconds is the max backoff for unschedulable pods. -If specified, it must be greater than podInitialBackoffSeconds. If this value is null, -the default value (10s) will be used.
profiles [Required]
-[]KubeSchedulerProfile -		 - Profiles are scheduling profiles that kube-scheduler supports. Pods can -choose to be scheduled under a particular profile by setting its associated -scheduler name. Pods that don't specify any scheduler name are scheduled -with the "default-scheduler" profile, if present here.
extenders [Required]
-[]Extender -		 - Extenders are the list of scheduler extenders, each holding the values of how to communicate -with the extender. These extenders are shared by all scheduler profiles.
- - - -## `NodeAffinityArgs` {#kubescheduler-config-k8s-io-v1beta1-NodeAffinityArgs} - - - - - -NodeAffinityArgs holds arguments to configure the NodeAffinity plugin. - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		NodeAffinityArgs
addedAffinity
-core/v1.NodeAffinity -		 - AddedAffinity is applied to all Pods additionally to the NodeAffinity -specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity -AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes -match). -When AddedAffinity is used, some Pods with affinity requirements that match -a specific Node (such as Daemonset Pods) might remain unschedulable.
- - - -## `NodeLabelArgs` {#kubescheduler-config-k8s-io-v1beta1-NodeLabelArgs} - - - - - -NodeLabelArgs holds arguments used to configure the NodeLabel plugin. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		NodeLabelArgs
presentLabels [Required]
-[]string -		 - PresentLabels should be present for the node to be considered a fit for hosting the pod
absentLabels [Required]
-[]string -		 - AbsentLabels should be absent for the node to be considered a fit for hosting the pod
presentLabelsPreference [Required]
-[]string -		 - Nodes that have labels in the list will get a higher score.
absentLabelsPreference [Required]
-[]string -		 - Nodes that don't have labels in the list will get a higher score.
- - - -## `NodeResourcesFitArgs` {#kubescheduler-config-k8s-io-v1beta1-NodeResourcesFitArgs} - - - - - -NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin. - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		NodeResourcesFitArgs
ignoredResources [Required]
-[]string -		 - IgnoredResources is the list of resources that NodeResources fit filter -should ignore.
ignoredResourceGroups [Required]
-[]string -		 - IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. -e.g. if group is ["example.com"], it will ignore all resource names that begin -with "example.com", such as "example.com/aaa" and "example.com/bbb". -A resource group name can't contain '/'.
- - - -## `NodeResourcesLeastAllocatedArgs` {#kubescheduler-config-k8s-io-v1beta1-NodeResourcesLeastAllocatedArgs} - - - - - -NodeResourcesLeastAllocatedArgs holds arguments used to configure NodeResourcesLeastAllocated plugin. - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		NodeResourcesLeastAllocatedArgs
resources [Required]
-[]ResourceSpec -		 - Resources to be managed, if no resource is provided, default resource set with both -the weight of "cpu" and "memory" set to "1" will be applied. -Resource with "0" weight will not accountable for the final score.
- - - -## `NodeResourcesMostAllocatedArgs` {#kubescheduler-config-k8s-io-v1beta1-NodeResourcesMostAllocatedArgs} - - - - - -NodeResourcesMostAllocatedArgs holds arguments used to configure NodeResourcesMostAllocated plugin. - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		NodeResourcesMostAllocatedArgs
resources [Required]
-[]ResourceSpec -		 - Resources to be managed, if no resource is provided, default resource set with both -the weight of "cpu" and "memory" set to "1" will be applied. -Resource with "0" weight will not accountable for the final score.
- - - -## `PodTopologySpreadArgs` {#kubescheduler-config-k8s-io-v1beta1-PodTopologySpreadArgs} - - - - - -PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin. - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		PodTopologySpreadArgs
defaultConstraints
-[]core/v1.TopologySpreadConstraint -		 - DefaultConstraints defines topology spread constraints to be applied to -Pods that don't define any in `pod.spec.topologySpreadConstraints`. -`.defaultConstraints[∗].labelSelectors` must be empty, as they are -deduced from the Pod's membership to Services, ReplicationControllers, -ReplicaSets or StatefulSets. -When not empty, .defaultingType must be "List".
defaultingType
-PodTopologySpreadConstraintsDefaulting -		 - DefaultingType determines how .defaultConstraints are deduced. Can be one -of "System" or "List". - -- "System": Use kubernetes defined constraints that spread Pods among - Nodes and Zones. -- "List": Use constraints defined in .defaultConstraints. - -Defaults to "List" if feature gate DefaultPodTopologySpread is disabled -and to "System" if enabled.
- - - -## `RequestedToCapacityRatioArgs` {#kubescheduler-config-k8s-io-v1beta1-RequestedToCapacityRatioArgs} - - - - - -RequestedToCapacityRatioArgs holds arguments used to configure RequestedToCapacityRatio plugin. - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		RequestedToCapacityRatioArgs
shape [Required]
-[]UtilizationShapePoint -		 - Points defining priority function shape
resources [Required]
-[]ResourceSpec -		 - Resources to be managed
- - - -## `ServiceAffinityArgs` {#kubescheduler-config-k8s-io-v1beta1-ServiceAffinityArgs} - - - - - -ServiceAffinityArgs holds arguments used to configure the ServiceAffinity plugin. - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		ServiceAffinityArgs
affinityLabels [Required]
-[]string -		 - AffinityLabels are homogeneous for pods that are scheduled to a node. -(i.e. it returns true IFF this pod can be added to this node such that all other pods in -the same service are running on nodes with the exact same values for Labels).
antiAffinityLabelsPreference [Required]
-[]string -		 - AntiAffinityLabelsPreference are the labels to consider for service anti affinity scoring.
- - - -## `VolumeBindingArgs` {#kubescheduler-config-k8s-io-v1beta1-VolumeBindingArgs} - - - - - -VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin. - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta1
kind
string		VolumeBindingArgs
bindTimeoutSeconds [Required]
-int64 -		 - BindTimeoutSeconds is the timeout in seconds in volume binding operation. -Value must be non-negative integer. The value zero indicates no waiting. -If this value is nil, the default value (600) will be used.
- - - -## `Extender` {#kubescheduler-config-k8s-io-v1beta1-Extender} - - - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) - - -Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, -it is assumed that the extender chose not to provide that extension. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
urlPrefix [Required]
-string -		 - URLPrefix at which the extender is available
filterVerb [Required]
-string -		 - Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb [Required]
-string -		 - Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb [Required]
-string -		 - Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the prioritize call generates. -The weight should be a positive integer
bindVerb [Required]
-string -		 - Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. -If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender -can implement this function.
enableHTTPS [Required]
-bool -		 - EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig [Required]
-ExtenderTLSConfig -		 - TLSConfig specifies the transport layer security config
httpTimeout [Required]
-meta/v1.Duration -		 - HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize -timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable [Required]
-bool -		 - NodeCacheCapable specifies that the extender is capable of caching node information, -so the scheduler should only send minimal information about the eligible nodes -assuming that the extender already cached full details of all nodes in the cluster
managedResources
-[]ExtenderManagedResource -		 - ManagedResources is a list of extended resources that are managed by -this extender. -- A pod will be sent to the extender on the Filter, Prioritize and Bind - (if the extender is the binder) phases iff the pod requests at least - one of the extended resources in this list. If empty or unspecified, - all pods will be sent to this extender. -- If IgnoredByScheduler is set to true for a resource, kube-scheduler - will skip checking the resource in predicates.
ignorable [Required]
-bool -		 - Ignorable specifies if the extender is ignorable, i.e. scheduling should not -fail when the extender returns an error or is not reachable.
- - - -## `KubeSchedulerProfile` {#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerProfile} - - - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration) - - -KubeSchedulerProfile is a scheduling profile. - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
schedulerName [Required]
-string -		 - SchedulerName is the name of the scheduler associated to this profile. -If SchedulerName matches with the pod's "spec.schedulerName", then the pod -is scheduled with this profile.
plugins [Required]
-Plugins -		 - Plugins specify the set of plugins that should be enabled or disabled. -Enabled plugins are the ones that should be enabled in addition to the -default plugins. Disabled plugins are any of the default plugins that -should be disabled. -When no enabled or disabled plugin is specified for an extension point, -default plugins for that extension point will be used if there is any. -If a QueueSort plugin is specified, the same QueueSort Plugin and -PluginConfig must be specified for all profiles.
pluginConfig [Required]
-[]PluginConfig -		 - PluginConfig is an optional set of custom plugin arguments for each plugin. -Omitting config args for a plugin is equivalent to using the default config -for that plugin.
- - - -## `Plugin` {#kubescheduler-config-k8s-io-v1beta1-Plugin} - - - - -**Appears in:** - -- [PluginSet](#kubescheduler-config-k8s-io-v1beta1-PluginSet) - - -Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name defines the name of plugin
weight [Required]
-int32 -		 - Weight defines the weight of plugin, only used for Score plugins.
- - - -## `PluginConfig` {#kubescheduler-config-k8s-io-v1beta1-PluginConfig} - - - - -**Appears in:** - -- [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerProfile) - - -PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. -A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. -It is up to the plugin to process these Args. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name defines the name of plugin being configured
args [Required]
-k8s.io/apimachinery/pkg/runtime.RawExtension -		 - Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.
- - - -## `PluginSet` {#kubescheduler-config-k8s-io-v1beta1-PluginSet} - - - - -**Appears in:** - -- [Plugins](#kubescheduler-config-k8s-io-v1beta1-Plugins) - - -PluginSet specifies enabled and disabled plugins for an extension point. -If an array is empty, missing, or nil, default plugins at that extension point will be used. - - - - - - - - - - - - - - - - - - -
FieldDescription
enabled [Required]
-[]Plugin -		 - Enabled specifies plugins that should be enabled in addition to default plugins. -These are called after default plugins and in the same order specified here.
disabled [Required]
-[]Plugin -		 - Disabled specifies default plugins that should be disabled. -When all default plugins need to be disabled, an array containing only one "∗" should be provided.
- - - -## `Plugins` {#kubescheduler-config-k8s-io-v1beta1-Plugins} - - - - -**Appears in:** - -- [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerProfile) - - -Plugins include multiple extension points. When specified, the list of plugins for -a particular extension point are the only ones enabled. If an extension point is -omitted from the config, then the default set of plugins is used for that extension point. -Enabled plugins are called in the order specified here, after default plugins. If they need to -be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
queueSort [Required]
-PluginSet -		 - QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.
preFilter [Required]
-PluginSet -		 - PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.
filter [Required]
-PluginSet -		 - Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.
postFilter [Required]
-PluginSet -		 - PostFilter is a list of plugins that are invoked after filtering phase, no matter whether filtering succeeds or not.
preScore [Required]
-PluginSet -		 - PreScore is a list of plugins that are invoked before scoring.
score [Required]
-PluginSet -		 - Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.
reserve [Required]
-PluginSet -		 - Reserve is a list of plugins invoked when reserving/unreserving resources -after a node is assigned to run the pod.
permit [Required]
-PluginSet -		 - Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.
preBind [Required]
-PluginSet -		 - PreBind is a list of plugins that should be invoked before a pod is bound.
bind [Required]
-PluginSet -		 - Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. -The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.
postBind [Required]
-PluginSet -		 - PostBind is a list of plugins that should be invoked after a pod is successfully bound.
- - - -## `PodTopologySpreadConstraintsDefaulting` {#kubescheduler-config-k8s-io-v1beta1-PodTopologySpreadConstraintsDefaulting} - -(Alias of `string`) - - -**Appears in:** - -- [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta1-PodTopologySpreadArgs) - - -PodTopologySpreadConstraintsDefaulting defines how to set default constraints -for the PodTopologySpread plugin. - - - - - -## `ResourceSpec` {#kubescheduler-config-k8s-io-v1beta1-ResourceSpec} - - - - -**Appears in:** - -- [NodeResourcesLeastAllocatedArgs](#kubescheduler-config-k8s-io-v1beta1-NodeResourcesLeastAllocatedArgs) - -- [NodeResourcesMostAllocatedArgs](#kubescheduler-config-k8s-io-v1beta1-NodeResourcesMostAllocatedArgs) - -- [RequestedToCapacityRatioArgs](#kubescheduler-config-k8s-io-v1beta1-RequestedToCapacityRatioArgs) - - -ResourceSpec represents single resource and weight for bin packing of priority RequestedToCapacityRatioArguments. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name of the resource to be managed by RequestedToCapacityRatio function.
weight [Required]
-int64 -		 - Weight of the resource.
- - - -## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1beta1-UtilizationShapePoint} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArgs](#kubescheduler-config-k8s-io-v1beta1-RequestedToCapacityRatioArgs) - - -UtilizationShapePoint represents single point of priority function shape. - - - - - - - - - - - - - - - - - - -
FieldDescription
utilization [Required]
-int32 -		 - Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score [Required]
-int32 -		 - Score assigned to given utilization (y axis). Valid values are 0 to 10.
- - diff --git a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md index 1a28c03c88..9025c78dcb 100644 --- a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md +++ b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md @@ -17,256 +17,334 @@ auto_generated: true - [NodeResourcesFitArgs](#kubescheduler-config-k8s-io-v1beta2-NodeResourcesFitArgs) - [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta2-PodTopologySpreadArgs) - [VolumeBindingArgs](#kubescheduler-config-k8s-io-v1beta2-VolumeBindingArgs) -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) ## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) -ClientConnectionConfiguration contains details for constructing a client. +

ClientConnectionConfiguration contains details for constructing a client.

+ - +

kubeconfig is the path to a KubeConfig file.

+ - - +client.

+ - - +

contentType is the content type used when sending data to the server from this client.

+ - - +

qps controls the number of queries per second allowed for this connection.

+ - - +

burst allows extra queries to accumulate when a client is exceeding its rate.

+ - -
FieldDescription
kubeconfig [Required]
string 		- kubeconfig is the path to a KubeConfig file.
acceptContentTypes [Required]
string 		- acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the +

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular -client.
contentType [Required]
string 		- contentType is the content type used when sending data to the server from this client.
qps [Required]
float32 		- qps controls the number of queries per second allowed for this connection.
burst [Required]
int32 		- burst allows extra queries to accumulate when a client is exceeding its rate.
## `DebuggingConfiguration` {#DebuggingConfiguration} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) -DebuggingConfiguration holds configuration for Debugging related features. +

DebuggingConfiguration holds configuration for Debugging related features.

+ - +

enableProfiling enables profiling via web interface host:port/debug/pprof/

+ - - +

enableContentionProfiling enables lock contention profiling, if +enableProfiling is true.

+ + +
FieldDescription
enableProfiling [Required]
bool 		- enableProfiling enables profiling via web interface host:port/debug/pprof/
enableContentionProfiling [Required]
bool 		- enableContentionProfiling enables lock contention profiling, if -enableProfiling is true.
+ +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

FormatOptions contains options for the different logging formats.

+ + + + + + + + + +
FieldDescription
json [Required]
+JSONOptions +		 +

[Experimental] JSON contains options for logging format "json".

+
+ +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

JSONOptions contains options for logging format "json".

+ + + + + + + + + + + + +
FieldDescription
splitStream [Required]
+bool +		 +

[Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

+
infoBufferSize [Required]
+k8s.io/apimachinery/pkg/api/resource.QuantityValue +		 +

[Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

+
## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) -LeaderElectionConfiguration defines the configuration of leader election -clients for components that can run with leader election enabled. +

LeaderElectionConfiguration defines the configuration of leader election +clients for components that can run with leader election enabled.

+ - +components for high availability.

+ - - +enabled.

+ - - +election is enabled.

+ - - +leader election is enabled.

+ - - +

resourceLock indicates the resource object type that will be used to lock +during leader election cycles.

+ - - +

resourceName indicates the name of resource object that will be used to lock +during leader election cycles.

+ - - +

resourceName indicates the namespace of resource object that will be used to lock +during leader election cycles.

+ - -
FieldDescription
leaderElect [Required]
bool 		- leaderElect enables a leader election client to gain leadership +

leaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated -components for high availability.
leaseDuration [Required]
-meta/v1.Duration +meta/v1.Duration 		- leaseDuration is the duration that non-leader candidates will wait +

leaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is -enabled.
renewDeadline [Required]
-meta/v1.Duration +meta/v1.Duration 		- renewDeadline is the interval between attempts by the acting master to +

renewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader -election is enabled.
retryPeriod [Required]
-meta/v1.Duration +meta/v1.Duration 		- retryPeriod is the duration the clients should wait between attempting +

retryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if -leader election is enabled.
resourceLock [Required]
string 		- resourceLock indicates the resource object type that will be used to lock -during leader election cycles.
resourceName [Required]
string 		- resourceName indicates the name of resource object that will be used to lock -during leader election cycles.
resourceNamespace [Required]
string 		- resourceName indicates the namespace of resource object that will be used to lock -during leader election cycles.
## `LoggingConfiguration` {#LoggingConfiguration} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) -LoggingConfiguration contains logging options -Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information. +

LoggingConfiguration contains logging options +Refer Logs Options for more information.

+ - +

Format Flag specifies the structure of log messages. +default value of format is text

+ - - - +

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

+ + + + + + + + + + - -
FieldDescription
format [Required]
string 		- Format Flag specifies the structure of log messages. -default value of format is `text`
sanitization [Required]
-bool +
flushFrequency [Required]
+time.Duration 		- [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)
verbosity [Required]
+uint32 +		 +

Verbosity is the threshold that determines which log messages are +logged. Default is zero which logs only the most important +messages. Higher values enable additional messages. Error messages +are always logged.

+
vmodule [Required]
+VModuleConfiguration +		 +

VModule overrides the verbosity threshold for individual files. +Only supported for "text" log format.

+
options [Required]
+FormatOptions +		 +

[Experimental] Options holds additional parameters that are specific +to the different logging formats. Only the options for the selected +format get used, but all of them get validated.

+
+ +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

+ + + - ## `DefaultPreemptionArgs` {#kubescheduler-config-k8s-io-v1beta2-DefaultPreemptionArgs} +

DefaultPreemptionArgs holds arguments used to configure the +DefaultPreemption plugin.

-DefaultPreemptionArgs holds arguments used to configure the -DefaultPreemption plugin. - @@ -274,47 +352,40 @@ DefaultPreemption plugin. - - +unspecified.

+ - - +0 nodes. Defaults to 100 nodes if unspecified.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		DefaultPreemptionArgs
minCandidateNodesPercentage [Required]
int32 		- MinCandidateNodesPercentage is the minimum number of candidates to +

MinCandidateNodesPercentage is the minimum number of candidates to shortlist when dry running preemption as a percentage of number of nodes. Must be in the range [0, 100]. Defaults to 10% of the cluster size if -unspecified.
minCandidateNodesAbsolute [Required]
int32 		- MinCandidateNodesAbsolute is the absolute minimum number of candidates to +

MinCandidateNodesAbsolute is the absolute minimum number of candidates to shortlist. The likely number of candidates enumerated for dry running preemption is given by the formula: -numCandidates = max(numNodes ∗ minCandidateNodesPercentage, minCandidateNodesAbsolute) -We say "likely" because there are other factors such as PDB violations +numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute) +We say "likely" because there are other factors such as PDB violations that play a role in the number of candidates shortlisted. Must be at least -0 nodes. Defaults to 100 nodes if unspecified.
- - ## `InterPodAffinityArgs` {#kubescheduler-config-k8s-io-v1beta2-InterPodAffinityArgs} +

InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.

-InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin. - @@ -322,31 +393,25 @@ InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plug - - +

HardPodAffinityWeight is the scoring weight for existing pods with a +matching hard affinity to the incoming pod.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		InterPodAffinityArgs
hardPodAffinityWeight [Required]
int32 		- HardPodAffinityWeight is the scoring weight for existing pods with a -matching hard affinity to the incoming pod.
- - ## `KubeSchedulerConfiguration` {#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration} +

KubeSchedulerConfiguration configures a scheduler

-KubeSchedulerConfiguration configures a scheduler - @@ -354,129 +419,113 @@ KubeSchedulerConfiguration configures a scheduler - - +

Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16

+ - - +

LeaderElection defines the configuration of leader election client.

+ - - +

ClientConnection specifies the kubeconfig file and client connection +settings for the proxy server to use when communicating with the apiserver.

+ - - +

Note: Both HealthzBindAddress and MetricsBindAddress fields are deprecated. +Only empty address or port 0 is allowed. Anything else will fail validation. +HealthzBindAddress is the IP address and port for the health check server to serve on.

+ - - +

MetricsBindAddress is the IP address and port for the metrics server to serve on.

+ - - +

DebuggingConfiguration holds configuration for Debugging related features +TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration

+ - - +nodes will be scored.

+ - - +will be used.

+ - - +the default value (10s) will be used.

+ - - +with the "default-scheduler" profile, if present here.

+ - - +

Extenders are the list of scheduler extenders, each holding the values of how to communicate +with the extender. These extenders are shared by all scheduler profiles.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		KubeSchedulerConfiguration
parallelism [Required]
int32 		- Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16
leaderElection [Required]
LeaderElectionConfiguration 		- LeaderElection defines the configuration of leader election client.
clientConnection [Required]
ClientConnectionConfiguration 		- ClientConnection specifies the kubeconfig file and client connection -settings for the proxy server to use when communicating with the apiserver.
healthzBindAddress [Required]
string 		- HealthzBindAddress is the IP address and port for the health check server to serve on, -defaulting to 0.0.0.0:10251
metricsBindAddress [Required]
string 		- MetricsBindAddress is the IP address and port for the metrics server to -serve on, defaulting to 0.0.0.0:10251.
DebuggingConfiguration [Required]
DebuggingConfiguration 		(Members of DebuggingConfiguration are embedded into this type.) - DebuggingConfiguration holds configuration for Debugging related features -TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration
percentageOfNodesToScore [Required]
int32 		- PercentageOfNodesToScore is the percentage of all nodes that once found feasible +

PercentageOfNodesToScore is the percentage of all nodes that once found feasible for running a pod, the scheduler stops its search for more feasible nodes in the cluster. This helps improve scheduler's performance. Scheduler always tries to find -at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. +at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. Example: if the cluster size is 500 nodes and the value of this flag is 30, then scheduler stops finding further feasible nodes once it finds 150 feasible ones. When the value is 0, default percentage (5%--50% based on the size of the cluster) of the -nodes will be scored.
podInitialBackoffSeconds [Required]
int64 		- PodInitialBackoffSeconds is the initial backoff for unschedulable pods. +

PodInitialBackoffSeconds is the initial backoff for unschedulable pods. If specified, it must be greater than 0. If this value is null, the default value (1s) -will be used.
podMaxBackoffSeconds [Required]
int64 		- PodMaxBackoffSeconds is the max backoff for unschedulable pods. +

PodMaxBackoffSeconds is the max backoff for unschedulable pods. If specified, it must be greater than podInitialBackoffSeconds. If this value is null, -the default value (10s) will be used.
profiles [Required]
[]KubeSchedulerProfile 		- Profiles are scheduling profiles that kube-scheduler supports. Pods can +

Profiles are scheduling profiles that kube-scheduler supports. Pods can choose to be scheduled under a particular profile by setting its associated scheduler name. Pods that don't specify any scheduler name are scheduled -with the "default-scheduler" profile, if present here.
extenders [Required]
[]Extender 		- Extenders are the list of scheduler extenders, each holding the values of how to communicate -with the extender. These extenders are shared by all scheduler profiles.
- - ## `NodeAffinityArgs` {#kubescheduler-config-k8s-io-v1beta2-NodeAffinityArgs} +

NodeAffinityArgs holds arguments to configure the NodeAffinity plugin.

-NodeAffinityArgs holds arguments to configure the NodeAffinity plugin. - @@ -484,35 +533,29 @@ NodeAffinityArgs holds arguments to configure the NodeAffinity plugin. - - +a specific Node (such as Daemonset Pods) might remain unschedulable.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		NodeAffinityArgs
addedAffinity
-core/v1.NodeAffinity +core/v1.NodeAffinity 		- AddedAffinity is applied to all Pods additionally to the NodeAffinity +

AddedAffinity is applied to all Pods additionally to the NodeAffinity specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes match). When AddedAffinity is used, some Pods with affinity requirements that match -a specific Node (such as Daemonset Pods) might remain unschedulable.
- - ## `NodeResourcesBalancedAllocationArgs` {#kubescheduler-config-k8s-io-v1beta2-NodeResourcesBalancedAllocationArgs} +

NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.

-NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin. - @@ -520,30 +563,24 @@ NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResour - - +

Resources to be managed, the default is "cpu" and "memory" if not specified.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		NodeResourcesBalancedAllocationArgs
resources [Required]
[]ResourceSpec 		- Resources to be managed, the default is "cpu" and "memory" if not specified.
- - ## `NodeResourcesFitArgs` {#kubescheduler-config-k8s-io-v1beta2-NodeResourcesFitArgs} +

NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.

-NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin. - @@ -551,51 +588,43 @@ NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plug - - +

IgnoredResources is the list of resources that NodeResources fit filter +should ignore. This doesn't apply to scoring.

+ - - +

IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. +e.g. if group is ["example.com"], it will ignore all resource names that begin +with "example.com", such as "example.com/aaa" and "example.com/bbb". +A resource group name can't contain '/'. This doesn't apply to scoring.

+ - - +

ScoringStrategy selects the node resource scoring strategy. +The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		NodeResourcesFitArgs
ignoredResources [Required]
[]string 		- IgnoredResources is the list of resources that NodeResources fit filter -should ignore. This doesn't apply to scoring.
ignoredResourceGroups [Required]
[]string 		- IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. -e.g. if group is ["example.com"], it will ignore all resource names that begin -with "example.com", such as "example.com/aaa" and "example.com/bbb". -A resource group name can't contain '/'. This doesn't apply to scoring.
scoringStrategy [Required]
ScoringStrategy 		- ScoringStrategy selects the node resource scoring strategy. -The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.
- - ## `PodTopologySpreadArgs` {#kubescheduler-config-k8s-io-v1beta2-PodTopologySpreadArgs} +

PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.

-PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin. - @@ -603,51 +632,43 @@ PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread pl - - +When not empty, .defaultingType must be "List".

+ - - +

DefaultingType determines how .defaultConstraints are deduced. Can be one +of "System" or "List".

+
    +
  • "System": Use kubernetes defined constraints that spread Pods among +Nodes and Zones.
    • +
  • "List": Use constraints defined in .defaultConstraints.
    • +
+

Defaults to "System".

+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		PodTopologySpreadArgs
defaultConstraints
-[]core/v1.TopologySpreadConstraint +[]core/v1.TopologySpreadConstraint 		- DefaultConstraints defines topology spread constraints to be applied to -Pods that don't define any in `pod.spec.topologySpreadConstraints`. -`.defaultConstraints[∗].labelSelectors` must be empty, as they are +

DefaultConstraints defines topology spread constraints to be applied to +Pods that don't define any in pod.spec.topologySpreadConstraints. +.defaultConstraints[*].labelSelectors must be empty, as they are deduced from the Pod's membership to Services, ReplicationControllers, ReplicaSets or StatefulSets. -When not empty, .defaultingType must be "List".
defaultingType
PodTopologySpreadConstraintsDefaulting 		- DefaultingType determines how .defaultConstraints are deduced. Can be one -of "System" or "List". - -- "System": Use kubernetes defined constraints that spread Pods among - Nodes and Zones. -- "List": Use constraints defined in .defaultConstraints. - -Defaults to "List" if feature gate DefaultPodTopologySpread is disabled -and to "System" if enabled.
- - ## `VolumeBindingArgs` {#kubescheduler-config-k8s-io-v1beta2-VolumeBindingArgs} +

VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.

-VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin. - @@ -655,528 +676,583 @@ VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin. - - +If this value is nil, the default value (600) will be used.

+ - - +The default shape points are:

+
    +
  1. 0 for 0 utilization
    2. +
  2. 10 for 100 utilization +All points must be sorted in increasing order by utilization.
    3. +
+ - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta2
kind
string		VolumeBindingArgs
bindTimeoutSeconds [Required]
int64 		- BindTimeoutSeconds is the timeout in seconds in volume binding operation. +

BindTimeoutSeconds is the timeout in seconds in volume binding operation. Value must be non-negative integer. The value zero indicates no waiting. -If this value is nil, the default value (600) will be used.
shape
[]UtilizationShapePoint 		- Shape specifies the points defining the score function shape, which is +

Shape specifies the points defining the score function shape, which is used to score nodes based on the utilization of statically provisioned PVs. The utilization is calculated by dividing the total requested storage of the pod by the total capacity of feasible PVs on each node. Each point contains utilization (ranges from 0 to 100) and its associated score (ranges from 0 to 10). You can turn the priority by specifying different scores for different utilization numbers. -The default shape points are: -1) 0 for 0 utilization -2) 10 for 100 utilization -All points must be sorted in increasing order by utilization.
- - ## `Extender` {#kubescheduler-config-k8s-io-v1beta2-Extender} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) -Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, -it is assumed that the extender chose not to provide that extension. +

Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, +it is assumed that the extender chose not to provide that extension.

+ - +

URLPrefix at which the extender is available

+ - - +

Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.

+ - - +

Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.

+ - - +

Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.

+ - - +

The numeric multiplier for the node scores that the prioritize call generates. +The weight should be a positive integer

+ - - +can implement this function.

+ - - +

EnableHTTPS specifies whether https should be used to communicate with the extender

+ - - +

TLSConfig specifies the transport layer security config

+ - - +

HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize +timeout is ignored, k8s/other extenders priorities are used to select the node.

+ - - +assuming that the extender already cached full details of all nodes in the cluster

+ - - +

ManagedResources is a list of extended resources that are managed by +this extender.

+
    +
  • A pod will be sent to the extender on the Filter, Prioritize and Bind +(if the extender is the binder) phases iff the pod requests at least +one of the extended resources in this list. If empty or unspecified, +all pods will be sent to this extender.
    • +
  • If IgnoredByScheduler is set to true for a resource, kube-scheduler +will skip checking the resource in predicates.
    • +
+ - - +

Ignorable specifies if the extender is ignorable, i.e. scheduling should not +fail when the extender returns an error or is not reachable.

+ - -
FieldDescription
urlPrefix [Required]
string 		- URLPrefix at which the extender is available
filterVerb [Required]
string 		- Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb [Required]
string 		- Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb [Required]
string 		- Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight [Required]
int64 		- The numeric multiplier for the node scores that the prioritize call generates. -The weight should be a positive integer
bindVerb [Required]
string 		- Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. +

Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender -can implement this function.
enableHTTPS [Required]
bool 		- EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig [Required]
-ExtenderTLSConfig +ExtenderTLSConfig 		- TLSConfig specifies the transport layer security config
httpTimeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize -timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable [Required]
bool 		- NodeCacheCapable specifies that the extender is capable of caching node information, +

NodeCacheCapable specifies that the extender is capable of caching node information, so the scheduler should only send minimal information about the eligible nodes -assuming that the extender already cached full details of all nodes in the cluster
managedResources
-[]ExtenderManagedResource +[]ExtenderManagedResource 		- ManagedResources is a list of extended resources that are managed by -this extender. -- A pod will be sent to the extender on the Filter, Prioritize and Bind - (if the extender is the binder) phases iff the pod requests at least - one of the extended resources in this list. If empty or unspecified, - all pods will be sent to this extender. -- If IgnoredByScheduler is set to true for a resource, kube-scheduler - will skip checking the resource in predicates.
ignorable [Required]
bool 		- Ignorable specifies if the extender is ignorable, i.e. scheduling should not -fail when the extender returns an error or is not reachable.
+ +## `ExtenderManagedResource` {#kubescheduler-config-k8s-io-v1beta2-ExtenderManagedResource} +**Appears in:** + +- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) + + +

ExtenderManagedResource describes the arguments of extended resources +managed by an extender.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name is the extended resource name.

+
ignoredByScheduler [Required]
+bool +		 +

IgnoredByScheduler indicates whether kube-scheduler should ignore this +resource when applying predicates.

+
+ +## `ExtenderTLSConfig` {#kubescheduler-config-k8s-io-v1beta2-ExtenderTLSConfig} + + +**Appears in:** + +- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) + + +

ExtenderTLSConfig contains settings to enable TLS with extender

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
insecure [Required]
+bool +		 +

Server should be accessed without verifying the TLS certificate. For testing only.

+
serverName [Required]
+string +		 +

ServerName is passed to the server for SNI and is used in the client to check server +certificates against. If ServerName is empty, the hostname used to contact the +server is used.

+
certFile [Required]
+string +		 +

Server requires TLS client certificate authentication

+
keyFile [Required]
+string +		 +

Server requires TLS client certificate authentication

+
caFile [Required]
+string +		 +

Trusted root certificates for server

+
certData [Required]
+[]byte +		 +

CertData holds PEM-encoded bytes (typically read from a client certificate file). +CertData takes precedence over CertFile

+
keyData [Required]
+[]byte +		 +

KeyData holds PEM-encoded bytes (typically read from a client certificate key file). +KeyData takes precedence over KeyFile

+
caData [Required]
+[]byte +		 +

CAData holds PEM-encoded bytes (typically read from a root certificates bundle). +CAData takes precedence over CAFile

+
## `KubeSchedulerProfile` {#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerProfile} - - **Appears in:** - [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) -KubeSchedulerProfile is a scheduling profile. +

KubeSchedulerProfile is a scheduling profile.

+ - +

SchedulerName is the name of the scheduler associated to this profile. +If SchedulerName matches with the pod's "spec.schedulerName", then the pod +is scheduled with this profile.

+ - - +PluginConfig must be specified for all profiles.

+ - - +for that plugin.

+ - -
FieldDescription
schedulerName [Required]
string 		- SchedulerName is the name of the scheduler associated to this profile. -If SchedulerName matches with the pod's "spec.schedulerName", then the pod -is scheduled with this profile.
plugins [Required]
Plugins 		- Plugins specify the set of plugins that should be enabled or disabled. +

Plugins specify the set of plugins that should be enabled or disabled. Enabled plugins are the ones that should be enabled in addition to the default plugins. Disabled plugins are any of the default plugins that should be disabled. When no enabled or disabled plugin is specified for an extension point, default plugins for that extension point will be used if there is any. If a QueueSort plugin is specified, the same QueueSort Plugin and -PluginConfig must be specified for all profiles.
pluginConfig [Required]
[]PluginConfig 		- PluginConfig is an optional set of custom plugin arguments for each plugin. +

PluginConfig is an optional set of custom plugin arguments for each plugin. Omitting config args for a plugin is equivalent to using the default config -for that plugin.
- - ## `Plugin` {#kubescheduler-config-k8s-io-v1beta2-Plugin} - - **Appears in:** - [PluginSet](#kubescheduler-config-k8s-io-v1beta2-PluginSet) -Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins. +

Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.

+ - +

Name defines the name of plugin

+ - - +

Weight defines the weight of plugin, only used for Score plugins.

+ - -
FieldDescription
name [Required]
string 		- Name defines the name of plugin
weight [Required]
int32 		- Weight defines the weight of plugin, only used for Score plugins.
- - ## `PluginConfig` {#kubescheduler-config-k8s-io-v1beta2-PluginConfig} - - **Appears in:** - [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerProfile) -PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. +

PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. -It is up to the plugin to process these Args. +It is up to the plugin to process these Args.

+ - +

Name defines the name of plugin being configured

+ - - +

Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.

+ - -
FieldDescription
name [Required]
string 		- Name defines the name of plugin being configured
args [Required]
-k8s.io/apimachinery/pkg/runtime.RawExtension +k8s.io/apimachinery/pkg/runtime.RawExtension 		- Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.
- - ## `PluginSet` {#kubescheduler-config-k8s-io-v1beta2-PluginSet} - - **Appears in:** - [Plugins](#kubescheduler-config-k8s-io-v1beta2-Plugins) -PluginSet specifies enabled and disabled plugins for an extension point. -If an array is empty, missing, or nil, default plugins at that extension point will be used. +

PluginSet specifies enabled and disabled plugins for an extension point. +If an array is empty, missing, or nil, default plugins at that extension point will be used.

+ - +These are called after default plugins and in the same order specified here.

+ - - +

Disabled specifies default plugins that should be disabled. +When all default plugins need to be disabled, an array containing only one "*" should be provided.

+ - -
FieldDescription
enabled [Required]
[]Plugin 		- Enabled specifies plugins that should be enabled in addition to default plugins. +

Enabled specifies plugins that should be enabled in addition to default plugins. If the default plugin is also configured in the scheduler config file, the weight of plugin will be overridden accordingly. -These are called after default plugins and in the same order specified here.
disabled [Required]
[]Plugin 		- Disabled specifies default plugins that should be disabled. -When all default plugins need to be disabled, an array containing only one "∗" should be provided.
- - ## `Plugins` {#kubescheduler-config-k8s-io-v1beta2-Plugins} - - **Appears in:** - [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerProfile) -Plugins include multiple extension points. When specified, the list of plugins for +

Plugins include multiple extension points. When specified, the list of plugins for a particular extension point are the only ones enabled. If an extension point is omitted from the config, then the default set of plugins is used for that extension point. Enabled plugins are called in the order specified here, after default plugins. If they need to -be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order. +be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.

+ - +

QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.

+ - - +

PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.

+ - - +

Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.

+ - - +

PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.

+ - - +

PreScore is a list of plugins that are invoked before scoring.

+ - - +

Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.

+ - - +

Reserve is a list of plugins invoked when reserving/unreserving resources +after a node is assigned to run the pod.

+ - - +

Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.

+ - - +

PreBind is a list of plugins that should be invoked before a pod is bound.

+ - - +

Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. +The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.

+ - - +

PostBind is a list of plugins that should be invoked after a pod is successfully bound.

+ + + + - -
FieldDescription
queueSort [Required]
PluginSet 		- QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.
preFilter [Required]
PluginSet 		- PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.
filter [Required]
PluginSet 		- Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.
postFilter [Required]
PluginSet 		- PostFilter is a list of plugins that are invoked after filtering phase, no matter whether filtering succeeds or not.
preScore [Required]
PluginSet 		- PreScore is a list of plugins that are invoked before scoring.
score [Required]
PluginSet 		- Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.
reserve [Required]
PluginSet 		- Reserve is a list of plugins invoked when reserving/unreserving resources -after a node is assigned to run the pod.
permit [Required]
PluginSet 		- Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.
preBind [Required]
PluginSet 		- PreBind is a list of plugins that should be invoked before a pod is bound.
bind [Required]
PluginSet 		- Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. -The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.
postBind [Required]
PluginSet 		- PostBind is a list of plugins that should be invoked after a pod is successfully bound.
multiPoint [Required]
+PluginSet +		 +

MultiPoint is a simplified config section to enable plugins for all valid extension points.

+
- - ## `PodTopologySpreadConstraintsDefaulting` {#kubescheduler-config-k8s-io-v1beta2-PodTopologySpreadConstraintsDefaulting} (Alias of `string`) - **Appears in:** - [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta2-PodTopologySpreadArgs) -PodTopologySpreadConstraintsDefaulting defines how to set default constraints -for the PodTopologySpread plugin. +

PodTopologySpreadConstraintsDefaulting defines how to set default constraints +for the PodTopologySpread plugin.

- ## `RequestedToCapacityRatioParam` {#kubescheduler-config-k8s-io-v1beta2-RequestedToCapacityRatioParam} - - **Appears in:** - [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta2-ScoringStrategy) -RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters +

RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters

+ - +

Shape is a list of points defining the scoring function shape.

+ - -
FieldDescription
shape [Required]
[]UtilizationShapePoint 		- Shape is a list of points defining the scoring function shape.
- - ## `ResourceSpec` {#kubescheduler-config-k8s-io-v1beta2-ResourceSpec} - - **Appears in:** - [NodeResourcesBalancedAllocationArgs](#kubescheduler-config-k8s-io-v1beta2-NodeResourcesBalancedAllocationArgs) @@ -1184,106 +1260,91 @@ RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters - [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta2-ScoringStrategy) -ResourceSpec represents a single resource. +

ResourceSpec represents a single resource.

+ - +

Name of the resource.

+ - - +

Weight of the resource.

+ - -
FieldDescription
name [Required]
string 		- Name of the resource.
weight [Required]
int64 		- Weight of the resource.
- - ## `ScoringStrategy` {#kubescheduler-config-k8s-io-v1beta2-ScoringStrategy} - - **Appears in:** - [NodeResourcesFitArgs](#kubescheduler-config-k8s-io-v1beta2-NodeResourcesFitArgs) -ScoringStrategy define ScoringStrategyType for node resource plugin +

ScoringStrategy define ScoringStrategyType for node resource plugin

+ - +

Type selects which strategy to run.

+ - - +Weight defaults to 1 if not specified or explicitly set to 0.

+ - - +

Arguments specific to RequestedToCapacityRatio strategy.

+ - -
FieldDescription
type [Required]
ScoringStrategyType 		- Type selects which strategy to run.
resources [Required]
[]ResourceSpec 		- Resources to consider when scoring. -The default resource set includes "cpu" and "memory" with an equal weight. +

Resources to consider when scoring. +The default resource set includes "cpu" and "memory" with an equal weight. Allowed weights go from 1 to 100. -Weight defaults to 1 if not specified or explicitly set to 0.
requestedToCapacityRatio [Required]
RequestedToCapacityRatioParam 		- Arguments specific to RequestedToCapacityRatio strategy.
- - ## `ScoringStrategyType` {#kubescheduler-config-k8s-io-v1beta2-ScoringStrategyType} (Alias of `string`) - **Appears in:** - [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta2-ScoringStrategy) -ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin. +

ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.

- ## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1beta2-UtilizationShapePoint} - - **Appears in:** - [VolumeBindingArgs](#kubescheduler-config-k8s-io-v1beta2-VolumeBindingArgs) @@ -1291,818 +1352,28 @@ ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin - [RequestedToCapacityRatioParam](#kubescheduler-config-k8s-io-v1beta2-RequestedToCapacityRatioParam) -UtilizationShapePoint represents single point of priority function shape. +

UtilizationShapePoint represents single point of priority function shape.

+ - +

Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.

+ - - +

Score assigned to given utilization (y axis). Valid values are 0 to 10.

+ - -
FieldDescription
utilization [Required]
int32 		- Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score [Required]
int32 		- Score assigned to given utilization (y axis). Valid values are 0 to 10.
- - - - - - -## `Policy` {#kubescheduler-config-k8s-io-v1-Policy} - - - - - -Policy describes a struct for a policy resource used in api. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1
kind
string		Policy
predicates [Required]
-[]PredicatePolicy -		 - Holds the information to configure the fit predicate functions
priorities [Required]
-[]PriorityPolicy -		 - Holds the information to configure the priority functions
extenders [Required]
-[]LegacyExtender -		 - Holds the information to communicate with the extender(s)
hardPodAffinitySymmetricWeight [Required]
-int32 -		 - RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule -corresponding to every RequiredDuringScheduling affinity rule. -HardPodAffinitySymmetricWeight represents the weight of implicit PreferredDuringScheduling affinity rule, in the range 1-100.
alwaysCheckAllPredicates [Required]
-bool -		 - When AlwaysCheckAllPredicates is set to true, scheduler checks all -the configured predicates even after one or more of them fails. -When the flag is set to false, scheduler skips checking the rest -of the predicates after it finds one predicate that failed.
- - - -## `ExtenderManagedResource` {#kubescheduler-config-k8s-io-v1-ExtenderManagedResource} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderManagedResource describes the arguments of extended resources -managed by an extender. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name is the extended resource name.
ignoredByScheduler [Required]
-bool -		 - IgnoredByScheduler indicates whether kube-scheduler should ignore this -resource when applying predicates.
- - - -## `ExtenderTLSConfig` {#kubescheduler-config-k8s-io-v1-ExtenderTLSConfig} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderTLSConfig contains settings to enable TLS with extender - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
insecure [Required]
-bool -		 - Server should be accessed without verifying the TLS certificate. For testing only.
serverName [Required]
-string -		 - ServerName is passed to the server for SNI and is used in the client to check server -certificates against. If ServerName is empty, the hostname used to contact the -server is used.
certFile [Required]
-string -		 - Server requires TLS client certificate authentication
keyFile [Required]
-string -		 - Server requires TLS client certificate authentication
caFile [Required]
-string -		 - Trusted root certificates for server
certData [Required]
-[]byte -		 - CertData holds PEM-encoded bytes (typically read from a client certificate file). -CertData takes precedence over CertFile
keyData [Required]
-[]byte -		 - KeyData holds PEM-encoded bytes (typically read from a client certificate key file). -KeyData takes precedence over KeyFile
caData [Required]
-[]byte -		 - CAData holds PEM-encoded bytes (typically read from a root certificates bundle). -CAData takes precedence over CAFile
- - - -## `LabelPreference` {#kubescheduler-config-k8s-io-v1-LabelPreference} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -LabelPreference holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
presence [Required]
-bool -		 - This is a boolean flag -If true, higher priority is given to nodes that have the label -If false, higher priority is given to nodes that do not have the label
- - - -## `LabelsPresence` {#kubescheduler-config-k8s-io-v1-LabelsPresence} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -LabelsPresence holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should be either present (or absent) for the node to be considered a fit for hosting the pod
presence [Required]
-bool -		 - The boolean flag that indicates whether the labels should be present or absent from the node
- - - -## `LegacyExtender` {#kubescheduler-config-k8s-io-v1-LegacyExtender} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -LegacyExtender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, -it is assumed that the extender chose not to provide that extension. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
urlPrefix [Required]
-string -		 - URLPrefix at which the extender is available
filterVerb [Required]
-string -		 - Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb [Required]
-string -		 - Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb [Required]
-string -		 - Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the prioritize call generates. -The weight should be a positive integer
bindVerb [Required]
-string -		 - Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. -If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender -can implement this function.
enableHttps [Required]
-bool -		 - EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig [Required]
-ExtenderTLSConfig -		 - TLSConfig specifies the transport layer security config
httpTimeout [Required]
-time.Duration -		 - HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize -timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable [Required]
-bool -		 - NodeCacheCapable specifies that the extender is capable of caching node information, -so the scheduler should only send minimal information about the eligible nodes -assuming that the extender already cached full details of all nodes in the cluster
managedResources
-[]ExtenderManagedResource -		 - ManagedResources is a list of extended resources that are managed by -this extender. -- A pod will be sent to the extender on the Filter, Prioritize and Bind - (if the extender is the binder) phases iff the pod requests at least - one of the extended resources in this list. If empty or unspecified, - all pods will be sent to this extender. -- If IgnoredByScheduler is set to true for a resource, kube-scheduler - will skip checking the resource in predicates.
ignorable [Required]
-bool -		 - Ignorable specifies if the extender is ignorable, i.e. scheduling should not -fail when the extender returns an error or is not reachable.
- - - -## `PredicateArgument` {#kubescheduler-config-k8s-io-v1-PredicateArgument} - - - - -**Appears in:** - -- [PredicatePolicy](#kubescheduler-config-k8s-io-v1-PredicatePolicy) - - -PredicateArgument represents the arguments to configure predicate functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAffinity [Required]
-ServiceAffinity -		 - The predicate that provides affinity for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelsPresence [Required]
-LabelsPresence -		 - The predicate that checks whether a particular node has a certain label -defined or not, regardless of value
- - - -## `PredicatePolicy` {#kubescheduler-config-k8s-io-v1-PredicatePolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PredicatePolicy describes a struct of a predicate policy. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the predicate policy -For a custom predicate, the name can be user-defined -For the Kubernetes provided predicates, the name is the identifier of the pre-defined predicate
argument [Required]
-PredicateArgument -		 - Holds the parameters to configure the given predicate
- - - -## `PriorityArgument` {#kubescheduler-config-k8s-io-v1-PriorityArgument} - - - - -**Appears in:** - -- [PriorityPolicy](#kubescheduler-config-k8s-io-v1-PriorityPolicy) - - -PriorityArgument represents the arguments to configure priority functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAntiAffinity [Required]
-ServiceAntiAffinity -		 - The priority function that ensures a good spread (anti-affinity) for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelPreference [Required]
-LabelPreference -		 - The priority function that checks whether a particular node has a certain label -defined or not, regardless of value
requestedToCapacityRatioArguments [Required]
-RequestedToCapacityRatioArguments -		 - The RequestedToCapacityRatio priority function is parametrized with function shape.
- - - -## `PriorityPolicy` {#kubescheduler-config-k8s-io-v1-PriorityPolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PriorityPolicy describes a struct of a priority policy. - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the priority policy -For a custom priority, the name can be user-defined -For the Kubernetes provided priority functions, the name is the identifier of the pre-defined priority function
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the priority function generates -The weight should be non-zero and can be a positive or a negative integer
argument [Required]
-PriorityArgument -		 - Holds the parameters to configure the given priority function
- - - -## `RequestedToCapacityRatioArguments` {#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -RequestedToCapacityRatioArguments holds arguments specific to RequestedToCapacityRatio priority function. - - - - - - - - - - - - - - - - - - -
FieldDescription
shape [Required]
-[]UtilizationShapePoint -		 - Array of point defining priority function shape.
resources [Required]
-[]ResourceSpec -		 - No description provided. -
- - - -## `ResourceSpec` {#kubescheduler-config-k8s-io-v1-ResourceSpec} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -ResourceSpec represents single resource and weight for bin packing of priority RequestedToCapacityRatioArguments. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name of the resource to be managed by RequestedToCapacityRatio function.
weight [Required]
-int64 -		 - Weight of the resource.
- - - -## `ServiceAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAffinity} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -ServiceAffinity holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should match for the node to be considered a fit for hosting the pod
- - - -## `ServiceAntiAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAntiAffinity} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -ServiceAntiAffinity holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
- - - -## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1-UtilizationShapePoint} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -UtilizationShapePoint represents single point of priority function shape. - - - - - - - - - - - - - - - - - - -
FieldDescription
utilization [Required]
-int32 -		 - Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score [Required]
-int32 -		 - Score assigned to given utilization (y axis). Valid values are 0 to 10.
- diff --git a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md new file mode 100644 index 0000000000..aeddbe89be --- /dev/null +++ b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md @@ -0,0 +1,1385 @@ +--- +title: kube-scheduler Configuration (v1beta3) +content_type: tool-reference +package: kubescheduler.config.k8s.io/v1beta3 +auto_generated: true +--- + + +## Resource Types + + +- [DefaultPreemptionArgs](#kubescheduler-config-k8s-io-v1beta3-DefaultPreemptionArgs) +- [InterPodAffinityArgs](#kubescheduler-config-k8s-io-v1beta3-InterPodAffinityArgs) +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) +- [NodeAffinityArgs](#kubescheduler-config-k8s-io-v1beta3-NodeAffinityArgs) +- [NodeResourcesBalancedAllocationArgs](#kubescheduler-config-k8s-io-v1beta3-NodeResourcesBalancedAllocationArgs) +- [NodeResourcesFitArgs](#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs) +- [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta3-PodTopologySpreadArgs) +- [VolumeBindingArgs](#kubescheduler-config-k8s-io-v1beta3-VolumeBindingArgs) + + + +## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

ClientConnectionConfiguration contains details for constructing a client.

+ + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
kubeconfig [Required]
+string +		 +

kubeconfig is the path to a KubeConfig file.

+
acceptContentTypes [Required]
+string +		 +

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the +default value of 'application/json'. This field will control all connections to the server used by a particular +client.

+
contentType [Required]
+string +		 +

contentType is the content type used when sending data to the server from this client.

+
qps [Required]
+float32 +		 +

qps controls the number of queries per second allowed for this connection.

+
burst [Required]
+int32 +		 +

burst allows extra queries to accumulate when a client is exceeding its rate.

+
+ +## `DebuggingConfiguration` {#DebuggingConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + + +

DebuggingConfiguration holds configuration for Debugging related features.

+ + + + + + + + + + + + + + +
FieldDescription
enableProfiling [Required]
+bool +		 +

enableProfiling enables profiling via web interface host:port/debug/pprof/

+
enableContentionProfiling [Required]
+bool +		 +

enableContentionProfiling enables lock contention profiling, if +enableProfiling is true.

+
+ +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

FormatOptions contains options for the different logging formats.

+ + + + + + + + + + + +
FieldDescription
json [Required]
+JSONOptions +		 +

[Experimental] JSON contains options for logging format "json".

+
+ +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

JSONOptions contains options for logging format "json".

+ + + + + + + + + + + + + + +
FieldDescription
splitStream [Required]
+bool +		 +

[Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

+
infoBufferSize [Required]
+k8s.io/apimachinery/pkg/api/resource.QuantityValue +		 +

[Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

+
+ +## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

LeaderElectionConfiguration defines the configuration of leader election +clients for components that can run with leader election enabled.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
leaderElect [Required]
+bool +		 +

leaderElect enables a leader election client to gain leadership +before executing the main loop. Enable this when running replicated +components for high availability.

+
leaseDuration [Required]
+meta/v1.Duration +		 +

leaseDuration is the duration that non-leader candidates will wait +after observing a leadership renewal until attempting to acquire +leadership of a led but unrenewed leader slot. This is effectively the +maximum duration that a leader can be stopped before it is replaced +by another candidate. This is only applicable if leader election is +enabled.

+
renewDeadline [Required]
+meta/v1.Duration +		 +

renewDeadline is the interval between attempts by the acting master to +renew a leadership slot before it stops leading. This must be less +than or equal to the lease duration. This is only applicable if leader +election is enabled.

+
retryPeriod [Required]
+meta/v1.Duration +		 +

retryPeriod is the duration the clients should wait between attempting +acquisition and renewal of a leadership. This is only applicable if +leader election is enabled.

+
resourceLock [Required]
+string +		 +

resourceLock indicates the resource object type that will be used to lock +during leader election cycles.

+
resourceName [Required]
+string +		 +

resourceName indicates the name of resource object that will be used to lock +during leader election cycles.

+
resourceNamespace [Required]
+string +		 +

resourceName indicates the namespace of resource object that will be used to lock +during leader election cycles.

+
+ +## `LoggingConfiguration` {#LoggingConfiguration} + + +**Appears in:** + +- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) + + +

LoggingConfiguration contains logging options +Refer Logs Options for more information.

+ + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
format [Required]
+string +		 +

Format Flag specifies the structure of log messages. +default value of format is text

+
flushFrequency [Required]
+time.Duration +		 +

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

+
verbosity [Required]
+uint32 +		 +

Verbosity is the threshold that determines which log messages are +logged. Default is zero which logs only the most important +messages. Higher values enable additional messages. Error messages +are always logged.

+
vmodule [Required]
+VModuleConfiguration +		 +

VModule overrides the verbosity threshold for individual files. +Only supported for "text" log format.

+
options [Required]
+FormatOptions +		 +

[Experimental] Options holds additional parameters that are specific +to the different logging formats. Only the options for the selected +format get used, but all of them get validated.

+
+ +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

+ + + + + + +## `DefaultPreemptionArgs` {#kubescheduler-config-k8s-io-v1beta3-DefaultPreemptionArgs} + + + +

DefaultPreemptionArgs holds arguments used to configure the +DefaultPreemption plugin.

+ + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		DefaultPreemptionArgs
minCandidateNodesPercentage [Required]
+int32 +		 +

MinCandidateNodesPercentage is the minimum number of candidates to +shortlist when dry running preemption as a percentage of number of nodes. +Must be in the range [0, 100]. Defaults to 10% of the cluster size if +unspecified.

+
minCandidateNodesAbsolute [Required]
+int32 +		 +

MinCandidateNodesAbsolute is the absolute minimum number of candidates to +shortlist. The likely number of candidates enumerated for dry running +preemption is given by the formula: +numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute) +We say "likely" because there are other factors such as PDB violations +that play a role in the number of candidates shortlisted. Must be at least +0 nodes. Defaults to 100 nodes if unspecified.

+
+ +## `InterPodAffinityArgs` {#kubescheduler-config-k8s-io-v1beta3-InterPodAffinityArgs} + + + +

InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		InterPodAffinityArgs
hardPodAffinityWeight [Required]
+int32 +		 +

HardPodAffinityWeight is the scoring weight for existing pods with a +matching hard affinity to the incoming pod.

+
+ +## `KubeSchedulerConfiguration` {#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration} + + + +

KubeSchedulerConfiguration configures a scheduler

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		KubeSchedulerConfiguration
parallelism [Required]
+int32 +		 +

Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16

+
leaderElection [Required]
+LeaderElectionConfiguration +		 +

LeaderElection defines the configuration of leader election client.

+
clientConnection [Required]
+ClientConnectionConfiguration +		 +

ClientConnection specifies the kubeconfig file and client connection +settings for the proxy server to use when communicating with the apiserver.

+
DebuggingConfiguration [Required]
+DebuggingConfiguration +		(Members of DebuggingConfiguration are embedded into this type.) +

DebuggingConfiguration holds configuration for Debugging related features +TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration

+
percentageOfNodesToScore [Required]
+int32 +		 +

PercentageOfNodesToScore is the percentage of all nodes that once found feasible +for running a pod, the scheduler stops its search for more feasible nodes in +the cluster. This helps improve scheduler's performance. Scheduler always tries to find +at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. +Example: if the cluster size is 500 nodes and the value of this flag is 30, +then scheduler stops finding further feasible nodes once it finds 150 feasible ones. +When the value is 0, default percentage (5%--50% based on the size of the cluster) of the +nodes will be scored.

+
podInitialBackoffSeconds [Required]
+int64 +		 +

PodInitialBackoffSeconds is the initial backoff for unschedulable pods. +If specified, it must be greater than 0. If this value is null, the default value (1s) +will be used.

+
podMaxBackoffSeconds [Required]
+int64 +		 +

PodMaxBackoffSeconds is the max backoff for unschedulable pods. +If specified, it must be greater than podInitialBackoffSeconds. If this value is null, +the default value (10s) will be used.

+
profiles [Required]
+[]KubeSchedulerProfile +		 +

Profiles are scheduling profiles that kube-scheduler supports. Pods can +choose to be scheduled under a particular profile by setting its associated +scheduler name. Pods that don't specify any scheduler name are scheduled +with the "default-scheduler" profile, if present here.

+
extenders [Required]
+[]Extender +		 +

Extenders are the list of scheduler extenders, each holding the values of how to communicate +with the extender. These extenders are shared by all scheduler profiles.

+
+ +## `NodeAffinityArgs` {#kubescheduler-config-k8s-io-v1beta3-NodeAffinityArgs} + + + +

NodeAffinityArgs holds arguments to configure the NodeAffinity plugin.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		NodeAffinityArgs
addedAffinity
+core/v1.NodeAffinity +		 +

AddedAffinity is applied to all Pods additionally to the NodeAffinity +specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity +AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes +match). +When AddedAffinity is used, some Pods with affinity requirements that match +a specific Node (such as Daemonset Pods) might remain unschedulable.

+
+ +## `NodeResourcesBalancedAllocationArgs` {#kubescheduler-config-k8s-io-v1beta3-NodeResourcesBalancedAllocationArgs} + + + +

NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		NodeResourcesBalancedAllocationArgs
resources [Required]
+[]ResourceSpec +		 +

Resources to be managed, the default is "cpu" and "memory" if not specified.

+
+ +## `NodeResourcesFitArgs` {#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs} + + + +

NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		NodeResourcesFitArgs
ignoredResources [Required]
+[]string +		 +

IgnoredResources is the list of resources that NodeResources fit filter +should ignore. This doesn't apply to scoring.

+
ignoredResourceGroups [Required]
+[]string +		 +

IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. +e.g. if group is ["example.com"], it will ignore all resource names that begin +with "example.com", such as "example.com/aaa" and "example.com/bbb". +A resource group name can't contain '/'. This doesn't apply to scoring.

+
scoringStrategy [Required]
+ScoringStrategy +		 +

ScoringStrategy selects the node resource scoring strategy. +The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.

+
+ +## `PodTopologySpreadArgs` {#kubescheduler-config-k8s-io-v1beta3-PodTopologySpreadArgs} + + + +

PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.

+ + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		PodTopologySpreadArgs
defaultConstraints
+[]core/v1.TopologySpreadConstraint +		 +

DefaultConstraints defines topology spread constraints to be applied to +Pods that don't define any in pod.spec.topologySpreadConstraints. +.defaultConstraints[*].labelSelectors must be empty, as they are +deduced from the Pod's membership to Services, ReplicationControllers, +ReplicaSets or StatefulSets. +When not empty, .defaultingType must be "List".

+
defaultingType
+PodTopologySpreadConstraintsDefaulting +		 +

DefaultingType determines how .defaultConstraints are deduced. Can be one +of "System" or "List".

+
    +
  • "System": Use kubernetes defined constraints that spread Pods among +Nodes and Zones.
    • +
  • "List": Use constraints defined in .defaultConstraints.
    • +
+

Defaults to "System".

+
+ +## `VolumeBindingArgs` {#kubescheduler-config-k8s-io-v1beta3-VolumeBindingArgs} + + + +

VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.

+ + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1beta3
kind
string		VolumeBindingArgs
bindTimeoutSeconds [Required]
+int64 +		 +

BindTimeoutSeconds is the timeout in seconds in volume binding operation. +Value must be non-negative integer. The value zero indicates no waiting. +If this value is nil, the default value (600) will be used.

+
shape
+[]UtilizationShapePoint +		 +

Shape specifies the points defining the score function shape, which is +used to score nodes based on the utilization of statically provisioned +PVs. The utilization is calculated by dividing the total requested +storage of the pod by the total capacity of feasible PVs on each node. +Each point contains utilization (ranges from 0 to 100) and its +associated score (ranges from 0 to 10). You can turn the priority by +specifying different scores for different utilization numbers. +The default shape points are:

+
    +
  1. 0 for 0 utilization
    2. +
  2. 10 for 100 utilization +All points must be sorted in increasing order by utilization.
    3. +
+
+ +## `Extender` {#kubescheduler-config-k8s-io-v1beta3-Extender} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, +it is assumed that the extender chose not to provide that extension.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
urlPrefix [Required]
+string +		 +

URLPrefix at which the extender is available

+
filterVerb [Required]
+string +		 +

Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.

+
preemptVerb [Required]
+string +		 +

Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.

+
prioritizeVerb [Required]
+string +		 +

Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.

+
weight [Required]
+int64 +		 +

The numeric multiplier for the node scores that the prioritize call generates. +The weight should be a positive integer

+
bindVerb [Required]
+string +		 +

Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. +If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender +can implement this function.

+
enableHTTPS [Required]
+bool +		 +

EnableHTTPS specifies whether https should be used to communicate with the extender

+
tlsConfig [Required]
+ExtenderTLSConfig +		 +

TLSConfig specifies the transport layer security config

+
httpTimeout [Required]
+meta/v1.Duration +		 +

HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize +timeout is ignored, k8s/other extenders priorities are used to select the node.

+
nodeCacheCapable [Required]
+bool +		 +

NodeCacheCapable specifies that the extender is capable of caching node information, +so the scheduler should only send minimal information about the eligible nodes +assuming that the extender already cached full details of all nodes in the cluster

+
managedResources
+[]ExtenderManagedResource +		 +

ManagedResources is a list of extended resources that are managed by +this extender.

+
    +
  • A pod will be sent to the extender on the Filter, Prioritize and Bind +(if the extender is the binder) phases iff the pod requests at least +one of the extended resources in this list. If empty or unspecified, +all pods will be sent to this extender.
    • +
  • If IgnoredByScheduler is set to true for a resource, kube-scheduler +will skip checking the resource in predicates.
    • +
+
ignorable [Required]
+bool +		 +

Ignorable specifies if the extender is ignorable, i.e. scheduling should not +fail when the extender returns an error or is not reachable.

+
+ +## `ExtenderManagedResource` {#kubescheduler-config-k8s-io-v1beta3-ExtenderManagedResource} + + +**Appears in:** + +- [Extender](#kubescheduler-config-k8s-io-v1beta3-Extender) + + +

ExtenderManagedResource describes the arguments of extended resources +managed by an extender.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name is the extended resource name.

+
ignoredByScheduler [Required]
+bool +		 +

IgnoredByScheduler indicates whether kube-scheduler should ignore this +resource when applying predicates.

+
+ +## `ExtenderTLSConfig` {#kubescheduler-config-k8s-io-v1beta3-ExtenderTLSConfig} + + +**Appears in:** + +- [Extender](#kubescheduler-config-k8s-io-v1beta3-Extender) + + +

ExtenderTLSConfig contains settings to enable TLS with extender

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
insecure [Required]
+bool +		 +

Server should be accessed without verifying the TLS certificate. For testing only.

+
serverName [Required]
+string +		 +

ServerName is passed to the server for SNI and is used in the client to check server +certificates against. If ServerName is empty, the hostname used to contact the +server is used.

+
certFile [Required]
+string +		 +

Server requires TLS client certificate authentication

+
keyFile [Required]
+string +		 +

Server requires TLS client certificate authentication

+
caFile [Required]
+string +		 +

Trusted root certificates for server

+
certData [Required]
+[]byte +		 +

CertData holds PEM-encoded bytes (typically read from a client certificate file). +CertData takes precedence over CertFile

+
keyData [Required]
+[]byte +		 +

KeyData holds PEM-encoded bytes (typically read from a client certificate key file). +KeyData takes precedence over KeyFile

+
caData [Required]
+[]byte +		 +

CAData holds PEM-encoded bytes (typically read from a root certificates bundle). +CAData takes precedence over CAFile

+
+ +## `KubeSchedulerProfile` {#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerProfile} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

KubeSchedulerProfile is a scheduling profile.

+ + + + + + + + + + + + + + + + + +
FieldDescription
schedulerName [Required]
+string +		 +

SchedulerName is the name of the scheduler associated to this profile. +If SchedulerName matches with the pod's "spec.schedulerName", then the pod +is scheduled with this profile.

+
plugins [Required]
+Plugins +		 +

Plugins specify the set of plugins that should be enabled or disabled. +Enabled plugins are the ones that should be enabled in addition to the +default plugins. Disabled plugins are any of the default plugins that +should be disabled. +When no enabled or disabled plugin is specified for an extension point, +default plugins for that extension point will be used if there is any. +If a QueueSort plugin is specified, the same QueueSort Plugin and +PluginConfig must be specified for all profiles.

+
pluginConfig [Required]
+[]PluginConfig +		 +

PluginConfig is an optional set of custom plugin arguments for each plugin. +Omitting config args for a plugin is equivalent to using the default config +for that plugin.

+
+ +## `Plugin` {#kubescheduler-config-k8s-io-v1beta3-Plugin} + + +**Appears in:** + +- [PluginSet](#kubescheduler-config-k8s-io-v1beta3-PluginSet) + + +

Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name defines the name of plugin

+
weight [Required]
+int32 +		 +

Weight defines the weight of plugin, only used for Score plugins.

+
+ +## `PluginConfig` {#kubescheduler-config-k8s-io-v1beta3-PluginConfig} + + +**Appears in:** + +- [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerProfile) + + +

PluginConfig specifies arguments that should be passed to a plugin at the time of initialization. +A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. +It is up to the plugin to process these Args.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name defines the name of plugin being configured

+
args [Required]
+k8s.io/apimachinery/pkg/runtime.RawExtension +		 +

Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.

+
+ +## `PluginSet` {#kubescheduler-config-k8s-io-v1beta3-PluginSet} + + +**Appears in:** + +- [Plugins](#kubescheduler-config-k8s-io-v1beta3-Plugins) + + +

PluginSet specifies enabled and disabled plugins for an extension point. +If an array is empty, missing, or nil, default plugins at that extension point will be used.

+ + + + + + + + + + + + + + +
FieldDescription
enabled [Required]
+[]Plugin +		 +

Enabled specifies plugins that should be enabled in addition to default plugins. +If the default plugin is also configured in the scheduler config file, the weight of plugin will +be overridden accordingly. +These are called after default plugins and in the same order specified here.

+
disabled [Required]
+[]Plugin +		 +

Disabled specifies default plugins that should be disabled. +When all default plugins need to be disabled, an array containing only one "*" should be provided.

+
+ +## `Plugins` {#kubescheduler-config-k8s-io-v1beta3-Plugins} + + +**Appears in:** + +- [KubeSchedulerProfile](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerProfile) + + +

Plugins include multiple extension points. When specified, the list of plugins for +a particular extension point are the only ones enabled. If an extension point is +omitted from the config, then the default set of plugins is used for that extension point. +Enabled plugins are called in the order specified here, after default plugins. If they need to +be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
queueSort [Required]
+PluginSet +		 +

QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.

+
preFilter [Required]
+PluginSet +		 +

PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.

+
filter [Required]
+PluginSet +		 +

Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.

+
postFilter [Required]
+PluginSet +		 +

PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.

+
preScore [Required]
+PluginSet +		 +

PreScore is a list of plugins that are invoked before scoring.

+
score [Required]
+PluginSet +		 +

Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.

+
reserve [Required]
+PluginSet +		 +

Reserve is a list of plugins invoked when reserving/unreserving resources +after a node is assigned to run the pod.

+
permit [Required]
+PluginSet +		 +

Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.

+
preBind [Required]
+PluginSet +		 +

PreBind is a list of plugins that should be invoked before a pod is bound.

+
bind [Required]
+PluginSet +		 +

Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. +The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.

+
postBind [Required]
+PluginSet +		 +

PostBind is a list of plugins that should be invoked after a pod is successfully bound.

+
multiPoint [Required]
+PluginSet +		 +

MultiPoint is a simplified config section to enable plugins for all valid extension points. +Plugins enabled through MultiPoint will automatically register for every individual extension +point the plugin has implemented. Disabling a plugin through MultiPoint disables that behavior. +The same is true for disabling "*" through MultiPoint (no default plugins will be automatically registered). +Plugins can still be disabled through their individual extension points.

+

In terms of precedence, plugin config follows this basic hierarchy

+
    +
  1. Specific extension points
    2. +
  2. Explicitly configured MultiPoint plugins
    3. +
  3. The set of default plugins, as MultiPoint plugins +This implies that a higher precedence plugin will run first and overwrite any settings within MultiPoint. +Explicitly user-configured plugins also take a higher precedence over default plugins. +Within this hierarchy, an Enabled setting takes precedence over Disabled. For example, if a plugin is +set in both multiPoint.Enabled and multiPoint.Disabled, the plugin will be enabled. Similarly, +including multiPoint.Disabled = '*' and multiPoint.Enabled = pluginA will still register that specific +plugin through MultiPoint. This follows the same behavior as all other extension point configurations.
    4. +
+
+ +## `PodTopologySpreadConstraintsDefaulting` {#kubescheduler-config-k8s-io-v1beta3-PodTopologySpreadConstraintsDefaulting} + +(Alias of `string`) + +**Appears in:** + +- [PodTopologySpreadArgs](#kubescheduler-config-k8s-io-v1beta3-PodTopologySpreadArgs) + + +

PodTopologySpreadConstraintsDefaulting defines how to set default constraints +for the PodTopologySpread plugin.

+ + + + +## `RequestedToCapacityRatioParam` {#kubescheduler-config-k8s-io-v1beta3-RequestedToCapacityRatioParam} + + +**Appears in:** + +- [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy) + + +

RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters

+ + + + + + + + + + + +
FieldDescription
shape [Required]
+[]UtilizationShapePoint +		 +

Shape is a list of points defining the scoring function shape.

+
+ +## `ResourceSpec` {#kubescheduler-config-k8s-io-v1beta3-ResourceSpec} + + +**Appears in:** + +- [NodeResourcesBalancedAllocationArgs](#kubescheduler-config-k8s-io-v1beta3-NodeResourcesBalancedAllocationArgs) + +- [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy) + + +

ResourceSpec represents a single resource.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

Name of the resource.

+
weight [Required]
+int64 +		 +

Weight of the resource.

+
+ +## `ScoringStrategy` {#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy} + + +**Appears in:** + +- [NodeResourcesFitArgs](#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs) + + +

ScoringStrategy define ScoringStrategyType for node resource plugin

+ + + + + + + + + + + + + + + + + +
FieldDescription
type [Required]
+ScoringStrategyType +		 +

Type selects which strategy to run.

+
resources [Required]
+[]ResourceSpec +		 +

Resources to consider when scoring. +The default resource set includes "cpu" and "memory" with an equal weight. +Allowed weights go from 1 to 100. +Weight defaults to 1 if not specified or explicitly set to 0.

+
requestedToCapacityRatio [Required]
+RequestedToCapacityRatioParam +		 +

Arguments specific to RequestedToCapacityRatio strategy.

+
+ +## `ScoringStrategyType` {#kubescheduler-config-k8s-io-v1beta3-ScoringStrategyType} + +(Alias of `string`) + +**Appears in:** + +- [ScoringStrategy](#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy) + + +

ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.

+ + + + +## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1beta3-UtilizationShapePoint} + + +**Appears in:** + +- [VolumeBindingArgs](#kubescheduler-config-k8s-io-v1beta3-VolumeBindingArgs) + +- [RequestedToCapacityRatioParam](#kubescheduler-config-k8s-io-v1beta3-RequestedToCapacityRatioParam) + + +

UtilizationShapePoint represents single point of priority function shape.

+ + + + + + + + + + + + + + +
FieldDescription
utilization [Required]
+int32 +		 +

Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.

+
score [Required]
+int32 +		 +

Score assigned to given utilization (y axis). Valid values are 0 to 10.

+
+ diff --git a/content/en/docs/reference/config-api/kube-scheduler-policy-config.v1.md b/content/en/docs/reference/config-api/kube-scheduler-policy-config.v1.md deleted file mode 100644 index 8b6c0a9a24..0000000000 --- a/content/en/docs/reference/config-api/kube-scheduler-policy-config.v1.md +++ /dev/null @@ -1,799 +0,0 @@ ---- -title: kube-scheduler Policy Configuration (v1) -content_type: tool-reference -package: kubescheduler.config.k8s.io/v1 -auto_generated: true ---- - - -## Resource Types - - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - - - -## `Policy` {#kubescheduler-config-k8s-io-v1-Policy} - - - - - -Policy describes a struct for a policy resource used in api. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
apiVersion
string		kubescheduler.config.k8s.io/v1
kind
string		Policy
predicates [Required]
-[]PredicatePolicy -		 - Holds the information to configure the fit predicate functions
priorities [Required]
-[]PriorityPolicy -		 - Holds the information to configure the priority functions
extenders [Required]
-[]LegacyExtender -		 - Holds the information to communicate with the extender(s)
hardPodAffinitySymmetricWeight [Required]
-int32 -		 - RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule -corresponding to every RequiredDuringScheduling affinity rule. -HardPodAffinitySymmetricWeight represents the weight of implicit PreferredDuringScheduling affinity rule, in the range 1-100.
alwaysCheckAllPredicates [Required]
-bool -		 - When AlwaysCheckAllPredicates is set to true, scheduler checks all -the configured predicates even after one or more of them fails. -When the flag is set to false, scheduler skips checking the rest -of the predicates after it finds one predicate that failed.
- - - -## `ExtenderManagedResource` {#kubescheduler-config-k8s-io-v1-ExtenderManagedResource} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderManagedResource describes the arguments of extended resources -managed by an extender. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name is the extended resource name.
ignoredByScheduler [Required]
-bool -		 - IgnoredByScheduler indicates whether kube-scheduler should ignore this -resource when applying predicates.
- - - -## `ExtenderTLSConfig` {#kubescheduler-config-k8s-io-v1-ExtenderTLSConfig} - - - - -**Appears in:** - -- [Extender](#kubescheduler-config-k8s-io-v1beta2-Extender) - -- [LegacyExtender](#kubescheduler-config-k8s-io-v1-LegacyExtender) - - -ExtenderTLSConfig contains settings to enable TLS with extender - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
insecure [Required]
-bool -		 - Server should be accessed without verifying the TLS certificate. For testing only.
serverName [Required]
-string -		 - ServerName is passed to the server for SNI and is used in the client to check server -certificates against. If ServerName is empty, the hostname used to contact the -server is used.
certFile [Required]
-string -		 - Server requires TLS client certificate authentication
keyFile [Required]
-string -		 - Server requires TLS client certificate authentication
caFile [Required]
-string -		 - Trusted root certificates for server
certData [Required]
-[]byte -		 - CertData holds PEM-encoded bytes (typically read from a client certificate file). -CertData takes precedence over CertFile
keyData [Required]
-[]byte -		 - KeyData holds PEM-encoded bytes (typically read from a client certificate key file). -KeyData takes precedence over KeyFile
caData [Required]
-[]byte -		 - CAData holds PEM-encoded bytes (typically read from a root certificates bundle). -CAData takes precedence over CAFile
- - - -## `LabelPreference` {#kubescheduler-config-k8s-io-v1-LabelPreference} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -LabelPreference holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
presence [Required]
-bool -		 - This is a boolean flag -If true, higher priority is given to nodes that have the label -If false, higher priority is given to nodes that do not have the label
- - - -## `LabelsPresence` {#kubescheduler-config-k8s-io-v1-LabelsPresence} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -LabelsPresence holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should be either present (or absent) for the node to be considered a fit for hosting the pod
presence [Required]
-bool -		 - The boolean flag that indicates whether the labels should be present or absent from the node
- - - -## `LegacyExtender` {#kubescheduler-config-k8s-io-v1-LegacyExtender} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -LegacyExtender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, -it is assumed that the extender chose not to provide that extension. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
urlPrefix [Required]
-string -		 - URLPrefix at which the extender is available
filterVerb [Required]
-string -		 - Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb [Required]
-string -		 - Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb [Required]
-string -		 - Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the prioritize call generates. -The weight should be a positive integer
bindVerb [Required]
-string -		 - Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. -If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender -can implement this function.
enableHttps [Required]
-bool -		 - EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig [Required]
-ExtenderTLSConfig -		 - TLSConfig specifies the transport layer security config
httpTimeout [Required]
-time.Duration -		 - HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize -timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable [Required]
-bool -		 - NodeCacheCapable specifies that the extender is capable of caching node information, -so the scheduler should only send minimal information about the eligible nodes -assuming that the extender already cached full details of all nodes in the cluster
managedResources
-[]ExtenderManagedResource -		 - ManagedResources is a list of extended resources that are managed by -this extender. -- A pod will be sent to the extender on the Filter, Prioritize and Bind - (if the extender is the binder) phases iff the pod requests at least - one of the extended resources in this list. If empty or unspecified, - all pods will be sent to this extender. -- If IgnoredByScheduler is set to true for a resource, kube-scheduler - will skip checking the resource in predicates.
ignorable [Required]
-bool -		 - Ignorable specifies if the extender is ignorable, i.e. scheduling should not -fail when the extender returns an error or is not reachable.
- - - -## `PredicateArgument` {#kubescheduler-config-k8s-io-v1-PredicateArgument} - - - - -**Appears in:** - -- [PredicatePolicy](#kubescheduler-config-k8s-io-v1-PredicatePolicy) - - -PredicateArgument represents the arguments to configure predicate functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAffinity [Required]
-ServiceAffinity -		 - The predicate that provides affinity for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelsPresence [Required]
-LabelsPresence -		 - The predicate that checks whether a particular node has a certain label -defined or not, regardless of value
- - - -## `PredicatePolicy` {#kubescheduler-config-k8s-io-v1-PredicatePolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PredicatePolicy describes a struct of a predicate policy. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the predicate policy -For a custom predicate, the name can be user-defined -For the Kubernetes provided predicates, the name is the identifier of the pre-defined predicate
argument [Required]
-PredicateArgument -		 - Holds the parameters to configure the given predicate
- - - -## `PriorityArgument` {#kubescheduler-config-k8s-io-v1-PriorityArgument} - - - - -**Appears in:** - -- [PriorityPolicy](#kubescheduler-config-k8s-io-v1-PriorityPolicy) - - -PriorityArgument represents the arguments to configure priority functions in scheduler policy configuration. -Only one of its members may be specified - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
serviceAntiAffinity [Required]
-ServiceAntiAffinity -		 - The priority function that ensures a good spread (anti-affinity) for pods belonging to a service -It uses a label to identify nodes that belong to the same "group"
labelPreference [Required]
-LabelPreference -		 - The priority function that checks whether a particular node has a certain label -defined or not, regardless of value
requestedToCapacityRatioArguments [Required]
-RequestedToCapacityRatioArguments -		 - The RequestedToCapacityRatio priority function is parametrized with function shape.
- - - -## `PriorityPolicy` {#kubescheduler-config-k8s-io-v1-PriorityPolicy} - - - - -**Appears in:** - -- [Policy](#kubescheduler-config-k8s-io-v1-Policy) - - -PriorityPolicy describes a struct of a priority policy. - - - - - - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Identifier of the priority policy -For a custom priority, the name can be user-defined -For the Kubernetes provided priority functions, the name is the identifier of the pre-defined priority function
weight [Required]
-int64 -		 - The numeric multiplier for the node scores that the priority function generates -The weight should be non-zero and can be a positive or a negative integer
argument [Required]
-PriorityArgument -		 - Holds the parameters to configure the given priority function
- - - -## `RequestedToCapacityRatioArguments` {#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -RequestedToCapacityRatioArguments holds arguments specific to RequestedToCapacityRatio priority function. - - - - - - - - - - - - - - - - - - -
FieldDescription
shape [Required]
-[]UtilizationShapePoint -		 - Array of point defining priority function shape.
resources [Required]
-[]ResourceSpec -		 - No description provided. -
- - - -## `ResourceSpec` {#kubescheduler-config-k8s-io-v1-ResourceSpec} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -ResourceSpec represents single resource and weight for bin packing of priority RequestedToCapacityRatioArguments. - - - - - - - - - - - - - - - - - - -
FieldDescription
name [Required]
-string -		 - Name of the resource to be managed by RequestedToCapacityRatio function.
weight [Required]
-int64 -		 - Weight of the resource.
- - - -## `ServiceAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAffinity} - - - - -**Appears in:** - -- [PredicateArgument](#kubescheduler-config-k8s-io-v1-PredicateArgument) - - -ServiceAffinity holds the parameters that are used to configure the corresponding predicate in scheduler policy configuration. - - - - - - - - - - - - - -
FieldDescription
labels [Required]
-[]string -		 - The list of labels that identify node "groups" -All of the labels should match for the node to be considered a fit for hosting the pod
- - - -## `ServiceAntiAffinity` {#kubescheduler-config-k8s-io-v1-ServiceAntiAffinity} - - - - -**Appears in:** - -- [PriorityArgument](#kubescheduler-config-k8s-io-v1-PriorityArgument) - - -ServiceAntiAffinity holds the parameters that are used to configure the corresponding priority function - - - - - - - - - - - - - -
FieldDescription
label [Required]
-string -		 - Used to identify node "groups"
- - - -## `UtilizationShapePoint` {#kubescheduler-config-k8s-io-v1-UtilizationShapePoint} - - - - -**Appears in:** - -- [RequestedToCapacityRatioArguments](#kubescheduler-config-k8s-io-v1-RequestedToCapacityRatioArguments) - - -UtilizationShapePoint represents single point of priority function shape. - - - - - - - - - - - - - - - - - - -
FieldDescription
utilization [Required]
-int32 -		 - Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score [Required]
-int32 -		 - Score assigned to given utilization (y axis). Valid values are 0 to 10.
- - diff --git a/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md b/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md index 77595b4599..a6e2bd98cd 100644 --- a/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md +++ b/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md @@ -4,272 +4,245 @@ content_type: tool-reference package: kubeadm.k8s.io/v1beta2 auto_generated: true --- -Package v1beta2 defines the v1beta2 version of the kubeadm configuration file format. -This version improves on the v1beta1 format by fixing some minor issues and adding a few new fields. - -A list of changes since v1beta1: - -- `certificateKey" field is added to InitConfiguration and JoinConfiguration. -- "ignorePreflightErrors" field is added to the NodeRegistrationOptions. -- The JSON "omitempty" tag is used in a more places where appropriate. -- The JSON "omitempty" tag of the "taints" field (inside NodeRegistrationOptions) is removed. -See the Kubernetes 1.15 changelog for further details. - -## Migration from old kubeadm config versions - -Please convert your v1beta1 configuration files to v1beta2 using the "kubeadm config migrate" command of kubeadm v1.15.x -(conversion from older releases of kubeadm config files requires older release of kubeadm as well e.g. - -- kubeadm v1.11 should be used to migrate v1alpha1 to v1alpha2; kubeadm v1.12 should be used to translate v1alpha2 to v1alpha3; -- kubeadm v1.13 or v1.14 should be used to translate v1alpha3 to v1beta1) - -Nevertheless, kubeadm v1.15.x will support reading from v1beta1 version of the kubeadm config file format. - -## Basics - -The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the +

Overview

+

Package v1beta2 defines the v1beta2 version of the kubeadm configuration file format. +This version improves on the v1beta1 format by fixing some minor issues and adding a few new fields.

+

A list of changes since v1beta1:

+
    +
  • "certificateKey" field is added to InitConfiguration and JoinConfiguration.
    • +
  • "ignorePreflightErrors" field is added to the NodeRegistrationOptions.
    • +
  • The JSON "omitempty" tag is used in a more places where appropriate.
    • +
  • The JSON "omitempty" tag of the "taints" field (inside NodeRegistrationOptions) is removed.
    • +
+

See the Kubernetes 1.15 changelog for further details.

+

Migration from old kubeadm config versions

+

Please convert your v1beta1 configuration files to v1beta2 using the "kubeadm config migrate" command of kubeadm v1.15.x +(conversion from older releases of kubeadm config files requires older release of kubeadm as well e.g.

+
    +
  • kubeadm v1.11 should be used to migrate v1alpha1 to v1alpha2; kubeadm v1.12 should be used to translate v1alpha2 to v1alpha3;
    • +
  • kubeadm v1.13 or v1.14 should be used to translate v1alpha3 to v1beta1)
    • +
+

Nevertheless, kubeadm v1.15.x will support reading from v1beta1 version of the kubeadm config file format.

+

Basics

+

The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the configuration options defined in the kubeadm config file are also available as command line flags, but only -the most common/simple use case are supported with this approach. - -A kubeadm config file could contain multiple configuration types separated using three dashes (“---”). - -kubeadm supports the following configuration types: - -```yaml -apiVersion: kubeadm.k8s.io/v1beta2 -kind: InitConfiguration - -apiVersion: kubeadm.k8s.io/v1beta2 -kind: ClusterConfiguration - -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration - -apiVersion: kubeproxy.config.k8s.io/v1alpha1 -kind: KubeProxyConfiguration - -apiVersion: kubeadm.k8s.io/v1beta2 -kind: JoinConfiguration -``` - -To print the defaults for "init" and "join" actions use the following commands: - -```shell -kubeadm config print init-defaults +the most common/simple use case are supported with this approach.

+

A kubeadm config file could contain multiple configuration types separated using three dashes (---).

+

kubeadm supports the following configuration types:

+
apiVersion: kubeadm.k8s.io/v1beta2
+kind: InitConfiguration
+
+apiVersion: kubeadm.k8s.io/v1beta2
+kind: ClusterConfiguration
+
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+
+apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+
+apiVersion: kubeadm.k8s.io/v1beta2
+kind: JoinConfiguration
+

To print the defaults for "init" and "join" actions use the following commands:

+
kubeadm config print init-defaults
 kubeadm config print join-defaults
-```
-
-The list of configuration types that must be included in a configuration file depends by the action you are
-performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).
-
-If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults
+

The list of configuration types that must be included in a configuration file depends by the action you are +performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).

+

If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults provided by kubeadm includes also enforcing consistency of values across components when required (e.g. -cluster-cidr flag on controller manager and clusterCIDR on kube-proxy). - -Users are always allowed to override default values, with the only exception of a small subset of setting with -relevance for security (e.g. enforce authorization-mode Node and RBAC on api server) - -If the user provides a configuration types that is not expected for the action you are performing, kubeadm will -ignore those types and print a warning. - -## Kubeadm init configuration types - -When executing kubeadm init with the `--config` option, the following configuration types could be used: +--cluster-cidr flag on controller manager and clusterCIDR on kube-proxy).

+

Users are always allowed to override default values, with the only exception of a small subset of setting with +relevance for security (e.g. enforce authorization-mode Node and RBAC on API server)

+

If the user provides a configuration types that is not expected for the action you are performing, kubeadm will +ignore those types and print a warning.

+

Kubeadm init configuration types

+

When executing kubeadm init with the --config option, the following configuration types could be used: InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one -between InitConfiguration and ClusterConfiguration is mandatory. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta2 -kind: InitConfiguration -bootstrapTokens: - ... -nodeRegistration: - ... -``` - -The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init +between InitConfiguration and ClusterConfiguration is mandatory.

+
apiVersion: kubeadm.k8s.io/v1beta2
+kind: InitConfiguration
+bootstrapTokens:
+  ...
+nodeRegistration:
+  ...
+

The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm -is executed, including: - -- NodeRegistration, that holds fields that relate to registering the new node to the cluster; - use it to customize the node name, the CRI socket to use or any other settings that should apply to this - node only (e.g. the node ip). - -- LocalAPIEndpoint, that represents the endpoint of the instance of the API server to be deployed on this node; - use it e.g. to customize the API server advertise address. - - ```yaml - apiVersion: kubeadm.k8s.io/v1beta2 - kind: ClusterConfiguration - networking: - ... - etcd: - ... - apiServer: - extraArgs: - ... - extraVolumes: - ... - ``` - -The ClusterConfiguration type should be used to configure cluster-wide settings, -including settings for: - -- Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize - pod subnet or services subnet. -- Etcd configurations; use it e.g. to customize the local etcd or to configure the API server - for using an external etcd cluster. -- kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane - components by adding customized setting or overriding kubeadm default settings. - - ```yaml - apiVersion: kubeproxy.config.k8s.io/v1alpha1 - kind: KubeProxyConfiguration - ... - ``` - -The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed -in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults. - -See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration -for kube proxy official documentation. - -```yaml -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -... -``` - -The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances -deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults. - -See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration -for kubelet official documentation. - -Here is a fully populated example of a single YAML file containing multiple -configuration types to be used during a `kubeadm init` run. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta2 -kind: InitConfiguration -bootstrapTokens: - - token: "9a08jv.c0izixklcxtmnze7" - description: "kubeadm bootstrap token" - ttl: "24h" - - token: "783bde.3f89s0fje9f38fhf" - description: "another bootstrap token" - usages: - - authentication - - signing - groups: - - system:bootstrappers:kubeadm:default-node-token -nodeRegistration: - name: "ec2-10-100-0-1" - criSocket: "/var/run/dockershim.sock" - taints: - - key: "kubeadmNode" - value: "master" - effect: "NoSchedule" - kubeletExtraArgs: - cgroup-driver: "cgroupfs" - ignorePreflightErrors: - - IsPrivilegedUser -localAPIEndpoint: - advertiseAddress: "10.100.0.1" - bindPort: 6443 -certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204" ---- -apiVersion: kubeadm.k8s.io/v1beta2 -kind: ClusterConfiguration -etcd: - # one of local or external - local: - imageRepository: "k8s.gcr.io" - imageTag: "3.2.24" - dataDir: "/var/lib/etcd" - extraArgs: - listen-client-urls: "http://10.100.0.1:2379" - serverCertSANs: - - "ec2-10-100-0-1.compute-1.amazonaws.com" - peerCertSANs: - - "10.100.0.1" - # external: - # endpoints: - # - "10.100.0.1:2379" - # - "10.100.0.2:2379" - # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt" - # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt" - # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key" - networking: - serviceSubnet: "10.96.0.0/12" - podSubnet: "10.100.0.1/24" - dnsDomain: "cluster.local" - kubernetesVersion: "v1.12.0" - controlPlaneEndpoint: "10.100.0.1:6443" - apiServer: - extraArgs: - authorization-mode: "Node,RBAC" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File - certSANs: - - "10.100.1.1" - - "ec2-10-100-0-1.compute-1.amazonaws.com" - timeoutForControlPlane: 4m0s - controllerManager: - extraArgs: - "node-cidr-mask-size": "20" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File - scheduler: - extraArgs: - address: "10.100.0.1" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File -certificatesDir: "/etc/kubernetes/pki" -imageRepository: "k8s.gcr.io" -useHyperKubeImage: false -clusterName: "example-cluster" ---- -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -# kubelet specific options here ---- -apiVersion: kubeproxy.config.k8s.io/v1alpha1 -kind: KubeProxyConfiguration -# kube-proxy specific options here -``` - -## Kubeadm join configuration types - -When executing kubeadm join with the `--config` option, the JoinConfiguration type should be provided. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta2 -kind: JoinConfiguration -... -``` - -The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join +is executed, including:

+
    +
  • +

    nodeRegistration, that holds fields that relate to registering the new node to the cluster; +use it to customize the node name, the CRI socket to use or any other settings that should apply to this +node only (e.g. the node ip).

    +
    • +
  • +

    apiServer, that represents the endpoint of the instance of the API server to be deployed on this node; +use it e.g. to customize the API server advertise address.

    +
    • +
+
apiVersion: kubeadm.k8s.io/v1beta2
+kind: ClusterConfiguration
+networking:
+    ...
+etcd:
+    ...
+apiServer:
+  extraArgs:
+    ...
+  extraVolumes:
+    ...
+...
+

The ClusterConfiguration type should be used to configure cluster-wide settings, +including settings for:

+
    +
  • +

    Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize +pod subnet or services subnet.

    +
    • +
  • +

    Etcd configurations; use it e.g. to customize the local etcd or to configure the API server +for using an external etcd cluster.

    +
    • +
  • +

    kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane +components by adding customized setting or overriding kubeadm default settings.

    +
    • +
+
apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+  ...
+

The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed +in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

+

See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or +https://pkg.go.dev/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration +for kube proxy official documentation.

+
apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+  ...
+

The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances +deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

+

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or +https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration +for kubelet official documentation.

+

Here is a fully populated example of a single YAML file containing multiple +configuration types to be used during a kubeadm init run.

+
apiVersion: kubeadm.k8s.io/v1beta2
+kind: InitConfiguration
+bootstrapTokens:
+  - token: "9a08jv.c0izixklcxtmnze7"
+    description: "kubeadm bootstrap token"
+    ttl: "24h"
+  - token: "783bde.3f89s0fje9f38fhf"
+    description: "another bootstrap token"
+    usages:
+      - authentication
+      - signing
+    groups:
+      - system:bootstrappers:kubeadm:default-node-token
+nodeRegistration:
+  name: "ec2-10-100-0-1"
+  criSocket: "/var/run/dockershim.sock"
+  taints:
+    - key: "kubeadmNode"
+      value: "someValue"
+      effect: "NoSchedule"
+  kubeletExtraArgs:
+    v: 4
+  ignorePreflightErrors:
+    - IsPrivilegedUser
+localAPIEndpoint:
+  advertiseAddress: "10.100.0.1"
+  bindPort: 6443
+certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
+---
+apiVersion: kubeadm.k8s.io/v1beta2
+kind: ClusterConfiguration
+etcd:
+  # one of local or external
+  local:
+    imageRepository: "k8s.gcr.io"
+    imageTag: "3.2.24"
+    dataDir: "/var/lib/etcd"
+    extraArgs:
+      listen-client-urls: "http://10.100.0.1:2379"
+    serverCertSANs:
+      -  "ec2-10-100-0-1.compute-1.amazonaws.com"
+    peerCertSANs:
+      - "10.100.0.1"
+  # external:
+  #   endpoints:
+  #     - "10.100.0.1:2379"
+  #     - "10.100.0.2:2379"
+  #   caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
+  #   certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
+  #   keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
+networking:
+  serviceSubnet: "10.96.0.0/16"
+  podSubnet: "10.244.0.0/24"
+  dnsDomain: "cluster.local"
+kubernetesVersion: "v1.12.0"
+controlPlaneEndpoint: "10.100.0.1:6443"
+apiServer:
+  extraArgs:
+    authorization-mode: "Node,RBAC"
+  extraVolumes:
+    - name: "some-volume"
+      hostPath: "/etc/some-path"
+      mountPath: "/etc/some-pod-path"
+      readOnly: false
+      pathType: File
+  certSANs:
+    - "10.100.1.1"
+    - "ec2-10-100-0-1.compute-1.amazonaws.com"
+  timeoutForControlPlane: 4m0s
+controllerManager:
+  extraArgs:
+    "node-cidr-mask-size": "20"
+  extraVolumes:
+    - name: "some-volume"
+      hostPath: "/etc/some-path"
+      mountPath: "/etc/some-pod-path"
+      readOnly: false
+      pathType: File
+scheduler:
+  extraArgs:
+    address: "10.100.0.1"
+  extraVolumes:
+    - name: "some-volume"
+      hostPath: "/etc/some-path"
+      mountPath: "/etc/some-pod-path"
+      readOnly: false
+      pathType: File
+certificatesDir: "/etc/kubernetes/pki"
+imageRepository: "k8s.gcr.io"
+useHyperKubeImage: false
+clusterName: "example-cluster"
+---
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+# kubelet specific options here
+---
+apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+# kube-proxy specific options here
+

Kubeadm join configuration types

+

When executing kubeadm join with the --config option, the JoinConfiguration type should be provided.

+
apiVersion: kubeadm.k8s.io/v1beta2
+kind: JoinConfiguration
+  ...
+

The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join are the discovery method used for accessing the cluster info and all the setting which are specific -to the node where kubeadm is executed, including: +to the node where kubeadm is executed, including:

+
    +
  • +

    NodeRegistration, that holds fields that relate to registering the new node to the cluster; +use it to customize the node name, the CRI socket to use or any other settings that should apply to this +node only (e.g. the node IP).

    +
    • +
  • +

    APIEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node.

    +
    • +
-- NodeRegistration, that holds fields that relate to registering the new node to the cluster; - use it to customize the node name, the CRI socket to use or any other settings that should apply to this - node only (e.g. the node ip). - -- APIEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node. ## Resource Types @@ -281,15 +254,13 @@ to the node where kubeadm is executed, including: - ## `ClusterConfiguration` {#kubeadm-k8s-io-v1beta2-ClusterConfiguration} +

ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster

-ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster - @@ -297,146 +268,130 @@ ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster - - +

etcd holds configuration for etcd.

+ - - +

networking holds configuration for the networking topology of the cluster.

+ - - +

kubernetesVersion is the target version of the control plane.

+ - - +In case the controlPlaneEndpoint is not specified, the advertiseAddress + bindPort +are used; in case the controlPlaneEndpoint is specified but without a TCP port, +the bindPort is used. +Possible usages are:

+
    +
  • In a cluster with more than one control plane instances, this field should be +assigned the address of the external load balancer in front of the +control plane instances.
    • +
  • In environments with enforced node recycling, the controlPlaneEndpoint +could be used for assigning a stable DNS to the control plane.
    • +
+ - - +

apiServer contains extra settings for the API server.

+ - - +

controllerManager contains extra settings for the controller manager.

+ - - +

scheduler contains extra settings for the scheduler.

+ - - +

dns defines the options for the DNS add-on installed in the cluster.

+ - - +

certificatesDir specifies where to store or look for all required certificates.

+ - - +

imageRepository sets the container registry to pull images from. +If empty, k8s.gcr.io will be used by default; in case of kubernetes version is +a CI build (kubernetes version starts with ci/) gcr.io/k8s-staging-ci-images +is used as a default for control plane components and for kube-proxy, while +k8s.gcr.io will be used for all the other images.

+ - - +

useHyperKubeImage controls if hyperkube should be used for Kubernetes components +instead of their respective separate images. +DEPRECATED: As hyperkube is itself deprecated, this fields is too. It will be +removed in future kubeadm config versions, kubeadm will print multiple warnings +when this set to true, and at some point it may become ignored.

+ - - +

featureGates contains the feature gates enabled by the user.

+ - - +

The cluster name.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta2
kind
string		ClusterConfiguration
etcd [Required]
Etcd 		- `etcd` holds configuration for etcd.
networking [Required]
Networking 		- `networking` holds configuration for the networking topology of the cluster.
kubernetesVersion [Required]
string 		- `kubernetesVersion` is the target version of the control plane.
controlPlaneEndpoint [Required]
string 		- `controlPlaneEndpoint` sets a stable IP address or DNS name for the control plane; it +

controlPlaneEndpoint sets a stable IP address or DNS name for the control plane; it can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port. -In case the ControlPlaneEndpoint is not specified, the AdvertiseAddress + BindPort -are used; in case the ControlPlaneEndpoint is specified but without a TCP port, -the BindPort is used. -Possible usages are: - -- In a cluster with more than one control plane instances, this field should be - assigned the address of the external load balancer in front of the - control plane instances. -- In environments with enforced node recycling, the ControlPlaneEndpoint - could be used for assigning a stable DNS to the control plane.
apiServer [Required]
APIServer 		- `apiServer` contains extra settings for the API server.
controllerManager [Required]
ControlPlaneComponent 		- `controllerManager` contains extra settings for the controller manager.
scheduler [Required]
ControlPlaneComponent 		- `scheduler` contains extra settings for the scheduler.
dns [Required]
DNS 		- `dns` defines the options for the DNS add-on.
certificatesDir [Required]
string 		- `certificatesDir` specifies where to store or look for all required certificates.
imageRepository [Required]
string 		- `imageRepository` sets the container registry to pull images from. -If empty, `k8s.gcr.io` will be used by default; in case of kubernetes version is -a CI build (kubernetes version starts with `ci/` or `ci-cross/`) -`gcr.io/k8s-staging-ci-images` will be used as a default for control plane -components and for kube-proxy, while `k8s.gcr.io` will be used for all the other images.
useHyperKubeImage [Required]
bool 		- `useHyperKubeImage` controls if hyperkube should be used for Kubernetes -components instead of their respective separate images -DEPRECATED: As hyperkube is itself deprecated, this fields is too. It will -be removed in future kubeadm config versions, kubeadm will print multiple -warnings when this is set to true, and at some point it may become ignored.
featureGates [Required]
map[string]bool 		- Feature gates enabled by the user.
clusterName [Required]
string 		- The cluster name
- - ## `ClusterStatus` {#kubeadm-k8s-io-v1beta2-ClusterStatus} +

ClusterStatus contains the cluster status. The ClusterStatus will be stored in +the kubeadm-config ConfigMap in the cluster, and then updated by kubeadm when +additional control plane instance joins or leaves the cluster.

-ClusterStatus contains the cluster status. The ClusterStatus will be stored in the kubeadm-config -ConfigMap in the cluster, and then updated by kubeadm when additional control plane instance joins or leaves the cluster. - @@ -444,32 +399,27 @@ ConfigMap in the cluster, and then updated by kubeadm when additional control pl - - +

apiEndpoints currently available in the cluster, one for each control +plane/API server instance. +The key of the map is the IP of the host's default interface.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta2
kind
string		ClusterStatus
apiEndpoints [Required]
map[string]github.com/tengqm/kubeconfig/config/kubeadm/v1beta2.APIEndpoint 		- `apiEndpoints` currently available in the cluster, one for each control -plane/API server instance. The key of the map is the IP of the host's default interface
- - ## `InitConfiguration` {#kubeadm-k8s-io-v1beta2-InitConfiguration} +

InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime +information.

-InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime -information. - @@ -477,61 +427,52 @@ information. - - +

bootstrapTokens is respected at kubeadm init time and describes a set of bootstrap tokens to create. +This information IS NOT uploaded to the kubeadm cluster ConfigMap, partly because of its sensitive nature.

+ - - +

nodeRegistration holds fields that relate to registering the new control-plane node to the cluster.

+ - - +fails you may set the desired value here.

+ - - +

certificateKey sets the key with which certificates and keys are encrypted prior to being uploaded in +a secret in the cluster during the uploadcerts init phase.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta2
kind
string		InitConfiguration
bootstrapTokens [Required]
[]BootstrapToken 		- `bootstrapTokens` is respected at `kubeadm init` time and describes a set of Bootstrap Tokens to create. -This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature
nodeRegistration [Required]
NodeRegistrationOptions 		- `nodeRegistration` holds fields that relate to registering the new control-plane node to the cluster
localAPIEndpoint [Required]
APIEndpoint 		- `localAPIEndpoint` represents the endpoint of the API server instance that's deployed on this control plane node -In HA setups, this differs from ClusterConfiguration.ControlPlaneEndpoint in the sense that ControlPlaneEndpoint -is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This +

localAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node. +In HA setups, this differs from ClusterConfiguration.controlPlaneEndpoint in the sense that ControlPlaneEndpoint +is the global endpoint for the cluster, which then load-balances the requests to each individual API server. This configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process -fails you may set the desired value here.
certificateKey [Required]
string 		- `certificateKey` sets the key with which certificates and keys are encrypted prior to being uploaded in -a Secret in the cluster during the "uploadcerts" init phase.
- - ## `JoinConfiguration` {#kubeadm-k8s-io-v1beta2-JoinConfiguration} +

JoinConfiguration contains elements describing a particular node.

-JoinConfiguration contains elements describing a particular node. - @@ -539,56 +480,46 @@ JoinConfiguration contains elements describing a particular node. - - +

nodeRegistration holds fields that relate to registering the new +control-plane node to the cluster

+ - - +

caCertPath is the path to the SSL certificate authority used to +secure comunications between a node and the control-plane. +Defaults to "/etc/kubernetes/pki/ca.crt".

+ - - +

discovery specifies the options for the kubelet to use during the TLS +bootstrap process.

+ - - +

controlPlane defines the additional control plane instance to be deployed +on the joining node. If nil, no additional control plane instance will be deployed.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta2
kind
string		JoinConfiguration
nodeRegistration [Required]
NodeRegistrationOptions 		- `nodeRegistration` holds fields that relate to registering the new control-plane -node to the cluster
caCertPath [Required]
string 		- `caCertPath` is the path to the SSL certificate authority used to -secure comunications between node and control-plane. -Defaults to "/etc/kubernetes/pki/ca.crt".
discovery [Required]
Discovery 		- `discovery` specifies the options for the kubelet to use during the TLS Bootstrap -process
controlPlane [Required]
JoinControlPlane 		- `controlPlane` defines the additional control plane instance to be deployed on the -joining node. If nil, no additional control plane instance will be deployed.
- - ## `APIEndpoint` {#kubeadm-k8s-io-v1beta2-APIEndpoint} - - **Appears in:** - [ClusterStatus](#kubeadm-k8s-io-v1beta2-ClusterStatus) @@ -598,272 +529,233 @@ joining node. If nil, no additional control plane instance will be deployed.APIEndpoint struct contains elements of API server instance deployed on a node.

+ - +

advertiseAddress sets the IP address for the API server to advertise.

+ - - +

bindPort sets the secure port for the API Server to bind to. +Defaults to 6443.

+ - -
FieldDescription
advertiseAddress [Required]
string 		- `advertiseAddress` sets the IP address for the API server to advertise.
bindPort [Required]
int32 		- `bindPort` sets the secure port for the API Server to bind to. Defaults to 6443.
- - ## `APIServer` {#kubeadm-k8s-io-v1beta2-APIServer} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta2-ClusterConfiguration) -APIServer holds settings necessary for API server deployments in the cluster +

APIServer holds settings necessary for API server deployments in the cluster.

+ - + No description provided. - - +

certSANs sets extra Subject Alternative Names (SANs) for the API Server +signing certificate.

+ - - +

timeoutForControlPlane controls the timeout that we wait for the API server +to appear.

+ - -
FieldDescription
ControlPlaneComponent [Required]
ControlPlaneComponent 		(Members of ControlPlaneComponent are embedded into this type.) - No description provided. -
certSANs [Required]
[]string 		- `certSANs` sets extra Subject Alternative Names for the API Server signing cert.
timeoutForControlPlane [Required]
-meta/v1.Duration +meta/v1.Duration 		- `timeoutForControlPlane` controls the timeout that we use for API server to appear
- - ## `BootstrapToken` {#kubeadm-k8s-io-v1beta2-BootstrapToken} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta2-InitConfiguration) -BootstrapToken describes one bootstrap token, stored as a Secret in the cluster +

BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

+ - +

token is used for establishing bidirectional trust between nodes and control-planes. +Used for joining nodes in the cluster.

+ - - +

description sets a human-friendly message why this token exists and what it's used +for, so other administrators can know its purpose.

+ - - +

ttl defines the time to live for this token. Defaults to '24h'. +expires and ttl are mutually exclusive.

+ - - +

expires specifies the timestamp when this token expires. Defaults to being set +dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

+ - - +

usages describes the ways in which this token can be used. Can by default be used +for establishing bidirectional trust, but that can be changed here.

+ - - +

groups specifies the extra groups that this token will authenticate as when/if +used for authentication.

+ - -
FieldDescription
token [Required]
BootstrapTokenString 		- `token` used for establishing bidirectional trust between nodes and control-planes. -Used for joining nodes in the cluster.
description [Required]
string 		- `description` sets a human-friendly message why this token exists and what it's used -for, so other administrators can know its purpose.
ttl [Required]
-meta/v1.Duration +meta/v1.Duration 		- `ttl` defines the time to live for this token. Defaults to "24h". -`expires` and `ttl` are mutually exclusive.
expires [Required]
-meta/v1.Time +meta/v1.Time 		- `expires` specifies the timestamp when this token expires. Defaults to being set -dynamically at runtime based on the `ttl`. `expires` and `ttl` are mutually exclusive.
usages [Required]
[]string 		- `usages` describes the ways in which this token can be used. Can by default be used -for establishing bidirectional trust, but that can be changed here.
groups [Required]
[]string 		- `groups` specifies the extra groups that this token will authenticate as when/if -used for authentication
- - ## `BootstrapTokenDiscovery` {#kubeadm-k8s-io-v1beta2-BootstrapTokenDiscovery} - - **Appears in:** - [Discovery](#kubeadm-k8s-io-v1beta2-Discovery) -BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery +

BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery

+ - +

token is a token used to validate cluster information fetched from +the control-plane.

+ - - +

apiServerEndpoint is an IP or domain name to the API server from which information +will be fetched.

+ - - +

caCertHashes specifies a set of public key pins to verify when token-based discovery +is used. The root CA found during discovery must match one of these values. +Specifying an empty set disables root CA pinning, which can be unsafe. +Each hash is specified as ":", where the only currently supported type is "sha256". +This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI) object in +DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.

+ - - +

unsafeSkipCAVerification allows token-based discovery without CA verification via +caCertHashes. This can weaken the security of kubeadm since other nodes can +impersonate the control-plane.

+ - -
FieldDescription
token [Required]
string 		- `token` is a token used to validate cluster information fetched from the control-plane.
apiServerEndpoint [Required]
string 		- `apiServerEndpoint` is an IP or domain name to the API server from which -information will be fetched.
caCertHashes [Required]
[]string 		- discovery is used. The root CA found during discovery must match one of these -values. Specifying an empty set disables root CA pinning, which can be unsafe. -Each hash is specified as `:`, where the only currently supported -type is "sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key -Info (SPKI) object in DER-encoded ASN.1. These hashes can be calculated using, -for example, OpenSSL.
unsafeSkipCAVerification [Required]
bool 		- `unsafeSkipCAVerification` allows token-based discovery without CA verification -via `caCertHashes`. This can weaken the security of kubeadm since other nodes -can impersonate the control-plane.
- - ## `BootstrapTokenString` {#kubeadm-k8s-io-v1beta2-BootstrapTokenString} - - **Appears in:** - [BootstrapToken](#kubeadm-k8s-io-v1beta2-BootstrapToken) -BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used +

BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used for both validation of the practically of the API server from a joining node's point of view and as an authentication method for the node in the bootstrap phase of -"kubeadm join". This token is and should be short-lived +"kubeadm join". This token is and should be short-lived

+ - + No description provided. - - + No description provided. - -
FieldDescription
- [Required]
string 		- No description provided. -
- [Required]
string 		- No description provided. -
- - ## `ControlPlaneComponent` {#kubeadm-k8s-io-v1beta2-ControlPlaneComponent} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta2-ClusterConfiguration) @@ -871,351 +763,305 @@ of view and as an authentication method for the node in the bootstrap phase of - [APIServer](#kubeadm-k8s-io-v1beta2-APIServer) -ControlPlaneComponent holds settings common to control plane component of the cluster +

ControlPlaneComponent holds settings common to control plane component of the cluster

+ - +

extraArgs is an extra set of flags to pass to a control plane component. +A key in this map is the flag name as it appears on the command line except +without leading dash(es).

+ - - +

extraVolumes is an extra set of host volumes mounted to the control plane +component.

+ - -
FieldDescription
extraArgs [Required]
map[string]string 		- `extraArgs` is an extra set of flags to pass to the control plane component.
extraVolumes [Required]
[]HostPathMount 		- `extraVolumes` is an extra set of host volumes, mounted to the control plane component.
- - ## `DNS` {#kubeadm-k8s-io-v1beta2-DNS} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta2-ClusterConfiguration) -DNS defines the DNS addon that should be used in the cluster +

DNS defines the DNS addon that should be used in the cluster

+ - +

type defines the DNS add-on to be used.

+ - - +

ImageMeta allows to customize the image used for the DNS component

+ - -
FieldDescription
type [Required]
DNSAddOnType 		- `type` defines the DNS add-on to use.
ImageMeta [Required]
ImageMeta 		(Members of ImageMeta are embedded into this type.) - `imageMeta` allows to customize the image used for the DNS.
- - ## `DNSAddOnType` {#kubeadm-k8s-io-v1beta2-DNSAddOnType} (Alias of `string`) - **Appears in:** - [DNS](#kubeadm-k8s-io-v1beta2-DNS) -DNSAddOnType defines string identifying DNS add-on types +

DNSAddOnType defines string identifying DNS add-on types.

- ## `Discovery` {#kubeadm-k8s-io-v1beta2-Discovery} - - **Appears in:** - [JoinConfiguration](#kubeadm-k8s-io-v1beta2-JoinConfiguration) -Discovery specifies the options for the kubelet to use during the TLS Bootstrap process +

Discovery specifies the options for the kubelet to use during the TLS Bootstrap process

+ - +

bootstrapToken is used to set the options for bootstrap token based discovery. +bootstrapToken and file are mutually exclusive.

+ - - +

file is used to specify a file or URL to a kubeconfig file from which to load +cluster information. +bootstrapToken and file are mutually exclusive.

+ - - +If file is set, this field must be set in case the KubeConfigFile does not +contain any other authentication information.

+ - - +

timeout modifies the discovery timeout.

+ - -
FieldDescription
bootstrapToken [Required]
BootstrapTokenDiscovery 		- `bootstrapToken` is used to set the options for bootstrap token based discovery. -`bootstrapToken` and `file` are mutually exclusive.
file [Required]
FileDiscovery 		- `file` specifies a file or URL to a kubeconfig file from which to load cluster information. -`bootstrapToken` and `file` are mutually exclusive.
tlsBootstrapToken [Required]
string 		- `tlsBootstrapToken` is a token used for TLS bootstrapping. -If `bootstrapToken` is set, this field is defaulted to `bootstrapToken.token`, +

tlsBootstrapToken is a token used for TLS bootstrapping. +If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, but can be overridden. -If `file` is set, this field ∗∗must be set∗∗ in case the KubeConfigFile does -not contain any other authentication information
timeout [Required]
-meta/v1.Duration +meta/v1.Duration 		- `timeout` modifies the discovery timeout.
- - ## `Etcd` {#kubeadm-k8s-io-v1beta2-Etcd} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta2-ClusterConfiguration) -Etcd contains elements describing Etcd configuration. +

Etcd contains elements describing Etcd configuration.

+ - +

local provides configuration knobs for configuring the local etcd instance. +local and external are mutually exclusive.

+ - - +

external describes how to connect to an external etcd cluster. +local and external are mutually exclusive.

+ - -
FieldDescription
local [Required]
LocalEtcd 		- `local` provides configuration knobs for configuring the local etcd instance. -`local` and `external` are mutually exclusive.
external [Required]
ExternalEtcd 		- `external` describes how to connect to an external etcd cluster. -`local` and `external` are mutually exclusive.
- - ## `ExternalEtcd` {#kubeadm-k8s-io-v1beta2-ExternalEtcd} - - **Appears in:** - [Etcd](#kubeadm-k8s-io-v1beta2-Etcd) -ExternalEtcd describes an external etcd cluster. -Kubeadm has no knowledge of where certificate files live and they must be supplied. +

ExternalEtcd describes an external etcd cluster. +Kubeadm has no knowledge of where certificate files live and they must be supplied.

+ - +

endpoints of etcd members.

+ - - +

caFile is an SSL Certificate Authority (CA) file used to secure etcd communication. +Required if using a TLS connection.

+ - - +

certFile is an SSL certification file used to secure etcd communication. +Required if using a TLS connection.

+ - - +

keyFile is an SSL key file used to secure etcd communication. +Required if using a TLS connection.

+ - -
FieldDescription
endpoints [Required]
[]string 		- `endpoints` are endpoints of etcd members. This field is required.
caFile [Required]
string 		- `caFile` is an SSL Certificate Authority file used to secure etcd communication. -Required if using a TLS connection.
certFile [Required]
string 		- `certFile` is an SSL certification file used to secure etcd communication. -Required if using a TLS connection.
keyFile [Required]
string 		- `keyFile` is an SSL key file used to secure etcd communication. -Required if using a TLS connection.
- - ## `FileDiscovery` {#kubeadm-k8s-io-v1beta2-FileDiscovery} - - **Appears in:** - [Discovery](#kubeadm-k8s-io-v1beta2-Discovery) -FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information +

FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information

+ - +

kubeConfigPath is used to specify the actual file path or URL to the kubeconfig file +from which to load cluster information.

+ - -
FieldDescription
kubeConfigPath [Required]
string 		- `kubeConfigPath` specifies the actual file path or URL to the kubeconfig file -from which to load cluster information
- - ## `HostPathMount` {#kubeadm-k8s-io-v1beta2-HostPathMount} - - **Appears in:** - [ControlPlaneComponent](#kubeadm-k8s-io-v1beta2-ControlPlaneComponent) -HostPathMount contains elements describing volumes that are mounted from the host. +

HostPathMount contains elements describing volumes that are mounted from the host.

+ - +

name of the volume inside the Pod template.

+ - - +

hostPath is the path in the host that will be mounted inside the Pod.

+ - - +

mountPathis the path inside the Pod where hostPath volume will be mounted.

+ - - +

readOnly controls write access to the volume.

+ - - +

pathType is the type of the HostPath.

+ - -
FieldDescription
name [Required]
string 		- `name` is the volume name inside the Pod template.
hostPath [Required]
string 		- `hostPath` is the path in the host that will be mounted inside the Pod.
mountPath [Required]
string 		- `mountPath` is the path inside the Pod where the `hostPath` volume is mounted.
readOnly [Required]
bool 		- `readOnly` controls write access to the volume.
pathType [Required]
-core/v1.HostPathType +core/v1.HostPathType 		- `pathType` is the type of the `hostPath` volume.
- - ## `ImageMeta` {#kubeadm-k8s-io-v1beta2-ImageMeta} - - **Appears in:** - [DNS](#kubeadm-k8s-io-v1beta2-DNS) @@ -1223,198 +1069,175 @@ HostPathMount contains elements describing volumes that are mounted from the hos - [LocalEtcd](#kubeadm-k8s-io-v1beta2-LocalEtcd) -ImageMeta allows to customize the image used for components that are not -originated from the Kubernetes/Kubernetes release process +

ImageMeta allows to customize the image used for components that are not +originated from the Kubernetes/Kubernetes release process

+ - +

imageRepository sets the container registry to pull images from. +If not set, the imageRepository defined in ClusterConfiguration will be used.

+ - - +version of the above components during upgrades.

+ - -
FieldDescription
imageRepository [Required]
string 		- `imageRepository` sets the container registry to pull images from. -If not set, the ImageRepository defined in ClusterConfiguration will be used instead.
imageTag [Required]
string 		- `imageTag` allows to specify a tag for the image. +

imageTag allows for specifying a tag for the image. In case this value is set, kubeadm does not change automatically the -version of the above components during upgrades.
- - ## `JoinControlPlane` {#kubeadm-k8s-io-v1beta2-JoinControlPlane} - - **Appears in:** - [JoinConfiguration](#kubeadm-k8s-io-v1beta2-JoinConfiguration) -JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node. +

JoinControlPlane contains elements describing an additional control plane instance +to be deployed on the joining node.

+ - +

localAPIEndpoint represents the endpoint of the API server instance +to be deployed on this node.

+ - - +

certificateKey is the key that is used for decryption of certificates after +they are downloaded from the secret upon joining a new control plane node. +The corresponding encryption key is in the InitConfiguration.

+ - -
FieldDescription
localAPIEndpoint [Required]
APIEndpoint 		- `localAPIEndpoint` represents the endpoint of the API server instance to be deployed -on this node.
certificateKey [Required]
string 		- `certificateKey` is the key that is used for decryption of certificates after they -are downloaded from the secret upon joining a new control plane node. The -corresponding encryption key is in the InitConfiguration.
- - ## `LocalEtcd` {#kubeadm-k8s-io-v1beta2-LocalEtcd} - - **Appears in:** - [Etcd](#kubeadm-k8s-io-v1beta2-Etcd) -LocalEtcd describes that kubeadm should run an etcd cluster locally +

LocalEtcd describes that kubeadm should run an etcd cluster locally.

+ - +

ImageMeta allows to customize the container used for etcd.

+ - - +

dataDir is the directory etcd will place its data. +Defaults to "/var/lib/etcd".

+ - - +

extraArgs are extra arguments provided to the etcd binary when run +inside a static pod. +A key in this map is the flag name as it appears on the +command line except without leading dash(es).

+ - - +

serverCertSANs sets extra Subject Alternative Names (SANs) for the +etcd server signing certificate.

+ - - +

peerCertSANs sets extra Subject Alternative Names (SANs) for the +etcd peer signing certificate.

+ - -
FieldDescription
ImageMeta [Required]
ImageMeta 		(Members of ImageMeta are embedded into this type.) - `ImageMeta` allows to customize the container used for etcd.
dataDir [Required]
string 		- `dataDir` is the directory etcd will place its data. -Defaults to "/var/lib/etcd".
extraArgs [Required]
map[string]string 		- `extraArgs` are extra arguments provided to the etcd binary -when run inside a static pod.
serverCertSANs [Required]
[]string 		- `serverCertSANs` sets extra Subject Alternative Names for the etcd server signing cert.
peerCertSANs [Required]
[]string 		- `peerCertSANs` sets extra Subject Alternative Names for the etcd peer signing cert.
- - ## `Networking` {#kubeadm-k8s-io-v1beta2-Networking} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta2-ClusterConfiguration) -Networking contains elements describing cluster's networking configuration +

Networking contains elements describing cluster's networking configuration

+ - +

serviceSubnet is the subnet used by kubernetes Services. Defaults to "10.96.0.0/12".

+ - - +

podSubnet is the subnet used by Pods.

+ - - +

dnsDomain is the DNS domain used by kubernetes Services. Defaults to "cluster.local".

+ - -
FieldDescription
serviceSubnet [Required]
string 		- `serviceSubnet` is the subnet used by k8s services. Defaults to "10.96.0.0/12".
podSubnet [Required]
string 		- `podSubnet` is the subnet used by Pods.
dnsDomain [Required]
string 		- `dnsDomain` is the DNS domain used by k8s services. Defaults to "cluster.local".
- - ## `NodeRegistrationOptions` {#kubeadm-k8s-io-v1beta2-NodeRegistrationOptions} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta2-InitConfiguration) @@ -1422,68 +1245,66 @@ Networking contains elements describing cluster's networking configuration - [JoinConfiguration](#kubeadm-k8s-io-v1beta2-JoinConfiguration) -NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via "kubeadm init" or "kubeadm join" +

NodeRegistrationOptions holds fields that relate to registering a new control-plane +or node to the cluster, either via "kubeadm init" or "kubeadm join".

+ - +

name is the .Metadata.Name field of the Node API object that will be created +in this kubeadm init or kubeadm join operation. +This field is also used in the CommonName field of the kubelet's client certificate +to the API server. +Defaults to the hostname of the node if not provided.

+ - - +

criSocket is used to retrieve container runtime information. This information will +be annotated to the Node API object, for later re-use.

+ - - +

taints specifies the taints the Node API object should be registered with. +If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted with +a control-plane taint for control-plane nodes. If you don't want to taint your control-plane +node, set this field to an empty list, i.e. taints: [], in the YAML file. This field is +solely used for Node registration.

+ - - +

kubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are +passed to the kubelet command line via the environment file kubeadm writes at runtime for +the kubelet to source. This overrides the generic base-level configuration in the +'kubelet-config-1.X' ConfigMap. +Flags have higher priority when parsing. These values are local and specific to the node +kubeadm is executing on. +A key in this map is the flag name as it appears on the command line except without leading dash(es).

+ - - +

ignorePreflightErrors provides a list of pre-flight errors to be ignored when the +current node is registered.

+ - -
FieldDescription
name [Required]
string 		- `name` is the `.metadata.name` field of the Node API object that will be created in this -`kubeadm init` or `kubeadm join` operation. -This field is also used in the CommonName field of the kubelet's client certificate to the -API server. Defaults to the hostname of the node if not provided.
criSocket [Required]
string 		- `criSocket` is used to retrieve container runtime info. This information will be -annotated to the Node API object, for later re-use.
taints [Required]
-[]core/v1.Taint +[]core/v1.Taint 		- `taints` specifies the taints the Node API object should be registered with. If -this field is unset, i.e. nil, in the `kubeadm init` process, it will be defaulted -to `['"node-role.kubernetes.io/master"=""']`. If you don't want to taint your -control-plane node, set this field to an empty list, i.e. `taints: []` in the YAML -file. This field is solely used for Node registration.
kubeletExtraArgs [Required]
map[string]string 		- `kubeletExtraArgs` passes through extra arguments to the kubelet. The arguments here -are passed to the kubelet command line via the environment file kubeadm writes at -runtime for the kubelet to source. This overrides the generic base-level -configuration in the "kubelet-config-1.X" ConfigMap. Flags have higher priority when -parsing. These values are local and specific to the node kubeadm is executing on.
ignorePreflightErrors [Required]
[]string 		- `ignorePreflightErrors` provides a slice of pre-flight errors to be ignored when -the current node is registered.
- - + \ No newline at end of file diff --git a/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md b/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md index 5f73e8b3a8..20f5e44d93 100644 --- a/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md +++ b/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md @@ -4,284 +4,256 @@ content_type: tool-reference package: kubeadm.k8s.io/v1beta3 auto_generated: true --- -Package v1beta3 defines the v1beta3 version of the kubeadm configuration file format. -This version improves on the v1beta2 format by fixing some minor issues and adding a few new fields. - -A list of changes since v1beta2: - -- The deprecated `ClusterConfiguration.useHyperKubeImage` field has been removed. - Kubeadm no longer supports the hyperkube image. -- The `ClusterConfiguration.dns.type` field has been removed since CoreDNS is the only supported - DNS server type by kubeadm. -- Include "datapolicy" tags on the fields that hold secrets. - This would result in the field values to be omitted when API structures are printed with klog. -- Add `InitConfiguration.skipPhases`, `JoinConfiguration.skipPhases` to allow skipping - a list of phases during kubeadm init/join command execution. -- Add `InitConfiguration.nodeRegistration.imagePullPolicy" and - `JoinConfiguration.nodeRegistration.imagePullPolicy` to allow specifying - the images pull policy during kubeadm "init" and "join". The value must be - one of "Always", "Never" or "IfNotPresent". "IfNotPresent" is the default, - which has been the existing behavior prior to this addition. -- Add `InitConfiguration.patches.directory`, `JoinConfiguration.patches.directory` - to allow the user to configure a directory from which to take patches for - components deployed by kubeadm. -- Move the `BootstrapToken∗` API and related utilities out of the "kubeadm" API group - to a new group "bootstraptoken". The kubeadm API version v1beta3 no longer contains - the `BootstrapToken∗` structures. - -## Migration from old kubeadm config versions - -- kubeadm v1.15.x and newer can be used to migrate from the v1beta1 to v1beta2. -- kubeadm v1.22.x no longer supports v1beta1 and older APIs, but can be used to migrate v1beta2 to v1beta3. - -## Basics - -The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the +

Overview

+

Package v1beta3 defines the v1beta3 version of the kubeadm configuration file format. +This version improves on the v1beta2 format by fixing some minor issues and adding a few new fields.

+

A list of changes since v1beta2:

+
    +
  • The deprecated "ClusterConfiguration.useHyperKubeImage" field has been removed. +Kubeadm no longer supports the hyperkube image.
    • +
  • The "ClusterConfiguration.DNS.Type" field has been removed since CoreDNS is the only supported +DNS server type by kubeadm.
    • +
  • Include "datapolicy" tags on the fields that hold secrets. +This would result in the field values to be omitted when API structures are printed with klog.
    • +
  • Add "InitConfiguration.SkipPhases", "JoinConfiguration.SkipPhases" to allow skipping +a list of phases during kubeadm init/join command execution.
    • +
  • Add "InitConfiguration.NodeRegistration.ImagePullPolicy" and "JoinConfiguration.NodeRegistration.ImagePullPolicy" +to allow specifying the images pull policy during kubeadm "init" and "join". +The value must be one of "Always", "Never" or "IfNotPresent". +"IfNotPresent" is the default, which has been the existing behavior prior to this addition.
    • +
  • Add "InitConfiguration.Patches.Directory", "JoinConfiguration.Patches.Directory" to allow +the user to configure a directory from which to take patches for components deployed by kubeadm.
    • +
  • Move the BootstrapToken* API and related utilities out of the "kubeadm" API group to a new group +"bootstraptoken". The kubeadm API version v1beta3 no longer contains the BootstrapToken* structures.
    • +
+

Migration from old kubeadm config versions

+
    +
  • kubeadm v1.15.x and newer can be used to migrate from v1beta1 to v1beta2.
    • +
  • kubeadm v1.22.x and newer no longer support v1beta1 and older APIs, but can be used to migrate v1beta2 to v1beta3.
    • +
+

Basics

+

The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the configuration options defined in the kubeadm config file are also available as command line flags, but only -the most common/simple use case are supported with this approach. - -A kubeadm config file could contain multiple configuration types separated using three dashes (“---”). - -kubeadm supports the following configuration types: - -```yaml -apiVersion: kubeadm.k8s.io/v1beta3 -kind: InitConfiguration ---- -apiVersion: kubeadm.k8s.io/v1beta3 -kind: ClusterConfiguration ---- -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration ---- -apiVersion: kubeproxy.config.k8s.io/v1alpha1 -kind: KubeProxyConfiguration ---- -apiVersion: kubeadm.k8s.io/v1beta3 -kind: JoinConfiguration -``` - -To print the defaults for "init" and "join" actions use the following commands: - -```shell -kubeadm config print init-defaults +the most common/simple use case are supported with this approach.

+

A kubeadm config file could contain multiple configuration types separated using three dashes (---).

+

kubeadm supports the following configuration types:

+
apiVersion: kubeadm.k8s.io/v1beta3
+kind: InitConfiguration
+
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: ClusterConfiguration
+
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+
+apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: JoinConfiguration
+

To print the defaults for "init" and "join" actions use the following commands:

+
kubeadm config print init-defaults
 kubeadm config print join-defaults
-```
-
-The list of configuration types that must be included in a configuration file depends by the action you are
-performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).
-
-If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults
+

The list of configuration types that must be included in a configuration file depends by the action you are +performing (init or join) and by the configuration options you are going to use (defaults or advanced +customization).

+

If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults provided by kubeadm includes also enforcing consistency of values across components when required (e.g. -cluster-cidr flag on controller manager and clusterCIDR on kube-proxy). - -Users are always allowed to override default values, with the only exception of a small subset of setting with -relevance for security (e.g. enforce authorization-mode Node and RBAC on api server) - -If the user provides a configuration types that is not expected for the action you are performing, kubeadm will -ignore those types and print a warning. - -## Kubeadm init configuration types - -When executing kubeadm init with the `--config` option, the following configuration types could be used: +--cluster-cidr flag on controller manager and clusterCIDR on kube-proxy).

+

Users are always allowed to override default values, with the only exception of a small subset of setting with +relevance for security (e.g. enforce authorization-mode Node and RBAC on api server)

+

If the user provides a configuration types that is not expected for the action you are performing, kubeadm will +ignore those types and print a warning.

+

Kubeadm init configuration types

+

When executing kubeadm init with the --config option, the following configuration types could be used: InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one -between InitConfiguration and ClusterConfiguration is mandatory. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta3 -kind: InitConfiguration -bootstrapTokens: - ... -nodeRegistration: - ... -``` - -The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init -are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm -is executed, including: - -- NodeRegistration, that holds fields that relate to registering the new node to the cluster; - use it to customize the node name, the CRI socket to use or any other settings that should apply to this - node only (e.g. the node ip). - -- LocalAPIEndpoint, that represents the endpoint of the instance of the API server to be deployed on this node; - use it e.g. to customize the API server advertise address. - - ```yaml - apiVersion: kubeadm.k8s.io/v1beta3 - kind: ClusterConfiguration - networking: - ... - etcd: - ... - apiServer: - extraArgs: - ... - extraVolumes: - ... - ... - ``` - -The ClusterConfiguration type should be used to configure cluster-wide settings, -including settings for: - -- Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize - pod subnet or services subnet. -- Etcd configurations; use it e.g. to customize the local etcd or to configure the API server - for using an external etcd cluster. -- kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane - components by adding customized setting or overriding kubeadm default settings. - - ```yaml - apiVersion: kubeproxy.config.k8s.io/v1alpha1 - kind: KubeProxyConfiguration - ... - ``` - -The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed -in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults. - -See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration -for kube proxy official documentation. - -```yaml -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -... -``` - -The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances -deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults. - -See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration -for kubelet official documentation. - -Here is a fully populated example of a single YAML file containing multiple -configuration types to be used during a `kubeadm init` run. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta3 -kind: InitConfiguration -bootstrapTokens: - - token: "9a08jv.c0izixklcxtmnze7" - description: "kubeadm bootstrap token" - ttl: "24h" - - token: "783bde.3f89s0fje9f38fhf" - description: "another bootstrap token" - usages: - - authentication - - signing - groups: - - system:bootstrappers:kubeadm:default-node-token -nodeRegistration: - name: "ec2-10-100-0-1" - criSocket: "/var/run/dockershim.sock" - taints: - - key: "kubeadmNode" - value: "master" - effect: "NoSchedule" - kubeletExtraArgs: - v: 4 - ignorePreflightErrors: - - IsPrivilegedUser - imagePullPolicy: "IfNotPresent" -localAPIEndpoint: - advertiseAddress: "10.100.0.1" - bindPort: 6443 -certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204" -skipPhases: - - add/kube-proxy ---- -apiVersion: kubeadm.k8s.io/v1beta3 -kind: ClusterConfiguration -etcd: - # one of local or external - local: - imageRepository: "k8s.gcr.io" - imageTag: "3.2.24" - dataDir: "/var/lib/etcd" - extraArgs: - listen-client-urls: "http://10.100.0.1:2379" - serverCertSANs: - - "ec2-10-100-0-1.compute-1.amazonaws.com" - peerCertSANs: - - "10.100.0.1" - # external: - # endpoints: - # - "10.100.0.1:2379" - # - "10.100.0.2:2379" - # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt" - # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt" - # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key" -networking: - serviceSubnet: "10.96.0.0/12" - podSubnet: "10.100.0.1/24" - dnsDomain: "cluster.local" -kubernetesVersion: "v1.12.0" -controlPlaneEndpoint: "10.100.0.1:6443" -apiServer: - extraArgs: - authorization-mode: "Node,RBAC" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File - certSANs: - - "10.100.1.1" - - "ec2-10-100-0-1.compute-1.amazonaws.com" - timeoutForControlPlane: 4m0s -controllerManager: - extraArgs: - "node-cidr-mask-size": "20" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File -scheduler: - extraArgs: - address: "10.100.0.1" - extraVolumes: - - name: "some-volume" - hostPath: "/etc/some-path" - mountPath: "/etc/some-pod-path" - readOnly: false - pathType: File -certificatesDir: "/etc/kubernetes/pki" -imageRepository: "k8s.gcr.io" -clusterName: "example-cluster" ---- -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -# kubelet specific options here ---- -apiVersion: kubeproxy.config.k8s.io/v1alpha1 -kind: KubeProxyConfiguration -# kube-proxy specific options here -``` - -## Kubeadm join configuration types - -When executing kubeadm join with the `--config` option, the JoinConfiguration type should be provided. - -```yaml -apiVersion: kubeadm.k8s.io/v1beta3 -kind: JoinConfiguration -... -``` - -The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join +between InitConfiguration and ClusterConfiguration is mandatory.

+
apiVersion: kubeadm.k8s.io/v1beta3
+kind: InitConfiguration
+bootstrapTokens:
+  ...
+nodeRegistration:
+  ...
+

The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init +are the configuration of the bootstrap token and all the setting which are specific to the node where +kubeadm is executed, including:

+
    +
  • +

    NodeRegistration, that holds fields that relate to registering the new node to the cluster; +use it to customize the node name, the CRI socket to use or any other settings that should apply to this +node only (e.g. the node ip).

    +
    • +
  • +

    LocalAPIEndpoint, that represents the endpoint of the instance of the API server to be deployed on this node; +use it e.g. to customize the API server advertise address.

    +
    • +
+
apiVersion: kubeadm.k8s.io/v1beta3
+kind: ClusterConfiguration
+networking:
+  ...
+etcd:
+  ...
+apiServer:
+  extraArgs:
+    ...
+  extraVolumes:
+    ...
+...
+

The ClusterConfiguration type should be used to configure cluster-wide settings, +including settings for:

+
    +
  • +

    networking that holds configuration for the networking topology of the cluster; use it e.g. to customize +Pod subnet or services subnet.

    +
    • +
  • +

    etcd: use it e.g. to customize the local etcd or to configure the API server +for using an external etcd cluster.

    +
    • +
  • +

    kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane +components by adding customized setting or overriding kubeadm default settings.

    +
    • +
+
apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+  ...
+

The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances +deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

+

See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or +https://pkg.go.dev/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration +for kube-proxy official documentation.

+
apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+  ...
+

The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances +deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

+

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or +https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration +for kubelet official documentation.

+

Here is a fully populated example of a single YAML file containing multiple +configuration types to be used during a kubeadm init run.

+
apiVersion: kubeadm.k8s.io/v1beta3
+kind: InitConfiguration
+bootstrapTokens:
+- token: "9a08jv.c0izixklcxtmnze7"
+  description: "kubeadm bootstrap token"
+  ttl: "24h"
+- token: "783bde.3f89s0fje9f38fhf"
+  description: "another bootstrap token"
+  usages:
+  - authentication
+  - signing
+  groups:
+  - system:bootstrappers:kubeadm:default-node-token
+nodeRegistration:
+  name: "ec2-10-100-0-1"
+  criSocket: "/var/run/dockershim.sock"
+  taints:
+  - key: "kubeadmNode"
+    value: "someValue"
+    effect: "NoSchedule"
+  kubeletExtraArgs:
+    v: 4
+  ignorePreflightErrors:
+    - IsPrivilegedUser
+  imagePullPolicy: "IfNotPresent"
+localAPIEndpoint:
+  advertiseAddress: "10.100.0.1"
+  bindPort: 6443
+certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
+skipPhases:
+  - addon/kube-proxy
+---
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: ClusterConfiguration
+etcd:
+  # one of local or external
+  local:
+    imageRepository: "k8s.gcr.io"
+    imageTag: "3.2.24"
+    dataDir: "/var/lib/etcd"
+    extraArgs:
+      listen-client-urls: "http://10.100.0.1:2379"
+    serverCertSANs:
+    -  "ec2-10-100-0-1.compute-1.amazonaws.com"
+    peerCertSANs:
+    - "10.100.0.1"
+  # external:
+    # endpoints:
+    # - "10.100.0.1:2379"
+    # - "10.100.0.2:2379"
+    # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
+    # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
+    # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
+networking:
+  serviceSubnet: "10.96.0.0/16"
+  podSubnet: "10.244.0.0/24"
+  dnsDomain: "cluster.local"
+kubernetesVersion: "v1.21.0"
+controlPlaneEndpoint: "10.100.0.1:6443"
+apiServer:
+  extraArgs:
+    authorization-mode: "Node,RBAC"
+  extraVolumes:
+  - name: "some-volume"
+    hostPath: "/etc/some-path"
+    mountPath: "/etc/some-pod-path"
+    readOnly: false
+    pathType: File
+  certSANs:
+  - "10.100.1.1"
+  - "ec2-10-100-0-1.compute-1.amazonaws.com"
+  timeoutForControlPlane: 4m0s
+controllerManager:
+  extraArgs:
+    "node-cidr-mask-size": "20"
+  extraVolumes:
+  - name: "some-volume"
+    hostPath: "/etc/some-path"
+    mountPath: "/etc/some-pod-path"
+    readOnly: false
+    pathType: File
+scheduler:
+  extraArgs:
+    address: "10.100.0.1"
+  extraVolumes:
+  - name: "some-volume"
+    hostPath: "/etc/some-path"
+    mountPath: "/etc/some-pod-path"
+    readOnly: false
+    pathType: File
+certificatesDir: "/etc/kubernetes/pki"
+imageRepository: "k8s.gcr.io"
+clusterName: "example-cluster"
+---
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+# kubelet specific options here
+---
+apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+# kube-proxy specific options here
+

Kubeadm join configuration types

+

When executing kubeadm join with the --config option, the JoinConfiguration type should be provided.

+
apiVersion: kubeadm.k8s.io/v1beta3
+kind: JoinConfiguration
+  ...
+

The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join are the discovery method used for accessing the cluster info and all the setting which are specific -to the node where kubeadm is executed, including: +to the node where kubeadm is executed, including:

+
    +
  • +

    nodeRegistration, that holds fields that relate to registering the new node to the cluster; +use it to customize the node name, the CRI socket to use or any other settings that should apply to this +node only (e.g. the node ip).

    +
    • +
  • +

    apiEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node.

    +
    • +
-- NodeRegistration, that holds fields that relate to registering the new node to the cluster; - use it to customize the node name, the CRI socket to use or any other settings that should apply to this - node only (e.g. the node ip). -- APIEndpoint, that represents the endpoint of the instance of the API server to be eventually - deployed on this node. ## Resource Types @@ -292,15 +264,13 @@ to the node where kubeadm is executed, including: - ## `ClusterConfiguration` {#kubeadm-k8s-io-v1beta3-ClusterConfiguration} +

ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster

-ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster - @@ -308,133 +278,120 @@ ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster - - +

etcd holds the configuration for etcd.

+ - - +

networking holds configuration for the networking topology of the cluster.

+ - - +

kubernetesVersion is the target version of the control plane.

+ - - +

controlPlaneEndpoint sets a stable IP address or DNS name for the control plane. +It can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port. +In case the controlPlaneEndpoint is not specified, the advertiseAddress + bindPort +are used; in case the controlPlaneEndpoint is specified but without a TCP port, +the bindPort is used. +Possible usages are:

+
    +
  • In a cluster with more than one control plane instances, this field should be +assigned the address of the external load balancer in front of the +control plane instances.
    • +
  • In environments with enforced node recycling, the controlPlaneEndpoint could +be used for assigning a stable DNS to the control plane.
    • +
+ - - +

apiServer contains extra settings for the API server.

+ - - +

controllerManager contains extra settings for the controller manager.

+ - - +

scheduler contains extra settings for the scheduler.

+ - - +

dns defines the options for the DNS add-on installed in the cluster.

+ - - +

certificatesDir specifies where to store or look for all required certificates.

+ - - +

imageRepository sets the container registry to pull images from. +If empty, k8s.gcr.io will be used by default. +In case of kubernetes version is a CI build (kubernetes version starts with ci/) +gcr.io/k8s-staging-ci-images will be used as a default for control plane components +and for kube-proxy, while k8s.gcr.io will be used for all the other images.

+ - - +

featureGates contains the feature gates enabled by the user.

+ - - +

The cluster name.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta3
kind
string		ClusterConfiguration
etcd
Etcd 		- `etcd` holds configuration for etcd.
networking
Networking 		- `networking` holds configuration for the networking topology of the cluster.
kubernetesVersion
string 		- `kubernetesVersion` is the target version of the control plane.
controlPlaneEndpoint
string 		- `controlPlaneEndpoint` sets a stable IP address or DNS name for the control plane; it -can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port. -In case the `controlPlaneEndpoint` is not specified, the `advertiseAddress` + `bindPort` -are used; in case the `controlPlaneEndpoint` is specified but without a TCP port, -the `bindPort` of the `localAPIEndpoint` is used. -Possible usages are: - -- In a cluster with more than one control plane instances, this field should be - assigned the address of the external load balancer in front of the - control plane instances. -- In environments with enforced node recycling, the ControlPlaneEndpoint - could be used for assigning a stable DNS to the control plane.
apiServer
APIServer 		- `apiServer` contains extra settings for the API server.
controllerManager
ControlPlaneComponent 		- `controllerManager` contains extra settings for the controller manager.
scheduler
ControlPlaneComponent 		- `scheduler` contains extra settings for the scheduler.
dns
DNS 		- `dns` defines the options for the DNS add-on.
certificatesDir
string 		- `certificatesDir` specifies where to store or look for all required certificates.
imageRepository
string 		- `imageRepository` sets the container registry to pull images from. -If empty, `k8s.gcr.io` will be used by default; in case of kubernetes version is -a CI build (kubernetes version starts with `ci/` or `ci-cross/`) -`gcr.io/k8s-staging-ci-images` will be used as a default for control plane -components and for kube-proxy, while `k8s.gcr.io` will be used for all the other images.
featureGates
map[string]bool 		- Feature gates enabled by the user.
clusterName
string 		- The cluster name.
- - ## `InitConfiguration` {#kubeadm-k8s-io-v1beta3-InitConfiguration} - - -InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime +

InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime information. +kubeadm init-only information. These fields are solely used the first time kubeadm init runs. +After that, the information in the fields IS NOT uploaded to the kubeadm-config ConfigMap +that is used by kubeadm upgrade for instance. These fields must be omitempty.

+ @@ -443,80 +400,71 @@ information. - - +

bootstrapTokens is respected at kubeadm init time and describes a set of Bootstrap Tokens to create. +This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature

+ - - +

nodeRegistration holds fields that relate to registering the new control-plane node +to the cluster.

+ - - +

localAPIEndpoint represents the endpoint of the API server instance that's deployed on this +control plane node. In HA setups, this differs from ClusterConfiguration.controlPlaneEndpoint +in the sense that controlPlaneEndpoint is the global endpoint for the cluster, which then +load-balances the requests to each individual API server. +This configuration object lets you customize what IP/DNS name and port the local API server +advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default +interface and use that, but in case that process fails you may set the desired value here.

+ - - +

certificateKey sets the key with which certificates and keys are encrypted prior to being +uploaded in a Secret in the cluster during the uploadcerts init phase.

+ - - +

skipPhases is a list of phases to skip during command execution. +The list of phases can be obtained with the kubeadm init --help command. +The flag "--skip-phases" takes precedence over this field.

+ - - +

patches contains options related to applying patches to components deployed by kubeadm during +kubeadm init.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta3
kind
string		InitConfiguration
bootstrapTokens
[]BootstrapToken 		- `bootstrapTokens` is respected at `kubeadm init` time and describes a set of Bootstrap Tokens to create. -This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature.
nodeRegistration
NodeRegistrationOptions 		- `nodeRegistration` holds fields that relate to registering the new control-plane node to the cluster
localAPIEndpoint
APIEndpoint 		- `localAPIEndpoint` represents the endpoint of the API server instance that's deployed on this control plane node -In HA setups, this differs from ClusterConfiguration.ControlPlaneEndpoint in the sense that ControlPlaneEndpoint -is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This -configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible -on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process -fails you may set the desired value here.
certificateKey
string 		- `certificateKey` sets the key with which certificates and keys are encrypted prior to being uploaded in -a Secret in the cluster during the "uploadcerts" init phase.
skipPhases
[]string 		- `skipPhases` is a list of phases to skip during command execution. -The list of phases can be obtained with the `kubeadm init --help` command. -The flag `--skip-phases` takes precedence over this field.
patches
Patches 		- `patches` contains options related to applying patches to components deployed by kubeadm during -"kubeadm init".
- - ## `JoinConfiguration` {#kubeadm-k8s-io-v1beta3-JoinConfiguration} +

JoinConfiguration contains elements describing a particular node.

-JoinConfiguration contains elements describing a particular node. - @@ -524,74 +472,63 @@ JoinConfiguration contains elements describing a particular node. - - +

nodeRegistration holds fields that relate to registering the new +control-plane node to the cluster.

+ - - +

caCertPath is the path to the SSL certificate authority used to secure +comunications between a node and the control-plane. +Defaults to "/etc/kubernetes/pki/ca.crt".

+ - - +

discovery specifies the options for the kubelet to use during the TLS +bootstrap process.

+ - - +

controlPlane defines the additional control plane instance to be deployed +on the joining node. If nil, no additional control plane instance will be deployed.

+ - - +

skipPhases is a list of phases to skip during command execution. +The list of phases can be obtained with the kubeadm join --help command. +The flag --skip-phases takes precedence over this field.

+ - - +

patches contains options related to applying patches to components deployed +by kubeadm during kubeadm join.

+ - -
FieldDescription
apiVersion
string		kubeadm.k8s.io/v1beta3
kind
string		JoinConfiguration
nodeRegistration
NodeRegistrationOptions 		- `nodeRegistration` holds fields that relate to registering the new control-plane -node to the cluster
caCertPath
string 		- `caCertPath` is the path to the SSL certificate authority used to -secure comunications between node and control-plane. -Defaults to "/etc/kubernetes/pki/ca.crt".
discovery [Required]
Discovery 		- `discovery` specifies the options for the kubelet to use during the TLS Bootstrap process.
controlPlane
JoinControlPlane 		- `controlPlane` defines the additional control plane instance to be deployed on the -joining node. If nil, no additional control plane instance will be deployed.
skipPhases
[]string 		- `skipPhases` is a list of phases to skip during command execution. -The list of phases can be obtained with the `kubeadm join --help` command. -The flag `--skip-phases` takes precedence over this field.
patches
Patches 		- `patches` contains options related to applying patches to components deployed by kubeadm during -`kubeadm join`.
- - ## `APIEndpoint` {#kubeadm-k8s-io-v1beta3-APIEndpoint} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) @@ -599,152 +536,131 @@ The flag `--skip-phases` takes precedence over this field. - [JoinControlPlane](#kubeadm-k8s-io-v1beta3-JoinControlPlane) -APIEndpoint struct contains elements of API server instance deployed on a node. +

APIEndpoint struct contains elements of API server instance deployed on a node.

+ - +

advertiseAddress sets the IP address for the API server to advertise.

+ - - +

bindPort sets the secure port for the API Server to bind to. +Defaults to 6443.

+ - -
FieldDescription
advertiseAddress
string 		- `advertiseAddress` sets the IP address for the API server to advertise.
bindPort
int32 		- `bindPort` sets the secure port for the API Server to bind to. Defaults to 6443.
- - ## `APIServer` {#kubeadm-k8s-io-v1beta3-APIServer} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta3-ClusterConfiguration) -APIServer holds settings necessary for API server deployments in the cluster +

APIServer holds settings necessary for API server deployments in the cluster

+ - + No description provided. - - +

certSANs sets extra Subject Alternative Names (SANs) for the API Server signing +certificate.

+ - - +

timeoutForControlPlane controls the timeout that we wait for API server to appear.

+ - -
FieldDescription
ControlPlaneComponent [Required]
ControlPlaneComponent 		(Members of ControlPlaneComponent are embedded into this type.) - No description provided. -
certSANs
[]string 		- `certSANs` sets extra Subject Alternative Names for the API Server signing cert.
timeoutForControlPlane
-meta/v1.Duration +meta/v1.Duration 		- `timeoutForControlPlane` controls the timeout that we use for API server to appear
- - ## `BootstrapTokenDiscovery` {#kubeadm-k8s-io-v1beta3-BootstrapTokenDiscovery} - - **Appears in:** - [Discovery](#kubeadm-k8s-io-v1beta3-Discovery) -BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery +

BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery

+ - +

token is a token used to validate cluster information fetched from the +control-plane.

+ - - +

apiServerEndpoint is an IP or domain name to the API server from which +information will be fetched.

+ - - +

caCertHashes specifies a set of public key pins to verify when token-based discovery +is used. The root CA found during discovery must match one of these values. +Specifying an empty set disables root CA pinning, which can be unsafe. +Each hash is specified as ":", where the only currently supported type is +"sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI) +object in DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.

+ - - +

unsafeSkipCAVerification allows token-based discovery without CA verification +via caCertHashes. This can weaken the security of kubeadm since other nodes can +impersonate the control-plane.

+ - -
FieldDescription
token [Required]
string 		- `token` is a token used to validate cluster information fetched from the control-plane.
apiServerEndpoint
string 		- `apiServerEndpoint` is an IP or domain name to the API server from which -information will be fetched.
caCertHashes
[]string 		- CACertHashes specifies a set of public key pins to verify when token-based -discovery is used. The root CA found during discovery must match one of these -values. Specifying an empty set disables root CA pinning, which can be unsafe. -Each hash is specified as `:`, where the only currently supported -type is "sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key -Info (SPKI) object in DER-encoded ASN.1. These hashes can be calculated using, -for example, OpenSSL.
unsafeSkipCAVerification
bool 		- `unsafeSkipCAVerification` allows token-based discovery without CA verification -via `caCertHashes`. This can weaken the security of kubeadm since other nodes -can impersonate the control-plane.
- - ## `ControlPlaneComponent` {#kubeadm-k8s-io-v1beta3-ControlPlaneComponent} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta3-ClusterConfiguration) @@ -752,341 +668,283 @@ can impersonate the control-plane. - [APIServer](#kubeadm-k8s-io-v1beta3-APIServer) -ControlPlaneComponent holds settings common to control plane component of the cluster +

ControlPlaneComponent holds settings common to control plane component of the cluster

+ - +

extraArgs is an extra set of flags to pass to the control plane component. +A key in this map is the flag name as it appears on the command line except +without leading dash(es).

+ - - +

extraVolumes is an extra set of host volumes, mounted to the control plane component.

+ - -
FieldDescription
extraArgs
map[string]string 		- `extraArgs` is an extra set of flags to pass to the control plane component. -A key in this map is the flag name as it appears on the -command line except without leading dash(es).
extraVolumes
[]HostPathMount 		- `extraVolumes` is an extra set of host volumes, mounted to the control plane component.
- - ## `DNS` {#kubeadm-k8s-io-v1beta3-DNS} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta3-ClusterConfiguration) -DNS defines the DNS addon that should be used in the cluster +

DNS defines the DNS addon that should be used in the cluster

+ - +

imageMeta allows to customize the image used for the DNS component.

+ - -
FieldDescription
ImageMeta [Required]
ImageMeta 		(Members of ImageMeta are embedded into this type.) - `imageMeta` allows to customize the image used for the DNS component.
- - - -## `DNSAddOnType` {#kubeadm-k8s-io-v1beta3-DNSAddOnType} - -(Alias of `string`) - - - -DNSAddOnType defines string identifying DNS add-on types - - - - ## `Discovery` {#kubeadm-k8s-io-v1beta3-Discovery} - - **Appears in:** - [JoinConfiguration](#kubeadm-k8s-io-v1beta3-JoinConfiguration) -Discovery specifies the options for the kubelet to use during the TLS Bootstrap process +

Discovery specifies the options for the kubelet to use during the TLS Bootstrap process.

+ - +

bootstrapToken is used to set the options for bootstrap token based discovery. +bootstrapToken and file are mutually exclusive.

+ - - +

file is used to specify a file or URL to a kubeconfig file from which to load +cluster information. +bootstrapToken and file are mutually exclusive.

+ - - +

tlsBootstrapToken is a token used for TLS bootstrapping. +If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, but +can be overridden. If file is set, this field must be set in case the KubeConfigFile +does not contain any other authentication information

+ - - +

timeout modifies the discovery timeout.

+ - -
FieldDescription
bootstrapToken
BootstrapTokenDiscovery 		- `bootstrapToken` is used to set the options for bootstrap token based discovery. -`bootstrapToken` and `file` are mutually exclusive.
file
FileDiscovery 		- `file` specifies a file or URL to a kubeconfig file from which to load cluster information. -`bootstrapToken` and `file` are mutually exclusive.
tlsBootstrapToken
string 		- `tlsBootstrapToken` is a token used for TLS bootstrapping. -If `bootstrapToken` is set, this field is defaulted to `bootstrapToken.token`, -but can be overridden. -If `file` is set, this field ∗∗must be set∗∗ in case the KubeConfigFile does -not contain any other authentication information
timeout
-meta/v1.Duration +meta/v1.Duration 		- `timeout` modifies the discovery timeout.
- - ## `Etcd` {#kubeadm-k8s-io-v1beta3-Etcd} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta3-ClusterConfiguration) -Etcd contains elements describing Etcd configuration. +

Etcd contains elements describing Etcd configuration.

+ - +

local provides configuration knobs for configuring the local etcd instance. +local and external are mutually exclusive.

+ - - +

external describes how to connect to an external etcd cluster. +local and external are mutually exclusive.

+ - -
FieldDescription
local
LocalEtcd 		- `local` provides configuration knobs for configuring the local etcd instance. -`local` and `external` are mutually exclusive.
external
ExternalEtcd 		- `external` describes how to connect to an external etcd cluster. -`local` and `external` are mutually exclusive.
- - ## `ExternalEtcd` {#kubeadm-k8s-io-v1beta3-ExternalEtcd} - - **Appears in:** - [Etcd](#kubeadm-k8s-io-v1beta3-Etcd) -ExternalEtcd describes an external etcd cluster. -Kubeadm has no knowledge of where certificate files live and they must be supplied. +

ExternalEtcd describes an external etcd cluster. +Kubeadm has no knowledge of where certificate files live and they must be supplied.

+ - +

endpoints contains the list of etcd members.

+ - - +

caFile is an SSL Certificate Authority (CA) file used to secure etcd communication. +Required if using a TLS connection.

+ - - +

certFile is an SSL certification file used to secure etcd communication. +Required if using a TLS connection.

+ - - +

keyFile is an SSL key file used to secure etcd communication. +Required if using a TLS connection.

+ - -
FieldDescription
endpoints [Required]
[]string 		- `endpoints` are endpoints of etcd members. This field is required.
caFile [Required]
string 		- `caFile` is an SSL Certificate Authority file used to secure etcd communication. -Required if using a TLS connection.
certFile [Required]
string 		- `certFile` is an SSL certification file used to secure etcd communication. -Required if using a TLS connection.
keyFile [Required]
string 		- `keyFile` is an SSL key file used to secure etcd communication. -Required if using a TLS connection.
- - ## `FileDiscovery` {#kubeadm-k8s-io-v1beta3-FileDiscovery} - - **Appears in:** - [Discovery](#kubeadm-k8s-io-v1beta3-Discovery) -FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information +

FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load +cluster information.

+ - +

kubeConfigPath is used to specify the actual file path or URL to the kubeconfig +file from which to load cluster information.

+ - -
FieldDescription
kubeConfigPath [Required]
string 		- `kubeConfigPath` specifies the actual file path or URL to the kubeconfig file -from which to load cluster information
- - ## `HostPathMount` {#kubeadm-k8s-io-v1beta3-HostPathMount} - - **Appears in:** - [ControlPlaneComponent](#kubeadm-k8s-io-v1beta3-ControlPlaneComponent) -HostPathMount contains elements describing volumes that are mounted from the host. +

HostPathMount contains elements describing volumes that are mounted from the host.

+ - +

name is the name of the volume inside the Pod template.

+ - - +

hostPath is the path in the host that will be mounted inside the Pod.

+ - - +

mountPath is the path inside the Pod where hostPath will be mounted.

+ - - +

readOnly controls write access to the volume.

+ - - +

pathType is the type of the hostPath.

+ - -
FieldDescription
name [Required]
string 		- `name` is the volume name inside the Pod template.
hostPath [Required]
string 		- `hostPath` is the path in the host that will be mounted inside the Pod.
mountPath [Required]
string 		- `mountPath` is the path inside the Pod where the `hostPath` volume is mounted.
readOnly
bool 		- `readOnly` controls write access to the volume.
pathType
-core/v1.HostPathType +core/v1.HostPathType 		- `pathType` is the type of the `hostPath` volume.
- - ## `ImageMeta` {#kubeadm-k8s-io-v1beta3-ImageMeta} - - **Appears in:** - [DNS](#kubeadm-k8s-io-v1beta3-DNS) @@ -1094,200 +952,174 @@ HostPathMount contains elements describing volumes that are mounted from the hos - [LocalEtcd](#kubeadm-k8s-io-v1beta3-LocalEtcd) -ImageMeta allows to customize the image used for components that are not -originated from the Kubernetes/Kubernetes release process +

ImageMeta allows to customize the image used for components that are not +originated from the Kubernetes/Kubernetes release process

+ - +

imageRepository sets the container registry to pull images from. +If not set, the imageRepository defined in ClusterConfiguration will be used instead.

+ - - +

imageTag allows to specify a tag for the image. +In case this value is set, kubeadm does not change automatically the version of +the above components during upgrades.

+ - -
FieldDescription
imageRepository
string 		- `imageRepository` sets the container registry to pull images from. -If not set, the ImageRepository defined in ClusterConfiguration will be used instead.
imageTag
string 		- `imageTag` allows to specify a tag for the image. -In case this value is set, kubeadm does not change automatically the -version of the above components during upgrades.
- - ## `JoinControlPlane` {#kubeadm-k8s-io-v1beta3-JoinControlPlane} - - **Appears in:** - [JoinConfiguration](#kubeadm-k8s-io-v1beta3-JoinConfiguration) -JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node. +

JoinControlPlane contains elements describing an additional control plane instance +to be deployed on the joining node.

+ - +

localAPIEndpoint represents the endpoint of the API server instance to be +deployed on this node.

+ - - +

certificateKey is the key that is used for decryption of certificates after +they are downloaded from the secret upon joining a new control plane node. +The corresponding encryption key is in the InitConfiguration.

+ - -
FieldDescription
localAPIEndpoint
APIEndpoint 		- `localAPIEndpoint` represents the endpoint of the API server instance to be deployed -on this node.
certificateKey
string 		- `certificateKey` is the key that is used for decryption of certificates after they -are downloaded from the secret upon joining a new control plane node. The -corresponding encryption key is in the InitConfiguration.
- - ## `LocalEtcd` {#kubeadm-k8s-io-v1beta3-LocalEtcd} - - **Appears in:** - [Etcd](#kubeadm-k8s-io-v1beta3-Etcd) -LocalEtcd describes that kubeadm should run an etcd cluster locally +

LocalEtcd describes that kubeadm should run an etcd cluster locally

+ - +

ImageMeta allows to customize the container used for etcd.

+ - - +

dataDir is the directory etcd will place its data. +Defaults to "/var/lib/etcd".

+ - - +

extraArgs are extra arguments provided to the etcd binary when run +inside a static Pod. A key in this map is the flag name as it appears on the +command line except without leading dash(es).

+ - - +

serverCertSANs sets extra Subject Alternative Names (SANs) for the etcd +server signing certificate.

+ - - +

peerCertSANs sets extra Subject Alternative Names (SANs) for the etcd peer +signing certificate.

+ - -
FieldDescription
ImageMeta [Required]
ImageMeta 		(Members of ImageMeta are embedded into this type.) - `ImageMeta` allows to customize the container used for etcd.
dataDir [Required]
string 		- `dataDir` is the directory etcd will place its data. -Defaults to "/var/lib/etcd".
extraArgs
map[string]string 		- `extraArgs` are extra arguments provided to the etcd binary -when run inside a static pod. -A key in this map is the flag name as it appears on the command line except -without leading dash(es).
serverCertSANs
[]string 		- `serverCertSANs` sets extra Subject Alternative Names for the etcd server signing cert.
peerCertSANs
[]string 		- `peerCertSANs` sets extra Subject Alternative Names for the etcd peer signing cert.
- - ## `Networking` {#kubeadm-k8s-io-v1beta3-Networking} - - **Appears in:** - [ClusterConfiguration](#kubeadm-k8s-io-v1beta3-ClusterConfiguration) -Networking contains elements describing cluster's networking configuration +

Networking contains elements describing cluster's networking configuration

+ - +

serviceSubnet is the subnet used by Kubernetes Services. Defaults to "10.96.0.0/12".

+ - - +

podSubnet is the subnet used by Pods.

+ - - +

dnsDomain is the DNS domain used by Kubernetes Services. Defaults to "cluster.local".

+ - -
FieldDescription
serviceSubnet
string 		- `serviceSubnet` is the subnet used by k8s services. Defaults to "10.96.0.0/12".
podSubnet
string 		- `podSubnet` is the subnet used by Pods.
dnsDomain
string 		- `dnsDomain` is the DNS domain used by k8s services. Defaults to "cluster.local".
- - ## `NodeRegistrationOptions` {#kubeadm-k8s-io-v1beta3-NodeRegistrationOptions} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) @@ -1295,91 +1127,83 @@ Networking contains elements describing cluster's networking configuration - [JoinConfiguration](#kubeadm-k8s-io-v1beta3-JoinConfiguration) -NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via "kubeadm init" or "kubeadm join" +

NodeRegistrationOptions holds fields that relate to registering a new control-plane or +node to the cluster, either via "kubeadm init" or "kubeadm join"

+ - +

name is the .metadata.name field of the Node API object that will be created in this +kubeadm init or kubeadm join operation. +This field is also used in the CommonName field of the kubelet's client certificate to +the API server. +Defaults to the hostname of the node if not provided.

+ - - +

criSocket is used to retrieve container runtime info. +This information will be annotated to the Node API object, for later re-use

+ - - +

taints specifies the taints the Node API object should be registered with. +If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted +with a control-plane taint for control-plane nodes. +If you don't want to taint your control-plane node, set this field to an empty list, +i.e. taints: [] in the YAML file. This field is solely used for Node registration.

+ - - +

kubeletExtraArgs passes through extra arguments to the kubelet. +The arguments here are passed to the kubelet command line via the environment file +kubeadm writes at runtime for the kubelet to source. +This overrides the generic base-level configuration in the 'kubelet-config-1.X' ConfigMap. +Flags have higher priority when parsing. These values are local and specific to the node +kubeadm is executing on. A key in this map is the flag name as it appears on the +command line except without leading dash(es).

+ - - +

ignorePreflightErrors provides a list of pre-flight errors to be ignored when +the current node is registered.

+ - - +

imagePullPolicy specifies the policy for image pulling during kubeadm "init" and +"join" operations. +The value of this field must be one of "Always", "IfNotPresent" or "Never". +If this field is unset kubeadm will default it to "IfNotPresent", or pull the required +images if not present on the host.

+ - -
FieldDescription
name
string 		- `name` is the `.metadata.name` field of the Node API object that will be created in this -`kubeadm init` or `kubeadm join` operation. -This field is also used in the `CommonName` field of the kubelet's client certificate to the -API server. Defaults to the hostname of the node if not provided.
criSocket
string 		- `criSocket` is used to retrieve container runtime info. This information will be -annotated to the Node API object, for later re-use.
taints [Required]
-[]core/v1.Taint +[]core/v1.Taint 		- `taints` specifies the taints the Node API object should be registered with. If -this field is unset, i.e. nil, in the `kubeadm init` process, it will be defaulted -to `['"node-role.kubernetes.io/master"=""']`. If you don't want to taint your -control-plane node, set this field to an empty list, i.e. `taints: []` in the YAML -file. This field is solely used for Node registration.
kubeletExtraArgs
map[string]string 		- `kubeletExtraArgs` passes through extra arguments to the kubelet. The arguments here -are passed to the kubelet command line via the environment file kubeadm writes at -runtime for the kubelet to source. This overrides the generic base-level -configuration in the "kubelet-config-1.X" ConfigMap. Flags have higher priority when -parsing. These values are local and specific to the node kubeadm is executing on. -A key in this map is the flag name as it appears on the command line except without -leading dash(es).
ignorePreflightErrors
[]string 		- `ignorePreflightErrors` provides a slice of pre-flight errors to be ignored when -the current node is registered.
imagePullPolicy
-core/v1.PullPolicy +core/v1.PullPolicy 		- `imagePullPolicy` specifies the policy for image pulling during `kubeadm init` and -`kubeadm join` operations. -The value of this field must be one of "Always", "IfNotPresent" or "Never". -If this field is unset kubeadm will default it to "IfNotPresent", or pull the required -images if not present on the host.
- - ## `Patches` {#kubeadm-k8s-io-v1beta3-Patches} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) @@ -1387,32 +1211,31 @@ images if not present on the host. - [JoinConfiguration](#kubeadm-k8s-io-v1beta3-JoinConfiguration) -Patches contains options related to applying patches to components deployed by kubeadm. +

Patches contains options related to applying patches to components deployed by kubeadm.

+ - +

directory is a path to a directory that contains files named +"target[suffix][+patchtype].extension". +For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of +"kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd". "patchtype" can +be one of "strategic" "merge" or "json" and they match the patch formats supported by +kubectl. +The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". +"suffix" is an optional string that can be used to determine which patches are applied +first alpha-numerically.

+ - -
FieldDescription
directory
string 		- `directory` is a path to a directory that contains files named -`target[suffix][+patchtype].extension`. -For example, `kube-apiserver0+merge.yaml` or just `etcd.json`. `target` can be one of -"kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd". `patchtype` can be one -of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. -The default `patchtype` is "strategic". `extension` must be either "json" or "yaml". -`suffix` is an optional string that can be used to determine which patches are applied -first alpha-numerically.
- @@ -1420,116 +1243,100 @@ first alpha-numerically. ## `BootstrapToken` {#BootstrapToken} - - **Appears in:** - [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) -BootstrapToken describes one bootstrap token, stored as a Secret in the cluster +

BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

+ - +

token is used for establishing bidirectional trust between nodes and control-planes. +Used for joining nodes in the cluster.

+ - - +

description sets a human-friendly message why this token exists and what it's used +for, so other administrators can know its purpose.

+ - - +

ttl defines the time to live for this token. Defaults to 24h. +expires and ttl are mutually exclusive.

+ - - +

expires specifies the timestamp when this token expires. Defaults to being set +dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

+ - - +

usages describes the ways in which this token can be used. Can by default be used +for establishing bidirectional trust, but that can be changed here.

+ - - +

groups specifies the extra groups that this token will authenticate as when/if +used for authentication

+ - -
FieldDescription
token [Required]
BootstrapTokenString 		- `token` is used for establishing bidirectional trust between nodes and control-planes. -Used for joining nodes in the cluster.
description
string 		- `description` sets a human-friendly message why this token exists and what it's used -for, so other administrators can know its purpose.
ttl
-meta/v1.Duration +meta/v1.Duration 		- `ttl` defines the time to live for this token. Defaults to `24h`. -`expires` and `ttl` are mutually exclusive.
expires
-meta/v1.Time +meta/v1.Time 		- `expires` specifies the timestamp when this token expires. Defaults to being set -dynamically at runtime based on the `ttl`. `expires` and `ttl` are mutually exclusive.
usages
[]string 		- `usages` describes the ways in which this token can be used. Can by default be used -for establishing bidirectional trust, but that can be changed here.
groups
[]string 		- `groups` specifies the extra groups that this token will authenticate as when/if -used for authentication
## `BootstrapTokenString` {#BootstrapTokenString} - - **Appears in:** - [BootstrapToken](#BootstrapToken) -BootstrapTokenString is a token of the format `abcdef.abcdef0123456789` that is used +

BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used for both validation of the practically of the API server from a joining node's point of view and as an authentication method for the node in the bootstrap phase of -"kubeadm join". This token is and should be short-lived. +"kubeadm join". This token is and should be short-lived.

+ - + No description provided. - - + No description provided. - -
FieldDescription
- [Required]
string 		- No description provided. -
- [Required]
string 		- No description provided. -
diff --git a/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md b/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md new file mode 100644 index 0000000000..2fe765dabe --- /dev/null +++ b/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md @@ -0,0 +1,249 @@ +--- +title: Kubelet Configuration (v1alpha1) +content_type: tool-reference +package: kubelet.config.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig) + + + +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + + + +

FormatOptions contains options for the different logging formats.

+ + + + + + + + + + + +
FieldDescription
json [Required]
+JSONOptions +		 +

[Experimental] JSON contains options for logging format "json".

+
+ +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

JSONOptions contains options for logging format "json".

+ + + + + + + + + + + + + + +
FieldDescription
splitStream [Required]
+bool +		 +

[Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

+
infoBufferSize [Required]
+k8s.io/apimachinery/pkg/api/resource.QuantityValue +		 +

[Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

+
+ +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + + + +

VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

+ + + + + + +## `CredentialProviderConfig` {#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig} + + + +

CredentialProviderConfig is the configuration containing information about +each exec credential provider. Kubelet reads this configuration from disk and enables +each provider as specified by the CredentialProvider type.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubelet.config.k8s.io/v1alpha1
kind
string		CredentialProviderConfig
providers [Required]
+[]CredentialProvider +		 +

providers is a list of credential provider plugins that will be enabled by the kubelet. +Multiple providers may match against a single image, in which case credentials +from all providers will be returned to the kubelet. If multiple providers are called +for a single image, the results are combined. If providers return overlapping +auth keys, the value from the provider earlier in this list is used.

+
+ +## `CredentialProvider` {#kubelet-config-k8s-io-v1alpha1-CredentialProvider} + + +**Appears in:** + +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig) + + +

CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only +invoked when an image being pulled matches the images handled by the plugin (see matchImages).

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

name is the required name of the credential provider. It must match the name of the +provider executable as seen by the kubelet. The executable must be in the kubelet's +bin directory (set by the --image-credential-provider-bin-dir flag).

+
matchImages [Required]
+[]string +		 +

matchImages is a required list of strings used to match against images in order to +determine if this provider should be invoked. If one of the strings matches the +requested image from the kubelet, the plugin will be invoked and given a chance +to provide credentials. Images are expected to contain the registry domain +and URL path.

+

Each entry in matchImages is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like *.k8s.io or k8s.*.io, and top-level-domains such as k8s.*. +Matching partial subdomains like app*.k8s.io is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

+

A match exists between an image and a matchImage when all of the below are true:

+
    +
  • Both contain the same number of domain parts and each part matches.
    • +
  • The URL path of an imageMatch must be a prefix of the target image URL path.
    • +
  • If the imageMatch contains a port, then the port must match in the image as well.
    • +
+

Example values of matchImages:

+
    +
  • 123456789.dkr.ecr.us-east-1.amazonaws.com
    • +
  • *.azurecr.io
    • +
  • gcr.io
    • +
  • *.*.registry.io
    • +
  • registry.io:8080/path
    • +
+
defaultCacheDuration [Required]
+meta/v1.Duration +		 +

defaultCacheDuration is the default duration the plugin will cache credentials in-memory +if a cache duration is not provided in the plugin response. This field is required.

+
apiVersion [Required]
+string +		 +

Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse +MUST use the same encoding version as the input. Current supported values are:

+
    +
  • credentialprovider.kubelet.k8s.io/v1alpha1
    • +
+
args
+[]string +		 +

Arguments to pass to the command when executing it.

+
env
+[]ExecEnvVar +		 +

Env defines additional environment variables to expose to the process. These +are unioned with the host's environment, as well as variables client-go uses +to pass argument to the plugin.

+
+ +## `ExecEnvVar` {#kubelet-config-k8s-io-v1alpha1-ExecEnvVar} + + +**Appears in:** + +- [CredentialProvider](#kubelet-config-k8s-io-v1alpha1-CredentialProvider) + + +

ExecEnvVar is used for setting environment variables when executing an exec-based +credential plugin.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 + No description provided.
value [Required]
+string +		 + No description provided.
+ diff --git a/content/en/docs/reference/config-api/kubelet-config.v1beta1.md b/content/en/docs/reference/config-api/kubelet-config.v1beta1.md index 261a6dd5f8..42755470cb 100644 --- a/content/en/docs/reference/config-api/kubelet-config.v1beta1.md +++ b/content/en/docs/reference/config-api/kubelet-config.v1beta1.md @@ -9,20 +9,50 @@ auto_generated: true ## Resource Types +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig) - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - [SerializedNodeConfigSource](#kubelet-config-k8s-io-v1beta1-SerializedNodeConfigSource) +## `CredentialProviderConfig` {#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig} + + + +

CredentialProviderConfig is the configuration containing information about +each exec credential provider. Kubelet reads this configuration from disk and enables +each provider as specified by the CredentialProvider type.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		kubelet.config.k8s.io/v1beta1
kind
string		CredentialProviderConfig
providers [Required]
+[]CredentialProvider +		 +

providers is a list of credential provider plugins that will be enabled by the kubelet. +Multiple providers may match against a single image, in which case credentials +from all providers will be returned to the kubelet. If multiple providers are called +for a single image, the results are combined. If providers return overlapping +auth keys, the value from the provider earlier in this list is used.

+
## `KubeletConfiguration` {#kubelet-config-k8s-io-v1beta1-KubeletConfiguration} +

KubeletConfiguration contains the configuration for the Kubelet

-KubeletConfiguration contains the configuration for the Kubelet - @@ -30,1398 +60,1037 @@ KubeletConfiguration contains the configuration for the Kubelet - - +Default: true

+ - - +Default: ""

+ - - +Default: "1m"

+ - - +Default: "20s"

+ - - +

httpCheckFrequency is the duration between checking http for new data. +Default: "20s"

+ - - +

staticPodURL is the URL for accessing static pods to run. +Default: ""

+ - - +

staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL. +Default: nil

+ - - +Default: "0.0.0.0"

+ - - +Default: 10250

+ - - +Default: 0 (disabled)

+ - - +Default: ""

+ - - +

tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile. +Default: ""

+ - - +Default: nil

+ - - +Default: ""

+ - - +Default: false

+ - - +Default: false

+ - - +anonymous: +enabled: false +webhook: +enabled: true +cacheTTL: "2m"

+ - - +mode: Webhook +webhook: +cacheAuthorizedTTL: "5m" +cacheUnauthorizedTTL: "30s"

+ - - +Default: 5

+ - - +Default: 10

+ - - +Default: 5

+ - - +when eventRecordQPS > 0. +Default: 10

+ - - +Default: true

+ - - +

enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true. +Default: false

+ - - +Default: 10248

+ - - +

healthzBindAddress is the IP address for the healthz server to serve on. +Default: "127.0.0.1"

+ - - +Default: -999

+ - - +Default: ""

+ - - +Default: nil

+ - - +Default: "4h"

+ - - +Default: "10s"

+ - - +Default: "5m"

+ - - +Default: 40

+ - - +Default: "2m"

+ - - +Default: 85

+ - - +Default: 80

+ - - +Default: "1m"

+ - - +

kubeletCgroups is the absolute name of cgroups to isolate the kubelet in +Default: ""

+ - - +Default: ""

+ - - +

cgroupRoot is the root cgroup to use for pods. This is handled by the +container runtime on a best effort basis.

+ - - +Default: true

+ - - +Default: "cgroupfs"

+ - - +Default: "None"

+ - - +Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled. +Default: nil

+ - - +Default: "10s"

+ - - +Default: "none"

+ - - +

topologyManagerPolicy is the name of the topology manager policy to use. +Valid values include:

+
    +
  • restricted: kubelet only allows pods with optimal NUMA node alignment for +requested resources;
    • +
  • best-effort: kubelet will favor pods with NUMA alignment of CPU and device +resources;
    • +
  • none: kubelet has no knowledge of NUMA alignment of a pod's CPU and device resources.
    • +
  • single-numa-node: kubelet only allows pods with a single NUMA alignment +of CPU and device resources.
    • +
+

Policies other than "none" require the TopologyManager feature gate to be enabled. +Default: "none"

+ - - +

topologyManagerScope represents the scope of topology hint generation +that topology manager requests and hint providers generate. Valid values include:

+
    +
  • container: topology policy is applied on a per-container basis.
    • +
  • pod: topology policy is applied on a per-pod basis.
    • +
+

"pod" scope requires the TopologyManager feature gate to be enabled. +Default: "container"

+ - - +Default: nil

+ - - +Default: "2m"

+ - - +Default: "promiscuous-bridge"

+ - - +Default: 110

+ - - +Default: ""

+ - - +

podPidsLimit is the maximum number of PIDs in any pod. +Default: -1

+ - - +If set to the empty string, will override the default and effectively disable DNS lookups. +Default: "/etc/resolv.conf"

+ - - +Default: false

+ - - +Default: true

+ - - +Default: "100ms"

+ - - +Default: 50

+ - - +Default: 1000000

+ - - +

contentType is contentType of requests sent to apiserver. +Default: "application/vnd.kubernetes.protobuf"

+ - - +

kubeAPIQPS is the QPS to use while talking with kubernetes apiserver. +Default: 5

+ - - +Default: 10

+ - - +Default: true

+ - - +memory.available: "100Mi" +nodefs.available: "10%" +nodefs.inodesFree: "5%" +imagefs.available: "15%"

+ - - +

evictionSoft is a map of signal names to quantities that defines soft eviction thresholds. +For example: {"memory.available": "300Mi"}. +Default: nil

+ - - +

evictionSoftGracePeriod is a map of signal names to quantities that defines grace +periods for each soft eviction signal. For example: {"memory.available": "30s"}. +Default: nil

+ - - +Default: "5m"

+ - - +Default: 0

+ - - +For example: {"imagefs.available": "2Gi"}. +Default: nil

+ - - +Default: 0

+ - - +Note: attaching/detaching CSI volumes is not supported by the kubelet, +so this option needs to be true for that use case. +Default: true

+ - - +Default: false

+ - - +Default: true

+ - - +Default: 14

+ - - +Default: 15

+ - - +"k8s.io/kubernetes/pkg/features/kube_features.go". +Default: nil

+ - - +

failSwapOn tells the Kubelet to fail to start if swap is enabled on the node. +Default: true

+ - - +

memorySwap configures swap memory available to container workloads.

+ - - +

containerLogMaxSize is a quantity defining the maximum size of the container log +file before it is rotated. For example: "5Mi" or "256Ki". +Default: "10Mi"

+ - - +Default: 5

+ - - +

configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret +managers are running. Valid values include:

+
    +
  • Get: kubelet fetches necessary objects directly from the API server;
    • +
  • Cache: kubelet uses TTL cache for object fetched from the API server;
    • +
  • Watch: kubelet uses watches to observe changes to objects that are in its interest.
    • +
+

Default: "Watch"

+ - - +Default: nil

+ - - +Default: nil

+ - - +

The reservedSystemCPUs option specifies the CPU list reserved for the host +level system threads and kubernetes related threads. This provide a "static" +CPU list rather than the "dynamic" list by systemReserved and kubeReserved. +This option does not support systemReservedCgroup or kubeReservedCgroup.

+ - - +Default: ""

+ - - +Default: ""

+ - - +Default: ""

+ - - +Default: ["pods"]

+ - - +

A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in *). +Unsafe sysctl groups are kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue.*, +and net.*. For example: "kernel.msg*,net.ipv4.route.min_pmtu" +Default: []

+ - - +Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"

+ - - +Default: ""

+ - - +Default: false

+ - - +Format: text

+ - - +

enableSystemLogHandler enables system logs via web interface host:port/logs/ +Default: true

+ - - +Default: "0s"

+ - - +Default: "0s"

+ + + + - - +Also, avoid specifying:

+
    +
  1. Duplicates, the same NUMA node, and memory type, but with a different value.
    2. +
  2. zero limits for any memory type.
    3. +
  3. NUMAs nodes IDs that do not exist under the machine.
    4. +
  4. memory types except for memory and hugepages-
    5. +
+

Default: nil

+ - - +

enableProfilingHandler enables profiling via web interface host:port/debug/pprof/ +Default: true

+ - - +

enableDebugFlagsHandler enables flags endpoint via web interface host:port/debug/flags/v +Default: true

+ - - +Default: false

+ - - +Default: 0.8

+ + + + + + + - -
FieldDescription
apiVersion
string		kubelet.config.k8s.io/v1beta1
kind
string		KubeletConfiguration
enableServer [Required]
bool 		- enableServer enables Kubelet's secured server. +

enableServer enables Kubelet's secured server. Note: Kubelet's insecure port is controlled by the readOnlyPort option. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: true
staticPodPath
string 		- staticPodPath is the path to the directory containing local (static) pods to +

staticPodPath is the path to the directory containing local (static) pods to run, or the path to a single static pod file. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -the set of static pods specified at the new path may be different than the -ones the Kubelet initially started with, and this may disrupt your node. -Default: ""
syncFrequency
-meta/v1.Duration +meta/v1.Duration 		- syncFrequency is the max period between synchronizing running +

syncFrequency is the max period between synchronizing running containers and config. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening this duration may have a negative performance impact, especially -as the number of Pods on the node increases. Alternatively, increasing this -duration will result in longer refresh times for ConfigMaps and Secrets. -Default: "1m"
fileCheckFrequency
-meta/v1.Duration +meta/v1.Duration 		- fileCheckFrequency is the duration between checking config files for +

fileCheckFrequency is the duration between checking config files for new data. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the duration will cause the Kubelet to reload local Static Pod -configurations more frequently, which may have a negative performance impact. -Default: "20s"
httpCheckFrequency
-meta/v1.Duration +meta/v1.Duration 		- httpCheckFrequency is the duration between checking http for new data. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the duration will cause the Kubelet to poll staticPodURL more -frequently, which may have a negative performance impact. -Default: "20s"
staticPodURL
string 		- staticPodURL is the URL for accessing static pods to run. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -the set of static pods specified at the new URL may be different than the -ones the Kubelet initially started with, and this may disrupt your node. -Default: ""
staticPodURLHeader
map[string][]string 		- staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt the ability to read the latest set of static pods from StaticPodURL. -Default: nil
address
string 		- address is the IP address for the Kubelet to serve on (set to 0.0.0.0 +

address is the IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: "0.0.0.0"
port
int32 		- port is the port for the Kubelet to serve on. +

port is the port for the Kubelet to serve on. The port number must be between 1 and 65535, inclusive. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: 10250
readOnlyPort
int32 		- readOnlyPort is the read-only port for the Kubelet to serve on with +

readOnlyPort is the read-only port for the Kubelet to serve on with no authentication/authorization. The port number must be between 1 and 65535, inclusive. Setting this field to 0 disables the read-only service. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: 0 (disabled)
tlsCertFile
string 		- tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert, +

tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If tlsCertFile and tlsPrivateKeyFile are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to the Kubelet's --cert-dir flag. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: ""
tlsPrivateKeyFile
string 		- tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: ""
tlsCipherSuites
[]string 		- tlsCipherSuites is the list of allowed cipher suites for the server. +

tlsCipherSuites is the list of allowed cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: nil
tlsMinVersion
string 		- tlsMinVersion is the minimum TLS version supported. +

tlsMinVersion is the minimum TLS version supported. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: ""
rotateCertificates
bool 		- rotateCertificates enables client certificate rotation. The Kubelet will request a +

rotateCertificates enables client certificate rotation. The Kubelet will request a new certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may disrupt the Kubelet's ability to authenticate with the API server -after the current certificate expires. -Default: false
serverTLSBootstrap
bool 		- serverTLSBootstrap enables server certificate bootstrap. Instead of self +

serverTLSBootstrap enables server certificate bootstrap. Instead of self signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. This requires an approver to approve the certificate signing requests (CSR). The RotateKubeletServerCertificate feature must be enabled when setting this field. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it will stop the renewal of Kubelet server certificates, which can -disrupt components that interact with the Kubelet server in the long term, -due to certificate expiration. -Default: false
authentication
KubeletAuthentication 		- authentication specifies how requests to the Kubelet's server are authenticated. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. +

authentication specifies how requests to the Kubelet's server are authenticated. Defaults: - anonymous: - enabled: false - webhook: - enabled: true - cacheTTL: "2m"
authorization
KubeletAuthorization 		- authorization specifies how requests to the Kubelet's server are authorized. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. +

authorization specifies how requests to the Kubelet's server are authorized. Defaults: - mode: Webhook - webhook: - cacheAuthorizedTTL: "5m" - cacheUnauthorizedTTL: "30s"
registryPullQPS
int32 		- registryPullQPS is the limit of registry pulls per second. +

registryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced -by image pulls. -Default: 5
registryBurst
int32 		- registryBurst is the maximum size of bursty pulls, temporarily allows +

registryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced -by image pulls. -Default: 10
eventRecordQPS
int32 		- eventRecordQPS is the maximum event creations per second. If 0, there +

eventRecordQPS is the maximum event creations per second. If 0, there is no limit enforced. The value cannot be a negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced by -event creations. -Default: 5
eventBurst
int32 		- eventBurst is the maximum size of a burst of event creations, temporarily +

eventBurst is the maximum size of a burst of event creations, temporarily allows event creations to burst to this number, while still not exceeding eventRecordQPS. This field canot be a negative number and it is only used -when eventRecordQPS > 0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced by -event creations. -Default: 10
enableDebuggingHandlers
bool 		- enableDebuggingHandlers enables server endpoints for log access +

enableDebuggingHandlers enables server endpoints for log access and local running of containers and commands, including the exec, attach, logs, and portforward features. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may disrupt components that interact with the Kubelet server. -Default: true
enableContentionProfiling
bool 		- enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -enabling it may carry a performance impact. -Default: false
healthzPort
int32 		- healthzPort is the port of the localhost healthz endpoint (set to 0 to disable). +

healthzPort is the port of the localhost healthz endpoint (set to 0 to disable). A valid number is between 1 and 65535. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that monitor Kubelet health. -Default: 10248
healthzBindAddress
string 		- healthzBindAddress is the IP address for the healthz server to serve on. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that monitor Kubelet health. -Default: "127.0.0.1"
oomScoreAdj
int32 		- oomScoreAdj is The oom-score-adj value for kubelet process. Values +

oomScoreAdj is The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the stability of nodes under memory pressure. -Default: -999
clusterDomain
string 		- clusterDomain is the DNS domain for this cluster. If set, kubelet will +

clusterDomain is the DNS domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains. -Dynamic Kubelet Config (deprecated): Dynamically updating this field is not recommended, -as it should be kept in sync with the rest of the cluster. -Default: ""
clusterDNS
[]string 		- clusterDNS is a list of IP addresses for the cluster DNS server. If set, +

clusterDNS is a list of IP addresses for the cluster DNS server. If set, kubelet will configure all containers to use this for DNS resolution instead of the host's DNS servers. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes will only take effect on Pods created after the update. Draining -the node is recommended before changing this field. -Default: nil
streamingConnectionIdleTimeout
-meta/v1.Duration +meta/v1.Duration 		- streamingConnectionIdleTimeout is the maximum time a streaming connection +

streamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact components that rely on infrequent updates over streaming -connections to the Kubelet server. -Default: "4h"
nodeStatusUpdateFrequency
-meta/v1.Duration +meta/v1.Duration 		- nodeStatusUpdateFrequency is the frequency that kubelet computes node +

nodeStatusUpdateFrequency is the frequency that kubelet computes node status. If node lease feature is not enabled, it is also the frequency that kubelet posts node status to master. Note: When node lease feature is not enabled, be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact node scalability, and also that the node controller's -nodeMonitorGracePeriod must be set to N∗NodeStatusUpdateFrequency, -where N is the number of retries before the node controller marks -the node unhealthy. -Default: "10s"
nodeStatusReportFrequency
-meta/v1.Duration +meta/v1.Duration 		- nodeStatusReportFrequency is the frequency that kubelet posts node +

nodeStatusReportFrequency is the frequency that kubelet posts node status to master if node status does not change. Kubelet will ignore this frequency and post node status immediately if any change is detected. It is only used when node lease feature is enabled. nodeStatusReportFrequency's default value is 5m. But if nodeStatusUpdateFrequency is set explicitly, nodeStatusReportFrequency's default value will be set to nodeStatusUpdateFrequency for backward compatibility. -Default: "5m"
nodeLeaseDurationSeconds
int32 		- nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease, -when the NodeLease feature is enabled. This feature provides an indicator of node -health by having the Kubelet create and periodically renew a lease, named after the node, -in the kube-node-lease namespace. If the lease expires, the node can be considered unhealthy. -The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal interval -may be set based on the lease duration. +

nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease. +NodeLease provides an indicator of node health by having the Kubelet create and +periodically renew a lease, named after the node, in the kube-node-lease namespace. +If the lease expires, the node can be considered unhealthy. +The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal +interval may be set based on the lease duration. The field value must be greater than 0. -Requires the NodeLease feature gate to be enabled. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -decreasing the duration may reduce tolerance for issues that temporarily prevent -the Kubelet from renewing the lease (e.g. a short-lived network issue). -Default: 40
imageMinimumGCAge
-meta/v1.Duration +meta/v1.Duration 		- imageMinimumGCAge is the minimum age for an unused image before it is +

imageMinimumGCAge is the minimum age for an unused image before it is garbage collected. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay garbage collection, and may change the image overhead -on the node. -Default: "2m"
imageGCHighThresholdPercent
int32 		- imageGCHighThresholdPercent is the percent of disk usage after which +

imageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run. The percent is calculated by dividing this field value by 100, so this field must be between 0 and 100, inclusive. When specified, the value must be greater than imageGCLowThresholdPercent. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay garbage collection, and may change the image overhead -on the node. -Default: 85
imageGCLowThresholdPercent
int32 		- imageGCLowThresholdPercent is the percent of disk usage before which +

imageGCLowThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated by dividing this field value by 100, so the field value must be between 0 and 100, inclusive. When specified, the value must be less than imageGCHighThresholdPercent. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay garbage collection, and may change the image overhead -on the node. -Default: 80
volumeStatsAggPeriod
-meta/v1.Duration +meta/v1.Duration 		- volumeStatsAggPeriod is the frequency for calculating and caching volume +

volumeStatsAggPeriod is the frequency for calculating and caching volume disk usage for all pods. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the period may carry a performance impact. -Default: "1m"
kubeletCgroups
string 		- kubeletCgroups is the absolute name of cgroups to isolate the kubelet in -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""
systemCgroups
string 		- systemCgroups is absolute name of cgroups in which to place +

systemCgroups is absolute name of cgroups in which to place all non-kernel processes that are not already in a container. Empty for no container. Rolling back the flag requires a reboot. The cgroupRoot must be specified if this field is not empty. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""
cgroupRoot
string 		- cgroupRoot is the root cgroup to use for pods. This is handled by the -container runtime on a best effort basis. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""
cgroupsPerQOS
bool 		- cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes +

cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes and all Burstable and BestEffort Pods are brought up under their specific top level QoS CGroup. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: true
cgroupDriver
string 		- cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs +

cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs or systemd). -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: "cgroupfs"
cpuManagerPolicy
string 		- cpuManagerPolicy is the name of the policy to use. +

cpuManagerPolicy is the name of the policy to use. Requires the CPUManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: "None"
cpuManagerPolicyOptions
map[string]string 		- cpuManagerPolicyOptions is a set of key=value which allows to set extra options +

cpuManagerPolicyOptions is a set of key=value which allows to set extra options to fine tune the behaviour of the cpu manager policies. -Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled. -Dynamic Kubelet Config (beta): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: nil
cpuManagerReconcilePeriod
-meta/v1.Duration +meta/v1.Duration 		- cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager. +

cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager. Requires the CPUManager feature gate to be enabled. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the period may carry a performance impact. -Default: "10s"
memoryManagerPolicy
string 		- memoryManagerPolicy is the name of the policy to use by memory manager. +

memoryManagerPolicy is the name of the policy to use by memory manager. Requires the MemoryManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: "none"
topologyManagerPolicy
string 		- topologyManagerPolicy is the name of the topology manager policy to use. -Valid values include: - -- `restricted`: kubelet only allows pods with optimal NUMA node alignment for - requested resources; -- `best-effort`: kubelet will favor pods with NUMA alignment of CPU and device - resources; -- `none`: kublet has no knowledge of NUMA alignment of a pod's CPU and device resources. -- `single-numa-node`: kubelet only allows pods with a single NUMA alignment - of CPU and device resources. - -Policies other than "none" require the TopologyManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: "none"
topologyManagerScope
string 		- topologyManagerScope represents the scope of topology hint generation -that topology manager requests and hint providers generate. Valid values include: - -- `container`: topology policy is applied on a per-container basis. -- `pod`: topology policy is applied on a per-pod basis. - -"pod" scope requires the TopologyManager feature gate to be enabled. -Default: "container"
qosReserved
map[string]string 		- qosReserved is a set of resource name to percentage pairs that specify +

qosReserved is a set of resource name to percentage pairs that specify the minimum percentage of a resource reserved for exclusive use by the guaranteed QoS tier. -Currently supported resources: "memory" +Currently supported resources: "memory" Requires the QOSReserved feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: nil
runtimeRequestTimeout
-meta/v1.Duration +meta/v1.Duration 		- runtimeRequestTimeout is the timeout for all runtime requests except long running +

runtimeRequestTimeout is the timeout for all runtime requests except long running requests - pull, logs, exec and attach. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. -Default: "2m"
hairpinMode
string 		- hairpinMode specifies how the Kubelet should configure the container +

hairpinMode specifies how the Kubelet should configure the container bridge for hairpin packets. Setting this flag allows endpoints in a Service to loadbalance back to -themselves if they should try to access their own Service. Values: - -- "promiscuous-bridge": make the container bridge promiscuous. -- "hairpin-veth": set the hairpin flag on container veth interfaces. -- "none": do nothing. - -Generally, one must set `--hairpin-mode=hairpin-veth to` achieve hairpin NAT, +themselves if they should try to access their own Service. Values:

+
    +
  • "promiscuous-bridge": make the container bridge promiscuous.
    • +
  • "hairpin-veth": set the hairpin flag on container veth interfaces.
    • +
  • "none": do nothing.
    • +
+

Generally, one must set --hairpin-mode=hairpin-veth to achieve hairpin NAT, because promiscuous-bridge assumes the existence of a container bridge named cbr0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may require a node reboot, depending on the network plugin. -Default: "promiscuous-bridge"
maxPods
int32 		- maxPods is the maximum number of Pods that can run on this Kubelet. +

maxPods is the maximum number of Pods that can run on this Kubelet. The value must be a non-negative integer. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes may cause Pods to fail admission on Kubelet restart, and may change -the value reported in Node.Status.Capacity[v1.ResourcePods], thus affecting -future scheduling decisions. Increasing this value may also decrease performance, -as more Pods can be packed into a single node. -Default: 110
podCIDR
string 		- podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode. +

podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the control plane. -Dynamic Kubelet Config (deprecated): This field should always be set to the empty default. -It should only set for standalone Kubelets, which cannot use Dynamic Kubelet Config. -Default: ""
podPidsLimit
int64 		- podPidsLimit is the maximum number of PIDs in any pod. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may prevent container processes from forking after the change. -Default: -1
resolvConf
string 		- resolvConf is the resolver configuration file used as the basis +

resolvConf is the resolver configuration file used as the basis for the container DNS resolution configuration. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes will only take effect on Pods created after the update. Draining -the node is recommended before changing this field. -Default: "/etc/resolv.conf"
runOnce
bool 		- runOnce causes the Kubelet to check the API server once for pods, +

runOnce causes the Kubelet to check the API server once for pods, run those in addition to the pods specified by static pod files, and exit. -Default: false
cpuCFSQuota
bool 		- cpuCFSQuota enables CPU CFS quota enforcement for containers that +

cpuCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may reduce node stability. -Default: true
cpuCFSQuotaPeriod
-meta/v1.Duration +meta/v1.Duration 		- cpuCFSQuotaPeriod is the CPU CFS quota period value, `cpu.cfs_period_us`. +

cpuCFSQuotaPeriod is the CPU CFS quota period value, cpu.cfs_period_us. The value must be between 1 us and 1 second, inclusive. Requires the CustomCPUCFSQuotaPeriod feature gate to be enabled. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -limits set for containers will result in different cpu.cfs_quota settings. This -will trigger container restarts on the node being reconfigured. -Default: "100ms"
nodeStatusMaxImages
int32 		- nodeStatusMaxImages caps the number of images reported in Node.status.images. +

nodeStatusMaxImages caps the number of images reported in Node.status.images. The value must be greater than -2. Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -different values can be reported on node status. -Default: 50
maxOpenFiles
int64 		- maxOpenFiles is Number of files that can be opened by Kubelet process. +

maxOpenFiles is Number of files that can be opened by Kubelet process. The value must be a non-negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability of the Kubelet to interact with the node's filesystem. -Default: 1000000
contentType
string 		- contentType is contentType of requests sent to apiserver. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability for the Kubelet to communicate with the API server. -If the Kubelet loses contact with the API server due to a change to this field, -the change cannot be reverted via dynamic Kubelet config. -Default: "application/vnd.kubernetes.protobuf"
kubeAPIQPS
int32 		- kubeAPIQPS is the QPS to use while talking with kubernetes apiserver. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic the Kubelet -sends to the API server. -Default: 5
kubeAPIBurst
int32 		- kubeAPIBurst is the burst to allow while talking with kubernetes API server. +

kubeAPIBurst is the burst to allow while talking with kubernetes API server. This field cannot be a negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic the Kubelet -sends to the API server. -Default: 10
serializeImagePulls
bool 		- serializeImagePulls when enabled, tells the Kubelet to pull images one -at a time. We recommend ∗not∗ changing the default value on nodes that -run docker daemon with version < 1.9 or an Aufs storage backend. +

serializeImagePulls when enabled, tells the Kubelet to pull images one +at a time. We recommend not changing the default value on nodes that +run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the performance of image pulls. -Default: true
evictionHard
map[string]string 		- evictionHard is a map of signal names to quantities that defines hard eviction -thresholds. For example: `{"memory.available": "300Mi"}`. +

evictionHard is a map of signal names to quantities that defines hard eviction +thresholds. For example: {"memory.available": "300Mi"}. To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions. Default: - memory.available: "100Mi" - nodefs.available: "10%" - nodefs.inodesFree: "5%" - imagefs.available: "15%"
evictionSoft
map[string]string 		- evictionSoft is a map of signal names to quantities that defines soft eviction thresholds. -For example: `{"memory.available": "300Mi"}`. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions, and may change the allocatable reported -by the node. -Default: nil
evictionSoftGracePeriod
map[string]string 		- evictionSoftGracePeriod is a map of signal names to quantities that defines grace -periods for each soft eviction signal. For example: `{"memory.available": "30s"}`. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions. -Default: nil
evictionPressureTransitionPeriod
-meta/v1.Duration +meta/v1.Duration 		- evictionPressureTransitionPeriod is the duration for which the kubelet has to wait +

evictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may decrease the stability of the node when the node is overcommitted. -Default: "5m"
evictionMaxPodGracePeriod
int32 		- evictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use +

evictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. This value effectively caps the Pod's terminationGracePeriodSeconds value during soft evictions. Note: Due to issue #64530, the behavior has a bug where this value currently just overrides the grace period during soft eviction, which can increase the grace period from what is set on the Pod. This bug will be fixed in a future release. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it decreases the amount of time Pods will have to gracefully clean -up before being killed during a soft eviction. -Default: 0
evictionMinimumReclaim
map[string]string 		- evictionMinimumReclaim is a map of signal names to quantities that defines minimum reclaims, +

evictionMinimumReclaim is a map of signal names to quantities that defines minimum reclaims, which describe the minimum amount of a given resource the kubelet will reclaim when performing a pod eviction while that resource is under pressure. -For example: `{"imagefs.available": "2Gi"}`. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may change how well eviction can manage resource pressure. -Default: nil
podsPerCore
int32 		- podsPerCore is the maximum number of pods per core. Cannot exceed maxPods. +

podsPerCore is the maximum number of pods per core. Cannot exceed maxPods. The value must be a non-negative integer. If 0, there is no limit on the number of Pods. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes may cause Pods to fail admission on Kubelet restart, and may change -the value reported in `Node.status.capacity.pods`, thus affecting -future scheduling decisions. Increasing this value may also decrease performance, -as more Pods can be packed into a single node. -Default: 0
enableControllerAttachDetach
bool 		- enableControllerAttachDetach enables the Attach/Detach controller to +

enableControllerAttachDetach enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changing which component is responsible for volume management on a live node -may result in volumes refusing to detach if the node is not drained prior to -the update, and if Pods are scheduled to the node before the -volumes.kubernetes.io/controller-managed-attach-detach annotation is updated by the -Kubelet. In general, it is safest to leave this value set the same as local config. -Default: true
protectKernelDefaults
bool 		- protectKernelDefaults, if true, causes the Kubelet to error if kernel +

protectKernelDefaults, if true, causes the Kubelet to error if kernel flags are not as it expects. Otherwise the Kubelet will attempt to modify kernel flags to match its expectation. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -enabling it may cause the Kubelet to crash-loop if the Kernel is not configured as -Kubelet expects. -Default: false
makeIPTablesUtilChains
bool 		- makeIPTablesUtilChains, if true, causes the Kubelet ensures a set of iptables rules +

makeIPTablesUtilChains, if true, causes the Kubelet ensures a set of iptables rules are present on host. These rules will serve as utility rules for various components, e.g. kube-proxy. The rules will be created based on iptablesMasqueradeBit and iptablesDropBit. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it will prevent the Kubelet from healing locally misconfigured iptables rules. -Default: true
iptablesMasqueradeBit
int32 		- iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT. +

iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT. Values must be within the range [0, 31]. Must be different from other mark bits. Warning: Please match the value of the corresponding parameter in kube-proxy. TODO: clean up IPTablesMasqueradeBit in kube-proxy. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it needs to be coordinated with other components, like kube-proxy, and the update -will only be effective if MakeIPTablesUtilChains is enabled. -Default: 14
iptablesDropBit
int32 		- iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets. +

iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets. Values must be within the range [0, 31]. Must be different from other mark bits. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it needs to be coordinated with other components, like kube-proxy, and the update -will only be effective if MakeIPTablesUtilChains is enabled. -Default: 15
featureGates
map[string]bool 		- featureGates is a map of feature names to bools that enable or disable experimental +

featureGates is a map of feature names to bools that enable or disable experimental features. This field modifies piecemeal the built-in default values from -"k8s.io/kubernetes/pkg/features/kube_features.go". -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider the -documentation for the features you are enabling or disabling. While we -encourage feature developers to make it possible to dynamically enable -and disable features, some changes may require node reboots, and some -features may require careful coordination to retroactively disable. -Default: nil
failSwapOn
bool 		- failSwapOn tells the Kubelet to fail to start if swap is enabled on the node. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -setting it to true will cause the Kubelet to crash-loop if swap is enabled. -Default: true
memorySwap
MemorySwapConfiguration 		- memorySwap configures swap memory available to container workloads.
containerLogMaxSize
string 		- containerLogMaxSize is a quantity defining the maximum size of the container log -file before it is rotated. For example: "5Mi" or "256Ki". -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger log rotation. -Default: "10Mi"
containerLogMaxFiles
int32 		- containerLogMaxFiles specifies the maximum number of container log files that can +

containerLogMaxFiles specifies the maximum number of container log files that can be present for a container. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may cause log files to be deleted. -Default: 5
configMapAndSecretChangeDetectionStrategy
ResourceChangeDetectionStrategy 		- configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret -managers are running. Valid values include: - -- `Get`: kubelet fetches necessary objects directly from the API server; -- `Cache`: kubelet uses TTL cache for object fetched from the API server; -- `Watch`: kubelet uses watches to observe changes to objects that are in its interest. - -Default: "Watch"
systemReserved
map[string]string 		- systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) +

systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may not be possible to increase the reserved resources, because this -requires resizing cgroups. Always look for a NodeAllocatableEnforced event -after updating this field to ensure that the update was successful. -Default: nil
kubeReserved
map[string]string 		- kubeReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs +

kubeReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local storage for root file system are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more details. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may not be possible to increase the reserved resources, because this -requires resizing cgroups. Always look for a NodeAllocatableEnforced event -after updating this field to ensure that the update was successful. -Default: nil
reservedSystemCPUs [Required]
string 		- The reservedSystemCPUs option specifies the CPU list reserved for the host -level system threads and kubernetes related threads. This provide a "static" -CPU list rather than the "dynamic" list by systemReserved and kubeReserved. -This option does not support systemReservedCgroup or kubeReservedCgroup.
showHiddenMetricsForVersion
string 		- showHiddenMetricsForVersion is the previous version for which you want to show +

showHiddenMetricsForVersion is the previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. -The format is `.`, e.g.: `1.16`. +The format is <major>.<minor>, e.g.: 1.16. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that. -Default: ""
systemReservedCgroup
string 		- systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used -to enforce `systemReserved` compute resource reservation for OS system daemons. -Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) +

systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used +to enforce systemReserved compute resource reservation for OS system daemons. +Refer to Node Allocatable doc for more information. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""
kubeReservedCgroup
string 		- kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used -to enforce `KubeReserved` compute resource reservation for Kubernetes node system daemons. -Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) +

kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used +to enforce KubeReserved compute resource reservation for Kubernetes node system daemons. +Refer to Node Allocatable doc for more information. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""
enforceNodeAllocatable
[]string 		- This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform. -This flag accepts a list of options. Acceptable options are `none`, `pods`, -`system-reserved` and `kube-reserved`. -If `none` is specified, no other options may be specified. -When `system-reserved` is in the list, systemReservedCgroup must be specified. -When `kube-reserved` is in the list, kubeReservedCgroup must be specified. -This field is supported only when `cgroupsPerQOS` is set to true. -Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) +

This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform. +This flag accepts a list of options. Acceptable options are none, pods, +system-reserved and kube-reserved. +If none is specified, no other options may be specified. +When system-reserved is in the list, systemReservedCgroup must be specified. +When kube-reserved is in the list, kubeReservedCgroup must be specified. +This field is supported only when cgroupsPerQOS is set to true. +Refer to Node Allocatable for more information. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -removing enforcements may reduce the stability of the node. Alternatively, adding -enforcements may reduce the stability of components which were using more than -the reserved amount of resources; for example, enforcing kube-reserved may cause -Kubelets to OOM if it uses more than the reserved resources, and enforcing system-reserved -may cause system daemons to OOM if they use more than the reserved resources. -Default: ["pods"]
allowedUnsafeSysctls
[]string 		- A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in `∗`). -Unsafe sysctl groups are `kernel.shm∗`, `kernel.msg∗`, `kernel.sem`, `fs.mqueue.∗`, -and `net.∗`. For example: "`kernel.msg∗,net.ipv4.route.min_pmtu`" -Default: []
volumePluginDir
string 		- volumePluginDir is the full path of the directory in which to search +

volumePluginDir is the full path of the directory in which to search for additional third party volume plugins. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that changing -the volumePluginDir may disrupt workloads relying on third party volume plugins. -Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
providerID
string 		- providerID, if set, sets the unique ID of the instance that an external +

providerID, if set, sets the unique ID of the instance that an external provider (i.e. cloudprovider) can use to identify a specific node. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability of the Kubelet to interact with cloud providers. -Default: ""
kernelMemcgNotification
bool 		- kernelMemcgNotification, if set, instructs the the kubelet to integrate with the +

kernelMemcgNotification, if set, instructs the the kubelet to integrate with the kernel memcg notification for determining if memory eviction thresholds are exceeded rather than polling. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the way Kubelet interacts with the kernel. -Default: false
logging [Required]
LoggingConfiguration 		- logging specifies the options of logging. -Refer to [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) +

logging specifies the options of logging. +Refer to Logs Options for more information. Default: - Format: text
enableSystemLogHandler
bool 		- enableSystemLogHandler enables system logs via web interface host:port/logs/ -Default: true
shutdownGracePeriod
-meta/v1.Duration +meta/v1.Duration 		- shutdownGracePeriod specifies the total duration that the node should delay the +

shutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown. -Default: "0s"
shutdownGracePeriodCriticalPods
-meta/v1.Duration +meta/v1.Duration 		- shutdownGracePeriodCriticalPods specifies the duration used to terminate critical +

shutdownGracePeriodCriticalPods specifies the duration used to terminate critical pods during a node shutdown. This should be less than shutdownGracePeriod. For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s, during a node shutdown the first 20 seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods. -Default: "0s"

shutdownGracePeriodByPodPriority
+[]ShutdownGracePeriodByPodPriority +		 +

shutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based +on their associated priority class value. +When a shutdown request is received, the Kubelet will initiate shutdown on all pods +running on the node with a grace period that depends on the priority of the pod, +and then wait for all pods to exit. +Each entry in the array represents the graceful shutdown time a pod with a priority +class value that lies in the range of that value and the next higher entry in the +list when the node is shutting down. +For example, to allow critical pods 10s to shutdown, priority>=10000 pods 20s to +shutdown, and all remaining pods 30s to shutdown.

+

shutdownGracePeriodByPodPriority:

+
    +
  • priority: 2000000000 +shutdownGracePeriodSeconds: 10
    • +
  • priority: 10000 +shutdownGracePeriodSeconds: 20
    • +
  • priority: 0 +shutdownGracePeriodSeconds: 30
    • +
+

The time the Kubelet will wait before exiting will at most be the maximum of all +shutdownGracePeriodSeconds for each priority class range represented on the node. +When all pods have exited or reached their grace periods, the Kubelet will release +the shutdown inhibit lock. +Requires the GracefulNodeShutdown feature gate to be enabled. +This configuration must be empty if either ShutdownGracePeriod or ShutdownGracePeriodCriticalPods is set. +Default: nil

+
reservedMemory
[]MemoryReservation 		- reservedMemory specifies a comma-separated list of memory reservations for NUMA nodes. +

reservedMemory specifies a comma-separated list of memory reservations for NUMA nodes. The parameter makes sense only in the context of the memory manager feature. The memory manager will not allocate reserved memory for container workloads. For example, if you have a NUMA0 with 10Gi of memory and the reservedMemory was @@ -1430,75 +1099,85 @@ only 9Gi is available for allocation. You can specify a different amount of NUMA node and memory types. You can omit this parameter at all, but you should be aware that the amount of reserved memory from all NUMA nodes should be equal to the amount of memory specified -by the [node allocatable](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). +by the node allocatable. If at least one node allocatable parameter has a non-zero value, you will need to specify at least one NUMA node. -Also, avoid specifying: - -1. Duplicates, the same NUMA node, and memory type, but with a different value. -2. zero limits for any memory type. -3. NUMAs nodes IDs that do not exist under the machine. -4. memory types except for memory and hugepages- - -Default: nil
enableProfilingHandler
bool 		- enableProfilingHandler enables profiling via web interface host:port/debug/pprof/ -Default: true
enableDebugFlagsHandler
bool 		- enableDebugFlagsHandler enables flags endpoint via web interface host:port/debug/flags/v -Default: true
seccompDefault
bool 		- SeccompDefault enables the use of `RuntimeDefault` as the default seccomp profile for all workloads. +

SeccompDefault enables the use of RuntimeDefault as the default seccomp profile for all workloads. This requires the corresponding SeccompDefault feature gate to be enabled as well. -Default: false
memoryThrottlingFactor
float64 		- MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory +

MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory when setting the cgroupv2 memory.high value to enforce MemoryQoS. Decreasing this factor will set lower high limit for container cgroups and put heavier reclaim pressure while increasing will put less reclaim pressure. See http://kep.k8s.io/2570 for more details. -Default: 0.8

registerWithTaints
+[]core/v1.Taint +		 +

registerWithTaints are an array of taints to add to a node object when +the kubelet registers itself. This only takes effect when registerNode +is true and upon the initial registration of the node. +Default: nil

+
registerNode
+bool +		 +

registerNode enables automatic registration with the apiserver. +Default: true

+
- - ## `SerializedNodeConfigSource` {#kubelet-config-k8s-io-v1beta1-SerializedNodeConfigSource} - - -SerializedNodeConfigSource allows us to serialize v1.NodeConfigSource. +

SerializedNodeConfigSource allows us to serialize v1.NodeConfigSource. This type is used internally by the Kubelet for tracking checkpointed dynamic configs. -It exists in the kubeletconfig API group because it is classified as a versioned input to the Kubelet. +It exists in the kubeletconfig API group because it is classified as a versioned input to the Kubelet.

+ @@ -1507,163 +1186,246 @@ It exists in the kubeletconfig API group because it is classified as a versioned - - +

source is the source that we are serializing.

+ - -
FieldDescription
apiVersion
string		kubelet.config.k8s.io/v1beta1
kind
string		SerializedNodeConfigSource
source
-core/v1.NodeConfigSource +core/v1.NodeConfigSource 		- source is the source that we are serializing.
+ +## `CredentialProvider` {#kubelet-config-k8s-io-v1beta1-CredentialProvider} +**Appears in:** -## `HairpinMode` {#kubelet-config-k8s-io-v1beta1-HairpinMode} +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig) + + +

CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only +invoked when an image being pulled matches the images handled by the plugin (see matchImages).

+ + + + + -(Alias of `string`) - - - -HairpinMode denotes how the kubelet should configure networking to handle -hairpin packets. - + + + + + + + + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 +

name is the required name of the credential provider. It must match the name of the +provider executable as seen by the kubelet. The executable must be in the kubelet's +bin directory (set by the --image-credential-provider-bin-dir flag).

+
matchImages [Required]
+[]string +		 +

matchImages is a required list of strings used to match against images in order to +determine if this provider should be invoked. If one of the strings matches the +requested image from the kubelet, the plugin will be invoked and given a chance +to provide credentials. Images are expected to contain the registry domain +and URL path.

+

Each entry in matchImages is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

+

A match exists between an image and a matchImage when all of the below are true:

+
    +
  • Both contain the same number of domain parts and each part matches.
    • +
  • The URL path of an imageMatch must be a prefix of the target image URL path.
    • +
  • If the imageMatch contains a port, then the port must match in the image as well.
    • +
+

Example values of matchImages:

+
    +
  • 123456789.dkr.ecr.us-east-1.amazonaws.com
    • +
  • *.azurecr.io
    • +
  • gcr.io
    • +
  • ..registry.io
    • +
  • registry.io:8080/path
    • +
+
defaultCacheDuration [Required]
+meta/v1.Duration +		 +

defaultCacheDuration is the default duration the plugin will cache credentials in-memory +if a cache duration is not provided in the plugin response. This field is required.

+
apiVersion [Required]
+string +		 +

Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse +MUST use the same encoding version as the input. Current supported values are:

+
    +
  • credentialprovider.kubelet.k8s.io/v1beta1
    • +
+
args
+[]string +		 +

Arguments to pass to the command when executing it.

+
env
+[]ExecEnvVar +		 +

Env defines additional environment variables to expose to the process. These +are unioned with the host's environment, as well as variables client-go uses +to pass argument to the plugin.

+
+## `ExecEnvVar` {#kubelet-config-k8s-io-v1beta1-ExecEnvVar} +**Appears in:** + +- [CredentialProvider](#kubelet-config-k8s-io-v1beta1-CredentialProvider) + + +

ExecEnvVar is used for setting environment variables when executing an exec-based +credential plugin.

+ + + + + + + + + + + + + + +
FieldDescription
name [Required]
+string +		 + No description provided.
value [Required]
+string +		 + No description provided.
## `KubeletAnonymousAuthentication` {#kubelet-config-k8s-io-v1beta1-KubeletAnonymousAuthentication} - - **Appears in:** - [KubeletAuthentication](#kubelet-config-k8s-io-v1beta1-KubeletAuthentication) - - +Anonymous requests have a username of system:anonymous, and a group name of +system:unauthenticated.

+ - -
FieldDescription
enabled
bool 		- enabled allows anonymous requests to the kubelet server. +

enabled allows anonymous requests to the kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. -Anonymous requests have a username of `system:anonymous`, and a group name of -`system:unauthenticated`.
- - ## `KubeletAuthentication` {#kubelet-config-k8s-io-v1beta1-KubeletAuthentication} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - +

x509 contains settings related to x509 client certificate authentication.

+ - - +

webhook contains settings related to webhook bearer token authentication.

+ - - +

anonymous contains settings related to anonymous authentication.

+ - -
FieldDescription
x509
KubeletX509Authentication 		- x509 contains settings related to x509 client certificate authentication.
webhook
KubeletWebhookAuthentication 		- webhook contains settings related to webhook bearer token authentication.
anonymous
KubeletAnonymousAuthentication 		- anonymous contains settings related to anonymous authentication.
- - ## `KubeletAuthorization` {#kubelet-config-k8s-io-v1beta1-KubeletAuthorization} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - +

mode is the authorization mode to apply to requests to the kubelet server. +Valid values are AlwaysAllow and Webhook. +Webhook mode uses the SubjectAccessReview API to determine authorization.

+ - - +

webhook contains settings related to Webhook authorization.

+ - -
FieldDescription
mode
KubeletAuthorizationMode 		- mode is the authorization mode to apply to requests to the kubelet server. -Valid values are `AlwaysAllow` and `Webhook`. -Webhook mode uses the SubjectAccessReview API to determine authorization.
webhook
KubeletWebhookAuthorization 		- webhook contains settings related to Webhook authorization.
- - ## `KubeletAuthorizationMode` {#kubelet-config-k8s-io-v1beta1-KubeletAuthorizationMode} (Alias of `string`) - **Appears in:** - [KubeletAuthorization](#kubelet-config-k8s-io-v1beta1-KubeletAuthorization) @@ -1672,253 +1434,343 @@ Webhook mode uses the SubjectAccessReview API to determine authorization. - - - ## `KubeletWebhookAuthentication` {#kubelet-config-k8s-io-v1beta1-KubeletWebhookAuthentication} - - **Appears in:** - [KubeletAuthentication](#kubelet-config-k8s-io-v1beta1-KubeletAuthentication) - - +

enabled allows bearer token authentication backed by the +tokenreviews.authentication.k8s.io API.

+ - - +

cacheTTL enables caching of authentication results

+ - -
FieldDescription
enabled
bool 		- enabled allows bearer token authentication backed by the -tokenreviews.authentication.k8s.io API.
cacheTTL
-meta/v1.Duration +meta/v1.Duration 		- cacheTTL enables caching of authentication results
- - ## `KubeletWebhookAuthorization` {#kubelet-config-k8s-io-v1beta1-KubeletWebhookAuthorization} - - **Appears in:** - [KubeletAuthorization](#kubelet-config-k8s-io-v1beta1-KubeletAuthorization) - - +

cacheAuthorizedTTL is the duration to cache 'authorized' responses from the +webhook authorizer.

+ - - +

cacheUnauthorizedTTL is the duration to cache 'unauthorized' responses from +the webhook authorizer.

+ - -
FieldDescription
cacheAuthorizedTTL
-meta/v1.Duration +meta/v1.Duration 		- cacheAuthorizedTTL is the duration to cache 'authorized' responses from the -webhook authorizer.
cacheUnauthorizedTTL
-meta/v1.Duration +meta/v1.Duration 		- cacheUnauthorizedTTL is the duration to cache 'unauthorized' responses from -the webhook authorizer.
- - ## `KubeletX509Authentication` {#kubelet-config-k8s-io-v1beta1-KubeletX509Authentication} - - **Appears in:** - [KubeletAuthentication](#kubelet-config-k8s-io-v1beta1-KubeletAuthentication) - - +and groups corresponding to the Organization in the client certificate.

+ - -
FieldDescription
clientCAFile
string 		- clientCAFile is the path to a PEM-encoded certificate bundle. If set, any request +

clientCAFile is the path to a PEM-encoded certificate bundle. If set, any request presenting a client certificate signed by one of the authorities in the bundle is authenticated with a username corresponding to the CommonName, -and groups corresponding to the Organization in the client certificate.
- - ## `MemoryReservation` {#kubelet-config-k8s-io-v1beta1-MemoryReservation} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) -MemoryReservation specifies the memory reservation of different types for each NUMA node +

MemoryReservation specifies the memory reservation of different types for each NUMA node

+ - + No description provided. - - + No description provided. - -
FieldDescription
numaNode [Required]
int32 		- No description provided. -
limits [Required]
-core/v1.ResourceList +core/v1.ResourceList 		- No description provided. -
- - ## `MemorySwapConfiguration` {#kubelet-config-k8s-io-v1beta1-MemorySwapConfiguration} - - **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - +

swapBehavior configures swap memory available to container workloads. May be one of +"", "LimitedSwap": workload combined memory and swap usage cannot exceed pod memory limit +"UnlimitedSwap": workloads can use unlimited swap, up to the allocatable limit.

+ - -
FieldDescription
swapBehavior
string 		- swapBehavior configures swap memory available to container workloads. May be one of -"", "LimitedSwap": workload combined memory and swap usage cannot exceed pod memory limit -"UnlimitedSwap": workloads can use unlimited swap, up to the allocatable limit.
- - ## `ResourceChangeDetectionStrategy` {#kubelet-config-k8s-io-v1beta1-ResourceChangeDetectionStrategy} (Alias of `string`) +**Appears in:** + +- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) + + +

ResourceChangeDetectionStrategy denotes a mode in which internal +managers (secret, configmap) are discovering object changes.

+ + + + +## `ShutdownGracePeriodByPodPriority` {#kubelet-config-k8s-io-v1beta1-ShutdownGracePeriodByPodPriority} + **Appears in:** - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) -ResourceChangeDetectionStrategy denotes a mode in which internal -managers (secret, configmap) are discovering object changes. +

ShutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based on their associated priority class value

- - - - - -## `LoggingConfiguration` {#LoggingConfiguration} - - - - -**Appears in:** - -- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - -LoggingConfiguration contains logging options -Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information. - + + + + + + + + +
FieldDescription
priority [Required]
+int32 +		 +

priority is the priority value associated with the shutdown grace period

+
shutdownGracePeriodSeconds [Required]
+int64 +		 +

shutdownGracePeriodSeconds is the shutdown grace period in seconds

+
+ + + +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

FormatOptions contains options for the different logging formats.

+ + + + + + + + + + + +
FieldDescription
json [Required]
+JSONOptions +		 +

[Experimental] JSON contains options for logging format "json".

+
+ +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

JSONOptions contains options for logging format "json".

+ + + + + + + + + + + + + + +
FieldDescription
splitStream [Required]
+bool +		 +

[Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

+
infoBufferSize [Required]
+k8s.io/apimachinery/pkg/api/resource.QuantityValue +		 +

[Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

+
+ +## `LoggingConfiguration` {#LoggingConfiguration} + + +**Appears in:** + +- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) + + +

LoggingConfiguration contains logging options +Refer Logs Options for more information.

+ + + + + + +

Format Flag specifies the structure of log messages. +default value of format is text

+ - - - +

Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

+ + + + + + + + + + - -
FieldDescription
format [Required]
string 		- Format Flag specifies the structure of log messages. -default value of format is `text`
sanitization [Required]
-bool +
flushFrequency [Required]
+time.Duration 		- [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)
verbosity [Required]
+uint32 +		 +

Verbosity is the threshold that determines which log messages are +logged. Default is zero which logs only the most important +messages. Higher values enable additional messages. Error messages +are always logged.

+
vmodule [Required]
+VModuleConfiguration +		 +

VModule overrides the verbosity threshold for individual files. +Only supported for "text" log format.

+
options [Required]
+FormatOptions +		 +

[Experimental] Options holds additional parameters that are specific +to the different logging formats. Only the options for the selected +format get used, but all of them get validated.

+
+ +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

+ + + diff --git a/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md new file mode 100644 index 0000000000..029d5ac62c --- /dev/null +++ b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md @@ -0,0 +1,169 @@ +--- +title: Kubelet CredentialProvider (v1alpha1) +content_type: tool-reference +package: credentialprovider.kubelet.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [CredentialProviderRequest](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderRequest) +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse) + + + +## `CredentialProviderRequest` {#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderRequest} + + + +

CredentialProviderRequest includes the image that the kubelet requires authentication for. +Kubelet will pass this request object to the plugin via stdin. In general, plugins should +prefer responding with the same apiVersion they were sent.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		credentialprovider.kubelet.k8s.io/v1alpha1
kind
string		CredentialProviderRequest
image [Required]
+string +		 +

image is the container image that is being pulled as part of the +credential provider plugin request. Plugins may optionally parse the image +to extract any information required to fetch credentials.

+
+ +## `CredentialProviderResponse` {#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse} + + + +

CredentialProviderResponse holds credentials that the kubelet should use for the specified +image provided in the original request. Kubelet will read the response from the plugin via stdout. +This response should be set to the same apiVersion as CredentialProviderRequest.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		credentialprovider.kubelet.k8s.io/v1alpha1
kind
string		CredentialProviderResponse
cacheKeyType [Required]
+PluginCacheKeyType +		 +

cacheKeyType indiciates the type of caching key to use based on the image provided +in the request. There are three valid values for the cache key type: Image, Registry, and +Global. If an invalid value is specified, the response will NOT be used by the kubelet.

+
cacheDuration
+meta/v1.Duration +		 +

cacheDuration indicates the duration the provided credentials should be cached for. +The kubelet will use this field to set the in-memory cache duration for credentials +in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in +CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

+
auth
+map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1alpha1.AuthConfig +		 +

auth is a map containing authentication information passed into the kubelet. +Each key is a match image string (more on this below). The corresponding authConfig value +should be valid for all images that match against this key. A plugin should set +this field to null if no valid credentials can be returned for the requested image.

+

Each key in the map is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

+

The kubelet will match images against the key when all of the below are true:

+
    +
  • Both contain the same number of domain parts and each part matches.
    • +
  • The URL path of an imageMatch must be a prefix of the target image URL path.
    • +
  • If the imageMatch contains a port, then the port must match in the image as well.
    • +
+

When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

+
    +
  • longer keys come before shorter keys with the same prefix
    • +
  • non-wildcard keys come before wildcard keys with the same prefix.
    • +
+

For any given match, the kubelet will attempt an image pull with the provided credentials, +stopping after the first successfully authenticated pull.

+

Example keys:

+
    +
  • 123456789.dkr.ecr.us-east-1.amazonaws.com
    • +
  • *.azurecr.io
    • +
  • gcr.io
    • +
  • ..registry.io
    • +
  • registry.io:8080/path
    • +
+
+ +## `AuthConfig` {#credentialprovider-kubelet-k8s-io-v1alpha1-AuthConfig} + + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse) + + +

AuthConfig contains authentication information for a container registry. +Only username/password based authentication is supported today, but more authentication +mechanisms may be added in the future.

+ + + + + + + + + + + + + + +
FieldDescription
username [Required]
+string +		 +

username is the username used for authenticating to the container registry +An empty username is valid.

+
password [Required]
+string +		 +

password is the password used for authenticating to the container registry +An empty password is valid.

+
+ +## `PluginCacheKeyType` {#credentialprovider-kubelet-k8s-io-v1alpha1-PluginCacheKeyType} + +(Alias of `string`) + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse) + + + + + diff --git a/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md new file mode 100644 index 0000000000..a849970d4b --- /dev/null +++ b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md @@ -0,0 +1,169 @@ +--- +title: Kubelet CredentialProvider (v1beta1) +content_type: tool-reference +package: credentialprovider.kubelet.k8s.io/v1beta1 +auto_generated: true +--- + + +## Resource Types + + +- [CredentialProviderRequest](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderRequest) +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + + +## `CredentialProviderRequest` {#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderRequest} + + + +

CredentialProviderRequest includes the image that the kubelet requires authentication for. +Kubelet will pass this request object to the plugin via stdin. In general, plugins should +prefer responding with the same apiVersion they were sent.

+ + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		credentialprovider.kubelet.k8s.io/v1beta1
kind
string		CredentialProviderRequest
image [Required]
+string +		 +

image is the container image that is being pulled as part of the +credential provider plugin request. Plugins may optionally parse the image +to extract any information required to fetch credentials.

+
+ +## `CredentialProviderResponse` {#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse} + + + +

CredentialProviderResponse holds credentials that the kubelet should use for the specified +image provided in the original request. Kubelet will read the response from the plugin via stdout. +This response should be set to the same apiVersion as CredentialProviderRequest.

+ + + + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		credentialprovider.kubelet.k8s.io/v1beta1
kind
string		CredentialProviderResponse
cacheKeyType [Required]
+PluginCacheKeyType +		 +

cacheKeyType indiciates the type of caching key to use based on the image provided +in the request. There are three valid values for the cache key type: Image, Registry, and +Global. If an invalid value is specified, the response will NOT be used by the kubelet.

+
cacheDuration
+meta/v1.Duration +		 +

cacheDuration indicates the duration the provided credentials should be cached for. +The kubelet will use this field to set the in-memory cache duration for credentials +in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in +CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

+
auth
+map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1beta1.AuthConfig +		 +

auth is a map containing authentication information passed into the kubelet. +Each key is a match image string (more on this below). The corresponding authConfig value +should be valid for all images that match against this key. A plugin should set +this field to null if no valid credentials can be returned for the requested image.

+

Each key in the map is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

+

The kubelet will match images against the key when all of the below are true:

+
    +
  • Both contain the same number of domain parts and each part matches.
    • +
  • The URL path of an imageMatch must be a prefix of the target image URL path.
    • +
  • If the imageMatch contains a port, then the port must match in the image as well.
    • +
+

When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

+
    +
  • longer keys come before shorter keys with the same prefix
    • +
  • non-wildcard keys come before wildcard keys with the same prefix.
    • +
+

For any given match, the kubelet will attempt an image pull with the provided credentials, +stopping after the first successfully authenticated pull.

+

Example keys:

+
    +
  • 123456789.dkr.ecr.us-east-1.amazonaws.com
    • +
  • *.azurecr.io
    • +
  • gcr.io
    • +
  • ..registry.io
    • +
  • registry.io:8080/path
    • +
+
+ +## `AuthConfig` {#credentialprovider-kubelet-k8s-io-v1beta1-AuthConfig} + + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + +

AuthConfig contains authentication information for a container registry. +Only username/password based authentication is supported today, but more authentication +mechanisms may be added in the future.

+ + + + + + + + + + + + + + +
FieldDescription
username [Required]
+string +		 +

username is the username used for authenticating to the container registry +An empty username is valid.

+
password [Required]
+string +		 +

password is the password used for authenticating to the container registry +An empty password is valid.

+
+ +## `PluginCacheKeyType` {#credentialprovider-kubelet-k8s-io-v1beta1-PluginCacheKeyType} + +(Alias of `string`) + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + + + + diff --git a/content/en/docs/reference/glossary/api-eviction.md b/content/en/docs/reference/glossary/api-eviction.md index 69fc9d9b0c..e6db562461 100644 --- a/content/en/docs/reference/glossary/api-eviction.md +++ b/content/en/docs/reference/glossary/api-eviction.md @@ -19,4 +19,9 @@ You can request eviction either by directly calling the Eviction API using a client of the kube-apiserver, like the `kubectl drain` command. When an `Eviction` object is created, the API server terminates the Pod. -API-initiated eviction is not the same as [node-pressure eviction](/docs/concepts/scheduling-eviction/eviction/#kubelet-eviction). +API-initiated evictions respect your configured [`PodDisruptionBudgets`](/docs/tasks/run-application/configure-pdb/) +and [`terminationGracePeriodSeconds`](/docs/concepts/workloads/pods/pod-lifecycle#pod-termination). + +API-initiated eviction is not the same as [node-pressure eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/). + +* See [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/) for more information. diff --git a/content/en/docs/reference/glossary/cadvisor.md b/content/en/docs/reference/glossary/cadvisor.md new file mode 100644 index 0000000000..a85e572467 --- /dev/null +++ b/content/en/docs/reference/glossary/cadvisor.md @@ -0,0 +1,16 @@ +--- +title: cAdvisor +id: cadvisor +date: 2021-12-09 +full_link: https://github.com/google/cadvisor/ +short_description: > + Tool that provides understanding of the resource usage and performance characteristics for containers +aka: +tags: +- tool +--- +cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running {{< glossary_tooltip text="containers" term_id="container" >}}. + + + +It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide. diff --git a/content/en/docs/reference/glossary/container-runtime-interface.md b/content/en/docs/reference/glossary/container-runtime-interface.md new file mode 100644 index 0000000000..28a67cbbb4 --- /dev/null +++ b/content/en/docs/reference/glossary/container-runtime-interface.md @@ -0,0 +1,22 @@ +--- +title: Container Runtime Interface +id: container-runtime-interface +date: 2021-11-24 +full_link: /docs/concepts/architecture/cri +short_description: > + The main protocol for the communication between the kubelet and Container Runtime. + +aka: +tags: + - cri +--- + +The main protocol for the communication between the kubelet and Container Runtime. + + + +The Kubernetes Container Runtime Interface (CRI) defines the main +[gRPC](https://grpc.io) protocol for the communication between the +[cluster components](/docs/concepts/overview/components/#node-components) +{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} and +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}. diff --git a/content/en/docs/reference/glossary/container-runtime.md b/content/en/docs/reference/glossary/container-runtime.md index 89bd4ae7ad..1d2aae66c6 100644 --- a/content/en/docs/reference/glossary/container-runtime.md +++ b/content/en/docs/reference/glossary/container-runtime.md @@ -15,7 +15,7 @@ tags: -Kubernetes supports several container runtimes: {{< glossary_tooltip term_id="docker">}}, +Kubernetes supports container runtimes such as {{< glossary_tooltip term_id="containerd" >}}, {{< glossary_tooltip term_id="cri-o" >}}, -and any implementation of the [Kubernetes CRI (Container Runtime +and any other implementation of the [Kubernetes CRI (Container Runtime Interface)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md). diff --git a/content/en/docs/reference/glossary/dockershim.md b/content/en/docs/reference/glossary/dockershim.md new file mode 100644 index 0000000000..d726679064 --- /dev/null +++ b/content/en/docs/reference/glossary/dockershim.md @@ -0,0 +1,18 @@ +--- +title: Dockershim +id: dockershim +date: 2022-04-15 +full_link: /dockershim +short_description: > + A component of Kubernetes v1.23 and earlier, which allows Kubernetes system components to communicate with Docker Engine. + +aka: +tags: +- fundamental +--- +The dockershim is a component of Kubernetes version 1.23 and earlier. It allows the kubelet +to communicate with {{< glossary_tooltip text="Docker Engine" term_id="docker" >}}. + + + +Starting with version 1.24, dockershim has been removed from Kubernetes. For more information, see [Dockershim FAQ](/dockershim). diff --git a/content/en/docs/reference/glossary/downward-api.md b/content/en/docs/reference/glossary/downward-api.md new file mode 100644 index 0000000000..a3d9a46336 --- /dev/null +++ b/content/en/docs/reference/glossary/downward-api.md @@ -0,0 +1,28 @@ +--- +title: Downward API +id: downward-api +date: 2022-03-21 +short_description: > + A mechanism to expose Pod and container field values to code running in a container. +aka: +full_link: /docs/concepts/workloads/pods/downward-api/ +tags: +- architecture +--- +Kubernetes' mechanism to expose Pod and container field values to code running in a container. + +It is sometimes useful for a container to have information about itself, without +needing to make changes to the container code that directly couple it to Kubernetes. + +The Kubernetes downward API allows containers to consume information about themselves +or their context in a Kubernetes cluster. Applications in containers can have +access to that information, without the application needing to act as a client of +the Kubernetes API. + +There are two ways to expose Pod and container fields to a running container: + +- using [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) +- using [a `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) + +Together, these two ways of exposing Pod and container fields are called the _downward API_. + diff --git a/content/en/docs/reference/glossary/event.md b/content/en/docs/reference/glossary/event.md new file mode 100644 index 0000000000..23c72302a6 --- /dev/null +++ b/content/en/docs/reference/glossary/event.md @@ -0,0 +1,25 @@ +--- +title: Event +id: event +date: 2022-01-16 +full_link: /docs/reference/kubernetes-api/cluster-resources/event-v1/ +short_description: > + A report of an event somewhere in the cluster. It generally denotes some state change in the system. +aka: +tags: +- core-object +- fundamental +--- +Each Event is a report of an event somewhere in the {{< glossary_tooltip text="cluster" term_id="cluster" >}}. +It generally denotes some state change in the system. + + +Events have a limited retention time and triggers and messages may evolve with time. +Event consumers should not rely on the timing of an event with a given reason reflecting a consistent underlying trigger, +or the continued existence of events with that reason. + + +Events should be treated as informative, best-effort, supplemental data. + +In Kubernetes, [auditing](/docs/tasks/debug/debug-cluster/audit/) generates a different kind of +Event record (API group `audit.k8s.io`). diff --git a/content/en/docs/reference/glossary/extensions.md b/content/en/docs/reference/glossary/extensions.md index 1515bffcfd..c994b601a1 100644 --- a/content/en/docs/reference/glossary/extensions.md +++ b/content/en/docs/reference/glossary/extensions.md @@ -2,9 +2,10 @@ title: Extensions id: Extensions date: 2019-02-01 -full_link: /docs/concepts/extend-kubernetes/extend-cluster/#extensions +full_link: /docs/concepts/extend-kubernetes/#extensions short_description: > - Extensions are software components that extend and deeply integrate with Kubernetes to support new types of hardware. + Extensions are software components that extend and deeply integrate with Kubernetes to support + new types of hardware. aka: tags: @@ -15,4 +16,6 @@ tags: -Most cluster administrators will use a hosted or distribution instance of Kubernetes. As a result, most Kubernetes users will need to install [extensions](/docs/concepts/extend-kubernetes/extend-cluster/#extensions) and fewer will need to author new ones. +Many cluster administrators use a hosted or distribution instance of Kubernetes. These clusters +come with extensions pre-installed. As a result, most Kubernetes users will not need to install +[extensions](/docs/concepts/extend-kubernetes/) and even fewer users will need to author new ones. diff --git a/content/en/docs/reference/glossary/finalizer.md b/content/en/docs/reference/glossary/finalizer.md index c44386fbf3..e01f66d7f2 100644 --- a/content/en/docs/reference/glossary/finalizer.md +++ b/content/en/docs/reference/glossary/finalizer.md @@ -19,12 +19,12 @@ to clean up resources the deleted object owned. When you tell Kubernetes to delete an object that has finalizers specified for -it, the Kubernetes API marks the object for deletion, putting it into a -read-only state. The target object remains in a terminating state while the +it, the Kubernetes API marks the object for deletion by populating `.metadata.deletionTimestamp`, +and returns a `202` status code (HTTP "Accepted"). The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. After these actions are complete, the controller removes the relevant finalizers from the target object. When the `metadata.finalizers` field is empty, -Kubernetes considers the deletion complete. +Kubernetes considers the deletion complete and deletes the object. You can use finalizers to control {{}} of resources. For example, you can define a finalizer to clean up related resources or diff --git a/content/en/docs/reference/glossary/flexvolume.md b/content/en/docs/reference/glossary/flexvolume.md index 91478fd1f0..1f54e4dc68 100644 --- a/content/en/docs/reference/glossary/flexvolume.md +++ b/content/en/docs/reference/glossary/flexvolume.md @@ -4,14 +4,14 @@ id: flexvolume date: 2018-06-25 full_link: /docs/concepts/storage/volumes/#flexvolume short_description: > - FlexVolume is an interface for creating out-of-tree volume plugins. The {{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} is a newer interface which addresses several problems with FlexVolumes. + FlexVolume is a deprecated interface for creating out-of-tree volume plugins. The {{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} is a newer interface that addresses several problems with FlexVolume. aka: tags: - storage --- - FlexVolume is an interface for creating out-of-tree volume plugins. The {{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} is a newer interface which addresses several problems with FlexVolumes. + FlexVolume is a deprecated interface for creating out-of-tree volume plugins. The {{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}} is a newer interface that addresses several problems with FlexVolume. diff --git a/content/en/docs/reference/glossary/garbage-collection.md b/content/en/docs/reference/glossary/garbage-collection.md index ec2fe19af7..1d4b9b5785 100644 --- a/content/en/docs/reference/glossary/garbage-collection.md +++ b/content/en/docs/reference/glossary/garbage-collection.md @@ -2,7 +2,7 @@ title: Garbage Collection id: garbage-collection date: 2021-07-07 -full_link: /docs/concepts/workloads/controllers/garbage-collection/ +full_link: /docs/concepts/architecture/garbage-collection/ short_description: > A collective term for the various mechanisms Kubernetes uses to clean up cluster resources. @@ -12,13 +12,16 @@ tags: - fundamental - operation --- - Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up - cluster resources. + +Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up +cluster resources. -Kubernetes uses garbage collection to clean up resources like [unused containers and images](/docs/concepts/workloads/controllers/garbage-collection/#containers-images), +Kubernetes uses garbage collection to clean up resources like +[unused containers and images](/docs/concepts/architecture/garbage-collection/#containers-images), [failed Pods](/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection), [objects owned by the targeted resource](/docs/concepts/overview/working-with-objects/owners-dependents/), [completed Jobs](/docs/concepts/workloads/controllers/ttlafterfinished/), and resources -that have expired or failed. \ No newline at end of file +that have expired or failed. + diff --git a/content/en/docs/reference/glossary/kubectl.md b/content/en/docs/reference/glossary/kubectl.md index 665fffcf98..61f93b9cf6 100644 --- a/content/en/docs/reference/glossary/kubectl.md +++ b/content/en/docs/reference/glossary/kubectl.md @@ -4,16 +4,19 @@ id: kubectl date: 2018-04-12 full_link: /docs/user-guide/kubectl-overview/ short_description: > - A command line tool for communicating with a Kubernetes API server. + A command line tool for communicating with a Kubernetes cluster. -aka: +aka: +- kubectl tags: - tool - fundamental --- - A command line tool for communicating with a {{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}} server. +Command line tool for communicating with a Kubernetes cluster's +{{< glossary_tooltip text="control plane" term_id="control-plane" >}}, +using the Kubernetes API. -You can use kubectl to create, inspect, update, and delete Kubernetes objects. +You can use `kubectl` to create, inspect, update, and delete Kubernetes objects. diff --git a/content/en/docs/reference/glossary/managed-service.md b/content/en/docs/reference/glossary/managed-service.md index 186588252c..4752867bb7 100644 --- a/content/en/docs/reference/glossary/managed-service.md +++ b/content/en/docs/reference/glossary/managed-service.md @@ -16,7 +16,4 @@ tags: Some examples of Managed Services are AWS EC2, Azure SQL Database, and GCP Pub/Sub, but they can be any software offering that can be used by an application. -[Service Catalog](/docs/concepts/extend-kubernetes/service-catalog/) provides a way to -list, provision, and bind with Managed Services offered by -{{< glossary_tooltip text="Service Brokers" term_id="service-broker" >}}. diff --git a/content/en/docs/reference/glossary/namespace.md b/content/en/docs/reference/glossary/namespace.md index a477150a9d..02381c4ee6 100644 --- a/content/en/docs/reference/glossary/namespace.md +++ b/content/en/docs/reference/glossary/namespace.md @@ -4,15 +4,15 @@ id: namespace date: 2018-04-12 full_link: /docs/concepts/overview/working-with-objects/namespaces short_description: > - An abstraction used by Kubernetes to support multiple virtual clusters on the same physical cluster. + An abstraction used by Kubernetes to support isolation of groups of resources within a single cluster. aka: tags: - fundamental --- - An abstraction used by Kubernetes to support multiple virtual clusters on the same physical {{< glossary_tooltip text="cluster" term_id="cluster" >}}. + An abstraction used by Kubernetes to support isolation of groups of resources within a single {{< glossary_tooltip text="cluster" term_id="cluster" >}}. -Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a namespace, but not across namespaces. +Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects _(e.g. Deployments, Services, etc)_ and not for cluster-wide objects _(e.g. StorageClass, Nodes, PersistentVolumes, etc)_. diff --git a/content/en/docs/reference/glossary/pod-disruption-budget.md b/content/en/docs/reference/glossary/pod-disruption-budget.md index 1665224174..7ee55d45a4 100644 --- a/content/en/docs/reference/glossary/pod-disruption-budget.md +++ b/content/en/docs/reference/glossary/pod-disruption-budget.md @@ -15,4 +15,12 @@ tags: - operation --- - A [Pod Disruption Budget](/docs/concepts/workloads/pods/disruptions/) allows an application owner to create an object for a replicated application, that ensures a certain number or percentage of Pods with an assigned label will not be voluntarily evicted at any point in time. PDBs cannot prevent an involuntary disruption, but will count against the budget. + A [Pod Disruption Budget](/docs/concepts/workloads/pods/disruptions/) allows an + application owner to create an object for a replicated application, that ensures + a certain number or percentage of Pods with an assigned label will not be voluntarily + evicted at any point in time. + + + +Involuntary disruptions cannot be prevented by PDBs; however they +do count against the budget. diff --git a/content/en/docs/reference/glossary/pod-disruption.md b/content/en/docs/reference/glossary/pod-disruption.md index 1efd69dd4c..d92d607855 100644 --- a/content/en/docs/reference/glossary/pod-disruption.md +++ b/content/en/docs/reference/glossary/pod-disruption.md @@ -14,6 +14,11 @@ tags: - operation --- -[Pod disruption](/docs/concepts/workloads/pods/disruptions/) is the process by which Pods on Nodes are terminated either voluntarily or involuntarily. +[Pod disruption](/docs/concepts/workloads/pods/disruptions/) is the process by which +Pods on Nodes are terminated either voluntarily or involuntarily. -Voluntary disruptions are started intentionally by application owners or cluster administrators. Involuntary disruptions are unintentional and can be triggered by unavoidable issues like Nodes running out of resources, or by accidental deletions. + + +Voluntary disruptions are started intentionally by application owners or cluster +administrators. Involuntary disruptions are unintentional and can be triggered by +unavoidable issues like Nodes running out of resources, or by accidental deletions. diff --git a/content/en/docs/reference/glossary/pod-security-policy.md b/content/en/docs/reference/glossary/pod-security-policy.md index 52c4433b48..ff178bb7cc 100644 --- a/content/en/docs/reference/glossary/pod-security-policy.md +++ b/content/en/docs/reference/glossary/pod-security-policy.md @@ -2,7 +2,7 @@ title: Pod Security Policy id: pod-security-policy date: 2018-04-12 -full_link: /docs/concepts/policy/pod-security-policy/ +full_link: /docs/concepts/security/pod-security-policy/ short_description: > Enables fine-grained authorization of pod creation and updates. @@ -17,3 +17,4 @@ tags: A cluster-level resource that controls security sensitive aspects of the Pod specification. The `PodSecurityPolicy` objects define a set of conditions that a Pod must run with in order to be accepted into the system, as well as defaults for the related fields. Pod Security Policy control is implemented as an optional admission controller. +PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. We recommend migrating to [Pod Security Admission](/docs/concepts/security/pod-security-admission/), or a 3rd party admission plugin. diff --git a/content/en/docs/reference/glossary/quantity.md b/content/en/docs/reference/glossary/quantity.md index c12260a992..33d2dc8e04 100644 --- a/content/en/docs/reference/glossary/quantity.md +++ b/content/en/docs/reference/glossary/quantity.md @@ -10,7 +10,7 @@ aka: tags: - core-object --- - A whole-number representation of small or large numbers using SI suffixes. + A whole-number representation of small or large numbers using [SI](https://en.wikipedia.org/wiki/International_System_of_Units) suffixes. @@ -21,7 +21,7 @@ mega, or giga units. For instance, the number `1.5` is represented as `1500m`, while the number `1000` can be represented as `1k`, and `1000000` as `1M`. You can also specify -binary-notation suffixes; the number 2048 can be written as `2Ki`. +[binary-notation](https://en.wikipedia.org/wiki/Binary_prefix) suffixes; the number 2048 can be written as `2Ki`. The accepted decimal (power-of-10) units are `m` (milli), `k` (kilo, intentionally lowercase), `M` (mega), `G` (giga), `T` (tera), `P` (peta), @@ -29,3 +29,4 @@ intentionally lowercase), `M` (mega), `G` (giga), `T` (tera), `P` (peta), The accepted binary (power-of-2) units are `Ki` (kibi), `Mi` (mebi), `Gi` (gibi), `Ti` (tebi), `Pi` (pebi), `Ei` (exbi). + diff --git a/content/en/docs/reference/glossary/secret.md b/content/en/docs/reference/glossary/secret.md index 48088bbf9c..281b8b7b65 100644 --- a/content/en/docs/reference/glossary/secret.md +++ b/content/en/docs/reference/glossary/secret.md @@ -15,4 +15,4 @@ tags: -Allows for more control over how sensitive information is used and reduces the risk of accidental exposure, including [encryption](/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted) at rest. A {{< glossary_tooltip text="Pod" term_id="pod" >}} references the secret as a file in a volume mount or by the kubelet pulling images for a pod. Secrets are great for confidential data and [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) for non-confidential data. +Allows for more control over how sensitive information is used and reduces the risk of accidental exposure. Secret values are encoded as base64 strings and stored unencrypted by default, but can be configured to be [encrypted at rest](/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted). A {{< glossary_tooltip text="Pod" term_id="pod" >}} references the secret as a file in a volume mount or by the kubelet pulling images for a pod. Secrets are great for confidential data and [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) for non-confidential data. diff --git a/content/en/docs/reference/glossary/service-broker.md b/content/en/docs/reference/glossary/service-broker.md deleted file mode 100644 index d35ea3d688..0000000000 --- a/content/en/docs/reference/glossary/service-broker.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: Service Broker -id: service-broker -date: 2018-04-12 -full_link: -short_description: > - An endpoint for a set of Managed Services offered and maintained by a third-party. - -aka: -tags: -- extension ---- - An endpoint for a set of {{< glossary_tooltip text="Managed Services" term_id="managed-service" >}} offered and maintained by a third-party. - - - -{{< glossary_tooltip text="Service Brokers" term_id="service-broker" >}} implement the -[Open Service Broker API spec](https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md) -and provide a standard interface for applications to use their Managed Services. -[Service Catalog](/docs/concepts/extend-kubernetes/service-catalog/) provides a way to -list, provision, and bind with Managed Services offered by Service Brokers. - diff --git a/content/en/docs/reference/glossary/service-catalog.md b/content/en/docs/reference/glossary/service-catalog.md index 3404543d18..471311426c 100644 --- a/content/en/docs/reference/glossary/service-catalog.md +++ b/content/en/docs/reference/glossary/service-catalog.md @@ -4,15 +4,15 @@ id: service-catalog date: 2018-04-12 full_link: short_description: > - An extension API that enables applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider. + A former extension API that enabled applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider. aka: tags: - extension --- - An extension API that enables applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider. + A former extension API that enabled applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider. -It provides a way to list, provision, and bind with external {{< glossary_tooltip text="Managed Services" term_id="managed-service" >}} from {{< glossary_tooltip text="Service Brokers" term_id="service-broker" >}} without needing detailed knowledge about how those services are created or managed. +It provided a way to list, provision, and bind with external {{< glossary_tooltip text="Managed Services" term_id="managed-service" >}} without needing detailed knowledge about how those services would be created or managed. diff --git a/content/en/docs/reference/glossary/shuffle-sharding.md b/content/en/docs/reference/glossary/shuffle-sharding.md index 7d1a128762..d41c704333 100644 --- a/content/en/docs/reference/glossary/shuffle-sharding.md +++ b/content/en/docs/reference/glossary/shuffle-sharding.md @@ -1,5 +1,5 @@ --- -title: shuffle sharding +title: Shuffle-sharding id: shuffle-sharding date: 2020-03-04 full_link: @@ -18,28 +18,28 @@ We are often concerned with insulating different flows of requests from each other, so that a high-intensity flow does not crowd out low-intensity flows. A simple way to put requests into queues is to hash some characteristics of the request, modulo the number of queues, to get -the index of the queue to use. The hash function uses as input -characteristics of the request that align with flows. For example, in +the index of the queue to use. The hash function uses as input +characteristics of the request that align with flows. For example, in the Internet this is often the 5-tuple of source and destination address, protocol, and source and destination port. That simple hash-based scheme has the property that any high-intensity flow will crowd out all the low-intensity flows that hash to the same queue. Providing good insulation for a large number of flows requires a large -number of queues, which is problematic. Shuffle sharding is a more +number of queues, which is problematic. Shuffle-sharding is a more nimble technique that can do a better job of insulating the low-intensity -flows from the high-intensity flows. The terminology of shuffle sharding uses +flows from the high-intensity flows. The terminology of shuffle-sharding uses the metaphor of dealing a hand from a deck of cards; each queue is a -metaphorical card. The shuffle sharding technique starts with hashing +metaphorical card. The shuffle-sharding technique starts with hashing the flow-identifying characteristics of the request, to produce a hash -value with dozens or more of bits. Then the hash value is used as a +value with dozens or more of bits. Then the hash value is used as a source of entropy to shuffle the deck and deal a hand of cards -(queues). All the dealt queues are examined, and the request is put -into one of the examined queues with the shortest length. With a +(queues). All the dealt queues are examined, and the request is put +into one of the examined queues with the shortest length. With a modest hand size, it does not cost much to examine all the dealt cards and a given low-intensity flow has a good chance to dodge the effects of a -given high-intensity flow. With a large hand size it is expensive to examine +given high-intensity flow. With a large hand size it is expensive to examine the dealt queues and more difficult for the low-intensity flows to dodge the -collective effects of a set of high-intensity flows. Thus, the hand size +collective effects of a set of high-intensity flows. Thus, the hand size should be chosen judiciously. diff --git a/content/en/docs/reference/issues-security/security.md b/content/en/docs/reference/issues-security/security.md index 7fe91a037a..e5d2a565dd 100644 --- a/content/en/docs/reference/issues-security/security.md +++ b/content/en/docs/reference/issues-security/security.md @@ -19,15 +19,13 @@ This page describes Kubernetes security and disclosure information. Join the [kubernetes-security-announce](https://groups.google.com/forum/#!forum/kubernetes-security-announce) group for emails about security and major API announcements. -You can also subscribe to an RSS feed of the above using [this link](https://groups.google.com/forum/feed/kubernetes-security-announce/msgs/rss_v2_0.xml?num=50). - ## Report a Vulnerability We're extremely grateful for security researchers and users that report vulnerabilities to the Kubernetes Open Source Community. All reports are thoroughly investigated by a set of community volunteers. To make a report, submit your vulnerability to the [Kubernetes bug bounty program](https://hackerone.com/kubernetes). This allows triage and handling of the vulnerability with standardized response times. -You can also email the private [security@kubernetes.io](mailto:security@kubernetes.io) list with the security details and the details expected for [all Kubernetes bug reports](https://git.k8s.io/kubernetes/.github/ISSUE_TEMPLATE/bug-report.md). +You can also email the private [security@kubernetes.io](mailto:security@kubernetes.io) list with the security details and the details expected for [all Kubernetes bug reports](https://github.com/kubernetes/kubernetes/blob/master/.github/ISSUE_TEMPLATE/bug-report.yaml). You may encrypt your email to this list using the GPG keys of the [Security Response Committee members](https://git.k8s.io/security/README.md#product-security-committee-psc). Encryption using GPG is NOT required to make a disclosure. diff --git a/content/en/docs/reference/kubectl/_index.md b/content/en/docs/reference/kubectl/_index.md index 765adb6fe8..cdbfca809e 100644 --- a/content/en/docs/reference/kubectl/_index.md +++ b/content/en/docs/reference/kubectl/_index.md @@ -1,5 +1,556 @@ --- -title: "kubectl" +title: Command line tool (kubectl) +content_type: reference weight: 60 +no_list: true +card: + name: reference + weight: 20 --- + +{{< glossary_definition prepend="Kubernetes provides a" term_id="kubectl" length="short" >}} + +This tool is named `kubectl`. + +For configuration, `kubectl` looks for a file named `config` in the `$HOME/.kube` directory. +You can specify other [kubeconfig](/docs/concepts/configuration/organize-cluster-access-kubeconfig/) +files by setting the `KUBECONFIG` environment variable or by setting the +[`--kubeconfig`](/docs/concepts/configuration/organize-cluster-access-kubeconfig/) flag. + +This overview covers `kubectl` syntax, describes the command operations, and provides common examples. +For details about each command, including all the supported flags and subcommands, see the +[kubectl](/docs/reference/generated/kubectl/kubectl-commands/) reference documentation. + +For installation instructions, see [Installing kubectl](/docs/tasks/tools/#kubectl); +for a quick guide, see the [cheat sheet](/docs/reference/kubectl/cheatsheet/). +If you're used to using the `docker` command-line tool, [`kubectl` for Docker Users](/docs/reference/kubectl/docker-cli-to-kubectl/) explains some equivalent commands for Kubernetes. + + + +## Syntax + +Use the following syntax to run `kubectl` commands from your terminal window: + +```shell +kubectl [command] [TYPE] [NAME] [flags] +``` + +where `command`, `TYPE`, `NAME`, and `flags` are: + +* `command`: Specifies the operation that you want to perform on one or more resources, +for example `create`, `get`, `describe`, `delete`. + +* `TYPE`: Specifies the [resource type](#resource-types). Resource types are case-insensitive and + you can specify the singular, plural, or abbreviated forms. + For example, the following commands produce the same output: + + ```shell + kubectl get pod pod1 + kubectl get pods pod1 + kubectl get po pod1 + ``` + +* `NAME`: Specifies the name of the resource. Names are case-sensitive. If the name is omitted, details for all resources are displayed, for example `kubectl get pods`. + + When performing an operation on multiple resources, you can specify each resource by type and name or specify one or more files: + + * To specify resources by type and name: + + * To group resources if they are all the same type: `TYPE1 name1 name2 name<#>`.
+ Example: `kubectl get pod example-pod1 example-pod2` + + * To specify multiple resource types individually: `TYPE1/name1 TYPE1/name2 TYPE2/name3 TYPE<#>/name<#>`.
+ Example: `kubectl get pod/example-pod1 replicationcontroller/example-rc1` + + * To specify resources with one or more files: `-f file1 -f file2 -f file<#>` + + * [Use YAML rather than JSON](/docs/concepts/configuration/overview/#general-configuration-tips) since YAML tends to be more user-friendly, especially for configuration files.
+ Example: `kubectl get -f ./pod.yaml` + +* `flags`: Specifies optional flags. For example, you can use the `-s` or `--server` flags to specify the address and port of the Kubernetes API server.
+ +{{< caution >}} +Flags that you specify from the command line override default values and any corresponding environment variables. +{{< /caution >}} + +If you need help, run `kubectl help` from the terminal window. + +## In-cluster authentication and namespace overrides + +By default `kubectl` will first determine if it is running within a pod, and thus in a cluster. It starts by checking for the `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` environment variables and the existence of a service account token file at `/var/run/secrets/kubernetes.io/serviceaccount/token`. If all three are found in-cluster authentication is assumed. + +To maintain backwards compatibility, if the `POD_NAMESPACE` environment variable is set during in-cluster authentication it will override the default namespace from the service account token. Any manifests or tools relying on namespace defaulting will be affected by this. + +**`POD_NAMESPACE` environment variable** + +If the `POD_NAMESPACE` environment variable is set, cli operations on namespaced resources will default to the variable value. For example, if the variable is set to `seattle`, `kubectl get pods` would return pods in the `seattle` namespace. This is because pods are a namespaced resource, and no namespace was provided in the command. Review the output of `kubectl api-resources` to determine if a resource is namespaced. + +Explicit use of `--namespace ` overrides this behavior. + +**How kubectl handles ServiceAccount tokens** + +If: +* there is Kubernetes service account token file mounted at + `/var/run/secrets/kubernetes.io/serviceaccount/token`, and +* the `KUBERNETES_SERVICE_HOST` environment variable is set, and +* the `KUBERNETES_SERVICE_PORT` environment variable is set, and +* you don't explicitly specify a namespace on the kubectl command line + +then kubectl assumes it is running in your cluster. The kubectl tool looks up the +namespace of that ServiceAccount (this is the same as the namespace of the Pod) +and acts against that namespace. This is different from what happens outside of a +cluster; when kubectl runs outside a cluster and you don't specify a namespace, +the kubectl command acts against the `default` namespace. + +## Operations + +The following table includes short descriptions and the general syntax for all of the `kubectl` operations: + +Operation | Syntax | Description +-------------------- | -------------------- | -------------------- +`alpha` | `kubectl alpha SUBCOMMAND [flags]` | List the available commands that correspond to alpha features, which are not enabled in Kubernetes clusters by default. +`annotate` | kubectl annotate (-f FILENAME | TYPE NAME | TYPE/NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--overwrite] [--all] [--resource-version=version] [flags] | Add or update the annotations of one or more resources. +`api-resources` | `kubectl api-resources [flags]` | List the API resources that are available. +`api-versions` | `kubectl api-versions [flags]` | List the API versions that are available. +`apply` | `kubectl apply -f FILENAME [flags]`| Apply a configuration change to a resource from a file or stdin. +`attach` | `kubectl attach POD -c CONTAINER [-i] [-t] [flags]` | Attach to a running container either to view the output stream or interact with the container (stdin). +`auth` | `kubectl auth [flags] [options]` | Inspect authorization. +`autoscale` | kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU] [flags] | Automatically scale the set of pods that are managed by a replication controller. +`certificate` | `kubectl certificate SUBCOMMAND [options]` | Modify certificate resources. +`cluster-info` | `kubectl cluster-info [flags]` | Display endpoint information about the master and services in the cluster. +`completion` | `kubectl completion SHELL [options]` | Output shell completion code for the specified shell (bash or zsh). +`config` | `kubectl config SUBCOMMAND [flags]` | Modifies kubeconfig files. See the individual subcommands for details. +`convert` | `kubectl convert -f FILENAME [options]` | Convert config files between different API versions. Both YAML and JSON formats are accepted. Note - requires `kubectl-convert` plugin to be installed. +`cordon` | `kubectl cordon NODE [options]` | Mark node as unschedulable. +`cp` | `kubectl cp [options]` | Copy files and directories to and from containers. +`create` | `kubectl create -f FILENAME [flags]` | Create one or more resources from a file or stdin. +`delete` | kubectl delete (-f FILENAME | TYPE [NAME | /NAME | -l label | --all]) [flags] | Delete resources either from a file, stdin, or specifying label selectors, names, resource selectors, or resources. +`describe` | kubectl describe (-f FILENAME | TYPE [NAME_PREFIX | /NAME | -l label]) [flags] | Display the detailed state of one or more resources. +`diff` | `kubectl diff -f FILENAME [flags]`| Diff file or stdin against live configuration. +`drain` | `kubectl drain NODE [options]` | Drain node in preparation for maintenance. +`edit` | kubectl edit (-f FILENAME | TYPE NAME | TYPE/NAME) [flags] | Edit and update the definition of one or more resources on the server by using the default editor. +`exec` | `kubectl exec POD [-c CONTAINER] [-i] [-t] [flags] [-- COMMAND [args...]]` | Execute a command against a container in a pod. +`explain` | `kubectl explain [--recursive=false] [flags]` | Get documentation of various resources. For instance pods, nodes, services, etc. +`expose` | kubectl expose (-f FILENAME | TYPE NAME | TYPE/NAME) [--port=port] [--protocol=TCP|UDP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type] [flags] | Expose a replication controller, service, or pod as a new Kubernetes service. +`get` | kubectl get (-f FILENAME | TYPE [NAME | /NAME | -l label]) [--watch] [--sort-by=FIELD] [[-o | --output]=OUTPUT_FORMAT] [flags] | List one or more resources. +`kustomize` | `kubectl kustomize [flags] [options]` | List a set of API resources generated from instructions in a kustomization.yaml file. The argument must be the path to the directory containing the file, or a git repository URL with a path suffix specifying same with respect to the repository root. +`label` | kubectl label (-f FILENAME | TYPE NAME | TYPE/NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--overwrite] [--all] [--resource-version=version] [flags] | Add or update the labels of one or more resources. +`logs` | `kubectl logs POD [-c CONTAINER] [--follow] [flags]` | Print the logs for a container in a pod. +`options` | `kubectl options` | List of global command-line options, which apply to all commands. +`patch` | kubectl patch (-f FILENAME | TYPE NAME | TYPE/NAME) --patch PATCH [flags] | Update one or more fields of a resource by using the strategic merge patch process. +`plugin` | `kubectl plugin [flags] [options]` | Provides utilities for interacting with plugins. +`port-forward` | `kubectl port-forward POD [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N] [flags]` | Forward one or more local ports to a pod. +`proxy` | `kubectl proxy [--port=PORT] [--www=static-dir] [--www-prefix=prefix] [--api-prefix=prefix] [flags]` | Run a proxy to the Kubernetes API server. +`replace` | `kubectl replace -f FILENAME` | Replace a resource from a file or stdin. +`rollout` | `kubectl rollout SUBCOMMAND [options]` | Manage the rollout of a resource. Valid resource types include: deployments, daemonsets and statefulsets. +`run` | kubectl run NAME --image=image [--env="key=value"] [--port=port] [--dry-run=server|client|none] [--overrides=inline-json] [flags] | Run a specified image on the cluster. +`scale` | kubectl scale (-f FILENAME | TYPE NAME | TYPE/NAME) --replicas=COUNT [--resource-version=version] [--current-replicas=count] [flags] | Update the size of the specified replication controller. +`set` | `kubectl set SUBCOMMAND [options]` | Configure application resources. +`taint` | `kubectl taint NODE NAME KEY_1=VAL_1:TAINT_EFFECT_1 ... KEY_N=VAL_N:TAINT_EFFECT_N [options]` | Update the taints on one or more nodes. +`top` | `kubectl top [flags] [options]` | Display Resource (CPU/Memory/Storage) usage. +`uncordon` | `kubectl uncordon NODE [options]` | Mark node as schedulable. +`version` | `kubectl version [--client] [flags]` | Display the Kubernetes version running on the client and server. +`wait` | kubectl wait ([-f FILENAME] | resource.group/resource.name | resource.group [(-l label | --all)]) [--for=delete|--for condition=available] [options] | Experimental: Wait for a specific condition on one or many resources. + +To learn more about command operations, see the [kubectl](/docs/reference/kubectl/kubectl/) reference documentation. + +## Resource types + +The following table includes a list of all the supported resource types and their abbreviated aliases. + +(This output can be retrieved from `kubectl api-resources`, and was accurate as of Kubernetes 1.19.1.) + +| NAME | SHORTNAMES | APIGROUP | NAMESPACED | KIND | +|---|---|---|---|---| +| `bindings` | | | true | Binding | +| `componentstatuses` | `cs` | | false | ComponentStatus | +| `configmaps` | `cm` | | true | ConfigMap | +| `endpoints` | `ep` | | true | Endpoints | +| `events` | `ev` | | true | Event | +| `limitranges` | `limits` | | true | LimitRange | +| `namespaces` | `ns` | | false | Namespace | +| `nodes` | `no` | | false | Node | +| `persistentvolumeclaims` | `pvc` | | true | PersistentVolumeClaim | +| `persistentvolumes` | `pv` | | false | PersistentVolume | +| `pods` | `po` | | true | Pod | +| `podtemplates` | | | true | PodTemplate | +| `replicationcontrollers` | `rc` | | true | ReplicationController | +| `resourcequotas` | `quota` | | true | ResourceQuota | +| `secrets` | | | true | Secret | +| `serviceaccounts` | `sa` | | true | ServiceAccount | +| `services` | `svc` | | true | Service | +| `mutatingwebhookconfigurations` | | admissionregistration.k8s.io | false | MutatingWebhookConfiguration | +| `validatingwebhookconfigurations` | | admissionregistration.k8s.io | false | ValidatingWebhookConfiguration | +| `customresourcedefinitions` | `crd,crds` | apiextensions.k8s.io | false | CustomResourceDefinition | +| `apiservices` | | apiregistration.k8s.io | false | APIService | +| `controllerrevisions` | | apps | true | ControllerRevision | +| `daemonsets` | `ds` | apps | true | DaemonSet | +| `deployments` | `deploy` | apps | true | Deployment | +| `replicasets` | `rs` | apps | true | ReplicaSet | +| `statefulsets` | `sts` | apps | true | StatefulSet | +| `tokenreviews` | | authentication.k8s.io | false | TokenReview | +| `localsubjectaccessreviews` | | authorization.k8s.io | true | LocalSubjectAccessReview | +| `selfsubjectaccessreviews` | | authorization.k8s.io | false | SelfSubjectAccessReview | +| `selfsubjectrulesreviews` | | authorization.k8s.io | false | SelfSubjectRulesReview | +| `subjectaccessreviews` | | authorization.k8s.io | false | SubjectAccessReview | +| `horizontalpodautoscalers` | `hpa` | autoscaling | true | HorizontalPodAutoscaler | +| `cronjobs` | `cj` | batch | true | CronJob | +| `jobs` | | batch | true | Job | +| `certificatesigningrequests` | `csr` | certificates.k8s.io | false | CertificateSigningRequest | +| `leases` | | coordination.k8s.io | true | Lease | +| `endpointslices` | | discovery.k8s.io | true | EndpointSlice | +| `events` | `ev` | events.k8s.io | true | Event | +| `ingresses` | `ing` | extensions | true | Ingress | +| `flowschemas` | | flowcontrol.apiserver.k8s.io | false | FlowSchema | +| `prioritylevelconfigurations` | | flowcontrol.apiserver.k8s.io | false | PriorityLevelConfiguration | +| `ingressclasses` | | networking.k8s.io | false | IngressClass | +| `ingresses` | `ing` | networking.k8s.io | true | Ingress | +| `networkpolicies` | `netpol` | networking.k8s.io | true | NetworkPolicy | +| `runtimeclasses` | | node.k8s.io | false | RuntimeClass | +| `poddisruptionbudgets` | `pdb` | policy | true | PodDisruptionBudget | +| `podsecuritypolicies` | `psp` | policy | false | PodSecurityPolicy | +| `clusterrolebindings` | | rbac.authorization.k8s.io | false | ClusterRoleBinding | +| `clusterroles` | | rbac.authorization.k8s.io | false | ClusterRole | +| `rolebindings` | | rbac.authorization.k8s.io | true | RoleBinding | +| `roles` | | rbac.authorization.k8s.io | true | Role | +| `priorityclasses` | `pc` | scheduling.k8s.io | false | PriorityClass | +| `csidrivers` | | storage.k8s.io | false | CSIDriver | +| `csinodes` | | storage.k8s.io | false | CSINode | +| `storageclasses` | `sc` | storage.k8s.io | false | StorageClass | +| `volumeattachments` | | storage.k8s.io | false | VolumeAttachment | + +## Output options + +Use the following sections for information about how you can format or sort the output of certain commands. For details about which commands support the various output options, see the [kubectl](/docs/reference/kubectl/kubectl/) reference documentation. + +### Formatting output + +The default output format for all `kubectl` commands is the human readable plain-text format. To output details to your terminal window in a specific format, you can add either the `-o` or `--output` flags to a supported `kubectl` command. + +#### Syntax + +```shell +kubectl [command] [TYPE] [NAME] -o +``` + +Depending on the `kubectl` operation, the following output formats are supported: + +Output format | Description +--------------| ----------- +`-o custom-columns=` | Print a table using a comma separated list of [custom columns](#custom-columns). +`-o custom-columns-file=` | Print a table using the [custom columns](#custom-columns) template in the `` file. +`-o json` | Output a JSON formatted API object. +`-o jsonpath=