From 32d19d3d2e1d3934be72686d40dcff826c1b1a97 Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Sat, 22 Jun 2024 17:55:42 +0900 Subject: [PATCH 1/7] blog post for KEP-3619: Fine-grained SupplementalGroups control --- .../implicit-groups.yaml | 15 ++ .../index.md | 160 ++++++++++++++++++ .../strict-supplementalgroups-policy.yaml | 16 ++ 3 files changed, 191 insertions(+) create mode 100644 content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml create mode 100644 content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md create mode 100644 content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml new file mode 100644 index 0000000000..2e50ed0104 --- /dev/null +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Pod +metadata: + name: implicit-groups +spec: + securityContext: + runAsUser: 1000 + runAsGroup: 3000 + supplementalGroups: [4000] + containers: + - name: ctr + image: registry.k8s.io/e2e-test-images/agnhost:2.45 + command: [ "sh", "-c", "sleep 1h" ] + securityContext: + allowPrivilegeEscalation: false diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md new file mode 100644 index 0000000000..12ee6bea28 --- /dev/null +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md @@ -0,0 +1,160 @@ +--- +layout: blog +title: 'Kubernetes 1.31: Fine-grained SupplementalGroups control' +date: xxxx-xx-xx +slug: fine-grained-supplementalgroups-control +author: > + Shingo Omura (Woven By Toyota) + +--- + +This blog discusses a new feature in Kubernetes 1.31 to improve the handling of supplementary groups in containers in Pods. + + +## Motivation: Implicit group memberships defined in `/etc/group` in the container image + +Although this behavior may not be quite popular by many Kubernetes cluster users/admins, kubernetes, by default, _merges_ group information from the Pod with information defined in `/etc/group` in the container image. + +Let's see an example, below Pod specifies `runAsUser=1000`, `runAsGroup=3000` and `supplementalGroups=4000` in the Pod's security context. + +{{% code_sample file="implicit-groups.yaml" %}} + +What is the result of `id` command in the `ctr` container? + +```shell +# Create the Pod: +$ kubectl apply -f https://k8s.io/blog/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml + +# Verify that the Pod's Container is running: +$ kubectl get pod implicit-groups + +# Check the id command +$ kubectl exec implicit-groups -- id +``` + +Then, output should be similar to this: + +```none +uid=1000 gid=3000 groups=3000,4000,50000 +``` + +Where does group ID `50000` in supplementary groups (`groups` field) come from even though `50000` is not defined in the Pod's manifest at all? The answer is `/etc/group` file in the container image. + +Checking the contents of `/etc/group` in the container image should show below: + +```shell +$ kubectl exec implicit-groups -- cat /etc/group +... +user-defined-in-image:x:1000: +group-defined-in-image:x:50000:user-defined-in-image +``` + +Aha! The container's primary user `1000` belongs to the group `50000` in the last entry. + +Thus, the group membership defined in `/etc/group` in the container image for the container's primary user is _implicitly_ merged to the information from the Pod. + +### What's wrong with it? + +The _implicitly_ merged group information from `/etc/group` in the container image may cause some concerns particularly in accessing volumes (see [kubernetes/kubernetes#112879](https://issue.k8s.io/112879) for details) because file permission is controlled by uid/gid in Linux. Even worse, the implicit gids from `/etc/group` can not be detected/validated by any policy engines becuase there is no clue for the implicit group information in the manifest. This can also be concern for Kubernetes security. + +## Fine-grined SupplementalGroups control in a Pod: `SupplementaryGroupsPolicy` + +To tackle the above problem, Kubernetes 1.31 introduces new field `supplementalGroupsPolicy` in Pod's `.spec.securityContext`. + +This field provies a way to control how to calculate supplementary groups for the container processes in a Pod. The available policy is below: + +* _Merge_: The group membership defined in `/etc/group` for the container's primary user will be merged. If not specified, this policy will be applied (i.e. as-is behavior for backword compatibility). + +* _Strict_: it only attaches specified group IDs in `fsGroup`, `supplementalGroups`, or `runAsGroup` fields as the supplementary groups of the container processes. This means no group membership defined in `/etc/group` for the container's primary user will be merged. + +Let's see how `Strict` policy works. + +{{% code_sample file="strict-supplementalgroups-policy.yaml" %}} + +```shell +# Create the Pod: +$ kubectl apply -f https://k8s.io/blog/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml + +# Verify that the Pod's Container is running: +$ kubectl get pod strict-supplementalgroups-policy + +# Check the process identity: +kubectl exec -it strict-supplementalgroups-policy -- id +``` + +The output should be similar to this: + +```none +uid=1000 gid=3000 groups=3000,4000 +``` + +You can see `Strict` policy can exclude group `50000` from `groups`! + +Thus, enforcing `SupplementalGroupsPolicy=Merge` by some policy engines helps preventing the implicit supplementary groups in a Pod (note: actually, this is not enough because container with strong priviledge/capability can change its process identity. Please see the following section for details). + + +## Attached process identity in Pod status + +This feature also exposes the process identity attached to the first container process of the container +via `.status.containerStatuses[].user.linux` field. It would be helpful to see if implicit group IDs are attached. + +```yaml +... +status: + containerStatuses: + - name: ctr + user: + linux: + gid: 3000 + supplementalGroups: + - 3000 + - 4000 + uid: 1000 +... +``` + +{{}} +Please note that the values in `status.containerStatuses[].user.linux` field is _the firstly attached_ +process identity to the first container process in the container. If the container has sufficient privilege +to call system calls related to process identity (e.g. [`setuid(2)`](https://man7.org/linux/man-pages/man2/setuid.2.html), [`setgid(2)`](https://man7.org/linux/man-pages/man2/setgid.2.html) or [`setgroups(2)`](https://man7.org/linux/man-pages/man2/setgroups.2.html), etc.), the container process can change its identity. Thus, the _actual_ process identity will be dynamic. +{{}} + +## Feature availability + +To enable `supplementalGroupsPolicy` field, the following components have to be used: + +- Kubernetes: v1.31 or later, with the `SupplementalGroupsPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled. As of v1.31, the gate is marked as alpha. +- CRI runtime: + - containerd: v2.0 or later + - CRI-O: v1.31 or later + +You can see if the feature is supported in the Node's `.status.features.supplementalGroupsPolicy` field. + +```yaml +apiVersion: v1 +kind: Node +... +status: + features: + supplementalGroupsPolicy: true +``` + +## What's next? + +Kubernetes SIG Node hope - and expect - that the feature will be promoted to beta and eventually +general availability (GA) in future releases of Kubernetes, so that users no longer need to enable +the feature gate manually. + +`Merge` policy is applied when `supplementalGroupsPolicy` is not specified, for backwards compatibility. + +## How can I learn more? + + +Please check out the [documentation](/content/en/docs/tasks/configure-pod-container/security-context/) +for the further details of `supplementalGroupsPolicy`. + +## How to get involved? + +This feature is driven by the SIG Node community. Please join us to connect with +the community and share your ideas and feedback around the above feature and +beyond. We look forward to hearing from you! diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml new file mode 100644 index 0000000000..8c949ba10f --- /dev/null +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Pod +metadata: + name: strict-supplementalgroups-policy +spec: + securityContext: + runAsUser: 1000 + runAsGroup: 3000 + supplementalGroups: [4000] + supplementalGroupsPolicy: Strict + containers: + - name: ctr + image: registry.k8s.io/e2e-test-images/agnhost:2.45 + command: [ "sh", "-c", "sleep 1h" ] + securityContext: + allowPrivilegeEscalation: false From 5a7e68c3e7db452941a0d656e7eb38ff5aab9f7d Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Wed, 24 Jul 2024 23:15:23 +0900 Subject: [PATCH 2/7] Update content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md --- .../xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md index 12ee6bea28..a5eac92ba4 100644 --- a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md @@ -38,7 +38,7 @@ Then, output should be similar to this: uid=1000 gid=3000 groups=3000,4000,50000 ``` -Where does group ID `50000` in supplementary groups (`groups` field) come from even though `50000` is not defined in the Pod's manifest at all? The answer is `/etc/group` file in the container image. +Where does group ID `50000` in supplementary groups (`groups` field) come from, even though `50000` is not defined in the Pod's manifest at all? The answer is `/etc/group` file in the container image. Checking the contents of `/etc/group` in the container image should show below: From 2f56106182d6290dcee31b3aa3050a48aff559df Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Wed, 24 Jul 2024 23:16:51 +0900 Subject: [PATCH 3/7] Update content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md --- .../xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md index a5eac92ba4..d33bd1933e 100644 --- a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md @@ -51,7 +51,7 @@ group-defined-in-image:x:50000:user-defined-in-image Aha! The container's primary user `1000` belongs to the group `50000` in the last entry. -Thus, the group membership defined in `/etc/group` in the container image for the container's primary user is _implicitly_ merged to the information from the Pod. +Thus, the group membership defined in `/etc/group` in the container image for the container's primary user is _implicitly_ merged to the information from the Pod. Please note that this was a design decision the current CRI implementations inherited from Docker, and the community never really reconsidered it until now. ### What's wrong with it? From 85c3fe26812a2efd4e789cfca6c967980f99f5f2 Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Wed, 24 Jul 2024 23:37:26 +0900 Subject: [PATCH 4/7] Update content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md --- .../xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md index d33bd1933e..1294cfc8fc 100644 --- a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md @@ -55,7 +55,7 @@ Thus, the group membership defined in `/etc/group` in the container image for th ### What's wrong with it? -The _implicitly_ merged group information from `/etc/group` in the container image may cause some concerns particularly in accessing volumes (see [kubernetes/kubernetes#112879](https://issue.k8s.io/112879) for details) because file permission is controlled by uid/gid in Linux. Even worse, the implicit gids from `/etc/group` can not be detected/validated by any policy engines becuase there is no clue for the implicit group information in the manifest. This can also be concern for Kubernetes security. +The _implicitly_ merged group information from `/etc/group` in the container image may cause some concerns particularly in accessing volumes (see [kubernetes/kubernetes#112879](https://issue.k8s.io/112879) for details) because file permission is controlled by uid/gid in Linux. Even worse, the implicit gids from `/etc/group` can not be detected/validated by any policy engines because there is no clue for the implicit group information in the manifest. This can also be a concern for Kubernetes security. ## Fine-grined SupplementalGroups control in a Pod: `SupplementaryGroupsPolicy` From 14612bb32776a7b740db3adddfa8600f6a6760f4 Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Mon, 29 Jul 2024 12:50:23 +0900 Subject: [PATCH 5/7] KEP-3619: Set publishing date: 2024-08-22 --- .../implicit-groups.yaml | 0 .../index.md | 6 +++--- .../strict-supplementalgroups-policy.yaml | 0 3 files changed, 3 insertions(+), 3 deletions(-) rename content/en/blog/_posts/{xxxx-xx-xx-Fine-grained-SupplementalGroups-control => 2024-08-22-Fine-grained-SupplementalGroups-control}/implicit-groups.yaml (100%) rename content/en/blog/_posts/{xxxx-xx-xx-Fine-grained-SupplementalGroups-control => 2024-08-22-Fine-grained-SupplementalGroups-control}/index.md (97%) rename content/en/blog/_posts/{xxxx-xx-xx-Fine-grained-SupplementalGroups-control => 2024-08-22-Fine-grained-SupplementalGroups-control}/strict-supplementalgroups-policy.yaml (100%) diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/implicit-groups.yaml similarity index 100% rename from content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml rename to content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/implicit-groups.yaml diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md similarity index 97% rename from content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md rename to content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md index 1294cfc8fc..8accccd38f 100644 --- a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md @@ -1,7 +1,7 @@ --- layout: blog title: 'Kubernetes 1.31: Fine-grained SupplementalGroups control' -date: xxxx-xx-xx +date: 2024-08-22 slug: fine-grained-supplementalgroups-control author: > Shingo Omura (Woven By Toyota) @@ -23,7 +23,7 @@ What is the result of `id` command in the `ctr` container? ```shell # Create the Pod: -$ kubectl apply -f https://k8s.io/blog/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/implicit-groups.yaml +$ kubectl apply -f https://k8s.io/blog/2024-08-22-Fine-grained-SupplementalGroups-control/implicit-groups.yaml # Verify that the Pod's Container is running: $ kubectl get pod implicit-groups @@ -73,7 +73,7 @@ Let's see how `Strict` policy works. ```shell # Create the Pod: -$ kubectl apply -f https://k8s.io/blog/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml +$ kubectl apply -f https://k8s.io/blog/2024-08-22-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml # Verify that the Pod's Container is running: $ kubectl get pod strict-supplementalgroups-policy diff --git a/content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml similarity index 100% rename from content/en/blog/_posts/xxxx-xx-xx-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml rename to content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml From a8fa614e22cd9e683a201d6c85c2d99d27be6a67 Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Mon, 29 Jul 2024 12:55:46 +0900 Subject: [PATCH 6/7] KEP-3619: addressed review feedbacks --- .../index.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md index 8accccd38f..2c8882f90e 100644 --- a/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md @@ -8,12 +8,12 @@ author: > --- -This blog discusses a new feature in Kubernetes 1.31 to improve the handling of supplementary groups in containers in Pods. +This blog discusses a new feature in Kubernetes 1.31 to improve the handling of supplementary groups in containers within Pods. ## Motivation: Implicit group memberships defined in `/etc/group` in the container image -Although this behavior may not be quite popular by many Kubernetes cluster users/admins, kubernetes, by default, _merges_ group information from the Pod with information defined in `/etc/group` in the container image. +Although this behavior may not be popular with many Kubernetes cluster users/admins, kubernetes, by default, _merges_ group information from the Pod with information defined in `/etc/group` in the container image. Let's see an example, below Pod specifies `runAsUser=1000`, `runAsGroup=3000` and `supplementalGroups=4000` in the Pod's security context. @@ -90,8 +90,11 @@ uid=1000 gid=3000 groups=3000,4000 You can see `Strict` policy can exclude group `50000` from `groups`! -Thus, enforcing `SupplementalGroupsPolicy=Merge` by some policy engines helps preventing the implicit supplementary groups in a Pod (note: actually, this is not enough because container with strong priviledge/capability can change its process identity. Please see the following section for details). +Thus, enforcing `SupplementalGroupsPolicy=Merge` by some policy engines helps preventing the implicit supplementary groups in a Pod. +{{}} +Actually, this is not enough because container with strong priviledge/capability can change its process identity. Please see the following section for details). +{{}} ## Attached process identity in Pod status From fee94ce486e3bfccfdbb9c658408ee13ee282d55 Mon Sep 17 00:00:00 2001 From: Shingo Omura Date: Thu, 8 Aug 2024 10:52:53 +0900 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: Tim Bannister --- .../index.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md index 2c8882f90e..e1d12ee6fc 100644 --- a/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md +++ b/content/en/blog/_posts/2024-08-22-Fine-grained-SupplementalGroups-control/index.md @@ -21,7 +21,7 @@ Let's see an example, below Pod specifies `runAsUser=1000`, `runAsGroup=3000` an What is the result of `id` command in the `ctr` container? -```shell +```console # Create the Pod: $ kubectl apply -f https://k8s.io/blog/2024-08-22-Fine-grained-SupplementalGroups-control/implicit-groups.yaml @@ -42,7 +42,7 @@ Where does group ID `50000` in supplementary groups (`groups` field) come from, Checking the contents of `/etc/group` in the container image should show below: -```shell +```console $ kubectl exec implicit-groups -- cat /etc/group ... user-defined-in-image:x:1000: @@ -71,7 +71,7 @@ Let's see how `Strict` policy works. {{% code_sample file="strict-supplementalgroups-policy.yaml" %}} -```shell +```console # Create the Pod: $ kubectl apply -f https://k8s.io/blog/2024-08-22-Fine-grained-SupplementalGroups-control/strict-supplementalgroups-policy.yaml @@ -90,10 +90,10 @@ uid=1000 gid=3000 groups=3000,4000 You can see `Strict` policy can exclude group `50000` from `groups`! -Thus, enforcing `SupplementalGroupsPolicy=Merge` by some policy engines helps preventing the implicit supplementary groups in a Pod. +Thus, ensuring `supplementalGroupsPolicy: Merge` (enforced by some policy mechanism) helps prevent the implicit supplementary groups in a Pod. {{}} -Actually, this is not enough because container with strong priviledge/capability can change its process identity. Please see the following section for details). +Actually, this is not enough because container with sufficient privileges / capability can change its process identity. Please see the following section for details. {{}} ## Attached process identity in Pod status @@ -153,7 +153,7 @@ the feature gate manually. ## How can I learn more? -Please check out the [documentation](/content/en/docs/tasks/configure-pod-container/security-context/) +Please check out the [documentation](/docs/tasks/configure-pod-container/security-context/) for the further details of `supplementalGroupsPolicy`. ## How to get involved?