kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Apply suggestions from code review 

Co-authored-by: Marcelo Giles <marcelogiles@gmail.com>
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
This commit is contained in:
Sascha Grunert 2023-06-20 09:07:03 +02:00
parent e51a45d0fa
commit 4897928dbd
No known key found for this signature in database
GPG Key ID: 09D97D153EF94D93
1 changed files with 28 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

57
content/en/blog/_posts/2023-06-29-container-image-signature-verification/index.md
View File

@ -7,7 +7,7 @@ slug: container-image-signature-verification
**Author**: Sascha Grunert **Author**: Sascha Grunert
The Kubernetes community has been signing their container image based artifacts The Kubernetes community has been signing their container image-based artifacts
since release v1.24. While the graduation of the [corresponding enhancement][kep] since release v1.24. While the graduation of the [corresponding enhancement][kep]
from `alpha` to `beta` in v1.26 introduced signatures for the binary artifacts, from `alpha` to `beta` in v1.26 introduced signatures for the binary artifacts,
other projects followed the approach by providing image signatures for their other projects followed the approach by providing image signatures for their
@ -25,10 +25,10 @@ infrastructure for pushing images into staging buckets.
Assuming that a project now produces signed container image artifacts, how can Assuming that a project now produces signed container image artifacts, how can
one actually verify the signatures? It is possible to do it manually like one actually verify the signatures? It is possible to do it manually like
outlined in the [official Kubernetes documentation][docs]. The problem with that outlined in the [official Kubernetes documentation][docs]. The problem with this
approach is that it involves no automation at all and should be only done for approach is that it involves no automation at all and should be only done for
testing purposes. In production environments, tools like the [sigstore testing purposes. In production environments, tools like the [sigstore
policy-controller][policy-controller] can help with the automation. They policy-controller][policy-controller] can help with the automation. These tools
provide a higher level API by using [Custom Resource Definitions (CRD)][crd] as provide a higher level API by using [Custom Resource Definitions (CRD)][crd] as
well as an integrated [admission controller and webhook][admission] to verify well as an integrated [admission controller and webhook][admission] to verify
the signatures. the signatures.
@ -42,27 +42,26 @@ The general usage flow for an admission controller based verification is:
![flow](flow.png "Admission controller flow") ![flow](flow.png "Admission controller flow")
Key benefit of this architecture is simplicity: A single instance within the A key benefit of this architecture is simplicity: A single instance within the
cluster validates the signatures before any image pull can happen in the cluster validates the signatures before any image pull can happen in the
container runtimes on the nodes, which gets initiated by the kubelet. This container runtime on the nodes, which gets initiated by the kubelet. This
benefit also incorporates the drawback of separation: The node which should pull benefit also brings along the issue of separation: The node which should pull
the container image is not necessarily the same which does the admission. This the container image is not necessarily the same node that performs the admission. This
means that if the controller is compromised, then a cluster-wide policy means that if the controller is compromised, then a cluster-wide policy
enforcement could not be possible any more. enforcement can no longer be possible.
One way to solve that issue is doing the policy evaluation directly within the One way to solve this issue is doing the policy evaluation directly within the
[Container Runtime Interface (CRI)][cri] compatible container runtime. The [Container Runtime Interface (CRI)][cri] compatible container runtime. The
runtime is directly connected to the [kubelet][kubelet] on a node and does all runtime is directly connected to the [kubelet][kubelet] on a node and does all
the tasks like pulling images. [CRI-O][cri-o] is one of those available runtimes the tasks like pulling images. [CRI-O][cri-o] is one of those available runtimes
and will feature full support for container image signature verification in the and will feature full support for container image signature verification in v1.28.
upcoming v1.28 release.
[cri]: /docs/concepts/architecture/cri [cri]: /docs/concepts/architecture/cri
[kubelet]: /docs/reference/command-line-tools-reference/kubelet [kubelet]: /docs/reference/command-line-tools-reference/kubelet
[cri-o]: https://github.com/cri-o/cri-o [cri-o]: https://github.com/cri-o/cri-o
How does it work? CRI-O reads a file called [`policy.json`][policy.json], which How does it work? CRI-O reads a file called [`policy.json`][policy.json], which
contains all the rules defined for container images. For example, I can define a contains all the rules defined for container images. For example, you can define a
policy which only allows signed images `quay.io/crio/signed` for any tag or policy which only allows signed images `quay.io/crio/signed` for any tag or
digest like this: digest like this:
@ -90,7 +89,7 @@ digest like this:
} }
``` ```
CRI-O has to be started to use that policy as global source of truth: CRI-O has to be started to use that policy as the global source of truth:
```console ```console
> sudo crio --log-level debug --signature-policy ./policy.json > sudo crio --log-level debug --signature-policy ./policy.json
@ -135,7 +134,7 @@ keys from the upstream [fulcio (OIDC PKI)][fulcio] and [rekor
[fulcio]: https://github.com/sigstore/fulcio [fulcio]: https://github.com/sigstore/fulcio
[rekor]: https://github.com/sigstore/rekor [rekor]: https://github.com/sigstore/rekor
This means if I now invalidate the `subjectEmail` of the policy, for example to This means that if you now invalidate the `subjectEmail` of the policy, for example to
`wrong@mail.com`: `wrong@mail.com`:
```console ```console
@ -143,20 +142,20 @@ This means if I now invalidate the `subjectEmail` of the policy, for example to
> mv new-policy.json policy.json > mv new-policy.json policy.json
``` ```
Then removing the image, because it already exists locally: Then remove the image, since it already exists locally:
```console ```console
> sudo crictl rmi quay.io/crio/signed > sudo crictl rmi quay.io/crio/signed
``` ```
Now when pulling the image, CRI-O complains that the required email is wrong: Now when you pull the image, CRI-O complains that the required email is wrong:
```console ```console
> sudo crictl pull quay.io/crio/signed > sudo crictl pull quay.io/crio/signed
FATA[…] pulling image: rpc error: code = Unknown desc = Source image rejected: Required email wrong@mail.com not found (got []string{"sgrunert@redhat.com"}) FATA[…] pulling image: rpc error: code = Unknown desc = Source image rejected: Required email wrong@mail.com not found (got []string{"sgrunert@redhat.com"})
``` ```
It is also possible to test an unsigned image against the policy. For that we It is also possible to test an unsigned image against the policy. For that you
have to modify the key `quay.io/crio/signed` to something like have to modify the key `quay.io/crio/signed` to something like
`quay.io/crio/unsigned`: `quay.io/crio/unsigned`:
@ -164,7 +163,7 @@ have to modify the key `quay.io/crio/signed` to something like
> sed -i 's;quay.io/crio/signed;quay.io/crio/unsigned;' policy.json > sed -i 's;quay.io/crio/signed;quay.io/crio/unsigned;' policy.json
``` ```
If I now pull the container image, CRI-O will complain that no signature exists If you now pull the container image, CRI-O will complain that no signature exists
for it: for it:
```console ```console
@ -175,7 +174,7 @@ FATA[…] pulling image: rpc error: code = Unknown desc = SignatureValidationFai
The error code `SignatureValidationFailed` got [recently added to The error code `SignatureValidationFailed` got [recently added to
Kubernetes][pr-117717] and will be available from v1.28. This error code allows Kubernetes][pr-117717] and will be available from v1.28. This error code allows
end-users to understand image pull failures directly from the kubectl CLI. For end-users to understand image pull failures directly from the kubectl CLI. For
example, if I run CRI-O together with Kubernetes using the policy which requires example, if you run CRI-O together with Kubernetes using the policy which requires
`quay.io/crio/unsigned` to be signed, then a pod definition like this: `quay.io/crio/unsigned` to be signed, then a pod definition like this:
[pr-117717]: https://github.com/kubernetes/kubernetes/pull/117717 [pr-117717]: https://github.com/kubernetes/kubernetes/pull/117717
@ -219,41 +218,41 @@ pod 0/1 SignatureValidationFailed 0 4s
This overall behavior provides a more Kubernetes native experience and does not This overall behavior provides a more Kubernetes native experience and does not
rely on third party software to be installed in the cluster. rely on third party software to be installed in the cluster.
There are still a few corner cases to consider: For example, what if we want to There are still a few corner cases to consider: For example, what if you want to
allow policies per namespace in the same way the policy-controller supports it? allow policies per namespace in the same way the policy-controller supports it?
Well, there is an upcoming CRI-O feature in v1.28 for that! CRI-O will support Well, there is an upcoming CRI-O feature in v1.28 for that! CRI-O will support
the `--signature-policy-dir` / `signature_policy_dir` option, which defines the the `--signature-policy-dir` / `signature_policy_dir` option, which defines the
root path for pod namespace-separated signature policies. This means that CRI-O root path for pod namespace-separated signature policies. This means that CRI-O
will lookup that path and assemble a policy like `<SIGNATURE_POLICY_DIR>/<NAMESPACE>.json`, will lookup that path and assemble a policy like `<SIGNATURE_POLICY_DIR>/<NAMESPACE>.json`,
which will be used on image pull if existing. If no pod namespace is being which will be used on image pull if existing. If no pod namespace is
provided on image pull ([via the sandbox config][sandbox-config]), or the provided on image pull ([via the sandbox config][sandbox-config]), or the
concatenated path is non-existent, then CRI-O's global policy will be used as concatenated path is non-existent, then CRI-O's global policy will be used as
fallback. fallback.
[sandbox-config]: https://github.com/kubernetes/cri-api/blob/e5515a5/pkg/apis/runtime/v1/api.proto#L1448 [sandbox-config]: https://github.com/kubernetes/cri-api/blob/e5515a5/pkg/apis/runtime/v1/api.proto#L1448
Another corner case to consider is cricital for the correct signature Another corner case to consider is critical for the correct signature
verification within container runtimes: The kubelet only invokes container image verification within container runtimes: The kubelet only invokes container image
pulls if the image does not already exist on disk. This means, that a pulls if the image does not already exist on disk. This means that an
unrestricted policy from Kubernetes namespace A can allow pulling an image, unrestricted policy from Kubernetes namespace A can allow pulling an image,
while namespace B is not able to enforce the policy because it already exits on while namespace B is not able to enforce the policy because it already exits on
the node. Finally, CRI-O has to verify the policy not only on image pull, but the node. Finally, CRI-O has to verify the policy not only on image pull, but
also on container creation. This fact makes things even a bit more complicated, also on container creation. This fact makes things even a bit more complicated,
because the CRI does not really pass down the user specified image reference on because the CRI does not really pass down the user specified image reference on
container creation, but more an already resolved iamge ID or digest. A [small container creation, but an already resolved image ID, or digest. A [small
change to the CRI][pr-118652] can help with that. change to the CRI][pr-118652] can help with that.
[pr-118652]: https://github.com/kubernetes/kubernetes/pull/118652 [pr-118652]: https://github.com/kubernetes/kubernetes/pull/118652
Now that everything happens within the container runtime, someone has to Now that everything happens within the container runtime, someone has to
maintain and define the policies to provide a good user experience around that maintain and define the policies to provide a good user experience around that
feature. The CRDs of the policy-controller are great, while I could imagine that feature. The CRDs of the policy-controller are great, while we could imagine that
a daemon within the cluster can write the policies for CRI-O per namespace. This a daemon within the cluster can write the policies for CRI-O per namespace. This
would make any additional hook obsolete and moves the responsibility of would make any additional hook obsolete and moves the responsibility of
verifying the image signature to the actual instance which pulls the image. [I verifying the image signature to the actual instance which pulls the image. [We
was evaluating][thread] other possible paths towards a better container image evaluated][thread] other possible paths toward a better container image
signature verification within plain Kubernetes, but I could not find a great fit signature verification within plain Kubernetes, but we could not find a great fit
for a native API. This means that I believe that a CRD is the way to go, but we for a native API. This means that we believe that a CRD is the way to go, but we
still need an instance which actually serves it. still need an instance which actually serves it.
[thread]: https://groups.google.com/g/kubernetes-sig-node/c/kgpxqcsJ7Vc/m/7X7t_ElsAgAJ [thread]: https://groups.google.com/g/kubernetes-sig-node/c/kgpxqcsJ7Vc/m/7X7t_ElsAgAJ