The implementation of the `NotIn` pod selector expression in the policy
controller is backwards. If a value exists for the label in the
expression, and it is contained in the `NotIn` set, the expression will
return `true`, and it will return `false` when the value is _not_ in the
set. This is because it calls `values.contains(v)`, just like the `In`
expression.
* Update kubert to v0.10
* Update kube-rs to v0.75 (fixes#9339)
* Update k8s-openapi to v0.16
* Update k8s-gateway-api to v0.7
Signed-off-by: Oliver Gould <ver@buoyant.io>
This branch updates the policy controller's validating admission
controller to use the same validation functions as the indexer when
validating `HTTPRoute` resources. This way, we can ensure that any
`HTTPRoute` spec that passes validation will also convert to a valid
`InboundRouteBinding` in the indexer.
When there are multiple equivalent routes (e.g., two routes with the
same match), the proxy will use the first route in the returned list. We
need to ensure that the policy controller returns routes in a
deterministic order--and the Gateway API defines such an order:
> If ties still exist across multiple Routes, matching precedence MUST
> be determined in order of the following criteria, continuing on ties:
>
> * The oldest Route based on creation timestamp.
> * The Route appearing first in alphabetical order by
> "{namespace}/{name}".
This branch updates the policy controller to return the list of
`HttpRoute`s for an inbound server with a deterministic ordering based
on these rules. This is done by tracking the creation timestamp for
indexed `HTTPRoute` resources, and sorting the list of protobuf
`HttpRoute`s when the API server constructs an `InboundServer` response.
The implementation is *somewhat* hairy, because we can't just define a
custom `Ord` implementation for the protobuf `HttpRoute` type that
includes the timestamp --- doing so would require actually storing the
creation timestamp in the protobuf type, which would be a change in
`linkerd2-proxy-api` (and would result in serializing additional
information that the proxy itself doesn't actually care about). Instead,
we use `slice::sort_by` with a closure that looks up routes by name in
the hash map stored by the indexer in order to determine their
timestamps, and implements a custom ordering that first compares the
timestamp, and falls back to comparing the route's name if the
timestamps are equal. Note that we don't include the namespace in that
comparison, because all the routes for a given `InboundServer` are
already known to be in the same namespace.
I've also added an end-to-end test that the API returns the route list
in the correct order. Unfortunately, this test has 4 seconds of `sleep`s
in it, because the minimum resolution of Kubernetes creation timestamps
is 1 second. I figured a test that takes five or six seconds to run was
probably not a huge deal in the end to end tests --- some of the policy
tests take as long as a minute to run, at least on my machine.
Closes#8946
The proxy won't handle httproute paths (in URI rewrites or matches) when
paths are relative. The policy admission controller and indexer should
catch this case and fail to handle routes that deal in paths that do not
start in `/`.
This branch adds validation to the admission controller and indexer to
ensure that all paths in an `httproute` rule are absolute.
Part of #8945
In order for default inbound Servers to authorize probe routes, we must track
the probe ports and their expected paths when indexing Pods.
This introduces a new `_probes` field to Pod (which will be used in a follow-up
change) which maps probe ports to their expected paths.
For example, if a Pod’s container configures the following probes
```yaml
livenessProbe:
httpGet:
path: /live
port: 4191
...
readinessProbe:
httpGet:
path: /ready
port: 4191
...
```
Then we expect `_probes == {4191: {“/live”, “/ready”}}`
Signed-off-by: Kevin Leimkuhler <kleimkuhler@icloud.com>
As discussed in #8944, Linkerd's current use of the
`gateway.networking.k8s.io` `HTTPRoute` CRD is not a spec-compliant use
of the Gateway API, because we don't support some "core" features of the
Gateway API that don't make sense in Linkerd's use-case. Therefore,
we've chosen to replace the `gateway.networking.k8s.io` `HTTPRoute` CRD
with our own `HTTPRoute` CRD in the `policy.linkerd.io` API group, which
removes the unsupported features.
PR #8949 added the Linkerd versions of those CRDs, but did not remove
support for the Gateway API CRDs. This branch removes the Gateway API
CRDs from the policy controller and `linkerd install`/Helm charts.
The various helper functions for converting the Gateway API resource
binding types from `k8s-gateway-api` to the policy controller's internal
representation is kept in place, but the actual use of that code in the
indexer is disabled. This way, we can add support for the Gateway API
CRDs again easily. Similarly, I've kept the validation code for Gateway
API types in the policy admission controller, but the admission
controller no longer actually tries to validate those resources.
Depends on #8949Closes#8944
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Our use of the `gateway.networking.k8s.io` types is not compliant with
the gateway API spec in at least a few ways:
1. We do not support the `Gateway` types. This is considered a "core"
feature of the `HTTPRoute` type.
2. We do not currently update `HTTPRoute` status fields as dictated by
the spec.
3. Our use of Linkerd-specific `parentRef` types may not work well with
the gateway project's admission controller (untested).
Issue #8944 proposes solving this by replacing our use of
`gateway.networking.k8s.io`'s `HTTPRoute` type with our own
`policy.linkerd.io` version of the same type. That issue suggests that
the new `policy.linkerd.io` types be added separately from the change
that removes support for the `gateway.networking.k8s.io` versions, so
that the migration can be done incrementally.
This branch does the following:
* Add new `HTTPRoute` CRDs. These are based on the
`gateway.networking.k8s.io` CRDs, with the following changes:
- The group is `policy.linkerd.io`,
- The API version is `v1alpha1`,
- `backendRefs` fields are removed, as Linkerd does not support them,
- filter types Linkerd does not support (`RequestMirror` and
`ExtensionRef`), are removed.
* Add Rust bindings for the new `policy.linkerd.io` versions of
`HTTPRoute` types in `linkerd-policy-controller-k8s-api`.
The Rust bindings define their own versions of the `HttpRoute`,
`HttpRouteRule`, and `HttpRouteFilter` types, because these types'
structures are changed from the Gateway API versions (due to the
removal of unsupported filter types and fields). For other types,
which are identical to the upstream Gateway API versions (such as the
various match types and filter types), we re-export the existing
bindings from the `k8s-gateway-api`crate to minimize duplication.
* Add conversions to `InboundRouteBinding` from the `policy.linkerd.io`
`HTTPRoute` types.
When possible, I tried to factor out the code that was shared between
the conversions for Linkerd's `HTTPRoute` types and the upstream
Gateway API versions.
* Implement `kubert`'s `IndexNamespacedResource` trait for
`linkerd_policy_controller_k8s_api::policy::HttpRoute`, so that the
policy controller can index both versions of the `HTTPRoute` CRD.
* Adds validation for `policy.linkerd.io` `HTTPRoute`s to the policy
controller's validating admission webhook.
* Updated the policy controller tests to test both versions of
`HTTPRoute`.
## Notes
A couple questions I had about this approach:
- Is re-using bindings from the `k8s-gateway-api` crate appropriate
here, when the type has not changed from the Gateway API version? If
not, I can change this PR to vendor those types as well, but it will
result in a lot more code duplication.
- Right now, the indexer stores all `HTTPRoute`s in the same index.
This means that applying a `policy.linkerd.io` version of `HTTPRoute`
and then applying the Gateway API version with the same ns/name will
update the same value in the index. Is this what we want? I wasn't
entirely sure...
See #8944.
In various places we read port configurations from external sources
(either the Kubernetes API or gRPC clients). We have manual checks in
place to ensure that port values are never zero. We can instead assert
this with the type system by using `NonZeroU16`.
This change updates the policy controller to use `NonZeroU16` for port
values. This allows us to replace our manual port value checks with
`NonZero::try_from`, etc.
Signed-off-by: Oliver Gould <ver@buoyant.io>
* Replace deprecated uses of `ResourceExt::name` with
`ResourceExt::name_unchecked`;
* Update k8s-gateway-api to v0.6;
* Update kubert to v0.9.
Signed-off-by: Oliver Gould <ver@buoyant.io>
Currently, the policy admission controller requires that the
`AuthorizationPolicy` resources include a non-empty
`requiredAuthenticationRefs` field. This means that all authorization
policies require at least a `NetworkAuthentication` to permit traffic.
For example:
```yaml
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: ingress
spec:
targetRef:
group: policy.linkerd.io
kind: Server
name: ingress-http
requiredAuthenticationRefs:
- group: policy.linkerd.io
kind: NetworkAuthentication
name: all-nets
---
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata:
name: ingress-all-nets
spec:
networks:
- cidr: 0.0.0.0/0
- cidr: ::/0
```
This is needlessly verbose and can more simply be expressed as:
```yaml
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: ingress
spec:
targetRef:
group: policy.linkerd.io
kind: Server
name: ingress-http
requiredAuthenticationRefs: []
```
That is: there are explicitly no required authentications for this
policy.
This change updates the admission controller to permit such a policy.
Note that the `requiredAuthenticationRefs` field is still required so
that it's harder for simple misconfigurations to result in allowing
traffic.
This change also removes `Default` implementation for resources where do
they not make sense because there are required fields.
Signed-off-by: Oliver Gould <ver@buoyant.io>
Issue #7709 proposes new Custom Resource types to support generalized
authorization policies:
- `AuthorizationPolicy`
- `MeshTLSAuthentication`
- `NetworkAuthentication`
This change introduces these CRDs to the default linkerd installation
(via the `linkerd-crds` chart) and updates the policy controller's
to handle these resource types. The policy admission controller
validates that these resource reference only suppported types.
This new functionality is tested at multiple levels:
* `linkerd-policy-controller-k8s-index` includes unit tests for the
indexer to test how events update the index;
* `linkerd-policy-test` includes integration tests that run in-cluster
to validate that the gRPC API updates as resources are manipulated;
* `linkerd-policy-test` includes integration tests that exercise the
admission controller's resource validation; and
* `linkerd-policy-test` includes integration tests that ensure that
proxies honor authorization resources.
This change does NOT update Linkerd's control plane and extensions to
use these new authorization primitives. Furthermore, the `linkerd` CLI
does not yet support inspecting these new resource types. These
enhancements will be made in followup changes.
Signed-off-by: Oliver Gould <ver@buoyant.io>
In preparation for new policy CRD resources, this change adds end-to-end
tests to validate policy enforcement for `ServerAuthorization`
resources.
In adding these tests, it became clear that the OpenAPI validation for
`ServerAuthorization` resources is too strict. Various `oneof`
constraints have been removed in favor of admission controller
validation. These changes are semantically compatible and do not
necessitate an API version change.
The end-to-end tests work by creating `curl` pods that call an `nginx`
pod. In order to test network policies, the `curl` pod may be created
before the nginx pod, in which case an init container blocks execution
until a `curl-lock` configmap is deleted from the cluster. If the
configmap is not present to begin with, no blocking occurs.
Signed-off-by: Oliver Gould <ver@buoyant.io>
The policy controller's indexing module spans several files and relies
on an unnecessarily complex double-watch. It's generally confusing and
therefore difficult to change.
This change attempts to simplify the logic somewhat:
* All of the indexing code is now in the
`linkerd_policy_controller_k8s_index::index` module. No other files
have any dependencies on the internals of this data structure. It
exposes one public API, `Index::pod_server_rx`, used by discovery
clients.
* It uses the new `kubert::index` API so that we can avoid redundant
event-handling code. We now let kubert drive event processing so that
our indexing code is solely responsible for updating per-port server
configurations.
* A single watch is maintained for each pod:port.
* Watches are constructed lazily. The policy controller no longer
requires that all ports be documented on a pod. (The proxy still
requires this, however). This sets up for more flexible port
discovery.
Signed-off-by: Oliver Gould <ver@buoyant.io>
`ServerAuthorization` resources are not validated by the admission
controller.
This change enables validation for `ServerAuthorization` resources,
based on changes to the admission controller proposed as a part of
linkerd/linkerd2#8007. This admission controller is generalized to
support arbitrary resource types. The `ServerAuthoriation` validation
currently only ensures that network blocks are valid CIDRs and that they
are coherent. We use the new _schemars_ feature of `ipnet` v2.4.0 to
support using IpNet data structures directly in the custom resource
type bindings.
This change also adds an integration test to validate that the admission
controller behaves as expected.
Signed-off-by: Oliver Gould <ver@buoyant.io>
In preparation for introducing new policy types, this change reorganizes
the policy controller to keep more of each indexing module private.
Signed-off-by: Oliver Gould <ver@buoyant.io>
The policy controller has an validating webhook for `Server` resources,
but this functionality is not really tested.
Before adding more policy resources that need validation, let's add an
integration test that exercises resource validation. The initial test is
pretty simplistic, but this is just setup.
These test also help expose two issues:
1. The change in 8760c5f--to solely use the index for validation--is
problematic, especially in CI where quick updates can pass validation
when they should not. This is fixed by going back to making API calls
when validating `Server` resources.
2. Our pod selector overlap detection is overly simplistic. This change
updates it to at least detect when a server selects _all_ pods.
There's probably more we can do here in followup changes.
Tests are added in a new `policy-test` crate that only includes these
tests and the utiltities they need. This crate is excluded when running
unit tests and is only executed when it has a Kubernetes cluster it can
execute against. A temporary namespace is created before each test is
run and deleted as the test completes.
The policy controller's CI workflow is updated to build the core control
plane, run a k3d cluster, and exercise tests. This workflow has minimal
dependencies on the existing script/CI tooling so that the dependencies
are explicit and we can avoid some of the complexity of the existing
test infrastructure.
Signed-off-by: Oliver Gould <ver@buoyant.io>
`kubert` provides a runtime utility that helps reduce boilerplate around
process lifecycle management, construction of admin and HTTPS servers,
etc.
The admission controller server preserves the certificate reloading
functionality introduced in 96131b5 and updates the utility to read both
RSA and PKSC8 keys to close#7963.
Signed-off-by: Oliver Gould <ver@buoyant.io>
Fixes#7904
Allow the `Server` CRD to have the `PodSelector` entry be an empty object, by removing the `omitempty` tag from its go type definition and the `oneof` section in the CRD. No update to the CRD version is required, as this is BC change -- The CRD overriding was tested fine.
Also added some unit tests to confirm podSelector conditions are ANDed, and some minor refactorings in the `Selector` constructors.
Co-authored-by: Oliver Gould <ver@buoyant.io>
Kubernetes v1.19 is reaching its end-of-life date on 2021-10-28. In
anticipation of this, we should explicitly update our minimum supported
version to v1.20. This allows us keep our dependencies up-to-date and
ensures that we can actually test against our minimum supported version.
Fixes#7171
Co-authored-by: Alejandro Pedraza <alejandro@buoyant.io>
Fixes#6827
We upgrade the Server and ServerAuthorization CRD versions from v1alpha1 to v1beta1. This version update does not change the schema at all and the v1alpha1 versions will continue to be served for now. We also update the CLI and control plane to use the v1beta1 versions.
Signed-off-by: Alex Leong <alex@buoyant.io>
We initially implemented a mechanism to automatically authorize
unauthenticated traffic from each pod's Kubelet's IP. Our initial method
of determining a pod's Kubelet IP--using the first IP from its node's
pod CIDRs--is not a generally usable solution. In particular, CNIs
complicate matters (and EKS doesn't even set the podCIDRs field).
This change removes the policy controller's node watch and removes the
`default:kubelet` authorization. When using a restrictive default
policy, users will have to define `serverauthorization` resources that
permit kubelet traffic. It's probably possible to programatically
generate these authorizations (i.e. by inspecting pod probe
configurations); but this is out of scope for the core control plane
functionality.
We add a validating admission controller to the policy controller which validates `Server` resources. When a `Server` admission request is received, we look at all existing `Server` resources in the cluster and ensure that no other `Server` has an identical selector and port.
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
The policy controller's readiness and liveness admin endpoint is tied to
watch state: the controller only advertises liveness when all watches
have received updates; and after a watch disconnects liveness fails
until a new update is received.
However, in some environments--especially when the API server ends the
stream before the client gracefully reconnects--the watch terminess so
that liveness is not advertises even though the client resumes watching
resources. Because the watch is resumed with a `resourceVersion`, no
updates are provided despite the watch being reestablished, and liveness
checks fail until the pod is terminated (or an update is received).
To fix this, we modify readiness advertisements to fail only until the
initial state is acquired from all watches. After this, the controller
serves cached state indefinitely.
While diagnosing this, logging changes were needed, especially for the
`Watch` type. Watches now properly maintain logging contexts and state
transitions are logged in more cases. The signature and logging context
of `Index::run` has been updated as well. Additionally, node lookup
debug logs have been elaborated to help confirm that 'pending' messages
are benign.
We can't use the typical multiarch docker build with the proxy:
qemu-hosted arm64/arm builds take 45+ minutes before failing due to
missing tooling--specifically `protoc`. (While there is a `protoc`
binary available for arm64, there are no binaries available for 32-bit
arm hosts).
To fix this, this change updates the release process to cross-build the
policy-controller on an amd64 host to the target architecture. We
separate the policy-controller's dockerfiles as `amd64.dockerfile`,
`arm64.dockerfile`, and `arm.dockerfile`. Then, in CI we build and push
each of these images individually (in parallel, via a build matrix).
Once all of these are complete, we use the `docker manifest` CLI tools
to unify these images into a single multi-arch manifest.
This cross-building approach requires that we move from using
`native-tls` to `rustls`, as we cannot build against the platform-
appropriate native TLS libraries. The policy-controller is now feature-
flagged to use `rustls` by default, though it may be necessary to use
`native-tls` in local development, as `rustls` cannot validate TLS
connections that target IP addresses.
The policy-controller has also been updated to pull in `tracing-log` for
compatibility with crates that do not use `tracing` natively. This was
helpful while debugging connectivity issue with the Kubernetes cluster.
The `bin/docker-build-policy-controller` helper script now *only* builds
the amd64 variant of the policy controller. It fails when asked to build
multiarch images.
kube v0.59 depends on k8s-openapi v0.13, which includes breaking
changes.
This change updates these dependencies and modifies our code to account
for these changes.
Furthermore, we now use the k8s-openapi feature `v1_16` so that we use
an API version that is compatible with Linkerd's minimum support
kubernetes version.
Closes#6657#6658#6659
We've implemented a new controller--in Rust!--that implements discovery
APIs for inbound server policies. This change imports this code from
linkerd/polixy@25af9b5e.
This policy controller watches nodes, pods, and the recently-introduced
`policy.linkerd.io` CRD resources. It indexes these resources and serves
a gRPC API that will be used by proxies to configure the inbound proxy
for policy enforcement.
This change introduces a new policy-controller container image and adds a
container to the `Linkerd-destination` pod along with a `linkerd-policy` service
to be used by proxies.
This change adds a `policyController` object to the Helm `values.yaml` that
supports configuring the policy controller at runtime.
Proxies are not currently configured to use the policy controller at runtime. This
will change in an upcoming proxy release.
Eliza Weisman
Oliver Gould
Kevin Leimkuhler
dependabot[bot]
Alejandro Pedraza
Alex Leong