## Subject
Prepare to expand the subset of [`Gateway API`](https://gateway-api.sigs.k8s.io/api-types/grpcroute/) route types `linkerd` supports driving outbound traffic with in [`linkerd-policy-controller-k8s-index`](https://github.com/linkerd/linkerd2/tree/main/policy-controller/k8s/status).
## Problem
Currently, the policy controller's `index` component is written with `HTTPRoute` support (effectively) exclusively, both in its structure/organization as well as its naming (e.g. `HttpRoute` as a primary type name, `convert_http_route` as a method name, etc...).
In order to expand the subset of route types defined by the Gateway API that `linkerd` supports for driving outbound traffic policy, the policy controller's `index` component needs to be made more generic in both respects.
## Solution
PR introduces structural and naming changes making the codebase generic with respect to the type of route being handled (e.g. `HTTPRoute` -> `Route`). Changes are largely cosmetic, with no _behavioral_ changes introduced by any functional refactoring.
Signed-off-by: Mark S <the@wondersmith.dev>
## Subject
Prepare to expand `linkerd`'s repertoire of supported [`Gateway API`](https://gateway-api.sigs.k8s.io/api-types/grpcroute/) route types in [`linkerd-policy-controller-k8s-status`](https://github.com/linkerd/linkerd2/tree/main/policy-controller/k8s/status).
## Problem
Currently, the policy controller's `status` component is written with `HTTPRoute` support (effectively) exclusively, both in its structure/organization as well as its naming (e.g. `HttpRoute` as a primary type name, `update_http_route` as a method name, etc...).
In order to expand `linkerd`'s support for the route types defined by the Gateway API, the policy controller's `status` component needs to be made more generic in both respects.
## Solution
> **NOTE:** PR was opened out of order and should only be merged _after_ #12662
PR introduces structural and naming changes making the codebase generic with respect to the type of route being handled (e.g. `HTTPRoute` -> `Route`). Changes are almost entirely cosmetic introducing only a couple of minor functional changes, most notably:
- making the `status` argument to [`make_patch`](8d6cd57b70/policy-controller/k8s/status/src/index.rs (L734)) generic
- adding a type-aware `api_version` helper method to [`NamespaceGroupKindName`](8d6cd57b70/policy-controller/k8s/status/src/resource_id.rs (L27))
- **note:** *required for proper handling of different route types in the future*
## Validation
- [X] maintainer review
- [X] tests pass
## ~~Fixes~~ *Lays Groundwork For Addressing*
- #12404
Signed-off-by: Mark S <the@wondersmith.dev>
If an HTTPRoute references a backend service that does not exist, the policy controller synthesizes a FailureInjector in the outbound policy so that requests to that backend will fail with a 500 status code. However, we do not update the policy when backend services are created or deleted, which can result in an outbound policy that synthesizes 500s for backends, even if the backend currently exists (or vice versa).
This is often papered over because when a backend service is created or deleted, this will trigger the HTTPRoute's ResolvedRef status condition to change which will cause a reindex of the HTTPRotue and a recomputation of the backends. However, depending on the specific order that these updates are processed, the outbound policy can still be left with the incorrect backend state.
In order to be able to update the backend of an outbound policy when backend services are created or deleted, we change the way these backends are represented in the index. Previously, we had represented backends which were services that did not exist as `Backend::Invalid`. However, this discards the necessary backend information necessary to recreate the backend if the service is created. Instead, we update this to represent these backends as a `Backend::Service` but with a new field `exists` set to false. This allows us to update this field as backend services are created or deleted.
Signed-off-by: Alex Leong <alex@buoyant.io>
ExternalWorkload resources represent as a resource configuration associated
with a process (or a group of processes) that are foreign to a Kubernetes
cluster. It allows Linkerd to read / write and store configuration for mesh
expansion. Since VMs will be able to receive inbound traffic from a variety of
resources, the proxy should be able to dynamically discover inbound
authorisation policies.
This change introduces a set of callbacks in the indexer that will apply (or
delete) ExternalWorkload resources. In addition, we ensure that
ExternalWorkloads can be processed in a similar fashion to pods (where
applicable, of course) wrt to server matching and defaulting. To serve
discovery requests for a VM, the policy controller will now also start a
watcher for external workloads and allow requests to reference an
`external_workload` target
A quick list of changes:
* ExternalWorkloads can now be indexed in the inbound (policy) index. Renamed
* the pod module in the inbound index to be more generic ("workload"); the
* module has some re-usable building blocks that we can use for external
* workloads. Moved common functions (e.g. building a default inbound server)
* around to share what's already been done without abstracting more or
* introducing generics. Changed gRPC target types to a tuple of `(Workload,
* port)` from a tuple of `(String, String, port)` Added RBAC to watch external
* workloads.
---------
Signed-off-by: Matei David <matei@buoyant.io>
Currently, the value that is put in the `LINKERD2_PROXY_POLICY_WORKLOAD` env var has the format of `pod_ns:pod_name`. This PR changes the format of the policy token into a json struct, so it can encode the type of workload and not only its location. For now, we add an additional `external_workload` type.
Zahari Dichev <zaharidichev@gmail.com>
The policy controller sets incorrect backend metadata when (1) there is
no explicit backend reference specified, and (2) when a backend
reference crosses namespaces.
This change fixes these backend references so that proxy logs and
metrics have the proper metadata references. Outbound policy tests are
updated to validate this.
The [xRoute Binding KEP](https://gateway-api.sigs.k8s.io/geps/gep-1426/#namespace-boundaries) states that HttpRoutes may be created in either the namespace of their parent Service (producer routes) or in the namespace of the client initiating requests to the service (consumer routes). Linkerd currently only indexes producer routes and ignores consumer routes.
We add support for consumer routes by changing the way that HttpRoutes are indexed. We now index each route by the namespace of its parent service instead of by the namespace of the HttpRoute resource. We then further subdivide the `ServiceRoutes` struct to have a watch per-client-namespace instead of just a single watch. This is because clients from different namespaces will have a different view of the routes for a service.
When an HttpRoute is updated, if it is a producer route, we apply that HttpRoute to watches for all of the client namespaces. If the route was a consumer route, then we only apply the HttpRoute to watches for that consumer namespace.
We also add API tests for consumer and producer routes.
A few noteworthy changes:
* Because the namespace of the client factors into the lookup, we had to change the discovery target to a type which includes the client namespace.
* Because a service may have routes from different namespaces, the route metadata now needs to track group, kind, name, AND namespace instead of just using the namespace of the service. This means that many uses of the `GroupKindName` type are replaced with a `GroupKindNamespaceName` type.
Signed-off-by: Alex Leong <alex@buoyant.io>
We add support for the RequestHeaderModifier and RequestRedirect HTTP filters. The policy controller reads these filters in any HttpRoute resource that it indexes (both policy.linkerd.io and gateway.networking.k8s.io) and returns them in the outbound policy API. These filters may be added at the route rule level and at the backend level.
We add outbound api tests for this behavior for both types of HttpRoute.
Incidentally we also fix a flaky test in the outbound api tests where a watch was being recreated partway through a test, leading to a race condition.
Signed-off-by: Alex Leong <alex@buoyant.io>
Updates the policy-controller to watch `httproute.gateway.networking.k8s.io` resources in addition to watching `httproute.policy.linkerd.io` resources. Routes of either or both types can be returned in policy responses and will be appropriately identified by the `group` field on their metadata. Furthermore we update the Status of these resources to correctly reflect when they are accepted.
We add the `httproute.gateway.networking.k8s.io` CRD to the Linkerd installed CRD list and add the appropriate RBAC to the policy controller so that it may watch these resources.
Signed-off-by: Alex Leong <alex@buoyant.io>
PR #10969 adds support for the GEP-1742 `timeouts` field to the
HTTPRoute CRD. This branch implements actual support for these fields in
the policy controller. The timeout fields are now read and used to set
the timeout fields added to the proxy-api in
linkerd/linkerd2-proxy-api#243.
In addition, I've added code to ensure that the timeout fields are
parsed correctly when a JSON manifest is deserialized. The current
implementation represents timeouts in the bindings as a Rust
`std::time::Duration` type. `Duration` does implement
`serde::Deserialize` and `serde::Serialize`, but its serialization
implementation attempts to (de)serialize it as a struct consisting of a
number of seconds and a number of subsecond nanoseconds. The timeout
fields are instead supposed to be represented as strings in the Go
standard library's `time.ParseDuration` format. Therefore, I've added a
newtype which wraps the Rust `std::time::Duration` and implements the
same parsing logic as Go. Eventually, I'd like to upstream the
implementation of this to `kube-rs`; see kube-rs/kube#1222 for details.
Depends on #10969
Depends on linkerd/linkerd2-proxy-api#243
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Controller currently serves hardcoded configuration values for failure
accrual parameters. This change adds support to discover configuration
on Service objects based on annotations.
Signed-off-by: Matei David <matei@buoyant.io>
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Alex Leong <alex@buoyant.io>
When an HTTPRoute references a backend service which does not exist, the outbound policy API returns an FailureInjector filter and no backend. However, since the proxy still needs to do queuing for this synthetic backend, we still need to send a queue config to the proxy.
We populate the queue config for this backend, even when it does not exist.
As an unrelated but nearby fix, we also populate metadata for service backends.
Signed-off-by: Alex Leong <alex@buoyant.io>
This branch makes a few very minor style tweaks to the OutboundPolicy
API server, primarily in the handling of authority strings in the
`lookup_authority` function. There should be no functional changes in
this branch.
In particular, this branch makes the following changes:
* use `trim_end_matches` instead of slicing when removing domain
suffixes
* `Option<&str>` implements `Eq`, so use that rather than `matches!`
* inline some variable names in format args
Currently, the OutboundPolicies API returns the gRPC status code
`InvalidArgument` when a target authority is valid but is not a
Kubernetes Service. In the proxy, the `NotFound` status is handled by
synthesizing a policy from a ServiceProfile, as it is used to indicate
that a target is potentially valid but not known to the policy
controller. Other gRPC status codes, such as `InvalidArgument`, are
treated as fatal, as they indicate that the proxy did something wrong.
Multicluster gateway proxies may perform lookups for Pod DNS
names (rather than Service DNS names), and those lookups should return
`NotFound`, so that the proxy knows to synthesize a policy for that
target, rather than failing after receiving `InvalidArgument`.
This branch changes the OutboundPolicies API to return `NotFound` when a
target authority is a valid DNS name that is not a Service.
This change improves the modularity of the policy controller by splitting code related in inbound policy and outbound policy into separate modules. There are no functional changes in this PR, only moves and renames. We have moved any code which was used by both inbound and outbound into common modules so that there are no cross dependencies between inbound and outbound.
Additionally, several types that had names beginning with "Inbound" or "Outbound" have had this prefix removed since they are now distinguishable by their modules. e.g. `InboundHttpRoute` becomes `inbound::HttpRoute`.
Signed-off-by: Alex Leong <alex@buoyant.io>
When the outbound policy API was returning backends which corresponded to Service resources, those backends were missing metadata information.
We update the policy controller to populate Service metadata for those backends.
Signed-off-by: Alex Leong <alex@buoyant.io>
Proxies in ingress and gateway configurations may discover policies by
name instead of network address.
This change updates the policy controller to parse service names from
lookups.
The outbound policy controller never sets backends for default opaque
routes.
This change ensures these routes are configured. Furthermore, default
metadata names are updated to be more descriptive.
We have a `ClusterInfo` type that includes all cluster-level
configuration. The outbound indexer does not use this type and, instead,
passes around individual configuration values.
This change updates the outbound indexer to use the ClusterInfo
configuration type. It also moves the type to its own module.
This is largely a reorganization PR, though there is one notable change:
InvalidDst response no longer reference a backend. Instead, the backend
is left empty, since it's not possible to route requests to it.
Implement the outbound policy API as defined in the proxy api: https://github.com/linkerd/linkerd2-proxy-api/blob/main/proto/outbound.proto
This API is consumed by the proxy for the routing of outbound traffic. It is intended to replace the GetProfile API which is currently served by the destination controller. It has not yet been released in a proxy-api release, so we take a git dependency on it in the mean time.
This PR adds a new index to the policy controller which indexes HTTPRoutes and Services and uses this information to serve the outbound API. We also add outbound API tests to validate the behavior of this implementation.
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
This branch updates the policy controller's dependency on `tonic` to
v0.8, and `linkerd2-proxy-api` to 0.7. `tonic` 0.8 is a breaking API
change, and `tonic` and `linkerd2-proxy-api` must be updated together,
rather than separately, so this cannot be done by Dependabot.
Closes#9078
This changes updates the policy controller's indexer to add default, unauthenticated routes for
endpoints referenced in a Pod's readiness/liveness/startup probe configuration. These default routes
are included when:
1. the policy controller is configured with a list of networks from which probes may originate; and
2. no other routes are configured for the server.
If a user defines routes for a Server, then they must also explicitly account for probe endpoints.
An e2e test has been added which asserts the following:
1. When no Server is configured for a Pod:port, the probe routes are authorized.
2. When a Server is configured, but there are no routes, the probe routes are still authorized.
3. When a route is configured for the Server, the probe routes are no longer authorized by default.
Related to #8961#8945
Signed-off-by: Kevin Leimkuhler <kleimkuhler@icloud.com>
Co-authored-by: Oliver Gould <ver@buoyant.io>
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
Currently, if no `HTTPRoute` resources reference a `Server`, the policy
controller returns an empty list of routes. This results in the proxy
404ing HTTP traffic. Instead, we should return a default empty route in
this case.
This branch changes the policy controller to return a default route for
`Server`s which are not referenced by any `HTTPRoute`s. The default
route defines a single `HttpRouteRule` with no matches and no filters,
so it will match any HTTP request. The default route does *not* define
any authorizations, as they would potentially clobber authorizations
defined by other resources that define authorization policies targeting
that `Server` --- if it is not targeted by any other resource defining
authorization policies, the `Server` itself will still get the default
authz policy.
In addition, this branch changes the various `${Resource}Ref` enums so
that their `Default` variants take a `&'static str` rather than a
`String`, to minimize string copying, since the default names are always
pre-defined. It also adds an `InboundHttpRouteRef` type which is used as
a key in the maps of `HTTPRoute`s (instead of `String`s), since we now
define default routes as well.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
When there are multiple equivalent routes (e.g., two routes with the
same match), the proxy will use the first route in the returned list. We
need to ensure that the policy controller returns routes in a
deterministic order--and the Gateway API defines such an order:
> If ties still exist across multiple Routes, matching precedence MUST
> be determined in order of the following criteria, continuing on ties:
>
> * The oldest Route based on creation timestamp.
> * The Route appearing first in alphabetical order by
> "{namespace}/{name}".
This branch updates the policy controller to return the list of
`HttpRoute`s for an inbound server with a deterministic ordering based
on these rules. This is done by tracking the creation timestamp for
indexed `HTTPRoute` resources, and sorting the list of protobuf
`HttpRoute`s when the API server constructs an `InboundServer` response.
The implementation is *somewhat* hairy, because we can't just define a
custom `Ord` implementation for the protobuf `HttpRoute` type that
includes the timestamp --- doing so would require actually storing the
creation timestamp in the protobuf type, which would be a change in
`linkerd2-proxy-api` (and would result in serializing additional
information that the proxy itself doesn't actually care about). Instead,
we use `slice::sort_by` with a closure that looks up routes by name in
the hash map stored by the indexer in order to determine their
timestamps, and implements a custom ordering that first compares the
timestamp, and falls back to comparing the route's name if the
timestamps are equal. Note that we don't include the namespace in that
comparison, because all the routes for a given `InboundServer` are
already known to be in the same namespace.
I've also added an end-to-end test that the API returns the route list
in the correct order. Unfortunately, this test has 4 seconds of `sleep`s
in it, because the minimum resolution of Kubernetes creation timestamps
is 1 second. I figured a test that takes five or six seconds to run was
probably not a huge deal in the end to end tests --- some of the policy
tests take as long as a minute to run, at least on my machine.
Closes#8946
As discussed in #8944, Linkerd's current use of the
`gateway.networking.k8s.io` `HTTPRoute` CRD is not a spec-compliant use
of the Gateway API, because we don't support some "core" features of the
Gateway API that don't make sense in Linkerd's use-case. Therefore,
we've chosen to replace the `gateway.networking.k8s.io` `HTTPRoute` CRD
with our own `HTTPRoute` CRD in the `policy.linkerd.io` API group, which
removes the unsupported features.
PR #8949 added the Linkerd versions of those CRDs, but did not remove
support for the Gateway API CRDs. This branch removes the Gateway API
CRDs from the policy controller and `linkerd install`/Helm charts.
The various helper functions for converting the Gateway API resource
binding types from `k8s-gateway-api` to the policy controller's internal
representation is kept in place, but the actual use of that code in the
indexer is disabled. This way, we can add support for the Gateway API
CRDs again easily. Similarly, I've kept the validation code for Gateway
API types in the policy admission controller, but the admission
controller no longer actually tries to validate those resources.
Depends on #8949Closes#8944
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This change updates the policy controller to admit `AuthorizationPolicy` resources
that reference `HTTPRoute` parents. These policies configure proxies to augment
server-level authorizations with per-route authorizations.
Fixes#8890
Signed-off-by: Alex Leong <alex@buoyant.io>
In various places we read port configurations from external sources
(either the Kubernetes API or gRPC clients). We have manual checks in
place to ensure that port values are never zero. We can instead assert
this with the type system by using `NonZeroU16`.
This change updates the policy controller to use `NonZeroU16` for port
values. This allows us to replace our manual port value checks with
`NonZero::try_from`, etc.
Signed-off-by: Oliver Gould <ver@buoyant.io>
linkerd2-proxy-api v0.6.0 adds support for inbound proxies to discover
route configurations based on the Gateway API HTTPRoute types. This
change updates the policy controller to index
`gateway.networking.k8s.io/v1beta` `HTTPRoute` types to discover these
policies from the Kubernetes API.
`HTTPRoute` resources may target `Server` resources (as a `parentRef`)
to attach policies to an inbound proxy. When no routes are configured,
a default route is synthesized to allow traffic; but when at least one
route attaches to a server, only requests that match a route are
permitted (other requests are failed with a 404).
Only the *core* subset of the `HTTPRoute` filters are supported:
`RequestRedirect` and `RequestHeaderModifier`. Backends may *not* be
configured on these routes (since they may only apply to inbound/server-
side proxies). No `status` updates are currently performed on these
`HTTPRoute` resources.
This change does not yet allow `AuthorizationPolicy` resources to target
`HTTPRoute` resources. This will be added in a follow-up change.
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
For the sake of completeness, we should set a `group` label in addition
to `kind`. This makes it possible to differentiate & discover resource
types handled by the proxy.
The `group` value is empty when a response does not refer to an actual
resource--usually when the `kind` is _default_.
Signed-off-by: Oliver Gould <ver@buoyant.io>
3f2061d updated the repo to use `linkerd2-proxy-api` v0.4.0 with `tonic`
v0.7.0; but this change does not work properly with ARM builds, as there
no `protoc` binary available on this platform. Now that
`linkerd2-proxy-api` includes statically-generated protobuf bindings,
the policy controller no longer needs protoc at build-time.
This change also removes the use of the `tonic/transport` feature, since
it has some annoying dependency implications. Instead we build a simple
`hyper` server as tonic would.
Signed-off-by: Oliver Gould <ver@buoyant.io>
This change updates the repo to use `linkerd2-proxy-api` v0.4.0 and
updates `bin/protoc` to use v3.20 to match the configuration in the
other repo.
The policy-controller builds are updated to use our `bin/protoc` wrapper
so that all builds go through the same toolchain (and to avoid compiling
protoc on each build).
Signed-off-by: Oliver Gould <ver@buoyant.io>
Issue #7709 proposes new Custom Resource types to support generalized
authorization policies:
- `AuthorizationPolicy`
- `MeshTLSAuthentication`
- `NetworkAuthentication`
This change introduces these CRDs to the default linkerd installation
(via the `linkerd-crds` chart) and updates the policy controller's
to handle these resource types. The policy admission controller
validates that these resource reference only suppported types.
This new functionality is tested at multiple levels:
* `linkerd-policy-controller-k8s-index` includes unit tests for the
indexer to test how events update the index;
* `linkerd-policy-test` includes integration tests that run in-cluster
to validate that the gRPC API updates as resources are manipulated;
* `linkerd-policy-test` includes integration tests that exercise the
admission controller's resource validation; and
* `linkerd-policy-test` includes integration tests that ensure that
proxies honor authorization resources.
This change does NOT update Linkerd's control plane and extensions to
use these new authorization primitives. Furthermore, the `linkerd` CLI
does not yet support inspecting these new resource types. These
enhancements will be made in followup changes.
Signed-off-by: Oliver Gould <ver@buoyant.io>
The policy controller's indexing module spans several files and relies
on an unnecessarily complex double-watch. It's generally confusing and
therefore difficult to change.
This change attempts to simplify the logic somewhat:
* All of the indexing code is now in the
`linkerd_policy_controller_k8s_index::index` module. No other files
have any dependencies on the internals of this data structure. It
exposes one public API, `Index::pod_server_rx`, used by discovery
clients.
* It uses the new `kubert::index` API so that we can avoid redundant
event-handling code. We now let kubert drive event processing so that
our indexing code is solely responsible for updating per-port server
configurations.
* A single watch is maintained for each pod:port.
* Watches are constructed lazily. The policy controller no longer
requires that all ports be documented on a pod. (The proxy still
requires this, however). This sets up for more flexible port
discovery.
Signed-off-by: Oliver Gould <ver@buoyant.io>
The identity-string parsing logic is currently implemented directly in
the ServerAuthorization indexer. In preparation of this being needed in
additional modules, this change extracts identity parsing and adds some
basic tests.
Signed-off-by: Oliver Gould <ver@buoyant.io>
Various development tools (including Rust Analyzer and some reusable
actions) expect the root of the project to define a Cargo workspace.
In order to work more naturally with these tools, this change moves the
`Cargo.lock`, `rust-toolchain`, and `deny.toml` files to the root of the
project. A `Cargo.toml` is factored out of `policy-controller` to define
the top-level workspace.
Policy controller API responses include a set of labels. These labels
are to be used in proxy m$etrics to indicate why traffic is permitted to
a pod. This permits metrics to be associated with `Server` and
ServerAuthorization` resources (i.e. for `stat`).
This change updates the response API to include a `name` label
referencing the server's name. When the policy is derived from a default
configuration (and not a `Server` instance), the name takes the form
'default:<policy>'.
This change also updates authorization labels. Defaults are encoded as
servers are, otherwise the authorization's name is set as a label. The
`tls` and `authn` labels have been removed, as they're redundant with
other labels that are already present.
Mark S