GEP-1426 specifies that if a GrpcRoute and an HttpRoute both attempt to attach to the same parent, the HttpRoute should be ignored and have the Accepted condition set to false with the reason `RouteReasonConflicted`.
https://gateway-api.sigs.k8s.io/geps/gep-1426/#route-types
We update the status controller and outbound policy index to implement this behavior.
Signed-off-by: Alex Leong <alex@buoyant.io>
We update the policy-controller to support grpcroutes in the outbound policy API. When a GrpcRoute resource has a Service as a parent ref, outbound policy requests for that service may return a proxy protocol of grpc with grpc routes.
* if a service has no HttpRoute or GrpcRoutes as children, we continue to return the default http route with a proxy protocol of Detect (so that the proxy is directed to detect Http1 vs Http2)
* similarly, if a service has HttpRoute children only, we continue to return those routes with a proxy protocol of Detect
* if a service has GrpcRoute children only, we return those routes with a proxy protocol of Grpc
* if a service has both types of routes as children, we determine the proxy protocol based on which route type has the oldest created_at timestamp as described in https://gateway-api.sigs.k8s.io/geps/gep-1016/#cross-serving and only return routes of the determined type
Signed-off-by: Alex Leong <alex@buoyant.io>
We add support for GrpcRoute resources in the policy-controller's status controller. This means that the policy controller will watch GrpcRoute resources in the cluster and keep their status up to date, in the same way that it currently does for HttpRoute resources.
Signed-off-by: Alex Leong <alex@buoyant.io>
## Subject
Prepare to expand the subset of [`Gateway API`](https://gateway-api.sigs.k8s.io/api-types/grpcroute/) route types `linkerd` supports driving outbound traffic with in [`linkerd-policy-controller-k8s-index`](https://github.com/linkerd/linkerd2/tree/main/policy-controller/k8s/status).
## Problem
Currently, the policy controller's `index` component is written with `HTTPRoute` support (effectively) exclusively, both in its structure/organization as well as its naming (e.g. `HttpRoute` as a primary type name, `convert_http_route` as a method name, etc...).
In order to expand the subset of route types defined by the Gateway API that `linkerd` supports for driving outbound traffic policy, the policy controller's `index` component needs to be made more generic in both respects.
## Solution
PR introduces structural and naming changes making the codebase generic with respect to the type of route being handled (e.g. `HTTPRoute` -> `Route`). Changes are largely cosmetic, with no _behavioral_ changes introduced by any functional refactoring.
Signed-off-by: Mark S <the@wondersmith.dev>
## Subject
Prepare to expand `linkerd`'s repertoire of supported [`Gateway API`](https://gateway-api.sigs.k8s.io/api-types/grpcroute/) route types in [`linkerd-policy-controller-k8s-status`](https://github.com/linkerd/linkerd2/tree/main/policy-controller/k8s/status).
## Problem
Currently, the policy controller's `status` component is written with `HTTPRoute` support (effectively) exclusively, both in its structure/organization as well as its naming (e.g. `HttpRoute` as a primary type name, `update_http_route` as a method name, etc...).
In order to expand `linkerd`'s support for the route types defined by the Gateway API, the policy controller's `status` component needs to be made more generic in both respects.
## Solution
> **NOTE:** PR was opened out of order and should only be merged _after_ #12662
PR introduces structural and naming changes making the codebase generic with respect to the type of route being handled (e.g. `HTTPRoute` -> `Route`). Changes are almost entirely cosmetic introducing only a couple of minor functional changes, most notably:
- making the `status` argument to [`make_patch`](8d6cd57b70/policy-controller/k8s/status/src/index.rs (L734)) generic
- adding a type-aware `api_version` helper method to [`NamespaceGroupKindName`](8d6cd57b70/policy-controller/k8s/status/src/resource_id.rs (L27))
- **note:** *required for proper handling of different route types in the future*
## Validation
- [X] maintainer review
- [X] tests pass
## ~~Fixes~~ *Lays Groundwork For Addressing*
- #12404
Signed-off-by: Mark S <the@wondersmith.dev>
If an HTTPRoute references a backend service that does not exist, the policy controller synthesizes a FailureInjector in the outbound policy so that requests to that backend will fail with a 500 status code. However, we do not update the policy when backend services are created or deleted, which can result in an outbound policy that synthesizes 500s for backends, even if the backend currently exists (or vice versa).
This is often papered over because when a backend service is created or deleted, this will trigger the HTTPRoute's ResolvedRef status condition to change which will cause a reindex of the HTTPRotue and a recomputation of the backends. However, depending on the specific order that these updates are processed, the outbound policy can still be left with the incorrect backend state.
In order to be able to update the backend of an outbound policy when backend services are created or deleted, we change the way these backends are represented in the index. Previously, we had represented backends which were services that did not exist as `Backend::Invalid`. However, this discards the necessary backend information necessary to recreate the backend if the service is created. Instead, we update this to represent these backends as a `Backend::Service` but with a new field `exists` set to false. This allows us to update this field as backend services are created or deleted.
Signed-off-by: Alex Leong <alex@buoyant.io>
An HTTPRoute whose parentRef is a Service in the same namespace is called a producer route. Producer routes should be used in outbound policy by all clients calling that Service, even if the client is in a different namespace. The policy controller has a bug where when a outbound policy watch is started, the initial outbound policy returned will not include any producer routes which already exist.
We correct this bug and add tests.
Signed-off-by: Alex Leong <alex@buoyant.io>
The policy controller sets a status on HttpRoute resources depending on the state of the backends. If all the backends can be resolved, it should set a resolved_ref status and if any of the backends cannot be resolved, it should set a backend_not_found status.
We update the logic to reflect this. Additionally, this resolves the edge case where an empty list of backends was resulting in a backend_not_found status instead of a resolved_refs status.
Signed-off-by: Alex Leong <alex@buoyant.io>
Adds index metrics to the outbound policy index.
```
# HELP outbound_index_service_index_size The number of entires in service index
# TYPE outbound_index_service_index_size gauge
outbound_index_service_index_size 20
# HELP outbound_index_service_info_index_size The number of entires in the service info index
# TYPE outbound_index_service_info_index_size gauge
outbound_index_service_info_index_size 23
# HELP outbound_index_service_route_index_size The number of entires in the service route index
# TYPE outbound_index_service_route_index_size gauge
outbound_index_service_route_index_size{namespace="kube-system"} 0
outbound_index_service_route_index_size{namespace="cert-manager"} 0
outbound_index_service_route_index_size{namespace="default"} 0
outbound_index_service_route_index_size{namespace="linkerd"} 0
outbound_index_service_route_index_size{namespace="emojivoto"} 0
outbound_index_service_route_index_size{namespace="linkerd-viz"} 0
# HELP outbound_index_service_port_route_index_size The number of entires in the service port route index
# TYPE outbound_index_service_port_route_index_size gauge
outbound_index_service_port_route_index_size{namespace="kube-system"} 0
outbound_index_service_port_route_index_size{namespace="cert-manager"} 0
outbound_index_service_port_route_index_size{namespace="default"} 1
outbound_index_service_port_route_index_size{namespace="linkerd"} 0
outbound_index_service_port_route_index_size{namespace="emojivoto"} 3
outbound_index_service_port_route_index_size{namespace="linkerd-viz"} 0
```
Signed-off-by: Alex Leong <alex@buoyant.io>
We add index size gauges for each of the resource indexes in the inbound policy index.
```
# HELP inbound_index_meshtls_authentication_index_size The number of MeshTLS authentications in index
# TYPE inbound_index_meshtls_authentication_index_size gauge
inbound_index_meshtls_authentication_index_size{namespace="linkerd-viz"} 1
# HELP inbound_index_network_autnetication_index_size The number of Network authentications in index
# TYPE inbound_index_network_autnetication_index_size gauge
inbound_index_network_autnetication_index_size{namespace="linkerd-viz"} 2
# HELP inbound_index_pod_index_size The number of pods in index
# TYPE inbound_index_pod_index_size gauge
inbound_index_pod_index_size{namespace="linkerd"} 3
inbound_index_pod_index_size{namespace="emojivoto"} 4
inbound_index_pod_index_size{namespace="linkerd-viz"} 5
# HELP inbound_index_external_workload_index_size The number of external workloads in index
# TYPE inbound_index_external_workload_index_size gauge
inbound_index_external_workload_index_size{namespace="linkerd"} 0
inbound_index_external_workload_index_size{namespace="emojivoto"} 0
inbound_index_external_workload_index_size{namespace="linkerd-viz"} 0
# HELP inbound_index_server_index_size The number of servers in index
# TYPE inbound_index_server_index_size gauge
inbound_index_server_index_size{namespace="linkerd"} 0
inbound_index_server_index_size{namespace="emojivoto"} 1
inbound_index_server_index_size{namespace="linkerd-viz"} 4
# HELP inbound_index_server_authorization_index_size The number of server authorizations in index
# TYPE inbound_index_server_authorization_index_size gauge
inbound_index_server_authorization_index_size{namespace="linkerd"} 0
inbound_index_server_authorization_index_size{namespace="emojivoto"} 0
inbound_index_server_authorization_index_size{namespace="linkerd-viz"} 0
# HELP inbound_index_authorization_policy_index_size The number of authorization policies in index
# TYPE inbound_index_authorization_policy_index_size gauge
inbound_index_authorization_policy_index_size{namespace="linkerd"} 0
inbound_index_authorization_policy_index_size{namespace="emojivoto"} 0
inbound_index_authorization_policy_index_size{namespace="linkerd-viz"} 4
# HELP inbound_index_http_route_index_size The number of HTTP routes in index
# TYPE inbound_index_http_route_index_size gauge
inbound_index_http_route_index_size{namespace="linkerd"} 0
inbound_index_http_route_index_size{namespace="emojivoto"} 600
inbound_index_http_route_index_size{namespace="linkerd-viz"} 0
```
Signed-off-by: Alex Leong <alex@buoyant.io>
ff5e485 added status controller metrics. The metric that counts the number of
status patches was mistakenly named "patchs" instead of "patches."
This commit renames the metric to "patches".
Co-authored-by: Mark S <the@wondersmith.dev>
This changes the policy controller service indexer to index
services by thier `spec.clusterIPs` instead of `spec.clusterIP`, to
account for dual-stack services that hold both an IPv4 and an IPv6
address in `spec.clusterIPs`.
This change is fully backwards-compatible.
This assists the ongoing effort to support IPv6/dual-stack in linkerd,
but doesn't imply full support yet.
The policy controller uses a queue to hold HttpRoute status patches before they are applied to the Kube API. In order to have better observaility into the state of this queue, we add metrics to count enqueues, dequeues, drops, and overflows.
```
# HELP resource_status_patch_dequeues_total Counter of patches dequeued from the updates channel.
# TYPE resource_status_patch_dequeues_total counter
resource_status_patch_dequeues_total_total 1430
# HELP resource_status_patch_drops_total Counter of patches dropped because we are not the leader.
# TYPE resource_status_patch_drops_total counter
resource_status_patch_drops_total_total 0
# HELP resource_status_patch_enqueues_total Counter of patches enqueued to the updates channel.
# TYPE resource_status_patch_enqueues_total counter
resource_status_patch_enqueues_total_total 1430
# HELP resource_status_patch_channel_full_total Counter of patches dropped because the updates channel is full.
# TYPE resource_status_patch_channel_full_total counter
resource_status_patch_channel_full_total_total 0
```
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
When the policy controller updates the status of an HttpRoute resource, we currently have little observability into if those updates are failing or how long they are taking. We also have no timeout in place to protect the policy controller from extremely slow or hanging status update requests.
We add a generous 5 second timeout for these API calls and add metrics to track success, failures, timeouts, and duration.
```
# HELP resource_status_patch_succeeded_total Counter patches successfully applied to HTTPRoutes.
# TYPE resource_status_patch_succeeded_total counter
resource_status_patch_succeeded_total_total 1711
# HELP resource_status_patch_failed_total Counter patches that fail to apply to HTTPRoutes.
# TYPE resource_status_patch_failed_total counter
resource_status_patch_failed_total_total 0
# HELP resource_status_patch_timeout_total Counter patches that time out when applying to HTTPRoutes.
# TYPE resource_status_patch_timeout_total counter
resource_status_patch_timeout_total_total 0
# HELP resource_status_patch_duration_seconds Histogram of time taken to apply patches to HTTPRoutes.
# TYPE resource_status_patch_duration_seconds histogram
resource_status_patch_duration_seconds_sum 8.930499397
resource_status_patch_duration_seconds_count 1711
resource_status_patch_duration_seconds_bucket{le="0.01"} 1656
resource_status_patch_duration_seconds_bucket{le="0.025"} 1694
resource_status_patch_duration_seconds_bucket{le="0.05"} 1707
resource_status_patch_duration_seconds_bucket{le="0.1"} 1710
resource_status_patch_duration_seconds_bucket{le="0.25"} 1711
resource_status_patch_duration_seconds_bucket{le="0.5"} 1711
resource_status_patch_duration_seconds_bucket{le="1.0"} 1711
resource_status_patch_duration_seconds_bucket{le="2.5"} 1711
resource_status_patch_duration_seconds_bucket{le="5.0"} 1711
resource_status_patch_duration_seconds_bucket{le="+Inf"} 1711
```
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
Fixes#12104
The leader of the policy controllers (as determined by a lease resource) is in charge of keeping the HttpRoute resource statuses up to date. To accomplish this, it performs reconciliation every 10 seconds where it computes the desired status for each HttpRoute and patches the resource. When there are many HttpRoute resources in the cluster, most of which change infrequently, this results in a large number of redundant no-op patches: one per HttpRoute every 10 seconds.
These patches are stored in an in-memory unbounded queue while waiting to be dispatched to the Kube API. If the rate that these patches are generated exceeds the rate that the Kube API can process them (synchronously) then this queue will grow without bound, resulting in a memory leak.
To address this we do two things:
* Use a bounded queue of size 10000. If the queue fills up, patches will be dropped. Statuses will be reconciled in a later iteration of the reconciliation loop once there is queue capacity.
* Only generate a patch when the desired status differs from the current status. This greatly reduces the number of patches generated.
Signed-off-by: Alex Leong <alex@buoyant.io>
PR https://github.com/linkerd/linkerd2/pull/12088 fixed an issue where removing and then re-adding certain policy resources could leave the policy index in an incorrect state. We add a test for the specific condition that triggered this behavior to prevent against future regressions.
Verified that this test fails prior to https://github.com/linkerd/linkerd2/pull/12088 but passes on main.
Signed-off-by: Alex Leong <alex@buoyant.io>
The ExternalWorkload resource we introduced has a minor naming
inconsistency; `Tls` in `meshTls` is not capitalised. Other resources
that we have (e.g. authentication resources) capitalise TLS (and so does
Go, it follows a similar naming convention).
We fix this in the workload resource by changing the field's name and
bumping the version to `v1beta1`.
Upgrading the control plane version will continue to work without
downtime. However, if an existing resource exists, the policy controller
will not completely initialise. It will not enter a crashloop backoff,
but it will also not become ready until the resource is edited or
deleted.
Signed-off-by: Matei David <matei@buoyant.io>
In the inbound policy index, we maintain a policy index per namespace which holds various policy resources for that namespace. When a per namespace index becomes empty, we remove it. However, we were not considering authorization policy resources when determining if the index is empty. This could result in the index being removed even while it contained authorization policy resources, as long as all other resource types did not exist.
This can lead to incorrect inbound policy responses when the per namespace index is recreated, since it will not longer contain the authorization policy.
We update the `is_empty()` function to properly consider authorization policies as well. We also add some generally useful logging at debug and trace level.
Signed-off-by: Alex Leong <alex@buoyant.io>
Co-authored-by: Oliver Gould <ver@buoyant.io>
ExternalWorkload resources require that status condition has almost all of its
fields set (with the exception of a date field). The original inspiration for
this design was the HTTPRoute object.
When using the resource, it is more practical to handle many of the fields as
optional; it is cumbersome to fill out the fields when creating an
ExternalWorkload. We change the settings to be in-line with a [Pod] object
instead.
[Pod]:
7d1a2f7a73/core/v1/types.go (L3063-L3084)
---------
Signed-off-by: Matei David <matei@buoyant.io>
ExternalWorkload resources represent as a resource configuration associated
with a process (or a group of processes) that are foreign to a Kubernetes
cluster. It allows Linkerd to read / write and store configuration for mesh
expansion. Since VMs will be able to receive inbound traffic from a variety of
resources, the proxy should be able to dynamically discover inbound
authorisation policies.
This change introduces a set of callbacks in the indexer that will apply (or
delete) ExternalWorkload resources. In addition, we ensure that
ExternalWorkloads can be processed in a similar fashion to pods (where
applicable, of course) wrt to server matching and defaulting. To serve
discovery requests for a VM, the policy controller will now also start a
watcher for external workloads and allow requests to reference an
`external_workload` target
A quick list of changes:
* ExternalWorkloads can now be indexed in the inbound (policy) index. Renamed
* the pod module in the inbound index to be more generic ("workload"); the
* module has some re-usable building blocks that we can use for external
* workloads. Moved common functions (e.g. building a default inbound server)
* around to share what's already been done without abstracting more or
* introducing generics. Changed gRPC target types to a tuple of `(Workload,
* port)` from a tuple of `(String, String, port)` Added RBAC to watch external
* workloads.
---------
Signed-off-by: Matei David <matei@buoyant.io>
We introduced an ExternalWorkload CRD along with bindings for mesh
expansion. Currently, the CRD allows users to create ExternalWorkload
resources without adding a meshTls strategy.
This change adds some more validation restrictions to the CRD definition
(i.e. server side validation). When a meshTls strategy is used, we
require both identity and serverName to be present. We also mark meshTls
as the only required field in the spec. Every ExternalWorkload regardless
of the direction of its traffic must have it set.
WorkloadIPs and ports now become optional to allow resources to be
created only to configure outbound discovery (VM to workload)
and inbound policy discovery (VM).
---------
Signed-off-by: Matei David <matei@buoyant.io>
This PR adds the ability for a `Server` resource to select over `ExternalWorkload`
resources in addition to `Pods`. For the time being, only one of these selector types
can be specified. This has been realized via incrementing the version of the resource
to `v1beta2`
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
We introduced an ExternalWorkload CRD for mesh expansion. This change
follows up by adding bindings for Rust and Go code.
For Go code:
* We add a new schema and ExternalWorkload types
* We also update the code-gen script to generate informers
* We add a new informer type to our abstractions built on-top of
client-go, including a function to check if a client has access to the
resource.
For Rust code:
* We add ExternalWorkload bindings to the policy controller.
---------
Signed-off-by: Matei David <matei@buoyant.io>
* kube 0.87.1
* k8s-openapi 0.20.0
* kubert 0.21.1
* k8s-gateway-api 0.15
* ring 0.17
Furthermore, the policy controller's metrics endpoint has been updated
to include tokio runtime metrics.
The policy controller sets incorrect backend metadata when (1) there is
no explicit backend reference specified, and (2) when a backend
reference crosses namespaces.
This change fixes these backend references so that proxy logs and
metrics have the proper metadata references. Outbound policy tests are
updated to validate this.
New versions of the k8s-openapi crate drop support for Kubernetes 1.21.
Kubernetes v1.22 has been considered EOL by the upstream project since
2022-07-08. Major cloud providers have EOL'd it as well (GKE's current
MSKV is 1.24).
This change updates the MSKV to v1.22. It also updates the max version
in _test-helpers.sh to v1.28.
Fixes#11659
When the policy controller updates the status of an HttpRoute, it will override any existing statuses on that resource.
We update the policy controller to take into account any statuses which are not controlled by Linkerd already on the resource. It will patch the final statuses to be the combination of thee non-Linkerd statuses plus the Linkerd statuses.
Signed-off-by: Alex Leong <alex@buoyant.io>
* Add native sidecar support
Kubernetes will be providing beta support for native sidecar containers in version 1.29. This feature improves network proxy sidecar compatibility for jobs and initContainers.
Introduce a new annotation config.alpha.linkerd.io/proxy-enable-native-sidecar and configuration option Proxy.NativeSidecar that causes the proxy container to run as an init-container.
Fixes: #11461
Signed-off-by: TJ Miller <millert@us.ibm.com>
* Update dev to v42
* Update Go to 1.21.3
* Update Rust to 1.73.0
* Update the Cargo workspace to use the v2 package resolver
* Update debian from bullseye to bookworm
* Update golangci-lint to 1.55.1
* Disable deprecated linters (deadcode, varcheck)
* Disable goconst linter -- pointless and noisy
* Disable depguard linter -- it requires that all of our Go dependencies be added to allowlists;
* Update K3d to v5.6.0
* Update CI from k3s 1.26 to 1.28
* Update markdownlint-cli2 to 0.10.0
When the policy controller's status index detects the deletion of a gateway API HTTPRoute, it attempts to delete that resource out of its own index. However, we use the wrong kind in the key when deleting. This results in the resource persisting in the index after it has been deleted from the cluster, which causes an error to be logged every 10 seconds when the policy controller attempts to do reconciliation and ensure that the statuses of all HTTPRoutes in its index are correct:
```
2023-10-09T20:53:17.059098Z ERROR status::Controller: linkerd_policy_controller_k8s_status::index: Failed to patch HTTPRoute namespace=linkerd-policy-test-sev0n7 route=GroupKindName { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "test" } error=ApiError: httproutes.gateway.networking.k8s.io "test" not found: NotFound (ErrorResponse { status: "Failure", message: "httproutes.gateway.networking.k8s.io \"test\" not found", reason: "NotFound", code: 404 })
```
To fix this, we use the correct kind when deleting from the index.
Signed-off-by: Alex Leong <alex@buoyant.io>
We intermittently see flaky policy integration test failures like:
```
failures:
either
thread 'either' panicked at 'assertion failed: `(left == right)`
left: `7`,
right: `0`: blessed uninjected curl must succeed', policy-test/tests/e2e_server_authorization.rs:293:9
```
This test failure is saying that the curl process is returning an exit code of 7 instead of the expected exit code of 0. This exit code indicates that curl failed to establish a connection. https://everything.curl.dev/usingcurl/returns
It's unclear why this connection occasionally fails in CI and I have not been able to reproduce this failure locally.
However, by looking at the logic of the integration test, we can see that the integration test creates the `web` Service and the `web` Pod and waits for that pod to become ready before unblocking the curl from executing. This means that, theoretically, there could be a race condition between the test and the kubernetes endpoints controller. As soon as the web pod becomes ready, the endpoints controller will update the endpoints resource for the `web` Service and at the same time, our test will unblock the curl command. If the test wins this race, it is possible that curl will run before the endpoints resource has been updated.
We add an additional wait condition to the test to wait until the endpoints resource has an endpoint before unblocking curl.
Since I could not reproduce the test failure locally, it is impossible to say if this is actually the cause of the flakiness or if this change fixes it.
Signed-off-by: Alex Leong <alex@buoyant.io>
This branch updates the policy-controller's dependency on Kubert to
v0.18, `kube-rs` to v0.85, `k8s-gateway-api` to v0.13, and `k8s-openapi`
to v0.19.
All of these crates depend on `kube-rs` and `k8s-openapi`, so they must
all be updated together in one commit. Therefore, this branch updates
all these dependencies.
dependabot[bot]