Use standard protoc for the dra instead of gogo.
Part of kubernetes#96564
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Kubernetes-commit: fd2d2fd66ff2fb34dc38711ce1c8b04a1bccd893
This change introduces the ability for the Kubelet to monitor and report
the health of devices allocated via Dynamic Resource Allocation (DRA).
This addresses a key part of KEP-4680 by providing visibility into
device failures, which helps users and controllers diagnose pod failures.
The implementation includes:
- A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources`
stream that DRA plugins can optionally implement.
- A health information cache within the Kubelet's DRA manager to track
the last known health of each device and handle plugin disconnections.
- An asynchronous update mechanism that triggers a pod sync when a
device's health changes.
- A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to
expose the device health information to users via the Pod API.
Update vendor
KEP-4680: Fix lint, boilerplate, and codegen issues
Add another e2e test, add TODO for KEP4680 & update test infra helpers
Add Feature Gate e2e test
Fixing presubmits
Fix var names, feature gating, and nits
Fix DRA Health gRPC API according to review feedback
Kubernetes-commit: b7de71f9ce74e99dde61ee138608df8edc5486bd
Use standard protoc for the kubelet plugin manager instead of gogo.
Part of https://github.com/kubernetes/kubernetes/issues/96564
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Kubernetes-commit: c889ee17a2f86e0357348b72c52bd36c7dbe8682
The API is a direct copy of v1beta1, with v1beta1 replaced by v1.
The interoperability support is the same that was used for v1alpha4.
Adding it "reverts" the removal of the v1alpha4 support in
a57f15e0816ecbef48f91b318c070070e7cbb84e, except that now v1beta1 is the legacy
API which needs conversion. If kubelet and the plugin both support v1, no
conversion is needed.
Kubernetes-commit: f141907ddd89998e821eb1047885722c8ba8922b
Use standard protoc for the device plugin API instead of gogo.
Part of kubernetes#96564
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Kubernetes-commit: 3026020b44a097a6f95a884e1e7f966c12675c0f
Use standard protoc for the pod resources instead of gogo.
Part of kubernetes#96564
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Kubernetes-commit: 532d48fe6adebcd4be43979ff189140f2d047f30
v1alpha4 was added in 1.31 and superseded by v1beta1 in 1.32. Since that
release, plugins are also required to advertise the supported gRPC services
during registration. In practice, all known DRA drivers use the helper code
from 1.32 or newer and thus don't need the legacy support.
Kubernetes-commit: a57f15e0816ecbef48f91b318c070070e7cbb84e
It's not okay to drop a claim from the response just because it encountered no
error. We want to be sure that a DRA driver really looked at the claim.
Kubernetes-commit: 005cef332da41990d08bee826262dd71fa74f522
The "// import <path>" comment has been superseded by Go modules.
We don't have to remove them, but doing so has some advantages:
- They are used inconsistently, which is confusing.
- We can then also remove the (currently broken) hack/update-vanity-imports.sh.
- Last but not least, it would be a first step towards avoiding the k8s.io domain.
This commit was generated with
sed -i -e 's;^package \(.*\) // import.*;package \1;' $(git grep -l '^package.*// import' | grep -v 'vendor/')
Everything was included, except for
package labels // import k8s.io/kubernetes/pkg/util/labels
because that package is marked as "read-only".
Kubernetes-commit: 8a908e0c0bd96a3455edf7e3b5f5af90564e65b0
As mentioned in https://protobuf.dev/programming-guides/style, package names
"should be unique". For generated API pb files, Kubernetes uses
"k8s.io.api.<api group>.<version>". The same approach is now used for the
kubelet DRA v1beta1 API. This can be changed without a breaking change because
the API is still unreleased.
This avoids a conflict between the DevicePlugin and the DRA API when both get
used by kubelet:
2024/11/10 12:51:40 proto: duplicate proto type registered: v1beta1.Device
The exact impact of that conflict is unknown but it seems better to
pro-actively avoid the problem.
Kubernetes-commit: 6c27e8dc8f789fd9daf11f6fc367adf586f727b5
Reusing types from the alpha in the beta made it possible to provide and use
both versions without conversion. The downside was that removal of the alpha
would have been harder, if not impossible. DRA drivers could continue to
use the alpha types and provided the beta interface automatically.
Now the two versions are completely separate gRPC APIs, although in practice
there are no differences besides the name. Support for the alpha API in kubelet
is provided via automatically generated conversion and manually written
interface wrappers.
Those are provided as part of the v1alpha4 package. The advantage of having all
of that in a central place is that it'll be easier to remove when no longer
needed.
Kubernetes-commit: 9261a182bb7693bc603a94e76af8bf98168ca16e
Listing supported gRPC services (e.g. drav1alpha3.Node, drav1beta1.DRAPlugin)
during registration enables the kubelet to determine in advance which methods
it can call.
Versioning by Kubernetes release makes less sense because it doesn't say
anything about which gRPC service is supported. New ones might get added and
obsolete ones removed. Some services might be optional.
In the past, this versioning support wasn't really used. At least one version
had to be provided and kubelet tried to use the plugin with the highest
version. This version comparison gets dropped. In the unlikely situation
that different plugins register under the same name, the most recent one is
used.
Because advertising gRPC services is a new convention, plugins only reporting
some version are treated as providing the old alpha gRPC service.
Kubernetes-commit: 2c23fe1b82467b0ed49ed8800c18334c4400132b
The version bump is an opportunity to pick a name that is a bit more
descriptive. It matches the "DevicePlugin" service name.
Kubernetes-commit: 437be1e651a6c5ff2259975c77af8575d2e66449
The v1beta1 API is identical to the previous v1alpha4, which erroneously was
still called "v1alpha3" in a few places, including the gRPC interface
definition itself.
The only reason for v1beta1 is to document the increased maturity of this API.
To simplify the transition, kubelet supports both v1alpha4 and v1beta1, picking
the more recent one automatically. All that DRA driver authors need to do to
implement v1beta1 is to update to the latest
k8s.io/dynamic-resource-allocation/kubeletplugin: it will automatically
register both API versions unless explicitly configured otherwise, which is
mostly just for testing.
DRA driver authors may replace their package import of v1alpha4 with v1beta1,
but they don't have to because the types in both packages are the same.
Kubernetes-commit: 7b3a9afca35cf60f4a5bc1d9cca5e25ff414d39d
Mentioning potential future API extensions help with understanding why the API
is defined the way it is when reading just the types.go. Those stand-alone
comments appear neither in the Go doc nor OpenAPI.
This corresponds to https://github.com/kubernetes/enhancements/pull/4808.
Kubernetes-commit: 148469d7ff00960bc13bd331009eec1db50122de
This adds the ability to select specific requests inside a claim for a
container.
NodePrepareResources is always called, even if the claim is not used by any
container. This could be useful for drivers where that call has some effect
other than injecting CDI device IDs into containers. It also ensures that
drivers can validate configs.
The pod resource API can no longer report a class for each claim because there
is no such 1:1 relationship anymore. Instead, that API reports claim,
API devices (with driver/pool/device as ID) and CDI device IDs. The kubelet
itself doesn't extract that information from the claim. Instead, it relies on
drivers to report this information when the claim gets prepared. This isolates
the kubelet from API changes.
Because of a faulty E2E test, kubelet was told to contact the wrong driver for
a claim. This was not visible in the kubelet log output. Now changes to the
claim info cache are getting logged. While at it, naming of variables and some
existing log output gets harmonized.
Co-authored-by: Oksana Baranova <oksana.baranova@intel.com>
Co-authored-by: Ed Bartosh <eduard.bartosh@intel.com>
Kubernetes-commit: 877829aeaa9b87d457bf1a3e59ae228e8aa1b1f0
This is the second and final step towards making kubelet independent of the
resource.k8s.io API versioning because it now doesn't need to copy structs
defined by that API from the driver to the API server.
Kubernetes-commit: 348f94ab55053ec9b4382de880406e27a730d8cd
This is a first step towards making kubelet independent of the resource.k8s.io
API versioning because it now doesn't need to copy structs defined by that API
from the driver to the API server. The next step is removing the other
direction (reading ResourceClaim status and passing the resource handle to
drivers).
The drivers must get deployed so that they have their own connection to the API
server. Securing at least the writes via a validating admission policy should
be possible.
As before, the kubelet removes all ResourceSlices for its node at startup, then
DRA drivers recreate them if (and only if) they start up again. This ensures
that there are no orphaned ResourceSlices when a driver gets removed while the
kubelet was down.
While at it, logging gets cleaned up and updated to use structured, contextual
logging as much as possible. gRPC requests and streams now use a shared,
per-process request ID and streams also get logged.
Kubernetes-commit: 616a01434756c4a818908a73b9256c9caa7d2c48
When renaming NodeResourceSlice to ResourceSlice, the embedded
[Node]ResourceModel also should have been renamed.
Kubernetes-commit: a0add8d2c7578cd9f94fc302d6212f9f7d16175b
The information is received from the DRA driver plugin through a new gRPC
streaming interface. This is backwards compatible with old DRA driver kubelet
plugins, their gRPC server will return "not implemented" and that can be
handled by kubelet. Therefore no API break is needed.
However, DRA drivers need to be updated because the Go API changed. They can
return
status.New(codes.Unimplemented, "no node resource support").Err()
if they don't support the new ListAndWatchResources method and
structured parameters.
The controller in kubelet then synchronizes this information from the driver
with NodeResourceSlice objects, creating, updating and deleting them as needed.
Kubernetes-commit: d59676a54531b6e135c0fbbe6b51c530f1150653
If the resource handle has data from a structured parameter model, then we need
to pass that to the DRA driver kubelet plugin. Because Kubernetes uses
gogo/protobuf, we cannot use "optional" for that new optional field and have to
resort to "repeated" with a single repetition if present.
This is a new, backwards-compatible field.
That extending the resource.k8s.io changes the checksum of a kubelet checkpoint
is unfortunate. Updating the test cases is a stop-gap measure, the actual
solution will have to be something else before beta.
Kubernetes-commit: 6f1ddfcd2e36c76b4e28dbb0ea355441cc4caeb2
This change adds CDI device IDs to the ContainerAllocateResponse in the
device plugin API. This allows a device plugin to specify CDI devices
by their unique fully-qualified CDI device names using the related field
in the CRI specification.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Kubernetes-commit: b57c7e2fe4bb466ff1614aa9df7cc164e90b24b6
Combining all prepare/unprepare operations for a pod enables plugins to
optimize the execution. Plugins can continue to use the v1beta2 API for now,
but should switch. The new API is designed so that plugins which want to work
on each claim one-by-one can do so and then report errors for each claim
separately, i.e. partial success is supported.
Kubernetes-commit: d743c50bb9e663809e0129ee058fabdcf59d3d27
This PR makes the NodePrepareResources() and NodeUnprepareResource()
calls of the kubeletplugin API for DynamicResourceAllocation
symmetrical. It wasn't clear how one would use the set of CDIDevices
passed back in the NodeUnprepareResource() of the v1alpha1 API, and the
new API now passes back the full ResourceHandle that was originally
passed to the Prepare() call. Passing the ResourceHandle is strictly
more informative and a plugin could always (re)derive the set of
CDIDevice from it.
This is a breaking change, but this release is scheduled to break
multiple APIs for DynamicResourceAllocation, so it makes sense to do
this now instead of later.
Signed-off-by: Kevin Klues <kklues@nvidia.com>
Kubernetes-commit: 579295e727a12deadad9e084ff8efd2708707091
Each of these scripts is basically identical, and all were too brittle.
Now they should be more resilient and easier to manage. The script
still needs to be updated if we add new ones, which I do not love.
More cleanup to follow.
Kubernetes-commit: e0ecccff3f5148cc167117ac73233b4edc1640d8
Dependencies need to be updated to use
github.com/container-orchestrated-devices/container-device-interface.
It's not decided yet whether we will implement Topology support
for DRA or not. Not having any toppology-related code
will help to avoid wrong impression that DRA is used as a hint
provider for the Topology Manager.
Kubernetes-commit: ae0f38437cbb5c2b515384cb9f7dea5d808b87c4
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
Sascha Grunert
Jordan Liggitt
John-Paul Sassine
Patrick Ohly
Haowei Cai
Anish Ramasekar
Mark Rossetti
Alexey Fomenko
Kevin Hannon
Itamar Holder
Evan Lezar
Kevin Klues
Moshe Levi
Tim Hockin
Ed Bartosh
Dixita Narang
Davanum Srinivas
Kubernetes Prow Robot
Aditi Sharma