mirror of https://github.com/crossplane/docs.git
Rewrite packages content and split into Config and Provider chapters (#566)
This commit is contained in:
Pete Lumbis
GitHub
parent
cc67ebd12e
commit
c0fd459f16
|
|
@ -387,88 +387,6 @@ For example, for a `CloudSQLInstance` managed resource (`database.gcp.crossplane
|
|||
kubectl patch cloudsqlinstance my-db -p '{"metadata":{"finalizers": []}}' --type=merge
|
||||
```
|
||||
|
||||
## Installing Crossplane Package
|
||||
|
||||
After installing [Crossplane package], to verify the install results or
|
||||
troubleshoot any issue spotted during the installation, there are a few things
|
||||
you can do.
|
||||
|
||||
Run below command to list all Crossplane resources available on your cluster:
|
||||
|
||||
```shell
|
||||
kubectl get crossplane
|
||||
```
|
||||
|
||||
If you installed a Provider package, pay attention to the `Provider` and
|
||||
`ProviderRevision` resource. Especially the `INSTALLED` and `HEALTHY` column.
|
||||
They all need to be `TRUE`. Otherwise, there must be some errors that occurred
|
||||
during the installation.
|
||||
|
||||
If you installed a Configuration package, pay attention to the `Configuration`
|
||||
and `ConfigurationRevision` resource. Again, the `INSTALLED` and `HEALTHY`
|
||||
column for these resources need to be `TRUE`. Besides that, you should also see
|
||||
the `CompositeResourceDefinition` and `Composition` resources included in this
|
||||
package are listed if the package is installed successfully.
|
||||
|
||||
If you only care about the installed packages, you can also run below command
|
||||
which will show you all installed Configuration and Provider packages:
|
||||
|
||||
```shell
|
||||
kubectl get pkg
|
||||
```
|
||||
|
||||
When there are errors, you can run below command to check detailed information
|
||||
for the packages that are getting installed.
|
||||
|
||||
```shell
|
||||
kubectl get lock -o yaml
|
||||
```
|
||||
|
||||
To inspect a particular package for troubleshooting, you can run
|
||||
`kubectl describe` against the corresponding resources, e.g. the `Provider` and
|
||||
`ProviderRevision` resource for Provider package, or the `Configuration` and
|
||||
`ConfigurationRevision` resource for Configuration package. Usually, you should
|
||||
be able to know the error reason by checking the `Status` and `Events` field for
|
||||
these resources.
|
||||
|
||||
## Handling Crossplane Package Dependency
|
||||
|
||||
When using `crossplane.yaml` to define a Crossplane Configuration package, you
|
||||
can specify packages that it depends on by including `spec.dependsOn`. You can
|
||||
also specify version constraints for dependency packages.
|
||||
|
||||
When you define a dependency package, please make sure you provide the fully
|
||||
qualified address to the dependency package, but do not append the package
|
||||
version (i.e. the OCI image tag) after the package name. This may lead to the
|
||||
missing dependency error when Crossplane tries to install the dependency.
|
||||
|
||||
When specifying the version constraint, you should strictly follow the
|
||||
[semver spec]. Otherwise, it may not be able to find the appropriate version for
|
||||
the dependency package even it says the dependency is found. This may lead to an
|
||||
incompatible dependency error during the installation.
|
||||
|
||||
Below is an example where a Configuration package depends on a provider pulled
|
||||
from `xpkg.upbound.io/crossplane-contrib/provider-aws`. It defines `">=v0.18.2`
|
||||
as the version constraint which means all versions after `v0.16.0` including all
|
||||
prerelease versions, in the form of `-xyz` after the normal version string, will
|
||||
be considered when Crossplane tries to find the best match.
|
||||
|
||||
```yaml
|
||||
apiVersion: meta.pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: test-configuration
|
||||
annotations:
|
||||
provider: aws
|
||||
spec:
|
||||
crossplane:
|
||||
version: ">=v1.4.0-0"
|
||||
dependsOn:
|
||||
- provider: xpkg.upbound.io/crossplane-contrib/provider-aws
|
||||
version: ">=v0.18.2"
|
||||
```
|
||||
|
||||
|
||||
## Tips, Tricks, and Troubleshooting
|
||||
|
||||
In this section we'll cover some common tips, tricks, and troubleshooting steps
|
||||
|
|
|
|||
|
|
@ -1,513 +1,485 @@
|
|||
---
|
||||
title: Crossplane Packages
|
||||
weight: 104
|
||||
title: Configuration Packages
|
||||
description: "Packages combine multiple Crossplane resources into a single, portable, OCI image."
|
||||
altTitle: "Crossplane Packages"
|
||||
---
|
||||
<!--
|
||||
Install packages behind a proxy.
|
||||
https://github.com/upbound/provider-azure/issues/475#issuecomment-1608390254 -->
|
||||
|
||||
Crossplane packages are opinionated [OCI images] that contain a stream of YAML
|
||||
that can be parsed by the Crossplane package manager. Crossplane packages come
|
||||
in two varieties: [Providers] and Configurations. Ultimately, the primary
|
||||
purposes of Crossplane packages are as follows:
|
||||
A _Configuration_ package is an
|
||||
[OCI container images](https://opencontainers.org/) containing a collection of
|
||||
[Compositions]({{<ref "./compositions" >}}),
|
||||
[Composite Resource Definitions]({{<ref "./composite-resource-definitions" >}})
|
||||
and any required [Providers]({{<ref "./providers">}}) or
|
||||
[Functions]({{<ref "./composition-functions" >}}).
|
||||
|
||||
- **Convenient Distribution**: Crossplane packages can be pushed to or installed
|
||||
from any OCI-compatible registry.
|
||||
- **Version Upgrade**: Crossplane can update packages in-place, meaning that you
|
||||
can pick up support for new resource types or controller bug-fixes without
|
||||
modifying your existing infrastructure.
|
||||
- **Permissions**: Crossplane allocates permissions to packaged controllers in a
|
||||
manner that ensures they will not maliciously take over control of existing
|
||||
resources owned by other packages. Installing CRDs via packages also allows
|
||||
Crossplane itself to manage those resources, allowing for powerful
|
||||
[composition] features to be enabled.
|
||||
- **Dependency Management**: Crossplane resolves dependencies between packages,
|
||||
automatically installing a package's dependencies if they are not present in
|
||||
the cluster, and checking if dependency versions are valid if they are already
|
||||
installed.
|
||||
Configuration packages make your Crossplane configuration fully portable.
|
||||
|
||||
## Table of Contents
|
||||
{{<hint "important" >}}
|
||||
Crossplane [Providers]({{<ref "./providers">}}) and
|
||||
[Functions]({{<ref "./composition-functions">}}) are also Crossplane packages.
|
||||
|
||||
The following packaging operations are covered in detail below:
|
||||
This document describes how to install and manage configuration packages.
|
||||
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Building a Package](#building-a-package)
|
||||
- [Provider Packages](#provider-packages)
|
||||
- [Configuration Packages](#configuration-packages)
|
||||
- [Pushing a Package](#pushing-a-package)
|
||||
- [Installing a Package](#installing-a-package)
|
||||
- [spec.package](#specpackage)
|
||||
- [spec.packagePullPolicy](#specpackagepullpolicy)
|
||||
- [spec.revisionActivationPolicy](#specrevisionactivationpolicy)
|
||||
- [spec.revisionHistoryLimit](#specrevisionhistorylimit)
|
||||
- [spec.packagePullSecrets](#specpackagepullsecrets)
|
||||
- [spec.skipDependencyResolution](#specskipdependencyresolution)
|
||||
- [spec.ignoreCrossplaneConstraints](#specignorecrossplaneconstraints)
|
||||
- [spec.controllerConfigRef](#speccontrollerconfigref)
|
||||
- [Upgrading a Package](#upgrading-a-package)
|
||||
- [Package Upgrade Issues](#package-upgrade-issues)
|
||||
- [The Package Cache](#the-package-cache)
|
||||
- [Pre-Populating the Package Cache](#pre-populating-the-package-cache)
|
||||
|
||||
## Building a Package
|
||||
|
||||
As stated above, Crossplane packages are just opinionated OCI images, meaning
|
||||
they can be constructed using any tool that outputs files that comply the the
|
||||
OCI specification. However, constructing packages using the Crossplane CLI is a
|
||||
more streamlined experience, as it will perform build-time checks on your
|
||||
packages to ensure that they are compliant with the Crossplane [package format].
|
||||
|
||||
Providers and Configurations vary in the types of resources they may contain in
|
||||
their packages. All packages must have a `crossplane.yaml` file in the root
|
||||
directory with package contents. The `crossplane.yaml` contains the package's
|
||||
metadata, which governs how Crossplane will install the package.
|
||||
|
||||
### Provider Packages
|
||||
|
||||
A Provider package contains a `crossplane.yaml` with the following format:
|
||||
|
||||
```yaml
|
||||
apiVersion: meta.pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-gcp
|
||||
spec:
|
||||
crossplane:
|
||||
version: ">=v1.0.0"
|
||||
controller:
|
||||
image: crossplane/provider-gcp-controller:v0.14.0
|
||||
permissionRequests:
|
||||
- apiGroups:
|
||||
- apiextensions.crossplane.io
|
||||
resources:
|
||||
- compositions
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- create
|
||||
- update
|
||||
- patch
|
||||
- watch
|
||||
```
|
||||
|
||||
See all available fields in the [official documentation][provider-docs].
|
||||
|
||||
> Note: The `meta.pkg.crossplane.io` group does not contain custom resources
|
||||
> that may be installed into the cluster. They are strictly used as metadata in
|
||||
> a Crossplane package.
|
||||
|
||||
A Provider package may optionally contain one or more CRDs. These CRDs will be
|
||||
installed prior to the creation of the Provider's `Deployment`. Crossplane will
|
||||
not install _any_ CRDs for a package unless it can determine that _all_ CRDs can
|
||||
be installed. This guards against multiple Providers attempting to reconcile the
|
||||
same CRDs. Crossplane will also create a `ServiceAccount` with permissions to
|
||||
reconcile these CRDs and it will be assigned to the controller `Deployment`.
|
||||
|
||||
The `spec.controller.image` fields specifies that the `Provider` desires for the
|
||||
controller `Deployment` to be created with the provided image. It is important
|
||||
to note that this image is separate from the package image itself. In the case
|
||||
above, it is an image containing the `provider-gcp` controller binary.
|
||||
|
||||
The `spec.controller.permissionRequests` field allows a package author to
|
||||
request additional RBAC for the packaged controller. The controller's
|
||||
`ServiceAccount` will automatically give the controller permission to reconcile
|
||||
all types that its package installs, as well as `Secrets`, `ConfigMaps`, and
|
||||
`Events`. Any additional permissions must be explicitly requested.
|
||||
|
||||
> Note that the Crossplane RBAC manager can be configured to reject permissions
|
||||
> for certain API groups. If a package requests permissions that Crossplane is
|
||||
> configured to reject, the package will fail to be installed.
|
||||
> Authorized permissions should be aggregated to the rbac manager clusterrole
|
||||
> (the cluster role defined by the provider-clusterrole flag in the rbac manager)
|
||||
> by using the label `rbac.crossplane.io/aggregate-to-allowed-provider-permissions: "true"`
|
||||
|
||||
The `spec.crossplane.version` field specifies the version constraints for core
|
||||
Crossplane that the `Provider` is compatible with. It is advisable to use this
|
||||
field if a package relies on specific features in a minimum version of
|
||||
Crossplane.
|
||||
|
||||
> All version constraints used in packages follow the [specification] outlined
|
||||
> in the `Masterminds/semver` repository.
|
||||
|
||||
For an example Provider package, see [provider-gcp].
|
||||
|
||||
To build a Provider package, navigate to the package root directory and execute
|
||||
the following command:
|
||||
|
||||
```
|
||||
crossplane build provider
|
||||
```
|
||||
|
||||
If the Provider package is valid, you will see a file with the `.xpkg`
|
||||
extension.
|
||||
|
||||
> Note that the Crossplane CLI will not follow symbolic links for files in the
|
||||
> root package directory.
|
||||
|
||||
### Configuration Packages
|
||||
|
||||
A Configuration package contains a `crossplane.yaml` with the following format:
|
||||
|
||||
```yaml
|
||||
apiVersion: meta.pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: my-org-infra
|
||||
spec:
|
||||
crossplane:
|
||||
version: ">=v1.0.0"
|
||||
dependsOn:
|
||||
- provider: xpkg.upbound.io/crossplane-contrib/provider-gcp
|
||||
version: ">=v0.14.0"
|
||||
```
|
||||
|
||||
See all available fields in the [official documentation][configuration-docs].
|
||||
|
||||
A Configuration package may also specify one or more of
|
||||
`CompositeResourceDefinition` and `Composition` types. These resources will be
|
||||
installed and will be solely owned by the Configuration package. No other
|
||||
package will be able to modify them.
|
||||
|
||||
The `spec.crossplane.version` field serves the same purpose that it does in a
|
||||
`Provider` package.
|
||||
|
||||
The `spec.dependsOn` field specifies packages that this package depends on. When
|
||||
installed, the package manager will ensure that all dependencies are present and
|
||||
have a valid version given the constraint. If a dependency is not installed, the
|
||||
package manager will install it at the latest version that fits within the
|
||||
provided constraints.
|
||||
|
||||
> Dependency resolution is a `beta` feature and depends on the `v1beta1`
|
||||
> [`Lock` API][lock-api].
|
||||
|
||||
For an example Configuration package, see [getting-started-with-gcp](https://github.com/crossplane/docs/tree/master/content/media/snippets/package/gcp).
|
||||
|
||||
To build a Configuration package, navigate to the package root directory and
|
||||
execute the following command:
|
||||
|
||||
```
|
||||
crossplane build configuration
|
||||
```
|
||||
|
||||
If the Provider package is valid, you will see a file with the `.xpkg`
|
||||
extension.
|
||||
|
||||
## Pushing a Package
|
||||
|
||||
Crossplane packages can be pushed to any OCI-compatible registry. If a specific
|
||||
registry is not specified they will be pushed to Docker Hub.
|
||||
|
||||
To push a Provider package, execute the following command:
|
||||
|
||||
```
|
||||
crossplane push provider xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.22.0
|
||||
```
|
||||
|
||||
To push a Configuration package, execute the following command:
|
||||
|
||||
```
|
||||
crossplane push configuration xpkg.upbound.io/crossplane-contrib/my-org-infra:v0.1.0
|
||||
```
|
||||
|
||||
> Note: Both of the above commands assume a single `.xpkg` file exists in the
|
||||
> directory. If multiple exist or you would like to specify a package in a
|
||||
> different directory, you can supply the `-f` flag with the path to the
|
||||
> package.
|
||||
|
||||
## Installing a Package
|
||||
|
||||
Packages can be installed into a Crossplane cluster using the Crossplane CLI.
|
||||
|
||||
To install a Provider package, execute the following command:
|
||||
|
||||
```
|
||||
crossplane install provider xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.22.0
|
||||
```
|
||||
|
||||
To install a Configuration package, execute the following command:
|
||||
|
||||
```
|
||||
crossplane install configuration xpkg.upbound.io/crossplane-contrib/my-org-infra:v0.1.0
|
||||
```
|
||||
|
||||
Packages can also be installed manually by creating a `Provider` or
|
||||
`Configuration` object directly. The preceding commands would result in the
|
||||
creation of the following two resources, which could have been authored by hand:
|
||||
|
||||
```yaml
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-gcp
|
||||
spec:
|
||||
package: xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.22.0
|
||||
packagePullPolicy: IfNotPresent
|
||||
revisionActivationPolicy: Automatic
|
||||
revisionHistoryLimit: 1
|
||||
```
|
||||
|
||||
```yaml
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: my-org-infra
|
||||
spec:
|
||||
package: xpkg.upbound.io/crossplane-contrib/my-org-infra:v0.1.0
|
||||
packagePullPolicy: IfNotPresent
|
||||
revisionActivationPolicy: Automatic
|
||||
revisionHistoryLimit: 1
|
||||
```
|
||||
|
||||
> Note: These types differ from the `Provider` and `Configuration` types we saw
|
||||
> earlier. They exist in the `pkg.crossplane.io` group rather than the
|
||||
> `meta.pkg.crossplane.io` group and are actual custom resources created in the
|
||||
> cluster.
|
||||
|
||||
The default fields specified above can be configured with different values to
|
||||
modify the installation and upgrade behavior of a package. In addition, there
|
||||
are multiple other fields which can further customize how the package manager
|
||||
handles a specific revision.
|
||||
|
||||
### spec.package
|
||||
|
||||
This is the package image that we built, pushed, and are asking Crossplane to
|
||||
install. The tag we specify here is important. Crossplane will periodically
|
||||
check if the installed image matches the digest of the image in the remote
|
||||
registry. If it does not, Crossplane will create a new _Revision_ (either
|
||||
`ProviderRevision` or `ConfigurationRevision`). If you do not wish Crossplane to
|
||||
ever update your packages without explicitly instructing it to do so, you should
|
||||
consider specifying a tag which you know will not have the underlying contents
|
||||
change unexpectedly (e.g. a specific semantic version, such as `v0.1.0`) or, for
|
||||
an even stronger guarantee, providing the image with a `@sha256` extension
|
||||
instead of a tag.
|
||||
|
||||
### spec.packagePullPolicy
|
||||
|
||||
Valid values: `IfNotPresent`, `Always`, or `Never` (default: `IfNotPresent`)
|
||||
|
||||
When a package is installed, Crossplane downloads the image contents into a
|
||||
cache. Depending on the image identifier (tag or digest) and the
|
||||
`packagePullPolicy`, the Crossplane package manager will decide if and when to
|
||||
check and see if newer package contents are available. The following table
|
||||
describes expected behavior based on the supplied fields:
|
||||
|
||||
| | `IfNotPresent` | `Always` | `Never` |
|
||||
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
|
||||
| Semver Tag (e.g. `v1.3.0`) | Package is downloaded when initially installed, and as long as it is present in the cache, it will not be downloaded again. If the cache is lost and the a new version of the package image has been pushed for the same tag, package could inadvertently be upgraded. <br><br> **Upgrade Safety: Strong** | Package is downloaded when initially installed, but Crossplane will check every minute if new content is available. New content would have to be pushed for the same semver tag for upgrade to take place. <br><br> **Upgrade Safety: Weak** | Crossplane will never download content. Must manually load package image in cache. <br><br> **Upgrade Safety: Strongest** |
|
||||
| Digest (e.g. `@sha256:28b6...`) | Package is downloaded when initially installed, and as long as it is present in the cache, it will not be downloaded again. If the cache is lost but an image with this digest is still available, it will be downloaded again. The package will never be upgraded without a user changing the digest. <br><br> **Upgrade Safety: Very Strong** | Package is downloaded when initially installed, but Crossplane will check every minute if new content is available. Because image digest is used, new content will never be downloaded. <br><br> **Upgrade Safety: Strong** | Crossplane will never download content. Must manually load package image in cache. <br><br> **Upgrade Safety: Strongest** |
|
||||
| Channel Tag (e.g. `latest`) | Package is downloaded when initially installed, and as long as it is present in the cache, it will not be downloaded again. If the cache is lost, the latest version of this package image will be downloaded again, which will frequently have different contents. <br><br> **Upgrade Safety: Weak** | Package is downloaded when initially installed, but Crossplane will check every minute if new content is available. When the image content is new, Crossplane will download the new contents and create a new revision. <br><br> **Upgrade Safety: Very Weak** | Crossplane will never download content. Must manually load package image in cache. <br><br> **Upgrade Safety: Strongest** |
|
||||
|
||||
### spec.revisionActivationPolicy
|
||||
|
||||
Valid values: `Automatic` or `Manual` (default: `Automatic`)
|
||||
|
||||
When Crossplane downloads new contents for a package, regardless of whether it
|
||||
was a manual upgrade (i.e. user updating package image tag), or an automatic one
|
||||
(enabled by the `packagePullPolicy`), it will create a new package revision.
|
||||
However, the new objects and / or controllers will not be installed until the
|
||||
new revision is marked as `Active`. This activation process is configured by the
|
||||
`revisionActivationPolicy` field.
|
||||
|
||||
An `Active` package revision attempts to become the _controller_ of all
|
||||
resources it installs. There can only be one controller of a resource, so if two
|
||||
`Active` revisions both install the same resource, one will fail to install
|
||||
until the other cedes control.
|
||||
|
||||
An `Inactive` package revision attempts to become the _owner_ of all resources
|
||||
it installs. There can be an arbitrary number of owners of a resource, so
|
||||
multiple `Inactive` revisions and a single `Active` revision can exist for a
|
||||
resource. Importantly, an `Inactive` package revision will not perform any
|
||||
auxiliary actions (such as creating a `Deployment` in the case of a `Provider`),
|
||||
meaning we will not encounter a situation where two revisions are fighting over
|
||||
reconciling a resource.
|
||||
|
||||
With `revisionActivationPolicy: Automatic`, Crossplane will mark any new
|
||||
revision as `Active` when it is created, as well as transition any old revisions
|
||||
to `Inactive`. When `revisionActivationPolicy: Manual`, the user must manually
|
||||
edit a new revision and mark it as `Active`. This can be useful if you are using
|
||||
a `packagePullPolicy: Automatic` with a channel tag (e.g. `latest`) and you want
|
||||
Crossplane to create new revisions when a new version is available, but you
|
||||
don't want to automatically update to that newer revision.
|
||||
|
||||
It is recommended for most users to use semver tags or image digests and
|
||||
manually update their packages, but use a `revisionActivationPolicy: Automatic`
|
||||
to avoid having to manually activate new versions. However, each user should
|
||||
consider their specific environment and choose a combination that makes sense
|
||||
for them.
|
||||
|
||||
For security reasons, it's suggested using image digests instead or alongside
|
||||
tags (`vx.y.z@sha256:...`), to ensure that the package content wasn't tampered
|
||||
with.
|
||||
|
||||
### spec.revisionHistoryLimit
|
||||
|
||||
Valid values: any integer, disabled by explicitly setting to `0` (default `1`)
|
||||
|
||||
When a revision transitions from `Inactive` to `Active`, its revision number
|
||||
gets set to one greater than the largest revision number of all revisions for
|
||||
its package. Therefore, as the number of revisions increases, the least recently
|
||||
`Active` revision will have the lowest revision number. Crossplane will garbage
|
||||
collect old `Inactive` revisions if they fall outside the
|
||||
`spec.revisionHistoryLimit`. For instance, if my revision history limit is `3`
|
||||
and I currently have three old `Inactive` revisions and one `Active` revision,
|
||||
when I upgrade the next time, the new revision will be given the highest
|
||||
revision number when it becomes `Active`, the previously `Active` revision will
|
||||
become `Inactive`, and the oldest `Inactive` revision will be garbage collected.
|
||||
|
||||
> Note: In the case that `spec.revisionActivationPolicy: Manual` and you upgrade
|
||||
> enough times (but do not make `Active` the new revisions), it is possible that
|
||||
> activating a newer revision could cause the previously `Active` revision to
|
||||
> immediately be garbage collected if it is outside the
|
||||
> `spec.revisionHistoryLimit`.
|
||||
|
||||
### spec.packagePullSecrets
|
||||
|
||||
Valid values: slice of `Secret` names (secrets must exist in `namespace`
|
||||
Crossplane was installed in, typically `crossplane-system`)
|
||||
|
||||
This field allows a user to provide credentials required to pull a package from
|
||||
a private repository on a registry. The credentials are passed along to a
|
||||
packaged controller if the package is a `Provider`, but are not passed along to
|
||||
any dependencies.
|
||||
|
||||
### spec.skipDependencyResolution
|
||||
|
||||
Valid values: `true` or `false` (default: `false`)
|
||||
|
||||
If `skipDependencyResolution: true`, the package manager will install a package
|
||||
without considering its dependencies.
|
||||
|
||||
### spec.ignoreCrossplaneConstraints
|
||||
|
||||
Valid values: `true` or `false` (default: `false`)
|
||||
|
||||
If `ignoreCrossplaneConstraints: true`, the package manager will install a
|
||||
package without considering the version of Crossplane that is installed.
|
||||
|
||||
### spec.controllerConfigRef
|
||||
|
||||
{{< hint "warning" >}}
|
||||
The `ControllerConfig` API has been deprecated and will be removed in a future
|
||||
release when a comparable alternative is available.
|
||||
Refer to the
|
||||
[Provider]({{<ref "./providers">}}) and
|
||||
[Composition Functions]({{<ref "./composition-functions">}}) chapters for
|
||||
details on their usage of packages.
|
||||
{{< /hint >}}
|
||||
|
||||
Valid values: name of a `ControllerConfig` object
|
||||
## Install a Configuration
|
||||
|
||||
Packaged `Provider` controllers are installed in the form of a `Deployment`.
|
||||
Crossplane populates the `Deployment` with default values that may not be
|
||||
appropriate for every use-case. In the event that a user wants to override some
|
||||
of the defaults that Crossplane has set, they may create and reference a
|
||||
`ControllerConfig`.
|
||||
Install a Configuration with a Crossplane
|
||||
{{<hover line="2" label="install">}}Configuration{{</hover>}} object by setting
|
||||
the {{<hover line="6" label="install">}}spec.package{{</hover>}} value to the
|
||||
location of the configuration package.
|
||||
|
||||
An example of when this may be useful is when a user is running Crossplane on
|
||||
EKS and wants to take advantage of [IAM Roles for Service Accounts]. This
|
||||
requires setting an `fsGroup` and annotating the `ServiceAccount` that
|
||||
Crossplane creates for the controller. This could be accomplished with the
|
||||
following `ControllerConfig` and `Provider`:
|
||||
For example to install the
|
||||
[Upbound AWS reference platform](https://marketplace.upbound.io/configurations/upbound/platform-ref-aws/v0.6.0),
|
||||
|
||||
```yaml
|
||||
apiVersion: pkg.crossplane.io/v1alpha1
|
||||
kind: ControllerConfig
|
||||
metadata:
|
||||
name: aws-config
|
||||
annotations:
|
||||
eks.amazonaws.com/role-arn: arn:aws:iam::$AWS_ACCOUNT_ID\:role/$IAM_ROLE_NAME
|
||||
spec:
|
||||
podSecurityContext:
|
||||
fsGroup: 2000
|
||||
---
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.33.0
|
||||
controllerConfigRef:
|
||||
name: aws-config
|
||||
```
|
||||
|
||||
You can find all configurable values in the [official `ControllerConfig`
|
||||
documentation][controller-config-docs].
|
||||
|
||||
## Upgrading a Package
|
||||
|
||||
Upgrading a `Provider` or `Configuration` to a new version can be accomplished
|
||||
by editing the existing manifest and applying it with a new version tag in
|
||||
`spec.package`. Crossplane will observe the updated manifest and create a new
|
||||
`ProviderRevision` or `ConfigurationRevision` for the specified version. The new
|
||||
revision will be activated in accordance with `spec.revisionActivationPolicy`.
|
||||
|
||||
### Package Upgrade Issues
|
||||
|
||||
Upgrading a package can require manual intervention in the event that the
|
||||
previous version of the package supported a version of a custom resource that
|
||||
has been dropped and replaced by a new version in the new package revision.
|
||||
Kubernetes does not allow for applying a `CustomResourceDefinition` (CRD) that
|
||||
drops a version in the `spec` that is in the current `status.storedVersions`
|
||||
list, meaning that a revision cannot update and become the _controller_ of all
|
||||
of its resources.
|
||||
|
||||
This situation can be remedied by manually deleting the offending CRD and
|
||||
letting the new revision re-create it. In the event that custom resources exist
|
||||
for the given CRD, they must be deleted before the CRD can be removed.
|
||||
|
||||
## The Package Cache
|
||||
|
||||
When a package is installed into a cluster, Crossplane fetches the package image
|
||||
and stores its contents in a dedicated package cache. By default, this cache is
|
||||
backed by an [`emptyDir` Volume][emptyDir-volume], meaning that all cached data
|
||||
is lost when a `Pod` restarts. Users who wish for cache contents to be persisted
|
||||
between `Pod` restarts may opt to instead use a [`persistentVolumeClaim`
|
||||
(PVC)][pvc] by setting the `packageCache.pvc` Helm chart parameter to the name
|
||||
of the PVC.
|
||||
|
||||
### Pre-Populating the Package Cache
|
||||
|
||||
Because the package cache can be backed by any storage medium, users are able to
|
||||
optionally to pre-populate the cache with images that are not present on an
|
||||
external [OCI registry]. To utilize a package that has been manually stored in
|
||||
the cache, users must specify the name of the package in `spec.package` and use
|
||||
`packagePullPolicy: Never`. For instance, if a user built a `Configuration`
|
||||
package named `mycoolpkg.xpkg` and loaded it into the volume that was to be used
|
||||
for the package cache (i.e. copied the `.xpkg` file into the storage medium
|
||||
backing the PVC), the package could be utilized with the following manifest:
|
||||
|
||||
```yaml
|
||||
```yaml {label="install"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: my-cool-pkg
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
package: mycoolpkg
|
||||
package: xpkg.upbound.io/upbound/platform-ref-aws:v0.6.0
|
||||
```
|
||||
|
||||
Crossplane installs the Compositions, Composite Resource Definitions and
|
||||
Providers listed in the Configuration.
|
||||
|
||||
### Install with Helm
|
||||
|
||||
Crossplane supports installing Configurations during an initial Crossplane
|
||||
installation with the Crossplane Helm chart.
|
||||
|
||||
Use the
|
||||
{{<hover label="helm" line="5" >}}--set configuration.packages{{</hover >}}
|
||||
argument with `helm install`.
|
||||
|
||||
For example, to install the Upbound AWS reference platform,
|
||||
|
||||
```shell {label="helm"}
|
||||
helm install crossplane \
|
||||
crossplane-stable/crossplane \
|
||||
--namespace crossplane-system \
|
||||
--create-namespace \
|
||||
--set configuration.packages='{xpkg.upbound.io/upbound/platform-ref-aws:v0.6.0}'
|
||||
```
|
||||
|
||||
### Install offline
|
||||
|
||||
Crossplane installs packages from a local package cache. By
|
||||
default the Crossplane package cache is an
|
||||
[emptyDir volume](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir).
|
||||
|
||||
Configure Crossplane to use a
|
||||
[PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
to use a storage location containing the Configuration image. Read more about
|
||||
configuring the Crossplane Pod settings in the
|
||||
[Crossplane install documentation]({{<ref "../software/install#customize-the-crossplane-helm-chart">}}).
|
||||
|
||||
Provide the name of the Configuration's `.xpkg` file and set
|
||||
{{<hover label="offline" line="7">}}packagePullPolicy: Never{{</hover>}}.
|
||||
|
||||
For example, to install a locally stored version of
|
||||
Upbound AWS reference platform set the
|
||||
{{<hover label="offline" line="6">}}package{{</hover>}} to the local filename
|
||||
and set the Configuration's
|
||||
{{<hover label="offline" line="7">}}packagePullPolicy: Never{{</hover>}}.
|
||||
|
||||
```yaml {label="offline"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: offline-platform-ref-aws
|
||||
spec:
|
||||
package: platform-ref-aws
|
||||
packagePullPolicy: Never
|
||||
```
|
||||
|
||||
Importantly, as long as a package is being used as the `spec.package` of a
|
||||
`Configuration` or `Provider`, it must remain in the cache. For this reason, it
|
||||
is recommended that users opt for a durable storage medium when manually loading
|
||||
packages into the cache.
|
||||
### Installation options
|
||||
|
||||
In addition, if manually loading a `Provider` package into the cache, users must
|
||||
ensure that the controller image that it references is able to be pulled by the
|
||||
cluster nodes. This can be accomplished either by pushing it to a registry, or
|
||||
by [pre-pulling images] onto nodes in the cluster.
|
||||
Configurations support multiple options to change configuration package related
|
||||
settings.
|
||||
|
||||
|
||||
<!-- Named Links -->
|
||||
#### Configuration revisions
|
||||
|
||||
When installing a newer version of an existing Configuration Crossplane creates
|
||||
a new configuration revision.
|
||||
|
||||
View the configuration revisions with
|
||||
{{<hover label="rev" line="1">}}kubectl get configurationrevisions{{</hover>}}.
|
||||
|
||||
```shell {label="rev",copy-lines="1"}
|
||||
kubectl get configurationrevisions
|
||||
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
|
||||
platform-ref-aws-1735d56cd88d True 2 xpkg.upbound.io/upbound/platform-ref-aws:v0.5.0 Active 2 2 46s
|
||||
platform-ref-aws-3ac761211893 True 1 xpkg.upbound.io/upbound/platform-ref-aws:v0.4.1 Inactive 5m13s
|
||||
```
|
||||
|
||||
Only a single revision is active at a time. The active revision determines the
|
||||
available resources, including Compositions and Composite Resource Definitions.
|
||||
|
||||
By default Crossplane keeps only a single _Inactive_ revision.
|
||||
|
||||
Change the number of revisions Crossplane maintains with a Configuration package
|
||||
{{<hover label="revHistory" line="6">}}revisionHistoryLimit{{</hover>}}.
|
||||
|
||||
The {{<hover label="revHistory" line="6">}}revisionHistoryLimit{{</hover>}}
|
||||
field is an integer.
|
||||
The default value is `1`.
|
||||
Disable storing revisions by setting
|
||||
{{<hover label="revHistory" line="6">}}revisionHistoryLimit{{</hover>}} to `0`.
|
||||
|
||||
For example, to change the default setting and store 10 revisions use
|
||||
{{<hover label="revHistory" line="6">}}revisionHistoryLimit: 10{{</hover>}}.
|
||||
|
||||
```yaml {label="revHistory"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
revisionHistoryLimit: 10
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Configuration package pull policy
|
||||
|
||||
Use a {{<hover label="pullpolicy" line="6">}}packagePullPolicy{{</hover>}} to
|
||||
define when Crossplane should download the Configuration package to the local
|
||||
Crossplane package cache.
|
||||
|
||||
The `packagePullPolicy` options are:
|
||||
* `IfNotPresent` - (**default**) Only download the package if it isn't in the cache.
|
||||
* `Always` - Check for new packages every minute and download any matching
|
||||
package that isn't in the cache.
|
||||
* `Never` - Never download the package. Packages are only installed from the
|
||||
local package cache.
|
||||
|
||||
{{<hint "tip" >}}
|
||||
The Crossplane
|
||||
{{<hover label="pullpolicy" line="6">}}packagePullPolicy{{</hover>}} works
|
||||
like the Kubernetes container image
|
||||
[image pull policy](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy).
|
||||
|
||||
Crossplane supports the use of tags and package digest hashes like
|
||||
Kubernetes images.
|
||||
{{< /hint >}}
|
||||
|
||||
For example, to `Always` download a given Configuration package use the
|
||||
{{<hover label="pullpolicy" line="6">}}packagePullPolicy: Always{{</hover>}}
|
||||
configuration.
|
||||
|
||||
```yaml {label="pullpolicy",copy-lines="6"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
packagePullPolicy: Always
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Revision activation policy
|
||||
|
||||
The `Active` package revision
|
||||
is the package controller actively reconciling resources.
|
||||
|
||||
By default Crossplane sets the most recently installed package revision as
|
||||
`Active`.
|
||||
|
||||
Control the Configuration upgrade behavior with a
|
||||
{{<hover label="revision" line="6">}}revisionActivationPolicy{{</hover>}}.
|
||||
|
||||
The {{<hover label="revision" line="6">}}revisionActivationPolicy{{</hover>}}
|
||||
options are:
|
||||
* `Automatic` - (**default**) Automatically activate the last installed configuration.
|
||||
* `Manual` - Don't automatically activate a configuration.
|
||||
|
||||
For example, to change the upgrade behavior to require manual upgrades, set
|
||||
{{<hover label="revision" line="6">}}revisionActivationPolicy: Manual{{</hover>}}.
|
||||
|
||||
```yaml {label="revision"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
revisionActivationPolicy: Manual
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
|
||||
#### Install a Configuration from a private registry
|
||||
|
||||
Like Kubernetes uses `imagePullSecrets` to
|
||||
[install images from private registries](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/),
|
||||
Crossplane uses `packagePullSecrets` to install Configuration packages from a
|
||||
private registry.
|
||||
|
||||
Use {{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}} to provide a
|
||||
Kubernetes secret to use for authentication when downloading a Configuration
|
||||
package.
|
||||
|
||||
{{<hint "important" >}}
|
||||
The Kubernetes secret must be in the same namespace as Crossplane.
|
||||
{{</hint >}}
|
||||
|
||||
The {{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}} is a list of
|
||||
secrets.
|
||||
|
||||
For example, to use the secret named
|
||||
{{<hover label="pps" line="6">}}example-secret{{</hover>}} configure a
|
||||
{{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}}.
|
||||
|
||||
```yaml {label="pps"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
packagePullSecrets:
|
||||
- name: example-secret
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Ignore dependencies
|
||||
|
||||
By default Crossplane installs any [dependencies](#manage-dependencies) listed
|
||||
in a Configuration package.
|
||||
|
||||
Crossplane can ignore a Configuration package's dependencies with
|
||||
{{<hover label="pkgDep" line="6" >}}skipDependencyResolution{{</hover>}}.
|
||||
|
||||
{{< hint "warning" >}}
|
||||
Most Configurations include dependencies for the required Providers.
|
||||
|
||||
If a Configuration ignores dependencies, the required Providers must be
|
||||
manually installed.
|
||||
{{< /hint >}}
|
||||
|
||||
For example, to disable dependency resolution configure
|
||||
{{<hover label="pkgDep" line="6" >}}skipDependencyResolution: true{{</hover>}}.
|
||||
|
||||
```yaml {label="pkgDep"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
skipDependencyResolution: true
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Ignore Crossplane version requirements
|
||||
|
||||
A Configuration package may require a specific or minimum Crossplane version
|
||||
before installing. By default, Crossplane doesn't install a Configuration if
|
||||
the Crossplane version doesn't meet the required version.
|
||||
|
||||
Crossplane can ignore the required version with
|
||||
{{<hover label="xpVer" line="6">}}ignoreCrossplaneConstraints{{</hover>}}.
|
||||
|
||||
For example, to install a Configuration package into an unsupported Crossplane
|
||||
version, configure
|
||||
{{<hover label="xpVer" line="6">}}ignoreCrossplaneConstraints: true{{</hover>}}.
|
||||
|
||||
```yaml {label="xpVer"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: platform-ref-aws
|
||||
spec:
|
||||
ignoreCrossplaneConstraints: true
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
|
||||
### Verify a Configuration
|
||||
|
||||
Verify a Configuration with
|
||||
{{<hover label="verify" line="1">}}kubectl get configuration{{</hover >}}.
|
||||
|
||||
A working configuration reports `Installed` and `Healthy` as `True`.
|
||||
|
||||
```shell {label="verify",copy-lines="1"}
|
||||
kubectl get configuration
|
||||
NAME INSTALLED HEALTHY PACKAGE AGE
|
||||
platform-ref-aws True True xpkg.upbound.io/upbound/platform-ref-aws:v0.6.0 54s
|
||||
```
|
||||
|
||||
### Manage dependencies
|
||||
|
||||
Configuration packages may include dependencies on other packages including
|
||||
Functions, Providers or other Configurations.
|
||||
|
||||
If Crossplane can't meet the dependencies of a Configuration the Configuration
|
||||
reports `HEALTHY` as `False`.
|
||||
|
||||
For example, this installation of the Upbound AWS reference platform is
|
||||
`HEALTHY: False`.
|
||||
|
||||
```shell {copy-lines="1"}
|
||||
kubectl get configuration
|
||||
NAME INSTALLED HEALTHY PACKAGE AGE
|
||||
platform-ref-aws True False xpkg.upbound.io/upbound/platform-ref-aws:v0.6.0 71s
|
||||
```
|
||||
|
||||
To see more information on why the Configuration isn't `HEALTHY` use
|
||||
{{<hover label="depend" line="1">}}kubectl describe configurationrevisions{{</hover>}}.
|
||||
|
||||
```yaml {copy-lines="1",label="depend"}
|
||||
kubectl describe configurationrevision
|
||||
Name: platform-ref-aws-a30ad655c769
|
||||
API Version: pkg.crossplane.io/v1
|
||||
Kind: ConfigurationRevision
|
||||
# Removed for brevity
|
||||
Spec:
|
||||
Desired State: Active
|
||||
Image: xpkg.upbound.io/upbound/platform-ref-aws:v0.6.0
|
||||
Revision: 1
|
||||
Status:
|
||||
Conditions:
|
||||
Last Transition Time: 2023-10-06T20:08:14Z
|
||||
Reason: UnhealthyPackageRevision
|
||||
Status: False
|
||||
Type: Healthy
|
||||
Controller Ref:
|
||||
Name:
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Warning LintPackage 29s (x2 over 29s) packages/configurationrevision.pkg.crossplane.io incompatible Crossplane version: package is not compatible with Crossplane version (v1.12.0)
|
||||
```
|
||||
|
||||
The {{<hover label="depend" line="18">}}Events{{</hover>}} show a
|
||||
{{<hover label="depend" line="21">}}Warning{{</hover>}} with a message that the
|
||||
current version of Crossplane doesn't meet the Configuration package
|
||||
requirements.
|
||||
|
||||
## Create a Configuration
|
||||
|
||||
Crossplane Configuration packages are
|
||||
[OCI container images](https://opencontainers.org/) containing one or more YAML
|
||||
files.
|
||||
|
||||
{{<hint "important" >}}
|
||||
Configuration packages are fully OCI compliant. Any tool that builds OCI images
|
||||
can build Configuration packages.
|
||||
|
||||
It's strongly recommended to use the Crossplane command-line tool to
|
||||
provide error checking and formatting to Crossplane package builds.
|
||||
|
||||
Read the
|
||||
[Crossplane package specification](https://github.com/crossplane/crossplane/blob/master/contributing/specifications/xpkg.md)
|
||||
for package requirements when building packages with third-party tools.
|
||||
{{</hint >}}
|
||||
|
||||
A Configuration package requires a `crossplane.yaml` file and may include
|
||||
Composition and CompositeResourceDefinition files.
|
||||
|
||||
<!-- vale Google.Headings = NO -->
|
||||
### The crossplane.yaml file
|
||||
<!-- vale Google.Headings = YES -->
|
||||
|
||||
To build a Configuration package using the Crossplane CLI, create a file
|
||||
named
|
||||
{{<hover label="cfgMeta" line="1">}}crossplane.yaml{{</hover>}}.
|
||||
The
|
||||
{{<hover label="cfgMeta" line="1">}}crossplane.yaml{{</hover>}}
|
||||
file defines the requirements and name of the
|
||||
Configuration.
|
||||
|
||||
{{<hint "important" >}}
|
||||
The Crossplane CLI only supports a file named `crossplane.yaml`.
|
||||
{{< /hint >}}
|
||||
|
||||
Configuration package uses the
|
||||
{{<hover label="cfgMeta" line="2">}}meta.pkg.crossplane.io{{</hover>}}
|
||||
Crossplane API group.
|
||||
|
||||
Specify any other Configurations, Functions or Providers in the
|
||||
{{<hover label="cfgMeta" line="7">}}dependsOn{{</hover>}} list.
|
||||
Optionally, you can require a specific or minimum package version with the
|
||||
{{<hover label="cfgMeta" line="9">}}version{{</hover>}} option.
|
||||
|
||||
You can also define a specific or minimum version of Crossplane for this
|
||||
Configuration with the
|
||||
{{<hover label="cfgMeta" line="11">}}crossplane.version{{</hover>}} option.
|
||||
|
||||
{{<hint "note" >}}
|
||||
Defining the {{<hover label="cfgMeta" line="10">}}crossplane{{</hover>}} object
|
||||
or required versions is optional.
|
||||
{{< /hint >}}
|
||||
|
||||
```yaml {label="cfgMeta",copy-lines="all"}
|
||||
$ cat crossplane.yaml
|
||||
apiVersion: meta.pkg.crossplane.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: test-configuration
|
||||
spec:
|
||||
dependsOn:
|
||||
- provider: xpkg.upbound.io/crossplane-contrib/provider-aws
|
||||
version: ">=v0.36.0"
|
||||
crossplane:
|
||||
version: ">=v1.12.1-0"
|
||||
```
|
||||
|
||||
### Build the package
|
||||
|
||||
Create the package using the Crossplane CLI command
|
||||
`crossplane build configuration -f <directory>`.
|
||||
|
||||
Where the `<directory>` is the directory containing the `crossplane.yaml` file
|
||||
and any Composition or CompositeResourceDefinition YAML files.
|
||||
|
||||
The CLI recursively searches for `.yml` or `.yaml` files in the directory to
|
||||
include in the package.
|
||||
|
||||
{{<hint "important" >}}
|
||||
You must ignore any other YAML files with `--ignore=<file_list>`.
|
||||
For
|
||||
example, `crossplane build configuration -f test-directory --ignore=".tmp/*"`.
|
||||
|
||||
Including YAML files that aren't Compositions or CompositeResourceDefinitions,
|
||||
including Claims isn't supported.
|
||||
{{</hint >}}
|
||||
|
||||
By default, Crossplane creates an `.xpkg` file of the Configuration name and
|
||||
a SHA-256 hash of the package contents.
|
||||
|
||||
For example, a {{<hover label="xpkgName" line="2">}}Configuration{{</hover>}}
|
||||
named {{<hover label="xpkgName" line="4">}}test-configuration{{</hover>}}.
|
||||
The
|
||||
Crossplane CLI builds a package named `test-configuration-e8c244f6bf21.xpkg`.
|
||||
|
||||
```yaml {label="xpkgName"}
|
||||
apiVersion: meta.pkg.crossplane.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: test-configuration
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
Specify the output file with `--name=<filename>` option.
|
||||
|
||||
For example, to build a package from a directory named `test-directory` and
|
||||
generate a package named `test-package.xpkg` use the command:
|
||||
|
||||
```shell
|
||||
crossplane build configuration -f test-directory --name=test-package
|
||||
```
|
||||
|
||||
Crossplane automatically adds the `.xpkg` extension.
|
||||
|
||||
Crossplane places the package in the provided directory, in this example,
|
||||
`test-directory`.
|
||||
|
||||
```shell
|
||||
ls test-directory
|
||||
composition.yml crossplane.yaml compositeresourcedefinition.yml test-package.xpkg
|
||||
```
|
||||
|
||||
[OCI images]: https://github.com/opencontainers/image-spec
|
||||
[Providers]: {{<ref "providers" >}}
|
||||
[provider-docs]: https://doc.crds.dev/github.com/crossplane/crossplane/meta.pkg.crossplane.io/Provider/v1
|
||||
[configuration-docs]: https://doc.crds.dev/github.com/crossplane/crossplane/meta.pkg.crossplane.io/Configuration/v1
|
||||
[lock-api]: https://doc.crds.dev/github.com/crossplane/crossplane/pkg.crossplane.io/Lock/v1beta1
|
||||
[specification]: https://github.com/Masterminds/semver#basic-comparisons
|
||||
[composition]: {{<ref "compositions" >}}
|
||||
[IAM Roles for Service Accounts]: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
|
||||
[controller-config-docs]: https://doc.crds.dev/github.com/crossplane/crossplane/pkg.crossplane.io/ControllerConfig/v1alpha1
|
||||
[package format]: https://github.com/crossplane/crossplane/blob/1aa83092172bdf0d2ed64754d33517c612ff7368/design/one-pager-package-format-v2.md
|
||||
[provider-gcp]: https://doc.crds.dev/github.com/crossplane/crossplane/meta.pkg.crossplane.io/Provider/v1
|
||||
[emptyDir-volume]: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
|
||||
[pvc]: https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim
|
||||
[OCI registry]: https://github.com/opencontainers/distribution-spec
|
||||
[pre-pulling images]: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images
|
||||
|
|
|
|||
|
|
@ -28,15 +28,11 @@ Find more providers in the [Upbound Marketplace](https://marketplace.upbound.io)
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
<!-- "are Managed" isn't passive in this context -->
|
||||
Providers define every external resource they can create in Kubernetes as a
|
||||
Kubernetes API endpoint. These endpoints are
|
||||
Kubernetes API endpoint.
|
||||
These endpoints are
|
||||
[_Managed Resources_]({{<ref "managed-resources" >}}).
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
{{< hint "note" >}}
|
||||
Instructions on building your own Provider are outside of the scope of this
|
||||
document. Read the Crossplane contributing [Provider Development Guide](https://github.com/crossplane/crossplane/blob/master/contributing/guide-provider-development.md)
|
||||
for more information.
|
||||
{{< /hint >}}
|
||||
|
||||
## Install a Provider
|
||||
|
||||
|
|
@ -63,14 +59,35 @@ spec:
|
|||
package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.39.0
|
||||
```
|
||||
|
||||
{{< hint "tip" >}}
|
||||
Providers are Crossplane Packages. Read more about Packages in the
|
||||
[Packages documentation]({{<ref "packages" >}}).
|
||||
{{< /hint >}}
|
||||
|
||||
By default, the Provider pod installs in the same namespace as Crossplane
|
||||
(`crossplane-system`).
|
||||
|
||||
{{<hint "note" >}}
|
||||
Providers are part of the
|
||||
{{<hover label="install" line="1">}}pkg.crossplane.io{{</hover>}} group.
|
||||
|
||||
The {{<hover label="meta-pkg" line="1">}}meta.pkg.crossplane.io{{</hover>}}
|
||||
group is for creating Provider packages.
|
||||
|
||||
Instructions on building Providers are outside of the scope of this
|
||||
document.
|
||||
Read the Crossplane contributing
|
||||
[Provider Development Guide](https://github.com/crossplane/crossplane/blob/master/contributing/guide-provider-development.md)
|
||||
for more information.
|
||||
|
||||
For information on the specification of Provider packages read the
|
||||
[Crossplane Provider Package specification](https://github.com/crossplane/crossplane/blob/master/contributing/specifications/xpkg.md#provider-package-requirements).
|
||||
|
||||
```yaml {label="meta-pkg"}
|
||||
apiVersion: meta.pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
# Removed for brevity
|
||||
```
|
||||
{{</hint >}}
|
||||
|
||||
### Install with Helm
|
||||
|
||||
Crossplane supports installing Providers during an initial Crossplane
|
||||
|
|
@ -90,28 +107,270 @@ crossplane-stable/crossplane \
|
|||
--set provider.packages='{xpkg.upbound.io/crossplane-contrib/provider-aws:v0.39.0}'
|
||||
```
|
||||
|
||||
### Install from a private repository
|
||||
### Install offline
|
||||
|
||||
Installing a Provider from a private package repository requires a
|
||||
Kubernetes secret object. The Provider uses the secret with the
|
||||
{{<hover label="pps" line="7" >}}packagePullSecrets{{</hover>}} option.
|
||||
Crossplane installs packages from a local package cache. By
|
||||
default the Crossplane package cache is an
|
||||
[emptyDir volume](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir).
|
||||
|
||||
Configure Crossplane to use a
|
||||
[PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
to use a storage location containing the Provider image. Read more about
|
||||
configuring the Crossplane Pod settings in the
|
||||
[Crossplane install documentation]({{<ref "../software/install#customize-the-crossplane-helm-chart">}}).
|
||||
|
||||
Provide the name of the Provider's `.xpkg` file and set
|
||||
{{<hover label="offline" line="7">}}packagePullPolicy: Never{{</hover>}}.
|
||||
|
||||
For example, to install a locally stored version of Provider AWS set the
|
||||
{{<hover label="offline" line="6">}}package{{</hover>}} to the local filename
|
||||
and set the Provider's
|
||||
{{<hover label="offline" line="7">}}packagePullPolicy: Never{{</hover>}}.
|
||||
|
||||
```yaml {label="offline"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: offline-provider-aws
|
||||
spec:
|
||||
package: provider-aws
|
||||
packagePullPolicy: Never
|
||||
```
|
||||
|
||||
### Installation options
|
||||
|
||||
Providers support multiple configuration options to change installation related
|
||||
settings.
|
||||
|
||||
#### Provider pull policy
|
||||
|
||||
Use a {{<hover label="pullpolicy" line="6">}}packagePullPolicy{{</hover>}} to
|
||||
define when Crossplane should download the Provider package to the local
|
||||
Crossplane package cache.
|
||||
|
||||
The `packagePullPolicy` options are:
|
||||
* `IfNotPresent` - (**default**) Only download the package if it isn't in the cache.
|
||||
* `Always` - Check for new packages every minute and download any matching
|
||||
package that isn't in the cache.
|
||||
* `Never` - Never download the package. Packages are only installed from the
|
||||
local package cache.
|
||||
|
||||
{{<hint "tip" >}}
|
||||
The Crossplane
|
||||
{{<hover label="pullpolicy" line="6">}}packagePullPolicy{{</hover>}} works
|
||||
like the Kubernetes container image
|
||||
[image pull policy](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy).
|
||||
|
||||
Crossplane supports the use of tags and package digest hashes like
|
||||
Kubernetes images.
|
||||
{{< /hint >}}
|
||||
|
||||
For example, to `Always` download a given Provider package use the
|
||||
{{<hover label="pullpolicy" line="6">}}packagePullPolicy: Always{{</hover>}}
|
||||
configuration.
|
||||
|
||||
```yaml {label="pullpolicy",copy-lines="6"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
packagePullPolicy: Always
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Revision activation policy
|
||||
|
||||
The `Active` package revision
|
||||
is the package controller actively reconciling resources.
|
||||
|
||||
By default Crossplane sets the most recently installed package revision as
|
||||
`Active`.
|
||||
|
||||
Control the Provider upgrade behavior with a
|
||||
{{<hover label="revision" line="6">}}revisionActivationPolicy{{</hover>}}.
|
||||
|
||||
The {{<hover label="revision" line="6">}}revisionActivationPolicy{{</hover>}}
|
||||
options are:
|
||||
* `Automatic` - (**default**) Automatically activate the last installed Provider.
|
||||
* `Manual` - Don't automatically activate a Provider.
|
||||
|
||||
For example, to change the upgrade behavior to require manual upgrades, set
|
||||
{{<hover label="revision" line="6">}}revisionActivationPolicy: Manual{{</hover>}}.
|
||||
|
||||
```yaml {label="revision"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
revisionActivationPolicy: Manual
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Package revision history limit
|
||||
|
||||
When Crossplane installs a different version of the same Provider package
|
||||
Crossplane creates a new _revision_.
|
||||
|
||||
By default Crossplane maintains one _Inactive_ revision.
|
||||
|
||||
{{<hint "note" >}}
|
||||
Read the [Provider upgrade](#upgrade-a-provider) section for
|
||||
more information on the use of package revisions.
|
||||
{{< /hint >}}
|
||||
|
||||
Change the number of revisions Crossplane maintains with a Provider Package
|
||||
{{<hover label="revHistoryLimit" line="6">}}revisionHistoryLimit{{</hover>}}.
|
||||
|
||||
The {{<hover label="revHistoryLimit" line="6">}}revisionHistoryLimit{{</hover>}}
|
||||
field is an integer.
|
||||
The default value is `1`.
|
||||
Disable storing revisions by setting
|
||||
{{<hover label="revHistoryLimit" line="6">}}revisionHistoryLimit{{</hover>}} to `0`.
|
||||
|
||||
For example, to change the default setting and store 10 revisions use
|
||||
{{<hover label="revHistoryLimit" line="6">}}revisionHistoryLimit: 10{{</hover>}}.
|
||||
|
||||
```yaml {label="revHistoryLimit"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
revisionHistoryLimit: 10
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Install a provider from a private registry
|
||||
|
||||
Like Kubernetes uses `imagePullSecrets` to
|
||||
[install images from private registries](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/),
|
||||
Crossplane uses `packagePullSecrets` to install Provider packages from a private
|
||||
registry.
|
||||
|
||||
Use {{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}} to provide a
|
||||
Kubernetes secret to use for authentication when downloading a Provider package.
|
||||
|
||||
{{<hint "important" >}}
|
||||
The Kubernetes secret must be in the same namespace as Crossplane.
|
||||
{{</hint >}}
|
||||
|
||||
The {{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}} is a list of
|
||||
secrets.
|
||||
|
||||
For example, to use the secret named
|
||||
{{<hover label="pps" line="6">}}example-secret{{</hover>}} configure a
|
||||
{{<hover label="pps" line="6">}}packagePullSecrets{{</hover>}}.
|
||||
|
||||
```yaml {label="pps"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: private-provider
|
||||
name: provider-aws
|
||||
spec:
|
||||
package: private-repo.example.org/providers/my-provider
|
||||
packagePullSecrets:
|
||||
- name: my-secret
|
||||
packagePullSecrets:
|
||||
- name: example-secret
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
{{< hint "note" >}}
|
||||
The Kubernetes secret object the Provider uses must be in the same namespace as
|
||||
the Crossplane pod.
|
||||
{{<hint "note" >}}
|
||||
Configured `packagePullSecrets` aren't passed to any Provider package
|
||||
dependencies.
|
||||
{{< /hint >}}
|
||||
|
||||
#### Ignore dependencies
|
||||
|
||||
By default Crossplane installs any [dependencies](#manage-dependencies) listed
|
||||
in a Provider package.
|
||||
|
||||
Crossplane can ignore a Provider package's dependencies with
|
||||
{{<hover label="pkgDep" line="6" >}}skipDependencyResolution{{</hover>}}.
|
||||
|
||||
For example, to disable dependency resolution configure
|
||||
{{<hover label="pkgDep" line="6" >}}skipDependencyResolution: true{{</hover>}}.
|
||||
|
||||
```yaml {label="pkgDep"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
skipDependencyResolution: true
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
#### Ignore Crossplane version requirements
|
||||
|
||||
A Provider package may require a specific or minimum Crossplane version before
|
||||
installing. By default, Crossplane doesn't install a Provider if the Crossplane
|
||||
version doesn't meet the required version.
|
||||
|
||||
Crossplane can ignore the required version with
|
||||
{{<hover label="xpVer" line="6">}}ignoreCrossplaneConstraints{{</hover>}}.
|
||||
|
||||
For example, to install a Provider package into an unsupported Crossplane
|
||||
version, configure
|
||||
{{<hover label="xpVer" line="6">}}ignoreCrossplaneConstraints: true{{</hover>}}.
|
||||
|
||||
```yaml {label="xpVer"}
|
||||
apiVersion: pkg.crossplane.io/v1
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: provider-aws
|
||||
spec:
|
||||
ignoreCrossplaneConstraints: true
|
||||
# Removed for brevity
|
||||
```
|
||||
|
||||
### Manage dependencies
|
||||
|
||||
Providers packages may include dependencies on other packages including
|
||||
Configurations or other Providers.
|
||||
|
||||
If Crossplane can't meet the dependencies of a Provider package the Provider
|
||||
reports `HEALTHY` as `False`.
|
||||
|
||||
For example, this installation of the Upbound AWS reference platform is
|
||||
`HEALTHY: False`.
|
||||
|
||||
```shell {copy-lines="1"}
|
||||
kubectl get providers
|
||||
NAME INSTALLED HEALTHY PACKAGE AGE
|
||||
provider-aws-s3 True False xpkg.upbound.io/upbound/provider-aws-s3:v0.41.0 12s
|
||||
```
|
||||
|
||||
To see more information on why the Provider isn't `HEALTHY` use
|
||||
{{<hover label="depend" line="1">}}kubectl describe providerrevisions{{</hover>}}.
|
||||
|
||||
```yaml {copy-lines="1",label="depend"}
|
||||
kubectl describe providerrevisions
|
||||
Name: provider-aws-s3-92206523fff4
|
||||
API Version: pkg.crossplane.io/v1
|
||||
Kind: ProviderRevision
|
||||
Spec:
|
||||
Desired State: Active
|
||||
Image: xpkg.upbound.io/upbound/provider-aws-s3:v0.41.0
|
||||
Revision: 1
|
||||
Status:
|
||||
Conditions:
|
||||
Last Transition Time: 2023-10-10T21:06:39Z
|
||||
Reason: UnhealthyPackageRevision
|
||||
Status: False
|
||||
Type: Healthy
|
||||
Controller Ref:
|
||||
Name:
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Warning LintPackage 41s (x3 over 47s) packages/providerrevision.pkg.crossplane.io incompatible Crossplane version: package is not compatible with Crossplane version (v1.10.0)
|
||||
```
|
||||
|
||||
The {{<hover label="depend" line="17">}}Events{{</hover>}} show a
|
||||
{{<hover label="depend" line="20">}}Warning{{</hover>}} with a message that the
|
||||
current version of Crossplane doesn't meet the Configuration package
|
||||
requirements.
|
||||
|
||||
## Upgrade a Provider
|
||||
|
||||
To upgrade an existing Provider edit the installed Provider Package by either
|
||||
|
|
@ -120,10 +379,34 @@ applying a new Provider manifest or with `kubectl edit providers`.
|
|||
Update the version number in the Provider's `spec.package` and apply the change.
|
||||
Crossplane installs the new image and creates a new `ProviderRevision`.
|
||||
|
||||
The `ProviderRevision` allows Crossplane to store deprecated Provider CRDs
|
||||
without removing them until you decide.
|
||||
|
||||
View the `ProviderRevisions` with
|
||||
{{<hover label="getPR" line="1">}}kubectl get providerrevisions{{</hover>}}
|
||||
|
||||
```shell {label="getPR",copy-lines="1"}
|
||||
kubectl get providerrevisions
|
||||
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
|
||||
provider-aws-s3-dbc7f981d81f True 1 xpkg.upbound.io/upbound/provider-aws-s3:v0.37.0 Active 1 1 10d
|
||||
provider-nop-552a394a8acc True 2 xpkg.upbound.io/crossplane-contrib/provider-nop:v0.3.0 Active 11d
|
||||
provider-nop-7e62d2a1a709 True 1 xpkg.upbound.io/crossplane-contrib/provider-nop:v0.2.0 Inactive 13d
|
||||
upbound-provider-family-aws-710d8cfe9f53 True 1 xpkg.upbound.io/upbound/provider-family-aws:v0.40.0 Active 10d
|
||||
```
|
||||
|
||||
By default Crossplane keeps a single
|
||||
{{<hover label="getPR" line="5">}}Inactive{{</hover>}} Provider.
|
||||
|
||||
Read the [revision history limit](#package-revision-history-limit) section to
|
||||
change the default value.
|
||||
|
||||
Only a single revision of a Provider is
|
||||
{{<hover label="getPR" line="4">}}Active{{</hover>}} at a time.
|
||||
|
||||
## Remove a Provider
|
||||
|
||||
Remove a Provider by deleting the Provider object with `kubectl delete
|
||||
provider`.
|
||||
Remove a Provider by deleting the Provider object with
|
||||
`kubectl delete provider`.
|
||||
|
||||
{{< hint "warning" >}}
|
||||
Removing a Provider without first removing the Provider's managed resources
|
||||
|
|
@ -294,7 +577,9 @@ Reason: UnknownPackageRevisionHealth
|
|||
Providers have two different types of configurations:
|
||||
|
||||
* _Controller configurations_ that change the settings of the Provider pod
|
||||
running inside the Kubernetes cluster. For example, Pod `toleration`.
|
||||
running inside the Kubernetes cluster. For example, setting a `toleration` on
|
||||
the Provider pod.
|
||||
|
||||
* _Provider configurations_ that change settings used when communicating with
|
||||
an external provider. For example, cloud provider authentication.
|
||||
|
||||
|
|
@ -308,14 +593,17 @@ Apply `ProviderConfig` objects to managed resources.
|
|||
|
||||
{{< hint "important" >}}
|
||||
The Crossplane community deprecated the `ControllerConfig` type in v1.11 to
|
||||
indicate that no further enhancements will be made to it.
|
||||
announce that there are no further enhancements.
|
||||
Applying a Controller configuration generates a deprecation warning.
|
||||
|
||||
<!-- vale Crossplane.Spelling = NO -->
|
||||
<!-- vale gitlab.SubstitutionWarning = NO -->
|
||||
<!-- allow runtime config -->
|
||||
Controller configurations are still supported until there is a replacement type
|
||||
in a future Crossplane version. You can read more about the design of
|
||||
in a future Crossplane version. You can read more about the design of the
|
||||
[Package Runtime Config](https://github.com/crossplane/crossplane/blob/master/design/one-pager-package-runtime-config.md)
|
||||
which will replace it in the future.
|
||||
|
||||
which is a future replacement.
|
||||
<!-- vale Crossplane.Spelling = YES -->
|
||||
<!-- vale gitlab.SubstitutionWarning = YES -->
|
||||
{{< /hint >}}
|
||||
|
||||
Applying a Crossplane `ControllerConfig` to a Provider changes the settings of
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@
|
|||
64-bit
|
||||
API's
|
||||
APIs
|
||||
autoscaler
|
||||
base64
|
||||
bool
|
||||
boolean
|
||||
|
|
@ -13,16 +14,20 @@ cluster-wide
|
|||
ClusterRole
|
||||
ClusterRoles
|
||||
command-line
|
||||
ConfigMap
|
||||
CRD
|
||||
CSS
|
||||
CUE
|
||||
DatabaseInstance
|
||||
docs-specific
|
||||
emptyDir
|
||||
Enum
|
||||
Env
|
||||
ESS
|
||||
float64
|
||||
Go
|
||||
gRPC
|
||||
imagePullSecret
|
||||
JSONPath
|
||||
key-pair
|
||||
kube-apiserver
|
||||
|
|
@ -34,8 +39,12 @@ namespace
|
|||
namespaced
|
||||
namespaces
|
||||
OCI
|
||||
PersistentVolumeClaim
|
||||
PriorityClass
|
||||
proselint
|
||||
RBAC
|
||||
RPC
|
||||
RPCs
|
||||
SCSS
|
||||
SDK
|
||||
SDKs
|
||||
|
|
@ -58,35 +67,3 @@ syscall
|
|||
tolerations
|
||||
VM
|
||||
YAML
|
||||
32-bit
|
||||
64-bit
|
||||
float64
|
||||
SHA-1
|
||||
SHA-256
|
||||
SHA-512
|
||||
base64
|
||||
RBAC
|
||||
RPC
|
||||
RPCs
|
||||
ServiceAccounts
|
||||
ServiceAccount
|
||||
ClusterRole
|
||||
ClusterRoles
|
||||
key-pair
|
||||
CI
|
||||
command-line
|
||||
YAML
|
||||
CSS
|
||||
SCSS
|
||||
docs-specific
|
||||
CRD
|
||||
CLI
|
||||
kube-controller-manager
|
||||
kube-apiserver
|
||||
cluster-wide
|
||||
autoscaler
|
||||
DatabaseInstance
|
||||
ConfigMap
|
||||
imagePullSecret
|
||||
PersistentVolumeClaim
|
||||
PriorityClass
|
||||
|
|
@ -21,6 +21,7 @@ exceptions:
|
|||
- compositeTypeRef
|
||||
- Composition
|
||||
- Compositions
|
||||
- Configuration
|
||||
- Cosmos
|
||||
- Crossplane
|
||||
- Docker
|
||||
|
|
|
|||
|
|
@ -141,7 +141,6 @@ tokens:
|
|||
- rapidly
|
||||
- rarely
|
||||
- really
|
||||
- recently
|
||||
- recklessly
|
||||
- regularly
|
||||
- remarkably
|
||||
|
|
|
|||
Loading…
Reference in New Issue