Consolidate checking the http-status codes to allow for a more uniform
error handling. Also treat code 429 (too many requests) as a known
error instead of an invalid status code.
When hitting 429, perform an exponential back off starting a 2 seconds
for at most 5 iterations. If the http.Response set the `Retry-Header`
then use the provided value or date to compute the delay until the
next attempt. Note that the maximum delay is 60 seconds.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Return an error whenever we attempt to create a docker client when the
underlying registry is configured to be blocked in the registries.conf.
Until now, users of the c/image library implemented this behaviour but
it has not been enforced.
Now, it's time to pull the trigger and enforce the configuration.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Primarily remove the misleading reference to makeRequest, which is not
the only caller of detectProperties.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
c.detectPropertiesOnce already ensures detectPropertiesHelper
only executes once, so the documentation and code for indicating this
in c.scheme is now redundant.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This commit enables private registry mirror support for the docker
client. Major changes and additions related to an image pull are:
- If mirrors are specified via the configuration (registries.conf), then
the first working (where the manifest is available remotely) mirror
will be used for the pull.
- The main registry will be the fallback if no mirror works.
- The insecure/skipVerify flags will be used from system context and
from the corresponding registry/mirror as fallback too.
- Configuring a prefix will be considered, too.
- Update documentation for registry mirror support within
`registries.conf` as well as the man page.
- Renamed the `url` to `location` within system_registries_v2
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
There have been redundant calls here since two ref.ref.Hostname()
calls were added in aaedc642 (Implement lookaside storage for
signatures for Docker registries, 2016-08-11, #52). At that point the
two calls were separated by a dockerHostname check which could have
been shifted by two lines to avoid the doubled function calls. But in
f28367e1 (Add docker/config package to containers/image/pkg,
2017-08-29, #333) the dockerHostname check moved to a separate
function entirely (newDockerClientWithDetails) while the Domain()
calls remained together in newDockerClientFromRef. So now there is no
longer any reason for the second call, and this commit drops it.
Signed-off-by: W. Trevor King <wking@tremily.us>
Make the extra token scope a parameter instead of a struct member. This
allows a parallel execution of TryReusingBlob without the need to
serialize accesses.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make tokenCache thread safe by turning it into a `sync.Map` suitable for
concurrent reads and writes.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make detectProperties usable for concurrent usage by executing it at
most once (via sync.Once) and storing its error in a field to make
it accessible to following invocations.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This allows using cross-repository mounts, which require
a token for both reading the source and reading and writing to the destination.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will make it easier to cache several different tokens in a single client.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is a trivial move, does not change behavior; but it will allow
setupRequestAuth to be a bit more complex about the scopes and token caching
in the future.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Make dockerClient sensitive to the configured registries by using the
sysregistriesv2 package. Unless specified in the SystemContext, the TLS
config's InsecureSkipVerify flag now adheres to the registries Insecure
field (if specified in the registries.conf). Notice, that the default
value remains false (i.e., TLS verification is performed).
SystemContext.DockerInsecureSkipTLSVerify is usually set by users of
this library (e.g., Skopeo and Podman). To be able to distinguish
between a default value and a user-specified value, introduce a
types.OptionalBool type that is now used for DockerInsecureSkipTLSVerify.
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
It would interpret the integer as an Unicode codepoint and print
a quoted character constant, which is not useful.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
e.g. 504 (Gateway timeout) instead of just "504".
Note that for unknown status codes this will output "99999 ()"; this
seems not worth worrying about and building a helper for right now.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
When logging a debug message about an error that we get back when trying
to ping a registry, log the error text, too.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Change the order in which a search happens. Try the v1 endpoint
first without any authentication token and if that fails, get a
token and try the v2 endpoint.
Also add feature to search with an empty query, which only works
with the v2 endpoint. If the query is empty skip the v1 endpoint
and go straight to the v2 endpoint.
This helps fix issues we were seeing when searching quay.io and also
adds the ability to search a whole registry without a specific query
parameter.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Previous to this commit the requests made against a bearer token service
would be lacking the `author` field.
That causes the jwt token to have an empty `sub` (Subject) field. [1]
The docker registry relies on the `sub` field to know the author of all
the requests. When a registry is configured to send notifications [2]
the resulting webhooks calls will be missing the author too.
This commit fixes the issue by reproducing the same request made by
the docker client.
[1] https://docs.docker.com/registry/spec/auth/jwt/#getting-a-bearer-token
[2] https://docs.docker.com/registry/configuration/#notifications
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
docker.SearchRegistry was not calling config.GetAuthentication
to get the credentials of a registry from the authfile.
Signed-off-by: umohnani8 <umohnani@redhat.com>
- Network IO paths should react to cancels now.
- File IO paths generally still won't.
- `SystemContext` objects have been renamed to `sys` to leave `ctx`
available for the stdlib context objects.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
especially focus on those from client.HandleErrorResponse, which may
be very unhelpful, e.g. "unknown: Not Found".
In general, make sure the error includes an expanded Docker reference,
or a full URL, or a hostname + path.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Adding /etc/containers/certs.d as another default certs directory
The code will first check /etc/containers/certs.d for the certificates
and if not found at this path it will fall back to /etc/docker/certs.dir
Signed-off-by: umohnani8 <umohnani@redhat.com>
podman search searches a registry for a matching image
this adds the functionality to support that
some registries respond to the v2 endpoint while others
only respond to the v1 endpoint.
This checks both endpoints for a result, and if none is given
the user is informed.
Signed-off-by: umohnani8 <umohnani@redhat.com>
See: https://docs.docker.com/registry/spec/auth/token/
Signed-off-by: Ed King <eking@pivotal.io>
Signed-off-by: Will Martin <wmartin@pivotal.io>
Signed-off-by: Julia Nedialkova <julianedialkova@hotmail.com>
This package is used in authenticating a user for kpod login
and can be used for authentication in kpod push, pull etc.
Signed-off-by: umohnani8 <umohnani@redhat.com>
Ignore (but log) all permission errors when trying to get the list of
additionally trusted Docker certificates and keys. This should not
result in any security issues, because additional trusted certificates
are additive (and thus the only issue that can occur is that a transport
is not trusted by skopeo when it should be).
Not ignoring permission errors would cause issues when trying to perform
a copy of a Docker image on a machine that has Docker installed (by
default Docker makes /etc/docker/certs.d not accessible). This would
manifest itself like so:
% ./skopeo copy docker://busybox oci:busybox
FATA[0000] Error initializing source docker://busybox:latest: open /etc/docker/certs.d/docker.io: permission denied
Signed-off-by: Aleksa Sarai <asarai@suse.de>
- Say what directory we are checking, even if it doesn't exist.
- Indent the search results to make it related to the directory
declaration above, and similar to registries.d output:
> Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
> cert: /etc/docker/certs.d/FOO/BAR.cert
> key: /etc/docker/certs.d/FOO/BAR.key
> Using registries.d directory /etc/containers/registries.d for sigstore configuration
> Using "default-docker" configuration
> No signature storage configuration found for FOO/BAZ:latest
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will make /etc/docker/certs.d/$host[:$port], notably RHEL CDN
certificates, work by default.
types.SystemContext.DockerCertPath, if defined, overrides and disables
DockerPerHostCertDirPath. We now always look into _some_ (exactly one)
directory. (This also forces us to explicitly override the default path
in tests because in Travis /etc/docker has mode 0700, causing spurious failures.)
(See also the added comment about handling docker.io in certs.d.)
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
i
This will make it easier to make the determination more complex without
affecting the consumer.
Also moves the 'certDir != ""' check out of setupCertificates.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Previously we would not use it if there was any other TLS configuration.
If the user specified certificates, this really does not make any sense
(there is no overlap between serverDeafult() and certificates).
If the user disabled TLS verification, skipping serverDefault() is a bit
more sensible (e.g. we may want to allow old ciphers because it doesn’t
matter anyway), but still suspect. If this turns out to be a problem,
the DockerInsecureTLSSkipVerify case can revert the specific settings,
making the intention explicit.
This also nicely simplifies the code.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is provided by the OpenShift-integrated registry. This is
equivalent to the atomic: transport (in the “openshift”) subpackage, but
it requires less code and notably does not require an OpenShift login
context to be configured.
See https://github.com/openshift/origin/pull/12504 and
https://github.com/openshift/openshift-docs/pull/3556 for more
information on this API extension.
To preserve compatibility, we always check for a configured lookaside
sigstore first; if that is set up, we use the lookaside and ignore the
registry-native signature storage. Usually the user would not bother to
set up the lookaside, and use the native mechanism.
The code is mostly trivial; the only non-obvious aspect is the loop in
putSignaturesToAPIExtension, which is a pretty direct translation of
openshiftImageDestination.PutSignatures.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This only moves the client.scheme == "" check inside the method; but the
naming will make it more appropriate to call the method from
dockerImagesource / dockerImageDestination when they need to depend on
the results of the properties detection in methods which may be called
before dockerClient.makeRequest is ever called (detecting the properties
as a side effect).
Does not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will allow using paths starting with /extensions/v2/ in the future.
Also, we can now name the parameter “path” instead of the incorrect
“url”, and the “path” name actually matches (there is no magic
/v2/ adding involved).
Also eliminates duplication of the ping paths (in logging code only).
Does not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This replaces the copy of github.com/docker/docker/reference in the same
place, which we have just gotten rid of, and allows using this package
even in consumers which insist on an incompatible version of
docker/distribution.
The copy has been edited to drop a reference to
github.com/docker/distribution/digestset .
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is an intermediate step which will eventually go away.
The goal of this PR is to get rid of c/i/docker/daemon/reference and to
replace uses of it by direct calls to docker/distribution/reference.
We can't do that safely and easily, because the two have different
semantics for reference.Named.Name() and reference.Named.String(): we
return a minimized version, e.g. "busybox", upstream returns an expanded
version, e.g. "docker.io/library/busybox".
BEFORE this commit the difference is hidden by using
docker/distribution/reference.WithName, which allows using the minimized
version, and works with it correctly; but because we want to use the
upstream canonicalization code, which will change semantics, we can't
just mix and match.
To make the distinction explicit, this commmit adds an X to ALL public
names from c/i/docker/daemon/reference. E.g. a reference.XNamed type,
which has methods XName and XString.
This is pretty large, but does not change behavior at all. By
inspection it is clear to see that reference.XNamed and subtypes does
not expose any of the non-X, conflicting, method names.
Using e.g.
> git diff --word-diff-regex=.|grep -F '{+'|grep -v '^\([^{]\|{+X+}\)*{\?$'
it is possible to see that most lines in this diff only add a single X
letter, and manually inspect the few lines which don't match the regexp.
The only REALLY new code is an explicit definition of namedRef.XName()
and namedRef.XString(), and two newly added casts to namedRef in cases
where we need to use the underlying distreference.Reference within
a reference.XNamed value. Strictly speaking these changes change
behavior, in that third-party implementations of reference.XNamed are no
longer accepted; but we broke them by renaming at all.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add containers/storage as a backend type called 'storage'.
The image's blobs are stored either as storage layers (if they look like
archived layers) or (alongside the manifest and signatures) as named big
data items associated with the image.
Inspection data and record-keeping (so that we can remember which blobs
were layers and which weren't) are encoded as a JSON object which is
stored in the storage image's metadata field.
When importing blobs, layer IDs are generated by concatenating the
parent's layer ID (if there is one) with the hex string representation
of the expected digest of the content blob, if one is known. If there
is no expected digest, the ID is randomly generated.
If we find ourselves importing a layer with the same ID as a layer that
we already have, we digest the incoming stream and compare it to the
cached digest of the already-present layer, and return an error only if
they don't match. If an expected blob digest is provided, the actual
digest of the blob is compared with it, and if they don't match, an
error is returned.
If we find ourselves importing a blob more than once, we track the IDs
of each of the resulting layers.
If we find ourselves importing an image that wants to be tagged with a
name that is already in use, the name is then assigned to the new image
and the old image remains otherwise unmodified. If that incoming image
claims to have the same ID as an image which we already have, the import
will fail.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Docker can store auth entries with schema prefix or even path suffix.
See an example:
{
"auths": {
"10.3.10.88:5000": {
...
},
"http://10.3.10.88:5000/v2/": {
...
},
"https://10.3.10.88:5000": {
...
},
"https://index.docker.io/v1/": {
...
}
}
}
The entries were created using command `docker login` of upstream Docker
1.12. Let's normalize the auth keys before trying to match against
hostname.
Signed-off-by: Michal Minář <miminar@redhat.com>
This does not manifest with skopeo because skopeo usually uses
contextFromGlobalOptions() which is never nil, but it still is a
violation of the documented API (and breaks the copy-lookalike I use for
quick debugging).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Then document SystemContext.DockerInsecureSkipTLSVerify more precisely.
Also simplify dockerClient.ping() a bit.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Allows configuring "sigstore" (for RW) and "sigstore-write" (write-only)
for images/repositories/namespaces/registries in
/etc/containers/repositories.d/*.yaml .
If configured, uses them to store signatures in the docker: transport.
Also includes documentation in docs/registries.d.md.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Submit this size in docker: destinations, validate it in dir: and oci:.
Preserve the size, if provided by ImageSource, when copying blobs.
Also document that PutBlob must be called before PutManifest and
PutSignatures; this is clearly the necessary ordering if the layer
tarballs are being generated on the fly, or if the manifest destination
wants to reject manifests referring to nonexistent blobs.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... instead of Docker-specific certPath and tlsVerify.
Also invert the sense of tlsVerify to make the default secure.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
* Do not consume the Body stream
* Use something closer to a real parser for WWW-Authenticate,
notably allowing a comman in the "scope" value.
* Allow missing "service" and "scope" parameters, they are not used by
OpenShift
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Implement a client to the chunked API, instead of the nonexistent
one-shot API (per
2a4deee441
).
Adds a FIXME to DELETE the pending upload on failure; the uploads are
supposed to time out so this is not immediately critical.
Fixes#64 .
Does not change behavior. This is a straightforward move and update of
package references, except for:
- Adding a duplicate definition of manifestSchema1 to
cmd/skopeo/copy.go. This will need to be cleaned up later, for now
preferring to make no design changes in this commit.
- Renaming parseDockerImage to NewDockerImage, to both make it public
and consistent with common golang conventions.
Ivan Voronchihin