SpecGen is our primary container creation abstraction, and is
used to connect our CLI to the Libpod container creation backend.
Because container creation has a million options (I exaggerate
only slightly), the struct is composed of several other structs,
many of which are quite large.
The core problem is that SpecGen is also an API type - it's used
in remote Podman. There, we have a client and a server, and we
want to respect the server's containers.conf. But how do we tell
what parts of SpecGen were set by the client explicitly, and what
parts were not? If we're not using nullable values, an explicit
empty string and a value never being set are identical - and we
can't tell if it's safe to grab a default from the server's
containers.conf.
Fortunately, we only really need to do this for booleans. An
empty string is sufficient to tell us that a string was unset
(even if the user explicitly gave us an empty string for an
option, filling in a default from the config file is acceptable).
This makes things a lot simpler. My initial attempt at this
changed everything, including strings, and it was far larger and
more painful.
Also, begin the first steps of removing all uses of
containers.conf defaults from client-side. Two are gone entirely,
the rest are marked as remove-when-possible.
[NO NEW TESTS NEEDED] This is just a refactor.
Signed-off-by: Matt Heon <mheon@redhat.com>
This ended up more complicated then expected. Lets start first with the
problem to show why I am doing this:
Currently we simply execute ps(1) in the container. This has some
drawbacks. First, obviously you need to have ps(1) in the container
image. That is no always the case especially in small images. Second,
even if you do it will often be only busybox's ps which supports far
less options.
Now we also have psgo which is used by default but that only supports a
small subset of ps(1) options. Implementing all options there is way to
much work.
Docker on the other hand executes ps(1) directly on the host and tries
to filter pids with `-q` an option which is not supported by busybox's
ps and conflicts with other ps(1) arguments. That means they fall back
to full ps(1) on the host and then filter based on the pid in the
output. This is kinda ugly and fails short because users can modify the
ps output and it may not even include the pid in the output which causes
an error.
So every solution has a different drawback, but what if we can combine
them somehow?! This commit tries exactly that.
We use ps(1) from the host and execute that in the container's pid
namespace.
There are some security concerns that must be addressed:
- mount the executable paths for ps and podman itself readonly to
prevent the container from overwriting it via /proc/self/exe.
- set NO_NEW_PRIVS, SET_DUMPABLE and PDEATHSIG
- close all non std fds to prevent leaking files in that the caller had
open
- unset all environment variables to not leak any into the contianer
Technically this could be a breaking change if somebody does not
have ps on the host and only in the container but I find that very
unlikely, we still have the exec in container fallback.
Because this can be insecure when the contianer has CAP_SYS_PTRACE we
still only use the podman exec version in that case.
This updates the docs accordingly, note that podman pod top never falls
back to executing ps in the container as this makes no sense with
multiple containers so I fixed the docs there as well.
Fixes#19001
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2215572
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
After[1] c/image no longer prints "Storing signatures" so we should
not check for it.
[1] https://github.com/containers/image/pull/2001
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
First: fix podman-registry script so it preserves the initial $PODMAN,
so all subsequent invocations of ps, logs, and stop will use the
same binary and arguments. Until now we've handled this by requiring
that our caller manage $PODMAN (and keep it the same), but that's
just wrong.
Next, simplify the golang interface: move the $PODMAN setting into
registry.go, instead of requiring e2e callers to set it. (This
could use some work: the local/remote conditional is icky).
IMPORTANT: To prevent registry.go from using the wrong podman binary,
the Start() call is gone. Only StartWithOptions() is valid now.
And, minor cleanup: comments, and add an actual error-message check
Reason for this PR is a recurring flake, #18355, whose multiple
failure modes I truly can't understand. I don't think this PR
is going to fix it, but this is still necessary work.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The remote API will wait 300s by default before conmon will call the
cleanup. In the meantime when you inspect an exec session started with
ExecStart() (so not attached) and it did exit we do not know that. If
a caller inspects it they think it is still running. To prevent this we
should sync the session based on the exec pid and update the state
accordingly.
For a reproducer see the test in this commit or the issue.
Fixes#18424
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
If Pull() fails, which it does on registry or network flakes,
bail out early: there's no point in continuing. Same with
Save() and restoreImageFromCache(), although those are
unlikely to fail.
Possibly better solution: retry with backoff. Left as exercise
for future maintainer.
Use Expect() for failure checks, and correct two existing
instances of Printf()/Exit() to also use Expect().
Signed-off-by: Ed Santiago <santiago@redhat.com>
When we searching any image at a container registry,
--cert-dir and --creds could be required
as well as push, pull, etc.
Signed-off-by: Toshiki Sonoda <sonoda.toshiki@fujitsu.com>
Add --ignore flag to the command line
Add a new parameter to the NetworkCreate interface in pkg/domain for CreateOptions
Add a new API Network CreateWithOptions in pkg/bindings
Remote API - Add a query parameter to set the ignore flag
Kube - use the IgnoreIfExists flag when creating the default network instead of handling the failure
Add e2e tests
Update man page for podman-network-create
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
I found the ginkgolinter[1] by accident, this looks for not optimal
matching and suggest how to do it better.
Overall these fixes seem to be all correct and they will give much
better error messages when something fails.
Check out the repo to see what the linter reports.
[1] https://github.com/nunnatsa/ginkgolinter
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Motivated to have a working `make lint` on Fedora 37 (beta).
Most changes come from the new `gofmt` standards.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
I realized that `params.Del("SkipTLSVerify")` doesn't have any
effect because keys are always lowercased. So it should really
be `params.Del("skiptlsverify")`.
There's also a little bug introduced by 3bf52aa and b1d1248: if
one passes `ProgressWriter` object having `Stringer` interface
i.e. `bytes.Buffer` it ends up been serialized in query with
`util.ToParams()`.
To circumvent both problems I propose to mark non-serializable
parameters with `schema:"-"` so there's no need to delete them from
resulting `url.Values`.
Signed-off-by: Vladimir Kochnev <hashtable@yandex.ru>
Currently bindings writes image push progress to os.Stderr.
Since os.Stderr is inconvenience for bindings caller to
process the progress messages, Added this support.
Signed-off-by: Naoto Kobayashi <naoto.kobayashi4c@gmail.com>
Some refer to issues that are closed. Remove them.
Some are runc bugs that will never be fixed. Say so, and remove
the FIXME.
One (bps/iops) should probably be fixed. File an issue for it, and
update comment to include the issue# so my find-obsolete-skips script
can track it.
And one (rootless mount with a "kernel bug?" comment) is still
not fixed. Leave the skip, but add a comment documenting the symptom.
Signed-off-by: Ed Santiago <santiago@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
[NO NEW TESTS NEEDED]
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Bindings already support `Remove` which removes a manifest from the list
following function adds support for removing entire manifest for local
storage.
Similar functionality can be also used indirectly by using `Remove` defined in
image bindings
Signed-off-by: Aditya R <arajan@redhat.com>
I think we forgot to bump the version in the main branch. It should be
v4.1.0-dev now.
Also set the min api version to 4.0.0 as on the podman 4.0 branch.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The linter ensures a common code style.
- use switch/case instead of else if
- use if instead of switch/case for single case statement
- add space between comment and text
- detect the use of defer with os.Exit()
- use short form var += "..." instead of var = var + "..."
- detect problems with append()
```
newSlice := append(orgSlice, val)
```
This could lead to nasty bugs because the orgSlice will be changed in
place if it has enough capacity too hold the new elements. Thus we
newSlice might not be a copy.
Of course most of the changes are just cosmetic and do not cause any
logic errors but I think it is a good idea to enforce a common style.
This should help maintainability.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
golint, scopelint and interfacer are deprecated. golint is replaced by
revive. This linter is better because it will also check for our error
style: `error strings should not be capitalized or end with punctuation or a newline`
scopelint is replaced by exportloopref (already endabled)
interfacer has no replacement but I do not think this linter is
important.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
* Ensure meaningful behaviour when called with /v3.x.x semantics
* Change return code to 409 from 500 when client attempts to use an
existing network name
* Update API bats test runner to support /v4.0.0 endpoints by default
Signed-off-by: Jhon Honce <jhonce@redhat.com>
Since this option will also be used for netavark we should rename it to
something more generic. It is important that --cni-config-dir still
works otherwise we could break existing container cleanup commands.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
* Update method/function signatures use the manifest list name and
images associated with the operation explicitly, in general
func f(ctx context.Context, manifestListName string,
ImageNames []string, options *fOptions)
* Leverage gorilla/mux Subrouters to support API v3.x and v4.x for
manifests
* Make manifest API endpoints more RESTful
* Add PUT /manifest/{id} to update existing manifests
* Add manifests.Annotate to go bindings, uncommented unit test
* Add DELETE /manifest/{Id} to remove existing manifest list, use
PUT /manifest/{id} to remove images from a list
* Deprecated POST /manifest/{id}/add and /manifest/{id}/remove, use
PUT /manifest/{id} instead
* Corrected swagger godoc and updated to cover API changes
* Update podman manifest commands to use registry.Context()
* Expose utils.GetVar() to obtain query parameters by name
* Unexpose server.registerSwaggerHandlers, not sure why this was ever
exposed.
* Refactored code to use http.Header instead of map[string]string when
operating on HTTP headers.
* Add API-Version header support in bindings to allow calling explicate
versions of the API. Header is _NOT_ forwarded to the API service.
Signed-off-by: Jhon Honce <jhonce@redhat.com>
The libpod/network packages were moved to c/common so that buildah can
use it as well. To prevent duplication use it in podman as well and
remove it from here.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This option causes Podman to not only remove the specified containers
but all of the containers that depend on the specified
containers.
Fixes: https://github.com/containers/podman/issues/10360
Also ran codespell on the code
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
The libpod/images/remove endpoint is not very REST-ish but, after some
debate, was decided to be implemented as for the following reasons.
First, it allows for batch removing images which improves performance
significantly. Note that Docker does support `rmi -a`!
Second, it allows for hiding the logic of setting the right exit code to
use from the client and keep all the logic on the server.
Hence, when removing an image that does not exist, the server will
return a 200. The response, however, includes the error message to be
used *and* the exit code that podman-remote will use.
Fixes: #12441
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
One of the main uses of context.Context is to provide cancellation for
go-routines, including API requests. While all user-facing bindings
already used a context parameter, it was only used to pass the client
information around.
This commit changes the internal DoRequest wrapper to take an additional
context argument, and pass that to the http request. Previously, the context
was derived from context.Background(), which made it impossible to cancel
once started.
All the convenience wrappers already supported the context parameter, so the
only user facing change is that cancelling those context now works as one
would expect.
Signed-off-by: Moritz "WanzenBug" Wanzenböck <moritz@wanzenbug.xyz>
The returned error was not checked, thus the test could hang forever
since it blocks on the log channel.
Also handle unexpectedEOF like EOF.
Fixes#12176
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This PR fixes the case when the API return HTTP 409 response. Where the
API return the body format different then for other HTTP error codes.
Signed-off-by: Ondra Machacek <omachace@redhat.com>
Make use of the new network interface in libpod.
This commit contains several breaking changes:
- podman network create only outputs the new network name and not file
path.
- podman network ls shows the network driver instead of the cni version
and plugins.
- podman network inspect outputs the new network struct and not the cni
conflist.
- The bindings and libpod api endpoints have been changed to use the new
network structure.
The container network status is stored in a new field in the state. The
status should be received with the new `c.getNetworkStatus`. This will
migrate the old status to the new format. Therefore old containers should
contine to work correctly in all cases even when network connect/
disconnect is used.
New features:
- podman network reload keeps the ip and mac for more than one network.
- podman container restore keeps the ip and mac for more than one
network.
- The network create compat endpoint can now use more than one ipam
config.
The man pages and the swagger doc are updated to reflect the latest
changes.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
* Follow https://pkg.go.dev/cmd/go#hdr-Generate_Go_files_by_processing_source
for leading comment
* Add godoc strings for all exposed methods for IDE support
* Copy field godoc strings into generated code as function godoc string
* Remove unused/unnecessary fields from generator.go structures
* Cleanup code regarding template usage
Signed-off-by: Jhon Honce <jhonce@redhat.com>
We should only print unhealthy if the check fails. Currently this is
filling logs when users are running lots of healthchecks.
Improves: https://github.com/containers/podman/issues/11157
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
InfraContainer should go through the same creation process as regular containers. This change was from the cmd level
down, involving new container CLI opts and specgen creating functions. What now happens is that both container and pod
cli options are populated in cmd and used to create a podSpecgen and a containerSpecgen. The process then goes as follows
FillOutSpecGen (infra) -> MapSpec (podOpts -> infraOpts) -> PodCreate -> MakePod -> createPodOptions -> NewPod -> CompleteSpec (infra) -> MakeContainer -> NewContainer -> newContainer -> AddInfra (to pod state)
Signed-off-by: cdoern <cdoern@redhat.com>
* Add response.Body.Close() where needed to release HTTP
connections to API server.
* Add tests to ensure no general leaks occur. 100% coverage would be
required to ensure no leaks on any call.
* Update code comments to be godoc correct
Signed-off-by: Jhon Honce <jhonce@redhat.com>
Migrate the Podman code base over to `common/libimage` which replaces
`libpod/image` and a lot of glue code entirely.
Note that I tried to leave bread crumbs for changed tests.
Miscellaneous changes:
* Some errors yield different messages which required to alter some
tests.
* I fixed some pre-existing issues in the code. Others were marked as
`//TODO`s to prevent the PR from exploding.
* The `NamesHistory` of an image is returned as is from the storage.
Previously, we did some filtering which I think is undesirable.
Instead we should return the data as stored in the storage.
* Touched handlers use the ABI interfaces where possible.
* Local image resolution: previously Podman would match "foo" on
"myfoo". This behaviour has been changed and Podman will now
only match on repository boundaries such that "foo" would match
"my/foo" but not "myfoo". I consider the old behaviour to be a
bug, at the very least an exotic corner case.
* Futhermore, "foo:none" does *not* resolve to a local image "foo"
without tag anymore. It's a hill I am (almost) willing to die on.
* `image prune` prints the IDs of pruned images. Previously, in some
cases, the names were printed instead. The API clearly states ID,
so we should stick to it.
* Compat endpoint image removal with _force_ deletes the entire not
only the specified tag.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>