CI is experiencing failures in the system_test step, caused by
podman commands issuing the following warning:
time="2019-07-09T13:30:19-04:00" level=error msg="User-selected graph driver \"overlay\" overwritten by graph driver \"vfs\" from database - delete libpod local files to resolve
Hypothesis: integration tests, which run just before us, are
leaving user config files in an unstable state.
Workaround: delete all user cache and config and db before
running system tests. This should be safe, and should be
a NOP when running as root.
Signed-off-by: Ed Santiago <santiago@redhat.com>
It's desirable to make archives available of builds containing actual
tested content. While not official distro-releases, these will enable
third-party testing, experimentation, and development for both branches
(e.g. "master") and pull requests (e.g. "pr3106").
* Add a Makefile targets for archiving both regular podman binaries
and the remote-client. Encode release metadata within these
archives so that their exact source can be identified.
* Fix bug with cross-compiling remote clients for the Windows and Darwin
platforms.
* Add unit-testing of cross-compiles for Windows and Darwin platforms.
* A few small CI-script typo-fixes
* Add a script which operates in two modes:
1. Call Makefile targets which produce release archives.
Upload the archive to Cirrus-CI's built-in caching system
using reproducible cache keys.
2. Utilize reproduced cache keys to attempt download of cache
from each tasks. When successful, parse the file's
release metadata, using it to name the archive file. Upload
all recovered archives to a publicly accessible storage bucket
for future reference.
* Update the main testing task to call the script in mode #1 for
all primary platforms.
* Add a new `$SPECIALMODE` task to call the script in mode #1 for
Windows and Darwin targets.
* Add a new 'release' task to the CI system, dependent upon all other
tasks. This new tasks executes the script in mode #2.
* Update CI documentation
Signed-off-by: Chris Evich <cevich@redhat.com>
Turn of go modules to avoid breaking build environments to accidentally
try pulling the dependencies instead of using the ./vendor directory.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add a `go-get` function to the Makefile to wrap `go get -u` into a
wrapper disabling go modules.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
An issue for achieving reproducible builds is build artifacts where
build paths are embedded. We remove them by passing the current working
directory to -gcflags and -asmflags which prefix trims the paths.
Note: Go 1.13 includes `-trimpath`
https://reproducible-builds.org/docs/build-path/
Signed-off-by: Morten Linderud <morten@linderud.pw>
Build artifacts embeds the current date of the build into the artifact.
If anyone want to reproduce the software at a later date there is no way
to pass a recorded date or fake it in the build system at a later point.
https://reproducible-builds.org/docs/source-date-epoch/
Signed-off-by: Morten Linderud <morten@linderud.pw>
I'm running the BATS tests manually once in a while, and
catching several problems each week that make it past
the rest of CI. Since the BATS tests run at RPM gating
time, we need to catch problems earlier. Try running
the tests from Cirrus.
Tests will be skipped on Ubuntu due to a too-ancient
version of coreutils (8.28; the 'timeout -v' we use
requires 8.29).
Tests are run *after* integration tests, even though
these take three minutes and would be nice to have
fail quickly, because running before causes bizarre
CI failures. Shrug.
UPDATE: also fix run test, broken by #3311.
Signed-off-by: Ed Santiago <santiago@redhat.com>
- PREFIX is now passed saved in the binary at build-time so that default
paths match installation paths.
- ETCDIR is also overridable in a similar way.
- DESTDIR is now applied on top of PREFIX for install/uninstall steps.
Previously, a DESTDIR=/foo PREFIX=/bar make would install into /bar,
rather than /foo/bar.
Signed-off-by: Lawrence Chan <element103@gmail.com>
For people who want to install podman remote or podman
only we need to separate out the two install commands.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want the remote tests for our distributions to be tested in a
different VM than the local tests. This allows for faster CI runs and
easier debug as well as seperation of flakes.
Signed-off-by: baude <bbaude@redhat.com>
If the systemd development files are not present on the system which
builds podman, then `podman events` will error on runtime creation.
Beside this, a warning will be printed when compiling podman.
This commit mainly exists because projects which depend on libpod
would not need the podman event support and therefore do not need to
rely on the systemd headers.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Existing code was not working due to a bash gotcha ('exit'
from a pipeline). It also had unnecessary duplication.
New version is safer; also includes unit tests run under localunit.
Existing invocations of req_env_var replaced via:
$ [ edit setup_environment.sh, move one closing quote to its own line ]
$ perl -ni -e 's/(?<=req_env_var )"(\S+)\s+\$\1"/$1/; if (/req_env_var "$/ .. /^\s*"/) { chomp; s/(?<=\S)\s.*//; if (/^\s*"/) { print "\n" } else { unless (/req_env_var/) { s/^\s+//; print " ";} print;} } else { print }' $(ack -l req_env_var)
$ [ hand-massage an incorrect instance of '@' in lib.sh:ircmsg() ]
Signed-off-by: Ed Santiago <santiago@redhat.com>
build a podman-remote binary for windows that allows users to use the
remote client on windows and interact with podman on linux system.
Signed-off-by: baude <bbaude@redhat.com>
Weekend hack by someone who doesn't grok zsh completion
but who finds it deeply offensive that most completion
files have an unmaintainable duplication of options
and arguments. The idea behind this one is to discover
the command line using --help, with a few hardcoded
helpers for discovering containers, images, pods,
and figuring out which args take files/dirs as args.
Working remarkably well. I am using this in my daily
routine and wondering how I ever managed without it.
It's not perfect -- a future version can perhaps
show only stopped containers for podman rm, only
running ones for podman stop -- but ROI seems low
on that given my limited zsh completion skills.
Sadly, I can't figure out how to write a regression
test suite for this. It would be lovely to have a
list if partial command lines and expected completions,
because the history of this change is that (seemingly)
minor tweaks in one place cause breakage in another.
Does anyone know of such a framework?
Still... working well enough to ship, IMO.
Signed-off-by: Ed Santiago <santiago@redhat.com>
to protect against regressions, we need to add a few gating tasks:
* build with varlink
* build podman-remote
* build podman-remote-darwin
we already have a gating task for building without varlink
Signed-off-by: baude <bbaude@redhat.com>
It also causes conflicts with CRI-O packages.
Also, change the path on seccomp.json so it lives in /usr/share
by default, with everything else.
Fixes#2596
Signed-off-by: Matthew Heon <mheon@redhat.com>
a series of improvements to our ginkgo test framework so we can
get better ideas of whats going on when run in CI
Signed-off-by: baude <bbaude@redhat.com>
when gopath was not explicitly set, make would fail due
to the varlink generator. this symlink in the makefile
addresses that.
fixes: #1842
Signed-off-by: baude <bbaude@redhat.com>
This is the final cleanup to remove urfave/sli from libpod. Removed
old, disabled tests that have not been run in over a year.
Signed-off-by: baude <bbaude@redhat.com>
We intend to migrate to the cobra cli from urfave/cli because the
project is more well maintained. There are also some technical reasons
as well which extend into our remote client work.
Signed-off-by: baude <bbaude@redhat.com>
@baude and I have recently debugged a `make vendor` issue, where
different versions of `vndr` leave slightly different states behind.
This ultimately leads to inconsistencies with the CI, which always
fetches the latest version.
To avoid such issues in the future, always use the latest version of
`vndr` by checking for new versions of it prior to execution.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Generate make helping message dynamicaly by using
python code snippet inside Makefile.
All commented make targets will be added to the
help message. To be added to the helping message
comment need to start with '## '.
These specials comments are detected by the python code.
Python code generate the helping output from these results.
Notice that this commit introduce a dependency with python (compatible python 2 and 3).
Signed-off-by: Hervé Beraud <hberaud@redhat.com>
PR #2259 removed the .install.gomega Makefile target but
didn't clean up two references to it. Do so now.
Also, when setting up GOPKGBASEDIR symlink, use -f (force)
flag; otherwise subsequent makes will fail.
Signed-off-by: Ed Santiago <santiago@redhat.com>
* Make sure that all vendored dependencies are in sync with the code and
the vendor.conf by running `make vendor` with a follow-up status check
of the git tree.
* Vendor ginkgo and gomega to include the test dependencies.
Signed-off-by: Chris Evic <cevich@redhat.com>
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
IMHO, longer than this waiting for automated testing is "too long".
Scientificaly speaking, based on thousands of runs across many
platforms, successful runs always happen in less time. Normally
Ubuntu passes in 35-40 minutes, and the Fedoras do it in 25-30.
If they take longer, something is likely badly broken. In that
case, it's better to fail within a short/defined time, than wait
for the (much longer) automation-level timeout and inevitable
failure.
Signed-off-by: Chris Evich <cevich@redhat.com>
libpod code added at end of Dockerfile, avoids
git clone of other packages in Dockerfile on subsequent builds.
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
Different components of testing reference the podman binary differently.
While they are identical in content, their SELinux types are not the
same, depending on build location. Avoid confusion and test failures
by always matching the bin/podman SELinux type to that of $BINDIR/podman
**after** install. This ensures even if the code or default contexts
change, a subsequent `make install` will re-match the SELinux type.
Signed-off-by: Chris Evich <cevich@redhat.com>
Also bump gitvalidation epoch - we usually do this every release,
but v1.0.0 is on a branch so we need a separate commit for master
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Add the ability to run the integration (ginkgo) suite using
the remote client.
Only the images_test.go file is run right now; all the rest are
isolated with a // +build !remotelinux. As more content is
developed for the remote client, we can unblock the files and
just block single tests as needed.
Signed-off-by: baude <bbaude@redhat.com>
Add a `make vendor` target calls `vndr` with a specified whitelist to
avoid deleting important files (currently the varlink/go project).
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The jsoniter library does not require code generation, which is a
massive advantage over easyjson (it's also about the same in
performance). Begin moving over to it by removing the existing
easyjson code.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
In podman 0.12.0 we have invalid completions. These should have been
caught during testing. This check will throw an error if the completions
do not successfully execute.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add the ability to build a remote client in golang that uses all
the same front-end cli code and output code. The initial limitations
here are that it can only be a local client while the bridge and
resolver code is being written for the golang varlink client.
Tests and docs will be added in subsequent PRs.
Signed-off-by: baude <bbaude@redhat.com>
It's impossible to get good debug out of the python tests, so
nuke them for now so I can figure out what's wrong.
DO NOT MERGE THIS COMMIT
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Add support for executing an init binary as PID 1 in a container to
forward signals and reap processes. When the `--init` flag is set for
podman-create or podman-run, the init binary is bind-mounted to
`/dev/init` in the container and "/dev/init --" is prepended to the
container's command.
The default base path of the container-init binary is `/usr/libexec/podman`
while the default binary is catatonit [1]. This default can be changed
permanently via the `init_path` field in the `libpod.conf` configuration
file (which is recommended for packaging) or temporarily via the
`--init-path` flag of podman-create and podman-run.
[1] https://github.com/openSUSE/catatonitFixes: #1670
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Also, bump the Dockerfile to use the latest Golang image, as most
of our testing is now done on 1.11
Fixes: #1999
Signed-off-by: Matthew Heon <matthew.heon@gmail.com>
> The go build command now maintains a cache of recently built
packages, separate from the installed packages in $GOROOT/pkg or
$GOPATH/pkg. The effect of the cache should be to speed builds that
do not explicitly install packages or when switching between
different copies of source code (for example, when changing back and
forth between different branches in a version control system). The
old advice to add the -i flag for speed, as in go build -i or go
test -i, is no longer necessary: builds run just as fast without -i.
This should also fix podman builds for NixOS, snap-installed go, …
Signed-off-by: Vincent Demeester <vdemeest@redhat.com>
The packer tool takes JSON as input for the details of producing VM
images to be used for PR CI-testing. JSON is not a very human-friendly
format, without support for comments and frequently containing lots of
duplicate data.
Fix this by using a Makefile + simple python one-liner to convert
from a human-friendly YAML format into packer-native JSON. This allows
use of anchors/aliases to reduce duplication, and allows inline comments
for easier maintainability. This also allows separating the 'test'
action from the 'build' action, for earlier and better syntax problem
detection.
Lastly, there are some minor ``lib.sh`` and ``integration_test.sh``
updates to support future work, and slightly improve the build and
test environments.
Signed-off-by: Chris Evich <cevich@redhat.com>
The tests can be filter by --focus and --skip to fit different test
target. Also be able to set global options and cmd options by export
it to ENV to fit different test matrix.
Signed-off-by: Yiqiao Pu <ypu@redhat.com>
We fixated on old metalinter dependency in past based on experience of
metalinter being oftentimes broke and hence broking our build. See
762f508d9ca97cdbaee6053b663e98aee9cae081 in cri-o for more details.
Now, dated metalinter is messing up with my environment (like it is
panicing on containters/storage) so let's see if we can move to more
current version of metalinter.
Signed-off-by: Šimon Lukašík <slukasik@redhat.com>
It was setting the wrong variable (CamelCase)
in the wrong module ("main", not "libpod")...
Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
The docker-in-docker was script was needed to run AppArmor tests in
Travis, which is not required anymore since Travis isn't being used
for a while. Removing the script will also cure some hiccups on
some atomic testing nodes.
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
This PR makes several key changes to our CI testing. Firstly, we now test
podman on fedora 28, fedora 29, and centos VMS (rather than containers). Any
of these that having failing tests are not marked as required yet. We
still preserve the podman in podman and podman in docker tests as well and
they are marked as required.
The lint and validate work is now done on a openshift container. We also
removed the rpm verification on papr and perform this test under the "images"
test on the openshift ci.
This PR exposes integration test fails on some of our OSs. My expectation is we
will fix those in additional PRs and as they are fixed, we should be flipping
the boolean bit to required.
Signed-off-by: baude <bbaude@redhat.com>
Closes: #1492
Approved by: mheon
This line landed in 8493dba2 (Initial varlink implementation,
2018-03-26, #627), but this Makefile has never consumed that variable.
Signed-off-by: W. Trevor King <wking@tremily.us>
The only Python dependency in contrib/python's clean is:
$(PYTHON) setup.py clean --all
and our setup.pys work on both major Python versions:
$ make -C contrib/python/podman PYTHON=python2 clean
make: Entering directory `/.../libpod/contrib/python/podman'
python2 setup.py clean --all
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
running clean
'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-2.7' does not exist -- can't clean it
rm -rf podman.egg-info dist
find . -depth -name __pycache__ -exec rm -rf {} \;
find . -depth -name \*.pyc -exec rm -f {} \;
make: Leaving directory `/.../libpod/contrib/python/podman'
$ echo $?
0
$ make -C contrib/python/pypodman PYTHON=python2 clean
make: Entering directory `/.../libpod/contrib/python/pypodman'
python2 setup.py clean --all
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
running clean
removing 'build/lib' (and everything under it)
removing 'build/bdist.linux-x86_64' (and everything under it)
'build/scripts-2.7' does not exist -- can't clean it
removing 'build'
rm -rf pypodman.egg-info dist
find . -depth -name __pycache__ -exec rm -rf {} \;
find . -depth -name \*.pyc -exec rm -f {} \;
make: Leaving directory `/.../libpod/contrib/python/pypodman'
$ echo $?
0
This rolls back part of 390bd16d (tidy up the copr spec, 2018-05-20, #813).
I've also shifted the submake cleans to the end of the main clean
recipe. That way, if one of the submake cleans dies (e.g. because the
system lacks Python entirely), the temp-file removal will still
happen. The way I have it setup now, the podman clean dying will keep
any of the pypodman clean from happening. You could probably work
around that with something like:
$(MAKE) -C contrib/python/podman clean & \
$(MAKE) -C contrib/python/pypodman clean & \
wait
but that seems too fancy for such a corner case.
Signed-off-by: W. Trevor King <wking@tremily.us>
Waiting uses a lot of CPU, so drop back to checking once/second
and allow user to pass in the interval.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
FFJSON has serialization differences versus stock Go - namely, it
does not respect the MarshalText() and UnmarshalText() methods,
particularly on []byte, which causes incompatability with
pre-FFJSON containers which contained DNS servers.
EasyJSON does not have these issues, and might even be slightly
faster.
Signed-off-by: Matthew Heon <matthew.heon@gmail.com>
Closes: #1322
Approved by: mheon
Need to get some small changes into libpod to pull back into buildah
to complete buildah transition.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1270
Approved by: mheon
In order to get a cleaner build out of the rpms we should
pass down the DESTDIR to the python Makefiles. Then we
can use them instead of hard coding other inteligence into
the spec files.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1214
Approved by: baude
Homu merged a commit without a signoff, update our Gitvalidation
epoch to after said commit to prevent problems.
Signed-off-by: Matthew Heon <mheon@redhat.com>
Closes: #1105
Approved by: vrothberg
Make users of libpod more secure by adding the libpod/apparmor package
to load a pre-defined AppArmor profile. Large chunks of libpod/apparmor
come from github.com/moby/moby.
Also check if a specified AppArmor profile is actually loaded and throw
an error if necessary.
The default profile is loaded only on Linux builds with the `apparmor`
buildtag enabled.
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Closes: #1063
Approved by: rhatdan
Pattern-rule documentation is in [1]. This commit follows the basic
approach from [2], with the portable build tags from [3].
Using --keep-going allows folks to see errors for multiple target
platforms. For example, if the Darwin target dies, we'll still
attempt to build the Linux target before erroring out.
I've added an ALLOWED_TO_FAIL environment variable to mark script
blocks for the the allow_failures block. Currently we're requiring
builds from Linux for Linux and OS X to succeed, but allowing builds
from OS X to both targets to fail.
[1]: https://www.gnu.org/software/make/manual/html_node/Pattern-Intro.html#Pattern-Intro
[2]: e5031fcf9a
[3]: https://github.com/kubernetes-incubator/cri-o/pull/1653
Signed-off-by: W. Trevor King <wking@tremily.us>
Closes: #1034
Approved by: baude
this should represent the last major changes to get darwin to **compile**. again,
the purpose here is to get darwin to compile so that we can eventually implement a
ci task that would protect against regressions for darwin compilation.
i have left the manual darwin compilation largely static still and in fact now only
interject (manually) two build tags to assist with the build. trevor king has great
ideas on how to make this better and i will defer final implementation of those
to him.
Signed-off-by: baude <bbaude@redhat.com>
Closes: #1047
Approved by: rhatdan
This makes fixing errors easier. Before this commit, errors looked
like [1]:
$ make gofmt
libpod/container_linux.go:1:⚠️ file is not gofmted with -s (gofmt)
make: *** [gofmt] Error 1
But that's not very helpful when your local gofmt thinks the file is
fine. With this commit, errors will look like:
$ make gofmt
find . -name '*.go' ! -path './vendor/*' -exec gofmt -s -w {} \+
git diff --exit-code
diff --git a/libpod/container_internal.go b/libpod/container_internal.go
index df4de3fe..22b39870 100644
--- a/libpod/container_internal.go
+++ b/libpod/container_internal.go
@@ -1,7 +1,7 @@
package libpod
import (
-"bytes"
+ "bytes"
"context"
"encoding/json"
"fmt"
make: *** [Makefile:87: gofmt] Error 1
(or whatever, I just stuffed in a formatting error for demonstration
purposes).
Also remove the helper script in favor of direct Makefile calls,
because with Git handling difference reporting and exit status, this
becomes a simpler check. find's -exec, !, and -path arguments are
specified in POSIX [2].
[1]: https://travis-ci.org/kubernetes-incubator/cri-o/jobs/331949394#L1075
[2]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html
Signed-off-by: W. Trevor King <wking@tremily.us>
Closes: #1038
Approved by: rhatdan
podman image and podman container have alternate CLI
to standard CLI for a lot of commands. The man pages
can be shared between both. This patch adds links so that
of some executes
`podman image load`, they will actually see the `podman load` man page.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #993
Approved by: mheon
some platforms and operating systems do not have varlink. in those cases,
we need to be able to turn off enablement of varlink in podman. this can now
be done with BUILDTAGS passed to the build though perhaps in the future
will be better.
the default is to build with varlink
Signed-off-by: baude <bbaude@redhat.com>
squash! make varlink optional for podman
The API.md and cmd/podman/varlink/ioprojectatomicpodman.go targets
will continue to work regardless of the presence (or not) of 'varlink'
is in BUILDTAGS. However, cmd/podman/varlink/ioprojectatomicpodman.go
is now only required by the podman target when BUILDTAGS contains
'varlink'.
API.md had also been an podman dependency since 5b2627dd (Makefile:
Drop find-godeps.sh for podman target, 2018-05-15, #776) when I
expanded varlink_api_generate. It had been an indirect podman
dependency (via varlink_api_generate) since 25263558 (Generate varlink
API documentation automatically, 2018-05-07, #734). But the podman
executable obviously doesn't depend on the Markdown file, so I'm
removing that dependency here.
Signed-off-by: baude <bbaude@redhat.com>
squash! make varlink optional for podman
The command-pointer approach will scale well if/when we add additional
optional commands behind their own build tags, because those tags
won't all be competing for the same getOptionalCommands namespace.
Signed-off-by: W. Trevor King <wking@tremily.us>
Closes: #987
Approved by: rhatdan
A commit snuck through without a DCO, update to a commit after
the broken one.
Signed-off-by: Matthew Heon <matthew.heon@gmail.com>
Closes: #957
Approved by: mheon
OpenShift Merge Robot
Ed Santiago
baude
Chris Evich
Matthew Heon
Giuseppe Scrivano
Valentin Rothberg
Morten Linderud
Lawrence Chan
Daniel J Walsh
Matthew Heon
Sascha Grunert
Kunal Kushwaha
Lars Karlitski
Hervé Beraud
Jhon Honce
Matthew Heon
Vincent Demeester
Yiqiao Pu
Šimon Lukašík
Anders F Björklund
Valentin Rothberg
W. Trevor King