Previously, if anyone touched these files no extra testing would
trigger. However, basically all testing depends on them. Update the
condition and test that verifies it.
Signed-off-by: Chris Evich <cevich@redhat.com>
When we check if source code was changed also include header files.
There is only one header file currently but that can change and it may
be possible that changes in this file can break things so make sure it
is considered source code so that all tests are triggered.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This directory contains important tools such as ginkgo as such updates
there should run through all testing and not skip anything.
Technically we do not need to run system tests as it doesn't use any
tool from there but that
a) might change in the future and
b) would make the only_if rules much more complicated if we try to
exclude it and
c) updates in test/tools are rare and/or automated so it does not cause
inconveniences to run all anyway
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
As we want to get rid of the special titles convert the existing skips
to the only_if condition, this makes it more readable as we do not need
to negate so much.
Then add similar conditions for all test tasks, this removes the need to
a special title such as CI:DOCS as the logic is smart enough to only
docs changes when no source code was changed.
Update the documentation for the new logic and no longer point
contributors to the CI:DOCS title as it is gone now.
There is a bunch of duplication in the rules as yaml doesn't allow us to
share only parts of a string. To prevent unwanted drift a test case in
contrib/cirrus/cirrus_yaml_test.py is added to ensure all conditions
follow the same base ruleset.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Now that we have source based skips there might be a case where we have
to run all tests. One option is to simply change a line in one of the
danger files but having something that can be set as title might be
easier for users.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
We do not have to test everything for each PR, we can know based on the
source if we changed (i.e. machine code) and only run the tests then.
This implements it as skip conditions, due to the nature of yaml files
we unfortunately cannot deduplicate everything, i.e. the is PR check and
danger files apply to everything but as skip is only a single yaml
string we cannot deduplicate parts of that string. If anyone knows a way
to achieve this I like to hear it.
For now I implemented this for int, system, bud and machine tests. Once
we are more comfortable with this I plan on adding it to other tests as
well.
This will replace the current _bail_if_test_can_be_skipped logic as it
covers more, marks tasks actually skipped in the github UI and works
even for the windows/macos machine tests.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
There's are sometimes conflicting purposes in podman CI:
1. Have the pipeline proceed in an orderly and progressive manner
to sometimes save resources and unnecessary runtime.
2. Complete all testing as quickly as possible in support of
human-developers moving on to other areas of work.
3. Ideally/hopefully, accomplish both items above safely,
preventing untested and/or unintended changes from merging.
This commit shifts the balance of these slightly more toward the second
point. It rearranges most CI tasks into essentially three buckets with
a single (new) aggregation task in-between the first two:
1. Build + Verify all the things
2. Test all the things
3. Minor/accessory things
The intention is that while we may unnecessarily spin some number of
testing tasks while others have failed, the best-case scenario
(everything passes) has a much shorter runtime. In other words, it
potentially wastes more resources in favor of a chance to have
developers wait less.
Signed-off-by: Chris Evich <cevich@redhat.com>
Issue Ref: #20853
Allow the tests to fail, but don't block merging PRs.
This commit should be reverted when #20853 is resolved.
Signed-off-by: Chris Evich <cevich@redhat.com>
Intended to serve as motivation to fix them. Removed from status
aggregator so the failures don't block PR merging. Updated comment text
to reference related open issue, #20548.
Signed-off-by: Chris Evich <cevich@redhat.com>
These jobs have been failing since early August due to
technical/scripting problems. Disable/remove entirely since a fix is
unlikely to be implemented anytime soon.
Ref: Abandoned recent attempt at debugging
https://github.com/containers/podman/pull/19720
Signed-off-by: Chris Evich <cevich@redhat.com>
Future work will present podman-machine benchmark data in some useful
format for analysis. However, this data is currently only stored as CI
artifacts. When CI runs on the main branch, after a PR merges, utilize
a pair of purpose-built containers to retrieve then upload the data into
a GCE firestore database. This operation should not be critical, such
that any faults will not cause the entire CI build to be marked as a
failure.
Signed-off-by: Chris Evich <cevich@redhat.com>
Run machine tests on every PR as label-driven machine test
triggering is currently hard to predict and debug.
Co-authored-by: Ed Santiago <santiago@redhat.com>
Co-authored-by: Miloslav Trmač <mitr@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
The podman-machine integration tests are designed to execute on
bare-metal, since they perform significant work with virtual-machines.
This test is costly to run at scale, so it is limited to being manually
triggered by developers (for now). A 'trigger' button will appear in the
task status page of the Github WebUI once all test dependencies are met.
In the Cirrus-CI WebUI, there is also a 'pre-trigger' button that may be
pressed if a developer doesn't wish to wait. Also:
* Add a `localmachine` target in the `Makefile` on the off-chance
developers wish to execute locally. Update the `ginkgo-run` target
to accommodate re-use by the new `localmachine` target.
* Exclude `podman_machine` task from `success` dependency verification.
This also involves adding an exception to `cirrus_yaml_test.py`
otherwise it will complain loudly.
* ***NOTE*** Inclusion of `ec2_instance` in *any* task will cause
`hack/get_ci_vm.sh` to barf and be non-functional. Future updates will
be made to restore functionality. Before then, simply comment out
the `ec2_instance` section as a temporarily workaround.
Signed-off-by: Chris Evich <cevich@redhat.com>
Building multi-arch images in a standardized way is complex. Some
of the builds themselves can take a really long time to run (over
an hour). Make changes easier to test inside a PR by adding
manually-triggered image-build tasks. These mirror most of the real
cron-triggered task, without actually pushing the final images.
Signed-off-by: Chris Evich <cevich@redhat.com>
In general continuous-delivery (CD) tends to pair well with CI. More
specifically, there is a need for some reverse-dependency CI testing in
netavark/aardvark-dns. In all cases, the download URL needs to remain
consistent, without elements like `Build%20for%20fedora-35`.
The 'Total Success' task only ever executes when all dependencies are
successful. When a non `[CI:DOCS]` build is successful, gather all
binary/release artifacts in a new task which depends on 'Total Success'.
This will provide a uniform name (`artifacts`) and URL for downstream
users to use. For example:
https://api.cirrus-ci.com/v1/artifact/github/containers/podman/artifacts/binary.zip
or
https://api.cirrus-ci.com/v1/artifact/github/containers/podman/artifacts/binary/FILENAME
Where ***FILENAME*** is one of:
* `podman`
* `podman-remote`
* `rootlessport`
* `podman-release-386.tar.gz`
* `podman-release-amd64.tar.gz`
* `podman-release-arm64.tar.gz`
* `podman-release-arm.tar.gz`
* `podman-release-mips64le.tar.gz`
* `podman-release-mips64.tar.gz`
* `podman-release-mipsle.tar.gz`
* `podman-release-mips.tar.gz`
* `podman-release-ppc64le.tar.gz`
* `podman-release-s390x.tar.gz`
* `podman-remote-release-darwin_amd64.zip`
* `podman-remote-release-darwin_arm64.zip`
* `podman-remote-release-windows_amd64.zip`
* `podman-v4.0.0-dev.msi`
Signed-off-by: Chris Evich <cevich@redhat.com>
Reimplement CI-automation to remove accumulated technical-debt and
optimize workflow. The task-dependency graph designed goal was to
shorten it's depth and increase width (i.e. more parallelism). A
reduction in redundant building (and 3rd party module download) was
also realized by caching `$GOPATH` and `$GOCACHE` early on. This
cache is then reused in favor of a fresh clone of the repository
(when possible).
Note: The system tests typically execute MUCH faster than the
integration tests. However, contrary to a fail-fast/fail-early
principal, they are executed last. This was implemented due to
debug-ability related concerns/preferences of the primary
(golang-centric) project developers.
Signed-off-by: Chris Evich <cevich@redhat.com>
The initial implementation was far more complicated than necessary.
Strip out the complexities in favor of a simpler and more direct
approach.
Signed-off-by: Chris Evich <cevich@redhat.com>
The release-task ***must*** always execute last, in order to guarantee a
consistent cache of release archives from dependent tasks. It
accomplishes this by verifying it's task-number matches one-less than
the total number of tasks. Previous to this commit, a YAML anchor/alias
was used to avoid duplication of the dependency list between 'success'
and 'release'
However, it's been observed that this opens the possibility for
'release' and 'success' tasks to race when running on a PR. Because
YAML anchor/aliases cannot be used to modify lists, duplication is
required to make 'release' actually depend upon 'success'.
This duplication will introduce an additional maintenance burden.
Though when adding a new task, it's already very easy to forget to
update the 'depends_on' list. Assist both cases by the addition
unit-tests to verify ``.cirrus.yml`` dependency contents and structure.
Signed-off-by: Chris Evich <cevich@redhat.com>
Chris Evich
Paul Holzinger
Lokesh Mandvekar