Changes SSH key behavior such that there is a single persisted key for all
machines across all providers. If there is no key that is located at
`.local/share/containers/podman/machine/` then it is created. The keys are
not deleted when the last machine on the host is removed.
The main motivation for this change is it leads to fewer files created on the
host as a result of vm configuration. Having `n` machines on your system doesn't
result in `2n` machine-related files in `.ssh` on your system anymore.
As a result of ssh keys being persisted by default, the `--save-keys` flag
on `podman machine rm` will no longer be supported.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Uses the systemd unit file parser to build unit files instead of having
them be just blocks of hard-coded strings.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves all of the ignitionfiles out of the `machine` package and into
its own called `ignition`. This required `VMType` to get moved out of
the `machine` package and into the `define` package in order to prevent
a circular dependency.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
I was trying to debug a failure which was seemingly related
to gvproxy failing which I now can't reproduce,
and added these while working on it. Maybe they're useful in
the future.
[NO NEW TESTS NEEDED]
Signed-off-by: Colin Walters <walters@verbum.org>
The following PR is the very first step in what will a series of steps
to apply a "common" machine configuration file to all providers.
Function names, method names, struct names, and field names are all up
for debate. The purpose of this PR is to offer a glimpse at the
direction we intend to take.
This PR also contains temporary structs (i.e. aThing) that are not
exported. These are merely placeholders.
The configuration work in this PR is also unused of yet. But the code
is compiled. Once merged, we can begin the next step of development.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
Removes the `MachineVMV1` and `MonitorV1` structures that have been
deprecated for a long enough period of time that it makes sense to no
longer support them.
Results in the removal of deprecated `getSocketAndPid` as well.
The migration code was added in commit
`6e0e1cbddd5e1c5dff51215ad2b41a99d890fad8` and made it into release `v4.1.0`
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
QEMU usb-host driver which is the one for passthrough, supports two
options for selecting an USB devices in the host to provide it to the
VM:
- Bus and Device number the device is plugged
- Vendor and Product information of the USB devices
https://qemu-project.gitlab.io/qemu/system/devices/usb.html
This commit allows a user to configure podman machine with either of
options, with new --usb command line option for podman machine init.
Examples
podman machine init tosovm4 --usb vendor=13d3,product=5406
podman machine init tosovm3 --usb bus=1,devnum=4 --usb bus=1,devnum=3
This commit also allows a user to change the USBs configured with
--usb command line option for podman machine set.
Note that this commit does not handle host device permissions nor
verify that the USB devices exists.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Creates a common SetIgnitionFile function in pkg/machine/ignition.go which
creates the new VMFile that will represent the machine's ignition file. It
assigns the VMFile to the provided location.
Creates an IgnitionBuilder type to generate the ignition configuration for a
given virt provider.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
In #20538, I was asked to consider refactoring the new OCI pull code
from within the generic machine directory. This is something I had
tried when originally coding it but it became apparent that a much
larger refactor to prevent circular deps was needed. Because I did not
want to pollute the initial PR with that refactor, I asked for the PR to
merge first. This is the refactor that needed to be done.
Signed-off-by: Brent Baude <bbaude@redhat.com>
allow podman machine to extract its disk image from an oci registry or
oci-dir locally. for now, the image must be relatively inflexible. it
must have 1 layer. the layer must possess one image. so a dockerfile
like:
FROM scratch
COPY ./myimage.xz /myimage.xz
when using an oci dir, the directory structure must adhere to the
typical directory structure of a an oci image (with one layer).
── blobs
│ └── sha256
│ ├── 53735773573b3853bb1cae16dd21061beb416239ceb78d4ef1f2a0609f7e843b
│ ├── 80577866ec13c041693e17de61444b4696137623803c3d87f92e4f28a1f4e87b
│ └── af57637ac1ab12f833e3cfa886027cc9834a755a437d0e1cf48b5d4778af7a4e
├── index.json
└── oci-layout
in order to identify this new input, you must use a transport/schema to
differentiate from current podman machine init --image-path behavior. we
will support `oci-dir://` and `docker://` as transports.
when using the docker transport, you can only use an empty transport for
input. for example, `podman machine init --image-path docker://`. A
fully quailified image name will be supported in the next iteration.
the transport absent anything means, i want to pull the default fcos
image stored in a registry. podman will determine its current version
and then look for its correlating manifest. in this default use case,
it would look for:
quay.io/libpod/podman-machine-images:<version>
that manifest would then point to specific images that contain the
correct arch and provider disk image. i.e.
quay.io/libpod/podman-machine-images:4.6-qcow2
this PR does not enable something like
docker://quay.io/mycorp/myimage:latest yet.
names, addresses, andf schema/transports are all subject to change. the
plan is to keep this all undocumented until things firm up.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
Refactors machine socket mapping to prevent using similar/the same code
paths. Moves the shared code to `pkg/machine/sockets.go`.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
On darwin arm64, we need to set the location of the ovmf vars. It should be put into the imageDir (also known as as dataDir). But because qemu determines the image path late in Init(), the image path is set something like a stream marker.
Fixes#20361
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
Implements a shared `GetLock` function for virtualization providers. Returns
a pointer to a lockfile used for serializing write operations.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
If init fails, or if a SIGINT is sent during init, podman machine should remove all files and configs
created during the init. This includes config jsons, image files, ssh
id's, and system connections. On Windows, the VM instances are also
unregistered.
Signed-off-by: Ashley Cui <acui@redhat.com>
Creates a wrapper around the Qemu command line implementation to prevent
the need to hard-code the different command line options in Init and
Start.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Removes the line in applehv and qemu `machine.go` file. These are
remnants from #19723. This lines was written to add stdin, stdout,
stderr as extra files, but that is not how `c.ExtraFiles` works (unlike
`os.ProcAttr`).
go source: https://cs.opensource.google/go/go/+/go1.21.1:src/os/exec/exec.go;l=147
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
After creating a podman machine, and before starting it, the LastUp value for podman machine ls should display Never. Previously, the LastUp value was the same as creation time. This also changes the LastUp value for inspect to ZeroTime instead of creation time.
Signed-off-by: Ashley Cui <acui@redhat.com>
This PR is a mishmash of updates needed so that the hyperv provider can
begin to passd the machine e2e tests.
Summary as follows:
* Added custom error handling for machine errors so that all providers
can generate the same formatted error messages. The ones implemented
thus far are needed for the basic and init tests. More will come as
they are identified.
* Vendored new libhvee for better memory inspection. The memory type
changed from uint32 to uint64.
* Some machine e2e tests used linux-specific utilities to check various
error conditions and messages (like pgrep). Those were made into
functions and implemented on an operating system level.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
Unix only code crept into shared portions of the machine codebase,
preventing builds on Windows. Move them into unix-only files.
[NO NEW TESTS NEEDED]
Signed-off-by: Ashley Cui <acui@redhat.com>
Converts the host networking code in `podman machine` to use the
`GvproxyCommand` type introduced in containers/gvisor-tap-vsock#258
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
As promised in #19596, this pr deduplicates and refactors image
acquisition. All virt providers that use FCOS as its default now use
the same code.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
After a failed start, we can run into (somehow inconsistent) states
where the machine won't start because a previous QEMU process is still
running and the PID file is being used. Stop didn't resolve the issue
as this state wasn't detected.
Allow to recover from this state by a) detecting it during start and
error out with a more helpful message than the error QEMU would
otherwise spit out, and b) by enabling stop to kill the dangling QEMU
process - even after a failed stop.
With the changes, a recovery may look as follows:
```
_ podman git:(main) _ ./bin/darwin/podman machine start
Starting machine "podman-machine-default"
Error: cannot start VM "podman-machine-default": another instance of "/opt/homebrew/bin/qemu-system-aarch64" is already running with process ID 970: please stop and restart the VM
_ podman git:(main) _ ./bin/darwin/podman machine stop
Machine "podman-machine-default" stopped successfully
_ podman git:(main) _ ./bin/darwin/podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
```
Please note that this change does not prevent us from running into such
inconsistent states but only allows for recovering from them.
[NO NEW TESTS NEEDED] - there is no reliable reproducer.
Fixes: #16054
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Moves the shared logic from `writeConfig` into a shared function in
`pkg/machine/machine_common.go`
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves most of the logic of `setRootful` to the common file
`pkg/machine/machine_common.go`.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves `removeFilesAndConnections` to the common file
`pkg/machine/connections.go` to be reused by multiple hypervisors.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves `waitAPIAndPrintInfo` into the common file
`pkg/machine/machine_common.go` allowing applehv and qemu to share the
code.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves the implementation of `addSSHConnectionsToPodmanSocket` into the
common file `pkg/machine/machine_common.go`. The implementation was
shared between the hypervisors and does not need to be implemented
multiple times.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves acquisition of an alternate image provided by the user out of
`acquireVMImage` in `pkg/machine/<hypervisor>/machine.go` and into
`pkg/machine/pull.go` as its own function.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Moves `getDevNullFiles` into a new common file,
`pkg/machine/machine_common.go`, preventing the re-implementation of the
function across the different hypervisor implementations.
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Converts new functions added in #19311 to methods and adds
documentation.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Lock the machine when stopping, removing or changing its attributes to
make sure write accesses are serialized which should prevent a number of
issues and inconsistencies reported.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Breaks some of the code in QEMU's `VirtProvider` implementation located
at `pkg/machine/qemu/config.go` into its own functions. Aids in
improving the readability of the code.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
Lock the VM on start. If the machine is in the "starting" state we know
that a previous start has failed and guide the user into resolving the
issue.
Concurrent starts will busy wait and return the expected "already
running" error.
NOTE: this change is only looking at the start issue (#18662). Other
commands such as stop and update should also lock and will be updated
in a future change. I expect the underlying issue to apply to all
machine providers, not only QEMU. It's desirable to aim for extending
the machine interface to also allow to `Lock()` and `Unlock()`. After
acquiring the lock, the VM should automatically be reloaded/updated.
[NO NEW TESTS NEEDED]
Fixes: #18662
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
The functions for QEMU's `VM` interface implementation (`machine.go`)
had quite large functions. Pulls out some code that could be moved to
its own function for easier readability.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
During the exponential backoff waiting for the machine to be fully up
and running, also make sure that SSH is ready. The systemd dependencies
of the ready.service include the sshd.service among others but that is
not enough.
Other CoreOS users reported the same issue on IRC, so I feel fairly
confident to use the pragmatic approach of making sure SSH works on the
client side. #17403 is quite old and there are other pressing machine
issues that need attention.
[NO NEW TESTS NEEDED]
Fixes: #17403
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Make sure that starting a qemu machine uses proper exponential backoffs
and that a single variable isn't shared across multiple backoffs.
DO NOT BACKPORT: I want to avoid backporting this PR to the upcoming 4.6
release as it increases the flakiness of machine start (see #17403). On
my M2 machine, the flake rate seems to have increased with this change
and I strongly suspect that additional/redundant sleep after waiting for
the machine to be running and listening reduced the flakiness. My hope
is to have more predictable behavior and find the sources of the flakes
soon.
[NO NEW TESTS NEEDED] - still too flaky to add a test to CI.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
When debugging #17403, the logs of sshd indicates that Podman tried to
ssh into the machine too soon as the `core` user has not yet been fully
set up:
> error: kex_exchange_identification: Connection closed by remote host
> fatal: Access denied for user core by PAM account configuration [preauth]
@dustymabe found that the we may have to wait for systemd-user sessions
to be up. Doing that reduces the flake rate on my M2 machine but does
not entirely fix the issue.
Since I have seen multiple symptoms of flakiness, I think it does not
hurt to add the systemd-user sessions to the dependencies of the ready
service and continue investigating.
[NO NEW TESTS NEEDED] - once we have a fix out, I want to exercise
frequent stop/start in the machine tests but they won't pass now.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Ensures that for each hypervisor implementation, their `config.go` file
deals with implementing the `VirtProvider` interface while the
`machine.go` file is for implementing the `VM` interface.
Moves the `Virtualization` type into a common file and
created wrappers for the individual hypervisors. Allows for shared
functions that are exactly the same while providing the flexibility to
create hypervisor-specific implementations of the functions.
[NO NEW TESTS NEEDED]
Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
When the `machine start` command is executed, Podman automatically retrieves the current host's `*_PROXY` environment variable and assigns it directly to the virtual machine in QEMU. However, most `*_PROXY` variables are set with `127.0.0.1` or `localhost`, such as `127.0.0.1:8888`. This causes failures in network-related operations within the virtual machine due to incorrect proxy settings.
Fixes: #14087
Signed-off-by: Black-Hole1 <bh@bugs.cc>
gvproxy listens on 127.0.0.1, using localhost as hostname can result in
the client trying to connect to the ipv6 localhost (`::1`). This will
fail as shown in the issue. This switches the hostname in the system
connection to 127.0.0.1 to fix this problem.
I switched the qemu, hyperV and WSL backend. I haven't touched the
applehv code because it uses two different ips and I am not sure what is
the correct thing there. I leave this to Brent to figure out.
[NO NEW TESTS NEEDED]
[1] https://github.com/containers/gvisor-tap-vsock/blob/main/cmd/gvproxy/main.go#L197-L199Fixes#16470
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
It really does not make sense to call RemoveConnection() twice and then
update the config file a third time in updateDefaultMachineinConfig().
This results in unnecessary reads/writes and more code.
Simplyfy this into one function that is only called once and do all
updates at once.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
the connection remove call must be done inside the function that is
returned so that we wait until the user confirmed it.
Fixes#18330
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
In rare instances, if podman machine start does not exit correctly, the machine can be left in a "Starting" state, when in reality the machine is stopped. This prevents the user from actually starting the machine. This commit makes sure that on `podman machine stop`, we check if this is the case, and correctly set the starting state to false, allowing the user to start their machine again.
Signed-off-by: Ashley Cui <acui@redhat.com>
This reverts commit 6b6458916e (Resolve
symlink path for qemu directory if possible).
Fully resolving the symlink to qemu solves some issues for
aarch64-darwin nix with regards to finding `edk2-aarch64-code.fd`, but
unfortunately the fully resolved path includes the version number,
making it so that even patch updates break the path to
homebrew-installed qemu files.
Fixes https://github.com/containers/podman/issues/18111
[NO NEW TESTS NEEDED]
Signed-off-by: Nathan Henrie <nate@n8henrie.com>
on machine start, create a socket representing the machine's podman
service socket so local (to the host) applications can take advanatge of
it.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
podman machine with Microsoft HyperV will use hvsock on the guest and
vsock on the guest for its networking. this pr enables the basics for
this to happen as well as changes to ignition to automatically set this
up with network manager.
the vm binary referenced in this pr is in containers/gvisor-tap-vsock
and will need to be added to distributions.
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
Windows HyperV uses HVSocks (Windows adaptation of vsock) for
communicating between vms and the host. Podman machine in Qemu uses a
virtual UDS to signal the host that the machine is booted. In HyperV,
we can use a HVSOCK for the same purpose.
One of the big aspects of using HVSOCK on Windows is that the HVSOCK
must be entered into the Windows registry. So now part of init and rm
of a podman machine, entries must be added and removed respectively.
Also duplicates are a no-no.
Signed-off-by: Brent Baude <bbaude@redhat.com>
Windows: Flush machine config writes before renaming
Windows: Previously this code was changed to improve atomicity by changing
the persitence approach to a two-step process (write + rename).
However, the first-step write operation was not fully flushed,
leading to the possibility of incomplete writes.
[NO NEW TESTS NEEDED]
Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
with libhvee, we are able to do the basics of podman machine management
on hyperv. The basic functions like init, rm, stop, and start are all
functional. Start and stop will periodically throw a benign error
processing the hyperv message being returned from the action. The error
is described in the todo's below.
notable items:
* no podman commands will work (like ps, images, etc)
* the machine must be initialized with --image-path and fed a custom image.
* disk size is set to 100GB statically.
* the vm joins the default hyperv network which is TCP/IP network based.
* podman machine ssh does not work
* podman machine set does not work
* you can grab the ip address from hyperv and fake a machine connection
with `podman system connection`.
* when booting, use the hyperv console to know the boot is complete.
TODOs:
* podman machine ssh
* podman machine set
* podman machine rm needs force bool
* disk size in NewMachine is set to 100GB
* podman start needs to wait until fully booted
* establish a boot complete signal from guest
* implement gvproxy like user networking
* fix benign failures in stop/start -> Error: error 2147749890 (FormatMessage failed with: The system cannot find message text for message number 0x%1 in the message file for %2.)
[NO NEW TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
in preparation for adding hyper as a machine option, several common
functions needed to be moved specifically from qemu to a common area in
pkg/machine. this usually involved functions and variables related to
using fcos as a machine image as well as its compression, artifact, and
image format.
[NO NEW TESTS NEEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
before we can support hyperv as a virtualization option for podman
machine, several areas in machine will require cleanup. this is the
first pass of these changes to keep the review burden low. changes
include:
* convert artifact, format (image format) and compression to enums
with string methods
* rename Provider interface to VirtProvider
* change Provider implementation in QEMU to QEMUVirt
* change Provider implementation in WSL to WSLVirt
as mentioned earlier, there will be several more of these refactoring
PRs because assumptions were made about associations of platforms and
virt providers as well as compression and image formats.
Signed-off-by: Brent Baude <bbaude@redhat.com>
There are 2 things added. First there is added support for handling drive
letters while doing value split. If drive letter is detected, then max number
of elements will be increased by one, but then first two will be concatenated
to reconstruct the path. Second part is basic, but working, conversion of Windows
path to Unix path to be used, when target path is not explicitly specified.
Signed-off-by: Arthur Sengileyev <arthur.sengileyev@gmail.com>
Using "w" suffixed versions of QEMU binaries for correct background process
handling and not bind it to lifecycle of command prompt window. Stub for
arm64 version added to fix compilation issues of this target, when QEMU
machine will be finally enabled.
Signed-off-by: Arthur Sengileyev <arthur.sengileyev@gmail.com>
On Mac machines security model none works, while "mapped-xattr"
causes symlinks to not work.
Update docs/source/markdown/podman-machine-init.1.md
[NO NEW TESTS NEEDED]
Related: https://github.com/containers/podman/discussions/16102
Co-authored-by: Tom Sweeney <tsweeney@redhat.com>
Signed-off-by: Sorin Sbarnea <sorin.sbarnea@gmail.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add quiet and no-info flags to podman machine start.
No-info suppresses helpful informational tips
Quiet suppresses machine start progress output, as well as informational
tips.
Signed-off-by: Ashley Cui <acui@redhat.com>
Safe guards calls to os.RemoveAll in order to prevent calls from accidently
deleting the root file system in very strange edge cases. Did this by creating
GuardedRemoveAll and migrated machine os.RemoveAll calls to it.
Signed-off-by: Mike Perry <mike@bitbistro.org>
Motivated to have a working `make lint` on Fedora 37 (beta).
Most changes come from the new `gofmt` standards.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
add the key used in newly initialized machines to the user's known_hosts file. This ensures that golang will be able to ssh into the machine using
podman-remote. Also, remove the /dev/null redirection for podman machine ssh's known_hosts file.
resolves#15347
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: cdoern <cbdoer23@g.holycross.edu>
Package `io/ioutil` was deprecated in golang 1.16, preventing podman from
building under Fedora 37. Fortunately, functionality identical
replacements are provided by the packages `io` and `os`. Replace all
usage of all `io/ioutil` symbols with appropriate substitutions
according to the golang docs.
Signed-off-by: Chris Evich <cevich@redhat.com>
Podman adds an Error: to every error message. So starting an error
message with "error" ends up being reported to the user as
Error: error ...
This patch removes the stutter.
Also ioutil.ReadFile errors report the Path, so wrapping the err message
with the path causes a stutter.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Jake Correnti