Merge branch 'main' into develop

CONTRIBUTING.md

@@ -2,69 +2,88 @@
 
 ## Who can contribute?
 
-- Users that found a bug
+- Users that found a bug,
-- Users that wants to propose new functionalities or enhancements
+- Users that want to propose new functionalities or enhancements,
-- Users that want to help other users to troubleshoot their environments
+- Users that want to help other users to troubleshoot their environments,
-- Developers that want to fix bugs
+- Developers that want to fix bugs,
-- Developers that want to implement new functionalities or enhancements
+- Developers that want to implement new functionalities or enhancements.
 
 ## Branches
 
-Please request your PR to be merged into the `devel` branch. 
+Please request your pull request to be merged into the `devel` branch.
 Changes to the `stable` branch are managed by the repository maintainers.
 
 ## Development environment setup
 
 Note: Some steps are OPTIONAL but all are RECOMMENDED.
 
-1. Fork the project repo and clone it
+1. Fork the project repository and clone it:
-```shell
+
-$ git clone https://github.com/USERNAME/podman-compose.git
+   ```shell
-$ cd podman-compose
+   $ git clone https://github.com/USERNAME/podman-compose.git
-```
+   $ cd podman-compose
-1. (OPTIONAL) Create a python virtual environment. Example using [virtualenv wrapper](https://virtualenvwrapper.readthedocs.io/en/latest/): 
+   ```
-```shell
+
-mkvirtualenv podman-compose
+2. (OPTIONAL) Create a Python virtual environment. Example using
-```
+   [virtualenv wrapper](https://virtualenvwrapper.readthedocs.io/en/latest/):
-2. Install the project runtime and development requirements   
+
-```shell
+    ```shell
-$ pip install '.[devel]'
+    $ mkvirtualenv podman-compose
-```
+    ```
-3. (OPTIONAL) Install `pre-commit` git hook scripts (https://pre-commit.com/#3-install-the-git-hook-scripts)
+
-```shell
+3. Install the project runtime and development requirements:
-$ pre-commit install
+
-```
+   ```shell
-4. Create a new branch, develop and add tests when possible
+   $ pip install '.[devel]'
-5. Run linting & testing before committing code. Ensure all the hooks are passing.
+   ```
-```shell
+
-$ pre-commit run --all-files
+4. (OPTIONAL) Install `pre-commit` git hook scripts
-```
+   (https://pre-commit.com/#3-install-the-git-hook-scripts):
-6. Run code coverage
+
-```shell
+   ```shell
-coverage run --source podman_compose -m unittest pytests/*.py
+   $ pre-commit install
-python -m unittest tests/*.py
+   ```
-coverage combine
+
-coverage report
+5. Create a new branch, develop and add tests when possible.
-coverage html
+6. Run linting and testing before committing code. Ensure all the hooks are passing.
-```
+
-7. Commit your code to your fork's branch. 
+   ```shell
-   - Make sure you include a `Signed-off-by` message in your commits. Read [this guide](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) to learn how to sign your commits 
+   $ pre-commit run --all-files
-   - In the commit message reference the Issue ID that your code fixes and a brief description of the changes. Example: `Fixes #516: allow empty network`
+   ```
-7. Open a PR to `containers/podman-compose:devel` and wait for a maintainer to review your work.
+
+7. Run code coverage:
+
+    ```shell
+    $ coverage run --source podman_compose -m unittest pytests/*.py
+    $ python -m unittest tests/*.py
+    $ coverage combine
+    $ coverage report
+    $ coverage html
+    ```
+
+8. Commit your code to your fork's branch.
+   - Make sure you include a `Signed-off-by` message in your commits.
+     Read [this guide](https://github.com/containers/common/blob/main/CONTRIBUTING.md#sign-your-prs)
+     to learn how to sign your commits.
+   - In the commit message, reference the Issue ID that your code fixes and a brief description of
+     the changes.
+     Example: `Fixes #516: Allow empty network`
+9. Open a pull request to `containers/podman-compose:devel` and wait for a maintainer to review your
+   work.
 
 ## Adding new commands
 
-To add a command you need to add a function that is decorated
+To add a command, you need to add a function that is decorated with `@cmd_run`.
-with `@cmd_run` passing the compose instance, command name and
-description. This function must be declared `async` the wrapped 
-function should accept two arguments the compose instance and 
-the command-specific arguments (resulted from python's `argparse` 
-package) inside that command you can run PodMan like this 
-`await compose.podman.run(['inspect', 'something'])`and inside 
-that function you can access `compose.pods` and `compose.containers` 
-...etc. Here is an example
 
-```
+The decorated function must be declared `async` and should accept two arguments: The compose
+instance and the command-specific arguments (resulted from the Python's `argparse` package).
+
+In this function, you can run Podman (e.g. `await compose.podman.run(['inspect', 'something'])`),
+access `compose.pods`, `compose.containers` etc.
+
+Here is an example:
+
+```python
 @cmd_run(podman_compose, 'build', 'build images defined in the stack')
 async def compose_build(compose, args):
     await compose.podman.run(['build', 'something'])
@@ -72,31 +91,36 @@ async def compose_build(compose, args):
 
 ## Command arguments parsing
 
-Add a function that accept `parser` which is an instance from `argparse`.
+To add arguments to be parsed by a command, you need to add a function that is decorated with
-In side that function you can call `parser.add_argument()`.
+`@cmd_parse` which accepts the compose instance and the command's name (as a string list or as a
-The function decorated with `@cmd_parse` accepting the compose instance,
+single string).
-and command names (as a list or as a string).
-You can do this multiple times. 
 
-Here is an example
+The decorated function should accept a single argument: An instance of `argparse`.
 
-```
+In this function, you can call `parser.add_argument()` to add a new argument to the command.
+
+Note you can add such a function multiple times.
+
+Here is an example:
+
+```python
 @cmd_parse(podman_compose, 'build')
 def compose_build_parse(parser):
     parser.add_argument("--pull",
         help="attempt to pull a newer version of the image", action='store_true')
     parser.add_argument("--pull-always",
-        help="attempt to pull a newer version of the image, Raise an error even if the image is present locally.", action='store_true')
+        help="Attempt to pull a newer version of the image, "
+             "raise an error even if the image is present locally.",
+        action='store_true')
 ```
 
-NOTE: `@cmd_parse` should be after `@cmd_run`
+NOTE: `@cmd_parse` should be after `@cmd_run`.
 
-## Calling a command from inside another
+## Calling a command from another one
 
-If you need to call `podman-compose down` from inside `podman-compose up`
+If you need to call `podman-compose down` from `podman-compose up`, do something like:
-do something like:
 
-```
+```python
 @cmd_run(podman_compose, 'up', 'up desc')
 async def compose_up(compose, args):
     await compose.commands['down'](compose, args)
@@ -104,8 +128,8 @@ async def compose_up(compose, args):
     await compose.commands['down'](argparse.Namespace(foo=123))
 ```
 
-
 ## Missing Commands (help needed)
+
 ```
   bundle             Generate a Docker bundle from the Compose file
   create             Create services

podman_compose.py

@@ -527,7 +527,11 @@ async def get_mount_args(compose, cnt, volume):
     return ["--mount", args]
 
 
-def get_secret_args(compose, cnt, secret):
+def get_secret_args(compose, cnt, secret, podman_is_building=False):
+    """
+    podman_is_building: True if we are preparing arguments for an invocation of "podman build"
+                        False if we are preparing for something else like "podman run"
+    """
     secret_name = secret if is_str(secret) else secret.get("source", None)
     if not secret_name or secret_name not in compose.declared_secrets.keys():
         raise ValueError(f'ERROR: undeclared secret: "{secret}", service: {cnt["_service"]}')
@@ -543,16 +547,33 @@ def get_secret_args(compose, cnt, secret):
     mode = None if is_str(secret) else secret.get("mode", None)
 
     if source_file:
-        if not target:
+        # assemble path for source file first, because we need it for all cases
-            dest_file = f"/run/secrets/{secret_name}"
-        elif not target.startswith("/"):
-            sec = target if target else secret_name
-            dest_file = f"/run/secrets/{sec}"
-        else:
-            dest_file = target
         basedir = compose.dirname
         source_file = os.path.realpath(os.path.join(basedir, os.path.expanduser(source_file)))
-        volume_ref = ["--volume", f"{source_file}:{dest_file}:ro,rprivate,rbind"]
+
+        if podman_is_building:
+            # pass file secrets to "podman build" with param --secret
+            if not target:
+                secret_id = secret_name
+            elif "/" in target:
+                raise ValueError(
+                    f'ERROR: Build secret "{secret_name}" has invalid target "{target}". '
+                    + "(Expected plain filename without directory as target.)"
+                )
+            else:
+                secret_id = target
+            volume_ref = ["--secret", f"id={secret_id},src={source_file}"]
+        else:
+            # pass file secrets to "podman run" as volumes
+            if not target:
+                dest_file = f"/run/secrets/{secret_name}"
+            elif not target.startswith("/"):
+                sec = target if target else secret_name
+                dest_file = f"/run/secrets/{sec}"
+            else:
+                dest_file = target
+            volume_ref = ["--volume", f"{source_file}:{dest_file}:ro,rprivate,rbind"]
+
         if uid or gid or mode:
             sec = target if target else secret_name
             log.warning(
@@ -2140,7 +2161,7 @@ async def build_one(compose, args, cnt):
         raise OSError("Dockerfile not found in " + ctx)
     build_args = ["-f", dockerfile, "-t", cnt["image"]]
     for secret in build_desc.get("secrets", []):
-        build_args.extend(get_secret_args(compose, cnt, secret))
+        build_args.extend(get_secret_args(compose, cnt, secret, podman_is_building=True))
     for tag in build_desc.get("tags", []):
         build_args.extend(["-t", tag])
     if "target" in build_desc:

setup.py

@@ -40,7 +40,7 @@ setup(
         "pyyaml",
         "python-dotenv",
     ],
-    extras_require={"devel": ["ruff", "pre-commit", "coverage", "parameterize"]},
+    extras_require={"devel": ["ruff", "pre-commit", "coverage", "parameterized"]},
     # test_suite='tests',
     # tests_require=[
     #     'coverage',

tests/build_secrets/Dockerfile

@@ -0,0 +1,9 @@
+FROM busybox
+
+RUN --mount=type=secret,required=true,id=build_secret \
+    ls -l /run/secrets/ && cat /run/secrets/build_secret
+
+RUN --mount=type=secret,required=true,id=build_secret,target=/tmp/secret \
+    ls -l /run/secrets/ /tmp/ && cat /tmp/secret
+
+CMD [ 'echo', 'nothing here' ]

tests/build_secrets/docker-compose.yaml

@@ -0,0 +1,22 @@
+version: "3.8"
+
+services:
+  test:
+    image: test
+    secrets:
+      - run_secret                  # implicitly mount to /run/secrets/run_secret
+      - source: run_secret
+        target: /tmp/run_secret2    # explicit mount point
+
+    build:
+      context: .
+      secrets:
+        - build_secret              # can be mounted in Dockerfile with "RUN --mount=type=secret,id=build_secret"
+        - source: build_secret
+          target: build_secret2     # rename to build_secret2
+
+secrets:
+  build_secret:
+    file: ./my_secret
+  run_secret:
+    file: ./my_secret

tests/build_secrets/docker-compose.yaml.invalid

@@ -0,0 +1,18 @@
+version: "3.8"
+
+services:
+  test:
+    image: test
+    build:
+      context: .
+      secrets:
+        # invalid target argument
+        #
+        # According to https://github.com/compose-spec/compose-spec/blob/master/build.md, target is
+        # supposed to be the "name of a *file* to be mounted in /run/secrets/". Not a path.
+        - source: build_secret
+          target: /build_secret
+
+secrets:
+  build_secret:
+    file: ./my_secret

tests/build_secrets/my_secret

@@ -0,0 +1 @@
+important-secret-is-important

tests/test_podman_compose_build_secrets.py

@@ -0,0 +1,90 @@
+# SPDX-License-Identifier: GPL-2.0
+
+
+"""Test how secrets in files are passed to podman."""
+
+import os
+import subprocess
+import unittest
+
+from .test_podman_compose import podman_compose_path
+from .test_podman_compose import test_path
+
+
+def compose_yaml_path():
+    """ "Returns the path to the compose file used for this test module"""
+    return os.path.join(test_path(), "build_secrets")
+
+
+class TestComposeBuildSecrets(unittest.TestCase):
+    def test_run_secret(self):
+        """podman run should receive file secrets as --volume
+
+        See build_secrets/docker-compose.yaml for secret names and mount points (aka targets)
+
+        """
+        cmd = (
+            "coverage",
+            "run",
+            podman_compose_path(),
+            "--dry-run",
+            "--verbose",
+            "-f",
+            os.path.join(compose_yaml_path(), "docker-compose.yaml"),
+            "run",
+            "test",
+        )
+        p = subprocess.run(
+            cmd, stdout=subprocess.PIPE, check=False, stderr=subprocess.STDOUT, text=True
+        )
+        self.assertEqual(p.returncode, 0)
+        secret_path = os.path.join(compose_yaml_path(), "my_secret")
+        self.assertIn(f"--volume {secret_path}:/run/secrets/run_secret:ro,rprivate,rbind", p.stdout)
+        self.assertIn(f"--volume {secret_path}:/tmp/run_secret2:ro,rprivate,rbind", p.stdout)
+
+    def test_build_secret(self):
+        """podman build should receive secrets as --secret, so that they can be used inside the
+        Dockerfile in "RUN --mount=type=secret ..." commands.
+
+        """
+        cmd = (
+            "coverage",
+            "run",
+            podman_compose_path(),
+            "--dry-run",
+            "--verbose",
+            "-f",
+            os.path.join(compose_yaml_path(), "docker-compose.yaml"),
+            "build",
+        )
+        p = subprocess.run(
+            cmd, stdout=subprocess.PIPE, check=False, stderr=subprocess.STDOUT, text=True
+        )
+        self.assertEqual(p.returncode, 0)
+        secret_path = os.path.join(compose_yaml_path(), "my_secret")
+        self.assertIn(f"--secret id=build_secret,src={secret_path}", p.stdout)
+        self.assertIn(f"--secret id=build_secret2,src={secret_path}", p.stdout)
+
+    def test_invalid_build_secret(self):
+        """build secrets in docker-compose file can only have a target argument without directory
+        component
+
+        """
+        cmd = (
+            "coverage",
+            "run",
+            podman_compose_path(),
+            "--dry-run",
+            "--verbose",
+            "-f",
+            os.path.join(compose_yaml_path(), "docker-compose.yaml.invalid"),
+            "build",
+        )
+        p = subprocess.run(
+            cmd, stdout=subprocess.PIPE, check=False, stderr=subprocess.STDOUT, text=True
+        )
+        self.assertNotEqual(p.returncode, 0)
+        self.assertIn(
+            'ValueError: ERROR: Build secret "build_secret" has invalid target "/build_secret"',
+            p.stdout,
+        )