Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
package sif
|
2021-09-27 22:02:53 +08:00
|
|
|
|
|
|
|
|
import (
|
|
|
|
|
"context"
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
"errors"
|
2021-09-27 22:02:53 +08:00
|
|
|
"fmt"
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
"path/filepath"
|
2021-09-27 22:02:53 +08:00
|
|
|
"strings"
|
|
|
|
|
|
|
|
|
|
"github.com/containers/image/v5/directory/explicitfilepath"
|
|
|
|
|
"github.com/containers/image/v5/docker/reference"
|
2022-06-11 03:49:31 +08:00
|
|
|
"github.com/containers/image/v5/internal/image"
|
2021-09-27 22:02:53 +08:00
|
|
|
"github.com/containers/image/v5/transports"
|
|
|
|
|
"github.com/containers/image/v5/types"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
func init() {
|
|
|
|
|
transports.Register(Transport)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Transport is an ImageTransport for SIF images.
|
|
|
|
|
var Transport = sifTransport{}
|
|
|
|
|
|
|
|
|
|
type sifTransport struct{}
|
|
|
|
|
|
|
|
|
|
func (t sifTransport) Name() string {
|
|
|
|
|
return "sif"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// ParseReference converts a string, which should not start with the ImageTransport.Name prefix, into an ImageReference.
|
|
|
|
|
func (t sifTransport) ParseReference(reference string) (types.ImageReference, error) {
|
|
|
|
|
return NewReference(reference)
|
|
|
|
|
}
|
|
|
|
|
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// ValidatePolicyConfigurationScope checks that scope is a valid name for a signature.PolicyTransportScopes keys
|
|
|
|
|
// (i.e. a valid PolicyConfigurationIdentity() or PolicyConfigurationNamespaces() return value).
|
|
|
|
|
// It is acceptable to allow an invalid value which will never be matched, it can "only" cause user confusion.
|
|
|
|
|
// scope passed to this function will not be "", that value is always allowed.
|
|
|
|
|
func (t sifTransport) ValidatePolicyConfigurationScope(scope string) error {
|
|
|
|
|
if !strings.HasPrefix(scope, "/") {
|
|
|
|
|
return fmt.Errorf("Invalid scope %s: Must be an absolute path", scope)
|
|
|
|
|
}
|
|
|
|
|
// Refuse also "/", otherwise "/" and "" would have the same semantics,
|
|
|
|
|
// and "" could be unexpectedly shadowed by the "/" entry.
|
|
|
|
|
if scope == "/" {
|
|
|
|
|
return errors.New(`Invalid scope "/": Use the generic default scope ""`)
|
|
|
|
|
}
|
|
|
|
|
cleaned := filepath.Clean(scope)
|
|
|
|
|
if cleaned != scope {
|
|
|
|
|
return fmt.Errorf(`Invalid scope %s: Uses non-canonical format, perhaps try %s`, scope, cleaned)
|
|
|
|
|
}
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// sifReference is an ImageReference for SIF images.
|
|
|
|
|
type sifReference struct {
|
|
|
|
|
// Note that the interpretation of paths below depends on the underlying filesystem state, which may change under us at any time!
|
|
|
|
|
// Either of the paths may point to a different, or no, inode over time. resolvedFile may contain symbolic links, and so on.
|
|
|
|
|
|
|
|
|
|
// Generally we follow the intent of the user, and use the "file" member for filesystem operations (e.g. the user can use a relative path to avoid
|
|
|
|
|
// being exposed to symlinks and renames in the parent directories to the working directory).
|
|
|
|
|
// (But in general, we make no attempt to be completely safe against concurrent hostile filesystem modifications.)
|
|
|
|
|
file string // As specified by the user. May be relative, contain symlinks, etc.
|
|
|
|
|
resolvedFile string // Absolute file path with no symlinks, at least at the time of its creation. Primarily used for policy namespaces.
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// There is no sif.ParseReference because it is rather pointless.
|
|
|
|
|
// Callers who need a transport-independent interface will go through
|
|
|
|
|
// sifTransport.ParseReference; callers who intentionally deal with SIF files
|
|
|
|
|
// can use sif.NewReference.
|
|
|
|
|
|
2021-09-27 22:02:53 +08:00
|
|
|
// NewReference returns an image file reference for a specified path.
|
|
|
|
|
func NewReference(file string) (types.ImageReference, error) {
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// We do not expose an API supplying the resolvedFile; we could, but recomputing it
|
|
|
|
|
// is generally cheap enough that we prefer being confident about the properties of resolvedFile.
|
2021-09-27 22:02:53 +08:00
|
|
|
resolved, err := explicitfilepath.ResolvePathToFullyExplicit(file)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return nil, err
|
|
|
|
|
}
|
|
|
|
|
return sifReference{file: file, resolvedFile: resolved}, nil
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (ref sifReference) Transport() types.ImageTransport {
|
|
|
|
|
return Transport
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// StringWithinTransport returns a string representation of the reference, which MUST be such that
|
|
|
|
|
// reference.Transport().ParseReference(reference.StringWithinTransport()) returns an equivalent reference.
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// NOTE: The returned string is not promised to be equal to the original input to ParseReference;
|
2023-02-03 03:17:01 +08:00
|
|
|
// e.g. default attribute values omitted by the user may be filled in the return value, or vice versa.
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// WARNING: Do not use the return value in the UI to describe an image, it does not contain the Transport().Name() prefix;
|
|
|
|
|
// instead, see transports.ImageName().
|
2021-09-27 22:02:53 +08:00
|
|
|
func (ref sifReference) StringWithinTransport() string {
|
2021-11-17 23:35:56 +08:00
|
|
|
return ref.file
|
2021-09-27 22:02:53 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// DockerReference returns a Docker reference associated with this reference
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// (fully explicit, i.e. !reference.IsNameOnly, but reflecting user intent,
|
|
|
|
|
// not e.g. after redirect or alias processing), or nil if unknown/not applicable.
|
2021-09-27 22:02:53 +08:00
|
|
|
func (ref sifReference) DockerReference() reference.Named {
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// PolicyConfigurationIdentity returns a string representation of the reference, suitable for policy lookup.
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// This MUST reflect user intent, not e.g. after processing of third-party redirects or aliases;
|
|
|
|
|
// The value SHOULD be fully explicit about its semantics, with no hidden defaults, AND canonical
|
|
|
|
|
// (i.e. various references with exactly the same semantics should return the same configuration identity)
|
|
|
|
|
// It is fine for the return value to be equal to StringWithinTransport(), and it is desirable but
|
|
|
|
|
// not required/guaranteed that it will be a valid input to Transport().ParseReference().
|
|
|
|
|
// Returns "" if configuration identities for these references are not supported.
|
2021-09-27 22:02:53 +08:00
|
|
|
func (ref sifReference) PolicyConfigurationIdentity() string {
|
|
|
|
|
return ref.resolvedFile
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// PolicyConfigurationNamespaces returns a list of other policy configuration namespaces to search
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// for if explicit configuration for PolicyConfigurationIdentity() is not set. The list will be processed
|
|
|
|
|
// in order, terminating on first match, and an implicit "" is always checked at the end.
|
|
|
|
|
// It is STRONGLY recommended for the first element, if any, to be a prefix of PolicyConfigurationIdentity(),
|
|
|
|
|
// and each following element to be a prefix of the element preceding it.
|
2021-09-27 22:02:53 +08:00
|
|
|
func (ref sifReference) PolicyConfigurationNamespaces() []string {
|
|
|
|
|
res := []string{}
|
|
|
|
|
path := ref.resolvedFile
|
|
|
|
|
for {
|
|
|
|
|
lastSlash := strings.LastIndex(path, "/")
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
if lastSlash == -1 || lastSlash == 0 {
|
2021-09-27 22:02:53 +08:00
|
|
|
break
|
|
|
|
|
}
|
|
|
|
|
path = path[:lastSlash]
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
res = append(res, path)
|
2021-09-27 22:02:53 +08:00
|
|
|
}
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// Note that we do not include "/"; it is redundant with the default "" global default,
|
|
|
|
|
// and rejected by sifTransport.ValidatePolicyConfigurationScope above.
|
2021-09-27 22:02:53 +08:00
|
|
|
return res
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NewImage returns a types.ImageCloser for this reference, possibly specialized for this ImageTransport.
|
|
|
|
|
// The caller must call .Close() on the returned ImageCloser.
|
|
|
|
|
// NOTE: If any kind of signature verification should happen, build an UnparsedImage from the value returned by NewImageSource,
|
|
|
|
|
// verify that UnparsedImage, and convert it into a real Image via image.FromUnparsedImage.
|
|
|
|
|
// WARNING: This may not do the right thing for a manifest list, see image.FromSource for details.
|
|
|
|
|
func (ref sifReference) NewImage(ctx context.Context, sys *types.SystemContext) (types.ImageCloser, error) {
|
2022-06-11 04:10:33 +08:00
|
|
|
return image.FromReference(ctx, sys, ref)
|
2021-09-27 22:02:53 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NewImageSource returns a types.ImageSource for this reference.
|
|
|
|
|
// The caller must call .Close() on the returned ImageSource.
|
|
|
|
|
func (ref sifReference) NewImageSource(ctx context.Context, sys *types.SystemContext) (types.ImageSource, error) {
|
|
|
|
|
return newImageSource(ctx, sys, ref)
|
|
|
|
|
}
|
|
|
|
|
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
// NewImageDestination returns a types.ImageDestination for this reference.
|
|
|
|
|
// The caller must call .Close() on the returned ImageDestination.
|
2021-09-27 22:02:53 +08:00
|
|
|
func (ref sifReference) NewImageDestination(ctx context.Context, sys *types.SystemContext) (types.ImageDestination, error) {
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
return nil, errors.New(`"sif:" locations can only be read from, not written to`)
|
2021-09-27 22:02:53 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// DeleteImage deletes the named image from the registry, if supported.
|
|
|
|
|
func (ref sifReference) DeleteImage(ctx context.Context, sys *types.SystemContext) error {
|
Extensive refactoring to address review comments and hopefully simplify
Fix policy configuration identities in sif
- Actually allow something in ValidatePolicyConfigurationScope ;
SIF is one of the cases where it's actually a bit plausible
that a policy rejecting some filesystem sources might be desirable.
- Fix PolicyConfigurationNamespaces not to include the file name itself,
and "/"
Add tests for sifTransport and sifReference
The NewImage and NewImageSource tests are rather
pointless, but we don't want to require and invoke
fakeroot etc. on every unit test run, at least for now.
Use ref.file instead of ref.resolvedFile in newImageSource
Consistently with the dir: design, if the user specifies a relative
path, use it directly so that we don't introduce races against
changes to the directory structure.
Beautify sif_transport.go
- Use the usual order (transport implementation, followed by
reference implementation)
- Copy&paste more of the comments, to reinforce the contract
requirements.
Fix the package name directive
Reorganize imports
... to follow the usual convention.
Don't use pkg/errors in sif.
Mostly replace its uses with fmt.Errorf(...%w...).
Fix uses of fmt.Errorf
- Use %w instead of %v for error wrapping
- Use errors.New when the string is constant
Don't prefix wrapped error context with "error "
... to match most of c/image code, where we have previously
removed such prefixes.
Return true from HasThreadSafeGetBlob
... because that's the case for the current implementation,
although it makes no difference for the current c/image/copy
caller, when this source only provides one layer.
Rename sifImageSource.blobID to blobDigest
Rename sifImageSource.configID to configDigest
Remove workdir if newImageSource fails
Rename sifImageSource.blob* to layer*
... to differentiate the layer data from the config, which
is also a "blob" in the ImageSource naming.
Remove sifImageSource.diffID
The value only needs to be known inside newImageSource,
so pass it around in a variable/return value.
Remove unused sifImageSource.diffSize
... which allows us to make getLayerInfo
a function without a (partially-created) sifImageSource parent
object.
Move layerTime computation from createBlob to getBlobInfo
If anything, the latter is a bit more accurate (capturing
the time of the last update of the file we are creating, vs.
the time of the initial creation), but we want to eventually
that with a value from the SIF header anyway.
Remove sifImageSource.layerTime
It's only necessary in newImageSource, so use
a return value and a local variable for that.
Remove unused sifImageSource.layerType
Remove sifImageSource.configSize
This value is already stored in the sifImageSource.config slice,
so don't store a redundant copy.
Rename workdir to workDir
following usual Go patterns.
Rename tarpath to tarPath everywhere
Return layerDigest and layerSize from getBlobInfo
... instead of writing it to partially-initialized sifImageSource.
Provide a path to getBlobInfo instead of reading it from sifImageSource
This makes getBlobInfo independent of the partially-created sifImageSource.
Don't create a compressed layer from the SIF file
Compression is very costly, principially in CPU time.
Many use cases (notably import to c/storage) would
only end up decompressing the data again.
Those that do neeed the data compressed, like push
to a registry, can use the copy pipeline's streaming compression
implementation, often without needing to store the compressed
version in a temporary file.
So this is likely to improve both CPU time usage and (maximum) disk
space usage - at the very least against the current implementation
which doens't even remove the uncompressed version after creating
the compressed one :)
This is a minimal version of the change, we are now computing
the layer's digest twice. We'll fix that soon.
Rename tarPath to layerPath
... to be consistent with the other variables.
Don't compute the DiffID separately
It's the same value as the layer digest, now that
the layer is just the uncompressed tarball.
Rename fgz to f
The file is now expected to not be compressed.
Rename blobDigester to digester
... just to be a bit shorter.
Inline some single-use variables when building the manifest
Use a struct initializer instead of a set of assignments for config
Inline single-use variables when building a config
Also remove some fairly redundant comments.
Use a switch if sifImageSource.GetBlob
... to make the structure a tiny bit less repetitive.
Close the SIF image object in newImageSource
Nothing actually needs it afterwards.
Rename UnloadSIFImage() to Close() to indirectly silence
a linter about handling the error; it can't fail in practice,
and isn't quite worth handling.
Explicitly specify a MediaType field in the generated OCI manifest
... to follow best practices vs. schema confusion attacks
(although this generated manfiest is clearly not a schema confusion
attack).
Beautify loadedSifImage.Close
Turn loadedSifImage.GetConfig into CommandLine
- Make loadedSifImage independent of the OCI format details.
- Make it clear at the call site that only the command is actually
provided.
- Don't return an error value which is always nil, which makes the caller
simpler.
Remove lookup of the sif.DataEnvVar descriptor
It is unused in this codebase, and it's unclear
what, if anything, it is used for anywhere else.
Make SifImage.parseEnvironment and SifImage.parseRunscript stand-alone
... i.e. independent from SifImage, so that we can more
easily unit-test it.
The res *[]string parameter is rather ugly, but we'll
refactor it away soon enough.
Should not change behavior.
Split parseDefFile from SifImage.generateConfig
... to have an easily unit-testable bit of code.
Should not change behavior. This destructively assigns
to image.envlist and image.cmdlist instead of appending,
but it should be the only writer at that point.
Add a smoke test for parseDefFile
Use a state machine for parseDefFile
... instead of a nesting scanner.Scan() loops
and a goto.
Should not change behavior.
Remove a misleading comment
Now, with GetDescriptor, more than one matching
descriptor results in an error, so there isn't anything
to assume about a single value.
Move DataDeffile descriptor lookup into generateConfig
It's the only user of that data
Remove deffile and defReader from loadedSifImage
Turn them into trivial local variables in generateConfig()
The code remains a big convoluted, we'll clean that up soon.
Simplify generateConfig
Eliminate both deffile and defReader.
Pass %environment and %runscript to generateRunscript explicitly
That will eventually make it easier to unit-test
Return the generated script from generateRunscript instad of updating image
This makes generateRunscript stand-alone and easy to unit-test.
Add a smoke test for generateRunscript
It's not much, but better than nothing.
Use InjectedScript instead of Runscript for the script we generate
... everywhere, to differentiate that script from the %runscript
section contents.
Store injectedScript as a []byte instead of bytes.Buffer
No need to keep around the intermediate form, and this allows
us to change the implementation.
Use strings.Join and Sprintf instead of bytes.Buffer in generateInjectedScript
Assuming this is not performance-critical, the code is much shorter,
and clearly cannot fail (just like the previous version, which is documented
to panic rather than return the errors that version unnecessarily handled).
Note that this might change behavior for empty %environment or %runscript
sections: we now add extra empty lines. That shouldn't make a difference.
Remove the unnecessary error return value from generateInjectedScript
Remove loadedSifImage.envlist
All users are local to generateConfig.
Don't use loadedSifImage.cmdlist for storing %runscript
It's just a local value to generateConfig, and we no longer
use cmdlist for both %runscript and the final command line.
Replace loadedSifImage.cmdlist with a single command string
The array only ever has one element, so get rid of the array.
Beautify generateConfig
Always refer to environment and runscript in the same order.
Move generateInjectedScript after parseDefFile
First create the data, then consume it.
Simplify generateConfig
Only have the fallback to "bash" if no script is
available in a single place.
Rename resultDesc to desc
For such a short-lived variable we can have a shorter name.
Rename tempdir to tempDir throughout
... to follow Go conventions.
Remove a Sync call on the squashfs copy
We'd actually prefer that data not to hit the disk;
we want to remove it as soon as possible.
Instead, scope the deferred Close() so that it happens
before we consume the file.
Pass around squashFSPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Pass around tarPath in a variable.
... instead of providing an ambient constant for the relative
path.
That will make it clearer which code uses that file.
Remove the generated tar file on failure
e.g. if creating it runs out of space.
Beautify SquashFSToTarLayer
Explicitly return values instead of relying on named return
values to make the data (in this case, error) flow more explicit.
Use Sprintf instead of string concatenation for the generated script
It is a bit more manageable that way.
Also actually start the script with a recognized shebang instead
of a newline.
Don't hard-code "squashfs-root" all over the place.
Pass extractedRootPath around instead, and use (unsquashfs -d) to
override the built-in default.
Inline a single-use cmd variable
Rename xcmd to cmd
... now that the cmd name is available.
Use explicit return statements instead of named return values
Make loadedImage always passed by reference
The struct contains a stateful *sif.FileImage, which
makes no sense to copy; so don't get into that habit
even in cases where it might be safe.
Use a constant for the /podman/runscript path
... instead of hard-coding it over the place, and even
assuming a specific directory structure.
Add more context to write failures
Don't write to stderr; return error output to the caller
Remove the generated script immediately after using it
Make the tar file creation cancellable using the provided context
Also add TODO notes in other places where we would prefer
the copy to be cancellable.
Rename runUnSquashFSTar to createTarFromSIFInputs
We are going to have it handle the injectedScript
as well.
Pass scriptPath to exec.Command instead of hard-coding a constant
This allows us not to care about the working directory of the
script, as well.
Move the scriptPath decision to SquashFSToTarLayer
That's the only place that is aware of tempDir now.
Pass injectedScript to writeInjectedScript
... to make it independent of loadedSifImage; it
will go away entirely soon.
Call writeInjectedScript from createTarFromSIFInputs
... so that createTarFromSIFInputs is responsible for both
creating and consuming extractedRootPath, without any external
interference.
Move the cleanup of extractedRootPath to createTarFromSIFInputs
... to make it a tiny bit more self-contained, now that it
handles the injectedScript as well.
Add a comment to more clearly document the alocation of paths in SquashFSToTarLayer
Remove loadedSifImage.rootfs
Instead, determine the value in SquashFSToTarLayer.
This means that we now try to interpret the deffile before checking
for rootfs presence, changing the possible order of errors. That shouldn't
be much of a difference for valid images.
Rename generateConfig to processDefFile
Have processDefFile return the values instead of writing to loadedSifImage
We will eliminate the loadedSifImage members entirely soon.
Pass sif.FileImage to processDefFile explicitly
... instead of using the image.fimg member.
Rename SquashFSToTarLayer to convertSIFToElements
We are going to have it return other values as well.
Return also the command line from convertSIFToElements
This turns it into the central point of the conversion
process, instead of the fairly ambient loadedSifImage object.
Call processDefFile only in convertSIFToElements
This allows us to remove loadedSifImage.injectedScript and
loadedSifImage.command, making loadedSifImage finally
a trivial wrapper around sif.FileImage - and we'll eliminate
that wrapper next.
Inline loadedSifImage.GetSIFID
We don't really need that abstraction.
Inline loadedSifImage.GetSIFArch
We don't really need that abstraction.
Make convertSIFToElements a stand-alone function
Eliminating the last non-trivial user of loadedSifImage.
Eliminate loadedSifImage
Finally, eliminate the loadedSifImage type entirely.
It doesn't really make sense to inject a layer of abstraction
between sifImageSource and sif.FileImage, purely for the abstraction.
loadedSifImage was only ever used in one way, as an essentially
procedural step; that is now served by the convertSIFToElements
function, rather than being split between the loadedSifImage constructor
and the original tarball creation method.
(convertSIFToElements might eventually return a struct with named
fields if there were many, but it doesn't make sense for newImageSource
to hold an object and fill it up one step at a time.)
Use the last modification time from the SIF header for OCI creation time
Add a TODO note
Add a note about (unsquashfs -o)
Add debug logs around long-running operations
... and make sure to include paths of the relevant files.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-01-07 07:27:36 +08:00
|
|
|
return errors.New("Deleting images not implemented for sif: images")
|
2021-09-27 22:02:53 +08:00
|
|
|
}
|