create_host_config from docker.utils will be deprecated so that
the new create_host_config has access to the _version so
we can ensure that network_mode only gets set to 'default' by
default if the version is high enough and won't explode.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
By doing this we were over-riding any of the daemon's defaults.
Instead we can send an empty string which docker-py sends on
and the daemon interprets as, 'json-file' as a default if it
hasn't got any other daemon level config options.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
One of the use cases is swarm requires at least : character, so going
from conservative to relaxed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We're going to warn people that allowing a boolean in the environment is
being deprecated, so in a future release we can disallow it. This is to
ensure boolean variables are quoted in strings to ensure they don't get
mis-parsed by YML.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When users were putting true/false/yes/no in the environment key,
the YML parser was converting them into True/False, rather than leaving
them as a string.
This change will force people to put them in quotes, thus ensuring
that the value gets passed through as intended.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
oneOf schema ValidationError takes a little more work to parse and
pull out more detail so we can give a better error message back to
the user.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
English language is a tricky old thing and I've pulled out the validator type
parsing so that we can prefix our validator types with the correct article,
'an' or 'a'.
Doing a bit of extra hard work to ensure the error message is clear and
well constructed english.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
For simple definitions where a field can be multiple types, we can
specify the allowed types in an array. It's simpler and clearer.
This is only applicable to *simple* definitions, like number, string, list,
object without any other constraints.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
'~/' in a path currently doesnt work, you get the following error:
[Errno 2] No such file or directory: u'/home/USER/folder/~/some/path/.yml'
Signed-off-by: Nick H <nick.humrich@gmail.com>
The command value can be a list, which would be a Unix command-line
invocation broken up into individual values, thus needing the ability to
have non unique values.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Previously on Buffy... The process_errors was parsing a load of
ValidationErrors that we get back from jsonschema which included
assumptions about the state of the instance we're validating.
Now it's split in two and we're doing field separate to service,
those assumptions don't hold and we can't logically retrieve the
service_name from the error parsing when we're doing service schema
validation, have to explicitly pass this in.
process_errors is high on my list for some future re-factoring to help
make it a bit clearer, smaller state of doing things.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now validation is split in two, the integration tests helped
highlight some places where the schema definition was incorrect.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If make_service_dict is our factory function then we'll give it the
responsibility of validation/construction and resolving.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now the schema has been split into two, we need to modify the
process_errors function to accomodate.
Previously if an error.path was empty then it meant they were root
errors. Now that service_schema checks after the service has been
resolved, our service name is a key within the dictionary and
so our root error logic check is no longer true.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We want to give feedback to the user as soon as possible about the
validity of the config supplied for the services.
When extending a service, we can validate that the fields are
correct against our schema but we must wait until the *end* of
the extends cycle once all of the extended dicts have been merged
into the service dict, to perform the final validation check on the
config to ensure it is a complete valid service.
Doing this before that had happened resulted in false reports of
invalid config, as common config when split out, by itself, is not
a valid service but *is* valid config to be included.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This refactoring is now really coming together. Construction is
happening in the __init__, which is a constructor and helps
clean up the design and clarity of intent of the code. We can now
see (nearly) everything that is being constructed when a ServiceLoader
is created. It needs all of these data constructs to perform the
domain logic and actions. Which are now clearer to see and moving
more towards the principle of functions doing (mostly)one thing and
function names being more descriptive.
resolve_extends is now concerned with the resolving of extends, rather
than the construction, validation, pre processing and *then* resolving
of extends.
Happy days :)
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Separating out the steps we need to resolve extends, so that it
will be clear to insert pre-processing of interpolation and
validation.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
resolve_environment is specific to ServiceLoader, the function
does not need to be on the global scope, it is a part of the
ServiceLoader object. The environment needs to be resolved
before we can make any service dicts, it belongs in the
constructor.
This is cleaning up the design a little and being clearer
about intent and scope of functions.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Moving service name and dict out of the function make_service_dict
and into __init__. We always call make_service_dict with those so
let's put them in the initialiser. Slightly cleaner design intent.
The whole purpose of the ServiceLoader is to take a
service name&service dictionary then validate, process and return
service dictionaries ready to be created.
This is also another step towards cleaning the code up so we can
interpolate and validate an extended dictionary.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it can be set to ultimately a value of None, when a
config file is read in from stdin, it is not optional.
We kinda make use of it's ability to be set to None in our
tests but functionally and design wise, it is required.
If filename is not set, extends does not work.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Check for this in the init so we can remove the duplication of
raising in further functions.
A ServiceLoader isn't valid without one.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In particular it includes:
- some extension of CONTRIBUTING.md
- one fix for Python 2.6 in tests/integration/cli_test.py
- one fix for Python 3.3 in tests/integration/service_test.py
- removal of unused imports
Make stream_output Python 3-compatible
Signed-off-by: Frank Sachsenheim <funkyfuture@riseup.net>
If we get back an error that wasn't an APIError, it was causing the
thread to hang. This catch all, while I appreciate feels risky to
have a catch all, is better than not catching and silently failing,
with a never ending thread.
If something worse than an APIError has gone wrong, we want to stop
the incredible journey of what we're doing.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In order to validate a service name that has been specified as an
integer we need to run that as a pre-process validation step
*before* we pass the config to be validated against the schema.
It is not possible to validate it *in* the schema, it causes a
type error. Even though a number is a valid service name, it
must be a cast as a string within the yaml to avoid type error.
Taken this opportunity to move the code design in a direction
towards:
1. pre-process
2. validate
3. construct
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We use $ref in the schema to allow us to specify multiple type, eg
command, it can be a string or a list of strings.
It required some extra parsing to retrieve a helpful type to display
in our error message rather than 'string or string'. Which while
correct, is not helpful. We value helpful.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When a schema type is set as unique, we should display the validation
error to indicate that non-unique values have been provided for a key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The validation message was confusing by displaying only 1 level of
property of the service, even if the error was another level down.
Eg. if the 'files' property of 'extends' was the incorrect format,
it was displaying 'an invalid value for 'extends'', rather than
correctly retrieving 'files'.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Unfortunately the way that jsonschema is calling %r on its property
and then encoding the complete message means I've had to do this
manual way of removing the literal string prefix, u'.
eg:
key = 'extends'
message = "Invalid value for %r" % key
error.message = message.encode("utf-8")"
results in:
"Invalid value for u'extends'"
Performing a replace to strip out the extra "u'", does not change the
encoding of the string, it is at this point the character u followed
by a '.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it was intended as a positive to be stricter in validation
it would in fact break backwards compatibility, which we do not
want to be doing.
Consider re-visiting this later and include a deprecation warning if
we want to be stricter.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If a container is in the process of being removed, or removal has
failed, it can sometimes appear in the output of GET /containers/json
but not have a 'Name' key. In that case, rather than crashing, we can
ignore it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Rather than implement the logic a second time, use docker-py
split_port function to test if the ports is valid.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Move validation out into its own file without causing circular
import errors.
Fix some of the tests to import from the right place.
Also fix tests that were not using valid test data, as the validation
schema is now firing telling you that you couldn't "just" have this
dict without a build/image config key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
jsonschema provides a rich error tree of info, by parsing each error
we can pull out relevant info and re-write the error messages.
This covers current error handling behaviour.
This includes new error handling behaviour for types and formatting of
the ports field.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Define a schema that we can pass to jsonschema to validate against the
config a user has supplied. This will help catch a wide variety of common
errors that occur.
If the config does not pass schema validation then it raises an exception
and prints out human readable reasons.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
* Add support for volume_driver parameter in compose yml
* Don't expand volume host paths if a volume_driver is specified
(i.e., disable compose feature "relative to absolute path
transformation" when volume drivers are in use, since volume drivers
can use name where host path is normally specified; this is a
heuristic)
Signed-off-by: Luke Marsden <luke@clusterhq.com>
When an image declares a volume such as `/var/lib/mysql`, and a Compose
file has a line like `./data:/var/lib/mysql/` (note the trailing slash),
Compose creates duplicate volume binds when *recreating* the container.
(The first container is created without a hitch, but contains multiple
entries in its "Volumes" config.)
Fixed by normalizing all paths in volumes config.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Hard-coding the API version to '1.18' with the docker-py constructor will
cause the docker-py logic at
https://github.com/docker/docker-py/blob/master/docker/client.py#L143-L146
to always fail, which will cause authentication issues if you're using a
remote daemon using API version 1.19 - regardless of the API version of
the registry.
Allow the user to set the API version via an environment variable. If
the variable is not present, it will still default to '1.18' like it
does today.
Signed-off-by: Reilly Herrewig-Pope <reilly.herrewigpope@mandiant.com>
It was harder to see when there are errors if they came straight after
the other output. Putting a newline in there gives it a bit of visual
room.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Refactored parallel execute and execute create into a single function
parallel_execute that can now handle both cases. This helps untangle it
from being so tightly coupled to the container.
Updated all the relevant operations to use the refactored function.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The concurrent.futures backport doesn't play well with
KeyboardInterrupt, so I'm using Thread and Queue instead.
Since thread pooling would likely be a pain to implement, I've just
removed `COMPOSE_MAX_WORKERS` for now. We'll implement it later if we
decide we need it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Sometimes, some messages were being executed at the same time, meaning
that the status wasn't being overwritten, it was displaying on a
separate line for both doing and done messages.
Rather than trying to have both sets of statuses being written out
concurrently, we write out all of the doing messages first. Then
the done messages are written out/updated, as they are completed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
There's significant speed improvement by having more workers. This
value still shouldn't cause anyone's machines to melt/explode.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This approach takes the style of replacing the output message, in
place, when the command has finished executing. Bringing it a bit
more inline with what `docker pull` does.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Commands able to use this parallelisation are `stop`, `kill` and `rm`.
We're using a backported function from python 3, to allow us to make
the most of a pool of threads without having to write the low level
code for managing this ourselves.
A default value for number of threads is a low enough number so it
shouldn't cause performance problems but if someone knows the
capability of their system and wants to increase it, they can via
an environment variable DEFAULT_MAX_WORKERS
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When specifying a log_driver you want to specify some options for
the logger as per the docker run --log-opt option. The logger
options are key value pairs.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
By allowing the memswap_limit option to be defined we also need to
check that mem_limit is set, you can't have swap without a limit.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Also warn the user about the one-off containers in the standard error
message about legacy containers.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
This top level function is a test helper, so I've moved it into the
config_test file and updated accordingly.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When building test data using make_service_dict, we need to include
working_dir as it is core to some of the functionality of
ServiceLoader.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If you have an alternate YAML file with different services defined,
containers for those services will be shown in `docker-compose ps` even
if you don't pass that file in.
Furthermore, `docker-compose rm` will claim that it's going to remove
them, but actually won't.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
In our circular reference check the stack was previously off by one,
by not including the current service name that was calling another.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
A circular reference bug occurs when there is a difference in the paths
of the file specified in the extends. So one time it is relative, second
time is absolute thus allowing a further circular reference to occur.
By using absolute paths we can be sure that the service filename check
is correct.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This refactoring allows us to raise an error when there is no
'file' key specified in the .yml and no self.filename set. This
error was specific to the tests, as the tests are
the only place that constructs service dicts without sometimes
setting a filename.
Moving the function within the class as well as it is code that
is exclusively for the use of validating properties for the
ServiceLoader class.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If we're using self.filename, then it's already a full path and we
don't need to splice off the filename.yml just so we can .join it
back together again.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If the 'file' key is not set in the extends_options dict then we
look for the 'service' from within the same file.
Fixes this issue: https://github.com/docker/compose/issues/1237
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The process function contained purely validation checks, so re-named
the function to aid intent clarity.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This commit adds environment variable parsing to the container side
of the volume mapping in configs. The common use case for this is
mounting SSH agent sockets in a container, using code like:
volumes:
- $SSH_AUTH_SOCK:$SSH_AUTH_SOCK
environment:
- SSH_AUTH_SOCK
Signed-off-by: Jeff Kramer <jeff.kramer@voxmedia.com>
As VALID_CHARS is shared with project names, these chars are also
now allowed within project names.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This adds a command 'version' to show software versions information
like Docker does. In addition it includes:
- version of the docker-py-package
- Python-implementation and -version
Signed-off-by: Frank Sachsenheim <funkyfuture@riseup.net>
This fixes a bug where migration would fail with an error if a
downstream container was migrated before its upstream dependencies, due
to `check_for_legacy_containers()` being implicitly called when we fetch
`links`, `volumes_from` or `net` dependencies.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
When an upstream dependency (e.g. a db) has a container but a downstream
service (e.g. a web app) doesn't, a web container is not created on
`docker-compose up`.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
"Define and run multi-container applications with Docker"
Not just development environments, and "complex" is not clear and
not really true.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
- One-off containers were included in the warning log messages, which can
make for unreadable output when there are lots (as there often are).
- Compose was attempting to recreate one-off containers as normal
containers when migrating.
Fixed by implementing the exact naming logic from before we used labels.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- Rename `migration` module to `legacy` to make its legacy-ness explicit
- Move `check_for_legacy_containers` into `legacy` module
- Fix migration test so it can be run in isolation
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
If we're just streaming logs from `docker-compose up`, we don't need
to set AttachStdin/out/err, and doing so results in containers with
different configuration depending on whether `up` or `run` were invoked
with `-d` or not.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
We shouldn't start the container before handing it off to dockerpty -
dockerpty will start it after attaching, which is the correct order.
Otherwise the container might exit before we attach to it, which can
lead to weird bugs.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
The help text of the build subcommand suggested to use 'compose build' (instead of 'docker-compose build') to rebuild images.
Signed-off-by: Simon Herter <sim.herter@gmail.com>
This ensures that the connection is not recycled, which can cause the
Docker daemon to complain if we've already performed another streaming
call such as doing a build.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
linting...
six.string_types
list-of-strings in examples
disallow extra_hosts support for list-of-dicts
A more thorough sets of tests for extra_hosts
Provide better examples
As per @aanand's [comment](https://github.com/docker/compose/pull/1158/files#r28326312)
I think it'd be better to check `if not isinstance(extra_hosts_line,
six.string_types)` and raise an error saying `extra_hosts_config must be
either a list of strings or a string->string mapping`. We shouldn't need
to do anything special with the list-of-dicts case.
order result to work with assert
use set() instead of sort()
Signed-off-by: CJ <lim@chernjie.com>
Does not change directory to the parent with the compose-file found.
Works like passing '--file' or setting 'COMPOSE_FILE' with absolute path.
Resolves issue #946.
Signed-off-by: Aleksandr Vinokurov <aleksandr.vin@gmail.com>
* This fix introduces one side-effect: the build parameter is now
validated early, when the service dicionary is first constructed.
That leads to less scary stack traces when the path is not valid.
* The tests for the changes introduced here alter the fixtures
of those (otherwise unrelated) tests that make use of the 'build:'
parameter)
Signed-off-by: Moysés Borges Furtado <moyses.furtado@wplex.com.br>
This is required for Swarm integration: the cluster needs to know
about config like `links` and `volumes_from` at create time so that it
can co-schedule containers.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Allows overriding a user on the command line from the one specified in
the docker-compose.yml
The added tests verify that a specified user overrides a default
user in the docker-compose.yml file.
Based on commit f2f01e207b491866349db7168e3d48082d7abdda by @chmouel
Signed-off-by: Ian VanSchooten <ian@badgelabsllc.com>
The commands `stop`, `restart`, and `up` now support a flag `--timeout`.
It represents the number of seconds to give the services to comply to
the command. In case of `up`, this is only relevant if running in
attached mode.
Signed-off-by: Paul Horn <knutwalker@gmail.com>
Aanand Prasad
mnowster
Vojta Orgon
Daniel Nephin
Mazz Mosley
Daniel Nephin
Christophe Labouisse
Nick H
Lucas N. Munhoz
Karol Duleba
Lachlan Pease
Joffrey F
Frank Sachsenheim
Joel Hansson
Yuval Kohavi
Luke Marsden
Reilly Herrewig-Pope
Erik Kristensen
Jeff Kramer
Sébastien Gruchet
Dan O'Reilly
Travis Thieman
Ben Firshman
Vincent Demeester
André Martins
Todd Whiteman
lsowen
Harald Albers
delbert@umn.edu
CJ
Simon Herter
xuxinkun
Timothy Van Heest
Michael Chase-Salerno
Kyle Walker
Thomas Desvenain
Sam Wing
Aleksandr Vinokurov
Steven Dake
Moysés Borges
Joseph Page
akoskaaa
Ian VanSchooten
Paul Horn
Gil Clark
Igor Ch
KingsleyKelly