For intermediate stages, it's generally ok to perform steps
separately, however in this case, fixing the links would
happen every time the HTML was generated, so we might as
well do it in the same step, to reduce the size of the local
build cache, as it was adding 54MB for each rebuild:
IMAGE CREATED CREATED BY SIZE COMMENT
7d97d86ae290 3 minutes ago RUN /bin/sh -c find ${TARGET} -type f -name … 53.7MB buildkit.dockerfile.v0
<missing> 3 minutes ago RUN /bin/sh -c jekyll build -d ${TARGET} # b… 375MB buildkit.dockerfile.v0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "fetch-upstream-resources" script also updated the toc.yaml,
based on values in the _config.yml.
As a result, any change in either the "_config.yml" or "toc.yaml"
would also result in the upstream resources to be fetched again.
This patch separates the step to update the toc, so that the
upstream resources can be cached. This _does_ mean that remote
sources can get outdated (only will be rebuilt when changing
ENGINE_BRANCH or DISTRIBUTION_BRANCH). That should not be a
problem for actual deployments, which don't use caching.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "deploybase" stage was only used for the final stage.
This patch inlines the steps from that stage to the final stage
itself. In the process, changing the order of copying the NGINX
configuration file to _after_ the docs were copied. This prevents
having to copy the archived docs when making a change in the NGINX
configuration. The NGINX configuration file _will_ now be copied
on each rebuild, but the file is small, and as such is still faster
than copying all archives.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "deploy-source" stage was added in a previous refactor. While
this stage is useful to "export" all docs (including archives if
needed), and _convenient_ to collect all those files to the final
stage, it caused a regression in performance.
This patch changes the final ("deploy") to copy the "archives" and
"current" docs separately, so that the archived versions can be
cached in the final stage, and don't have to be copied again on
each rebuild.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This part of the script was setting variables for any "_version"
property in the configuration file.
We (currently) only need the engine API version, so simplifying
the code to just do that, and fix some spaces -> tabs indentations.
With this change the script only uses standard (POSIX) scripting,
so switching to use the standard (/bin/sh) shell.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The docs/docker.github.io:nginx-onbuild has not been
updated for a while, and having a local file present
allows for easier updating it (for example, to add
wildcard redirects).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Docker Enterprise actively supports the 17.06, 18.03, 18.09, and
19.03 versions, with 17.06 and 18.03 reaching EOL soon.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Don't require these to be set upfront, as we can create them
locally in the fetch-upstream-resources.sh script
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch adds a stage that only contains the generated files. It can
be used to export the generated HTML for hosting the documentation on a
non-containerised service (e.g. to deploy to an s3 bucket).
When using BuildKit, use the `--output` option to build the files and to
copy them to your local filesystem.
For example, to build current docs, including archives:
DOCKER_BUILDKIT=1 docker build --target=deploy-source --output=./_site .
And to build without archives:
DOCKER_BUILDKIT=1 docker build --target=deploy-source --build-arg ENABLE_ARCHIVES=false --output=./_site .
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This adds a `ENABLE_ARCHIVES` build-arg, which allows building
the documentation without archives.
Note that currently, the archives drop-down is still added
unconditionally (so also included if archives are disabled).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Although the docs still rendered ok, some pieces were missing
when generating the "samples", which resulted in warnings during
build:
```
Adding front-matter to ./_samples/library/rethinkdb/README.md ...
cat: can't open './_samples/boilerplate.txt': No such file or directory
Adding front-matter to ./_samples/library/tomcat/README.md ...
cat: can't open './_samples/boilerplate.txt': No such file or directory
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The implementation of the Dockerfile first copied the current
version of the documentation, to be followed by fetching upstream
resources, and adding the archived documentation.
As a result, the build-cache was "busted" on every change that was
made locally, causing a full rebuild of both "fetching upstream resources",
building the reference documentation, and generating the archived
versions of the documentation.
With the changes made to the "fetch-upstream-resources.sh" script,
it's now possible to build those docs in advance (in a separate build-
stage), and append the pre-built docs _after_ the current version
of the documentation was built.
This patch updates the Dockerfile to build reference-docs and
archived docs in separate build-stages, optimizing the build for
caching (wich the assumption that no changes are made in upstream
resources between builds). Not only caused this builds to take longer
than needed, but each rebuild also caused new "dangling" images to
be created (in case of the "classic" builder), or new build-caches
to be created (when using BuildKit).
Note that when _deploying_ the documentation, no caching should be
used (to enforce a full rebuild of the documentation, and guarantee
that reference-docs are "fresh").
Before this change, rebuilding the documentation (after a previous
build, so with cache present) took ~185 seconds;
```
docker build -t docs .
[+] Building 184.7s (26/26) FINISHED
=> [builder 2/5] COPY . md_source 2.8s
=> [builder 3/5] RUN bash ./md_source/_scripts/fetch-upstream-resources.sh md_source 52.7s
=> [builder 4/5] RUN jekyll build -s md_source -d /usr/share/nginx/html --config md_source/_config.yml 113.0s
=> [builder 5/5] RUN find /usr/share/nginx/html -type f -name '*.html' | grep -vE "v[0-9]+\." | while read i; do sed -i 's#href="https://docs.docker.com/#href="/#g' "$i"; done 4.4s
=> CACHED [stage-1 2/8] COPY --from=docs/docker.github.io:nginx-onbuild /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf 0.0s
=> CACHED [stage-1 3/8] COPY --from=docs/docker.github.io:v17.03 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [stage-1 4/8] COPY --from=docs/docker.github.io:v17.06 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [stage-1 5/8] COPY --from=docs/docker.github.io:v17.09 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [stage-1 6/8] COPY --from=docs/docker.github.io:v17.12 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [stage-1 7/8] COPY --from=docs/docker.github.io:v18.03 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> [stage-1 8/8] COPY --from=builder /usr/share/nginx/html /usr/share/nginx/html 3.7s
=> exporting to image 5.1s
=> => exporting layers 5.1s
=> => writing image sha256:70f77631001bd0271455f893b87dd75a1fd6f5db84a00feb8afe5ddf5a697d7d 0.0s
=> => naming to docker.io/library/docs 0.0s
```
After this change, archives and reference-docs are using the build-cache,
and building was reduced to ~126 seconds (most of that taken by Jekyll to
generate the static HTML);
```
docker build -t docs .
+] Building 126.4s (30/30) FINISHED
...
=> CACHED [upstream-resources 1/4] COPY ./_scripts/fetch-upstream-resources.sh ./_scripts/ 0.0s
=> CACHED [upstream-resources 2/4] COPY ./_config.yml . 0.0s
=> CACHED [upstream-resources 3/4] COPY ./_data/toc.yaml ./_data/ 0.0s
=> CACHED [upstream-resources 4/4] RUN bash ./_scripts/fetch-upstream-resources.sh . 0.0s
=> [current 2/4] COPY --from=upstream-resources /usr/src/app/md_source/. ./ 1.5s
=> [current 3/4] RUN jekyll build -d /usr/share/nginx/html 108.9s
=> [current 4/4] RUN find /usr/share/nginx/html -type f -name '*.html' | grep -vE "v[0-9]+\." | while read i; do sed -i 's#href="https://docs.docker.com/#href="/#g' "$i"; done 4.8s
=> CACHED [deploybase 2/2] COPY --from=docs/docker.github.io:nginx-onbuild /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf 0.0s
=> CACHED [archives 1/5] COPY --from=docs/docker.github.io:v17.03 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [archives 2/5] COPY --from=docs/docker.github.io:v17.06 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [archives 3/5] COPY --from=docs/docker.github.io:v17.09 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [archives 4/5] COPY --from=docs/docker.github.io:v17.12 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> CACHED [archives 5/5] COPY --from=docs/docker.github.io:v18.03 /usr/share/nginx/html /usr/share/nginx/html 0.0s
=> [deploy 1/1] COPY --from=current /usr/share/nginx/html /usr/share/nginx/html 4.1s
=> exporting to image 1.9s
=> => exporting layers 1.9s
=> => writing image sha256:e76359b937fb4d956fb17e889a7333687929d4c9ae093909fe5630a8f3e4de64 0.0s
=> => naming to docker.io/library/docs 0.0s
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This variable is no longer needed, as the scripts are
always executed relative to the current directory (`.`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Go back to small /edge
* Update site front page
Add info about Editions
Reorganize components
Update headings
* Move logic of fetching content to a script
* Add v1.28 API
* Add info about versioned API
The master branch is called trunk in svn parlance
and the directory structure is a little bit different.
Also trust the certificates for the https svn connections,
otherwise we may be prompted.
Fix Dockerfile in vnext-engine
The master branch is called trunk in svn parlance
and the directory structure is a little bit different.
Also trust the certificates for the https svn connections,
otherwise we may be prompted.
The master branch is called trunk in svn parlance
and the directory structure is a little bit different.
Also trust the certificates for the https svn connections,
otherwise we may be prompted.
* Moves to GHP 112, nginx-based server
* Import official nginx instructions per @thaJeztah
* Reordering of commands per @thaJeztah
* Reordering of commands per @thaJeztah
* Make sure that at the end of each layer we only keep the one directory
* Forgot to use md_source when resolving conflicts
* Update Dockerfile
* Use base image, clean up cruft from docsarchive folder
* Update docs-base to be self-browseable, conf source of truth
The files we’re pulling from docker/docker may include links to docs.docker.com. And we can’t forbid that because relative links wouldn’t make sense in the context of docker/docker repository in some situations. So let’s just fix these links right after get imported.
Signed-off-by: Adrien Duermael <adrien@duermael.com>
files in /usr/src/app/allvbuild and /_site are the same. Let’s just serve what’s in /usr/src/app/allvbuild.
Signed-off-by: Adrien Duermael <adrien@duermael.com>
This optimizes the Dockerfile a bit by;
- putting the docs archives at the top to
optimize caching and to prevent having
to clone the github repository on each
build. Note that '--no-cache' is needed
to forcefully break the cache, but the
archives should not frequently change
- grouping RUN lines to reduce image size.
- using a loop for the archived versions to
reduce the amount of duplicated code.
- using the local files for the *current*
version of the docs instead of the git
clone from GitHub. this makes it also
use the right source instead of "master"
- adding a .dockerignore to prevent busting
the cache if not needed, and to prevent
uploading the '.git' repository, which
is not used for the "current" docs
Difference in size before/after;
REPOSITORY TAG IMAGE ID CREATED SIZE
docs latest 36f6ad029e6a 3 minutes ago 1.722 GB
docs-orig latest 4f1a3e3fda4f 16 minutes ago 3.344 GB
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This version introduces the following:
- uses nanosecond timestamps for event
- ensure events are sent once their effect is "live"
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
(cherry picked from commit 29b2714580d085533c29807fa337c2b7a302abb6)
Signed-off-by: Tibor Vass <tibor@docker.com>
following the announcement;
https://groups.google.com/forum/m/#!topic/golang-announce/7JTsd70ZAT0
> [security] Go 1.6.3 and Go 1.7rc2 pre-announcement
>
> Hello gophers,
> We plan to issue Go 1.6.3 and Go 1.7rc2 on Monday July 18 at approximately 2am UTC.
> These are minor release to fix a security issue.
>
> Following our policy at https://golang.org/security, this is the pre-announcement of those releases.
>
> Because we are so late in the release cycle for Go 1.7, we will not issue a minor release of Go 1.5.
> Additionally, we plan to issue Go 1.7rc3 later next week, which will include any changes between 1.7rc1 and tip.
>
> Cheers,
> Chris on behalf of the Go team
**Note:**
the man/Dockerfile is not yet updated, because
the official image for Go 1.6.2 has not yet
been updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 589bafddf391cbf6aff8b22044266dc819cdcaeb)
Signed-off-by: Tibor Vass <tibor@docker.com>
This updates containerd to b93a33be39bc4ef0fb00bfcb79147a28c33d9d43
fixing the start sync issues.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit 7db7e046315eed5ccfb3a913b965be4b1a595601)
Due to the issue of go-md2man, a numbered list in `man docker login` was not rendered correctly.
a8f937e113
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit cfe16e0d5b4bb7d1d194553f2a82cc1879d60e6b)
This bumps containerd to cf554d59dd96e459544748290eb9167f4bcde509 and
includes various fixes and updates the grpc package and types generated
for use.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
the llvm apt repo was removed and finding the exact rev of the llvm
toolchain that worked perfecting in our dockerfile is a problem.
We looked at the reasons why we were using this version of clang and it
appears to be this compiler warning.
`was built for newer OSX version (10.11) than being linked
(10.6)`
When you look at the dockerfile this makes sense.
```
ENV OSX_SDK MacOSX10.11.sdk
ENV OSX_CROSS_COMMIT 8aa9b71a394905e6c5f4b59e2b97b87a004658a4
RUN set -x \
&& export OSXCROSS_PATH="/osxcross" \
&& git clone https://github.com/tpoechtrager/osxcross.git $OSXCROSS_PATH \
&& ( cd $OSXCROSS_PATH && git checkout -q $OSX_CROSS_COMMIT) \
&& curl -sSL https://s3.dockerproject.org/darwin/v2/${OSX_SDK}.tar.xz -o "${OSXCROSS_PATH}/tarballs/${OSX_SDK}.tar.xz" \
&& UNATTENDED=yes OSX_VERSION_MIN=10.6 ${OSXCROSS_PATH}/build.sh
ENV PATH /osxcross/target/bin:$PATH
```
We are basically using the 10.11 sdk but linking to 10.6 as the
min version so this warning should be expected.
Also the docs on the osxcross project require clan 3.2+, not 3.8.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This environment variable is no longer
needed in Go 1.6 (as it's not the default).
Removed this environment variable from
all Dockerfiles except the Dockerfile.s390x,
which is still using gcc 5.3 (Go 1.5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change adds file version information to docker.exe and dockerd.exe by
adding a Windows version resource with the windres tool.
This change adds a dependency to binutils-mingw-w64 on Linux, but removes
a dependency on rsrc. Most Windows build environments should already have
windres if they have gcc (which is necessary to build dockerd).
Signed-off-by: John Starks <jostarks@microsoft.com>
For context: https://github.com/golang/go/issues/15286
This commit downloads go1.5.3 in addition to go1.5.4 in order to
workaround the issue.
It is not expected to do a Docker release without a proper fix, however
this should help unblock Docker development on Windows TP5.
Signed-off-by: Tibor Vass <tibor@docker.com>
This vendors in new spec/runc that supports
setting readonly and masked paths in the
configuration. Using this allows us to make an
exception for `—-privileged`.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This includes fixes for;
- outputing errors for missing seccomp options on seccomp versions < 2.3
- cap set apply EPERM errors on ARM systems
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
runc expects a systemd cgroupsPath to be in slice:scopePrefix:containerName
format and the "--systemd-cgroup" option to be set. Update docker accordingly.
Fixes 21475
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Add some missing dependencies in the Dockerfile:
- apt-utils for apt-ftparchive.conf
- bsdmainutils for our use of the column command in
hack/make/generate-index-listing
We also ensure that the docker daemon is started before calling
release-deb or release-rpm, since .detect-daemon-osarch, which is sourced
in each of them, requires the daemon to be running.
This commit also gets completely rid of s3cmd and fixes references to
AWS_* environment variables (changing from AWS_ACCESS_KEY to
AWS_ACCESS_KEY_ID and AWS_SECRET_KEY to AWS_SECRET_ACCESS_KEY) in order
to please awscli. Also AWS_DEFAULT_REGION is now important to specify,
the default has been set to the region used by get.docker.com and
test.docker.com.
Signed-off-by: Tibor Vass <tibor@docker.com>
Contains fixes for:
- pid.max fix that is causing hang on network stats test.
- fix for early stdin close containerd-shim
- better logging for `could not synchronise with container process`
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fixes#20550
This update to libseccomp supports the new versions of socket
system calls that can be called directly rather than via the
socketcall syscall in kernel versions 4.3 or later with new glibc.
Note this library version now supports s390x and ppc64le, so
seccomp can be potentially be enabled for these architectures now.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Fixes broken-pipe issue when piping s3cmd to grep -q, by removing the -q
flag and redirecting to /dev/null instead.
Add net-tools for ifconfig, because some tests rely on ifconfig.
Harmonize all Dockerfiles in this direction.
Signed-off-by: Tibor Vass <tibor@docker.com>
dockerinit has been around for a very long time. It was originally used
as a way for us to do configuration for LXC containers once the
container had started. LXC is no longer supported, and /.dockerinit has
been dead code for quite a while. This removes all code and references
in code to dockerinit.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
Fixes#19400
Note that this introduces an incompatibility with Docker 1.10-rc1,
because the media type used for schema1 manifests has been corrected in
the upstream distribution code. Docker 1.10-rc1 won't be able to pull
old manifests from Registry 2.3-rc0 and up, but because of this vendor
update, Docker 1.10-rc2 won't have this problem.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Sebastiaan van Stijn
Usha Mandya
Ally Smith
paigehargrave
Maria Bermudez
Joao Fernandes
Misty Stanley-Jones
Misty Stanley-Jones
John Mulhausen
Victor Vieux
Adrien Duermael
Adrien Duermael
Ben Firshman
Kenfe-Mickael Laventure
Justin Cormack
Michael Crosby
Akihiro Suda
Christy Perez
Antonio Murdaca
cyli
Shijiang Wei
Michael Holzheu
John Howard
Mrunal Patel
John Starks
Tibor Vass
Qiang Huang
Tibor Vass
Tonis Tiigi
Ken Cochrane
Sebastiaan van Stijn
David Calavera
Anusha Ragunathan
Harald Albers
Riyaz Faizullabhoy
Alexander Morozov
Vincent Demeester
Arnaud Porterie
unclejack
Jessica Frazelle
Aleksa Sarai
Alexander Morozov
Aaron Lehmann