diff --git a/.gitattributes b/.gitattributes deleted file mode 100644 index 2125666142..0000000000 --- a/.gitattributes +++ /dev/null @@ -1 +0,0 @@ -* text=auto \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index 53e518f780..0000000000 --- a/.github/ISSUE_TEMPLATE.md +++ /dev/null @@ -1,51 +0,0 @@ - - -**Output of `docker version`:** - -``` -(paste your output here) -``` - - -**Output of `docker info`:** - -``` -(paste your output here) -``` - -**Additional environment details (AWS, VirtualBox, physical, etc.):** - - - -**Steps to reproduce the issue:** -1. -2. -3. - - -**Describe the results you received:** - - -**Describe the results you expected:** - - -**Additional information you deem important (e.g. issue happens only occasionally):** diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md deleted file mode 100644 index 426981828b..0000000000 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ /dev/null @@ -1,30 +0,0 @@ - - -**- What I did** - -**- How I did it** - -**- How to verify it** - -**- Description for the changelog** - - - -**- A picture of a cute animal (not mandatory but encouraged)** - diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000..4baf5383a9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,62 @@ +OSS-LICENSES +.DS_Store +shell/shell +shell/shell-history +shell/out.txt +shell/shell-config.json +shell/shell-env.json +shell9pAPI/shell9pAPI +shell9pAPI/sftp/sftp +shell/sftpKeys +shellAgent/shellAgent +shellAgent/hostKey.pem +shell/key.pem +shell/key.pub +agentSDK/node/node_modules/* +!agentSDK/node/node_modules/dockersdk +agentSDK/lib/agentlib.dylib +agentSDK/node/key.pem +agentSDK/node/keyC.pem +agentSDK/node/keyNode.pem +_tests +all-packages.txt + +# ignore compiled binaries +v1/agent/agent +v1/agent/com.docker.agent.exe +v1/cmd/proxy/proxy.exe + +# ignore generated SSH keys +v1/clitool/key.pem +v1/agent/hostKey.pem + +# ignore files generated by cli testing +v1/agent/application_storage +v1/bin +v1/cmd/agent/agent + +v1/db/irmin/_build +v1/db/irmin9p/irmin9p +v1/vendor/github.com/mortdeus/go9p/srv/examples/ufs/ufs + +# ignore compiled boot2docker, since we will rebuild it +v1/boot2docker/boot2docker-data.img.tar +v1/boot2docker/vmlinuz64 +v1/boot2docker/initrd.img +v1/devtool_sdk/libsrpc/libsrpc +v1/devtool_sdk/libsrpc/libsrpc.dylib +v1/devtool_sdk/libsrpc/libsrpc.h + +# ignore temporary files from editors +*.swp +*~ +*.sdf +*.VC.opendb + +_cache + +# PLEASE DON'T PUT SUBPROJECT IGNORE PATTERNS HERE. +# PUT THEM IN THE SUBPROJECT'S .gitignore FILE. +# THIS MAKES MAINTENANCE EASIER. THANK YOU. +v1/mac/dependencies +v1/mac/bin/Windows/Dependencies/HyperVInstaller.ps1 diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 0000000000..d231c24b45 --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,1130 @@ +### 1.12.1-beta27 "Beta27" (unreleased) + +* Upgrades + - Docker 1.12.2-rc1 + - Docker Machine 0.8.2 + - Docker compose 1.8.1 + - Kernel vsock driver v7 + - Kernel 4.4.21 + - aufs 20160912 + +* Bug fixes and minor changes + - Fix an issue where some windows did not claim focus correctly (#5221,#5314) + - Add UI when switching channel to prevent user losing containers and settings (#5253) + - Check disk capacity before toolbox import (#5165) + - Import certificates in etc/ssl/certs/ca-certificates.crt (#5239) + - DNS: reduce the number of UDP sockets consumed on the host + - VPNkit: improve the connection-limiting code to avoid running out of sockets on the host + - UDP: handle diagrams bigger than 2035, up to the configured macOS kernel limit + - UDP: make the forwarding more robust; drop packets and continue rather than stopping + - disk: make the "flush" behaviour configurable for database-like workloads. This works around a performance regression in 1.12.1. + +### 1.12.1-beta26 "Beta26" + +* New + - Improved support for macOS 10.12 Sierra + +* Upgrades + - Linux kernel 4.4.20 + - aufs 20160905 + +* Bug fixes and minor changes + - Fix communications glitch when UI talks to com.docker.vmnetd (#5115) + Fixes https://github.com/docker/for-mac/issues/90 + - UI fix for macOs 10.12 (#5152) + - Windows open on top of full screen app are available in all spaces (#5136) + - Reporting a bug, while not previously logged into github now works. (#5110) + - When a diagnostic upload fails, the error is properly reported (#5116) + - docker-diagnose: display and record the time the diagnosis was captured (#5075) + - Allow ports to be bound on host addresses other than 0.0.0.0 and 127.0.0.1 (moby/#482) + Fixes issue reported in https://github.com/docker/for-mac/issues/68 + - Don't compute the container folder in com.docker.vmnetd (#5066) + Fixes https://github.com/docker/for-mac/issues/47 + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2016-09-07 1.12.1-beta25 "Beta25" + +* Upgrades + - Experimental support for OSX 10.12 Sierra (beta) + +* Bug fixes and minor changes + - VPNKit supports search domains (#4974) + - Entries from /etc/hosts should now resolve from within containers + - osxfs: fix thread leak (#4933) + +* Known issues + - Several problems have been reported on macOS 10.12 Sierra and are being + investigated. This includes failure to launch the app and being unable + to upgrade to a new version. + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2016-08-31 1.12.1-beta24.1 "Beta24.1" + +* Hotfixes + - Fix regression in UI when changing memory/cpu settings + +### 2016-08-30 1.12.1-beta24 "Beta24" + +* Upgrades + - Docker 1.12.1 + - Docker machine 0.8.1 + - Linux kernel 4.4.19 + - aufs 20160822 + +* Bug fixes and minor changes + - osxfs: fixed a malfunction of new directories that have the same + name as an old directory that is still open (#4532) + - osxfs: rename events now trigger DELETE and/or MODIFY inotify + events (saving with TextEdit works now) (#4498) + - slirp: support up to 8 external DNS servers + - slirp: reduce the number of sockets used by UDP NAT, reduce the + probability that NAT rules will time out earlier than expected (#4826) + - Warn the user if BlueStacks is installed (potential kernel panic) (#4661) + +* Known issues + - Several problems have been reported on macOS 10.12 Sierra and are being + investigated. This includes failure to launch the app and being unable + to upgrade to a new version. + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2016-08-16 1.12.1-rc1-beta23 "Beta23" + +* Upgrades + - Docker 1.12.1-rc1 + - Linux kernel 4.4.17 + - aufs 20160808 + +* Bug fixes and minor changes + - Automatic update interval changed from 1 hour to 24 hours (#4822) + - Moby: use default sysfs settings, transparent huge pages disabled (#4815) + - Moby: cgroup mount to support systemd in containers (#4789) + - osxfs: fix an issue that caused inotify failure and crashes (#4799) + - osxfs: fix a directory fd leak (#4533) + - Zsh completions (#4785) + +* Known issues + - Several problems have been reported on macOS 10.12 Sierra and are being + investigated. This includes failure to launch the app and being unable + to upgrade to a new version. + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2015-08-11 1.12.0-beta22 "Beta22" + +* New + +* Upgrades + - Linux kernel to 4.4.16 + +* Bug fixes and minor changes + - Increase Moby fs.file-max to 524288 + - Increase Moby fs.file-max to 524288 + - Use Mac System Configuration database to detect DNS (#4649) + - HyperKit updated with dtrace support and lock fixes + - Fix Moby Diagnostics and Update Kernel (#4636) + - UI Fixes (#4538) + - osxfs: fix socket chowns (#4743) + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2016-08-03 1.12.0-beta21.1 "Beta21.1" + +* Hotfixes + - osxfs: fixed an issue causing access to children of renamed + directories to fail (symptoms: npm failures, apt-get failures) + (docker/for-mac#53 #74 #76) + - osxfs: fixed an issue causing some ATTRIB and CREATE inotify + events to fail delivery and other inotify events to stop (#4650 #4652) + - osxfs: fixed an issue causing all inotify events to stop when an + ancestor directory of a mounted directory was mounted (#4635) + - osxfs: fixed an issue causing volumes mounted under other mounts + to spontaneously unmount (docker/docker#24503) + +### 2016-07-28 1.12.0-beta21 "Beta21" + +* New + - Docker for Mac is now available from 2 channels - stable and beta. + New features and bug fixes will go out first in auto-updates to users + in the beta channel. Updates to the stable channel are much less + frequent and happen in sync with major and minor releases of the Docker + engine. Only features that are well-tested and ready for production are + added to the stable channel releases. You can download Docker for Mac + stable at https://download.docker.com/mac/stable/Docker.dmg + +* Upgrades + - docker 1.12.0 with experimental features + - docker machine 0.8.0 + - docker compose 1.8.0 + +* Bug fixes and minor changes + - Check for updates, auto-update and diagnose can be run by non-admin users (#4441) + - osxfs: fixed an issue causing occasional incorrect short reads (#3876) + - osxfs: fixed an issue causing occasional EIO errors (#3876) + - osxfs: fixed an issue causing inotify creation events to fail (#3876) + - osxfs: increased the fs.inotify.max_user_watches limit in Moby to 524288 + - The UI shows documentation link for sharing volumes (#4482) + - Improved error when running with outdated Virtualbox version (#4450) + - Added link to sources for qemu-img (#4503) + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart Docker.app (#1224) + - There are a number of issues with the performance of + directories bind-mounted with `osxfs`. In particular, writes of + small blocks and traversals of large directories are currently + slow. Additionally, containers that perform large numbers of + directory operations, such as repeated scans of large directory + trees, may suffer from poor performance. More information is + available in the Known Issues section of the documentation at + https://docs.docker.com/docker-for-mac/troubleshoot/#known-issues + + - Under some unhandled error conditions, inotify event delivery can + fail and become permanently disabled. + The workaround is to restart Docker.app (#2181) + +### 2016-07-19 1.12.0-rc4-beta20 "Beta20" + +* New + +* Upgrades + +* Bug fixes and minor changes + - Fix docker.sock permission issues (#3919) + - Don't check for update when the settings panel opens (#4431) + - Remove obsolete DNS workaround (#4432) + - Use the secondary DNS server in more circumstances (#4429) + - Limit the number of concurrent port forwards to avoid running out of resources (#4429) + - Store the database as a "bare" git repo to avoid corruption problems (#4236) + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart Docker.app (#1224) + +### 2016-07-14 1.12.0-rc4-beta19 "Beta19" + +* New + - Add privacy tab in settings (#4190) + - Allow the definition of HTTP proxy overrides in the UI (#3517, #4303) + +* Upgrades + - Docker 1.12.0 RC4 + - Docker Compose 1.8.0 RC2 + - Docker Machine 0.8.0 RC2 + - Linux kernel 4.4.15 + +* Bug fixes and minor changes + - filesystem sharing permissions can only be configured in the UI (no more `/Mac` in moby) (#4312) + - com.docker.osx.xhyve.hyperkit: increase max number of fds to 10240 (#4232) + - Improve Moby syslog facilities (#4261) + - Improve file-sharing tab (#4252) + - com.docker.slirp: include the DNS TCP fallback fix, required when UDP responses are truncated (docker/vpnkit#72) + - docker build/events/logs/stats... won't leak when iterrupted with Ctrl-C (#3628) + +### 2016-07-07 1.12.0-rc3-beta18.1 "Beta-18.1" + +NOTE: Docker 1.12.0 RC3 release introduces a backward incompatible change from RC2, +See https://github.com/docker/docker/issues/24343#issuecomment-230623542 for more details. + +Fix for the above can be found at: https://docs.docker.com/docker-for-mac/troubleshoot/#/recreate-or-update-your-containers-after-beta-18-upgrade + +* Hotfix + - Fix issue resulting in error "Hijack is incompatible with use of CloseNotifier", reverts previous fix for "Ctrl-C" during build. + +* New + - New host/container file sharing UI (#4193, #4191) + - /Mac bind mount prefix is deprecated and will be removed soon + +* Upgrades + - Docker 1.12.0 RC3 (#4199, #4177) + +* Bug fixes and minor changes + - VPNKit: Improved scalability as number of network connections increases (#4173) + - The docker API proxy was failing to deal with some 1.12 features (e.g. health check) + +* Known issues + - See https://docs.docker.com/docker-for-mac/troubleshoot/ + +### 2016-07-06 1.12.0-rc3-beta18 "Beta-18" + +* New + - New host/container file sharing UI (#4193, #4191) + - /Mac bind mount prefix is deprecated and will be removed soon + +* Upgrades + - Docker 1.12.0 RC3 (#4199, #4177) + +* Bug fixes and minor changes + - VPNKit: Improved scalability as number of network connections increases (#4173) + - Interrupting a `docker build` with Ctrl-C will actually stop the build + - The docker API proxy was failing to deal with some 1.12 features (e.g. health check) + +* Known issues + - See https://docs.docker.com/docker-for-mac/troubleshoot/ + +### 2016-06-29 1.12.0-rc2-beta17 "Beta-17" + +* Upgrades + - Linux kernel 4.4.14, aufs 20160627 (#4133) + +* Bug fixes and minor changes + - Documentation moved to https://docs.docker.com/docker-for-mac/ + - Allow non-admin users to launch the app for the first time (using admin creds) (#4060) + - Prompt non-admin users for admin password when needed in Settings (#4046) + - Fixed download links, documentation links (#4059, #4013) + - Fixed "failure: No error" message in diagnostic panel (#4010) + - Improved diagnostics for networking and logs for the service port openers (#4116) + +* Known issues + - See https://docs.docker.com/docker-for-mac/troubleshoot/ + +### 2016-06-17 1.12.0-rc2-beta16 "Beta-16" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + +* Upgrades + - Docker 1.12.0 RC2 + - docker-compose 1.8.0 RC1 + - docker-machine 0.8.0 RC1 + - notary 0.3 + - Alpine 3.4 + +* Bug fixes and minor changes + - VPNKit: fix a regressed error message when a port is in use (#3904) + - Fix UI crashing with 'NSInternalInconsistencyException' / fd leak (#3883) + - HyperKit API: Improved error reporting (#3888) + - osxfs: fix sporadic EBADF due to fd access/release races (#3683) + +* Known issues + - See https://beta.docker.com/docs/mac/troubleshoot/#known-issues + +### 2016-06-10 1.11.2-beta15 "Beta-15" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - Registry mirror and insecure registries can now be configure from Settings + - VM can now be restarted from Settings + - sysctl.conf can be edited from Settings + +* Upgrades + - Docker 1.11.2 + - Linux 4.4.12, aufs 20160530 + +* Bug fixes and minor changes + - Number of concurrent TCP/UDP connections increased in VPNKit + - Hyperkit: vsock stability improvements + - Fixed crash when admin user group does not exit + +* Known issues + - See https://beta.docker.com/docs/mac/troubleshoot/#known-issues + +### 2016-06-06 1.11.1-beta14.1 "Beta-14" + +* Hotfix + - Updated Linux kernel to avoid falsely triggering an alarm in a common antivirus scanner + +### 2016-06-06 1.11.1-beta14 "Beta-14" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - New menu item "Diagnose & Feedback" can now be used to run diagnostics and upload logs to Docker + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart Docker.app (#1224) + +* Bug fixes and minor changes + - osxfs: support statfs (#3536) + - settings: updated toolbar icons (#3476) + - Fall back to secondary DNS server if primary fails (#3458) + - Link to documentation from menu + +### 2016-05-28 1.11.1-beta13.1 "Beta-13.1" + +* Hotfixes + – osxfs: Fix sporadic EBADF errors and End_of_file crashes due to a race corrupting node table invariants (#3454) + – osxfs: Fix a crash after accessing a sibling of a file moved to another directory caused by a node table invariant violation (#3474) + – Proxy settings were applied on network change, causing docker daemon to restart too often + – Log file sizes doubled on docker daemon restart + +### 2016-05-25 1.11.1-beta13 "Beta-13" + +* New + - osxfs: enabled 10ms dcache for 3x speedup on a go list ./... test against docker/machine. Workloads + heavy in file system path resolution (common among dynamic languages and build systems) will have + those resolutions performed in amortized constant time rather than time linear in the depth of the + path so speedups of 2-10x will be common + - Support multiple users on the same machine, non-admin users can use the app as long as vmnetd has been installed. Currently + only one user can be logged in at the same time. + - Basic support for using system HTTP/HTTPS proxy in docker daemon + - Docs have been updated! See https://beta.docker.com/docs/ + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + +* Bug fixes and minor changes + - osxfs: setting atime and mtime of nodes is now supported (#2174) + - osxfs: fixed major regression in Beta 12 with ENOENT, ENOTEMPY, and other spurious errors after + a directory rename. This manifested as `npm install` failure and other directory traversal issues. (#3229) + - osxfs: fixed temporary file ENOENT errors (#3220, #1794) + - osxfs: fixed in-place editing file truncation error (e.g. perl -i) (#3219) + - improve time synchronisation after sleep (#3240) + +### 2016-05-17 1.11.1-beta12 "Beta-12" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + +* Bug fixes and minor changes + - UI improvements (#3138, #3107) + - osxfs: fixed mkdir returns EBUSY but directory is created (#3131) + - osxfs: FUSE 7.23 (#3130) + +### 2016-05-10 1.11.1-beta11 "Beta-11" + +* New + - osxfs now persists ownership changes in an extended attribute, see + https://beta.docker.com/docs/mac/osxfs/ (#823) + +* Upgrades + - docker-compose 1.7.1 + - Kernel 4.4.9 + +* Bug fixes and minor changes + - desktop notifications after successful update (#2947) + - no "update available" popup during install process (#2973) + - fix repeated bind of privileged ports (#3015) + - osxfs: fix the block count reported by stat (#2978) + - Moby: Fix vsock half closed issue (#2961) + - Moby: Add NFS support (#2961) + - Moby: Hostname is now Moby, not Docker (#2961) + - Moby: Fixes to disk formatting scripts (#2961) + +### 2016-05-03 1.11.1-beta10 "Beta-10" + +* New + - Token validation is now done over an actual SSL tunnel (HTTPS). + (should fix issues with antivirus software) + - Docs have been updated! See https://beta.docker.com/docs/ + +* Upgrades + - docker 1.11.1 (#2858, 2808) + +* Bug fixes and minor changes + - UCP now starts again (#2854) + - include debugging symbols in xhyve (#2837) + - vsock stability improvements (#2831) + - addressing glitches in Settings panel + - fix issues impacting the "whale menu" + - fix uninstall process + - xhyve vcpu state machine improvements, may improve suspend/resume (#2798) + +### 2016-04-28 1.11.0-beta9 "Beta-9" + +* New + - `localhost` is now used for port forwarding by default. + `docker.local` will no longer work as of Beta 9. See documentation + for details. + - Docs have been updated! See https://beta.docker.com/docs/ + - New settings window - memory and vCPUs now adjustable (#2634) + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + +* Bug fixes and minor changes + - Fix loopback device naming (#2734) + - Improve docker socket download and osxfs sequential write by ~20% (#2653) + - osxfs: improve sequential read throughput by up to 20% (#2654) + - osxfs: improve readdir performance by up to 6x (#2695) + - osxfs: log all fatal exceptions (#2732) + - more reliable DNS forwarding over UDP and TCP (#2721, #2786) + - UDP ports can be proxied over vsock (#2711) + - fix EADDRINUSE (manifesting as errno 526) when ports are re-used (#2818) + - send ICMP when asked to not fragment and we can't guarantee it (#2780) + - fix parsing of UDP datagrams with IP socket options (#2780) + - drop abnormally large ethernet frames (#2793) + - Improve xhyve logging (#2719) + - Record VM start and stop events (#2698) + +### 2016-04-19 1.11.0-beta8 "Beta-8" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - Networking mode switched to VPN compatible by default (#2091), + and as part of this change the overall experience has been improved: + - `docker.local` now works in VPN compatibility mode (#2603) + - exposing ports on the Mac is available in both networking modes (#2621) + - port forwarding of privileged ports now works in both networking modes (#2557) + - traffic to external DNS servers is no longer dropped in VPN mode (#2625) + - `osxfs` now uses `AF_VSOCK` for transport (#2531, #2584) giving ~1.8x + speedup for large sequential read/write workloads but increasing + latency by ~1.3x. `osxfs` performance engineering work continues. + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + +* Bug fixes and minor changes + - Apple System Log now used for most logs instead of direct filesystem logging (#2543, #2587) + - docker_proxy fixes (#2539) + - Merge Xhyve upstream patches (#2551) + - Improve error reporting in `nat` network mode (#2606) + - `osxfs` `transfused` client now logs over `AF_VSOCK` (#2602) + - Fixed a com.docker.osx.xhyve.linux supervisor deadlock if processes exit + during a controlled shutdown (#2602) + - Fixed VPN mode malformed DNS query bug preventing some resolutions (#2607) + +### 2016-04-14 1.11.0-beta7 "Beta-7" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - Use `AF_VSOCK` for Docker socket transport (#2469, #2438, #2410) + +* Upgrades + - docker 1.11.0 + - docker-machine 0.7.0 + - docker-compose 1.7.0 + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - If VPN mode is enabled and then disabled and then re-enabled again, + `docker ps` will block for 90s (#2337) + +* Bug fixes and minor changes + - Logging improvements (#2455, #2427, #2409, #2380, #2379) + - Improve process management (#2454, #2451, #2456) + +### 2016-04-05 1.11.0-beta6 "Beta-6" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - Add uninstall option in user interface (#2198) + +* Upgrades + - docker 1.11.0-rc3 (#2255) + - docker-compose 1.7.0-rc2 (#2301) + - docker-machine 0.7.0-rc1 + - Linux 4.4.6 + +* Known issues + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - If VPN mode is enabled and then disabled and then re-enabled again, + `docker ps` will block for 90s (#2337) + +* Bug fixes and minor changes + - Fix osxfs multiple same directory bind mounts stopping inotify (#2258) + - Fix osxfs setattr on mode 0 files (sed failures) (#2171) + - Fix osxfs blocking all operations during `readdir` (#2141) + - Fix osxfs mishandled errors which crashed the file system and VM (#2166) + - Remove outdated lofs/9p support (#2256) + - Add more debugging info to logs uploaded by `pinata diagnose` (#2175) + - Improve diagnostics from within VM (#2183) + - Virtualbox version check now also works without VBoxManage in path (#2152) + - VPN mode now uses same IP range as NAT mode (#2151) + - Tokens are now verified on port 443 (#2117) + - Remove outdated uninstall scripts (#2198) + - Increase default ulimits (#2253) + - Port forwarding with `-p` and `-P` should work in VPN mode (#2190) + - Fix a memory leak in com.docker.db (#2289) + - Fix a race on startup between docker and networking which can + lead to Docker.app not starting on reboot (#1808) + +### 2016-03-29 1.10.3-beta5 "Beta-5" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + +* Known issues + - There is a race on startup between docker and networking which can + lead to Docker.app not starting on reboot. The workaround is to + restart the application manually. (#1808). + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - In VPN mode, the `-p` option needs to be explicitly of the form + `-p :`. `-p ` and `-P` will not + work yet. (#1520) + +* Bug fixes and minor changes + - Update DMG background image (#2098) + - Show correct VM memory in Settings (#2067) + - Feedback opens forum, not email (#2066) + - Fix RAM amount error message (#2026) + - Fix wording of CPU error dialog (#1940) + - Remove status from settings (#2058) + - Check for incompatible versions of Virtualbox (#2056) + + +### 2016-03.22 1.10.3-beta4 "Beta-4" + +* New features + - File-sharing: support inotify events so that filesystem events on the + Mac will trigger filesystem activations inside Linux containers (#822) + - Install docker-machine in `/usr/local` (#1703) + - Add an animated popover window to help first-time users get started (#1848) + - Add `pinata doctor` to diagnose common setup issues such as stray environment variables (#1809) + - New Beta icon (#1780) + +* Known issues + - There is a race on startup between docker and networking which can + lead to Docker.app not starting on reboot. The workaround is to + restart the application manually. (#1808). + - Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated. The workaround is to restart + Docker.app (#1224) + - In VPN mode, the `-p` option needs to be explicitly of the form + `-p :`. `-p ` and `-P` will not + work yet. (#1520) + +* Bug fixes and minor changes + - Hostnet: fix Moby DNS resolver failures by proxying the + Recursion Available flag (#1698) + - `docker ps` shows IP address rather than `docker.local` (#1841) + - Re-enable support for OS X 10.10 (#1748) + - Fix "Notification Center"-related crash on startup (#1167) + - Fix watchdog crash on startup (#1558) + - Ensure binaries are built for 10.10 rather than 10.11 (#1748) + - `pinata diagnose`: work around bugsnag upload limits (#1845) + and add more diagnostic tests (#1852) + - `pinata diagnose`: remove false positives if old launchd plists + are still present (#1662) + +### 2016-03-15 1.10.3-beta3 "Beta-3" + +* New + - Docs have been updated! See https://beta.docker.com/docs/ + - 6x speedup for sequential write throughput with osxfs (#1607) + - Rename `bridged` mode to `nat` mode (#1621) + - Docker runs in debug mode by default for new installs (#1546) + - Add `pinata diagnose` command to help diagnose problems and upload logs (#1664) + +* Upgrades + - Docker 1.10.3 (#1586) + +* Bug fixes and minor changes + - Add more verbose logging on errors in `nat` mode (#1620, #1522) + - Fix HockeyApp crash reporting (#1587) + - Fix get/set VPN mode in settings (#1583) + - Fix osxfs chmod on sockets (#1570) + - Fix osxfs EINVAL from `open` using O_NOFOLLOW (#1511) + - Show correct forwarding details in `docker ps/inspect/port` in `nat` mode (#1564) + - Auto update automatically checks for new versions again (#1536) + - Hypervisor stability fixes, resynced with upstream repository (#1531, #1514) + - Fatal GUI errors now correctly terminate the app again (#1524,#1528) + - New lines ignored in token entry field (#1498) + - Fix proxy panics on EOF when decoding JSON (#1489) + - Clarify open source licenses (#1604, #1677, #1636) + - Fix long delay/crash when switching from `hostnet` to `nat` mode (#1675) + - Moby logs included in diagnose upload (#1664) + - Feedback mail has app-version in subject field (#1653) + - App version included in logs on startup (#1645) + +### 2016-03-08 1.10.2-beta2 "Beta-2" + +* New features + - Docs have been updated! See https://beta.docker.com/docs/ + - GUI: Add VPN mode / slirp to settings (#1444) + - GUI: Add disable Time Machine backups of VM disk image to settings (#1429) + - CLI: `pinata` configuration tool for experimental settings (#1411, #1370) + - FS: Add guest-guest FIFO and socket file support (#1314) + +* Upgrades + - notary 0.2 (#1447) + +* Bug fixes + - FS: Fix data corruption bug during cp (use of sendfile/splice) (#1384) + - GUI: Fix About box to contain correct version string (#1289) + - Slirp/VPN mode: Stability fixes and tests (#1394, #1428, #1434) + - Slirp/VPN mode: Fix DNS issues when changing networks (#1455) + - Moby: Clean up Docker startup code (#1398) + - Fix various linking and dependency problems (#1426, #1378) + - Logging improvements (#1437) + +### 2016-03-01 1.10.2-b1 "Beta-1" + +* GUI + - Add dialog to explain why we need admin rights (#1284, #1207) + - Remove shutdown/quit window (#1287) + - Improve machine migration (#1255) + - Add "Help" option in menu to open docs webpage (#1237) + - Add license agreement (#1209) + - Add MixPanel support (#1206) + +* CLI + - Add `docker-configure` tool to modify experimental settings (#1304, #1293, #1286, #1280, #1266, #1229, #1222, #1311) + - Add `docker-diagnose` and `docker-configure` to path (#1254) + +* Crash reports + - Add HockeyApp crash reporting (#1260, #1251) + +* Task manager + - Improve signal handling (#1221, #1201, #1246) + +* Logging + - Use ISO timestamps with microsecond precision (#1285) + - Clean up logging format (#1285) + +* Packaging + - Create `/usr/local` if it doesn't exist (#1307) + - `docker-uninstall` improvements (#1236) + - Remove `docker-select` as it's no longer used (#1295) + +* Hypervisor + - Add pid file (#1220) + - Networking reliability improvements (#1181) + +* Bug fixes + - Slirp: fix port forwarding issue (#1218) + - Slirp: stability fixes (#1194) + - Moby: fix setting hostname (#1283) + - Fix permissions on `/usr/local` symlinks (#1261) + +### 2016-02-23 1.10.2-12 "Alpha-12" + +* Enrollment System + - A token is now required to launch the application + - How to get the token? + - Sign up on http://beta.docker.com + - Wait for validation email (or ask for it in #docker-mac channel on Slack) + - Open the application, token will be requested + - Internet access is required only once + +* API proxy + - Bug fixes for `docker inspect` (#1051) + - Stop rewriting volume paths: `/Users`, `/private`, `/tmp` and `/Volumes` + are now directly mapped into the OSX filesystem. (#1016) + - Stability fixes (#1097) + +* Docker + - Upgrade to docker 1.10.2 (#1103) + +* CLI + - bundle bash completion from docker, machine and compose (#1036) + - bundle docker-machine (and its bash completion) (#1027) + +* Moby + - Speed-up exit time: shutdown is now really fast. (#945) + - Kernel update to 4.1.18 (#1096) + +* Hypervisor (com.docker.driver.amd64-linux) + - Stability fix and more logging for new `freeze` sleep mode (#1026) + +* Packaging + - Move database to `~/Library/Containers/com.docker.docker/Data/database` + - Move disk image to `~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux` + - Move Moby logs to `~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log` + +### 2016-02-16 1.10.1-11 "Alpha-11" + +* API proxy + - Rewrite `docker inspect` to rewrite volumes and ports + - Handle start requests and adjust mounts for `docker-compose` (#918, #924) + - Add disabled, experimental flag to expose `/var/run/docker.sock` (#953) + +* Networking + - In `slirp` mode, if no IP is specified bind to 0.0.0.0 (#939) + - `docker ps` will show where the port is really bound, i.e. on + `docker.local` (#939) + - New experimental key `native/port-forwarding=false`: if true this will + perform port forwarding from IP addresses on the host, like regular + Linux. (#939) + +* UI + - Logs: Remove the log dashboard and link to the log folder instead + - Buttons: Fix the "Acknowledgements" button (#984) + +* Hypervisor (com.docker.driver.amd64-linux) + - Freeze VM while OSX sleeps (workaround for Apple ACPI bug, issue #147) (#967) + - Prevent spinning if the tty is disconnected (#869, #996) + +* Moby + - Fix fd leaks inside 9p daemon (#954) + - Initial user namespace support (#933) + - New kernel, switch to C transfused (#966) + +* Upgrades + - docker-engine 1.10.1 (#946) + - notary 1.10-5 (#964) + - link to latest Kitematic which works out-of-the-box with Docker.app (#957) + +* Windows + - Initial code integration. Now the OSX and Windows apps are built from + the same source tree. + +### 2016-02-09 1.10.0-10 "Alpha-10" + +* File system sharing + - osxfs read corruption fix (#896) + - osxfs sequential read performance 50% faster (now 75MiB/s) + - lofs sequential read performance 13x faster (now 60MiB/s) + - lofs sequential write performance 11x faster (now 50MiB/s) + +* Docker socket + - Socket upload into a container 70% faster (now 17MiB/s) + - Socket download from a container 30% faster (now 21MiB/s) + +* Upgrades + - docker-engine 1.10.0 (#888) + - docker-compose 1.6.0 (#890) + - docker-machine 0.6.0 (#894) + +* Networking + - experimental slirp mode (#903, #902, #901, #887) + - the internal IP address can be set in the database (keys slirp/docker and slirp/host) + - the internal IP now defaults to 169.254.0.1,2 + - when port forwarding, binding to 0.0.0.0 works as on Linux (see documentation for limitations) + +* Bugfixes + - Improve proxy stability (#910, #898) + +### 2016-02-02 1.10.0-9 "Alpha-9" + +NOTE: If you have manually changed the database keys they will be reset to default values after upgrading to this alpha. + +* File system sharing + - osxfs now enabled by default for all users + - osxfs sequential read throughput is now 10x higher (50MiB/s) (#833, #818, #817) + - osxfs stability improvements (#816, #787, #785) + +* Migration + - experimental support for migrating containers from Virtualbox on first launch (#842, #725) + +* Upgrades + - docker-engine 1.10.0-rc2 + - docker-compose 1.6.0rc2 + +* Networking + - New experimental mode ("slirp") added as a workaround for VPN/firewall issues. This mode is not enabled by default (#843, #838, #836, #815, #809) + +* Proxy + - Stability and logging improvements (#814, #781) + +* User interface + - Logs now displayed under "Logs..." + - Dashboard links to Kitematic download + +* Moby + - Reduced boot time (#moby/18) + +* Database + - Add initial support for schema upgrades (#846) + + +### 2016-01-26 1.10.0-8 "Alpha-8" + +* Dashboard + - Include output from docker.log (#728) + +* Hypervisor (com.docker.driver.amd64-linux) + - Shutdown VM while OSX sleeps (workaround for Apple ACPI bug, issue #147) (#711, #694) + +* Configuration + - Linux: Database file to configure hostname, restarts VM on commit + - Docker: Database file to configure Docker daemon, restarts docker on commit + +* Upgrades + - docker-compose 1.6.0rc1 + +* File system sharing (experimental) + - `osxfs`, an experimental shared file system has been integrated (#707) + which fixes UID/GID mapping (#123) and atomic rename (#668) and has + twice the sequential write throughput. It is disabled by default + but can be enabled by writing "osxfs" to the `filesystem` database + key. Please see the documentation for further details. + +### 2016-01-19 1.10.0-7 "Alpha-7" + +* Docker + - upgrade to 1.10.0-rc1 (#650) + - logs available at ~/Library/Group Containers/group.com.docker/ + com.docker.driver.amd64-linux/docker.log + +* Installer + - symlinks in /usr/local/bin are automatically created on first launch (#674) + +* Menubar + - add a simple dashboard which currently displays logs (#679) + +* Packaging + - Include even more of the open-source component LICENSE text (in + previous alphas we missed a few packages) + - Future automatic upgrades will only prompt for admin password if + the vmnetd protocol has changed, not on every version (#645) + - New and updated graphic assets (#652) + +* Hypervisor (com.docker.driver.amd64-linux) + - bundle an experimental qemu x86_64 binary (#453) + +### 2016-01-12 1.9.1-6 "Alpha-6" + +* Packaging + - Distributed as a .dmg, with cute whale picture! + - Auto update re-enabled + +* Moby + - Add support for running containers for other architectures (arm, arm64, ppc64le, mips64, mips64le) (#545, #557, #559) + - Kernel 4.1.15 and aufs fixes (#589) + +* Networking (com.docker.vmnetd) + - Improve performance from host to container (~350 mbit/sec vs ~1.2 gbit/sec) (#566) + - Add support for rx batching to reduce packet overhead under high load (#566) + +* Menubar (Docker.app) + - Improve process management and logging (#570, #543, #568) + +* Hypervisor (com.docker.driver.amd64-linux) + - Add support for switching between hypervisors (xhyve, qemu) by writing to the db (#562) + - Proxy: handle old-style protocol upgrades to fix compose and terminal issues (#569) + + +### 2016-01-05 1.9.1-5 "Alpha-5" + +Note: This version requires a manual uninstall of earlier alpha versions. See documentation for details. + +* Main Window (former Kitematic) + - main Window removed from this alpha, waiting for new mockups and decision about JS (+ smooth transition from Kitematic) versus Swift implementation (#477) + +* Docker Terminal + - "Open Terminal" removed. See documentation for CLI tool installation instructions. (#513) + +* Settings + - login autostart checkbox (ON by default) (#477) + +* Moby + - faster boot time + - upgrade to Alpine 3.3 final + - diagnostics tool + - installer for docker-x + +* Hypervisor + - support more than 3GiB of RAM (`git commit` new value to ~/Library/Application Support/Docker/database/com.docker.driver.amd64-linux/memory) (#473, #474) + - bundle `qcow-tool` for manual creation of larger `Docker.qcow2` files (#501) + - enable ACPI support (#525) + +* Installer + - launchd socket activation is removed; services (including hypervisor and containers) run while the whale menu is running. (#477) + - the app is not automatically copied in /Applications anymore (#477) + - the app can run from any location (#477) + - uninstall dragging application's icon to the trash (may not work for vmnetd) (#477) + +* Packaging + - OSS Licenses of software running on the Host in Docker.app/Contents/Resources/OSS-LICENSES (#495, #491, #483) + + +### 2015-12-21 1.9.1-4 "Alpha-4" + +* moby + - use AUFS instead of OverlayFS + +* network: + - vmnetd: improve performance. disabling verbose logging improve perfs + from ~2MB/s to ~30-40MB/s (#443) + +* in-app support + - docker-diagnose: add capability to upload to bugsnag, upload sysctl and + logs to bugsnag and add a JSON output (#428, #429, #430) + - xhyve: log fatal errors to bugsnag (#425) + +* toolbox + - compose: bundle and install docker-compose (#423, by @justincormack) + - notary: bundle notary but do not install it yet because the generated + binary has some signing issue (#445) + +* DB + - bundle and install com.docker.db, an Irmin DB exposing a 9p interface + (#403, #415) + - update the Go API to use the new FS interface (#90) + +* installer + - docker-select now creates `/usr/local/bin` if it doesn't exists and force + install docker CLI and docker-compose if they are not properly installed + yet (#446) + + +### 2015-12-15 1.9.1-3 "Alpha-3" + +* Critical fix to 1.9.1-2 regarding auto-update + - Alpha-2 use the `master` channels instead of the `alpha` channel + +### 2015-12-14 1.9.1-2 "Alpha-2" + +* hypervisor + - process renamed to com.docker.driver.amd64-linux + +* distro (dom0) + - first release of moby, our own OS distro + +* network + - privileged daemon renamed to com.docker.vmnetd + - network daemon can now auto-update + +* storage (file-sharing) + - file-sharing daemon renamed to com.docker.lofs + +* installer + - install a proof-of-concept preference-panel + - report installation bug to bugsnag + - Docker.app is now properly signed + - The installer gives more feedback to the user + - need to run /Applications/Docker.app/Contents/Resources/docker-select + to install CLI tools + - configuration files are in ~/Library/Application Support/Docker + +* kitematic + - volumes works + - image pull works + - port forwarding works + +* preference-pane + - basic proof-of-concept + +* menubar + - allow to open a terminal + - allow to start the GUI + - allow to open the preference pane + - allow to check for upgrades and upgrade to a new version 2015-12-04 1.9.1-1 "Alpha-1" + +### 2015-12-04 1.9.1-1 "Alpha-1" + +* hypervisor + - more stable + - extensible storage (qcow2) for guests can now grow to 64 gB + - fixed amount of RAM extended to 2GB + +* installer + - install and start kitematic + - smaller: ~450MB + - auto-updates + +* Kitematic + - modified version of Kitematic which do not requires VirtualBox + +* menubar + - show HockeyApp version + - allow to upgrade to new version + - persisted on reboot + - appears whenever the hypervisor is started + +* user-feedback + - docker-diagnose + - use bugsnag + +### 2015-11-30 1.9.1-0 "Alpha-0" + + * hypervisor + - stable enough to boot a few times + - use qcow2 for extensible copy-on-write storage for guests (can grow to 1gB) + - fixed amount of RAM (1GB) + + * distro (dom0) + - heavily trimmed-down version of boot2docker + - still >100 MB + + * network + - network daemon runs as root + - blocker for integration in the App Store + - docker.local points to the distro VM + + * storage (file-sharing) + - basic file-sharing works over 9p + - not POSIX (issues with symlink, …) + - all files have the same user/group + - file timestamps can be wrong + + * installer + - install hypervisor, network daemon, storage daemon, docker CLI and menu-bar + - install docker in /usr/local/bin + - very big: 1GB + - report installation stats to mixpanel + + * docker + - upstream docker CLI 1.9.1 on OSX + - upstream docker daemon running in boot2docker + - simple proxy to rewrite volumes API running in OSX + - symlink /var/tmp/docker.sock to /var/run/docker.sock + - need root privileges on every reboot: blocker for App Store + - not very secure + +* menubar + - doesn’t do anything useful + - Not persisted on reboot diff --git a/MAINTAINERS b/MAINTAINERS new file mode 100644 index 0000000000..0a55ce60fa --- /dev/null +++ b/MAINTAINERS @@ -0,0 +1,21 @@ +Jeffrey Morgan +Sean Li +Michael Chiang +Adrien Duermael +Gaetan Devillelle +Solomon Hykes +Dave Scott +Michel Courtine +Thomas Gazagnaire +Emmanuel Briney +Justin Cormack +David Sheets +Stephen Day +Magnus Skjegstad +Patrick Chanezon +Thomas Leonard +David Gageot +Rolf Neugebauer +Richard Mortier +Jean-Laurent de Morlhon +Ben Bonnefoy diff --git a/Makefile b/Makefile new file mode 100644 index 0000000000..17c8ee1fad --- /dev/null +++ b/Makefile @@ -0,0 +1,254 @@ +MACOSX_DEPLOYMENT_TARGET?=10.10 +REPO_ROOT=$(shell git rev-parse --show-toplevel) +OUTPUT?=$(REPO_ROOT)/v1/mac/build/Docker.app +PROJECT_ROOT?=$(GOPATH)/src/github.com/docker/pinata +CACHE_DIR?=$(REPO_ROOT)/_cache + +# Getting version from Info.plist file +# NOTE(aduermael): this won't work on Windows +# It will be updated soon to support both platform +# with tags of the form: "win-.*" and "mac-.*" +plistPath=$(PROJECT_ROOT)/v1/mac/src/docker-app/docker/docker/Info.plist +versionFromPlist=$(shell /usr/libexec/PlistBuddy -c "Print CFBundleShortVersionString" "$(plistPath)" 2> /dev/null) +PARTS=$(subst -, ,$(versionFromPlist)) +VERSION=v$(word 1, $(PARTS))-$(word 2, $(PARTS)) + +# hockeyapp read-only tokens +HA_MAC_TOKEN=bf3c4239192a4511ab54ff5e963d51b1 +HA_WIN_TOKEN=64336e7527dc477da596bedfa2804540 + +# opam flags +OPAMROOT=$(CACHE_DIR)/opam +OPAMDIR=$(REPO_ROOT)/v1/opam +OPAMFLAGS=MACOSX_DEPLOYMENT_TARGET=$(MACOSX_DEPLOYMENT_TARGET) OPAMROOT=$(OPAMROOT) OPAMYES=1 OPAMCOLORS=1 OPAMDIR=$(OPAMDIR) GO15VENDOREXPERIMENT=1 +OPAMLIBS=mirage-block-c docker-diagnose osx-daemon osx-hyperkit +OPAMCMDS=osxfs +OPAMSUPPORT=nurse + +# TODO: this needs a cleaner solution +BACKENDCMDS=driver.amd64-linux vmnetd osx.hyperkit.linux hyperkit frontend shell #driver.amd64-qemu + +LICENSEDIRS=\ + $(PROJECT_ROOT)/v1/opam \ + $(PROJECT_ROOT)/v1/vendor \ + $(PROJECT_ROOT)/v1/uefi \ + $(OPAMROOT) \ + $(PROJECT_ROOT)/v1/cmd/com.docker.hyperkit \ + $(PROJECT_ROOT)/v1/docker_proxy/vendor \ + $(PROJECT_ROOT)/v1/mac/src/docker-app/docker/Carthage/Checkouts \ + $(PROJECT_ROOT)/v1/mac/dependencies/qemu + +.PHONY: all depends opam perf OSS-LICENSES dmg dsym-zip clean cacheclean versions go-fmt go-lint go-vet go-test go-depends + +all: opam mac + @ + +depends: mac-depends qemu-depends opam-depends go-depends moby-depends + @ + +clean: opam-clean backend-clean moby-clean + $(MAKE) -C $(PROJECT_ROOT)/v1/mac clean + +cacheclean: + rm -rf "$(HOME)/.docker-ci-cache/opam" + rm -rf "$(OPAMROOT)" + +OSS-LICENSES: + $(OPAMFLAGS) v1/opam/opam-licenses $(OPAMCMDS) + $(MAKE) -C $(PROJECT_ROOT)/v1/cmd/com.docker.hyperkit LICENSE + $(foreach dir, $(LICENSEDIRS), mkdir -p $(dir);) + $(PROJECT_ROOT)/v1/mac/scripts/list-licenses $(LICENSEDIRS) > OSS-LICENSES + +# opam applications + +UPSTREAM=$(shell ls $(OPAMDIR)/repo/packages/upstream | awk -F. '{ print $$1 }') +DEV=$(shell ls $(OPAMDIR)/repo/packages/dev) + +opam-depends: + @brew install opam || true &> /dev/null + @brew install dylibbundler || true &> /dev/null + @$(OPAMFLAGS) $(OPAMDIR)/opam-boot + @$(OPAMFLAGS) opam update -u + @$(OPAMFLAGS) opam install depext + @$(OPAMFLAGS) opam depext $(UPSTREAM) $(DEV) &> /dev/null + @$(OPAMFLAGS) opam install $(UPSTREAM) $(DEV) + +opam-lib-clean-%s: + $(OPAMFLAGS) $(MAKE) -C $(PROJECT_ROOT)/v1/$* clean + +opam-cmd-clean-%s: + $(OPAMFLAGS) $(MAKE) -C $(PROJECT_ROOT)/v1/cmd/com.docker.$* clean + +opam-support-clean-%: + $(OPAMFLAGS) $(MAKE) -C $(PROJECT_ROOT)/support/$* clean + +opam-clean: $(OPAMLIBS:%=opam-lib-clean-%s) $(OPAMCMDS:%=opam-cmd-clean-%s) $(OPAMSUPPORT:%=opam-support-clean-%) + @ + +opam-lib-%: + cd $(PROJECT_ROOT)/v1/$* && $(OPAMFLAGS) ./build.sh + +opam-cmd-%: + cd $(PROJECT_ROOT)/v1/cmd/com.docker.$* && $(OPAMFLAGS) ./build.sh + +opam-support-%: + cd $(PROJECT_ROOT)/support/$* && $(OPAMFLAGS) ./build.sh + +opam: $(OPAMLIBS:%=opam-lib-%) $(OPAMCMDS:%=opam-cmd-%) $(OPAMSUPPORT:%=opam-support-%) OSS-LICENSES + @ + +# backend + +backend-cmd-clean-%: + cd $(PROJECT_ROOT)/v1/cmd/com.docker.$* && $(OPAMFLAGS) $(MAKE) clean + +backend-lib-clean-%: + cd $(PROJECT_ROOT)/v1/$* && $(OPAMFLAGS) $(MAKE) clean + +backend-clean: $(BACKENDCMDS:%=backend-cmd-clean-%) + @ + +backend-cmd-%: + cd $(PROJECT_ROOT)/v1/cmd/com.docker.$* && $(OPAMFLAGS) $(MAKE) CACHE_DIR=$(CACHE_DIR) + +backend-cmd-vmnetd: opam +backend-cmd-hyperkit: backend-cmd-vmnetd + +backend: $(BACKENDCMDS:%=backend-cmd-%) + @ + +# moby +moby-depends: + go get -u github.com/justincormack/regextract + +moby: + cd $(PROJECT_ROOT)/v1/moby && make + +moby-clean: + cd $(PROJECT_ROOT)/v1/moby && make clean + +# mac app + +mac-depends: + cd $(PROJECT_ROOT)/v1/mac/scripts && ./make.bash -dy + +mac: opam backend moby docker-release + cd $(PROJECT_ROOT)/v1/mac/scripts && ./make.bash -cby + +dmg: + cd $(PROJECT_ROOT)/v1/mac/scripts && ./make-dmg + +dsym-zip: + cd $(PROJECT_ROOT)/v1/mac/scripts && ./make-dsym-zip + +# run Docker.app + +dev: opam mac + rm -rf "$(PROJECT_ROOT)/v1/mac/build" + rm -rf "$(PROJECT_ROOT)/v1/mac/src/docker-app/build" + cd $(PROJECT_ROOT)/v1/mac/src/docker-app && make dev + +# open Docker.app .xcodeproj + +run: + $(PROJECT_ROOT)/v1/mac/build/Docker.app/Contents/MacOS/Docker + +backend-run: + @$(PROJECT_ROOT)/v1/cmd/com.docker.shell/com.docker.shell -debug -bundle $(PROJECT_ROOT)/v1/mac/build/Docker.app + +# tests +lint: go-fmt go-lint go-vet + # lint test scripts + brew install shellcheck + find tests/cases -type f | xargs -L1 file -I | grep 'text/x-shellscript' | cut -f1 -d":" | xargs -L1 shellcheck -e SC2129,SC1090,SC2039 + +GOPACKAGES = $(eval GOPACKAGES := $(shell cd $(PROJECT_ROOT)/v1 && go list -e ./... | grep -v vendor | grep -v moby))$(GOPACKAGES) + + +go-depends: + go get -u github.com/golang/lint/golint + +go-fmt: + @for pkg in $(GOPACKAGES) ; do \ + echo "gofmt $${pkg##*pinata/} ..." ;\ + cd $(PROJECT_ROOT)/$${pkg##*pinata/} ;\ + test -z "$$(gofmt -s -l . 2>&1 | grep -v ^vendor/ | tee /dev/stderr)" || exit 1 ;\ + done + +go-lint: + @for pkg in $(GOPACKAGES) ; do \ + echo "golint $${pkg##*pinata/} ..." ;\ + cd $(PROJECT_ROOT)/$${pkg##*pinata/} ;\ + test -z "$$(golint . 2>&1 | grep -v ^vendor/ | tee /dev/stderr)" || exit 1 ;\ + done + +go-vet: + @cd $(PROJECT_ROOT) && go vet $(GOPACKAGES) + +go-test: + @cd $(PROJECT_ROOT) && for pkg in $(GOPACKAGES) ; do \ + echo "testing $$pkg ..." ;\ + go test -race -v $$pkg ;\ + done + +test-depends: opam + cd $(PROJECT_ROOT)/v1/tests && $(OPAMFLAGS) ./build.sh + +test: lint test-depends go-test + (cd $(PROJECT_ROOT)/tests && ./rt-local -l nostart,checkout -v -x run) + +# test-dmg also tests the dmg - it's assumed that `make dmg` was performed first first +test-dmg: lint test-depends go-test + (cd $(PROJECT_ROOT)/tests && ./rt-local -l installer,checkout -v -x run) + +fulltest: + (cd $(PROJECT_ROOT)/tests && ./rt-local -l nostart,release,checkout -v -x run) + PINATA_APP_PATH=$(OUTPUT) $(PROJECT_ROOT)/v1/tests/pinata-rt test -e + +perf: + make -C $(PROJECT_ROOT)/v1/perf + +# qemu +QEMUV = 2.4.1 +export QEMUV +qemu-depends: + @mkdir -p $(CACHE_DIR) + @cd $(PROJECT_ROOT)/v1/cmd/com.docker.driver.amd64-qemu && make depends CACHE_DIR=$(CACHE_DIR) + +# upload to HockeyApp + +upload: + @cd $(PROJECT_ROOT)/v1/mac/scripts && ./make.bash -uy + +release: + rm -rf "$(PROJECT_ROOT)/v1/mac/build" + rm -rf "$(HOME)/.docker-ci-cache" + rm -rf "$(CACHE_DIR)" + make depends + make + make test + git tag $(VERSION) -a -m "Release $(VERSION)" + git push upstream $(VERSION) + +versions: + @echo git tag name: $(VERSION) + @echo Xcode project version \(Info.plist\): $(versionFromPlist) + @echo Changelog: $(shell head -n 1 CHANGELOG | cut -f 3 -d" ") + @echo docker-diagnose: $(shell cat v1/docker-diagnose/src/dockerCli.ml | grep check_version) + +release-to-rc: + @echo "Releasing latest builds to RC (the newest unreleased build will also be downloaded)" + docker-release --channel rc --arch mac --build latest publish + docker-release --channel rc --arch win --build latest publish + +# docker-release build +docker-release: + cd $(PROJECT_ROOT)/v1/docker-release && make + +# helpful targets for development +logwatch: + syslog -w -F '$$Time $$Host $$(Sender)[$$(Facility)][$$(PID)]\n<$$((Level)(str))>: $$Message' \ + -k Sender Seq Docker -o \ + -k Sender Seq docker -o \ + -k Message Seq Docker -o \ + -k Message Seq docker diff --git a/README.md b/README.md new file mode 100644 index 0000000000..adaf4bca89 --- /dev/null +++ b/README.md @@ -0,0 +1,163 @@ +# Pinata: an experimental standalone Docker client + +| | pr | master | rc | beta | stable | +|---|---|---|---|---|---| +| macOS | [latest](https://download-stage.docker.com/mac/pr/Docker.dmg) | [latest](https://download-stage.docker.com/mac/master/Docker.dmg) | [latest](https://download-stage.docker.com/mac/rc/Docker.dmg) | [latest](https://download.docker.com/mac/beta/Docker.dmg) | [latest](https://download.docker.com/mac/stable/Docker.dmg) | +| Windows | [latest](https://download-stage.docker.com/win/pr/InstallDocker.msi) | [latest](https://download-stage.docker.com/win/master/InstallDocker.msi) | [latest](https://download-stage.docker.com/win/test/InstallDocker.msi) | [latest](https://download.docker.com/win/beta/InstallDocker.msi) | [latest](https://download.docker.com/win/stable/InstallDocker.msi) | +*( if you get Access Denied errors, it means nothing has been published to this channel yet )* + +To list all the versions: http://omakase.omakase.e57b9b5b.svc.dockerapp.io/ + +This is an experimental project to develop a new client for Docker, +separately from the daemon or any other backend component. + +By maintaining a standalone client, the goal is to: + +1. Allow for more rapid iteration on client functionality. +2. Improve compatibility between different versions of the client and daemon. +3. Add more features to the client without bloating the daemon-side components. +4. Pave the way to simplifying the daemon code base, improving its +quality and making its maintenance easier. + +## Versioning + +The release cycle respects the following convention: `X-Y[-Z]` where: + +- `X` is the version of the docker engine used as a base for the build. The build can be modified during the build process to fit better into the use-case of `Docker.app` (ie. it won't usually be a drop-in replacement, but we will try to upstream our patches as quickly as possible). + +- `Y` is an arbitrary string that we can use to define a version of `Docker.app`, independently of the release cycle of docker engine. + +- `Z` indicates the build channel (dev, test, master, release). `Z` is empty for releases. + +For instance the first beta release of pinata has the version: `1.9.1-beta1`. While on master channel (one build for each PR merged), it has the version: `1.9.1-beta1-master`. + +On OS X, the version is defined in XCode project's Info.plist file (key: `CFBundleShortVersionString`). There's also a build number, associated with `CFBundleVersion` key (set by CI). + +In Xcode project, the version should use a suffix like `-dev` (`1.9.1-beta1-dev`). That suffix will be replaced/removed by CI. + +## INSTALL + +### Through HockeyApp + +For Docker for Mac see the [Docker.app installation guide](https://github.com/docker/pinata/blob/master/v1/docs/content/mackit/getting-started.md) and for Docker for Windows see the [Docker installation guide](https://github.com/docker/pinata/blob/master/v1/docs/content/winkit/getting-started.md) + +### OSX Build + +Check that your `GOPATH` is correctly set-up and clone this repository in +`$GOPATH/src/github.com/docker/pinata`. + +#### Dependencies + +As prerequisites, you need to have `Xcode`, `homebrew` and `go` installed. + +To minimize build times, the dependencies are cached with this command + +You only need to run it once or when an external dependency was updated + +At the root of this repository, type: + +``` +make depends +``` + +When you add a new go dependency, add it in the `GO_DEPS` variable of the toplevel +`Makefile`. + +#### Build + +After a successful `make depends`, type: + +``` +make +``` + +If you are asked for the password to the `dockerbuilder` keychain, it is +`docker4all`. + +#### Run + +After a successful `make depends` and `make`, type: + +``` +make run +``` + +You will see the logs on stdout + +#### Install + +First, make sure you have uninstalled any previous installation of +pinata with: + +``` +v1/mac/uninstall +``` + +Then, install with: + +``` +v1/mac/build/Docker.app/Contents/MacOS/docker-installer +``` + +#### Tests + +You can run the tests by running: + +``` +make test +``` +This is currently Mac only. + + + +### Windows Build + +[![Build status](https://ci.appveyor.com/api/projects/status/fpa7neeotor31bdh/branch/master?svg=true)](https://ci.appveyor.com/project/Pinata/pinata/branch/master) + +Latest msi builds : + * On [Master](https://download-stage.docker.com/win/master/InstallDocker.msi) channel. + * On [Test](https://download-stage.docker.com/win/test/InstallDocker.msi) channel. + +Check that your `GOPATH` is correctly set-up and clone this repository in +`$GOPATH/src/github.com/docker/pinata`. + +#### Dependencies + +Install: + +- Go 1.6 +- [Visual Studio 2015](https://www.visualstudio.com/en-us/products/vs-2015-product-editions.aspx). The app builds with the free Community edition but the licensing for that edition doesn't allow its use for commercial, closed source work. + +Once you installed the above, open a powershell. + +#### Build + +The main build is driven by the `please.ps1` powershell script in the `win` +sub-directory. + +``` +cd /win +./please.ps1 package +``` + +will clean the build directory and build a new package (installer) in +the `build` sub-directory. + + +``` +cd /win +./please.ps1 build +``` + +will build a new `Docker.exe` file but not the installer. + +### Troubleshooting + +If you have an issue, please report it to the +[bugtracker](https://github.com/docker/pinata/issues) with the output +of: + +``` +pinata diagnose +``` +This is currently Mac only. diff --git a/appveyor.yml b/appveyor.yml new file mode 100644 index 0000000000..74506a7ff0 --- /dev/null +++ b/appveyor.yml @@ -0,0 +1,54 @@ +environment: + GOPATH: c:\gopath + SIGNTOOLPATH: C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Bin + +clone_folder: c:\gopath\src\github.com\docker\pinata + +before_build: + - cd win/ + +build_script: + - ps: ./please.ps1 AppVeyor + +after_build: + - cd .. + - ps: > + copy .\win\build\InstallDocker.msi InstallDocker.msi + +artifacts: + - path: win/build/InstallDocker.msi + - path: win/TestResults/*.xml + - path: InstallDocker.msi + +test: off + +configuration: Release + +before_deploy: + - ps: > + if ($env:appveyor_pull_request_number -gt 0) { + git fetch -q origin +refs/pull/$env:appveyor_pull_request_number/head + $env:commit=(git rev-parse FETCH_HEAD) + } else { + $env:commit=$env:appveyor_repo_commit + } + true; + +deploy: + - provider: S3 + bucket: pinata-ci + access_key_id: $(pinata_aws_key) + secret_access_key: $(pinata_aws_secret) + folder: $(commit) + artifact: InstallDocker.msi + on: + appveyor_repo_tag: false + - provider: S3 + bucket: pinata-ci + access_key_id: $(pinata_aws_key) + secret_access_key: $(pinata_aws_secret) + folder: $(appveyor_repo_tag_name) + artifact: InstallDocker.msi + on: + appveyor_repo_tag: true + diff --git a/circle-linux.yml b/circle-linux.yml new file mode 100644 index 0000000000..837d719160 --- /dev/null +++ b/circle-linux.yml @@ -0,0 +1,27 @@ +# Linux specific circleci.yml +# This build process only build the docs. +# A process mirrors this repo in another repo for every commit, and cp circle-linux.yml to circle.yml. +# This ensures that a Linux build happens in circleci for every commit. + +general: + build_dir: v1 + artifacts: + - "docs/public" + - "mac/build" +machine: + services: + - docker +dependencies: + cache_directories: + - "~/docker" + override: + - make -C docs DOCS_EXPORT=public docs-deploy + - if [[ -e ~/docker/image.tar ]]; then docker load -i ~/docker/image.tar; fi + - mac/make -cblvy + - mkdir -p ~/docker; mkdir -p ~/pinata/v1/mac/build + - docker save pinata/docker-builder > ~/docker/image.tar +deployment: + master: + branch: master + commands: + - make -C docs DOCS_EXPORT=public docs-deploy diff --git a/circle.yml b/circle.yml new file mode 100644 index 0000000000..afda8e63e2 --- /dev/null +++ b/circle.yml @@ -0,0 +1,105 @@ +# Mac specific circleci.yml +# This build process will download and bundle the build artifacts from the linux ci. +# Checkout the same file on the linux branch to edit the linux version of the circleci configuration + +general: + artifacts: + - v1/mac/build/Docker.dmg + - tests/_results + - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log + - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring +machine: + xcode: + version: "8.0" + environment: + GOVERSION: 1.7.1 + GOPATH: "$HOME/go" + GOROOT: "${HOME}/.gimme/versions/go${GOVERSION}.darwin.amd64" + PATH: "${GOROOT}/bin:${GOPATH}/bin:${PATH}" + PROJECT_ROOT: "$GOPATH/src/github.com/docker/pinata" + CI: "1" + post: + - brew unpin go + - brew update + - brew -v install git-lfs + # something forces /usr/loca/bin to the front of the path. + # we'll remove the existing go to be sure + - rm /usr/local/bin/go + - brew -v install gimme + - gimme $GOVERSION + # display homebrew config + - brew config +checkout: + post: + - brew unpin go + - git lfs install --local + - git lfs pull + # Check disk space + - df -h + # check consistency of v1/opam/repo + - v1/opam/repo/check.sh + # v1/cmd/com.docker.hyperkit/hyperkit/ is a git subtree merge from + # http://github.com/docker/hyperkit and no code should be + # committed directly to it. To do this we need to unshallow and to + # have the hyperkit history available. + # + # Mistakes were made prior to + # 5e58d3e47a592e9235c4d03a099bc52ccd7a47ae which were repaired in + # that commit. Start checking from then onwards. + - if [ -s .git/shallow ] ; then git fetch --unshallow git@github.com:docker/pinata master ; fi + - git fetch https://github.com/docker/hyperkit +master:hyperkit/master + - > + if ! git log --no-merges --stat 5e58d3e47a592e9235c4d03a099bc52ccd7a47ae..HEAD --not hyperkit/master -- v1/cmd/com.docker.hyperkit/hyperkit/ | + awk 'BEGIN { rc=0 }; // { rc=1; print }; END { exit $rc }' ; then + echo ""; + echo "Direct commit to hyperkit vendored code detected. Please see:"; + echo "https://github.com/docker/pinata/blob/master/v1/cmd/com.docker.hyperkit/README.md"; + exit 1 + fi +dependencies: + override: + - which go + - go version + - rm -rf "$PROJECT_ROOT" + - mkdir -p $(dirname "$PROJECT_ROOT") + - ln -s $(pwd) "$PROJECT_ROOT" + - make cacheclean + - make mac-depends + - make opam-depends + - make moby-depends + - make go-depends + cache_directories: + - ~/.docker-ci-cache +test: + override: + - make clean + - make opam + - make mac + - make dmg + - make dsym-zip + - make test-dmg + - > + syslog -F "\$Time \$Host \$(Sender)[\$(Facility)][\$(PID)]<\$((Level)(str))>: \$Message" \ + -k Sender Seq Docker -o \ + -k Sender Seq docker -o \ + -k Message Seq Docker -o \ + -k Message Seq docker +deployment: + release: + branch: master + owner: docker + commands: + - aws s3 cp v1/mac/build/Docker.dmg s3://pinata-ci/$CIRCLE_SHA1/Docker.dmg + - make upload + tags: + tag: /.*/ + owner: docker + commands: + - aws s3 cp v1/mac/build/Docker.dmg s3://pinata-ci/$CIRCLE_TAG/Docker.dmg + - make upload + pr: + branch: /.*/ + owner: docker + commands: + - aws s3 cp v1/mac/build/Docker.dmg s3://pinata-ci/$CIRCLE_SHA1/Docker.dmg + - make upload diff --git a/docs/Dockerfile b/docs/Dockerfile new file mode 100644 index 0000000000..6ccd6b2d3a --- /dev/null +++ b/docs/Dockerfile @@ -0,0 +1,10 @@ +FROM docs/base:oss +MAINTAINER Docker Docs + +# because both the 2 dir's are going into the root +env PROJECT= + +# To get the git info for this repo +COPY . /src +#RUN rm -rf /docs/content/$PROJECT/ +COPY . /docs/content/$PROJECT/ diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000000..72fb4bf9c8 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,39 @@ +.PHONY: all default docs docs-build docs-shell shell test + +# to allow `make DOCSDIR=docs docs-shell` (to create a bind mount in docs) +DOCS_MOUNT := $(if $(DOCSDIR),-v $(CURDIR)/$(DOCSDIR):/$(DOCSDIR)) + +# to allow `make DOCSPORT=9000 docs` +DOCSPORT := 8000 + +# Get the IP ADDRESS +DOCKER_IP=$(shell python -c "import urlparse ; print urlparse.urlparse('$(DOCKER_HOST)').hostname or ''") +HUGO_BASE_URL=$(shell test -z "$(DOCKER_IP)" && echo localhost || echo "$(DOCKER_IP)") +HUGO_BIND_IP=0.0.0.0 + +GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD 2>/dev/null) +DOCKER_IMAGE := docker$(if $(GIT_BRANCH),:$(GIT_BRANCH)) +DOCKER_DOCS_IMAGE := docs-base$(if $(GIT_BRANCH),:$(GIT_BRANCH)) + + +DOCKER_RUN_DOCS := docker run --rm -it $(DOCS_MOUNT) -e AWS_S3_BUCKET -e NOCACHE + +# for some docs workarounds (see below in "docs-build" target) +GITCOMMIT := $(shell git rev-parse --short HEAD 2>/dev/null) + +default: docs + +docs: docs-build + $(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP) + +docs-draft: docs-build + $(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --buildDrafts="true" --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP) + +docs-shell: docs-build + $(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 "$(DOCKER_DOCS_IMAGE)" bash + +test: docs-build + $(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 "$(DOCKER_DOCS_IMAGE)" + +docs-build: + docker build -t "$(DOCKER_DOCS_IMAGE)" . diff --git a/docs/docker-for-mac/docker-toolbox.md b/docs/docker-for-mac/docker-toolbox.md new file mode 100644 index 0000000000..1be3f5bc83 --- /dev/null +++ b/docs/docker-for-mac/docker-toolbox.md @@ -0,0 +1,145 @@ + + +# Docker for Mac vs. Docker Toolbox + +If you already have an installation of Docker Toolbox, please read these topics first to learn how Docker for Mac and Docker Toolbox differ, and how they can coexist. + + +## The Docker Toolbox environment + +Docker Toolbox installs `docker`, `docker-compose` and `docker-machine` in `/usr/local/bin` on your Mac. It also installs VirtualBox. At installation time, Toolbox uses `docker-machine` to provision a VirtualBox VM called `default`, running the `boot2docker` Linux distribution, with [Docker Engine](https://docs.docker.com/engine/) with certificates located on your Mac at `$HOME/.docker/machine/machines/default`. + +Before you use `docker` or `docker-compose` on your Mac, you typically use the command `eval $(docker-machine env default)` to set environment variables so that `docker` or `docker-compose` know how to talk to Docker Engine running on VirtualBox. + +This setup is shown in the following diagram. + +![Docker Toolbox Install](images/toolbox-install.png) + + +## The Docker for Mac environment + +Docker for Mac is a Mac native application, that you install in `/Applications`. At installation time, it creates symlinks in `/usr/local/bin` for `docker` and `docker-compose`, to the version of the commands inside the Mac application bundle, in `/Applications/Docker.app/Contents/Resources/bin`. + +Here are some key points to know about Docker for Mac before you get started: + +* Docker for Mac does not use VirtualBox, but rather HyperKit, a lightweight OS X virtualization solution built on top of Hypervisor.framework in OS X 10.10 Yosemite and higher. + +* Installing Docker for Mac does not affect machines you created with Docker Machine. The install offers to copy containers and images from your local `default` machine (if one exists) to the new Docker for Mac HyperKit VM. If chosen, content from `default` is copied to the new Docker for Mac HyperKit VM, and your original `default` machine is kept as is. + +* The Docker for Mac application does not use `docker-machine` to provision that VM; but rather creates and manages it directly. + +* At installation time, Docker for Mac provisions an HyperKit VM based on Alpine Linux, running Docker Engine. It exposes the docker API on a socket in `/var/tmp/docker.sock`. Since this is the default location where `docker` will look if no environment variables are set, you can start using `docker` and `docker-compose` without setting any environment variables. + +This setup is shown in the following diagram. + +![Docker for Mac Install](images/docker-for-mac-install.png) + +With Docker for Mac, you get only one VM, and you don't manage it. It is managed by the Docker for Mac application, which includes autoupdate to update the client and server versions of Docker. + +If you need several VMs and want to manage the version of the Docker client or server you are using, you can continue to use `docker-machine`, on the same machine, as described in [Docker Toolbox and Docker for Mac coexistence](#docker-toolbox-and-docker-for-mac-coexistence). + + +## Setting up to run Docker for Mac + +1. Check whether Toolbox DOCKER environment variables are set: + + $ env | grep DOCKER + DOCKER_HOST=tcp://192.168.99.100:2376 + DOCKER_MACHINE_NAME=default + DOCKER_TLS_VERIFY=1 + DOCKER_CERT_PATH=/Users/victoriabialas/.docker/machine/machines/default + + If this command returns no output, you are ready to use Docker for Mac. + + If it returns output (as shown in the example), you need to unset the `DOCKER` environment variables to make the client talk to the Docker for Mac Engine (next step). + +2. Run the `unset` command on the following `DOCKER` environment variables to unset them in the current shell. + + unset DOCKER_TLS_VERIFY + unset DOCKER_CERT_PATH + unset DOCKER_MACHINE_NAME + unset DOCKER_HOST + + Now, this command should return no output. + + $ env | grep DOCKER + + If you are using a Bash shell, you can use `unset ${!DOCKER_*}` to unset all DOCKER environment variables at once. (This will not work in other shells such as `.zsh`; you will need to unset each variable individually.) + +>**Note**: If you have a shell script as part of your profile that sets these `DOCKER` environment variables automatically each time you open a command window, then you will need to unset these each time you want to use Docker for Mac. + +> **Warning**: If you install Docker for Mac on a machine where Docker Toolbox is installed, it will replace the `docker` and `docker-compose` command lines in `/usr/local/bin` with symlinks to its own versions. + + +## Docker Toolbox and Docker for Mac coexistence + +You can use Docker for Mac and Docker Toolbox together on the same machine. When you want to use Docker for Mac, make sure all DOCKER environment variables are unset. You can do this in bash with `unset ${!DOCKER_*}`. When you want to use one of the VirtualBox VMs you have set with `docker-machine`, just run a `eval $(docker-machine env default)` (or the name of the machine you want to target). This will switch the current command shell to talk to the specified Toolbox machine. + +This setup is represented in the following diagram. + +![Docker Toolbox and Docker for Mac coexistence](images/docker-for-mac-and-toolbox.png) + + +## Using different versions of Docker tools + +The coexistence setup works as is as long as your VirtualBox VMs provisioned with `docker-machine` run the same version of Docker Engine as Docker for Mac. If you need to use VMs running older versions of Docker Engine, you can use a tool like Docker Version Manager to manage several versions of docker client. + + +### Checking component versions + +Ideally, the Docker CLI client and Docker Engine should be the same version. Mismatches between client and server, and among host machines you might have created with Docker Machine can cause problems (client can't talk to the server or host machines). + +If you already have Docker Toolbox installed, and then install Docker for Mac, you might get a newer version of the Docker client. Run `docker version` in a command shell to see client and server versions. In this example, the client installed with Docker for Mac is `Version: 1.11.1` and the server (which was installed earlier with Toolbox) is Version: 1.11.0. + + $ docker version + Client: + Version: 1.11.1 + ... + + Server: + Version: 1.11.0 + ... + +Also, if you created machines with Docker Machine (installed with Toolbox) then upgraded or installed Docker for Mac, you might have machines running different versions of Engine. Run `docker-machine ls` to view version information for the machines you created. In this example, the DOCKER column shows that each machine is running a different version of server. + + $ docker-machine ls + NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS + aws-sandbox - amazonec2 Running tcp://52.90.113.128:2376 v1.10.0 + default * virtualbox Running tcp://192.168.99.100:2376 v1.10.1 + docker-sandbox - digitalocean Running tcp://104.131.43.236:2376 v1.10.0 + +You might also run into a similar situation with Docker Universal Control Plan (UCP). + +There are a few ways to address this problem and keep using your older machines. One solution is to use a version manager like DVM. + +## How do I uninstall Docker Toolbox? + +You might decide that you do not need Toolbox now that you have Docker for Mac, +and want to uninstall it. For details on how to perform a clean uninstall of +Toolbox on the Mac, see [How to uninstall +Toolbox](/toolbox/toolbox_install_mac.md#how-to-uninstall-toolbox) in the +Toolbox Mac topics. + +
+
    +
    + +
    diff --git a/docs/docker-for-mac/examples.md b/docs/docker-for-mac/examples.md new file mode 100644 index 0000000000..c784dd7042 --- /dev/null +++ b/docs/docker-for-mac/examples.md @@ -0,0 +1,38 @@ + + +# Example Applications + +Upcoming releases will include example applications especially tailored for Docker for Mac and Docker for Windows. + +Examples will highlight develop, build, and run workfows in several languages, including Node.js, Python, Ruby, and Java. + +For now, if you want get started experimenting with the Beta apps and Docker Compose (which is installed automatically with Docker Desktop Editions), have a look at these example applications in the Compose documentation. You should be able to run these with Docker for Mac and Docker for Windows. + +Quickstart: Compose and Django + +Quickstart: Compose and Rails + +Quickstart: Compose and WordPress + +See also [learn by example](/engine/tutorials/index.md) tutorials on building images, runnning containers, networking, managing data, and storing images on Docker Hub. + +
    +
      +
      + +
      diff --git a/docs/docker-for-mac/faqs.md b/docs/docker-for-mac/faqs.md new file mode 100644 index 0000000000..191cddc350 --- /dev/null +++ b/docs/docker-for-mac/faqs.md @@ -0,0 +1,156 @@ + + +# Frequently Asked Questions (FAQs) + +**Looking for popular FAQs on Docker for Mac?** Check out the [Docker Knowledge Hub](http://success.docker.com/) for knowledge base articles, FAQs, technical support for various subscription levels, and more. + +### Stable and beta channels + +**Q: How do I get the stable or beta version of Docker for Mac?** + +A: Use the download links for the channels given in the topic [Download Docker for Mac](index.md#download-docker-for-mac). + +This topic also has more information about the two channels. + +**Q: What is the difference between the stable and beta versions of Docker for Mac?** + +A: Two different download channels are available for Docker for Mac: + +* The stable channel provides a general availability release-ready installer for a fully baked and tested, more reliable app. The stable version of Docker for Mac comes with the latest released version of Docker Engine. The release schedule is synched with Docker Engine releases and hotfixes. + +* The beta channel provides an installer with new features we are working on, but is not necessarily fully tested. It comes with the experimental version of Docker Engine. Bugs, crashes and issues are more likely to occur with the beta app, but you get a chance to preview new functionality, experiment, and provide feedback as the apps evolve. Releases are typically more frequent than for stable, often one or more per month. + +**Q: Can I switch back and forth between stable and beta versions of Docker for Mac?** + +A: Yes, you can switch between versions to try out the betas to see what's new, then go back to stable for other work. However, **you can have only one app installed at a time**. Switching back and forth between stable and beta apps can de-stabilize your development environment, particularly in cases where you switch from a newer (beta) channel to older (stable). + +For example, containers created with a newer beta version of Docker for Mac may not work after you switch back to stable because they may have been created leveraging beta features that aren't in stable yet. Just keep this in mind as you create and work with beta containers, perhaps in the spirit of a playground space where you are prepared to troubleshoot or start over. + +To safely switch between beta and stable versions be sure to save images and export the containers you need, then uninstall the current version before installing another. The workflow is described in more detail below.
      + +Do the following each time: + +1. Use `docker save` to save any images you want to keep. (See [save](/engine/reference/commandline/save.md) in the Docker Engine command line reference.) + +2. Use `docker export` to export containers you want to keep. (See [export](/engine/reference/commandline/export.md) in the Docker Engine command line reference.) + +3. Uninstall the current app (whether stable or beta). + +4. Install a different version of the app (stable or beta). + +### What is Docker.app? + +`Docker.app` is Docker for Mac, a bundle of Docker client, and Docker +Engine. `Docker.app` uses the OS X +Hypervisor.framework (part of MacOS X 10.10 Yosemite and higher) +to run containers, meaning that _**no separate VirtualBox is required**_. + +### What kind of feedback are we looking for? + +Everything is fair game. We'd like your impressions on the download-install process, startup, functionality available, the GUI, usefulness of the app, +command line integration, and so on. Tell us about problems, what you like, or functionality you'd like to see added. + +We are especially interested in getting feedback on the new swarm mode described in [Docker Swarm](/engine/swarm/index.md). A good place to start is the [tutorial](/engine/swarm/swarm-tutorial/index.md). + +### What if I have problems or questions? + +You can find the list of frequent issues in +[Logs and Troubleshooting](troubleshoot.md). + +If you do not find a solution in Troubleshooting, browse issues on [Docker for Mac issues on GitHub](https://github.com/docker/for-mac/issues) or create a new one. You can also create new issues based on diagnostics. To learn more, see [Diagnose problems, send feedback, and create GitHub issues](troubleshoot.md#diagnose-problems-send-feedback-and-create-github-issues). + +[Docker for Mac forum](https://forums.docker.com/c/docker-for-mac) provides discussion threads as well, and you can create discussion topics there, but we recommend using the GitHub issues over the forums for better tracking and response. + +### Can I use Docker for Mac with new swarm mode? + +Yes, you can use Docker for Mac to test single-node features of [swarm mode](/engine/swarm/index.md) introduced with Docker Engine 1.12, including +initializing a swarm with a single node, creating services, and scaling +services. Docker “Moby” on Hyperkit will serve as the single swarm node. You can +also use Docker Machine, which comes with Docker for Mac, to create and +experiment a multi-node swarm. Check out the tutorial at [Get started with swarm mode](/engine/swarm/swarm-tutorial/index.md). + +### How do I connect to the remote Docker Engine API? + +You might need to provide the location of the remote API for Docker clients and development tools. + +On Docker for Mac, clients can connect to the Docker Engine through a Unix socket: `unix:///var/run/docker.sock`. + +See also [Docker Remote API](/engine/reference/api/docker_remote_api.md) and Docker for Mac forums topic [Using pycharm Docker plugin..](https://forums.docker.com/t/using-pycharm-docker-plugin-with-docker-beta/8617). + +If you are working with applications like [Apache Maven](https://maven.apache.org/) that expect settings for `DOCKER_HOST` and `DOCKER_CERT_PATH` environment variables, specify these to connect to Docker instances through Unix sockets. For example: + + export DOCKER_HOST=unix:///var/run/docker.sock + +### How do I connect from a container to a service on the host? + +The Mac has a changing IP address (or none if you have no network access). Our current recommendation is to attach an unused IP to the `lo0` interface on the Mac so that containers can connect to this address. + +For a full explanation and examples, see [I want to connect from a container to a service on the host](networking.md#i-want-to-connect-from-a-container-to-a-service-on-the-host) under [Known Limitations, Use Cases, and Workarounds](networking.md#known-limitations-use-cases-and-workarounds) in the Networking topic. + +### How do I to connect to a container from the Mac? + +Our current recommendation is to publish a port, or to connect from another container. Note that this is what you have to do even on Linux if the container is on an overlay network, not a bridge network, as these are not routed. + +For a full explanation and examples, see [I want to connect to a container from the Mac](networking.md#i-want-to-connect-to-a-container-from-the-mac) under [Known Limitations, Use Cases, and Workarounds](networking.md#known-limitations-use-cases-and-workarounds) in the Networking topic. + +### What are system requirements for Docker for Mac? + +Note that you need a Mac that supports hardware virtualization, which is most non ancient ones; i.e., use OS X `10.10.3+` or `10.11` (OS X Yosemite or OS X El Capitan). See also "What to know before you install" in [Getting Started](index.md). + + +### Do I need to uninstall Docker Toolbox to use Docker for Mac? + +No, you can use these side by side. Docker Toolbox leverages a Docker daemon installed using `docker-machine` in a machine called `default`. Running `eval $(docker-machine env default)` in a shell sets DOCKER environment variables locally to connect to the default machine using Engine from Toolbox. To check whether Toolbox DOCKER environment variables are set, run `env | grep DOCKER`. + +To make the client talk to the Docker for Mac Engine, run the command `unset ${!DOCKER_*}` to unset all DOCKER environment variables in the current shell. (Now, `env | grep DOCKER` should return no output.) You can have multiple command line shells open, some set to talk to Engine from Toolbox and others set to talk to Docker for Mac. The same applies to `docker-compose`. + +### How do I uninstall Docker Toolbox? + +You might decide that you do not need Toolbox now that you have Docker for Mac, +and want to uninstall it. For details on how to perform a clean uninstall of +Toolbox on the Mac, see [How to uninstall +Toolbox](/toolbox/toolbox_install_mac.md#how-to-uninstall-toolbox) in the +Toolbox Mac topics. + +### What is HyperKit? + +HyperKit is a hypervisor built on top of the Hypervisor.framework in OS X 10.10 Yosemite and higher. It runs entirely in userspace and has no other dependencies. + +We use HyperKit to eliminate the need for other VM products, such as Oracle Virtualbox or VMWare Fusion. + +### What is the benefit of HyperKit? + +It is thinner than VirtualBox and VMWare fusion, and the version we include is tailor made for Docker workloads on the Mac. + +### Why is com.docker.vmnetd running after I quit the app? + +The privileged helper process `com.docker.vmnetd` is started by `launchd` and runs in the background. The process will not +consume any resources unless Docker.app connects to it, so it's safe to ignore. + +### Can I pass through a USB device to a container? + + Unfortunately it is not possible to pass through a USB device (or a serial port) to a container. For use cases requiring this, we recommend the use of [Docker Toolbox](/toolbox/overview.md). + +
      +
        +
        + +
        diff --git a/docs/docker-for-mac/images/About.png b/docs/docker-for-mac/images/About.png new file mode 100644 index 0000000000..4fc78daf7b Binary files /dev/null and b/docs/docker-for-mac/images/About.png differ diff --git a/docs/docker-for-mac/images/changelog-placeholder.png b/docs/docker-for-mac/images/changelog-placeholder.png new file mode 100644 index 0000000000..87717b49cd Binary files /dev/null and b/docs/docker-for-mac/images/changelog-placeholder.png differ diff --git a/docs/docker-for-mac/images/chat.png b/docs/docker-for-mac/images/chat.png new file mode 100644 index 0000000000..597db5aae9 Binary files /dev/null and b/docs/docker-for-mac/images/chat.png differ diff --git a/docs/docker-for-mac/images/console_logs.png b/docs/docker-for-mac/images/console_logs.png new file mode 100644 index 0000000000..951eac2820 Binary files /dev/null and b/docs/docker-for-mac/images/console_logs.png differ diff --git a/docs/docker-for-mac/images/console_logs_search.png b/docs/docker-for-mac/images/console_logs_search.png new file mode 100644 index 0000000000..bd3583b3cb Binary files /dev/null and b/docs/docker-for-mac/images/console_logs_search.png differ diff --git a/docs/docker-for-mac/images/diagnose-d4mac-issues-template.png b/docs/docker-for-mac/images/diagnose-d4mac-issues-template.png new file mode 100644 index 0000000000..328328e516 Binary files /dev/null and b/docs/docker-for-mac/images/diagnose-d4mac-issues-template.png differ diff --git a/docs/docker-for-mac/images/diagnose-id-forums.png b/docs/docker-for-mac/images/diagnose-id-forums.png new file mode 100644 index 0000000000..ae902907fb Binary files /dev/null and b/docs/docker-for-mac/images/diagnose-id-forums.png differ diff --git a/docs/docker-for-mac/images/diagnose-issue.png b/docs/docker-for-mac/images/diagnose-issue.png new file mode 100644 index 0000000000..4dd243a2c2 Binary files /dev/null and b/docs/docker-for-mac/images/diagnose-issue.png differ diff --git a/docs/docker-for-mac/images/diagnose.png b/docs/docker-for-mac/images/diagnose.png new file mode 100644 index 0000000000..8c3a5945ca Binary files /dev/null and b/docs/docker-for-mac/images/diagnose.png differ diff --git a/docs/docker-for-mac/images/diagnostic-forums-topic.png b/docs/docker-for-mac/images/diagnostic-forums-topic.png new file mode 100644 index 0000000000..09973b0c96 Binary files /dev/null and b/docs/docker-for-mac/images/diagnostic-forums-topic.png differ diff --git a/docs/docker-for-mac/images/docker-app-drag-old.png b/docs/docker-for-mac/images/docker-app-drag-old.png new file mode 100644 index 0000000000..b05124d3b1 Binary files /dev/null and b/docs/docker-for-mac/images/docker-app-drag-old.png differ diff --git a/docs/docker-for-mac/images/docker-app-drag.png b/docs/docker-for-mac/images/docker-app-drag.png new file mode 100644 index 0000000000..b4744e47d8 Binary files /dev/null and b/docs/docker-for-mac/images/docker-app-drag.png differ diff --git a/docs/docker-for-mac/images/docker-app-in-apps-no-annotation.png b/docs/docker-for-mac/images/docker-app-in-apps-no-annotation.png new file mode 100644 index 0000000000..c1ace536af Binary files /dev/null and b/docs/docker-for-mac/images/docker-app-in-apps-no-annotation.png differ diff --git a/docs/docker-for-mac/images/docker-app-in-apps.png b/docs/docker-for-mac/images/docker-app-in-apps.png new file mode 100644 index 0000000000..2290320ea4 Binary files /dev/null and b/docs/docker-for-mac/images/docker-app-in-apps.png differ diff --git a/docs/docker-for-mac/images/docker-app-log.png b/docs/docker-for-mac/images/docker-app-log.png new file mode 100644 index 0000000000..6aa36e7aef Binary files /dev/null and b/docs/docker-for-mac/images/docker-app-log.png differ diff --git a/docs/docker-for-mac/images/docker-app.png b/docs/docker-for-mac/images/docker-app.png new file mode 100644 index 0000000000..530fccb9ca Binary files /dev/null and b/docs/docker-for-mac/images/docker-app.png differ diff --git a/docs/docker-for-mac/images/docker-for-mac-and-toolbox.png b/docs/docker-for-mac/images/docker-for-mac-and-toolbox.png new file mode 100644 index 0000000000..1bbf09bd0a Binary files /dev/null and b/docs/docker-for-mac/images/docker-for-mac-and-toolbox.png differ diff --git a/docs/docker-for-mac/images/docker-for-mac-install.png b/docs/docker-for-mac/images/docker-for-mac-install.png new file mode 100644 index 0000000000..6a51c8b858 Binary files /dev/null and b/docs/docker-for-mac/images/docker-for-mac-install.png differ diff --git a/docs/docker-for-mac/images/download.png b/docs/docker-for-mac/images/download.png new file mode 100644 index 0000000000..bfa896d89e Binary files /dev/null and b/docs/docker-for-mac/images/download.png differ diff --git a/docs/docker-for-mac/images/hello-world-nginx.png b/docs/docker-for-mac/images/hello-world-nginx.png new file mode 100644 index 0000000000..bf6cb20496 Binary files /dev/null and b/docs/docker-for-mac/images/hello-world-nginx.png differ diff --git a/docs/docker-for-mac/images/hello-world.png b/docs/docker-for-mac/images/hello-world.png new file mode 100644 index 0000000000..ededac6bbd Binary files /dev/null and b/docs/docker-for-mac/images/hello-world.png differ diff --git a/docs/docker-for-mac/images/hockeyapp-docker.png b/docs/docker-for-mac/images/hockeyapp-docker.png new file mode 100644 index 0000000000..d07f85b97c Binary files /dev/null and b/docs/docker-for-mac/images/hockeyapp-docker.png differ diff --git a/docs/docker-for-mac/images/log-files-finder.png b/docs/docker-for-mac/images/log-files-finder.png new file mode 100644 index 0000000000..a006274086 Binary files /dev/null and b/docs/docker-for-mac/images/log-files-finder.png differ diff --git a/docs/docker-for-mac/images/logs.png b/docs/docker-for-mac/images/logs.png new file mode 100644 index 0000000000..d16ff51520 Binary files /dev/null and b/docs/docker-for-mac/images/logs.png differ diff --git a/docs/docker-for-mac/images/mac-activity-monitor-docker-app.png b/docs/docker-for-mac/images/mac-activity-monitor-docker-app.png new file mode 100644 index 0000000000..4510767cea Binary files /dev/null and b/docs/docker-for-mac/images/mac-activity-monitor-docker-app.png differ diff --git a/docs/docker-for-mac/images/mac-install-success-docker-ps.png b/docs/docker-for-mac/images/mac-install-success-docker-ps.png new file mode 100644 index 0000000000..62e8b9aad1 Binary files /dev/null and b/docs/docker-for-mac/images/mac-install-success-docker-ps.png differ diff --git a/docs/docker-for-mac/images/mac-install-success-docker-wait.png b/docs/docker-for-mac/images/mac-install-success-docker-wait.png new file mode 100644 index 0000000000..a6bb45f816 Binary files /dev/null and b/docs/docker-for-mac/images/mac-install-success-docker-wait.png differ diff --git a/docs/docker-for-mac/images/menu.png b/docs/docker-for-mac/images/menu.png new file mode 100644 index 0000000000..659e329309 Binary files /dev/null and b/docs/docker-for-mac/images/menu.png differ diff --git a/docs/docker-for-mac/images/privacy.png b/docs/docker-for-mac/images/privacy.png new file mode 100644 index 0000000000..7f0a52aaa7 Binary files /dev/null and b/docs/docker-for-mac/images/privacy.png differ diff --git a/docs/docker-for-mac/images/proxy-settings.png b/docs/docker-for-mac/images/proxy-settings.png new file mode 100644 index 0000000000..7411df9afc Binary files /dev/null and b/docs/docker-for-mac/images/proxy-settings.png differ diff --git a/docs/docker-for-mac/images/remove-app-instances.png b/docs/docker-for-mac/images/remove-app-instances.png new file mode 100644 index 0000000000..5ba3588c4e Binary files /dev/null and b/docs/docker-for-mac/images/remove-app-instances.png differ diff --git a/docs/docker-for-mac/images/settings-advanced.png b/docs/docker-for-mac/images/settings-advanced.png new file mode 100644 index 0000000000..9d735f9555 Binary files /dev/null and b/docs/docker-for-mac/images/settings-advanced.png differ diff --git a/docs/docker-for-mac/images/settings-diagnose-id.png b/docs/docker-for-mac/images/settings-diagnose-id.png new file mode 100644 index 0000000000..f149356cd3 Binary files /dev/null and b/docs/docker-for-mac/images/settings-diagnose-id.png differ diff --git a/docs/docker-for-mac/images/settings-diagnose.png b/docs/docker-for-mac/images/settings-diagnose.png new file mode 100644 index 0000000000..28547df813 Binary files /dev/null and b/docs/docker-for-mac/images/settings-diagnose.png differ diff --git a/docs/docker-for-mac/images/settings-diagnostic-results-only.png b/docs/docker-for-mac/images/settings-diagnostic-results-only.png new file mode 100644 index 0000000000..82f1fd3fda Binary files /dev/null and b/docs/docker-for-mac/images/settings-diagnostic-results-only.png differ diff --git a/docs/docker-for-mac/images/settings-file-share-choose.png b/docs/docker-for-mac/images/settings-file-share-choose.png new file mode 100644 index 0000000000..248407e8e7 Binary files /dev/null and b/docs/docker-for-mac/images/settings-file-share-choose.png differ diff --git a/docs/docker-for-mac/images/settings-file-share.png b/docs/docker-for-mac/images/settings-file-share.png new file mode 100644 index 0000000000..69b26c3802 Binary files /dev/null and b/docs/docker-for-mac/images/settings-file-share.png differ diff --git a/docs/docker-for-mac/images/settings-uninstall.png b/docs/docker-for-mac/images/settings-uninstall.png new file mode 100644 index 0000000000..bfb65af22c Binary files /dev/null and b/docs/docker-for-mac/images/settings-uninstall.png differ diff --git a/docs/docker-for-mac/images/settings.png b/docs/docker-for-mac/images/settings.png new file mode 100644 index 0000000000..a9f9e95f67 Binary files /dev/null and b/docs/docker-for-mac/images/settings.png differ diff --git a/docs/docker-for-mac/images/startup-help.png b/docs/docker-for-mac/images/startup-help.png new file mode 100644 index 0000000000..0db4aca3db Binary files /dev/null and b/docs/docker-for-mac/images/startup-help.png differ diff --git a/docs/docker-for-mac/images/toolbox-install.png b/docs/docker-for-mac/images/toolbox-install.png new file mode 100644 index 0000000000..fa10bc28b9 Binary files /dev/null and b/docs/docker-for-mac/images/toolbox-install.png differ diff --git a/docs/docker-for-mac/images/whale-in-menu-bar.png b/docs/docker-for-mac/images/whale-in-menu-bar.png new file mode 100644 index 0000000000..780201e127 Binary files /dev/null and b/docs/docker-for-mac/images/whale-in-menu-bar.png differ diff --git a/docs/docker-for-mac/images/whale-x.png b/docs/docker-for-mac/images/whale-x.png new file mode 100644 index 0000000000..c99e8d5898 Binary files /dev/null and b/docs/docker-for-mac/images/whale-x.png differ diff --git a/docs/docker-for-mac/images/whale.png b/docs/docker-for-mac/images/whale.png new file mode 100644 index 0000000000..894c379349 Binary files /dev/null and b/docs/docker-for-mac/images/whale.png differ diff --git a/docs/docker-for-mac/index.md b/docs/docker-for-mac/index.md new file mode 100644 index 0000000000..41049349f8 --- /dev/null +++ b/docs/docker-for-mac/index.md @@ -0,0 +1,293 @@ + + +# Getting Started with Docker for Mac + +Welcome to Docker for Mac! + +Please read through these topics on how to get started. To **give us feedback** on your experience with the app and report bugs or problems, log in to our [Docker for Mac forum](https://forums.docker.com/c/docker-for-mac). + +>**Already have Docker for Mac?** If you already have Docker for Mac installed, and are ready to get started, skip over to the [Getting Started with Docker](/engine/getstarted/index.md) tutorial. + + +## Download Docker for Mac + +If you have not already done so, please install Docker for Mac. You can download installers from the stable or beta channel. + +For more about stable and beta channels, see the [FAQs](faqs.md#stable-and-beta-channels). + + + + + + + + + + + + + + +
        Stable channelBeta channel
        This installer is fully baked and tested, and comes with the latest GA version of Docker Engine.

        This is the best channel to use if you want a reliable platform to work with.

        These releases follow a version schedule with a longer lead time than the betas, synched with Docker Engine releases and hotfixes. +         		This installer offers cutting edge features and comes with the experimental version of Docker Engine, which is described in the Docker Experimental Features README on GitHub.

        This is the best channel to use if you want to experiment with features we are working on as they become available, and can weather some instability and bugs. This channel is a continuation of the beta program, where you can provide feedback as the apps evolve. Releases are typically more frequent than for stable, often one or more per month.
        + Get Docker for Mac (stable)

        + Download checksum: Docker.dmg SHA256 +         		+ Get Docker for Mac (beta)

        + Download checksum: Docker.dmg SHA256 +
        + +>**Important Notes**: +> +>* Docker for Mac requires OS X 10.10.3 Yosemite or newer running on a 2010 or newer Mac, with Intel's hardware support for MMU virtualization. Please see [What to know before you install](#what-to-know-before-you-install) for a full list of prerequisites. +> +>* You can switch between beta and stable versions, but _you must have only one app installed at a time_. Also, you will need to save images and export containers you want to keep before uninstalling the current version before installing another. For more about this, see the [FAQs about beta and stable channels](faqs.md#stable-and-beta-channels). + + +## What to know before you install + +* **README FIRST for Docker Toolbox and Docker Machine users**: If you are already running Docker on your machine, first read [Docker for Mac vs. Docker Toolbox](docker-toolbox.md) to understand the impact of this installation on your existing setup, how to set your environment for Docker for Mac, and how the two products can coexist. + +* **Relationship to Docker Machine**: Installing Docker for Mac does not affect machines you created with Docker Machine. You'll get the option to copy containers and images from your local `default` machine (if one exists) to the new Docker for Mac HyperKit VM. + +* **System Requirements**: Docker for Mac will launch only if all these requirements are met. + + - Mac must be a 2010 or newer model, with Intel's hardware support for memory management unit (MMU) virtualization; i.e., Extended Page Tables (EPT) + + - OS X 10.10.3 Yosemite or newer + + - At least 4GB of RAM + + - VirtualBox prior to version 4.3.30 must NOT be installed (it is incompatible with Docker for Mac) + + > **Note**: If your system does not satisfy these requirements, you can install [Docker Toolbox](/toolbox/overview.md), which uses Oracle Virtual Box instead of HyperKit. + +* **What the install includes**: The installation provides [Docker Engine](https://docs.docker.com/engine/userguide/intro/), Docker CLI client, [Docker Compose](https://docs.docker.com/compose/overview/), and [Docker Machine](https://docs.docker.com/machine/overview/). + + +## Step 1. Install and Run Docker for Mac + +1. Double-click `Docker.dmg` to open the installer, then drag Moby the whale to the Applications folder. + + ![Install Docker app](images/docker-app-drag.png) + + You will be asked to authorize `Docker.app` with your system password during the install process. Privileged access is needed to install networking components and links to the Docker apps. + +2. Double-click `Docker.app` to start Docker. + + ![Docker app in Hockeyapp](images/docker-app-in-apps.png) + + The whale in the top status bar indicates that Docker is running, and accessible from a terminal. + + ![Whale in menu bar](images/whale-in-menu-bar.png) + + If you just installed the app, you also get a success message with suggested next steps and a link to this documentation. Click the whale () in the status bar to dismiss this popup. + + ![Docker success](images/mac-install-success-docker-ps.png) + +3. Click the whale () to get Preferences, and other options. + + ![Docker context menu](images/menu.png) + +4. Select **About Docker** to verify that you have the latest version. + + Congratulations! You are up and running with Docker for Mac. + + +## Step 2. Check versions of Docker Engine, Compose, and Machine + +Run these commands to test if your versions of `docker`, `docker-compose`, and `docker-machine` are up-to-date and compatible with `Docker.app`. + +```shell + $ docker --version + Docker version 1.12.0, build 8eab29e + + $ docker-compose --version + docker-compose version 1.8.0, build f3628c7 + + $ docker-machine --version + docker-machine version 0.8.0, build b85aac1 +``` + +>**Note**: The above is an example. Your output will differ if you are running different (e.g., newer) versions. + + +## Step 3. Explore the application and run examples + +1. Open a command-line terminal, and run some Docker commands to verify that Docker is working as expected. + + Some good commands to try are `docker version` to check that you have the latest release installed, and `docker ps` and `docker run hello-world` to verify that Docker is running. + +2. For something more adventurous, start a Dockerized web server. + + ```shell + docker run -d -p 80:80 --name webserver nginx + ``` + + If the image is not found locally, Docker will pull it from Docker Hub. + + In a web browser, go to `http://localhost/` to bring up the home page. (Since you specified the default HTTP port, it isn't necessary to append `:80` at the end of the URL.) + + ![nginx home page](images/hello-world-nginx.png) + + >**Note:** Early beta releases used `docker` as the hostname to build the URL. Now, ports are exposed on the private IP addresses of the VM and forwarded to `localhost` with no other host name set. See also, [Release Notes](release-notes.md) for Beta 9. + > + +3. Run `docker ps` while your web server is running to see details on the webserver container. + + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 56f433965490 nginx "nginx -g 'daemon off" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, 443/tcp webserver + +4. Stop or remove containers and images. + + The `nginx` webserver will continue to run in the container on that port until you stop and/or remove the container. If you want to stop the webserver, type: `docker stop webserver` and start it again with `docker start webserver`. + + To stop and remove the running container with a single command, type: `docker rm -f webserver`. This will remove the container, but not the `nginx` image. You can list local images with `docker images`. You might want to keep some images around so that you don't have to pull them again from Docker Hub. To remove an image you no longer need, use `docker rmi |`. For example, `docker rmi nginx`. + +**Want more example applictions?** - For more example walkthroughs that include setting up services and databases in Docker Compose, see [Example Applications](examples.md). + +## Preferences + +Choose --> **Preferences** from the menu bar. You can set the following runtime options. + +#### General + +![Preferences](images/settings.png) + +* Docker for Mac is set to **automatically start** when you log in. Uncheck the login autostart option if you don't want Docker to start when you open your session. + +* Docker for Mac is set to **check for updates** automatically and notify you when an update is available. If an update is found, click **OK** to accept and install it (or cancel to keep the current version). If you disable the check for updates, you can still find out about updates manually by choosing -> **Check for Updates** + +* Check **Exclude VM from Time Machine backups** to prevent Time Machine from backing up the Docker for Mac virtual machine. + +* **CPUs** - By default, Docker for Mac is set to use 2 processors. You can increase processing power for the app by setting this to a higher number, or lower it to have Docker for Mac use fewer computing resources. + +* **Memory** - By default, Docker for Mac is set to use `2` GB runtime memory, allocated from the total available memory on your Mac. You can increase the RAM on the app to get faster performance by setting this number higher (for example to `3`) or lower (to `1`) if you want Docker for Mac to use less memory. + +#### Advanced + +![Advanced Preference settings-advanced](images/settings-advanced.png) + +* **Adding registries** - As an alternative to using [Docker Hub](https://hub.docker.com/) to store your public or private images or [Docker Trusted Registry](https://docs.docker.com/docker-trusted-registry/overview/), you can use Docker to set up your own insecure [registry](https://docs.docker.com/registry/introduction/). Add URLs for insecure registries and registry mirrors on which to host your images. + +* **HTTP proxy settings** - Docker for Mac will detect HTTP/HTTPS Proxy Settings and automatically propagate these to Docker and to your containers. +For example, if you set your proxy settings to `http://proxy.example.com`, Docker will use this proxy when pulling containers. + + +#### File sharing + +You can decide which directories on your Mac to share with containers. + +* **Add a Directory** - Click `+` and navigate to the directory you want to add. + + ![File Sharing](images/settings-file-share.png) + +* Click **Apply & Restart** to make the directory available to + containers using Docker's bind mount (`-v`) feature. + +There are some limitations on the directories that can be shared: + +* They cannot be a subdirectory of an already shared directory. + +* They cannot already exist inside of Docker. + +See [Namespaces](osxfs.md#namespaces) in the topic on [osxfs file system sharing](osxfs.md) for more information. + +#### Privacy + +You can set Docker for Mac to auto-send diagnostics, crash reports, and usage data. This information can help Docker improve the application and get more context for troubleshooting problems. + +Uncheck any of the options to opt out and prevent auto-send of data. Docker may prompt for more information in some cases, even with auto-send enabled. + +![Privacy](images/privacy.png) + +Also, you can enable or disable these auto-reporting settings with one click on the information popup when you first start Docker. + +![Startup information](images/mac-install-success-docker-wait.png) + +## Uninstall or reset +Choose --> **Preferences** from the menu bar, then click **Uninstall / Reset** on the Preferences dialog. + +![Uninstall or reset Docker](images/settings-uninstall.png) + +* **Uninstall** - Choose this option to remove Docker for Mac from your system. + +* **Reset to factory defaults** - Choose this option to reset all options on Docker for Mac to its initial state, the same as when it was first installed. + +You can uninstall Docker for Mac from the command line with this command: ` --uninstall`. If Docker is installed in the default location, the following command will provide a clean uninstall. + +```shell +$ /Applications/Docker.app/Contents/MacOS/Docker --uninstall +Docker is running, exiting... +Docker uninstalled successfully. You can move the Docker application to the trash. +``` + +You might want to use the command-line uninstall if, for example, you find that the app is non-functional, and you cannot uninstall it from the menu. + + +## Installing bash completion + +If you are using [bash completion](https://www.debian-administration.org/article/316/An_introduction_to_bash_completion_part_1), such as [homebrew bash-completion on Mac](http://davidalger.com/development/bash-completion-on-os-x-with-brew/), bash completion scripts for +- docker +- docker-machine +- docker-compose +may be found inside Docker.app, in the Contents/Resources/etc folder. + +To activate bash completion, these files need to be copied or symlinked to +your bash_completion.d directory. For example, if you use Homebrew: + +``` +cd /usr/local/etc/bash_completion.d +ln -s /Applications/Docker.app/Contents/Resources/etc/docker.bash-completion +ln -s /Applications/Docker.app/Contents/Resources/etc/docker-machine.bash-completion +ln -s /Applications/Docker.app/Contents/Resources/etc/docker-compose.bash-completion +``` + +## Where to go next + +* Try out the [Getting Started with Docker](/engine/getstarted/index.md) tutorial. + +* Dig in deeper with [learn by example](/engine/tutorials/index.md) tutorials on on building images, runnning containers, networking, managing data, and storing images on Docker Hub. + +* See [Example Applications](examples.md) for example applications that include setting up services and databases in Docker Compose. + +* Interested in trying out the new [swarm mode](/engine/swarm/index.md) on Docker Engine v1.12? + + See [Get started with swarm mode](/engine/swarm/swarm-tutorial/index.md), a tutorial which includes specifics on how to leverage your Docker for Mac installation to run single and multi-node swarms. + + Also, try out the Swarm examples in [docker labs](https://github.com/docker/labs/tree/master/swarm-mode/beginner-tutorial). Run the `bash script` and follow the accompanying [Docker Swarm Tutorial](https://github.com/docker/labs/blob/master/swarm-mode/beginner-tutorial/README.md). The script uses Docker Machine to create a multi-node swarm, then walks you through various Swarm tasks and commands. + +* For a summary of Docker command line interface (CLI) commands, see [Docker CLI Reference Guide](/engine/reference/index.md). + +* Check out the blog posts on Docker for Mac and Docker for Windows public betas, and earlier posts on the intial private beta. + +* Please give feedback on your experience with the app and report bugs and problems by logging into our [Docker for Mac forum](https://forums.docker.com/c/docker-for-mac). + + +
        +
          +
          + +
          diff --git a/docs/docker-for-mac/menu.md b/docs/docker-for-mac/menu.md new file mode 100644 index 0000000000..fa706b6da0 --- /dev/null +++ b/docs/docker-for-mac/menu.md @@ -0,0 +1,14 @@ + + +# Docker for Mac diff --git a/docs/docker-for-mac/multi-arch.md b/docs/docker-for-mac/multi-arch.md new file mode 100644 index 0000000000..6bd76ff4ef --- /dev/null +++ b/docs/docker-for-mac/multi-arch.md @@ -0,0 +1,48 @@ + + +# Leveraging Multi-CPU Architecture Support + +Docker for Mac provides `binfmt_misc` multi architecture support, so you can run containers for different Linux architectures, such as `arm`, `mips`, `ppc64le` and even `s390x`. + +This should just work without any configuration, but the containers you run need to have the appropriate `qemu` binary inside the container before you can do this. (See QEMU for more information.) + +So, you can run a container that already has this set up, like the resin arm builds: + +``` +$ docker run resin/armv7hf-debian uname -a + +Linux 7ed2fca7a3f0 4.1.12 #1 SMP Tue Jan 12 10:51:00 UTC 2016 armv7l GNU/Linux + +$ docker run justincormack/ppc64le-debian uname -a + +Linux edd13885f316 4.1.12 #1 SMP Tue Jan 12 10:51:00 UTC 2016 ppc64le GNU/Linux + +``` + +Running containers pre-configured with `qemu` has the advantage that you can use these to do builds `FROM`, so you can build new Multi-CPU architecture packages. + +Alternatively, you can bind mount in the `qemu` static binaries to any cross-architecture package, such as the semi-official ones using a script like this one https://github.com/justincormack/cross-docker. (See the README at the given link for details on how to use the script.) + +
          +
            +
            + +
            diff --git a/docs/docker-for-mac/networking.md b/docs/docker-for-mac/networking.md new file mode 100644 index 0000000000..8a7b579f15 --- /dev/null +++ b/docs/docker-for-mac/networking.md @@ -0,0 +1,121 @@ + + +# Networking + +Docker for Mac provides several networking features to make it easier to use. + +## Features + +### VPN Passthrough + +Docker for Mac's networking can work when attached to a VPN. +To do this, Docker for Mac intercepts traffic from the `HyperKit` and injects it into OSX as if it originated from the Docker application. + +### Port Mapping + +When you run a container with the `-p` argument, for example: +``` +$ docker run -p 80:80 -d nginx +``` +Docker for Mac will make the container port available at `localhost`. + +### HTTP/HTTPS Proxy Support + +Docker for Mac will detect HTTP/HTTPS Proxy Settings from OSX and automatically propagate these to Docker and to your containers. +For example, if you set your proxy settings to `http://proxy.example.com` in OSX, Docker will use this proxy when pulling containers. + +![OSX Proxy Settings](images/proxy-settings.png) + +When you start a container, you will see that your proxy settings propagate into the containers. For example: + +``` +$ docker run -it alpine env +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +HOSTNAME=b7edf988b2b5 +TERM=xterm +HOME=/root +HTTP_PROXY=http://proxy.example.com:3128 +http_proxy=http://proxy.example.com:3128 +no_proxy=*.local, 169.254/16 +``` + +You can see from the above output that the `HTTP_PROXY`, `http_proxy` and `no_proxy` environment variables are set. +When your proxy configuration changes, Docker restarts automatically to pick up the new settings. +If you have containers that you wish to keep running across restarts, you should consider using [restart policies](https://docs.docker.com/engine/reference/run/#restart-policies-restart) + +## Known Limitations, Use Cases, and Workarounds + +Following is a summary of current limitations on the Docker for Mac networking stack, along with some ideas for workarounds. + +### There is no docker0 bridge on OSX + +Because of the way networking is implemented in Docker for Mac, you cannot see a `docker0` interface in OSX. +This interface is actually within `HyperKit`. + +### I cannot ping my containers + +Unfortunately, due to limtations in OSX, we're unable to route traffic to containers, and from containers back to the host. + +### Per-container IP addressing is not possible + +The docker (Linux) bridge network is not reachable from the OSX host. + +### Use cases and workarounds + +There are two scenarios that the above limitations will affect: + +#### I want to connect from a container to a service on the host + +The Mac has a changing IP address (or none if you have no network access). Our current recommendation is to attach an unused IP to the `lo0` interface on the Mac; for example: `sudo ifconfig lo0 alias 10.200.10.1/24`, and make sure that your service is listening on this address or `0.0.0.0` (ie not `127.0.0.1`). Then containers can connect to this address. + +#### I want to connect to a container from the Mac + +Port forwarding works for `localhost`; `--publish`, `-p`, or `-P` all work. Ports exposed from Linux are forwarded to the Mac. + +Our current recommendation is to publish a port, or to connect from another container. Note that this is what you have to do even on Linux if the container is on an overlay network, not a bridge network, as these are not routed. + +The command to run the `nginx` webserver shown in [Getting Started](index.md#explore) is an example of this. + +```shell +docker run -d -p 80:80 --name webserver nginx +``` + +To clarify the syntax, the following two commands both expose port `80` on the container to port `8000` on the host: + + docker run --publish 8000:80 --name webserver nginx + docker run --p 8000:80 --name webserver nginx + +To expose all ports, use the `-P` flag. For example, the following command starts a container (in detached mode) and the `-P` exposes all ports on the container to random ports on the host. + + docker run -d -P --name webserver nginx + +See the [run commmand](/engine/reference/commandline/run.md) for more details on publish options used with `docker run`. + +#### A view into implementation + +We understand that these workarounds are not ideal, but there are several problems. In particular, there is a bug in OSX that is only fixed in 10.12 and is not being backported as far as we can tell, which means that we could not support this in all supported OSX versions. In addition, this network setup would require root access which we are trying to avoid entirely in Docker for Mac (we currently have a very small root helper that we are trying to remove). + + +
            +
              +
              + +
              diff --git a/docs/docker-for-mac/opensource.md b/docs/docker-for-mac/opensource.md new file mode 100644 index 0000000000..a21338693d --- /dev/null +++ b/docs/docker-for-mac/opensource.md @@ -0,0 +1,34 @@ + + +# Open Source Components and Licensing + +Docker Desktop Editions are built using open source software. For +details on the licensing, choose --> +**About Docker** from within the application, then click +**Acknowledgements**. + +Docker Desktop Editions distribute some components that are licensed under the GNU General Public License. You can download the source for these components [here](https://download.docker.com/opensource/License.tar.gz). + +The sources for `qemu-img` can be obtained [here](http://wiki.qemu-project.org/download/qemu-2.4.1.tar.bz2). +The sources for the `gettext` and `glib` libraries that `qemu-img` requires were obtained from [Homebrew](https://brew.sh) and may be retrieved using `brew install --build-from-source gettext glib`. + +
              +
                +
                + +
                diff --git a/docs/docker-for-mac/osxfs.md b/docs/docker-for-mac/osxfs.md new file mode 100644 index 0000000000..6d17758967 --- /dev/null +++ b/docs/docker-for-mac/osxfs.md @@ -0,0 +1,166 @@ + + +# File system sharing (osxfs) + +`osxfs` is a new shared file system solution, exclusive to Docker for +Mac. `osxfs` provides a close-to-native +user experience for bind mounting OS X file system trees into Docker +containers. To this end, `osxfs` features a number of unique +capabilities as well as differences from a classical Linux file system. + +- [Case sensitivity](#case-sensitivity) +- [Access control](#access-control) +- [Namespaces](#namespaces) +- [Ownership](#ownership) +- [File system events](#file-system-events) +- [Mounts](#mounts) +- [Symlinks](#symlinks) +- [File types](#file-types) +- [Extended attributes](#extended-attributes) +- [Technology](#technology) + +### Case sensitivity + +With Docker for Mac, file systems are shared from OS X into containers +in the same way as they operate in OS X. As a result, if a file system +on OS X is case-insensitive that behavior is shared by any bind mount +from OS X into a container. The default OS X file system is HFS+ and, +during installation, it is installed as case-insensitive by default. To +get case-sensitive behavior from your bind mounts, you must either +create and format a ramdisk or external volume as HFS+ with +case-sensitivity or reformat your OS root partition with HFS+ with +case-sensitivity. We do not recommend reformatting your root partition +as some Mac software dubiously relies on case-insensitivity to function. + + +### Access control + +`osxfs`, and therefore Docker, can access only those file system +resources that the Docker for Mac user has access to. `osxfs` does +not run as `root`. If the OS X user is an administrator, `osxfs` inherits +those administrator privileges. We are still evaluating which privileges +to drop in the file system process to balance security and +ease-of-use. `osxfs` performs no additional permissions checks and +enforces no extra access control on accesses made through it. All +processes in containers can access the same objects in the same way as +the Docker user who started the containers. + +### Namespaces + +Much of the OS X file system that is accessible to the user is also +available to containers using the `-v` bind mount syntax. By default, +you can share files in `/Users`, `/Volumes`, `/private`, and `/tmp` +directly. To add or remove directory trees that are exported to Docker, +use the **File sharing** tab in Docker preferences -> **Preferences** -> **File +sharing**. (See [Preferences](index.md#preferences).) All other paths +used in `-v` bind mounts are sourced from the Moby Linux VM running the +Docker containers, so arguments such as `-v +/var/run/docker.sock:/var/run/docker.sock` should work as expected. If +an OS X path is not shared and does not exist in th VM, an attempt to +bind mount it will fail rather than create it in the VM. Paths that +already exist in the VM and contain files are reserved by Docker and +cannot be exported from OS X. + +### Ownership + +Initially, any containerized process that requests ownership metadata of +an object is told that its `uid` and `gid` own the object. When any +containerized process changes the ownership of a shared file system +object, e.g. with `chown`, the new ownership information is persisted in +the `com.docker.owner` extended attribute of the object. Subsequent +requests for ownership metadata will return the previously set +values. Ownership-based permissions are only enforced at the OS X file +system level with all accessing processes behaving as the user running +Docker. If the user does not have permission to read extended attributes +on an object, e.g. when that object's permissions are `0000`, ownership +will be reported as the accessing process until the extended attribute +is again readable. + +### File system events + +Most `inotify` events are supported in bind mounts, and likely `dnotify` +and `fanotify` (though they have not been tested) are also supported. +This means that file system events from OS X are sent into containers +and trigger any listening processes there. + +The following are **supported file system events**: + +* Creation +* Modification +* Attribute changes +* Deletion +* Directory changes + +The following are **partially supported file system events**: + +* Move events trigger `IN_DELETE` on the source of the rename and + `IN_MODIFY` on the destination of the rename + +The following are **unsupported file system events**: + +* Open +* Access +* Close events +* Unmount events (see Mounts) + +Some events may be delivered multiple times. Events are not delivered for bind mounts from symlinks (notably `/tmp` will not deliver inotify events but +`/private/tmp` will). These limitations do not apply to events between +containers, only to those events originating in OS X. + + +### Mounts + +The OS X mount structure is not visible in the shared volume, but volume +contents are visible. Volume contents appear in the same file system as the +rest of the shared file system. Mounting/unmounting OS X volumes that +are also bind mounted into containers may result in unexpected behavior +in those containers. Unmount events are not supported. Mount export +support is planned but is still under development. + +### Symlinks + +Symlinks are shared unmodified. This may cause issues when symlinks +contain paths that rely on the default case-insensitivity of the +default OS X file system, HFS+. + +### File types + +Symlinks, hardlinks, socket files, named pipes, regular files, and +directories are supported. Socket files and named pipes only transmit +between containers and between OS X processes -- no transmission across +the hypervisor is supported, yet. Character and block device files are +not supported. + +### Extended attributes + +Extended attributes are not yet supported. + +### Technology + +`osxfs` does not use OSXFUSE. `osxfs` does not run under, inside, or +between OS X userspace processes and the OS X kernel. + +
                +
                  +
                  + +
                  diff --git a/docs/docker-for-mac/release-notes.md b/docs/docker-for-mac/release-notes.md new file mode 100644 index 0000000000..770049de4d --- /dev/null +++ b/docs/docker-for-mac/release-notes.md @@ -0,0 +1,1159 @@ + + +# Docker for Mac Release Notes + +Here are the main improvements and issues per release, starting with the current release. The documentation is always updated for each release. + +For system requirements, please see the Getting Started topic on [What to know before you install](index.md#what-to-know-before-you-install). + +Release notes for _stable_ and _beta_ releases are listed below. You can learn about both kinds of releases, and download stable and beta product installers at [Download Docker for Mac](index.md#download-docker-for-mac). + +* [Stable Release Notes](#stable-release-notes) +* [Beta Release Notes](#beta-release-notes) + +## Stable Release Notes + +### Docker for Mac 1.12.1, 2016-09-16 (stable) + +**New** + +* Support for OSX 10.12 Sierra + +**Upgrades** + +* Docker 1.12.1 +* Docker machine 0.8.1 +* Linux kernel 4.4.20 +* aufs 20160905 + +**Bug fixes and minor changes** + +**General** + + * Fixed communications glitch when UI talks to com.docker.vmnetd + Fixes https://github.com/docker/for-mac/issues/90 + + * `docker-diagnose`: display and record the time the diagnosis was captured + + * Don't compute the container folder in `com.docker.vmnetd` + Fixes https://github.com/docker/for-mac/issues/47 + + * Warn the user if BlueStacks is installed (potential kernel panic) + + * Automatic update interval changed from 1 hour to 24 hours + + * Include Zsh completions + + * UI Fixes + +**Networking** + +* VPNKit supports search domains + +* slirp: support up to 8 external DNS servers + +* slirp: reduce the number of sockets used by UDP NAT, reduce the probability that NAT rules will time out earlier than expected + +* Entries from `/etc/hosts` should now resolve from within containers + +* Allow ports to be bound on host addresses other than `0.0.0.0` and `127.0.0.1` + Fixes issue reported in https://github.com/docker/for-mac/issues/68 + +* Use Mac System Configuration database to detect DNS + +**Filesharing (OSXFS)** + +* Fixed thread leak + +* Fixed a malfunction of new directories that have the same name as an old directory that is still open + +* Rename events now trigger DELETE and/or MODIFY `inotify` events (saving with TextEdit works now) + +* Fixed an issue that caused `inotify` failure and crashes + +* Fixed a directory file descriptor leak + +* Fixed socket `chowns` + +**Moby** + +* Use default `sysfs` settings, transparent huge pages disabled + +* `cgroup` mount to support `systemd` in containers + +* Increase Moby `fs.file-max` to 524288 + +* Fixed Moby Diagnostics and Update Kernel + +**HyperKit** + +* HyperKit updated with `dtrace` support and lock fixes + +### Docker for Mac 2016-08-11 1.12.0-a (stable) + +This bug fix release contains osxfs improvements. The fixed issues may have +been seen as failures with apt-get and npm in containers, missed inotify +events or unexpected unmounts. + +* Bug fixes + - osxfs: fixed an issue causing access to children of renamed + directories to fail (symptoms: npm failures, apt-get failures) + - osxfs: fixed an issue causing some ATTRIB and CREATE inotify + events to fail delivery and other inotify events to stop + - osxfs: fixed an issue causing all inotify events to stop when an + ancestor directory of a mounted directory was mounted + - osxfs: fixed an issue causing volumes mounted under other mounts + to spontaneously unmount + +### Docker for Mac 1.12.0-a, 2016-08-03 (stable) + +This bug fix release contains osxfs improvements. The fixed issues may have +been seen as failures with apt-get and npm in containers, missed `inotify` +events or unexpected unmounts. + +**Hotfixes** + +* osxfs: fixed an issue causing access to children of renamed directories to fail (symptoms: npm failures, apt-get failures) (docker/for-mac) + +* osxfs: fixed an issue causing some ATTRIB and CREATE `inotify` events to fail delivery and other `inotify` events to stop + +* osxfs: fixed an issue causing all `inotify` events to stop when an ancestor directory of a mounted directory was mounted + +* osxfs: fixed an issue causing volumes mounted under other mounts to spontaneously unmount + +### Docker for Mac 1.12.0, 2016-07-28 (stable) + +* First stable release + +**Components** + +* Docker 1.12.0 +* Docker Machine 0.8.0 +* Docker Compose 1.8.0 + +## Beta Release Notes + +### Beta 27 Release Notes (2016-09-28 1.12.2-rc1-beta27) + +**Upgrades** + +* Docker 1.12.2-rc1 +* Docker Machine 0.8.2 +* Docker compose 1.8.1 +* Kernel vsock driver v7 +* Kernel 4.4.21 +* aufs 20160912 + +**Bug fixes and minor changes** + +* Fix an issue where some windows did not claim focus correctly +* Add UI when switching channel to prevent user losing containers and settings +* Check disk capacity before Toolbox import +* Import certificates in `etc/ssl/certs/ca-certificates.crt` +* DNS: reduce the number of UDP sockets consumed on the host +* VPNkit: improve the connection-limiting code to avoid running out of sockets on the host +* UDP: handle diagrams bigger than 2035, up to the configured macOS kernel limit +* UDP: make the forwarding more robust; drop packets and continue rather than stopping +* disk: make the "flush" behaviour configurable for database-like workloads. This works around a performance regression in `v1.12.1`. + +### Beta 26 Release Notes (2016-09-14 1.12.1-beta26) + +**New** + +* Improved support for macOS 10.12 Sierra + +**Upgrades** + +* Linux kernel 4.4.20 +* aufs 20160905 + +**Bug fixes and minor changes** + +* Fixed communications glitch when UI talks to `com.docker.vmnetd`. Fixes https://github.com/docker/for-mac/issues/90 + +* UI fix for macOs 10.12 + +* Windows open on top of full screen app are available in all spaces + +* Reporting a bug, while not previously logged into GitHub now works + +* When a diagnostic upload fails, the error is properly reported + +* `docker-diagnose` displays and records the time the diagnosis was captured + +* Ports are allowed to bind to host addresses other than `0.0.0.0` and `127.0.0.1`. Fixes issue reported in https://github.com/docker/for-mac/issues/68. + +* We no longer compute the container folder in `com.docker.vmnetd`. Fixes https://github.com/docker/for-mac/issues/47. + +**Known Issues** + +* `Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode. The +issue is being investigated. The workaround is to restart Docker.app. + +* There are a number of issues with the performance of directories bind-mounted with `osxfs`. In particular, writes of small blocks and +traversals of large directories are currently slow. Additionally, containers +that perform large numbers of directory operations, such as repeated scans of +large directory trees, may suffer from poor performance. More information is +available in [Known Issues](troubleshoot.md#known-issues) in Troubleshooting. + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart `Docker.app`. + +### Beta 25 Release Notes (2016-09-07 1.12.1-beta25) + +**Upgrades** + +* Experimental support for OSX 10.12 Sierra (beta) + +**Bug fixes and minor changes** + +* VPNKit supports search domains +* Entries from `/etc/hosts` should now resolve from within containers +* osxfs: fix thread leak + +**Known issues** + +* Several problems have been reported on macOS 10.12 Sierra and are being +investigated. This includes failure to launch the app and being unable to +upgrade to a new version. + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The +issue is being investigated. The workaround is to restart Docker.app + +* There are a number of issues with the performance of directories bind-mounted +with `osxfs`. In particular, writes of small blocks and traversals of large +directories are currently slow. Additionally, containers that perform large +numbers of directory operations, such as repeated scans of large directory +trees, may suffer from poor performance. More information is available in [Known +Issues](troubleshoot.md#known-issues) in Troubleshooting. + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart Docker.app. + +### Beta 24 Release Notes (2016-08-23 1.12.1-beta24) + +**Upgrades** + +* Docker 1.12.1 +* Docker Machine 0.8.1 +* Linux kernel 4.4.19 +* aufs 20160822 + +**Bug fixes and minor changes** + +* osxfs: fixed a malfunction of new directories that have the same name as an old directory that is still open + +* osxfs: rename events now trigger DELETE and/or MODIFY `inotify` events (saving with TextEdit works now) + +* slirp: support up to 8 external DNS servers + +* slirp: reduce the number of sockets used by UDP NAT, reduce the probability that NAT rules will time out earlier than expected + +* The app warns user if BlueStacks is installed (potential kernel panic) + +**Known issues** + +* Several problems have been reported on macOS 10.12 Sierra and are being investigated. This includes failure to launch the app and being unable to +upgrade to a new version. + +* `Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart `Docker.app`. + +* There are a number of issues with the performance of directories bind-mounted with `osxfs`. In particular, writes of small blocks and traversals of large +directories are currently slow. Additionally, containers that perform large +numbers of directory operations, such as repeated scans of large directory +trees, may suffer from poor performance. For more information and workarounds, see the bullet on [performance of bind-mounted directories](troubleshoot.md#bind-mounted-dirs) in [Known Issues](troubleshoot.md#known-issues) in Troubleshooting. + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart `Docker.app`. + +### Beta 23 Release Notes (2016-08-16 1.12.1-rc1-beta23) + +**Upgrades** + +* Docker 1.12.1-rc1 +* Linux kernel 4.4.17 +* aufs 20160808 + +**Bug fixes and minor changes** + +* Moby: use default sysfs settings, transparent huge pages disabled +* Moby: cgroup mount to support systemd in containers +* osxfs: fixed an issue that caused `inotify` failure and crashes +* osxfs: fixed a directory fd leak +* Zsh completions + +**Known issues** + +* Docker for Mac is not supported on OSX 10.12 Sierra + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app + +* There are a number of issues with the performance of directories bind-mounted with `osxfs`. In particular, writes of small blocks and traversals of large directories are currently slow. Additionally, containers that perform large numbers of directory operations, such as repeated scans of large directory trees, may suffer from poor performance. For more information and workarounds, see the bullet on [performance of bind-mounted directories](troubleshoot.md#bind-mounted-dirs) in [Known Issues](troubleshoot.md#known-issues) in Troubleshooting. + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart Docker.app + +### Beta 22 Release Notes (2016-08-11 1.12.0-beta22) + +**Upgrades** + +* Linux kernel to 4.4.16 + +**Bug fixes and minor changes** + +* Increase Moby fs.file-max to 524288 +* Use Mac System Configuration database to detect DNS +* HyperKit updated with dtrace support and lock fixes +* Fix Moby Diagnostics and Update Kernel +* UI Fixes +* osxfs: fix socket chowns + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app + +* There are a number of issues with the performance of directories bind-mounted with `osxfs`. In particular, writes of small blocks and traversals of large directories are currently slow. Additionally, containers that perform large numbers of directory operations, such as repeated scans of large directory trees, may suffer from poor performance. More information is available in [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart Docker.app + +### Beta 21.1 Release Notes (2016-08-03 1.12.0-beta21.1) + +This bug fix release contains osxfs improvements. The fixed issues may have +been seen as failures with apt-get and npm in containers, missed `inotify` +events or unexpected unmounts. + +**Hotfixes** + +* osxfs: fixed an issue causing access to children of renamed directories to fail (symptoms: npm failures, apt-get failures) (docker/for-mac) + +* osxfs: fixed an issue causing some ATTRIB and CREATE `inotify` events to fail delivery and other `inotify` events to stop + +* osxfs: fixed an issue causing all `inotify` events to stop when an ancestor directory of a mounted directory was mounted + +* osxfs: fixed an issue causing volumes mounted under other mounts to spontaneously unmount (docker/docker#24503) + +#### Docker for Mac 1.12.0 (2016-07-28 1.12.0-beta21) + +**New** + +* Docker for Mac is now available from 2 channels: **stable** and **beta**. New features and bug fixes will go out first in auto-updates to users in the beta channel. Updates to the stable channel are much less frequent and happen in sync with major and minor releases of the Docker engine. Only features that are well-tested and ready for production are added to the stable channel releases. For downloads of both and more information, see the [Getting Started](index.md#download-docker-for-mac). + +**Upgrades** + +* Docker 1.12.0 with experimental features +* Docker Machine 0.8.0 +* Docker Compose 1.8.0 + +**Bug fixes and minor changes** + +* Check for updates, auto-update and diagnose can be run by non-admin users +* osxfs: fixed an issue causing occasional incorrect short reads +* osxfs: fixed an issue causing occasional EIO errors +* osxfs: fixed an issue causing `inotify` creation events to fail +* osxfs: increased the `fs.inotify.max_user_watches` limit in Moby to 524288 +* The UI shows documentation link for sharing volumes +* Clearer error message when running with outdated Virtualbox version +* Added link to sources for qemu-img + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app + +* There are a number of issues with the performance of directories bind-mounted with `osxfs`. In particular, writes of small blocks, and traversals of large directories are currently slow. Additionally, containers that perform large numbers of directory operations, such as repeated scans of large directory trees, may suffer from poor performance. For more information and workarounds, see [Known Issues](troubleshoot.md#known-issues) in [Logs and Troubleshooting](troubleshoot.md). + +* Under some unhandled error conditions, `inotify` event delivery can fail and become permanently disabled. The workaround is to restart Docker.app + +### Beta 20 Release Notes (2016-07-19 1.12.0-rc4-beta20) + +**Bug fixes and minor changes** + +* Fixed `docker.sock` permission issues +* Don't check for update when the settings panel opens +* Removed obsolete DNS workaround +* Use the secondary DNS server in more circumstances +* Limit the number of concurrent port forwards to avoid running out of resources +* Store the database as a "bare" git repo to avoid corruption problems + +**Known issues** + +* `Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker for Mac (`Docker.app`). + +### Beta 19 Release Notes (2016-07-14 1.12.0-rc4-beta19) + +**New** + +* Added privacy tab in settings +* Allow the definition of HTTP proxy overrides in the UI + +**Upgrades** + +* Docker 1.12.0 RC4 +* Docker Compose 1.8.0 RC2 +* Docker Machine 0.8.0 RC2 +* Linux kernel 4.4.15 + +**Bug fixes and minor changes** + +* Filesystem sharing permissions can only be configured in the UI (no more `/Mac` in moby) +* `com.docker.osx.xhyve.hyperkit`: increased max number of fds to 10240 +* Improved Moby syslog facilities +* Improved file-sharing tab +* `com.docker.slirp`: included the DNS TCP fallback fix, required when UDP responses are truncated +* `docker build/events/logs/stats... ` won't leak when iterrupted with Ctrl-C + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 18.1 Release Notes (2016-07-07 1.12.0-rc3-beta18.1) + +>**Note**: Docker 1.12.0 RC3 release introduces a backward incompatible change from RC2. You can fix this by [recreating or updating your containers](troubleshoot.md#recreate-or-update-your-containers-after-beta-18-upgrade) as described in Troubleshooting. + +**Hotfix** + +* Fixed issue resulting in error "Hijack is incompatible with use of CloseNotifier", reverts previous fix for `Ctrl-C` during build. + +**New** + +* New host/container file sharing UI +* `/Mac` bind mount prefix is deprecated and will be removed soon + +**Upgrades** + +* Docker 1.12.0 RC3 + +**Bug fixes and minor changes** + +* VPNKit: Improved scalability as number of network connections increases +* The docker API proxy was failing to deal with some 1.12 features (e.g. health check) + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 18 Release Notes (2016-07-06 1.12.0-rc3-beta18) + +**New** + +* New host/container file sharing UI +* `/Mac` bind mount prefix is deprecated and will be removed soon + +**Upgrades** + +* Docker 1.12.0 RC3 + +**Bug fixes and minor changes** + +* VPNKit: Improved scalability as number of network connections increases +* Interrupting a `docker build` with Ctrl-C will actually stop the build +* The docker API proxy was failing to deal with some 1.12 features (e.g. health check) + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 17 Release Notes (2016-06-29 1.12.0-rc2-beta17) + +**Upgrades** + +* Linux kernel 4.4.14, aufs 20160627 + +**Bug fixes and minor changes** + +* Documentation moved to https://docs.docker.com/docker-for-mac/ +* Allow non-admin users to launch the app for the first time (using admin creds) +* Prompt non-admin users for admin password when needed in Preferences +* Fixed download links, documentation links +* Fixed "failure: No error" message in diagnostic panel +* Improved diagnostics for networking and logs for the service port openers + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 16 Release Notes (2016-06-17 1.12.0-rc2-beta16) + +**Upgrades** + +* Docker 1.12.0 RC2 +* docker-compose 1.8.0 RC1 +* docker-machine 0.8.0 RC1 +* notary 0.3 +* Alpine 3.4 + +**Bug fixes and minor changes** + +* VPNKit: Fixed a regressed error message when a port is in use +* Fixed UI crashing with `NSInternalInconsistencyException` / fixed leak +* HyperKit API: Improved error reporting +* osxfs: fix sporadic EBADF due to fd access/release races (#3683) + + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 15 Release Notes (2016-06-10 1.11.2-beta15) + +**New** + +* Registry mirror and insecure registries can now be configured from Preferences +* VM can now be restarted from Preferences +* `sysctl.conf` can be edited from Preferences + +**Upgrades** + +* Docker 1.11.2 +* Linux 4.4.12, `aufs` 20160530 + +**Bug fixes and minor changes** + +* Timekeeping in Moby VM improved +* Number of concurrent TCP/UDP connections increased in VPNKit +* Hyperkit: `vsock` stability improvements +* Fixed crash when user is admin + +**Known issues** + +* See [Known Issues](troubleshoot.md#known-issues) in [Troubleshooting](troubleshoot.md) + +### Beta 14 Release Notes (2016-06-02 1.11.1-beta14) + +**New** + +* New settings menu item, **Diagnose & Feedback**, is available to run diagnostics and upload logs to Docker. + +**Known issues** + +* `Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode with OSX 10.10. The issue is being investigated. The workaround is to restart `Docker.app`. + +**Bug fixes and minor changes** + +* `osxfs`: now support `statfs` +* **Preferences**: updated toolbar icons +* Fall back to secondary DNS server if primary fails. +* Added a link to the documentation from menu. + +### Beta 13.1 Release Notes (2016-05-28 1.11.1-beta13.1) + +**Hotfixes** + +* `osxfs`: + - Fixed sporadic EBADF errors and End_of_file crashes due to a race corrupting node table invariants + - Fixed a crash after accessing a sibling of a file moved to another directory caused by a node table invariant violation +* Fixed issue where Proxy settings were applied on network change, causing docker daemon to restart too often +* Fixed issue where log file sizes doubled on docker daemon restart + +### Beta 13 Release Notes (2016-05-25 1.11.1-beta13) + +**New** + +* `osxfs`: Enabled 10ms dcache for 3x speedup on a `go list ./...` test against docker/machine. Workloads heavy in file system path resolution (common among dynamic languages and build systems) will have those resolutions performed in amortized constant time rather than time linear in the depth of the path so speedups of 2-10x will be common. + +* Support multiple users on the same machine, non-admin users can use the app as long as `vmnetd` has been installed. Currently, only one user can be logged in at the same time. + +* Basic support for using system HTTP/HTTPS proxy in docker daemon + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app. + +**Bug fixes and minor changes** + +* `osxfs`: + - setting `atime` and `mtime` of nodes is now supported + - Fixed major regression in Beta 12 with ENOENT, ENOTEMPY, and other spurious errors after a directory rename. This manifested as `npm install` failure and other directory traversal issues. + - Fixed temporary file ENOENT errors + - Fixed in-place editing file truncation error (e.g. `perl -i`)w +* improved time synchronisation after sleep + +### Beta 12 Release (2016-05-17 1.11.1-beta12) + +**Upgrades** + +* FUSE 7.23 for [osxfs](osxfs.md) + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app. + +**Bug fixes and minor changes** + +* UI improvements +* Fixed a problem in [osxfs](osxfs.md) where`mkdir` returned EBUSY but directory was created. + +### Beta 11 Release (2016-05-10 1.11.1-beta11) + +**New** + +The `osxfs` file system now persists ownership changes in an extended attribute. (See the topic on [ownership](osxfs.md#ownership) in [Sharing the OS X file system with Docker containers](osxfs.md).) + +**Upgrades** + +* docker-compose 1.7.1 (see changelog) +* Linux kernel 4.4.9 + +**Bug fixes and minor changes** + +* Desktop notifications after successful update +* No "update available" popup during install process +* Fixed repeated bind of privileged ports +* `osxfs`: Fixed the block count reported by stat +* Moby (Backend) fixes: + - Fixed `vsock` half closed issue + - Added NFS support + - Hostname is now Moby, not Docker + - Fixes to disk formatting scripts + - Linux kernel upgrade to 4.4.9 + +## Beta 10 Release (2016-05-03 1.11.0-beta10) + +**New** + +* Token validation is now done over an actual SSL tunnel (HTTPS). (This should fix issues with antivirus applictions.) + +**Upgrades** + +* Docker 1.11.1 + +**Bug fixes and minor changes** + +* UCP now starts again +* Include debugging symbols in HyperKit +* vsock stability improvements +* Addressed glitches in **Preferences** panel +* Fixed issues impacting the “whale menu” +* Fixed uninstall process +* HyperKit vcpu state machine improvements, may improve suspend/resume + + +### Beta 9 Release (2016-04-26 1.11.0-beta9) + +**New** + +* New Preferences window - memory and vCPUs now adjustable +* `localhost` is now used for port forwarding by default.`docker.local` will no longer work as of Beta 9. + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app. + +**Bug fixes and minor changes** + +* Fix loopback device naming +* Improved docker socket download and osxfs sequential write by 20% +* `com.docker.osxfs` + - improved sequential read throughput by up to 20% + - improved `readdir` performance by up to 6x + - log all fatal exceptions +* More reliable DNS forwarding over UDP and TCP +* UDP ports can be proxied over vsock +* Fixed EADDRINUSE (manifesting as errno 526) when ports are re-used +* Send ICMP when asked to not fragment and we can’t guarantee it +* Fixed parsing of UDP datagrams with IP socket options +* Drop abnormally large ethernet frames +* Improved HyperKit logging +* Record VM start and stop events + +### Beta 8 Release (2016-04-20 1.11.0-beta8) + +**New** + +* Networking mode switched to VPN compatible by default, and as part of this change the overall experience has been improved: + - `docker.local` now works in VPN compatibility mode + - exposing ports on the Mac is available in both networking modes + - port forwarding of privileged ports now works in both networking modes + - traffic to external DNS servers is no longer dropped in VPN mode + + +* `osxfs` now uses `AF_VSOCK` for transport giving ~1.8x speedup for large sequential read/write workloads but increasing latency by ~1.3x. `osxfs` performance engineering work continues. + + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart `Docker.app` + +**Bug fixes and minor changes** + +* Apple System Log now used for most logs instead of direct filesystem logging +* `docker_proxy` fixes +* Merged HyperKit upstream patches +* Improved error reporting in `nat` network mode +* `osxfs` `transfused` client now logs over `AF_VSOCK` +* Fixed a `com.docker.osx.HyperKit.linux` supervisor deadlock if processes exit during a controlled shutdown +* Fixed VPN mode malformed DNS query bug preventing some resolutions + + +### Beta 7 Release (2016-04-12 1.11.0-beta7) + +**New** + +* Docs are updated per the Beta 7 release +* Use AF_VSOCK for docker socket transport + +**Upgrades** + +* docker 1.11.0-rc5 +* docker-machine 0.7.0-rc3 +* docker-compose 1.7.0rc2 + + +**Known issues** + +* Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app + +* If VPN mode is enabled and then disabled and then re-enabled again, `docker ps` will block for 90s + +**Bug fixes and minor changes** + +* Logging improvements +* Improve process management + +## Beta 6 Release (2016-04-05 1.11.0-beta6) + +**New** + +* Docs are updated per the Beta 6 release +* Added uninstall option in user interface + +**Upgrades** + +* docker 1.11.0-rc5 +* docker-machine 0.7.0-rc3 +* docker-compose 1.7.0rc2 + +**Known issues** + +* `Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode. +The issue is being investigated. The workaround is to restart +`Docker.app`. + +* If VPN mode is enabled, then disabled and re-enabled again, +`docker ps` will block for 90 seconds. + +**Bug fixes and minor changes** + +* Fixed osxfs multiple same directory bind mounts stopping inotify +* Fixed osxfs `setattr` on mode 0 files (`sed` failures) +* Fixed osxfs blocking all operations during `readdir` +* Fixed osxfs mishandled errors which crashed the file system and VM +* Removed outdated `lofs`/`9p` support +* Added more debugging info to logs uploaded by `pinata diagnose` +* Improved diagnostics from within the virtual machine +* VirtualBox version check now also works without VBoxManage in path +* VPN mode now uses same IP range as NAT mode +* Tokens are now verified on port 443 +* Removed outdated uninstall scripts +* Increased default ulimits +* Port forwarding with `-p` and `-P` should work in VPN mode +* Fixed a memory leak in `com.docker.db` +* Fixed a race condition on startup between Docker and networking which can +lead to `Docker.app` not starting on reboot + +### Beta 5 Release (2016-03-29 1.10.3-beta5) + +**New** + +- Docs are updated per the Beta 5 release! + +**Known issues** + +- There is a race on startup between docker and networking which can lead to Docker.app not starting on reboot. The workaround is to restart the application manually. + +- Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app. + +- In VPN mode, the `-p` option needs to be explicitly of the form `-p :`. `-p ` and `-P` will not work yet. + +**Bug fixes and minor changes** + +- Updated DMG background image +- Show correct VM memory in Preferences +- Feedback opens forum, not email +- Fixed RAM amount error message +- Fixed wording of CPU error dialog +- Removed status from Preferences +- Check for incompatible versions of Virtualbox + +### Beta 4 Release (2016-03-22 1.10.3-beta4) + +**New Features and Upgrades** + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  File System/SharingSupport `inotify` events so that file system events on the + Mac will trigger file system activations inside Linux containers
                  Docker MachineInstall Docker Machine as a part of Docker for Mac install in `/usr/local`
                  Getting Started and About- Added animated popover window to help first-time users get started
                  - Added a Beta icon to About box
                  + +**Known issues** + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  Starting DockerThere is a race on startup between Docker and networking that can lead to `Docker.app` not starting on reboot.

                  The workaround is to restart the application manually. +
                  OS X version support`Docker.app` sometimes uses 200% CPU after OS X wakes up from sleep mode. + The issue is being investigated.

                  The workaround is to restart + `Docker.app`.
                  +
                  VPN/HostnetIn VPN mode, the `-p` option needs to be explicitly of the form + `-p :`. `-p ` and `-P` will not + work yet. +
                  + +**Bug fixes and minor changes** + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  Hostnet/VPN modeFixed Moby DNS resolver failures by proxying the "Recursion Available" flag.
                  +
                  IP addresses`docker ps` shows IP address rather than `docker.local`. +
                  OS X version support- Re-enabled support for OS X Yosemite version 10.10
                  - Ensured binaries are built for 10.10 rather than 10.11. +
                  Application startup- Fixed "Notification Center"-related crash on startup
                  - Fixed watchdog crash on startup
                  + +### Beta 3 Release (2016-03-15 1.10.3-beta3) + +**New Features and Upgrades** + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  File SystemImproved file sharing write speed in OSXFS
                  User space networkingRenamed `bridged` mode to `nat` mode
                  DebuggingDocker runs in debug mode by default for new installs +
                  Docker EngineUpgraded to 1.10.3
                  + +**Bug fixes and minor changes** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  GUIAuto update automatically checks for new versions again
                  +
                  File System- Fixed OSXFS chmod on sockets
                  + - FixED OSXFS EINVAL from `open` using O_NOFOLLOW
                  HypervisorHypervisor stability fixes, resynced with upstream repository
                  Hostnet/VPN mode- Fixed get/set VPN mode in Preferences (GUI)
                  + - Added more verbose logging on errors in `nat` mode
                  + - Show correct forwarding details in `docker ps/inspect/port` in `nat` mode
                  +
                  TokensNew lines ignored in token entry field
                  FeedbackFeedback mail has app version in subject field
                  LicensingClarified open source licenses
                  Crash reporting and error handling- Fixed HockeyApp crash reporting
                  + - Fatal GUI errors now correctly terminate the app again
                  + - Fix proxy panics on EOF when decoding JSON
                  + - Fix long delay/crash when switching from `hostnet` to `nat` mode +
                  Logging- Moby logs included in diagnose upload
                  + - App version included in logs on startup +
                  + + +### Beta 2 Release (2016-03-08 1.10.2-beta2) + +**New Features and Upgrades** + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  GUIAdd VPN mode/`hostnet` to Preferences
                  Add disable Time Machine backups of VM disk image to Preferences
                  CLIAdded `pinata` configuration tool for experimental Preferences
                  File SystemAdd guest-to-guest FIFO and socket file support
                  NotaryUpgraded to version 0.2
                  + +**Bug fixes** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  File SystemFixed data corruption bug during cp (use of sendfile/splice)
                  GUIFixed About box to contain correct version string
                  Hostnet/VPN mode- Stability fixes and tests
                  - Fixed DNS issues when changing networks
                  MobyCleaned up Docker startup code
                  Linking and dependenciesFixed various problems
                  LoggingVarious improvements
                  + +

                   

                  + +### Beta 1 Release (2016-03-01 1.10.2-b1) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                  ComponentDescription
                  GUIAdded dialog to explain why we need admin rights
                  Removed shutdown/quit window
                  Improved machine migration
                  Added "Help" option in menu to open documentation web pages
                  Added license agreement
                  Added MixPanel support
                  Crash ReportsAdd HockeyApp crash reporting
                  Task ManagerImprove signal handling
                  LoggingUse ISO timestamps with microsecond precision
                  Clean up logging format
                  PackagingCreate /usr/local if it doesn't exist
                  docker-uninstall improvements
                  Remove docker-select as it's no longer used
                  HypervisorAdd PID file
                  Networking reliability improvements
                  HostnetFixed port forwarding issue
                  Stability fixes
                  Fixed setting hostname
                  SymlinksFixed permissions on `usr/local` symbolic links
                  + +
                  +
                    +
                    + +
                    diff --git a/docs/docker-for-mac/troubleshoot.md b/docs/docker-for-mac/troubleshoot.md new file mode 100644 index 0000000000..874478122c --- /dev/null +++ b/docs/docker-for-mac/troubleshoot.md @@ -0,0 +1,332 @@ + + +# Logs and Troubleshooting + +Here is information about how to diagnose and troubleshoot problems, send logs and communicate with the Docker for Mac team, use our forums and Knowledge Hub, browse and log issues on GitHub, and find workarounds for known problems. + +## Docker Knowledge Hub + +**Looking for help with Docker for Mac?** Check out the [Docker Knowledge Hub](http://success.docker.com/) for knowledge base articles, FAQs, and technical support for various subscription levels. + +## Diagnose problems, send feedback, and create GitHub issues + +If you encounter problems for which you do not find solutions in this documentation, [Docker for Mac issues on GitHub](https://github.com/docker/for-mac/issues) already filed by other users, or on the [Docker for Mac forum](https://forums.docker.com/c/docker-for-mac), we can help you troubleshoot the log data. + +Choose --> **Diagnose & Feedback** from the menu bar. + +![Diagnose problems](images/settings-diagnose.png) + +You can choose to run diagnostics only, or diagnose and send the results to the Docker Team: + +* **Diagnose Only** - Runs diagnostics, and shows results locally. (Results are not sent to Docker, and no ID is generated.) + +![Diagnostic results only](images/settings-diagnostic-results-only.png) + +* **Diagnose & Upload** - Runs diagnostics, shows results, and auto-uploads the diagnostic results to Docker. A diagnostic ID is auto-generated. You can refer to this ID when communicating with the Docker Team. Optionally, you can open an issue on GitHub using the uploaded results and ID as a basis. + +![Diagnostics & Feedback](images/settings-diagnose-id.png) + +If you click **Open Issues**, this opens [Docker for Mac issues on GitHub](https://github.com/docker/for-mac/issues/) in your web browser in a “create new issue” template prepopulated with the following: + +* ID and summary of the diagnostic you just ran + +* System and version details + +* Sections where you can fill in a description of expected and actual behavior, and steps to reproduce the issue + +![Create issue on GitHub](images/diagnose-issue.png) + +You can also create a new issue directly on GitHub at https://github.com/docker/for-mac/issues. (The README for the repository is [here](https://github.com/docker/for-mac).) + +Click [New Issue](https://github.com/docker/for-mac/issues/new) on that page (or right here ☺) to get a "create new issue" template prepopulated with sections for the ID and summary of your diagnostics, system and version details, description of expected and actual behavior, and steps to reproduce the issue. + +![issue template](images/diagnose-d4mac-issues-template.png) + + +## Checking the logs + +In addition to using the diagnose and feedback option to submit logs, you can browse the logs yourself. + +#### Use the command line to view logs + +To view Docker for Mac logs at the command line, type this command in a terminal window or your favorite shell. + + $ syslog -k Sender Docker + +Alternatively, you can send the output of this command to a file. The following command redirects the log output to a file called `my_docker_logs.txt`. + + $ syslog -k Sender Docker > ~/Desktop/my_docker_logs.txt + +#### Use the Mac Console for log queries + +Macs provide a built-in log viewer. You can use the Mac Console System Log Query to check Docker app logs. + +The Console lives on your Mac hard drive in `Applications` > `Utilities`. You can bring it up quickly by just searching for it with Spotlight Search. + +To find all Docker app log messages, do the following. + +1. From the Console menu, choose **File** > **New System Log Query...** + + ![Mac Console search for Docker app](images/console_logs_search.png) + + * Name your search (for example `Docker`) + * Set the **Sender** to **Docker** + +2. Click **OK** to run the log query. + + ![Mac Console display of Docker app search results](images/console_logs.png) + +You can use the Console Log Query to search logs, filter the results in various ways, and create reports. + +For example, you could construct a search for log messages sent by Docker that contain the word `hypervisor` then filter the results by time (earlier, later, now). + +The diagnostics and usage information to the left of the results provide auto-generated reports on packages. + + +## Troubleshooting + +#### Recreate or update your containers after Beta 18 upgrade + +Docker 1.12.0 RC3 release introduces a backward incompatible change from RC2 to RC3. (For more information, see https://github.com/docker/docker/issues/24343#issuecomment-230623542.) + +You may get the following error when you try to start a container created with pre-Beta 18 Docker for Mac applications. + + Error response from daemon: Unknown runtime specified default + +You can fix this by either [recreating](#recreate-your-containers) or [updating](#update-your-containers) your containers. + +If you get the error message shown above, we recommend recreating them. + +##### Recreate your containers + +To recreate your containers, use Docker Compose. + + docker-compose down && docker-compose up + +##### Update your containers + +To fix existing containers, follow these steps. + +1. Run this command. + + $ docker run --rm -v /var/lib/docker:/docker cpuguy83/docker112rc3-runtimefix:rc3 + + Unable to find image 'cpuguy83/docker112rc3-runtimefix:rc3' locally + rc3: Pulling from cpuguy83/docker112rc3-runtimefix + 91e7f9981d55: Pull complete + Digest: sha256:96abed3f7a7a574774400ff20c6808aac37d37d787d1164d332675392675005c + Status: Downloaded newer image for cpuguy83/docker112rc3-runtimefix:rc3 + proccessed 1648f773f92e8a4aad508a45088ca9137c3103457b48be1afb3fd8b4369e5140 + skipping container '433ba7ead89ba645efe9b5fff578e674aabba95d6dcb3910c9ad7f1a5c6b4538': already fixed + proccessed 43df7f2ac8fc912046dfc48cf5d599018af8f60fee50eb7b09c1e10147758f06 + proccessed 65204cfa00b1b6679536c6ac72cdde1dbb43049af208973030b6d91356166958 + proccessed 66a72622e306450fd07f2b3a833355379884b7a6165b7527c10390c36536d82d + proccessed 9d196e78390eeb44d3b354d24e25225d045f33f1666243466b3ed42fe670245c + proccessed b9a0ecfe2ed9d561463251aa90fd1442299bcd9ea191a17055b01c6a00533b05 + proccessed c129a775c3fa3b6337e13b50aea84e4977c1774994be1f50ff13cbe60de9ac76 + proccessed dea73dc21126434f14c58b83140bf6470aa67e622daa85603a13bc48af7f8b04 + proccessed dfa8f9278642ab0f3e82ee8e4ad029587aafef9571ff50190e83757c03b4216c + proccessed ee5bf706b6600a46e5d26327b13c3c1c5f7b261313438d47318702ff6ed8b30b + +2. Quit Docker. + +3. Start Docker. + + > **Note:** Be sure to quit and then restart Docker for Mac before attempting to start containers. + +4. Try to start the container again: + + $ docker start old-container + old-container + +#### Incompatible CPU detected + +Docker for Mac requires a processor (CPU) that supports virtualization and, more specifically, the [Apple Hypervisor framework](https://developer.apple.com/library/mac/documentation/DriversKernelHardware/Reference/Hypervisor/). Docker for Mac is only compatible with Macs that have a CPU that supports the Hypervisor framework. Most Macs built in 2010 and later support it, as described in the Apple Hypervisor Framework documentation about supported hardware: + +*Generally, machines with an Intel VT-x feature set that includes Extended Page Tables (EPT) and Unrestricted Mode are supported.* + +To check if your Mac supports the Hypervisor framework, run this command in a terminal window. + +``` +sysctl kern.hv_support +``` +If your Mac supports the Hypervisor Framework, the command will print `kern.hv_support: 1`. + +If not, the command will print `kern.hv_support: 0`. + +See also, [Hypervisor Framework Reference](https://developer.apple.com/library/mac/documentation/DriversKernelHardware/Reference/Hypervisor/) in the Apple documentation, and Docker for Mac system requirements in [What to know before you install](index.md#what-to-know-before-you-install). + + +#### Workarounds for common problems + +* IPv6 workaround to auto-filter DNS addresses - IPv6 is not yet supported on Docker for Mac, which typically manifests as a network timeout when running `docker` commands that need access to external network servers (e.g., `docker pull busybox`). + + ``` + $ docker pull busybox + Using default tag: latest + Pulling repository docker.io/library/busybox + Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/busybox/images. You may want to check your internet connection or if you are behind a proxy. + ``` + + Starting with v1.12.1, 2016-09016 on the stable channel, and Beta 24 on the beta channel, a workaround is provided that auto-filters out the IPv6 addresses in DNS server lists and enables successful network accesss. For example, `2001:4860:4860::8888` would become `8.8.8.8`. So, the only workaround action needed for users is to [upgrade to Docker for Mac stable v1.12.1 or newer, or Beta 24 or newer](index.md#download-docker-for-mac). + + On releases with the workaround included to filter out / truncate IPv6 addresses from the DNS list, the above command should run properly: + + ``` + $ docker pull busybox + Using default tag: latest + latest: Pulling from library/busybox + Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6 + Status: Image is up to date for busy box:latest + ``` + + To learn more, see these issues on GitHub and Docker for Mac forums: + + * [Network timeout when top two DNS servers in /etc/resolv.conf are IPv6 addresses](https://github.com/docker/for-mac/issues/9) + + * [ERROR: Network timed out while trying to connect to index.docker.io](https://forums.docker.com/t/error-network-timed-out-while-trying-to-connect-to-index-docker-io/17206) + +* If Docker for Mac fails to install or start properly: + * Make sure you quit Docker for Mac before installing a new version of the application ( --> **Quit Docker**). Otherwise, you will get an "application in use" error when you try to copy the new app from the `.dmg` to `/Applications`. + + * Restart your Mac to stop / discard any vestige of the daemon running from the previously installed version. + + * Run the the uninstall commands from the menu. + +

                    + +* If `docker` commands aren't working properly or as expected: + + Make sure you are not using the legacy Docker Machine environment in your shell +or command window. You do not need `DOCKER_HOST` set, so unset it as it may be +pointing at another Docker (e.g. VirtualBox). If you use bash, `unset +${!DOCKER_*}` will unset existing `DOCKER` environment variables you have set. +For other shells, unset each environment variable individually as described in +[Setting up to run Docker for +Mac](docker-toolbox.md#setting-up-to-run-docker-for-mac) in [Docker for Mac vs. +Docker Toolbox](docker-toolbox.md). + +

                    + +* Note that network connections will fail if the OS X Firewall is set to +"Block all incoming connections". You can enable the firewall, but `bootpd` must be allowed incoming connections so that the VM can get an IP address. + +

                    + +* For the `hello-world-nginx` example, Docker for Mac must be running in order to get to the webserver on `http://localhost/`. Make sure that the Docker whale +is showing in the menu bar, and that you run the Docker commands in a shell that +is connected to the Docker for Mac Engine (not Engine from Toolbox). Otherwise, +you might start the webserver container but get a "web page not available" error +when you go to `localhost`. For more on distinguishing between the two +environments, see [Docker for Mac vs. Docker Toolbox](docker-toolbox.md). + +

                    + +* If you see errors like `Bind for 0.0.0.0:8080 failed: port is already allocated` or + `listen tcp:0.0.0.0:8080: bind: address is already in use`: + + These errors are often caused by some other software on the Mac using those ports. + Run `lsof -i tcp:8080` to discover the name and pid of the other process and + decide whether to shut the other process down, or to use a different port in + your docker app. + +See also [Known Issues](#known-issues) on this page, and the [FAQs](faqs.md) topic. + + +## Known issues + +* IPv6 is not yet supported on Docker for Mac. If you are using IPv6, and haven't upgraded to Beta 24 or v1.12.1 stable or newer, you will see a network +timeout when you run `docker` commands that need access to external network +servers. The aforementioned releases include a workaround for this because +Docker for Mac does not yet support IPv6. See "IPv6 workaround to auto-filter DNS addresses" in +[Workarounds for common problems](#workarounds-for-common-problems). + +* You might encounter errors when using `docker-compose up` with Docker for Mac (`ValueError: Extra Data`). We've identified this is likely related to data and/or events being passed all at once rather than one by one, so sometimes the data comes back as 2+ objects concatenated and causes an error. + +

                    + +* Force-ejecting the `.dmg` after running `Docker.app` from it results in an unresponsive whale in the menu bar, Docker tasks "not responding" in activity monitor, helper processes running, and supporting technologies consuming large percentages of CPU. Please reboot, and then re-start Docker for Mac. If needed,`force quit` any Docker related applications as part of the reboot. + +

                    + +* Docker does not auto-start on login even when it is enabled in --> **Preferences**. This is related to a set of issues with Docker helper, registration, and versioning. + +

                    + +* Docker for Mac uses the `HyperKit` hypervisor (https://github.com/docker/hyperkit) in Mac OS X 10.10 Yosemite and higher. If you are developing with tools that have conflicts with `HyperKit`, such as [Intel Hardware Accelerated Execution Manager (HAXM)](https://software.intel.com/en-us/android/articles/intel-hardware-accelerated-execution-manager/), the current workaround is not to run them at the same time. You can pause `HyperKit` by quitting Docker for Mac temporarily while you work with HAXM. This will allow you to continue work with the other tools and prevent `HyperKit` from interfering. + +

                    + +* If you are working with applications like [Apache Maven](https://maven.apache.org/) that expect settings for `DOCKER_HOST` and `DOCKER_CERT_PATH` environment variables, specify these to connect to Docker instances through Unix sockets. For example: + + export DOCKER_HOST=unix:///var/run/docker.sock + +* `docker-compose` 1.7.1 performs DNS unnecessary lookups for `localunixsocket.local` which can take 5s to timeout on some networks. If `docker-compose` commands seem very slow but seem to speed up when the network is disabled (e.g. when disconnected from wifi), try appending `127.0.0.1 localunixsocket.local` to the file `/etc/hosts`. +Alternatively you could create a plain-text TCP proxy on localhost:1234 using: + + docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:1234:1234 bobrik/socat TCP-LISTEN:1234,fork UNIX-CONNECT:/var/run/docker.sock + + and then `export DOCKER_HOST=tcp://localhost:1234`. + +

                    + + + +* There are a number of issues with the performance of directories + bind-mounted with `osxfs`. In particular, writes of small blocks, and + traversals of large directories are currently slow. Additionally, + containers that perform large numbers of directory operations, such as + repeated scans of large directory trees, may suffer from poor + performance. Applications that behave in this way include: + + - `rake` + - `ember build` + - Symfony + - Magento + + As a work-around for this behavior, you can put vendor or third-party library directories in Docker volumes, perform temporary file system + operations outside of `osxfs` mounts, and use third-party tools like + Unison or `rsync` to synchronize between container directories and + bind-mounted directories. We are actively working on `osxfs` + performance using a number of different techniques and we look forward + to sharing improvements with you soon. + +

                    + +* If your system does not have access to an NTP server, then after a hibernate the time seen by Docker for Mac may be considerably out of sync with the host. Furthermore, the time may slowly drift out of sync during use. To manually reset the time after hibernation, run: + + docker run --rm --privileged alpine hwclock -s + + Or, to resolve both issues, you can add the local clock as a low-priority (high stratum) fallback NTP time source for the host. To do this, edit the host's `/etc/ntp-restrict.conf` to add: + + server 127.127.1.1 # LCL, local clock + fudge 127.127.1.1 stratum 12 # increase stratum + + Then restart the NTP service with: + + sudo launchctl unload /System/Library/LaunchDaemons/org.ntp.ntpd.plist + sudo launchctl load /System/Library/LaunchDaemons/org.ntp.ntpd.plist + +
                    +
                      +
                      + +
                      diff --git a/docs/docker-for-windows/examples.md b/docs/docker-for-windows/examples.md new file mode 100644 index 0000000000..f3c41c7ee8 --- /dev/null +++ b/docs/docker-for-windows/examples.md @@ -0,0 +1,38 @@ + + +# Example Applications + +Upcoming releases will include example applications especially tailored for Docker for Mac and Docker for Windows. + +Examples will highlight develop, build, and run workfows in several languages, including Node.js, Python, Ruby, and Java. + +For now, if you want get started experimenting with the Beta apps and Docker Compose (which is installed automatically with Docker Desktop Editions), have a look at these example applications in the Compose documentation. You should be able to run these with Docker for Mac and Docker for Windows. + +Quickstart: Compose and Django + +Quickstart: Compose and Rails + +Quickstart: Compose and WordPress + +See also [learn by example](/engine/tutorials/index.md) tutorials on building images, runnning containers, networking, managing data, and storing images on Docker Hub. + +
                      +
                        +
                        + +
                        diff --git a/docs/docker-for-windows/faqs.md b/docs/docker-for-windows/faqs.md new file mode 100644 index 0000000000..3ecfc94172 --- /dev/null +++ b/docs/docker-for-windows/faqs.md @@ -0,0 +1,153 @@ + + +# Frequently Asked Questions (FAQs) + + +>**Looking for popular FAQs on Docker for Windows?** Check out the [Docker +Knowledge Hub](http://success.docker.com/) for knowledge base articles, FAQs, +technical support for various subscription levels, and more. + +### Questions about stable and beta channels + +**Q: How do I get the stable or beta version of Docker for Windows?** + +A: Use the download links for the channels given in the topic [Download Docker +for Windows](index.md#download-docker-for-windows). + +This topic also has more information about the two channels. + +**Q: What is the difference between the stable and beta versions of Docker for Windows?** + +A: Two different download channels are available for Docker for Windows: + +* The stable channel provides a general availability release-ready installer for a fully baked and tested, more reliable app. The stable version of Docker for Windows comes with the latest released version of Docker Engine. The release schedule is synched with Docker Engine releases and hotfixes. + +* The beta channel provides an installer with new features we are working on, but is not necessarily fully tested. It comes with the experimental version of Docker Engine. Bugs, crashes and issues are more likely to occur with the beta app, but you get a chance to preview new functionality, experiment, and provide feedback as the apps evolve. Releases are typically more frequent than for stable, often one or more per month. + +**Q: Can I switch back and forth between stable and beta versions of Docker for Windows?** + +A: Yes, you can switch between versions to try out the betas to see what's new, +then go back to stable for other work. However, **you can have only one app +installed at a time**. Switching back and forth between stable and beta apps can +de-stabilize your development environment, particularly in cases where you +switch from a newer (beta) channel to older (stable). + +For example, containers created with a newer beta version of Docker for Windows +may not work after you switch back to stable because they may have been created +leveraging beta features that aren't in stable yet. Just keep this in mind as +you create and work with beta containers, perhaps in the spirit of a playground +space where you are prepared to troubleshoot or start over. + +To safely switch between beta and stable versions be sure +to save images and export the containers you need, then uninstall the current +version before installing another. The workflow is described in more detail +below.
                        + +Do the following each time: + +1. Use `docker save` to save any images you want to keep. (See +[save](/engine/reference/commandline/save.md) in the Docker Engine command line +reference.) + +2. Use `docker export` to export containers you want to keep. (See +[export](/engine/reference/commandline/export.md) in the Docker Engine command +line reference.) + +3. Uninstall the current app (whether stable or beta). + +4. Install a different version of the app (stable or beta). + +### What kind of feedback are we looking for? + +Everything is fair game. We'd like your impressions on the download-install +process, startup, functionality available, the GUI, usefulness of the app, +command line integration, and so on. Tell us about problems, what you like, or +functionality you'd like to see added. + +We are especially interested in getting feedback on the new swarm mode described +in [Docker Swarm](/engine/swarm/index.md). A good place to start is the +[tutorial](/engine/swarm/swarm-tutorial/index.md). + +### What if I have problems or questions? + +You can find the list of frequent issues in +[Logs and Troubleshooting](troubleshoot.md). + +If you do not find a solution in Troubleshooting, browse issues on [Docker for Windows issues on GitHub](https://github.com/docker/for-win/issues) or create a new one. You can also create new issues based on diagnostics. To learn more about running diagnostics and about Docker for Windows GitHub issues, see [Diagnose and Feedback](index.md#diagnose-and-feedback). + +[Docker for Windows forum](https://forums.docker.com/c/docker-for-windows) provides discussion threads as well, and you can create discussion topics there, but we recommend using the GitHub issues over the forums for better tracking and response. + +### Can I use Docker for Windows with new swarm mode? + +Yes! You can use Docker for Windows to test single-node features of [swarm mode](/engine/swarm/index.md) introduced with Docker Engine 1.12, including initializing a swarm with a single node, creating services, and scaling services. Docker “Moby” on Hyper-V will serve as the single swarm node. You can also use Docker Machine, which comes with Docker for Windows, to create and experiment with a multi-node swarm. Check out the tutorial at [Get started with swarm mode](/engine/swarm/swarm-tutorial/index.md). + +### How do I connect to the remote Docker Engine API? + +You might need to provide the location of the remote API for Docker clients and development tools. + +On Docker for Windows, clients can connect to the Docker Engine through a **named pipe**: `npipe:////./pipe/docker_engine`, or **TCP socket** at this URL: `http://localhost:2375`. + +This sets `DOCKER_HOST` and `DOCKER_CERT_PATH` environment variables to the given values (for the named pipe or TCP socket, whichever you use). + +See also [Docker Remote API](/engine/reference/api/docker_remote_api.md) and the Docker for Windows forums topic [How to find the remote API](https://forums.docker.com/t/how-to-find-the-remote-api/20988). + +### Why doesn't `nodemon` pick up file changes in a container mounted on a shared drive? + +Currently, `inotify` does not work on Docker for Windows. This is a known issue. +For more information and a temporary workaround, see [inotify on shared drives +does not work](troubleshoot.md#inotify-on-shared-drives-does-not-work) in +[Troubleshooting](troubleshoot.md). + +### Why does Docker for Windows sometimes lose network connectivity (e.g., `push`/`pull` doesn't work)? + +Networking is not yet fully stable across network changes and system sleep +cycles. Exit and start Docker to restore connectivity. + +### Can I use VirtualBox alongside Docker 4 Windows? + +Unfortunately, VirtualBox (and other hypervisors like VMWare) cannot run when +Hyper-V is enabled on Windows. + +### Why is Windows 10 Home not supported? + +Docker for Windows requires the Hyper-V Windows feature which is not +available on Home-edition. + +### Why is Windows 10 required? + +Docker for Windows uses Windows Hyper-V. While older Windows versions have +Hyper-V, their Hyper-V implementations lack features critical for Docker for +Windows to work. + +### Why does Docker for Windows fail to start when firewalls or anti-virus software is installed? + +Comodo Firewall currently is incompatible with Hyper-V and some Windows 10 builds (possibly, the Anniversary Update), which impacts Docker for Windows. Other firewalls and anti-virus software might also be incompatible with these Microsoft Windows 10 buids. See details and workarounds in [Docker fails to start when Comodo Firewall is installed](troubleshoot.md#docker-fails-to-start-when-comodo-firewall-is-installed) in [Troubleshooting](troubleshoot.md). + + +### How do I uninstall Docker Toolbox? + +You might decide that you do not need Toolbox now that you have Docker for Windows, and want to uninstall it. For +details on how to perform a clean uninstall of Toolbox on Windows, see [How to +uninstall Toolbox](/toolbox/toolbox_install_windows.md#how-to-uninstall-toolbox) +in the Toolbox Windows topics. + +
                        +
                          +
                          + +
                          diff --git a/docs/docker-for-windows/images/Config-popup.png b/docs/docker-for-windows/images/Config-popup.png new file mode 100644 index 0000000000..0ad3186996 Binary files /dev/null and b/docs/docker-for-windows/images/Config-popup.png differ diff --git a/docs/docker-for-windows/images/Docker-win-settings.png b/docs/docker-for-windows/images/Docker-win-settings.png new file mode 100755 index 0000000000..626956187f Binary files /dev/null and b/docs/docker-for-windows/images/Docker-win-settings.png differ diff --git a/docs/docker-for-windows/images/Start-Authorize.png b/docs/docker-for-windows/images/Start-Authorize.png new file mode 100644 index 0000000000..292c3cc8f3 Binary files /dev/null and b/docs/docker-for-windows/images/Start-Authorize.png differ diff --git a/docs/docker-for-windows/images/Start-init.png b/docs/docker-for-windows/images/Start-init.png new file mode 100644 index 0000000000..fb2e93dc62 Binary files /dev/null and b/docs/docker-for-windows/images/Start-init.png differ diff --git a/docs/docker-for-windows/images/about-docker-win.png b/docs/docker-for-windows/images/about-docker-win.png new file mode 100644 index 0000000000..a7495e933c Binary files /dev/null and b/docs/docker-for-windows/images/about-docker-win.png differ diff --git a/docs/docker-for-windows/images/chat.png b/docs/docker-for-windows/images/chat.png new file mode 100644 index 0000000000..597db5aae9 Binary files /dev/null and b/docs/docker-for-windows/images/chat.png differ diff --git a/docs/docker-for-windows/images/config-popup-menu-win-switch-containers.png b/docs/docker-for-windows/images/config-popup-menu-win-switch-containers.png new file mode 100644 index 0000000000..8a7a0ed12f Binary files /dev/null and b/docs/docker-for-windows/images/config-popup-menu-win-switch-containers.png differ diff --git a/docs/docker-for-windows/images/config-popup-menu-win.png b/docs/docker-for-windows/images/config-popup-menu-win.png new file mode 100644 index 0000000000..99a3461cd9 Binary files /dev/null and b/docs/docker-for-windows/images/config-popup-menu-win.png differ diff --git a/docs/docker-for-windows/images/desktop-whale-icon.png b/docs/docker-for-windows/images/desktop-whale-icon.png new file mode 100644 index 0000000000..7781fe1770 Binary files /dev/null and b/docs/docker-for-windows/images/desktop-whale-icon.png differ diff --git a/docs/docker-for-windows/images/diagnose-d4win-issues-template.png b/docs/docker-for-windows/images/diagnose-d4win-issues-template.png new file mode 100644 index 0000000000..c0715a37c5 Binary files /dev/null and b/docs/docker-for-windows/images/diagnose-d4win-issues-template.png differ diff --git a/docs/docker-for-windows/images/diagnose-feedback-id-win.png b/docs/docker-for-windows/images/diagnose-feedback-id-win.png new file mode 100644 index 0000000000..6f76d516f6 Binary files /dev/null and b/docs/docker-for-windows/images/diagnose-feedback-id-win.png differ diff --git a/docs/docker-for-windows/images/diagnose-feedback-win.png b/docs/docker-for-windows/images/diagnose-feedback-win.png new file mode 100644 index 0000000000..d4e68f2dde Binary files /dev/null and b/docs/docker-for-windows/images/diagnose-feedback-win.png differ diff --git a/docs/docker-for-windows/images/docker-daemon.png b/docs/docker-for-windows/images/docker-daemon.png new file mode 100644 index 0000000000..2ca6ab3c62 Binary files /dev/null and b/docs/docker-for-windows/images/docker-daemon.png differ diff --git a/docs/docker-for-windows/images/docker-is-running.png b/docs/docker-for-windows/images/docker-is-running.png new file mode 100644 index 0000000000..06b2557ada Binary files /dev/null and b/docs/docker-for-windows/images/docker-is-running.png differ diff --git a/docs/docker-for-windows/images/download.png b/docs/docker-for-windows/images/download.png new file mode 100644 index 0000000000..bfa896d89e Binary files /dev/null and b/docs/docker-for-windows/images/download.png differ diff --git a/docs/docker-for-windows/images/hyper-v-message.png b/docs/docker-for-windows/images/hyper-v-message.png new file mode 100644 index 0000000000..cd3c557814 Binary files /dev/null and b/docs/docker-for-windows/images/hyper-v-message.png differ diff --git a/docs/docker-for-windows/images/import-docker-content.png b/docs/docker-for-windows/images/import-docker-content.png new file mode 100644 index 0000000000..9b5fce2cbc Binary files /dev/null and b/docs/docker-for-windows/images/import-docker-content.png differ diff --git a/docs/docker-for-windows/images/installer-allow.png b/docs/docker-for-windows/images/installer-allow.png new file mode 100644 index 0000000000..4fd2d4dcc3 Binary files /dev/null and b/docs/docker-for-windows/images/installer-allow.png differ diff --git a/docs/docker-for-windows/images/installer-finishes.png b/docs/docker-for-windows/images/installer-finishes.png new file mode 100644 index 0000000000..daa793b35d Binary files /dev/null and b/docs/docker-for-windows/images/installer-finishes.png differ diff --git a/docs/docker-for-windows/images/installer-in-downloads.png b/docs/docker-for-windows/images/installer-in-downloads.png new file mode 100644 index 0000000000..f246d47911 Binary files /dev/null and b/docs/docker-for-windows/images/installer-in-downloads.png differ diff --git a/docs/docker-for-windows/images/installer-license-ok.png b/docs/docker-for-windows/images/installer-license-ok.png new file mode 100644 index 0000000000..49a3fe9b6f Binary files /dev/null and b/docs/docker-for-windows/images/installer-license-ok.png differ diff --git a/docs/docker-for-windows/images/installer-license-show.png b/docs/docker-for-windows/images/installer-license-show.png new file mode 100644 index 0000000000..11e075a60e Binary files /dev/null and b/docs/docker-for-windows/images/installer-license-show.png differ diff --git a/docs/docker-for-windows/images/installer-progress-bar.png b/docs/docker-for-windows/images/installer-progress-bar.png new file mode 100644 index 0000000000..d56cbff463 Binary files /dev/null and b/docs/docker-for-windows/images/installer-progress-bar.png differ diff --git a/docs/docker-for-windows/images/proxies.png b/docs/docker-for-windows/images/proxies.png new file mode 100644 index 0000000000..1e2c2b125a Binary files /dev/null and b/docs/docker-for-windows/images/proxies.png differ diff --git a/docs/docker-for-windows/images/run-nginx.png b/docs/docker-for-windows/images/run-nginx.png new file mode 100644 index 0000000000..01f5fa66e5 Binary files /dev/null and b/docs/docker-for-windows/images/run-nginx.png differ diff --git a/docs/docker-for-windows/images/settings-cpu-ram.png b/docs/docker-for-windows/images/settings-cpu-ram.png new file mode 100644 index 0000000000..6eb39d3c46 Binary files /dev/null and b/docs/docker-for-windows/images/settings-cpu-ram.png differ diff --git a/docs/docker-for-windows/images/settings-docker-win.png b/docs/docker-for-windows/images/settings-docker-win.png new file mode 100644 index 0000000000..9847392c71 Binary files /dev/null and b/docs/docker-for-windows/images/settings-docker-win.png differ diff --git a/docs/docker-for-windows/images/settings-general.png b/docs/docker-for-windows/images/settings-general.png new file mode 100644 index 0000000000..c92b672e0a Binary files /dev/null and b/docs/docker-for-windows/images/settings-general.png differ diff --git a/docs/docker-for-windows/images/settings-kernel.png b/docs/docker-for-windows/images/settings-kernel.png new file mode 100644 index 0000000000..69e47658d9 Binary files /dev/null and b/docs/docker-for-windows/images/settings-kernel.png differ diff --git a/docs/docker-for-windows/images/settings-network.png b/docs/docker-for-windows/images/settings-network.png new file mode 100644 index 0000000000..52896a8e56 Binary files /dev/null and b/docs/docker-for-windows/images/settings-network.png differ diff --git a/docs/docker-for-windows/images/settings-reset.png b/docs/docker-for-windows/images/settings-reset.png new file mode 100644 index 0000000000..3dd220ec84 Binary files /dev/null and b/docs/docker-for-windows/images/settings-reset.png differ diff --git a/docs/docker-for-windows/images/settings-shared-drives.png b/docs/docker-for-windows/images/settings-shared-drives.png new file mode 100644 index 0000000000..4b81036d9b Binary files /dev/null and b/docs/docker-for-windows/images/settings-shared-drives.png differ diff --git a/docs/docker-for-windows/images/settings-toolbox-import.png b/docs/docker-for-windows/images/settings-toolbox-import.png new file mode 100644 index 0000000000..60fc6398b2 Binary files /dev/null and b/docs/docker-for-windows/images/settings-toolbox-import.png differ diff --git a/docs/docker-for-windows/images/submit-token.png b/docs/docker-for-windows/images/submit-token.png new file mode 100644 index 0000000000..538460e9ce Binary files /dev/null and b/docs/docker-for-windows/images/submit-token.png differ diff --git a/docs/docker-for-windows/images/whale-systray.png b/docs/docker-for-windows/images/whale-systray.png new file mode 100644 index 0000000000..ba6f8f1a8b Binary files /dev/null and b/docs/docker-for-windows/images/whale-systray.png differ diff --git a/docs/docker-for-windows/images/whale-x.png b/docs/docker-for-windows/images/whale-x.png new file mode 100644 index 0000000000..c99e8d5898 Binary files /dev/null and b/docs/docker-for-windows/images/whale-x.png differ diff --git a/docs/docker-for-windows/images/win-file-and-printer-sharing.png b/docs/docker-for-windows/images/win-file-and-printer-sharing.png new file mode 100644 index 0000000000..7f53c8bba9 Binary files /dev/null and b/docs/docker-for-windows/images/win-file-and-printer-sharing.png differ diff --git a/docs/docker-for-windows/images/win-install-success-hello-world.png b/docs/docker-for-windows/images/win-install-success-hello-world.png new file mode 100644 index 0000000000..d269096013 Binary files /dev/null and b/docs/docker-for-windows/images/win-install-success-hello-world.png differ diff --git a/docs/docker-for-windows/images/win-install-success-popup.png b/docs/docker-for-windows/images/win-install-success-popup.png new file mode 100644 index 0000000000..69159c7fdf Binary files /dev/null and b/docs/docker-for-windows/images/win-install-success-popup.png differ diff --git a/docs/docker-for-windows/images/win-install-success.png b/docs/docker-for-windows/images/win-install-success.png new file mode 100644 index 0000000000..a170536a37 Binary files /dev/null and b/docs/docker-for-windows/images/win-install-success.png differ diff --git a/docs/docker-for-windows/index.md b/docs/docker-for-windows/index.md new file mode 100644 index 0000000000..9e64fc4cb9 --- /dev/null +++ b/docs/docker-for-windows/index.md @@ -0,0 +1,429 @@ + + +# Getting Started with Docker for Windows + +Welcome to Docker for Windows! + +Please read through these topics on how to get started. To **give us your feedback** on your experience with the app and report bugs or problems, log in to our [Docker for Windows forum](https://forums.docker.com/c/docker-for-windows). + +>**Already have Docker for Windows?** If you already have Docker for Windows installed, and are ready to get started, skip over to the [Getting Started with Docker](/engine/getstarted/index.md) tutorial. + + +## Download Docker for Windows + +If you have not already done so, please install Docker for Windows. You can download installers from the stable or beta channel. For more about stable and beta channels, see the [FAQs](faqs.md#questions-about-stable-and-beta-channels). + + + + + + + + + + + + + + +
                          Stable channelBeta channel
                          This installer is fully baked and tested, and comes with the latest GA version of Docker Engine.

                          This is the best channel to use if you want a reliable platform to work with.

                          These releases follow a version schedule with a longer lead time than the betas, synched with Docker Engine releases and hotfixes. +                           		This installer offers cutting edge features and comes with the experimental version of Docker Engine, which is described in the Docker Experimental Features README on GitHub.

                          This is the best channel to use if you want to experiment with features we are working on as they become available, and can weather some instability and bugs. This channel is a continuation of the beta program, where you can provide feedback as the apps evolve. Releases are typically more frequent than for stable, often one or more per month.
                          + Get Docker for Windows (stable)

                          + Download checksum: InstallDocker.msi SHA256 +                           		+ Get Docker for Windows (beta)

                          + Download checksum: InstallDocker.msi SHA256 +
                          + +>**Important Notes**: +> +>* Docker for Windows requires 64bit Windows 10 Pro, Enterprise and Education (1511 November update, Build 10586 or later) and Microsoft Hyper-V. Please see [What to know before you install](#what-to-know-before-you-install) for a full list of prerequisites. +> +>* You can switch between beta and stable versions, but _you must have only one app installed at a time_. Also, you will need to save images and export containers you want to keep before uninstalling the current version before installing another. For more about this, see the [FAQs about beta and stable channels](faqs.md#questions-about-stable-and-beta-channels). + +## What to know before you install + +* **README FIRST for Docker Toolbox and Docker Machine users**: Docker for Windows requires Microsoft Hyper-V to run. After Hyper-V is enabled, VirtualBox will no longer work, but any VirtualBox VM images will remain. VirtualBox VMs created with `docker-machine` (including the `default` one typically created during Toolbox install) will no longer start. These VMs cannot be used side-by-side with Docker for Windows. However, you can still use `docker-machine` to manage remote VMs. + +* You can import a `default` VirtualBox VM after installing Docker for Windows by using the **Settings** menu in the System Tray. + +* The current version of Docker for Windows runs on 64bit Windows 10 Pro, Enterprise and Education (1511 November update, Build 10586 or later). In the future we will support more versions of Windows 10. + +* Containers and images created with Docker for Windows are shared between all user accounts on machines where it is installed. This is because all Windows accounts will use the same VM to build and run containers. In the future, Docker for Windows will better isolate user content. + +* The Hyper-V package must be enabled for Docker for Windows to work. The Docker for Windows installer will enable it for you, if needed. (This requires a reboot). + + >**Note**: If your system does not satisfy these requirements, you can install [Docker Toolbox](/toolbox/overview.md), which uses Oracle Virtual Box instead of Hyper-V. + +* **What the install includes**: The installation provides [Docker Engine](https://docs.docker.com/engine/userguide/intro/), Docker CLI client, [Docker Compose](https://docs.docker.com/compose/overview/), and [Docker Machine](https://docs.docker.com/machine/overview/). + +## Step 1. Install Docker for Windows + +1. Double-click `InstallDocker.msi` to run the installer. + + If you haven't already downloaded the installer (`InstallDocker.msi`), you can get it [**here**](https://download.docker.com/win/stable/InstallDocker.msi). It typically downloads to your `Downloads folder`, or you can run it from the recent downloads bar at the bottom of your web browser. + +2. Follow the install wizard to accept the license, authorize the installer, and proceed with the install. + + You will be asked to authorize `Docker.app` with your system password during the install process. Privileged access is needed to install networking components, links to the Docker apps, and manage the Hyper-V VMs. + +3. Click **Finish** on the setup complete dialog to launch Docker. + + ![Install complete>](images/installer-finishes.png) + +## Step 2. Start Docker for Windows + +When the installation finishes, Docker starts automatically. + +The whale in the status bar indicates that Docker is running, and accessible from a terminal. + +If you just installed the app, you also get a popup success message with suggested next steps, and a link to this documentation. + +![Install success](images/win-install-success-popup.png) + +When initialization is complete, select **About Docker** from the notification area icon to verify that you have the latest version. + +Congratulations! You are up and running with Docker for Windows. + +## Step 3. Check versions of Docker Engine, Compose, and Machine + +Start your favorite shell (`cmd.exe`, PowerShell, or other) to check your versions of `docker` and `docker-compose`, and verify the installation. + + PS C:\Users\samstevens> docker --version + Docker version 1.12.0, build 8eab29e, experimental + + PS C:\Users\samstevens> docker-compose --version + docker-compose version 1.8.0, build d988a55 + + PS C:\Users\samstevens> docker-machine --version + docker-machine version 0.8.0, build b85aac1 + +## Step 4. Explore the application and run examples + +The next few steps take you through some examples. These are just suggestions for ways to experiment with Docker on your system, check version information, and make sure `docker` commands are working properly. + +1. Open a shell (`cmd.exe`, PowerShell, or other). + +2. Run some Docker commands, such as `docker ps`, `docker version`, and `docker info`. + + Here is the output of `docker ps` run in a powershell. (In this example, no containers are running yet.) + + PS C:\Users\samstevens> docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + Here is an example of command output for `docker version`. + + PS C:\Users\Vicky> docker version + Client: + Version: 1.12.0 + API version: 1.24 + Go version: go1.6.3 + Git commit: 8eab29e + Built: Thu Jul 28 21:04:48 2016 + OS/Arch: windows/amd64 + Experimental: true + + Server: + Version: 1.12.0 + API version: 1.24 + Go version: go1.6.3 + Git commit: 8eab29e + Built: Thu Jul 28 21:04:48 2016 + OS/Arch: linux/amd64 + Experimental: true + + Here is an example of command output for `docker info`. + + PS C:\Users\Vicky> docker info + Containers: 0 + Running: 0 + Paused: 0 + Stopped: 0 + Images: 0 + Server Version: 1.12.0 + Storage Driver: aufs + Root Dir: /var/lib/docker/aufs + Backing Filesystem: extfs + Dirs: 0 + Dirperm1 Supported: true + Logging Driver: json-file + Cgroup Driver: cgroupfs + Plugins: + Volume: local + Network: host bridge null overlay + Swarm: inactive + Runtimes: runc + Default Runtime: runc + Security Options: seccomp + Kernel Version: 4.4.16-moby + Operating System: Alpine Linux v3.4 + OSType: linux + Architecture: x86_64 + CPUs: 2 + Total Memory: 1.95 GiB + Name: moby + ID: BG6O:2VMH:OLNV:DDLF:SCSV:URRH:BW6M:INBW:OLAC:J7PX:XZVL:ADNB + Docker Root Dir: /var/lib/docker + Debug Mode (client): false + Debug Mode (server): false + Registry: https://index.docker.io/v1/ + Experimental: true + Insecure Registries: + 127.0.0.0/8 + + >**Note:** The outputs above are examples. Your output for commands like `docker version` and `docker info` will vary depending on your product versions (e.g., as you install newer versions). + +3. Run `docker run hello-world` to test pulling an image from Docker Hub and starting a container. + + PS C:\Users\samstevens> docker run hello-world + + Hello from Docker. + This message shows that your installation appears to be working correctly. + + To generate this message, Docker took the following steps: + 1. The Docker client contacted the Docker daemon. + 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. + 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. + 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. + +4. Try something more ambitious, and run an Ubuntu container in a Bash shell. + + $ docker run -it ubuntu bash + + PS C:\Users\samstevens> docker run -it ubuntu bash + Unable to find image 'ubuntu:latest' locally + latest: Pulling from library/ubuntu + 5a132a7e7af1: Pull complete + fd2731e4c50c: Pull complete + 28a2f68d1120: Pull complete + a3ed95caeb02: Pull complete + Digest: sha256:4e85ebe01d056b43955250bbac22bdb8734271122e3c78d21e55ee235fc6802d + Status: Downloaded newer image for ubuntu:latest + + Type `exit` to stop the container and close the Bash shell. + +5. For the pièce de résistance, start a Dockerized webserver with this command: + + docker run -d -p 80:80 --name webserver nginx + + This will download the `nginx` container image and start it. Here is the output of running this command in a powershell. + + PS C:\Users\samstevens> docker run -d -p 80:80 --name webserver nginx + Unable to find image 'nginx:latest' locally + latest: Pulling from library/nginx + + fdd5d7827f33: Pull complete + a3ed95caeb02: Pull complete + 716f7a5f3082: Pull complete + 7b10f03a0309: Pull complete + Digest: sha256:f6a001272d5d324c4c9f3f183e1b69e9e0ff12debeb7a092730d638c33e0de3e + Status: Downloaded newer image for nginx:latest + dfe13c68b3b86f01951af617df02be4897184cbf7a8b4d5caf1c3c5bd3fc267f + +6. Point your web browser at `http://localhost` to display the start page. + + (Since you specified the default HTTP port, it isn't necessary to append `:80` at the end of the URL.) + + ![Run nginx edge>](images/run-nginx.png) + +7. Run `docker ps` while your webserver is running to see details on the container. + + PS C:\Users\samstevens> docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS + NAMES + dfe13c68b3b8 nginx "nginx -g 'daemon off" 3 days ago Up 45 seconds 0.0.0.0:80->80/tcp, 443/tc + p webserver + +8. Stop or remove containers and images. + + The `nginx` webserver will continue to run in the container on that port until you stop and/or remove the container. If you want to stop the webserver, type: `docker stop webserver` and start it again with `docker start webserver`. + + To stop and remove the running container with a single command, type: `docker rm -f webserver`. This will remove the container, but not the `nginx` image. You can list local images with `docker images`. You might want to keep some images around so that you don't have to pull them again from Docker Hub. To remove an image you no longer need, use `docker rmi |`. For example, `docker rmi nginx`. + +**Want more example applications?** - For more example walkthroughs that include setting up services and databases with Docker Compose, see [Example Applications](examples.md). + +## Docker Settings + +When Docker is running, the Docker whale is displayed in the system tray. If it is hidden, click the up arrow in the tray to show it. + +![Showing hidden apps in the system tray](images/whale-systray.png) + +To get a popup menu with application options, right-click the whale: + +![Docker for Windows popup menu](images/config-popup-menu-win.png) + +The **Settings** dialogs provide options to allow Docker auto-start, automatically check for updates, share local drives with Docker containers, enable VPN compatibilty, manage CPUs and memory Docker uses, restart Docker, or perform a factory reset. + +**Beta 26** includes an option to switch between Windows and Linux conatiners. See [Switch between Windows and Linux containers (Beta 26)](#switch-between-windows-and-linux-containers-beta-26). This is not yet available on stable builds. + +![Beta 26 popup with switch for Windows or Linux containers](images/config-popup-menu-win-switch-containers.png) + + +### General + +![Settings](images/settings-general.png) + +* **Start Docker when you log in** - Automatically start the Docker for Windows application upon Windows system login. + +* **Check for updates when the application starts** - Docker for Windows is set to automatically check for updates and notify you when an update is available. If an update is found, click **OK** to accept and install it (or cancel to keep the current version). Uncheck this option if you do not want notifications of version upgrades. You can still find out about updates manually by choosing **Check for Updates** from the menu. + +* **Send usage statistics** - You can set Docker for Windows to auto-send diagnostics, crash reports, and usage data. This information can help Docker improve the application and get more context for troubleshooting problems. + + Uncheck any of the options to opt out and prevent auto-send of data. Docker may prompt for more information in some cases, even with auto-send enabled. Also, you can enable or disable these auto-reporting settings with one click on the information popup when you first start Docker. + + ![Startup information](images/win-install-success-popup.png) + +### Shared Drives + +Share your local drives (volumes) with Docker for Windows, so that they are available to your containers. + +![Shared Drives](images/settings-shared-drives.png) + +You will be asked to provide your Windows system username and password (domain user) to apply shared drives. You can select an option to have Docker store the credentials so that you don't have to re-enter them every time. + +Permissions to access shared drives are tied to the credentials you provide here. If you run `docker` commands and tasks under a different username than the one used here to set up sharing, your containers will not have permissions to access the mounted volumes. + +See also [Verify domain user has permissions for shared drives](troubleshoot.md#verify-domain-user-has-permissions-for-shared-drives-volumes) in Troubleshooting. + +### Advanced + +![CPU and Memory settings](images/settings-cpu-ram.png) + +* **CPUs** - Change the number of processors assigned to the Linux VM. + +* **Memory** - Change the amount of memory the Docker for Windows Linux VM uses. + +Please note, updating these settings requires a reconfiguration and reboot of the Linux VM. This will take a few seconds. + +### Network + +You can configure Docker for Windows networking to work on a virtual private network (VPN). + +* **Internal Virtual Switch** - You can specify a network address translation (NAT) prefix and subnet mask to enable internet connectivity. + +* **DNS Server** - You can configure the DNS server to use dynamic or static IP addressing. + +![Network settings](images/settings-network.png) + +>**Note:** Some users reported problems connecting to Docker Hub on Docker for Windows stable version. This would manifest as an error when trying to run `docker` commands that pull images from Docker Hub that are not alredy downloaded, such as a first time run of `docker run hello-world`. If you encounter this, reset the DNS server to use the Google DNS fixed address: `8.8.8.8`. For more information, see [Networking issues](troubleshoot.md#networking-issues) in Troubleshooting. + +Note that updating these settings requires a reconfiguration and reboot of the Linux VM. + +### Proxies + +Docker for Windows lets you configure HTTP/HTTPS Proxy Settings and automatically propagate these to Docker and to your containers. +For example, if you set your proxy settings to `http://proxy.example.com`, Docker will use this proxy when pulling containers. + +![Proxies](images/proxies.png) + +When you start a container, you will see that your proxy settings propagate into the containers. For example: + +``` +$ docker run -it alpine env +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +HOSTNAME=b7edf988b2b5 +TERM=xterm +HOME=/root +HTTP_PROXY=http://proxy.example.com:3128 +http_proxy=http://proxy.example.com:3128 +no_proxy=*.local, 169.254/16 +``` + +You can see from the above output that the `HTTP_PROXY`, `http_proxy` and `no_proxy` environment variables are set. +When your proxy configuration changes, Docker restarts automatically to pick up the new settings. +If you have containers that you wish to keep running across restarts, you should consider using [restart policies](https://docs.docker.com/engine/reference/run/#restart-policies-restart) + +### Docker daemon +You can configure options on the Docker daemon in the given JSON configuration file, and determine how your containers will run. + +For a full list of options on the Docker daemon, see daemon in the Docker Engine command line reference. + +![Docker Daemon](images/docker-daemon.png) + +Note that updating these settings requires a reconfiguration and reboot of the Linux VM. + +### Switch between Windows and Linux containers (Beta 26) + +Starting with Beta 26, you can select which daemon (Linux or Windows) the Docker CLI talks to. Select **Switch to Windows containers** to toggle to Windows containers. Select **Switch to Linux containers**. + +Microsoft Developer Network has preliminary/draft information on Windows containers [here](https://msdn.microsoft.com/en-us/virtualization/windowscontainers/about/about_overview). + +This feature is not yet available on stable builds. + +### Diagnose and Feedback + +If you encounter problems for which you do not find solutions in this documentation, searching [Docker for Windows issues on GitHub](https://github.com/docker/for-win/issues) already filed by other users, or on the [Docker for Windows forum](https://forums.docker.com/c/docker-for-windows), we can help you troubleshoot the log data. + +Select **Upload a diagnostic**. + +This uploads (sends) the logs to Docker. + +![Diagnose problems and Feedback](images/diagnose-feedback-id-win.png) + +To create a new issue directly on GitHub, open [Docker for Windows issues on GitHub](https://github.com/docker/for-win/issues) in your web browser and follow the instructions in the README. Click [New Issue](https://github.com/docker/for-win/issues/new) on that page (or right here ☺) to get a "create new issue" template prepopulated with sections for the ID and summary of your diagnostics, system and version details, description of expected and actual behavior, and steps to reproduce the issue. + +![issue template](images/diagnose-d4win-issues-template.png) + +### Reset + +![Reset](images/settings-reset.png) + +* **Restart Docker** - Shuts down and restarts the Docker application. + +* **Reset to Toolbox default machine content** - Imports containers and images from the existing Docker Toolbox machine named `default`. (This option is enabled only if you have Toolbox installed.) The VirtualBox VM will not be removed. + +* **Reset to factory defaults** - Resets Docker to factory defaults. This is useful in cases where Docker stops working or becomes unresponsive. + + + +## Where to go next + +* Try out the [Getting Started with Docker](/engine/getstarted/index.md) tutorial. + +* Dig in deeper with [learn by example](/engine/tutorials/index.md) tutorials on building images, runnning containers, networking, managing data, and storing images on Docker Hub. + +* See [Example Applications](examples.md) for example applications that include setting up services and databases in Docker Compose. + +* Interested in trying out the new [swarm mode](/engine/swarm/index.md) on Docker Engine v1.12? + + See [Get started with swarm mode](/engine/swarm/swarm-tutorial/index.md), a tutorial which includes specifics on how to leverage your Docker for Windows installation to run single and multi-node swarms. + + Also, try out the Swarm examples in [docker labs](https://github.com/docker/labs/tree/master/swarm-mode/beginner-tutorial). Run the `bash script` and follow the accompanying [Docker Swarm Tutorial](https://github.com/docker/labs/blob/master/swarm-mode/beginner-tutorial/README.md). The script uses Docker Machine to create a multi-node swarm, then walks you through various Swarm tasks and commands. + +* For a summary of Docker command line interface (CLI) commands, see [Docker CLI Reference Guide](/engine/reference/index.md). + +* Check out the blog posts on Docker for Mac and Docker for Windows public betas, and earlier posts on the intial private beta. + +* Please give feedback on your experience with the app and report bugs and problems by logging into our [Docker for Windows forum](https://forums.docker.com/c/docker-for-windows). + +
                          +
                            +
                            + +
                            diff --git a/docs/docker-for-windows/menu.md b/docs/docker-for-windows/menu.md new file mode 100644 index 0000000000..8274d96f76 --- /dev/null +++ b/docs/docker-for-windows/menu.md @@ -0,0 +1,14 @@ + + +# Docker for Mac and Docker for Windows diff --git a/docs/docker-for-windows/opensource.md b/docs/docker-for-windows/opensource.md new file mode 100644 index 0000000000..3809964332 --- /dev/null +++ b/docs/docker-for-windows/opensource.md @@ -0,0 +1,28 @@ + + +# Open Source Components and Licensing + +Docker Desktop Editions are built using open source software software. For details on the licensing, choose --> **About** from within the application, then click **Acknowlegdements**. + +Docker Desktop Editions distribute some components that are licensed under the GNU General Public License. You can download the source for these components [here](https://download.docker.com/opensource/License.tar.gz). + +
                            +
                              +
                              + +
                              diff --git a/docs/docker-for-windows/release-notes.md b/docs/docker-for-windows/release-notes.md new file mode 100644 index 0000000000..62f846a701 --- /dev/null +++ b/docs/docker-for-windows/release-notes.md @@ -0,0 +1,929 @@ + + +# Docker for Windows Release Notes + +Here are the main improvements and issues per release, starting with the current release. The documentation is always updated for each release. + +For system requirements, please see the Getting Started topic on [What to know before you install](index.md#what-to-know-before-you-install). + +Release notes for _stable_ and _beta_ releases are listed below. You can learn about both kinds of releases, and download stable and beta product installers at [Download Docker for Windows](index.md#download-docker-for-windows). + +* [Stable Release Notes](#stable-release-notes) +* [Beta Release Notes](#beta-release-notes) +* [Alpha Release Notes](#alpha-release-notes) + +## Stable Release Notes + +### Docker for Windows 1.12.1, 2016-09-16 (stable) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**New** + +* To support trusted registry transparently, all trusted CAs (root or intermediate) on the Windows host are automatically copied to Moby + +* `Reset Credentials` will also unshare the shared drives + +* Logs are now rotated every day + +* Support multiple DNS servers + +* Added `mfsymlinks` SMB option to support symlinks on bind mounted folder + +* Added `nobrl` SMB option to support `sqlite` on bind mounted folders + +* Detect outdated versions of Kitematic + +**Upgrades** + +* Docker 1.12.1 +* Docker machine 0.8.1 +* Linux kernel 4.4.20 +* aufs 20160905 + +**Bug fixes and minor changes** + +**General** + +* Uploading a diagnostic now shows a proper status message in the Settings + +* Docker will stop asking to import from Toolbox after an upgrade + +* Docker can now import from Toolbox just after HyperV is activated + +* Added more debug information to the diagnostics + +* Sending anonymous statistics shouldn't hang anymore when Mixpanel is not available + +* Support newlines in release notes + +* Improve error message when Docker daemon is not responding + +* The configuration database is now stored in-memory + +* Preserve the stacktrace of PowerShell errors + +* Display service stacktrace in error windows + +**Networking** + +* Improve name servers discovery +* VpnKit supports search domains +* VpnKit is now compiled with OCaml 4.03 rather than 4.02.3 + +**Filesharing** + +* Set `cifs` version to 3.02 + +* VnpKit: reduce the number of sockets used by UDP NAT, reduce the probability + +* `slirp`: reduce the number of sockets used by UDP NAT, reduce the probability that NAT rules will time out earlier than expected + +* Fixed password handling for host file system sharing + +**Hyper-V** + +* Automatically disable lingering net adapters that prevent Docker from starting or using the network + +* Automatically delete duplicated MobyLinuxVMs on a `reset to factory defaults` + +* Improved the HyperV detection and activation mechanism + +**Moby** + +* Fixed Moby Diagnostics and Update Kernel + +* Use default `sysfs` settings, transparent huge pages disabled + +* `Cgroup` mount to support `systemd` in containers + +**Known issues** + +* Docker will automatically disable lingering net adapters. The only way to remove them is manually using `devmgmt.msc` as documented in [Remove stale network adapters](troubleshoot.md#4-remove-stale-network-adapters) under [Networking issues](troubleshoot.md#networking-issues) in Troubleshooting. + +### Docker for Windows 1.12.0, 2016-07-28 (stable) + +* First stable release + +**Components** + +* Docker 1.12.0 +* Docker Machine 0.8.0 +* Docker Compose 1.8.0 + + +## Beta Release Notes + +### Beta 27 Release Notes (2016-09-28 1.12.2-rc1-beta27) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**New** + +* Reworked the File Sharing dialog and underlying mechanism +* Pre-fill username +* Faster and more reliable feedback when the user/password is not valid +* Better support for domain users +* Error message in Logs when File Sharing failed for other reasons + +**Upgrades** + +* Docker 1.12.2-rc1 +* Docker Machine 0.8.2 +* Docker Compose 1.8.1 +* kernel 4.4.21 +* aufs 20160912 + +**Bug fixes and minor changes** + +* Improve the switching between Linux and Windows containers: better errors, more reliable, deal with more edge cases +* Kill lingering dockerd that users might have still around because they played with Windows Containers before +* Don't recreate the VM if only the DNS server is set +* The uninstaller now kills the service if it failed to stop it properly +* Restart VpnKit and DataKit when the processes die +* VpnKit: impose a connection limit to avoid exhausting file descriptors +* VpnKit: handle UDP datagrams larger than 2035 bytes +* VpnKit: reduce the number of file descriptors consumed by DNS +* Improve debug information + +### Beta 26 Release Notes (2016-09-14 1.12.1-beta26) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**New** + +* Basic support for Windows containers. On Windows 10 build >= 14372, a switch in the `systray` icon will change which daemon (Linux or Windows) the Docker CLI talks to + +* To support trusted registry transparently, all trusted CAs (root or intermediate) on the Windows host are automatically copied to Moby + +* `Reset Credentials` will also unshare the shared drives + +* Logs are now rotated every day + +**Upgrades** + +* Linux kernel 4.4.20 +* aufs 20160905 + +**Bug fixes and minor changes** + +* We no longer send the same DNS settings twice to the daemon + +* Fixed the lingering net adapters removal on Windows 10 Anniversary Update + +* Uploading a diagnostic now shows a proper status message in the Settings + +### Beta 25 Release (2016-09-07 1.12.1-beta25) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**New** + +* Support multiple DNS servers + +**Bug fixes and minor changes** + +* Improved name servers discovery +* VpnKit supports search domains +* Set CIFS (common internet file system) version to 3.02 + +**Known issues** + +* Only UTF-8 passwords are supported for host filesystem sharing + +* Docker will automatically disable lingering net adapters. The only way to remove them is manually using `devmgmt.msc` as documented in [Remove stale network adapters](troubleshoot.md#4-remove-stale-network-adapters) under [Networking issues](troubleshoot.md#networking-issues) in Troubleshooting. + +### Beta 24 Release (2016-08-23 1.12.1-beta24) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**Upgrades** + +* Docker 1.12.1 +* Docker Machine 0.8.1 +* Linux kernel 4.4.19 +* aufs 20160822 + +**Bug fixes and minor changes** + +* `slirp`: reduce the number of sockets used by UDP NAT, reduce the probability that NAT rules will time out earlier than expected + +**Known issues** + +* Only UTF-8 passwords are supported for host filesystem sharing. + +* Docker will automatically disable lingering net adapters. The only way to remove them is manually using `devmgmt.msc` as documented in [Remove stale network adapters](troubleshoot.md#4-remove-stale-network-adapters) under [Networking issues](troubleshoot.md#networking-issues) in Troubleshooting. + +### Beta 23 Release (2016-08-16 1.12.1-rc1-beta23) + +>**Important Note**: +> +> The auto-update function in Beta 21 will not be able to install this update. To install the latest beta manually if you are still on Beta 21, please download the installer here: + +> https://download.docker.com/win/beta/InstallDocker.msi + +> This problem is fixed as of Beta 23 for subsequent auto-updates. + +**New** + +* Added `mfsymlinks` smb option to support symlinks on bind mounted folder +* Added `nobrl` smb option to support sqlite on bind mounted folders +* Detect outdated versions of Kitematic + +**Upgrades** + +* Docker 1.12.1-rc1 +* Linux kernel 4.4.17 +* aufs 20160808 + +**Bug fixes and minor changes** + +* Fixed password handling for host file system sharing +* Automatically disable lingering net adapters that prevent Docker from starting or using the network +* Automatically delete duplicated MobyLinuxVMs on a `reset to factory defaults` +* Docker will stop asking to import from toolbox after an upgrade +* Docker can now import from toolbox just after hyperV is activated +* Fixed Moby Diagnostics and Update Kernel +* Added more debug information to the diagnostics +* Sending anonymous statistics shouldn't hang anymore when Mixpanel is not available +* Improved the HyperV detection and activation mechanism +* VpnKit is now compiled with OCaml 4.03 rather than 4.02.3 +* Support newlines in release notes +* Improved error message when docker daemon is not responding +* The configuration database is now stored in-memory +* Preserve the stacktrace of PowerShell errors +* Display service stacktrace in error windows +* Moby: use default sysfs settings, transparent huge pages disabled +* Moby: cgroup mount to support systemd in containers + +**Known issues** + +* Only UTF-8 passwords are supported for host filesystem sharing +* Docker will automatically disable lingering net adapters. The only way to remove them is manually using `devmgmt.msc` as documented in [Troubleshooting](troubleshoot.md#networking-issues). + +### Beta 22 Release (2016-08-11 1.12.0-beta22) + +Unreleased. See Beta 23 for changes. + +**Known issues** + +* Docker will automatically disable lingering net adapters. The only way to remove them is manually using `devmgmt.msc` as documented in [Troubleshooting](troubleshoot.md#networking-issues). + +### Beta 21 Release (2016-07-28 1.12.0-beta21) + +**New** + +* Docker for Windows is now available from 2 channels: **stable** and **beta**. New features and bug fixes will go out first in auto-updates to users in the beta channel. Updates to the stable channel are much less frequent and happen in sync with major and minor releases of the Docker engine. Only features that are well-tested and ready for production are added to the stable channel releases. For downloads of both and more information, see the [Getting Started](index.md#download-docker-for-windows). + +* Removed the docker host name. Containers with exported ports are reachable via localhost. + +* The UI shows whether the user is on beta or stable channel + +**Upgrades** + +* Docker 1.12.0 with experimental features +* Docker Machine 0.8.0 +* Docker Compose 1.8.0 + +**Bug fixes and minor changes** + +* Fixed networking issue when transmitting data to a container via an exposed port. +* Include the sources for qemu-img +* Fixed the migration from toolbox when the user has a space in its login +* Disable the migration from toolbox when hyperV is not yet activated +* More windows can be closed with ESC +* Added the channel to crash reports +* Fixed a path rewriting bug that happens on Windows insider build 14367 +* Simplified the MobyLinux.ps1 script + +**Known issues** + +* Older Kitematic versions are not compatible with Docker for Windows. You need to manually delete the `C:\Program Files\Docker\Kitematic` folder before you click **Open Kitematic...** systray link. + +### Beta 20 Release (2016-07-19 1.12.0-rc4-beta20) + +**New** + +* The UI option to disable port forwarding to `localhost` has been removed + +**Bug fixes and minor changes** + +* Fixed `docker.sock` permission issues +* Don't check for update when the settings panel opens +* Removed obsolete DNS workaround +* Use the secondary DNS server in more circumstances +* Limit the number of concurrent port forwards to avoid running out of resources +* Store the database as a "bare" git repo to avoid corruption problems + +### Beta 19 Release (2016-07-14 1.12.0-rc4-beta19) + +**New** + +* Added an option to opt-out from sending usage statistics (will be available on the future stable channel) +* New error dialog box to upload crash reports + +**Upgrades** + +* Docker 1.12.0 RC4 +* Docker Compose 1.8.0 RC2 +* Docker Machine 0.8.0 RC2 +* Linux kernel 4.4.15 + +**Bug fixes and minor changes** + +* `com.docker.slirp`: included the DNS TCP fallback fix, required when UDP responses are truncated +* `docker build/events/logs/stats...` won't leak when interrupted with Ctrl-C +* Disable all buttons on Update Window when a version is downloading + +### Beta 18.1 Release (2016-07-07 1.12.0-rc3-beta18.1) + +>**Note**: Docker 1.12.0 RC3 release introduces a backward incompatible change from RC2. You can fix this by [recreating or updating your containers](troubleshoot.md#recreate-or-update-your-containers-after-beta-18-upgrade) as described in Troubleshooting. + +**Hotfix** + +* Fixed issue resulting in error "Hijack is incompatible with use of CloseNotifier", reverts previous fix for `Ctrl-C` during build. + +**New** + +* Forwarding the ports to localhost is now the default +* Added `http`/`https` proxy configuration to the settings +* The toolbox default machine can be imported on first launch +* Added UI when a crash report is collected and uploaded +* The check for update runs every 6 hours + +**Upgrades** + +* Docker 1.12.0 RC3 + +**Bug fixes and minor changes** + +* The docker API proxy was failing to deal with 1.12 features (health check for, for example) +* When killing the VM process, ignore when the process is already stopped +* When stopping the VM, always stop the docker proxy +* Prevent the update windows from downloading the `.msi` into `C:\Program Files\Docker` +* All settings should be disabled when Docker is starting. (This regression was introduced in Beta 17) +* VPNKit: Improved scalability as number of network connections increases +* Improve the connection to the database +* Ignore when the shutdown service is not available + +### Beta 18 Release (2016-07-06 1.12.0-rc3-beta18) + +**New** + +* Forwarding the ports to localhost is now the default +* Added `http`/`https` proxy configuration to the settings +* The toolbox default machine can be imported on first launch +* Added UI when a crash report is collected and uploaded +* The check for update runs every 6 hours + +**Upgrades** + +* Docker 1.12.0 RC3 + +**Bug fixes and minor changes** + +* Interrupting a `docker build` with Ctrl-C will actually stop the build +* The docker API proxy was failing to deal with 1.12 features (health check for, for example) +* When killing the VM process, ignore when the process is already stopped +* When stopping the VM, always stop the docker proxy +* Prevent the update windows from downloading the `.msi` into `C:\Program Files\Docker` +* All settings should be disabled when Docker is starting. (This regression was introduced in Beta 17) +* VPNKit: Improved scalability as number of network connections increases +* Improve the connection to the database +* Ignore when the shutdown service is not available + +### Beta 17 Release (2016-06-29 1.12.0-rc2-beta17) + +**Upgrades** + +* Linux kernel 4.4.14, aufs 20160627 + +**Bug fixes and minor changes** + +* Support users with spaces in their login +* Fix some cases where `dotnet restore` could hang +* Fixed `docker inspect` on an image +* Removed the console from hyper-v manager +* Improved diagnostic for VPN connection and add logs for the service port openers +* Improve Moby's boot sequence to adapt to longer boot time when swarm services are running +* Forcefully turn off a VM that won't shut down +* Clicking on a link from the changelog opens a browser +* Fix links to the documentation +* Fix the url to download Kitematic +* Renewed the signing certificates +* Fixed errors with the firewall and the network switch +* Fixed parsing errors in the Powershell script + +### Beta 16 Release (2016-06-17 1.12.0-rc2-beta16) + +**Upgrades** + +* Docker 1.12.0 RC2 +* docker-compose 1.8.0 RC1 +* docker-machine 0.8.0 RC1 +* Alpine 3.4 + +**Bug fixes and minor changes** + +* Fixes to the VPN mode +* Fixed the localhost port forwarding performance issue +* Auto-detect mounted/unmounted drive in the list of shares + - Changed the name of the application from "DockerforWindows" to "Docker for Windows" + - Avoid multiple update windows being displayed at the same time + +### Beta 15 Release (2016-06-10 1.11.2-beta15) + +**New** + +* New experimental networking mode, exposing container ports on `localhost` +* New Settings menu to configure sysctl.conf +* New Settings menu to configure http proxies +* The VPN mode setting is removed (VPN mode is now the only supported mode) +* The vSwitch NAT configuration has been removed + +**Upgrades** + +* Docker 1.11.2 +* Linux 4.4.12, aufs 20160530 + +**Bug fixes and minor changes** + +* Moved `Import from toolbox` option to the General Settings +* Increased the timeout to write to the configuration database +* Fixed an issue where sending anonynous stats to Mixpanel made the application stop +* Faster boot time +* All named pipes are now prefixed with the word `docker` +* Full version number is now displayed in the update window +* Default daemon config does not have debug enabled anymore +* More responsive Settings Panel, with new whales also :-) +* Improved logs and debug information + +### Beta 14 Release(2016-06-02 1.11.1-beta14) + +**New** + +* Enabled configuration of the docker daemon (edit `config.json`) +* The VPN mode is enabled by default +* Removed DHCP for VM network configuration +* User configurable NAT prefix and DNS server +* New feedback window to upload diagnostics dialog +* New status indicator in **Settings** window +* VM logs are uploaded with a crash report +* Animated welcome whale + +**Bug fixes and minor changes** + +* Support non-ASCII characters in passwords +* Fixed unshare a drive operation +* Fixed deserialized of exceptions sent from the service +* If the backend service is not running, the GUI now starts it +* The app no longer complains if the backend service is not running and the user just wants to shut down. + + +**Known issues** + +* Due to limitation in the Windows NAT implementation, co-existence with other NAT prefixes needs to be carefully managed. See [NAT Configuration](troubleshoot.md#nat-configuration) in [Troubleshooting](troubleshoot.md) for more details. + +### Beta 13 Release (2016-05-25 1.11.1-beta13) + +**New** + +This Beta release includes some significant changes: + +* Docker communication is over Hyper-V sockets instead of the network +* Experimental VPN mode, also known as `vpnkit` +* Initial support for `datakit` for configuration +* Redesigned Settings panel +* Docker can now be restarted + +**Bug fixes and minor changes** + +* Support Net adapters with a different name than "vEthernet (DockerNAT)" +* Sharing now has a better support for domain users +* Fixed Toolbox migration (was broken in Beta12) +* Enabling HyperV (was broken in Beta12) +* Fixed error message when invalid labels are passed to `docker run` +* Mixpanel no longer uses roaming App Data +* UI improvements +* Support was added for VMs with other IP addresses out of the `10.0.75.0/24` range +* Improved FAQ + +**Known issues** + +* Due to limitation in the Windows NAT implementation, co-existence with other NAT prefixes needs to be carefully managed. See [NAT Configuration](troubleshoot.md#nat-configuration) in [Troubleshooting](troubleshoot.md) for more details. + +### Beta 12 Release (2016-17-10 1.11.1-beta12) + +**New** + +* The application is now separated in two parts. A back-end service and a front-end GUI.The front-end GUI no longer asks for elevated access. + +**Bug fixes and minor changes** + +* Excluded the network drives from the shares list +* Removed the notification when closing the application +* Minor GUI improvements + +**Known issues** + +* Due to limitation in the Windows NAT implementation, co-existence with other NAT prefixes needs to be carefully managed. See [NAT Configuration](troubleshoot.md#nat-configuration) in [Troubleshooting](troubleshoot.md) for more details. + + +### Beta 11b Hot Fix Release (2016-05-11 1.11.1-beta11b) + +**Hot fixes** + +* Fixed an issue with named pipe permisions that prevented Docker from starting + +### Beta 11 Release (2016-05-10 1.11.1-beta11) + +**New** + +* The GUI now runs in non-elevated mode and connects to an elevated Windows service +* Allocate VM memory by 256 MB increments, instead of 1 GB +* Show a meaningful error when the user has an empty password +* Improved [Troubleshooting](troubleshoot.md) page + +**Upgrades** + +* docker-compose 1.7.1 (see changelog) +* Kernel 4.4.9 + +**Bug fixes and minor changes** + +* Report the VM's IP in `docker port` +* Handle passwords with spaces +* Show a clear error message when trying to install on Home editions +* Slower whale animation in the System Tray +* Proxy is restarting itself when it crashes +* DHCP process handles exceptions gracefully +* Moby (Backend) fixes: + - Fixed `vsock` half closed issue + - Added NFS support + - Hostname is now Moby, not Docker + - Fixes to disk formatting scripts + - Kernel upgrade to 4.4.9 + +**Known issues** + +* Due to limitation in the Windows NAT implementation, co-existence with other NAT prefixes needs to be carefully managed. See [Troubleshooting](troubleshoot.md) for more details. + +* Logs for the windows service are not aggregated with logs from the GUI. This will be fixed in future versions. + + +## Beta 10 Release (2016-05-03 1.11.0-beta10) + +**New** + +* Improved Settings panel, allow to configure the VM’s memory and CPUs +* Co-exist with multiple internal Hyper-V switches and improved DHCP handling +* Token validation is now done over HTTPS. This should fix issues with some firewalls and antiviros software. + +**Upgrades** + +* Docker 1.11.1 + +**Bug fixes and minor changes** + +* Fixed Desktop shortcut name and updated icons +* Preparation to run the backend as service +* Improved logging and Mixpanel events +* Improved code quality +* Improved the build +* New icons + +**Known issues** + +* Due to limitation in the Windows NAT implementation, co-existence with other NAT prefixes needs to be carefully managed. See [Troubleshooting](troubleshoot.md) for more details. + + +### Beta 9 Release (2016-04-26 1.11.0-beta9) + +**New** + +* Provide one-click dialog to enable Hyper-V +* Report clear underlying Hyper-V errors + +**Bug fixes and minor changes** + +* Better handling of some networking issues +* Fixed help menu and start menu getting started URLs +* Restored “Docker is Initializing” notification on first run +* Better error messages during authentication +* Improved logging on error conditions +* Improved build and tests + +**Known issues** + +* If multiple internal Hyper-V switches exist the Moby VM +may not start correctly. We have identified the issue and +are working on a solution. + +### Beta 8 Release (2016-04-20 1.11.0-beta8) + +**New** + +* Auto-update is installed silently, and relaunches the application when it completes +* Uninstaller can be found in Windows menu +* Kitematic can be downloaded from the Dashboard menu + +**Bug fixes and minor changes** + +* Better UI in the ShareDrive window +* The firewall alert dialog will not come up as often as it was +* Configured MobyLinux VM with a fixed memory of 2GB +* User password is no longer stored on the host-side KVP +* Uninstall shortcut is available in registry + +### Beta 7 Release (2016-04-12 1.11.0-beta7) + +**New** + + - Multiple drives can be shared + - New update window + - Welcome whale + +**Upgrades** + +* docker 1.11.0-rc5 +* docker-machine 0.7.0-rc3 +* docker-compose 1.7.0-rc2 + +**Bug fixes and minor changes** + +* Improved networking configuration and error detection: fixed DHCP renewal and rebind issues +* Allow DNS/DHCP processes to restart on bind error +* Less destructive migration from Docker Toolbox +* Improved documentation +* Better error handling: Moby will restart itself if start takes too long. +* Kill proxy and exit docker before a new version is installed +* The application cannot start twice now +* The proxy will stop automatically when the GUI is not running +* Removed existing proxy firewall rules before starting Moby +* The application now collects more and better information on crashes and other issues +* Improved all dialogs and windows +* Added the version to installer's first screen +* Better reset to defaults +* New regression test framework +* The installation MSI is now timestamped +* The Hyper-V install mentions Docker Toolbox only if it is present +* Improved Bugsnag reports: fixed a dependency bug, and added a unique ID to each new report +* Improved the build +* Improved code quality + +**Known issues** + +- Settings are now serialized in JSON. This install will lose the current settings. + +- Docker needs to open ports on the firewall. Sometimes, the user will see a firewall alert dialog. The user should allow the ports to be opened. + +- The application was upgraded to 64 bits. The installation path changed to `C:\Program Files\Docker\Docker`. Users might have to close any Powershell/Cmd windows that were already open before the update to get the new `PATH`. In some cases, users may need to log off and on again. + +**Bug Fixes** + + - Fixed DHCP renewal and rebind + - Only mention toolbox on Hyper-V install if it's present + - The application does not start twice now + - DNS/DHCP processes are allowed to restart on bind error now + - Removed the window that opens quickly during bugsnag reports + - Fixed OS reported by Bugsnag + - Improved the build + - Improved code quality + +### Beta 6 Release (2016-04-05 1.11.0.1288) + +**Enhancements** + +- Docs are updated for Beta 6! +- Support roaming: DNS queries are forwarded to the host +- Improved startup times by running a DHCP server on the host +- New settings dialog design +- Support windows paths with -v +- Updated docker CLI and deamon to 1.11.0-rc3 +- Updated docker-machine to 0.7.0-rc2 +- Updated docker-compose to 1.7.0-rc1 +- Now install docker-credential-wincred +- Allow non-root users in containers to create files on volume mounts +- Automatically install HyperV +- The application is now 64bits +- Improved wording in all dialog boxes and error messages +- Removed exit confirmation +- Show clickable URL in the Install HyperV message box +- Dashboard link to Kitematic (as on Mac) +- Moby Kernel updated to 4.4.6 +- The registry key was changed to HKLM\SOFTWARE\Docker Inc.\Docker\1.0 + +**Known issues** + +- Migration from Docker Toolbox can fail sometimes. If this happens, the workaround is to restart the application. + +- Docker needs to open ports on the firewall, which can activate a firewall alert dialog. Users should allow the ports to be opened. + +- The application was upgraded to 64 bits. The installation path changed to `C:\Program Files\Docker\Docker`. If users have Powershell/Cmd windows already open before the update, they might have to close them to catch the new PATH. In some cases, users will need to log off and on again. + +**Bug Fixes** + +- Kill VMs that cannot be shutdown properly + +- Improved the diagnostic information sent with bugsnag reports + +- Settings window shows when the drive is shared or not +`C:` drive can be bind mounted with `//c` or `/c`. Used to be `//c/` + +- Don't try to submit empty tokens + +- Fixed the version shown in the About box + +- Fixed a race condition on the logs + +- Fixed a race condition on the settings + +- Fixed broken links in the documentation + +- Replaced `sha1` with actual version in the assemblies + +- Don't start the unused agent process + +### Beta 5 Release (2016-03-29 1.10.6) + +**Enhancements** + +* Remove debug console +* Open browser with hyper-v installation instructions +* Add Cloudfront for downloads from Europe +* Capture qemu logs during toolbox upgrades +* Rename alpha distribution channel to beta + +**Bug Fixes** + +* Fix diagnose section in bugsnag report +* Fix msi version +* Don't truncate Toolbox link + +>**Note**: Docker for Windows skipped from Beta 1 to Beta 5 at this point to synch up the version numbering with Docker for Mac, which went into beta cycles a little earlier. + +### Beta 1 Release (2016-03-24 1.10.6) + +**Enhancements** + +- Display the third party licenses +- Display the license agreement +- The application will refuse to start if Hyper-v is not enabled +- Rename `console` to `debug console` +- Remove `machine` from notification +- Open the feedback forum +- Use same MixPanel project for Windows and OSX +- Align MixPanel events with OSX +- Added a script to diagnose problems +- Submit diagnostic with bugsnag reports +- MixPanel heartbeat every hour + +**Bug Fixes** + +- Accept all versions of Enterprise 10, Pro 10 and Education 10 during installation (Eval, N, ...) +- Fix Linux kernel crashes with certain applications or somesuch +- Fix notifications that are not shown +- Animate the systray whale on reset +- Shorten the enrollment process timeout +- Properly unmount shares when the user un-selects the setting +- Don't install on unsupported builds + +## Alpha Release Notes + +### Alpha 4 Release (A2016-03-10 1.10.4.0) + +- Faster Startup & Shutdown +- Use host DNS parameters +- Enrollment System +- Recreating manually removed vm +- More MixPanel Events +- Various Bug Fixes + +### Alpha 3 Release (2016-03-03 1.10.2.14) + +**File sharing** + + - Create network share automatically + - Improve Credentials management + - Support paths with c and C drive + +**Crashes and Analytics** + + - Report crashes with Bugsnag + - Send analytics through MixPanel + +**GUI** + + - Improve layout of About and Settings dialog + - Improve Updater + - Link to *Help* + - Link to *Send Feeback* + +**General** + + - Bug fixes + +### Alpha 2 Release (2016-02-26 1.10.2.12) + +**Installer** + + - Enhancements + - Auto-update + - License agreement + +**General** + + - Bug fixes + +### Alpha 1 Release (2016-02-22 1.10.1.42-1) + +**Hypervisor** + + - significant performance improvements + +**Security** + + - retrieving Credentials from user + +**Filesystem** + + - hot-mounting host filesystem with credential + +**General** + + - state management + - stability, logging + - bugfixes, eye candies + +### Alpha 0 Release (2016-02-09 1.10.0.0-0) + +**Hypervision** + + - hyper-v backed virtual machines + - boots moby in a few seconds + - installs CLI in `PATH` + - proxies docker commands to moby + +**Filesystem** + + - mounts host filesystem to support `--volume` + - samba client with a hardcoded password + - allows live reload + +**Networking** + + - live debugging Node.js application + +
                              +
                                +
                                + +
                                diff --git a/docs/docker-for-windows/troubleshoot.md b/docs/docker-for-windows/troubleshoot.md new file mode 100644 index 0000000000..0efe9ef327 --- /dev/null +++ b/docs/docker-for-windows/troubleshoot.md @@ -0,0 +1,300 @@ + + +# Logs and Troubleshooting + +Here is information about how to diagnose and troubleshoot problems, send logs and communicate with the Docker for Windows team, use our forums and Knowledge Hub, browse and log issues on GitHub, and find workarounds for known problems. + +## Docker Knowledge Hub + +**Looking for help with Docker for Windows?** Check out the [Docker Knowledge Hub](http://success.docker.com/) for knowledge base articles, FAQs, and technical support for various subscription levels. + +## Submitting diagnostics, feedback, and GitHub issues + +If you encounter problems for which you do not find solutions in this documentation or on the [Docker for Windows forum](https://forums.docker.com/c/docker-for-windows), we can help you troubleshoot the log data. See [Diagnose and Feedback](index.md#diagnose-and-feedback) in the Getting Started topic. + + +## Checking the Logs + +In addition to using the diagnose and feedback option to submit logs, you can browse the logs yourself. + +### Use the systray menu to view logs + +To view Docker for Windows latest log, click on the `Diagnose & Feedback` menu entry in the systray and then on the `Log file` link. You can see the full history of logs in your `AppData\Local` folder. + +### Use the systray menu to report and issue + +If you encounter an issue and the suggested troubleshoot procedures outlined below don't fix it you can generate a diagnostics report. Click on the `Diagnose & Feedback` menu entry in the systray and then on the `Upload diagnostic...` link. This will upload diagnostics to our server and provide you with a unique ID you can use in email or the forum to reference the upload. + + + +## Troubleshooting + +### inotify on shared drives does not work + +Currently, `inotify` does not work on Docker for Windows. This will become evident, for example, when when an application needs to read/write to a container across a mounted drive. This is a known issue that the team is working on. Below is a temporary workaround, and a link to the issue. + +* **Workaround for nodemon and Node.js** - If you are using [nodemon](https://github.com/remy/nodemon) with `Node.js`, try the fallback polling mode described here: [nodemon isn't restarting node applications](https://github.com/remy/nodemon#application-isnt-restarting) + +* **Docker for Windows issue on GitHub** - See the issue [Inotify on shared drives does not work](https://github.com/docker/for-win/issues/56#issuecomment-242135705) + + +### Verify domain user has permissions for shared drives (volumes) + +Permissions to access shared drives are tied to the username and password you use to set up shared drives. (See [Shared Drives](index.md#shared-drives).) If you run `docker` commands and tasks under a different username than the one used to set up shared drives, your containers will not have permissions to access the mounted volumes. The volumes will show as empty. + +The solution to this is to switch to the domain user account and reset credentials on shared drives. + +Here is an example of how to de-bug this problem, given a scenario where you shared the `C` drive as a local user instead of as the domain user. Assume the local user is `samstevens` and the domain user is `merlin`. + +1. Make sure you are logged in as the Windows domain user (for our example, `merlin`). + +2. Run `net share c` to view user permissions for `\, FULL`. + + PS C:\WINDOWS\system32> net share c + Share name C + Path C:\ + Remark + Maximum users No limit + Users SAMSTEVENS + Caching Caching disabled + Permission windowsbox\samstevens, FULL + +3. Run the following command to remove the share. + + net share c /delete + +4. Re-share the drive via the [Shared Drives dialog](index.md#shared-drives), and provide the Windows domain user account credentials. + +5. Re-run `net share c`. + + PS C:\WINDOWS\system32> net share c + Share name C + Path C:\ + Remark + Maximum users No limit + Users MERLIN + Caching Caching disabled + Permission windowsbox\merlin, FULL + +See also, the related issue on GitHub, [Mounted volumes are empty in the container](https://github.com/docker/for-win/issues/25). + +### Avoid unexpected syntax errors, use Unix style line endings for files in containers + +Any file destined to run inside a container must use Unix style `\n` line endings. This includes files referenced at the command line for builds and in RUN commands in Docker files. + +Docker containers and `docker build` run in a Unix environment, so files in containers must use Unix style line endings `\n`, _not_ Windows style: `\r\n`. Keep this in mind when authoring files such as shell scripts using Windows tools, where the default is likely to be Windows style line endings. These commands ultimately get passed to Unix commands inside a Unix based container (for example, a shell script passed to `/bin/sh`). If Windows style line endings are used, `docker run` will fail with syntax errors. + +For an example of this issue and the resolution, see this issue on GitHub: Docker RUN fails to execute shell script (https://github.com/docker/docker/issues/24388). + +### Recreate or update your containers after Beta 18 upgrade + +Docker 1.12.0 RC3 release introduces a backward incompatible change from RC2 to RC3. (For more information, see https://github.com/docker/docker/issues/24343#issuecomment-230623542.) + +You may get the following error when you try to start a container created with pre-Beta 18 Docker for Windows applications. + + Error response from daemon: Unknown runtime specified default + +You can fix this by either [recreating](#recreate-your-containers) or [updating](#update-your-containers) your containers. + +If you get the error message shown above, we recommend recreating them. + +#### Recreate your containers + +To recreate your containers, use Docker Compose. + + docker-compose down && docker-compose up + +#### Update your containers + +To fix existing containers, follow these steps. + +1. Run this command. + + $ docker run --rm -v /var/lib/docker:/docker cpuguy83/docker112rc3-runtimefix:rc3 + + Unable to find image 'cpuguy83/docker112rc3-runtimefix:rc3' locally + rc3: Pulling from cpuguy83/docker112rc3-runtimefix + 91e7f9981d55: Pull complete + Digest: sha256:96abed3f7a7a574774400ff20c6808aac37d37d787d1164d332675392675005c + Status: Downloaded newer image for cpuguy83/docker112rc3-runtimefix:rc3 + proccessed 1648f773f92e8a4aad508a45088ca9137c3103457b48be1afb3fd8b4369e5140 + skipping container '433ba7ead89ba645efe9b5fff578e674aabba95d6dcb3910c9ad7f1a5c6b4538': already fixed + proccessed 43df7f2ac8fc912046dfc48cf5d599018af8f60fee50eb7b09c1e10147758f06 + proccessed 65204cfa00b1b6679536c6ac72cdde1dbb43049af208973030b6d91356166958 + proccessed 66a72622e306450fd07f2b3a833355379884b7a6165b7527c10390c36536d82d + proccessed 9d196e78390eeb44d3b354d24e25225d045f33f1666243466b3ed42fe670245c + proccessed b9a0ecfe2ed9d561463251aa90fd1442299bcd9ea191a17055b01c6a00533b05 + proccessed c129a775c3fa3b6337e13b50aea84e4977c1774994be1f50ff13cbe60de9ac76 + proccessed dea73dc21126434f14c58b83140bf6470aa67e622daa85603a13bc48af7f8b04 + proccessed dfa8f9278642ab0f3e82ee8e4ad029587aafef9571ff50190e83757c03b4216c + proccessed ee5bf706b6600a46e5d26327b13c3c1c5f7b261313438d47318702ff6ed8b30b + +2. Quit Docker. + +3. Start Docker. + + > **Note:** Be sure to quit and then restart Docker for Windows before attempting to start containers. + +4. Try to start the container again: + + $ docker start old-container + old-container + +### Hyper-V +Docker for Windows requires a Hyper-V as well as the Hyper-V Module for Windows Powershell to be installed and enabled. See [these instructions](https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/quick_start/walkthrough_install) to install Hyper-V manually. A reboot is *required*. If you install Hyper-V without the reboot, Docker for Windows will not work correctly. On some systems, Virtualization needs to be enabled in the BIOS. The steps to do so are Vendor specific, but typically the BIOS option is called `Virtualization Technology (VTx)` or similar. + +### Networking issues + +Some users have reported problems connecting to Docker Hub on the Docker for Windows stable version. (See GitHub issue [22567](https://github.com/docker/docker/issues/22567).) + +Here is an example command and error message: + + PS C:\WINDOWS\system32> docker run hello-world + Unable to find image 'hello-world:latest' locally + Pulling repository docker.io/library/hello-world + C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error while pulling image: Get https://index.docker.io/v1/repositories/library/hello-world/images: dial tcp: lookup index.docker.io on 10.0.75.1:53: no such host. + See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'. + +As an immediate workaround to this problem, reset the DNS server to use the Google DNS fixed address: `8.8.8.8`. You can configure this via the **Settings** -> **Network** dialog, as described in the topic [Network](index.md#network). Docker will automatically restart when you apply this setting, which could take some time. + +We are currently investigating this issue. + +#### Networking issues on pre Beta 10 versions +Docker for Windows Beta 10 and later fixed a number of issues around the networking setup. If you still experience networking issue, this may be related to previous Docker for Windows installations. In this case, please quit Docker for Windows and perform the following steps: + +##### 1. Remove multiple `DockerNAT` VMswitches +You might have multiple Internal VMSwitches called `DockerNAT`. You can view all VMSwitches either via the `Hyper-V Manager` sub-menu `Virtual Switch Manager` or from an elevated Powershell (run as Administrator) prompt by typing `Get-VMSwitch`. Simply delete all VMSwitches with `DockerNAT` in the name, either via the `Virtual Switch Manager` or by using `Remove-VMSwitch` powershell cmdlet. + +##### 2. Remove lingering IP addresses + +You might have lingering IP addresses on the system. They are supposed to get removed when you remove the associated VMSwitches, but sometimes this fails. Using `Remove-NetIPAddress 10.0.75.1` in an elevated Powershell prompt should remove them. + +##### 3. Remove stale NAT configurations + +You might have stale NAT configurations on the system. You should remove them with `Remove-NetNat DockerNAT` on an elevated Powershell prompt. + +##### 4. Remove stale network adapters + +You might have stale Network Adapters on the system. You should remove them with the following commands on an elevated Powershell prompt: + + $vmNetAdapter = Get-VMNetworkAdapter -ManagementOS -SwitchName DockerNAT + Get-NetAdapter "vEthernet (DockerNAT)" | ? { $_.DeviceID -ne $vmNetAdapter.DeviceID } | Disable-NetAdapter -Confirm:$False -PassThru | Rename-NetAdapter -NewName "Broken Docker Adapter" + +Then you can remove them manually via the `devmgmt.msc` (aka Device Manager). You should see them as disabled Hyper-V Virtual Ethernet Adapter under the Network Adapter section. A right-click and selecting **uninstall** should remove the adapter. + +### NAT/IP configuration + +By default, Docker for Windows uses an internal network prefix of `10.0.75.0/24`. Should this clash with your normal network setup, you can change the prefix from the **Settings** menu. See the [Network](index.md#network) topic under [Settings](index.md#docker-settings). + +#### NAT/IP configuration issues on pre Beta 15 versions + +As of Beta 15, Docker for Windows is no longer using a switch with a NAT configuration. The notes below are left here only for older Beta versions. + +As of Beta14, networking for Docker for Windows is configurable through the UI. See the [Network](index.md#network) topic under [Settings](index.md#docker-settings). + +By default, Docker for Windows uses an internal Hyper-V switch with a NAT configuration with a `10.0.75.0/24` prefix. You can change the prefix used (as well as the DNS server) via the **Settings** menu as described in the [Network](index.md#network) topic. + +If you have additional Hyper-V VMs and they are attached to their own NAT prefixes, the prefixes need to be managed carefully, due to limitation of the Windows NAT implementation. Specifically, Windows currently only allows a single internal NAT prefix. If you need additional prefixes for your other VMs, you can create a larger NAT prefix. + +To create a larger NAT prefix, do the following. + +1. Stop Docker for Windows and remove all NAT prefixes with `Remove-NetNAT`. + +2. Create a new shorter NAT prefix which covers the Docker for Windows NAT prefix but allows room for additional NAT prefixes. For example: + + New-NetNat -Name DockerNAT -InternalIPInterfaceAddressPrefix 10.0.0.0/16 + + The next time Docker for Windows starts, it will use the new, wider prefix. + +Alternatively, you can use a different NAT name and NAT prefix and adjust the NAT prefix Docker for Windows uses accordingly via the `Settings` panel. + +>**Note**: You also need to adjust your existing VMs to use IP addresses from within the new NAT prefix. + + +### Host filesystem Sharing + +The Linux VM used for Docker for Windows uses SMB/CIFS mounting of the host filesystem. In order to use this feature you must explicitly enable it via the `Settings` menu. You will get prompted for your Username and Password. + +Unfortunately, this setup does not support passwords which contain Unicode characters, so your password must be 8-bit ASCII only. + +The setup also does not support empty password, so you should set a password if you want to use the host filesystem sharing feature. Beta 11 and newer of Docker for Windows will display a warning, but versions earlier will not. + +Note, releases of Docker for Windows prior to Beta 11 also did not support spaces in the password and username, but this has been fixed with Beta 11. + +Please make sure that "File and printer sharing" is enabled in `Control Panel\Network and Internet\Network and Sharing Center\Advanced sharing settings`. + +![Sharing settings](images/win-file-and-printer-sharing.png) + +## Workarounds + +### `inotify` currently does not work on Docker for Windows + +If you are using `Node.js` with `nodemon`, a temporary workaround is to try the fallback polling mode described here: [nodemon isn't restarting node applications](https://github.com/remy/nodemon#application-isnt-restarting). See also this issue on GitHub [Inotify on shared drives does not work](https://github.com/docker/for-win/issues/56#issuecomment-242135705). + +### Reboot + +Restart your PC to stop / discard any vestige of the daemon running from the previously installed version. + +### Unset `DOCKER_HOST` + +You do not need `DOCKER_HOST` set, so unset as it may be pointing at +another Docker (e.g. VirtualBox). If you use bash, `unset ${!DOCKER_*}` +will unset existing `DOCKER` environment variables you have set. For other shells, unset each environment variable individually. + +### Make sure Docker is running for webserver examples + +For the `hello-world-nginx` example and others, Docker for Windows must be running in order to get to the webserver on `http://localhost/`. Make sure that the Docker whale is showing in the menu bar, and that you run the Docker commands in a shell that is connected to the Docker for Windows Engine (not Engine from Toolbox). Otherwise, you might start the webserver container but get a "web page not available" error when you go to `docker`. For more on distinguishing between the two environments, see "Running Docker for Windows and Docker Toolbox" in [Getting Started](index.md). + +### How to solve `port already allocated` errors + +If you see errors like `Bind for 0.0.0.0:8080 failed: port is already allocated` or + `listen tcp:0.0.0.0:8080: bind: address is already in use` ... + +These errors are often caused by some other software on Windows using those +ports. To discover the identity of this software, either use the `resmon.exe` +GUI and click "Network" and then "Listening Ports" or in a powershell use +`netstat -aon | find /i "listening "` to discover the PID of the process +currently using the port (the PID is the number in the rightmost column). Decide +whether to shut the other process down, or to use a different port in your +docker app. + +### Docker fails to start when firewall or anti-virus software is installed + +The **Comodo Firewall currently is incompatible with Hyper-V and some Windows 10 +builds** (possibly, Windows 10 Anniversary Update), which impacts Docker for +Windows. **Other firewalls and anti-virus software might also be incompatible with these Microsoft Windows 10 buids**. The conflict typically occurs after a Windows update or new install of the firewall, and manifests as an error response from the Docker daemon and a **Docker for Windows start failure**. + +See the Comodo forums topics [Comodo Firewall conflict with +Hyper-V](https://forums.comodo.com/bug-reports-cis/comodo-firewall-began-conflict-with-hyperv-t116351.0.html) +and [Windows 10 Anniversary build doesn't allow Comodo drivers to be +installed](https://forums.comodo.com/install-setup-configuration-help-cis/windows-10-aniversary-build-doesnt-allow-comodo-drivers-to-be-installed-t116322.0.html). +A Docker for Windows user-created issue describes the problem specifically as it +relates to Docker: [Docker fails to start on Windows +10](https://github.com/docker/for-win/issues/27). + +For a temporary workaround, uninstall the Comodo Firewall, or explore other +workarounds suggested on the forum. + +
                                +
                                  +
                                  + +
                                  diff --git a/docs/images/chat.png b/docs/images/chat.png new file mode 100644 index 0000000000..597db5aae9 Binary files /dev/null and b/docs/images/chat.png differ diff --git a/docs/images/download-mac.png b/docs/images/download-mac.png new file mode 100644 index 0000000000..14627964e8 Binary files /dev/null and b/docs/images/download-mac.png differ diff --git a/docs/images/download-win.png b/docs/images/download-win.png new file mode 100644 index 0000000000..b934121090 Binary files /dev/null and b/docs/images/download-win.png differ diff --git a/docs/images/download.png b/docs/images/download.png new file mode 100644 index 0000000000..bfa896d89e Binary files /dev/null and b/docs/images/download.png differ diff --git a/docs/images/whale-x.png b/docs/images/whale-x.png new file mode 100644 index 0000000000..c99e8d5898 Binary files /dev/null and b/docs/images/whale-x.png differ diff --git a/project/DOCS.md b/project/DOCS.md new file mode 100644 index 0000000000..79b7b02ec5 --- /dev/null +++ b/project/DOCS.md @@ -0,0 +1,3 @@ +## documentation + +Documentation is now maintained and deployed by the documentation team. diff --git a/project/ISSUE-TRIAGE.md b/project/ISSUE-TRIAGE.md new file mode 100644 index 0000000000..e7b72a9924 --- /dev/null +++ b/project/ISSUE-TRIAGE.md @@ -0,0 +1,161 @@ +Triaging of issues +------------------ + +Triage provides an important way to contribute to an open source +project. Triage helps ensure issues resolve quickly by: + +- Describing the issue's intent and purpose is conveyed + precisely. This is necessary because it can be difficult for an + issue to explain how an end user experiences a problem and what + actions they took. +- Giving a contributor the information they need before they commit to + resolving an issue. +- Lowering the issue count by preventing duplicate issues. +- Streamlining the development process by preventing duplicate discussions. + +If you don't have time to code, consider helping with triage. The +community will thank you for saving them time by spending some of +yours. + +### 1. Ensure the issue contains basic information + +Before triaging an issue very far, make sure that the issue's author +provided the standard issue information. This will help you make an +educated recommendation on how this to categorize the issue. Standard +information that *must* be included in most issues are things such as: + +- the output of: + - `pinata diagnose -u` on OSX + - `DockerDebugInfo.ps1` using Powershell on Windows +- a reproducible case if this is a bug, Dockerfiles FTW +- page URL if this is a docs issue or the name of a man page + +Depending on the issue, you might not feel all this information is +needed. Use your best judgement. If you cannot triage an issue using +what its author provided, explain kindly to the author that they must +provide the above information to clarify the problem. + +If the author provides the standard information but you are still +unable to triage the issue, request additional information. Do this +kindly and politely because you are asking for more of the author's +time. + +If the author does not respond requested information within the +timespan of a week, close the issue with a kind note stating that the +author can request for the issue to be reopened when the necessary +information is provided. + +### 2. Classify the Issue + +An issue can have multiple of the following labels. + +#### Issue kind + +| Kind | Description | +|------------------|---------------------------------------------------------------------------------------------------------------------------------| +| kind/bug | Bugs are bugs. The cause may or may not be known at triage time so debugging should be taken account into the time estimate. | +| kind/docs | Writing documentation, man pages, articles, blogs, or other significant word-driven task. | +| kind/enhancement | Enhancement are not bugs or new features but can drastically improve usability or performance of a project component. | +| kind/feature | Functionality or other elements that the project does not currently support. Features are new and shiny. | +| kind/performance | Performance-related issues | +| kind/question | Contains a user or contributor question requiring a response. | +| kind/roadmap | Issue to track large items that need to be delivered. See [roadmap docs](ROADMAP.md) for more details. | +| kind/tracking | High-level issue to keep track of other issues. Tracking issues can be closed when all child issues are resolved. | +| kind/tracking/users | High-level issue to keep track of user feedback. | + +#### Functional area + +| Area Common | Description | +|---------------------------|------------------------------------------------------------------------------------------------------------------------| +| area/qa | Issue related to code quality and infrasture-related tasks to deal with automated QA | +| area/logging | Ensure consistent logging accross all the components | +| area/support | The built-in features for supporting users and reporting defects to the engineering team | +| area/tokens | Token mechanism to control the growth of beta users | +| area/sdk | | +| area/agent | | +| area/proxy | | +| area/upstream | Upstream bugs which needs to be reported | +| area/tools | Issues about the integration of open-source Docker tools (machine, toolbox, notary, etc.) in Pinata | + +| Area Windows | Description | +|---------------------------|------------------------------------------------------------------------------------------------------------------------| +| area/windows | | +| area/windows/build | | +| area/windows/installer | | +| area/windows/menubar | | +| area/windows/cli | | + +| Area OSX | Description | +|---------------------------|------------------------------------------------------------------------------------------------------------------------| +| area/osx | | +| area/osx/build | | +| area/osx/menubar | | +| area/osx/prefpane | | +| area/osx/installer | | +| area/osx/watchdog | | +| area/osx/cli | | + +| Area Backend | Description | +|------------------------------|---------------------------------------------------------------------------------------------------------------------| +| area/backend | General issues about the backend. If possible backend issues should have a more specific labels than this one. | +| area/backend/build | Build issues related to the backend components (OPAM packages, etc.) | +| area/backend/storage/volumes | Host/container file-sharing issues | +| area/backend/network/nat | NAT networking issues | +| area/backend/network/hostnet | Hostnet networking issues | +| area/backend/network/mdns | mDNS issues | +| area/backend/run/qemu | Issues related to the QEMU execution driver | +| area/backend/run/xhyve | Issues related with the Xhyve hypervisor | +| area/backend/run/hyper-v | Issues related with the Hype-V hypervisor | +| area/backend/moby/linux | Issues with the OS distribution for running Linux containers | +| area/backend/moby/windows | Issues with the OS distribution for running Windows containers | + +### 3. Prioritizing issues + +Issues with a known resolution date should be given a priority and attached to a milestone. +Issues that do not have a known resolution date should have a priority attached as this will assist with planning. + +The following labels are used to indicate the degree of priority (from more urgent to less urgent). + +| Priority | Description | +|-------------|-----------------------------------------------------------------------------------------------------------------------------------| +| priority/P0 | Urgent: Security, critical bugs, blocking issues. P0 basically means drop everything you are doing until this issue is addressed. | +| prority/unknown | The resolution date and/or the priority of that issue unclear. That issue should be discussed in a weekly meeting. | +| priority/P1 | Important: P1 issues are a top priority and a must-have for the next release. | +| priority/P2 | Normal priority: default priority applied. | +| priority/P3 | Best effort: those are nice to have / minor issues. | + +The release manager (or the maintainer if they know what they are +doing) is responsible for prioritizing issues. + +### 4. Handling User Reported Issues + +Issues that have been reported by a user via email to beta-feedback@docker.com, on the Docker Forums or by other means MUST have the `kind/tracking/user` label attached. + +A user reported issue is prioritized in accordance with the following table: + +| Priority | Descrtiption | +|----------|-------------------------------------------------------------------------------------------------------------------| +| P0 | As P1 but effects a large portion of the user base. Please consult the release manager before applying this label | +| P1 | I can't install/use the app at all - it's totally broken! | +| P2 | The app doesn't work and it severely impacts my workflow | +| P3 | I have a problem that effects my workflow but it doesn't stop me getting work done | + +If multiple reports of the same issue are provided, this will enable the release manager or PM to bump the priority of the issue. +Links to the source of the reports should be added to the GitHub issue. + +### 4. Milestone + +There should be a milestone per planned release. The release managers are +reponsible for assigning issues to milestones, in coordination with PM and +the rest of the engineering group. + +### 5. Status + +Status labels can be attached to issues to add more detailed status than just "open" and "closed". These are particularly useful to track the status of roadmap issues. + +| Label | Description | +|---------------------------|------------------------------------------------------------------------------------------------------------------------| +| status/at-risk | Issue may not land in the milestone specified | +| status/delayed | Issue will not land in the milestone specified | +| status/in-progress | Issue is currently being worked on | +| status/postponed | Issue is being pushed back and milestone is yet to be determined | diff --git a/project/MAC-RELEASE.md b/project/MAC-RELEASE.md new file mode 100644 index 0000000000..c20c78df0b --- /dev/null +++ b/project/MAC-RELEASE.md @@ -0,0 +1,59 @@ +### Preparing a Mac release + +This document describes the Mac-specific part of the release process. See +[RELEASE.md](RELEASE.md) for the overall process. + +### Update metadata + +- Run `make versions` to verify that all version strings are + correct. Manual check: + + - Verify that the version up to date, of the form + `--dev` (see + `README.md > Versioning `). (like "1.9.1-beta2-dev"). The version is + stored in `CFBundleShortVersionString` in + `v1/mac/src/docker-app/docker/docker/Info.plist` and the xcode + project can be opened with `make dev`. + + - Verify that the right version is in + [docker-diagnose](../v1/docker-diagnose/src/diagnose.ml) + + - Verify that the right version is in the [CHANGELOG](../CHANGELOG) + +- Update the date and add final entries in the CHANGELOG. + + +### Run tests + +Verify that the tests pass: + +``` +make test +``` + +If brave enough, verify that the long-running tests also work: + +``` +make fulltest +``` + +Additional manual tests (optional): + +1. Verify that both auto-update and download from dmg works by + - uninstalling current version (use button in settings) + - download previous master build from hockeyapp and install from dmg + - auto update to latest master + +2. Verify that Docker.app is still running after a reboot. + +3. Build and run some test containers to see that everything works as expected. + +See also [MAC-TESTING.md](MAC-TESTING.md) + +### Tag the release + +See [RELEASE.md](RELEASE.md) for details. + +### Verify and test build + +See [MAC-TESTING.md](MAC-TESTING.md) for pre-release test plan. diff --git a/project/MAC-TESTING.md b/project/MAC-TESTING.md new file mode 100644 index 0000000000..b609143813 --- /dev/null +++ b/project/MAC-TESTING.md @@ -0,0 +1,54 @@ +### Mac test plan + +- [ ] Basic installation: Uninstall any running Docker.app's and then install the build + +Run release tests on the build, e.g. by replacing BUILD with the correct version in the following bash script: + +``` +# make sure v1/mac/build is empty first, otherwise that build will be tested +export BUILD="1.12.xx.xx" +export URL="https://download-stage.docker.com/mac/rc/$BUILD/Docker.dmg" +./rt-local -l release,installer -vv -e "D4X_INSTALLER_URL=${URL}" -x run +``` + +VMs can now be created easily with `./rt-vm` from `tests/` (requires access to Cambridge office network). + +Pinata-CI builds for tags will appear here: https://rod.cam.docker.com:8443 + +- [ ] OS X 10.10 release tests +- [ ] OS X 10.11 release tests +- [ ] OS X 10.12 release tests + +- [ ] Auto update: Uninstall build, then install previous test build and auto update to the current one. If a test build is not available, pick a master build close to previous release and auto update to current release. +- [ ] Auto update on 10.12: Install previous master build on OSX 10.12 and upgrade to latest version to verify that auto updater works +- [ ] Check that events appear in Mixpanel + +- [ ] Verify that experimental features are enabled/disabled (beta=enabled, stable=disabled) + +UI tests (optional, but should be tested if there is a relevant change) + +- [ ] Test that migration from Toolbox works as expected +- [ ] Test that filesharing UI works +- [ ] Test that insecure registries and registry mirrors work + +On release binary: +- [ ] Verify that Info.plist auto updates from the correct appcast endpoint. +- [ ] Run `./rt-local -l release -v -x run` a last time to check that tests are still green +- [ ] Verify that `docker-diagnose` works on release build (after #4869) +- [ ] Test moving between channels, see below + +#### Channel tests + +If releasing beta: + +1. install current stable then attempt beta installation over top +2. install current stable over top of testing beta + +Both should result in warnings/errors and a functional installation of the new install afterward. + +If releasing stable: + +1. install current beta then attempt stable installation over top +2. install current beta over top of testing stable + +Both should result in warnings/errors and a functional installation of the new install afterward. diff --git a/project/RELEASE-CHECKLIST.md b/project/RELEASE-CHECKLIST.md new file mode 100644 index 0000000000..ed888eebe8 --- /dev/null +++ b/project/RELEASE-CHECKLIST.md @@ -0,0 +1,43 @@ +This issue tracks progress of the upcoming release of Docker for Mac/Windows. + +### Release plan + +- [ ] Reschedule tickets that will not be fixed for this release (@djs55, @dgageot and all) + +#### Release preparations on Monday + +- [ ] Announce release plans in #pinata-dev (@magnuss) +- [ ] Pre-release meeting Monday (@magnuss): [Agenda](https://docs.google.com/a/docker.com/document/d/1b_Mfe3XjW8OOCoUfyy4luY9yZOSakJvXy3Ooz4JjK5A/edit?usp=sharing) +- Mac (@djs55) + - [ ] Prepare Mac [CHANGELOG](https://github.com/docker/pinata/blob/master/CHANGELOG) + - [ ] Build Mac UI complete binary (master build is UI complete) +- Windows (@dgageot) + - [ ] Prepare Windows [CHANGELOG](https://github.com/docker/pinata/blob/master/win/CHANGELOG) + - [ ] Build Windows UI complete binary +- Docs (@londoncalling) (may be delayed) + - [ ] Prepare release notes based on [Mac CHANGELOG](https://github.com/docker/pinata/blob/master/CHANGELOG) + - [ ] Prepare release notes based on [Windows CHANGELOG](https://github.com/docker/pinata/blob/master/win/CHANGELOG) + - [ ] Update docs based Windows release binary when ready + - [ ] Update docs based on Mac release binary when read + +#### Release preparations on Tuesday + +- [ ] Windows binaries (@dgageot) + - [ ] Build RC and announce in #pinata-dev + - [ ] Test RC (see TODO for test plan) +- [ ] Mac binaries (@magnuss) + - [ ] Build RC - announce in #pinata-dev + - [ ] Test RC (see TODO for test plan) +- [ ] If RCs works fine, tag release binaries (@magnuss, @dgageot) +- [ ] Update License.tar.gz (@frenchben) + +#### Synchronised release of binaries + +- [ ] Release Win/Mac binaries with release notes (@magnuss) +- [ ] Announce in #ship-it, #pinata and #pinata-dev (@magnuss) + +#### Post-release + +- [ ] Deploy docs to docs.docker.com (@londoncalling) +- [ ] Bump Mac version (@jeanlaurent) +- [ ] Bump Win version (@dgageot) \ No newline at end of file diff --git a/project/RELEASE.md b/project/RELEASE.md new file mode 100644 index 0000000000..f6dac44743 --- /dev/null +++ b/project/RELEASE.md @@ -0,0 +1,325 @@ +# Project Pinata release process + +The Pinata release process currently takes two days to allow the teams in each time zone to prepare for the release. The first day is for planning and merging final PRs and freezing the UI. The second day is for creating and testing final binaries before the actual release in the afternoon/evening. If the release is to be synchronised with a documentation release, the documentation team also needs time to prepare final changes after the binaries have been built. + +This document describes the release process as of July 5th 2016. + +Jump to + + - [Summary: Release preparations day 1 ](#release-preparations-day-1) + - [Issue triaging](#issue-triaging) + - [Release meeting](#release-meeting) + - [Github tracking issue](#github-tracking-issues) + - [Updating CHANGELOGs](#changelogs) + - [Summary: Release preparations day 2](#release-preparations-day-2) + - [Tagging releases](#tagging-releases) + - [Synchronised release](#synchronised-release) + - [Troubleshooting](#troubleshooting) + - [Web UI](#web-ui) + - [Rollback Release](#rollback-release) + - [Update release notes with docker-release](#update-release-notes-with-docker-release) + - [Surgically updating the CHANGELOG](#surgically-updating-the-changelog) + - [Hotfixes and updates on stable channel](#hotfixes-and-updates-on-stable-channel) + - [Processes for PRs](#processes-for-prs) + - [Releasing a beta hotfix](#releasing-a-beta-hotfix) + - [Releasing a stable hotfix](#releasing-a-stable-hotfix) + +## Release preparations day 1 + +Summary: + +- Triage issues before release meeting ([see details below](#issue-triaging)) +- Prepare tracking issues on Github ([see details below](#github-tracking-issues)) +- Publish release plans in #pinata-dev which should contain: + - When next release is scheduled + - Link to agenda for the [release meeting](#release-meeting) + - Link to Github tracking issue + - Links to open Github issues for this milestone and remaining P0 issues. If there are many open issues it may also be useful with a reminder to triage them, as they will all be discussed in the release meeting. +- Prepare CHANGELOGs for Windows/Mac ([see details below](#changelogs)) +- Release meeting @ 5pm UK time ([see details below](#release-meeting)) +- When UI complete binaries are ready, send binary links and initial CHANGELOGs to documentation team + +#### Github tracking issues + +Each release is tracked by a tracking issue on Github. The tracking issue should include the tasks that should be completed per day and the person assigned to it, if relevant. The issue should be tagged `kind/tracking/release`, `area/osx` and `area/windows`. + +[List of earlier tracking issues](https://github.com/docker/pinata/issues?utf8=✓&q=is%3Aissue%20label%3Akind%2Ftracking%2Frelease%20) + +In addition, create a tracking issues for the test plans and link them from the main tracking issue. + +The Windows testing issue template is in `WIN-TESTING.md`, and the new issue should be called: `win: Release testing checklist` and be tagged with `kind/tracking/release` and `area/windows`. + +The Mac testing issue template is in `MAC-TESTING.md`, and the new issue should be called: `mac: Release testing checklist` and be tagged with `kind/tracking/release` and `area/osx`. + +#### Issue triaging + +Triage bugs on Github with the DRIs using the latest milestones defined on +https://github.com/docker/pinata/milestones + +See ISSUE-TRIAGE.md for details about the triaging process. + +#### CHANGELOGs + +Update the CHANGELOG files for Mac and Windows, with user-facing updates since the last release. + - add issue/PR number if available + - do not put names + - the "compare" feature of GitHub is helpful: + `https://github.com/docker/pinata/compare/...master` + +An initial CHANGELOG should be prepared prior to the release meeting so it can be discussed in the meeting and be ready for the documentation team. The final CHANGELOG will not be ready until actual release. Sync with docs team if there are large changes from the initial version. + +Remember to include docs in the list of changes if the release is synchronised with the binaries. + +The Windows and Mac CHANGELOGs are currently stored separately in [CHANGELOG](../CHANGELOG) and [win/CHANGELOG](../win/CHANGELOG). + +#### Release meeting + +The release meeting is held by the release manager on Mondays @ 5pm UK time. The agenda is posted in `#pinata-dev` Monday morning. + +Bluejeans link: https://bluejeans.com/747188286 + +Agenda should include + - Summary of previous release, discuss possible improvements + - Plans for next release (expected release time, release plans for documentation, engine upgrades etc) + - Discuss any open P0s on Github, plans for fixing them and/or if release should be delayed + - Triage open issues for current release milestone, move issues that will not be fixed in time + - Discuss UI changes for the release and if there are new features/UI that require documentation updates + - Go through [CHANGELOGs](#changelogs) for Windows/Mac + - If time: go through issues without an assigned milestones, in case there are important new issues + +Some earlier agendas +- [Beta 17 agenda](https://docs.google.com/document/d/1KdyKZ4Jvl4e42QSa4h42SZZaKvYmFKhXGzt1ApvvojM/edit) +- [Beta 13 agenda with notes](https://docs.google.com/document/d/1YPgm-oH_E1A4DylIlBXPz49WQ1hLVJjMEj3LmJkmRlU/edit) + +## Release preparations day 2 + +Summary: +- If all PRs are in (check with DRIs, @pinata-team on slack) tag a test release ([see details below](#release-candidate)) +- Follow release process per platform: [Windows](WIN-RELEASE.md) / [Mac](MAC-RELEASE.md) +- Test according to test plans: [Windows](WIN-TESTING.md) / [Mac](MAC-TESTING.md)). As a minimum the release should pass all `rt-local` tests with the `release` label. +- Tag release binaries for beta or stable channel ([see details below](#tagging-releases)) +- When release binaries are ready, run final tests on them and post binary link in `#pinata-dev` +- Send release binaries to @victoria with final release notes +- If synchronised release with docs, wait for final documentation updates +- Release binaries and announce the new release ([see below for details](#synchronise-release)) + +### Tagging releases + +#### Release candidate + +Creating a test release of Beta20 with Docker 1.12.0-rc4: + +* for Mac the tag should be: `mac-rc-v1.12.0-beta20` (note that for engine versions with a suffix, such as `1.12.0-rc4`, the tag is still `mac-rc-v1.12.0-beta20` due to limitations with current regexp) +* for Windows the tag should be: `win-test-v1.12.0-rc4-beta15` + +This will upload the artifact to the following endpoint, given a CI build of `8000`, and auto-publish it: + +``` +https://download-stage.docker.com/mac/rc/1.12.0.8000/ +https://download-stage.docker.com/win/test/1.12.0.8000/ +``` + +E.g.: +Make sure on right commit/branch, then: + +``` +VERSION="mac-rc-v1.12.0-beta20" +git tag $(VERSION) -a -m "Test release $(VERSION)" +git push upstream $(VERSION) +``` + +After the tag has been built the files on each endpoint should be: +``` +NOTES +Docker.dmg.sha256sum (mac) +Docker.dmg (mac) +InstallDocker.msi (win) +InstallDocker.msi.sha256sum (win) +``` + +For more details about the uploaded artifacts, see the CI build output. + +##### Beta Release + +Creating a beta release of Beta20 with Docker 1.12.0-rc4: + +* for Mac the tag should be: `mac-v1.12.0-beta20` +* for Windows the tag should be: `win-beta-v1.12.0-rc4-beta20` + +(Note that only the engine version is used in the Mac tag with suffixes such as `-rcX` ignored. E.g. the tag for `v1.12.0-rc4` is `mac-v1.12.0-beta20`. This is due to a limitation of the regexp used in the Mac build scripts. This is not the case for the Windows tag.) + +This will upload the artifact to the following endpoint, given a CI build of `8001`: +``` +https://download.docker.com/mac/beta/1.12.0.8001/ +https://download.docker.com/win/beta/1.12.0.8001/ +``` + +The binaries are not released until the `appcast.xml` file is updated - how to release them is described [below](#release-artifacts) + +##### Stable Release + +Creating a stable release with Docker 1.12.0: + +* for Mac the tag should be: `mac-v1.12.0` +* for Windows the tag should be: `win-v1.12.0` + +This will upload the artifact to the following endpoint, given a CI build of `8002`, and hold-it to be published manually: +``` +https://download.docker.com/mac/stable/1.12.0.8002/ +https://download.docker.com/win/stable/1.12.0.8002/ +``` + +The stable channel is not used until after 1.12 GA / Pinata GA. + +### Synchronised release + +When the builds have been tested and docs are ready for deployment, add Github release details, release the artifacts with `docker-release` and announce the release. + +Verify that the release notes are correct prior to release, as these will be shown in the Auto update window. + +Check that docs are up to date (verify release notes!), S3 downloads are working and correct and that +License.tar.gz has been updated. + +#### Create GitHub releases + +Create GitHub releases for Win and Mac and update the +release notes in https://github.com/docker/pinata/releases: + +- create a new release for the value of the Mac release tag +- copy/paste the related lines from the the `CHANGELOG` file +- create a new release for the value of the Win release tag +- copy/paste the related lines from the the `win/CHANGELOG` file + +#### Release artifacts + +To release the artifacts to the public from the beta channel, given a version of `1.12.0.8002` run: + +OSX: + +`docker-release --aws-access-key xxx --aws-secret-key xxx --channel beta --build 1.12.0.8002 --human 1.12.0-beta20 --arch mac -prod publish` + +Windows: + +`docker-release --aws-access-key xxx --aws-secret-key xxx --channel beta --build 1.12.0.8002 --human 1.12.0-beta20 --arch win -prod publish` + +Replace the AWS keys, the build number and the human string with the correct values. + +All other channels than stable and beta, are published automatically upon upload of the artifacts. Publishing a certain version also promotes that version as the 'latest' downloadable artifact, which will create the following endpoint for DOCS user download purpose (new user trying to download the edition) + +OSX: + +`https://download.docker.com/mac/stable/Docker.dmg` +`https://download.docker.com/mac/beta/Docker.dmg` + +Windows: + +`https://download.docker.com/win/stable/InstallDocker.msi` +`https://download.docker.com/win/beta/InstallDocker.msi` + +#### Announce the release + +Announce the release in #pinata-dev with `@here` and post the final CHANGELOGs for Windows +and Mac. + +Announce in #ship-it: +``` +Mac and Windows Beta ... has been released! +https://download-stage.docker.com/docs/windows/release-notes/ +https://download-stage.docker.com/docs/mac/release-notes/ +``` + +Announce in #pinata: +``` +Mac and Windows Beta ... has been released! +https://download-stage.docker.com/docs/windows/release-notes/ +https://download-stage.docker.com/docs/mac/release-notes/ +``` + +The release should now be complete! + +## Troubleshooting + +### Web UI + +A basic web UI to list releases and files on the public channels is available: + +http://omakase.omakase.e57b9b5b.svc.dockerapp.io/# + +### Rollback Release + +Rolling back to a previous release is as simple as publishing the old release. +Given a current release of: `1.12.0-beta18`, rollingback to the previous version/build would be: + +Mac: + +`docker-release --channel beta --build 1.12.0.9779 --human 1.12.0-rc2-beta17 --arch mac --prod publish` + +Windows: + +`docker-release --channel beta --build 1.12.0.5022 --human 1.12.0-rc2-beta17 --arch win --prod publish` + +Note that you need to know the build number of the previous release to do this. This is usually recorded in the tracking issue, but list of releases in the [Web UI](#web ui) can also be helpful. + +### Update release notes with `docker-release` +When the release is tagged the CHANGELOG is extracted and uploaded in the `NOTES` file. This file can be re-uploaded with `docker-release`. + +To download `NOTES` for editing: + +`curl -SLO https://download.docker.com/mac/beta/1.12.0.9996/NOTES` + +To upload the updated `NOTES` file and overwrite the existing one: + +`docker-release --arch mac --channel beta --build 1.12.0.9996 --prod upload NOTES` + +### Surgically updating the CHANGELOG + +Usually `docker-release` should provide everything you need to add/update files on S3. If you should need to manually modify the files, one way of doing it is to use the widely available `s3cmd` (`awscli` should also work in a similar way) + +* configure s3cmd to point to `download.docker.com` buckets +* Note that we have two buckets one for beta/stable `s3://editions-us-east-1` another for all other channels (`s3://editions-beta-us-east-1-150610-005505`) +* The path to the proper download is : `s3:[bucket]/[os]/[channel]/` where `os` is either `mac` or `win`, and channel either `beta` or `stable`. +* list the documents in the proper channel by `s3cmd ls s3://editions-us-east-1/mac/beta/` +* find the proper version and list files there `s3cmd ls s3://editions-us-east-1/mac/beta/1.12.0.9996/` +* the changelog is the `NOTES` files +* get the remote `NOTES` file `s3cmd get s3://editions-us-east-1/mac/beta/1.12.0.9996/NOTES` +* copy over the revelant part of `docker/pinata/CHANGELOG` for mac or `docker/pinata/win/CHANGELOG` for windows +* upload the new `NOTES` file : `s3cmd put NOTES s3://editions-us-east-1/mac/beta/1.12.0.9996/ --acl-public -m "text/plain; charset=utf-8"` +* check the public bucket URL is ok : `curl http://editions-us-east-1.s3.amazonaws.com/win/beta/1.12.0.5226/NOTES` +* if something is wrong at this point, this is probably missing metadata check them with `s3cmd info s3://editions-us-east-1/mac/beta/1.12.0.9996/NOTES` +* check the public endpoint : `curl https://download.docker.com/win/beta/1.12.0.5226/NOTES` ( you may have slight delay due to cloudfront caching) + +## Hotfixes and updates on stable channel + +### Processes for PRs + +#### Non urgent, but necessary fixes in stable + +Label the PR with `process/cherry-pick` and assign it to the relevant release milestone. This will mark the PR as a candidate to be picked into that stable release from master, when that release happens. Typically this is the next minor engine release. The PR in master should be included in one of the beta releases at least one week before the stable release. + +#### Urgent stable fixes + +If the issue needs to be urgently fixed, contact the release manager to make a plan for the release and to create a hotfix milestone. Typically this will require a beta hotfix first to test the fix, then a stable hotfix a week later. + +#### Urgent beta-only fixes + +If there is an issue in the beta that does not need to be fixed in stable, a hotfix can be released on the beta channel only. Contact the release manager to make a plan for the release. The PR does not need a label. + +### Releasing a beta hotfix + +A hotfix is usually a minimal update that we send out to fix an urgent issue between planned releases. + +* Create a branch from the beta release tag called `bXX-release-fixes`, where XX is the beta number the hotfix is for. In some cases it may be necessary to create `bXX-release-fixes-mac` or `bXX-release-fixes-win` if there were important commits in master between the win/mac tags. +* Cherry-pick the relevant PRs from master into the new branch +* Update the CHANGELOG in the branch to include the list of hotfixes under a "* Hotfixes" title. This will be shown in the update dialog. +* Add the release to the CHANGELOG in master +* Create a release tag on the last commit on the branch (as on master) to build the release binary in CI. Add the suffix `.X` where `X` is the hotfix number for this release, e.g. `mac-v1.12.0-beta21.1`. +* Re-test the binary and release as normal. Additional testing may be required depending on the changes that were fixed. + +### Releasing a stable hotfix + +A stable hotfix is an urgent update on the stable channel between planned releases. The PRs included in the hotfix should previously be tested in a beta before the changes are added to the stable branch. + +* Determine the minimal set of changes that needs to be included in the update, make sure all changes have been tested on beta channel first. Merge the PR in the branch for the current stable release. +* Add a letter to the version number, e.g. "1.12.0b" for the second update to `1.12.0` +* Update CHANGELOG with a friendly message describing the reason for the update (this will be displayed in the update dialog) +* Tag and release as normal - additional testing may be required depending on the changes that were fixed diff --git a/project/ROADMAP.md b/project/ROADMAP.md new file mode 100644 index 0000000000..de52466b1e --- /dev/null +++ b/project/ROADMAP.md @@ -0,0 +1,24 @@ +# Roadmap process + +The roadmap is tracked with GitHub issues with the "kind/roadmap" label. These things are large pieces of work that typically need to be delivered by a certain date, and may have complex dependencies that need tracking. + +Using the "kind/roadmap" label allows us to filter out the major items of work from all the other smaller stuff on GitHub (bugs, feature requests, sub-tasks, etc). + +The text of roadmap issues can be used as a wiki to give detail on that piece of work, keep track of its status, and link off to other relevant stuff. This can also link off to other sub-tasks if more granularity is necessary. + +Issues should be given: + + - **A milestone** to define its aimed delivery date + - **An owner** to define its DRI + - **A priority** as a label to define its priority relative to other items + +## Reports + +GitHub is the source of truth for the roadmap, but GitHub also has a great API that allows us to produce reports to give visibility into what is going on both within the team and to people outside the team. + + - [Charing Cross](https://charingcross.i.dckr.io/repos/docker/pinata), a tool for viewing milestones and the status of roadmap issues within those milestones. + +The issues can also be browsed using GitHub's interface. For example: + + - [All roadmap issues](https://github.com/docker/pinata/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Froadmap) + - [All roadmap issues for "GA" milestone](https://github.com/docker/pinata/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Froadmap+milestone%3AGA) (modify milestone filter to view others) diff --git a/project/SUPPORT.md b/project/SUPPORT.md new file mode 100644 index 0000000000..3580643981 --- /dev/null +++ b/project/SUPPORT.md @@ -0,0 +1,80 @@ +Piñata Support +============== + +This document outlines the roles and responsibilities of the support function +of Piñata. + +# Roles and responsibilities + +## L1 Support + +The L1 Support staff are responsible for: + +- Categorising and labelling issues in `docker/for-mac` and `docker/for-win` +- De-duplicating issues in the aforementioned repositories +- Ensuring that the issues are of high quality + - They contain a `docker-diagnose` output + - The logs for the report are available and uploaded to S3 + - This can be verified with `Dockerfile.fetch` in `support/s3` + - There is sufficient information about the users environment + - Is this a new/failed installation + - An upgrade + - A channel switch + - Reproduction instructions are present. The following are acceptable: + - A sequence of UI interactions that will cause the issue + - A shell script that will trigger the issue + - A `Dockerfile` that will trigger the issue + - A `docker-compose.yml` that will trigger the issue + - The reproduction instructions are valid, and reproduce the issue + - **IF** the cause is obvious, open an issue on `docker/pinata` and ping + the right person + - This issue should contain all of the necessary information, + not just a link to the `docker/for-X` bug + - Sending a report at the end of a rotation to `pinata@docker.com` or + `@pinata-team` on Slack that includes: + - Major occurrences + - Any patterns or trends identified + - Statistics: + - Issues Opened vs Closed this week + - Issues Opened vs Close all time + - Average time to resolution + - Tickets over N days old, where N is the targeted resolution time + +## L2 Support + +The L2 Support staff are responsible for: + +- Reviewing categorised issues with reproduction cases to identify the + root cause +- Once that cause has been found, to create an issue in `docker/pinata` and + assign the right person +- They should track the open issue to make sure it is resolved within a + reasonable timeframe (SLA TBD) +- Once the issue has been fixed (or not), they should update the issue in the + upstream tracker with the appropriate label +- Once the fix has been applied to a publicly available release, the + appropriate label should be applied + +## L3 Support + +In the proposed model L3 support are engineering. +L3 support are responsible for: + +- Reviewing issues where they have been assigned by L2 support +- Either fixing the issue, or marking as `0-wont-fix` + +# Status Labels (to be documented on `docker/for-X`) + +The following labels track the progression through the support process for +ease of triage. + +| Label | Description | +|---------------------------|------------------------------------------------------------------------| +| status/0-triage | The issue needs triaging | +| status/0-wont-fix | This issue will not be fixed and therefore can be closed | +| status/0-more-info-needed | The issue needs more information before it can be triaged | +| status/1-acknowledged | The issue has been triaged and is ready for L2 | +| status/2-in-progress | The issue has been assigned to a engineer and is waiting a fix | +| status/3-fixed | The issue has been fixed in `master` | +| status/4-fix-released-beta | The fix has been released! | +| status/4-fix-released-stable | The fix has been released! | diff --git a/project/WIN-RELEASE.md b/project/WIN-RELEASE.md new file mode 100644 index 0000000000..79c4b22144 --- /dev/null +++ b/project/WIN-RELEASE.md @@ -0,0 +1,46 @@ +# Preparing a Windows release + +This document describes the Windows-specific part of the release process. See +[RELEASE.md](RELEASE.md) for the overall process. + +On windows, pushing a release on own s3 buckets is an automatic process. + +## Tags and Channels + +All of the process is made on AppVeyor. Each time it builds either on `master` +branch or a tag, AppVeyor will build an installer an upload it to s3 using the `docker-release` command. + +Before tagging a release, please make sure the `CHANGELOG` file is up to date. + +**WARNING**: To make sure we don't pollute OSX releases, every Windows release tag name must +start with `win-` prefix. + +The build script analyze the tag name, and determine on which channel it uploads the artifact. +the tag format is the following `win-[channel]-v[version]-[milestone]` for example `win-beta-v1.12.0-beta18`. +`[channel]` can be either `beta`, `test`. Releasing on `stable` is made by omitting the channel altogether for +instance `win-v1.12.0-beta19`. + +The channel it uploads to is chosen by those rules: +- If the commit has a tag that starts with `win-v`, then the installer is pushed to `stable` channel. +- If the commit has a tag that starts with `win-beta-v`, then the installer is pushed to `beta` channel. +- If the commit has a tag that starts with `win-test-v`, then the installer is pushed to `test` channel. +- Otherwise (either on master or malformed tag), the installer is pushed to `master` channel. + +Tagging example +``` +git tag win-test-v1.11-beta11 +git push origin win-test-v1.11-beta11 +``` + +Release to the `beta` or `stable` channels are not published right away to users, they are uploaded +and available to download but it will not trigger an automated users. You must use the `docker-release` +command to achieve this. + +## Preparing the next iteration + +After a release, the version needs to be bumped for the next release: + + - Update `src/SolutionInfo.cs` file + - Set the date on the `CHANGELOG` for the current release, replacing `unreleased` by yyyy-mm-dd + - Create a section for this release in the `CHANGELOG` file + - Commit theses changes with a commit message similar to `Bump to version X.Y.Z-dev" diff --git a/project/WIN-TESTING.md b/project/WIN-TESTING.md new file mode 100644 index 0000000000..702e455ee5 --- /dev/null +++ b/project/WIN-TESTING.md @@ -0,0 +1,70 @@ +This is a template for tests which have to be performed before a Windows release. + +- [ ] Run regression tests with `release,installer` labels like this: `./rt-local -vxl release,installer` -e D4X_INSTALLER_URL=https://download-stage.docker.com/win/master/InstallDocker.msi run` + - [ ] On Windows 10 Pro Build 10586 (aka 1511) + - [ ] On Windows 10 Pro Build 14363 (aka 1607 aka aka RS1 Anniversary Release) + +The quickest way to download other binaries is via: +``` +Import-Module BitsTransfer +$url = "https://download-stage.docker.com/.../InstallDocker.msi" +Start-BitsTransfer -Source $url -Destination InstallDocker.msi +``` + +The following steps currently have to be performed manually: +**Make sure you start the app from the desktop icon, and not via the CLI!!!** +- Test auto-update. + - Download a previous release from the test channel via [this page](http://omakase.omakase.e57b9b5b.svc.dockerapp.io/) + - Install it + - Wait for the update dialogue, verify the version, and update + - [ ] Verify that the version after restart + +- Verify versions + - Verify there is a whale on the Desktop. Check if it says Beta + - [ ] Until #4613 is closed, verify with `docker version` that Beta release use experimental and that stable release use non-experimental. + - [ ] Verify the version in the `About Box` + - [ ] Verify `docker-machine version` + - [ ] Verify `docker-compose version` + - [ ] Verify that the path was added to a Powershell, cmd, and bash window + +- Test settings UI (has to be executed in order): + - [ ] Shared Drive: Make sure you have a D drive. Share it, unshare it and share it. Verify with `docker run --rm -v d:/:/data alpine ls /data` that the sharing/unsharing worked. **Leave the drives shared**. + - [ ] Advanced: change the CPU and Memory settings a couple of times. Verify with `Get-VMProcessor MobyLinuxVM` and `Get-VMMemory MobyLinuxVM` that the VM has the new settings. **Leave it at a non-default value**. + - Network: Change the Subnet address to `10.0.76.0` and the mask to `255.255.255.248` and the DNS to static `8.8.4.4`. Verify: + - [ ] `ipconfig` that the ip address/netmask has changed + - [ ] `docker run --rm -v c:/:/data alpine ls /data` verifies that mounting still works + - [ ] `docker run --rm alpine wget http://www.google.com` verifies that DNS and networking still works + - **Need a test for proxy settings** + - Docker Daemon: + - [ ] Verify the URL points to the damon config file section + - [ ] Change the debug setting to `true`. Verify using `docker info` (look for `Debug Mode (server): true`) + - Diagnose and Feedback + - [ ] Verify the documentation link points to the documentation + - [ ] Verify the issue link to Github works + - [ ] Verify the `Logs` link works + - [ ] Click on `Upload diagnostics` and verify that the diagnostics can be retrieved using the `Dockerfile.fetch`. + - Exit the App and Start again. Verify that the settings are still what they were before the restart and verify that they are actually used: + - [ ] `docker images` should be empty + - [ ] `ipconfig` that the ip address/netmask + - [ ] `docker run --rm -v c:/:/data alpine ls /data` verifies that mounting still works + - [ ] `docker run --rm alpine wget http://www.google.com` verifies that DNS and networking still + - [ ] `docker info` for debug settings + - Reset: Reset to factory and check, e.g.: + - [ ] Shared Drive: Make sure they are not ticked + - [ ] Shared Drive: Verify that `docker run --rm -v c:/:/data alpine ls +/data` is empty. Make sure it pulls in alpine. + - [ ] Advanced: Check that CPUs is reset and memory set to 2GB + - [ ] Advanced: Verify with `Get-VMProcessor MobyLinuxVM` and `Get-VMMemory MobyLinuxVM` + - [ ] Network: Make sure the IP address is `10.0.75.0` and mask is `255.255.255.0` and DNS is unticked. + - [ ] Docker Daemon: Make sure `Debug` is set to `false` + - [ ] Docker Daemon: Check `docker info` for debug setting + + +- Test with a user using a Microsoft Live ID for login + - [ ] Verify sharing/unsharing via the UI works + +- Test with a user with a space in the User name + - [ ] Verify sharing and/unsharing via the UI works + +- Mixpanel: Requires logging in to mixpanel: + - [ ] verify that mixpanel events in the final release binary report the correct version string (due to #4925) diff --git a/project/designs/TRIM.md b/project/designs/TRIM.md new file mode 100644 index 0000000000..a03d0cb9c6 --- /dev/null +++ b/project/designs/TRIM.md @@ -0,0 +1,152 @@ +# Reclaiming disk space on the Mac + +See [docker/pinata#2080] + +Engine 1.13 will include a feature allowing disk space to be reclaimed more +easily by deleting unused images. Unfortunately deleting files within the Moby VM +will not delete disk blocks stored on the host unless the block device supports +ATA TRIM / SCSI DISCARD. Note that Hyper-V supports AAT TRIM / SCSI DISCARD so this +should "just work" on Windows. + +## The datapath + +``` + docker + | + | rm unused images + V + ext4 + | + | [1] TRIM unused blocks + V + ahci-hd [2] (virtio-blk doesn't support TRIM) + | + | TRIM unused blocks + V + hyperkit [3] (needs to expose the capability, currently commented out) + | + | TRIM unused blocks + V + ocaml-qcow + | + | GC [4] + V + HFS+ +``` +where + +- [1] we need to investigate/decide whether TRIM should be run continuously + as a side-effect of an `rm` (enabled via mounting with the `discard` flag) + or whether we should have a `fstrim` running later, possibly on request? + This will likely depend on the performance of the rest of the system. + See [comments and notes on the Arch Wiki](https://wiki.archlinux.org/index.php/Solid_State_Drives). +- [2] unfortunately the virtio-blk protocol doesn't support TRIM so we have to + switch to the only other option in hyperkit: ahci-hd. Note that qemu/KVM + users would probably use virtio-scsi. We will have to investigate the + performance implications. +- [3] hyperkit's bhyve heritage includes support for exposing TRIM, but + the [capability is not exposed](https://github.com/docker/hyperkit/blob/547caeb5facb248067c529dd8c80931dbc1c56c6/src/block_if.c#L620) + and the operation [returns ENOTSUPP](https://github.com/docker/hyperkit/blob/547caeb5facb248067c529dd8c80931dbc1c56c6/src/block_if.c#L416). + This is clearly because it's not implemented by existing filesystems on + the Mac. +- [4] since the Mac doesn't support TRIM on HFS+, we will need to implement a + block GC to shrink the file. + +## The qcow block GC + +The general approach is to shrink the qcow2 file with `ftruncate` after copying +the trailing data blocks back to "holes" created by the TRIM. A shuffle operation +will be approximately: + +1. Find the pointer which is pointing to the last physical cluster. This will + require a reverse index to be created and maintained, since all the references + within the physical file point in the other direction. We can make the + simplifying assumption that the refcount is 1 since we don't support internal + snapshots. +2. Lock the last physical cluster and a TRIMed cluster, to prevent concurrent + modification from another concurrent thread. +3. Copy the last block into a TRIMed cluster. Note that reads from the TRIMed + cluster will still return zeroes because the TRIM set will be consulted before + reading the physical disk. +4. Issue a `flush` +5. Rewrite the pointer to point to the TRIMed cluster. +6. Issue a `flush` +7. Issue a `ftruncate` + +Note the `flush` calls are necessary as write barriers to prevent the pointer +update from being persisted without the data copy over a crash. + +Note the `flush` is implemented by `fcntl(F_FULLFSYNC)` on OSX, as described +by the `fsync` man page or [this email](http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html). +On a modern MBP, a `flush` takes about 10ms. + +### The reverse cluster index + +A qcow2 file is a tree of 64 KiB clusters where the root of the tree is +in the header at the beginning of the file. All clusters are linked either +from the header, or from other clusters. In order to shuffle a block within +the file we need to find the reference to it, so we can update the pointer +with the new location. +Unfortunately the qcow2 format only stores pointers to child blocks in the tree, +not pointers to parents, so we need to create and maintain our own reverse +cluster index. + +A table mapping clusters to their parents would need to have one entry per +cluster i.e. 3 bytes per 64 KiB of data i.e. an extra 0.005% space overhead. + +The largest possible qcow2 image is [2**63-513 bytes long](https://rwmj.wordpress.com/2011/10/03/maximum-qcow2-disk-size/). Rounding +the size up to `2**63`, the required overhead would be about 300 TiB. + +The top-of-the-range Mac Pro can be configured with 1 TiB of storage. Assuming +hard drive capacity doubles every 2 years then by 2020 we could have a 4 TiB +qcow2. The required overhead would be about 192 MiB. + +Therefore, although it's possible to construct a .qcow2 whose reverse cluster +index would need to be stored on disk, for the next couple of years at least +for our use-case we could use an in-memory table. + +### Making the TRIM asynchronous + +The block shuffling described above will be slow since it calls `flush` twice. +The `TRIM` command is supposed to be fast, and tools like `mkfs` will issue +thousands of them back-to-back. If we delete a lot of files all at once then +`ext4` could generate a lot of small `TRIM`s which could be coalesced and +executed all at once. + +We could make the TRIM asynchronous by first marking the clusters as empty +(such that reads will return zeroes) in a persistent datastructure cached +in memory. The ideal +structure would have fast lookup (since it's on the read path) and would +automatically coalesce adjacent TRIMed regions together. +Martin Erwig's paper [Diets for Fat Sets](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.26.4659) +describes a structure with fast lookup and automatic coalesce, and there is an OCaml implementation +inside the [batteries library](https://ocaml-batteries-team.github.io/batteries-included/hdoc2/BatISet.html). +To make the changes persistent we could use a simple append-only log to store +updates, and occasionally flush the whole structure to disk. The +[mirage/shared-block-ring](https://github.com/mirage/shared-block-ring) library +is designed for this use-case and was used to speed up LVM metadata operations +in the xapi-project. + +## Making the shuffling faster + +After we have coalesced a large number of TRIMs together, we can shuffle the +blocks more efficiently by batching. In the description above, one block was +moved at a time. We can extend this to an arbitrary number of blocks, with +the caveat that they must be locked against writes for the duration of the +shuffle. We can therefore amortise the cost of the `flush` over many blocks. + +## Possible development phases + +### Phase 1 + +If we expose the TRIM capability and track the TRIMed blocks, we could create +an offline tool for geneating a small qcow2 (rather than shuffling blocks). +This would allow motivated users to get their space back. + +### Phase 2 + +We could run the offline tool ourselves between reboots. + +### Phase 3 + +In this phase we could shuffle the blocks around live. diff --git a/scripts/token/main.go b/scripts/token/main.go new file mode 100644 index 0000000000..9dce759b64 --- /dev/null +++ b/scripts/token/main.go @@ -0,0 +1,158 @@ +package main + +import ( + "bytes" + "encoding/base64" + "fmt" + "io/ioutil" + "log" + "os" + "os/exec" + "os/user" + "path" + "path/filepath" + "strings" + + "github.com/Sirupsen/logrus" + "golang.org/x/crypto/ssh" +) + +// read bytes from private key PEM storage +const privateKey string = `-----BEGIN RSA PRIVATE KEY----- +MIIJKAIBAAKCAgEAvyhLlw5cF0T8rApKQwc/qZteZH0tVFnDW31yugy0CSJAcdMI +z8nP3gwL+Hzc6DBhNjPoajRitXxNEfoO2EwPKVXsU3V5fNl5/xqW4E7kS2UjZGJH +OOJz/ko4MA+0u60BSzQZDC8xo/NoZ08t4PiE4DjyFFb95SBUQ5+HkYlgMFPiHVGy +KqRAdX0zUiFjZbXrw9fhwPWPrgx1gweYoXbgctHWiEEiUeATNOwsYCFtrLPR9U4w +wldEp1M1Ft2Sv6nFCgkvLpK3nvqoEg4RfMlRiTX1KT1eQUBFM/owzeA5FjLZ/vBh +2/OZnll5eaU5gWH6RvDibxVbZogQScl9TJBYUuKzdT5VvLraIO29P947zuenhJs6 +VkH1gptoKLZ+z6lpuFZmkd472AyLnrK7cmYBagcCm+zMfGyqjSBJL636PpF/NHK5 +6HpjQUhuXuKeKbKYvb4JukRRfzJK60LCbBLliJ/TEI/+Qxz5XtVVsQ2EC2uLqrte +HwBDLzAIwy8wMORTIy1QZK+bJVE2gtq5xy+PIuiF0jFJPfASl1L/WsKzz0iVIN3T +KmHGpWhJjHGkPyGluBRqqQSXJWqUA4YSeuFBccLn7rAEMrh8trhx74jwFELHgNJ8 +0lCD73L/+ysKOwHHrp4hnmpAU9OTZ8fDstnKDwL8oxNK7jNw6FZ7SLYiYNECAwEA +AQKCAgAK0MDWHx3ewyx4n2xsNnDHq96/WMXDzABdoM9o72cQTTvQNNx+xTBZo9zZ +hlcJXBNj+bPgrm4XTp0ds0Q7wLHq6M2iOzdQdQ2N/Xcj4dLQ4TwLZfQZp9ZgQNrE +/V6Ab91u39e69MCeQhtaHQS/gdAiz5YCyJk86YNbAB+vgFJM6bIVbpXiC8EJ5LeO +ogz5qD1aq1A+lqY2dsX/T+K23/77ABxfQTAr6b2sdOBd4AZQiywJ8Q6ZME7WGMMc +NBUlevmHA9YDkJGLESuJOfqLUOioFsF0P+ChWH2R30n7nAAe3WmwHvGqfDHo368b +ztleFKkQcnidSFwPhQtG4XuRZWlZzh/koHZgvLpiMcopO5XNBRpkaUGjaTu4c8W+ +MGSpWJOiWgOwZiZG9OB1vD5b9oW3t+iBlU7D7HJT6DD17C2UzjQ52Y9e4bC34y1U +SZ11+7s5vV28FAJ3u1azVBDXXd7+kCvf0Eb07GhARLs66AiBCi8qszP4Lwz1EaTm +1daAwdcCqMpBQa/UtormP5Aa35dpwFsp48go/SSxs/yLCR6tqNnO8IYU+yB5RGr+ +qAXeGi0XGW9H8GZ4KPIYgojoll1Jw6uoFxwkUIjSiSQRNe1OmTmmjXRUYQUReKIb +nsvjCVR0gQB1He2hkNQSlGsfXm/4XIQ2oTxgxtcZ/0PQ4SBzAQKCAQEA6abhPSrJ +t+o9AW2jIZNZ0h2+/Js40EiwKGNwy29b6GZsxcVtSMxFZvUncKKqIc0hvIrfKGXO +fP7AFGuCh2ChT/8vP1d6Y3CT0beWCeLF1VBzR3aZcRyLP99JJKz9D8sThYbyvxsT +7f5Cm7cyGU6CF7Neacw80FVtRcm/DXvhuApCmXE+cMWgutfsW8c3OKygh4OV6bWf +Pz+JGxNqkyu+l4DJ64KDuiRuEwxNU3eJki9Zdij/i/tAfNRsXrvYklec/LpYIFIh +Lmp2dM3nMYgnQQbrjTTsvU+izUd30bY96VupOQ89mPrLxUsWaES4jMM2Vs9HjOPc +dpsojV4SbQumOQKCAQEA0XDpCYVyYf8nS0kNSh5V9CnXCrwQ0Y/8LzUv5nHrzESK +98W0nLUWtaUyuNbEECuRFbSqallWuWNsN3ByhrzlEntSZvEskQnEHbTjbkoxPq7r +sxcIZw163UOuoT9y954n1eroyGlQpIs4DKPEmNdKutaUgkQMgd6PeW/dAQZs+XzC +xCLxgMLZew5ybNWsbgclVD4Bznoo9Otv7pWRWRz/4ORNZWcxNBk2uBYYhfl5FBMj +5eTHCs6rQHDI0aGIrPRQ+tOTm9Sg0ctXmizYbZnf2j6G+NyeyPScVwtX7e+dvH7z +EVQ77Gvz3UoEA7GHlkEy8XQdA0uXLT2HAm4d6xpPWQKCAQBMAoXzqB/HPORruob/ +PThTKmofMz/gQkVMXk0rYSa9C9UG4ZsTu6A4Rjh2Y/SE2n7HH0ZJlhT+hMFn4zGr +aLwRkiqEqKigANeVueuNe8BwDKPz85knOunx9WmODNimcqH/Jk+B7AUnvzdcANTD +ds7LdwaX1GFURPYvZdpJQKvFe1D/Kd/uP9xx7BxwHvbP8Rin/R6f0P3lTX4E2OQq +zGhMURFfFC5WN9O3TqE5LgILFGw+DEhV+X6ZHWHDz8g8k1P2w4g3u4Af4XJ3vSQg +8PIULXQjk7wQOf/0V/OavXaWm8MJVPPs+Gmh3TOE6BZBdKAQzY2xP89Qplki5B+K +aefBAoIBAFbhUaojc0l3gKNYUGz4nItGd+/6B7gG1IP4ukAL7da0cDlMCBohfKQp +PRsz6+0RRYQNh8vJ95G7zn5I5RlDbnr2MT6GuQgJVxNDoDx2BmuMQDXwTgoBq3/x +vZUiLtzM/JVeduX72foHzl5f6QPF+zf6H2zSMaYF3tpvLuxn7/imalzWafwR2AmV ++p1vHbIewLtrZXBzeF3w9GOyI7Mltndh/UEdR2nnM621bMLWtAVB01hgSLkQ9jUr +FALx0TJ9vsHt1oOD9ppQkaxhAf6lIBj2ayL80dlmrxvklrsa9QHmX4pGuPzf4y9e +rr+hey83KJzEn+xoBPQ9W64EY+DM7zECggEBAIw3lgc5Ii30Hfzgf12c3ymbDivl +Bk3bxgIl5uX26Qwu12VV9t8tqTILA5YAfskc8NwcioAMfjXJ4oHU3qjZeo5r2dBE +XNDmORu9Vd/If/l+IPktSSD4rDoJuqKikiXLTI9C07d2gg/yc1zxdBa+n7y6Yz4y +4WHULNI6+4eyAtf6cdHQAKpDLGPYOyldy+MLVvHJhbfwqt6zXvP/ltqg2JjW7qdK +ihKTk8elCHK0xCLlpSnM/J2pp2+ZQeJQIVR5MFoKyNeqTvezDQLapfoW2PJn8wiL +aBif8NFXuOOz0Ke2xFHNrzthFSd8PWzDY19IAUxzRr53uYohjRa1RJHRuc8= +-----END RSA PRIVATE KEY----- +` + +func main() { + var udid string + // Get UDID from CLI + if len(os.Args) > 1 { + udid = os.Args[1] + } else { + udid = detectUDID() + } + + logrus.Debugf("UDID found: ==>%s<===", udid) + signatureBase64, err := signUdid(udid) + if err != nil { + logrus.Error(err) + } + logrus.Debugf(signatureBase64) + logrus.Infof("Writing signature to %s/private-invites-signature.txt", appContainerPath()) + if err := writeSignatureInFile(signatureBase64); err != nil { + logrus.Error(err) + } +} + +// signUdid signs the user udid according to the server private key +func signUdid(udid string) (string, error) { + + // create a signer object from the private key bytes + signer, err := ssh.ParsePrivateKey([]byte(privateKey)) + // signer, err := ssh.ParsePrivateKey(privateKeyBytes) + if err != nil { + logrus.Error("unable to parse private key", err.Error()) + } + signature, err := signer.Sign(nil, []byte(udid)) + if err != nil { + logrus.Error("ERROR:", err.Error()) + return "", err + } + // encode signature into base64 + signatureBase64 := base64.StdEncoding.EncodeToString(signature.Blob) + return signatureBase64, nil +} + +func appContainerPath() string { + usr, err := user.Current() + if err != nil { + log.Fatal(err) + } + return filepath.Join(usr.HomeDir, "Library", "Containers", "com.docker.docker", "Data") +} + +// encode signature into base64 and write it in a file +func writeSignatureInFile(signatureBase64 string) error { + // get path to group container + containerPath := appContainerPath() + filepath := path.Join(containerPath, "private-invites-signature.txt") + err := ioutil.WriteFile(filepath, []byte(signatureBase64), os.ModePerm) + return err +} + +var udidPrefix = []byte("Hardware UUID:") + +func detectUDID() string { + cmd := exec.Command("/usr/sbin/system_profiler", "SPHardwareDataType") + lines := getLines(cmd) + for _, line := range lines { + line = bytes.TrimSpace(line) + if bytes.HasPrefix(line, udidPrefix) { + return string(bytes.TrimSpace(bytes.TrimPrefix(line, udidPrefix))) + } + } + logrus.Fatal("udid not found; is this a Mac OSX?") + return "" +} + +func getLines(cmd *exec.Cmd) [][]byte { + out := combinedOutput(cmd) + return bytes.Split(out, []byte("\n")) +} +func combinedOutput(cmd *exec.Cmd) []byte { + out, err := cmd.CombinedOutput() + if err != nil { + fmt.Println(strings.Join(cmd.Args, "\n")) + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + return out +} diff --git a/scripts/token/main_test.go b/scripts/token/main_test.go new file mode 100644 index 0000000000..83fd2f3d24 --- /dev/null +++ b/scripts/token/main_test.go @@ -0,0 +1,67 @@ +package main + +import ( + "encoding/base64" + "fmt" + "path/filepath" + "strings" + "testing" + + "golang.org/x/crypto/ssh" +) + +const publicKeyBase64 string = "AAAAB3NzaC1yc2EAAAADAQABAAACAQC/KEuXDlwXRPysCkpDBz+pm15kfS1UWcNbfXK6DLQJIkBx0wjPyc/eDAv4fNzoMGE2M+hqNGK1fE0R+g7YTA8pVexTdXl82Xn/GpbgTuRLZSNkYkc44nP+SjgwD7S7rQFLNBkMLzGj82hnTy3g+ITgOPIUVv3lIFRDn4eRiWAwU+IdUbIqpEB1fTNSIWNltevD1+HA9Y+uDHWDB5ihduBy0daIQSJR4BM07CxgIW2ss9H1TjDCV0SnUzUW3ZK/qcUKCS8ukree+qgSDhF8yVGJNfUpPV5BQEUz+jDN4DkWMtn+8GHb85meWXl5pTmBYfpG8OJvFVtmiBBJyX1MkFhS4rN1PlW8utog7b0/3jvO56eEmzpWQfWCm2gotn7PqWm4VmaR3jvYDIuesrtyZgFqBwKb7Mx8bKqNIEkvrfo+kX80crnoemNBSG5e4p4pspi9vgm6RFF/MkrrQsJsEuWIn9MQj/5DHPle1VWxDYQLa4uqu14fAEMvMAjDLzAw5FMjLVBkr5slUTaC2rnHL48i6IXSMUk98BKXUv9awrPPSJUg3dMqYcalaEmMcaQ/IaW4FGqpBJclapQDhhJ64UFxwufusAQyuHy2uHHviPAUQseA0nzSUIPvcv/7Kwo7AceuniGeakBT05Nnx8Oy2coPAvyjE0ruM3DoVntItiJg0Q==" + +var udidTests = []struct { + in string // input + expected string // expected result +}{ + {"test-udid-1", `pKBLBUgEGYpY/UpyaQls2AeFie6kqFRUXH1prvHOowI84fgcUtFrfkZnV/xiVkV95ffFiZ4kYcEWx7ffFF18FwIg47wUSXGnfp8Xt5zHMlo+XmgXd08kUlUdQEqGhPL4VfIEIogBfURYGF+JNsXfo9k33OdaHgKgBzVNN0ZNgvvhcG1D9ACnIY0t+0BXNeIZRdVT5CE6x37JTls8x279mSXl5+BEV2czDVFbG/TgYlcTqtLRqhChtwY//HhthjjfBSV893hT03h/32mJ8udynMKn8UlD0UWvIrmFx1Z+hHoEBN6N3ybdfVoOrEFdTDhP9IQX+Uz5AT/xozR0Nq0hXEA47ZtYopsTfECcgKjCOPe3DbLD+yuOeHq9cR2U+LazubCcW8krjZEymzshiPrFFfC3nHLsZK3nlS64qtxyi8mHumJLdAh8e1UkV5q564NWjabvqIfENLIFyBDMc737xS7yhGj23pV3sqQpXI1waJIq4e8wrvmZsHvtYaQvE5vN+UsiAr9pXudMbVs0baVUi5yxqWdkoeJE7VDRPrxDkvQ8c8ygKpSlaq+4W8rVSLrOJMBlbooEAZpb0LMvl2x0rBPry5vsgFJpYtyLWTJrD0mQkuPJFXUNq+joAonMfSEEggKb2czGvWenmPwBpU4/2rj11Rgnc2eJUi9cbhZn+Ag=`}, + {"test-udid-2", `u53OcTYRuaRwE9tv3uq6ovBCaHEBnlKtx5tXEtoKDq8ntYjA4TV4xXkHEL9vOdrOUzaQe6ul6KAuYi1HBH5jdYHIKMJnLyypgcKDupVyF759WR0mHVJmj/b1RQAHDfK8BViqUN4NXhUsxgpJvEMt1gAN67b9xpWIE9VXAOwXcNnQkm/5oV7fvv2y7cxkW79N1mfnbFmpYASZlCVL4kapP8tzJcOvRlEwnsIL7O5Rv3seSVUDH1HLo5+R0iZXrar0tAG+oOnvf/O4KnvgTP8JHpxJqy+Ddqvx9XI2ZoGLXweFGEDBVRxDE8LnGGr6e/b7VNxi71kdoT8T/NJBDNA8a7w1uTcJiOO3ru2JehYLDuL7MWkNpXu109PIVdo9Yu/shYoQ/3PxzOp4GqS5MP9Ez035f1YSwhvzDguU/PJ2v/RQoEc6Xk6dT7tX1S5qyH0UZVubYPgsB/yKI5RP2RynhRylG3VStB+8gY1DgO6YQGltSfzSRHBg45ts914QqrsvMgsl1m4NDZfy9NfKSzsP1k3NzR9el7gmF85guKtytq+1/nYJARQvOI8bNZCIHACRIHWEGrfTDDV7/hsgwIAIOwbwutUWfrJidUkwcc4RTzF2JH6Yb3gx2lUbh7Ook3nCTTKtTkntOXaJmDlo1QeCumA1PDO/TqsVhE1a7Y746XI=`}, + {"test-udid-3", `sh1ezj9GVrTJxlg4kE2xtCiMkiZbHEi1XVzpybSEJ4VqYullLXcCuvJlcplypELx8eG9YItVi8RJ6Xpk6TrMKyJwZIENopAZ3VfujfYZWf/8FPTXuwSsije2qi1WY01jy4+P+QX7aoJ5wrwVYK3asuCzPCXT73i+QOwlK2gMYtdJ9CZ+gWN4q5RZqDVJHcg9j3hvxBFEVFdxlWWScgKbGdgmS/E4fHxaZSOdLy01VC4FKE3goEIStOvOJa/t972/h+zZ5rgpc/on9Xxu4NL/Q70/QSgS4pfaiQdJO5ZkCXC72Sz3DcIM2XjDDQ5ZF6o9DewC9v832/pyGjG3teuUQDujGfJF3yzBtwu7qEQqXWXZiAy3zatTgU0aAz+sFceK+Zs5ryPSEsfh7gqIDBmvtv3Zzxv71BzsLQXlO2aKRo9qhAnyI6ZlByJeDDWNVFNR+GnwolBu8Ixqru702FIxISdRyWyRiOpGDBp+PmWyb9/WK0E59NapeWRo8GC/W1K7GHHVCn0vcwC0c2QYP6ZD8Ho2AAZ3EJlZLT/Skz+Dy35rA0n4680zlQf4vEWWSWDnNMGWYw809f5k99ymbCR7mN+nAiOjnzMDJtHDutm61BU+0q+KfsxH6NLGg/lsZj3IuCGtVUEiORLsGNeP6weBY9rJf1el3m6PuWeF2SniCfU=`}, + {"test-udid-4", `XNTNeLP1Sm4sqj6zgiwcKz5h8rYSQgfNH4R49KvuT5qMPkz0+iadyZFZGLvz1J7R0IsfSOqb2YGw6hddwm+T/buR/nj7laTRe40BbqU4yElJmjiZ7xL7SGAWVEiXOu6hCDQ7gmg2a7qvP/g0AFoLtonVpIQ345GSRJtxwwt6IRV6Uh1M2sGCN7wvgMX+r4hOVSYPnLyUOBpf7l3eY3g+5K8YriBYDNooDPFEYCHymUqYxhbXUNJ7pXSRLm8OkWKS8mLf7CcSl4hLIlwluK9VdOChHQzAaXwTIvx/VaxBNwPz1e1x1VyhPS+v+EmiXHA/nSvJYTZSGhJRy+Uo84xnl38/IH5gLm8qZao7r+SlyMojEepEb0WzVpbPva8hY1uWt2ADm0P8LfZRMt0qpEDlFItzuwsOyBIU8g6AMNQ08/DGof0s0zBGvF3y+cnV8u9M3qbQ8L8AXeCgznHAL92D46uAaGDuwxx7P+oDvzohIWwTYaWwPgZW1geUR2zQyiI2t7hvn0BTPCbFesCRXxly5t2RCR7BFpimfV/O8peB7oBgzIwg9PWSBBrnJ3hW6kXPT76sgWFigbtJ7kzi/9jHJz+qyX9rQwZia4V8toUQFwDuiMs9YrC0Xb90QV2nSyi6zGt5BjfjA736SVGllNUgvJ4UXr877HXpYc4MXhWaowY=`}, + {"test-udid-5", `qj6NIOFaS4ja+2HaVEUCe25RXlZFo1Rxp7GC+92aOHki6tWtrMiM1xaDC91LixTH8WejUfE6BZgdfEzq+zR545m/o2ClE9SUihIe3FkmywxdwKe+EPVDIsgZeaTlsSM1R4zqlpvdwhVerLA8Bvf9t0Sf06wEkaIiKMfgZsQSJ6/Rw6cWkXBgY4pmAnXferWTHp1Y8twiuRTFuCIMhEhinZYlJaPulF/U9r/g1pgGA/Bn7dab5YestJUePuHwxz1TFTb/3ZfXWSGzOANEcj49nVXFFi461FtLoCbZyIcbKDttnmlVBTNd/fpGbSx0HbiIsMYN2vQYjqvy7eWSQq1fru3cKPF1VzYntrzg3G4z5veHhgPR1ZwSKc6ExpQnW36JFQwNp3+YjsSYCq0LrPsPh7OREfSldbuwmZEnRM2L5BJqh4mPOjO9lUOT7nMWMRdyMQFWkEMMAgOdChnsBxMVuttGwyhKumh/3c6LBV/apq6JajjHV69WzZONX0VJIQwEOdMR53wmH/bLGwsqVh/ycEZqWWN1Cgl9SU9Ch+T1AQJf+zi8H0CIxXkdmAx+4tNa2Cu4zP1a/K2xtEY832TPUv5lgXExfndYS833q07t1iJa2P6HM5zB22qcikeQzYHESr2uTQjcYJvTnb3PODDXLbdrBbyBxfyEbbJWWyWg0lo=`}, + {"test-udid-6", `df8fsfJI9DaOMjxO8nPIc20ULi9xX5uctfTDPu7j+nA4tCeNRaV3Pe9P1pbklZCxMGqZkg9Ba2sp9tZuPYJLZnbiyDS9w9y/eTr75CKK2TI7vPGWAIffej9dLs2sqHtsZEbttYhVldhlr4pilRGmtlOgg/mhaZUc9Hr0/GE2x8/SDwDoHTIy4oAcf64/zIOTv/rQDpc/jguqTWbG/oU2RBDsKjSFGAZQRcCq8BcxJ1yVlNfPZNUaf4elqbPVsMhpn0xFNblFKinWEW++VabsvgitXwcvyX4W+QfZF1d00bVSk6a2ZamkdT4E74/FcOYlUtgGtxVGB2usji9m1Jyrn5FHcEa5iCEjSghfJOW9837c8TlDmjmDXaqZ5S2MSaeF1eT9YxECxu6SVYlSxA5Q9Aiw+xMM2vF6IJOAnU8BhJctsJosVedojGNeSStRfhEMK3dq83OXZt7UljgHgfew7WF6oXfZp22xHAQfh7M9dykT87eOYlD5KOw6yuh+R0HU6Ev5Et7syeYZgfRYNGJwSp27QISf9Bccuo4ppPZMlmkTxYkbEA/5bDN7gB1ESITiIWgIqakgizRNriPTI6ED+bF01XwI6rJv9/8pn3z4QKKW89NzIOE0Lbhn+6SzvQ0F4ynC7vkwKWLrNEA4kGfff5e4yn8NXCg8PXX2zqP7soc=`}, + {"test-udid-7", `UNexHoNiQQ+V+Ht2nwpzH6/43o3T0qe2V9Hf5H3HPZ9ss4bHIk3Y4FIsEoGQlMWjHsV29v4SjUtgBtdZV47VS09PPnFfJxFtav4BCwyecsk3sZ3Zqkdx/RAZG4YmFddp7g01ddAKh5Xi+rV83bLu0o9Mkgi0YdVLIcCw3sxJMvUVFhSdyFDUyqXeoWAMErU434fcGVfl2wCq0hdfEmKWjOAaxQudfYWk/3dA5ZRk3Gtg55gRam3YfgHvl80KvgG4d2zvpafyGnCqHh1P1iTabnvIMw23giP+9onOlnW2H6a09gXXTlZLWT5HYlwgd7FOOYUy//2fBcGxx2IoSJXbquisY8FhSm+qKIqjYS2KbVLc9jK5QJ8Eibk0NjZz2p31UAyJicCCa0K4R/vthCcKrQn9hDk9L3JcvnZ3/LyAmQh1wph8iYmCTTmuRrUmJqaw4fdNpKIN4mPhi4X4fzddK4BoGz04Gz/pM17HV3dU8SZXxVWZkD4xq/MqeBkgkfXdHpaw4phlKCtOEsx2W5fU5ibLz19gRQoVKF/xtjpdaUnRKtkDmrTzKB5MywTy/wIoabJiO88ZnqkaUUjKxTreDqMmrc+5nKClYD6+6T0BwQqX+7lz+M4FhQ9YqU/hdg/UKsLOqEqyc+OkBJz5wMhy1SBKz/pS8nB5TDeXPg5Khz8=`}, + {"test-udid-8", `KhtESQA2i6wXkVcQqRmwlQP2ZbvyoNVupXdnYUW+x6qc5As1+LhMhpLWJxt3YWAtwiMkxAqffedz/PjBtph6rykwErts1jsHa9p76+IPegt5hK98lHFZr+SdRWuzcmfLJXJsr50uwgj1m9/1pftCXd+6E/vHQVE/31LG109iHvYV3744w3BOYOXMN6wpe9XFHDOoNCQngaZaGbXYAGuMfWo5q5wPfzG35uJMvvkuOcHTlSBq/Ys3gmlP3fx1/xrykyUIhdMwI42Pl8SAqw7LHtZdKLCOHlZr5pB+RehMAq1fYcSOJkUaYg+wsU6jJKhuHhFXufwZ34nD8ABAf8rOYwq2DpmGhhT3hAUx7+wzRYLX2Ny+conYYxygET515JwTwB+HseesmeU/SDsjweSky7TcHjO0DIkwOld/EBzuvg1ndzNDX2wc1KxcnoiWIIvPPtzWkFO6c5mmvZB+30I4QF/7zkHwGq5U0e3xztcOaZ6L0yZBfasdk+V/xqXK/bRIkmxYadOmDndfiHvLOzprTShdHNsrOjGCMv6SmtLlqDceOmfedVAeqRCiSK+7XWhBs8pDRgtf7tUy7iR/yr94204sFVeLt3OFvUxE1xIs19brntU/IUseNiwAlmvGB4Dyws86Pp6mvKkutgpemdVy5cTsiiHjuY8PtpAE7mtzzPI=`}, + {"test-udid-9", `oOVa8cX0x8HxOCx6wnOz9fAdIRGHXoIgPZBvAQnRNbKgXmN5dbHwAtyyOEoKw+M49poW98OfQDgFfHc0zLV3Y7udWhxqTzXhEokfC3/t9bmmGvM9JP2L/wJs3fHEzEdLQpGWoAMGuRNHgHKYDZBfNghFABh9nPxpFudRERbQikOpcycCLY4ppkVDj8w6w94/nJQIZah/OsDvpmi8SXu45F6CHqvGAgTc0BRp+Zb3o8vhCqgftKnG6FXAt17P7hg73q9B2zmbZcipF5sahpxyUlv+jfihYjJfnmVj7qBaZC4R3+qzfcopMLNDM6Ak+L+vIOPzQciumjXLqHC9xs0c+fcszEvQFtUNac9t0dwE3DOTAokTwmEAZNLzFRyg666ahsN0tvtwfzGu7skosv7y12PzA/KGYCUY+Ju/0tVgh7pUYMFCrAfgYgmaQgjUp5m+TiHuXmXVP+32ilgKYKXY/i0NCrSlMKar0wTrmECdFwii0DOvqReKeI/ijrGT081Fsh3X2mxTAqKRlXZs/4//rm1W/0iFvS5Rs0gFy+5NNgQxbj2hIaC0Z6LdrgZQuEeaA6GC0gl6CtM9Rk72bHLXTUgSIE8o/kzzTtFazJEcQYT0ajgxWUeDXVM6GITUWBUG4ecMhlekUFXGaNiuzzKn9EyFfUpA9uHkBkhCj/rd46Y=`}, +} + +func TestSignUdid(t *testing.T) { + t.Log("Check if UDID are as expected") + for _, udidTest := range udidTests { + signatureBase64, _ := signUdid(udidTest.in) + if signatureBase64 != udidTest.expected { + t.Error("UDID not signed properly") + } + } +} + +func TestValidSignature(t *testing.T) { + t.Log("Check if Signed UDID are valid") + + // decode public key bytes + publicKeyBytes, _ := base64.StdEncoding.DecodeString(publicKeyBase64) + // parse public key bytes to get an ssh.PublicKey object + publicKey, _ := ssh.ParsePublicKey(publicKeyBytes) + + for _, udidTest := range udidTests { + signatureBase64, _ := signUdid(udidTest.in) + // reconstruct a ssh.Signature object + blob, _ := base64.StdEncoding.DecodeString(signatureBase64) + sig := ssh.Signature{Format: "ssh-rsa", Blob: blob} + // verify signature + if err := publicKey.Verify([]byte(udidTest.in), &sig); err != nil { + fmt.Printf("Error: %q\n", err) + t.Error("Signed UDID is not valid") + } + } +} + +func TestPath(t *testing.T) { + expected := filepath.Join("Library", "Containers", "com.docker.docker", "Data") + actual := appContainerPath() + if !strings.HasSuffix(actual, expected) { + t.Errorf("Path should end with %s got %s", expected, actual) + } +} diff --git a/support/README.md b/support/README.md new file mode 100644 index 0000000000..719b446923 --- /dev/null +++ b/support/README.md @@ -0,0 +1,5 @@ +Some support infrastructure to make log processing easier. + +- s3/ converts log files from the `docker-pinata` AWS S3 + bucket into easier to handle containers, organised by user + UUID. diff --git a/support/nurse/.gitignore b/support/nurse/.gitignore new file mode 100644 index 0000000000..c4da92d3a8 --- /dev/null +++ b/support/nurse/.gitignore @@ -0,0 +1,25 @@ +### https://raw.github.com/github/gitignore/master/ocaml.gitignore + +*.annot +*.cmo +*.cma +*.cmi +*.a +*.o +*.cmx +*.cmxs +*.cmxa + +# ocamlbuild working directory +_build/ + +# ocamlbuild targets +*.byte +*.native + +# oasis generated files +setup.data +setup.log + +# Binary +nurse diff --git a/support/nurse/.merlin b/support/nurse/.merlin new file mode 100644 index 0000000000..9c111485cf --- /dev/null +++ b/support/nurse/.merlin @@ -0,0 +1,17 @@ +PKG core +PKG cmdliner +PKG github +PKG github.unix +PKG lwt +PKG re +PKG logs +PKG fmt +PKG tar +PKG result +PKG astring +PKG mirage-block-unix +PKG mirage-block +PKG io-page + +S src +B _build/** diff --git a/support/nurse/Makefile b/support/nurse/Makefile new file mode 100644 index 0000000000..2dfdcbea73 --- /dev/null +++ b/support/nurse/Makefile @@ -0,0 +1,12 @@ +.PHONY: all clean nurse + +all: nurse + @ + +clean: + @ocamlbuild -clean + @rm -f nurse *~ + +nurse: + @ocamlbuild -use-ocamlfind 'nurse.native' + @mv nurse.native nurse diff --git a/support/nurse/TODO.md b/support/nurse/TODO.md new file mode 100644 index 0000000000..c95b5adaf5 --- /dev/null +++ b/support/nurse/TODO.md @@ -0,0 +1,75 @@ + +# Interesting strings to find in the logs + +Here are things we could find in the logs and highlight automatically, +with suggested tickets we could link them to. It's not guaranteed the +linked issue is a duplicate but it might still be useful as a reference. + +## transfused crash + +In docker-system.log: + +Transport endpoint is not connected + +see #5 + +## qcow2 corruption + +In docker-system.log: + +Invalid_argument("Cstruct.sub + +see #11 + +## /var/lib/docker corruption + +In /moby/var/log/docker.log: + +invalid checksum digest format +invalid mount id value +Error starting daemon: + +see #20 + +## failure to talk to vmnetd + +In docker-system.log: + +Communication with networking components faile + +see #61 + +## too many hyperkits + +In ps-ax, more than one "... hyperkit -A -m ..." + +see #71 + +## still has virtualbox + +In docker-system.log: + +Docker does not rely on Virtualbox but may not work properly on systems with VirtualBox versions prior to v4.3.30 + VirtualBox v4.3.28 is currently installed. + Please upgrade or uninstall Virtualbox. + +see #78 + +## moby failed to boot + +Check the console, look for the login prompt. + +see #84 + +## Moby kernel problems + +In docker-system.log: "BUG" or "rcu_sched self-detected stall" + +see #87 + +## Invariants + +In docker-system.log: "INVARIANT VIOLATED" + +see #89 + diff --git a/support/nurse/_tags b/support/nurse/_tags new file mode 100644 index 0000000000..980e5e3100 --- /dev/null +++ b/support/nurse/_tags @@ -0,0 +1,5 @@ +true: debug, strict_sequence, principal +true: warn(@5@8@10@11@12@14@23@24@26@27@29) + +"src": include + diff --git a/support/nurse/build.sh b/support/nurse/build.sh new file mode 100755 index 0000000000..66ac649f42 --- /dev/null +++ b/support/nurse/build.sh @@ -0,0 +1,4 @@ +#!/bin/sh -ex + +eval `opam config env` +make diff --git a/support/nurse/opam b/support/nurse/opam new file mode 100644 index 0000000000..b55770823c --- /dev/null +++ b/support/nurse/opam @@ -0,0 +1,18 @@ +opam-version: "1.2" +name: "nurse" +version: "0.0.0" +maintainer: "pinata@docker.com" +authors: ["Dave Tucker" "David Scott"] +depends: [ + "ocamlfind" {build} + "lwt" + "cmdliner" + "re" + "github" + "result" + "mirage-block" + "mirage-block-unix" + "tar-format" + "logs" + "fmt" +] diff --git a/support/nurse/src/_tags b/support/nurse/src/_tags new file mode 100644 index 0000000000..5a995e39fe --- /dev/null +++ b/support/nurse/src/_tags @@ -0,0 +1,6 @@ +true: \ + package(cmdliner github.unix re logs.fmt logs.cli fmt.tty fmt.cli) +true: \ + package(tar.lwt tar.mirage result astring mirage-block-unix mirage-block) +true: \ + package(io-page.unix, re.perl) diff --git a/support/nurse/src/archive.ml b/support/nurse/src/archive.ml new file mode 100644 index 0000000000..c6c1384987 --- /dev/null +++ b/support/nurse/src/archive.ml @@ -0,0 +1,141 @@ +open Lwt.Infix +open Astring + +module KV_RO = Tar_mirage.Make_KV_RO(Block) + +type t = { + block: Block.t; + kv_ro: KV_RO.t; + diagnostic_id: string; + timestamp: string; +} + +module Symptom = struct + type symptom = { + problem: Problem.t; + archive: t; + contexts: string list; + } + + let to_markdown { problem; archive; contexts } = + let quote s = + let lines = String.cuts ~sep:"\n" s in + String.concat ~sep:"\n" @@ List.map (fun x -> "> " ^ x) lines in + Printf.sprintf "Detected symptom of problem '%s' in %s/%s.\n\n%s\n\nMay be related to %s\n\nThe following log matches:\n %s" + problem.Problem.label archive.diagnostic_id archive.timestamp + problem.Problem.description + (String.concat ~sep:" " @@ List.map (fun (repo, no) -> repo ^ "#" ^ (string_of_int no)) problem.Problem.link_to_issues) + (String.concat ~sep:"\nand\n" @@ List.map quote contexts) +end + +let get_diagnostic_id t = t.diagnostic_id +let get_timestamp t = t.timestamp + +type 'a error = ('a, [ `Msg of string]) Result.result + +let openarchive filename = + (* Add a `list` function to `Tar_mirage.Make_KV_RO`? *) + Lwt_unix.openfile filename [ Lwt_unix.O_RDONLY ] 0 + >>= fun fd -> + Lwt.finalize + (fun () -> + Tar_lwt_unix.Archive.list fd + ) (fun () -> + Lwt_unix.close fd + ) + >>= fun headers -> + + Block.connect filename + >>= function + | `Error e -> + Lwt.return (Result.Error (`Msg (Mirage_block.Error.string_of_error e))) + | `Ok block -> + Lwt.catch + (fun () -> + KV_RO.connect block + >>= function + | `Ok kv_ro -> + match headers with + | [] -> + Lwt.return (Result.Error (`Msg (Printf.sprintf "Archive %s has no files" filename))) + | first :: _rest -> + begin match String.cuts ~sep:"/" first.Tar_lwt_unix.Header.file_name with + | diagnostic_id :: timestamp :: _ -> + Lwt.return (Result.Ok { block; kv_ro; diagnostic_id; timestamp }) + | _ -> + Lwt.return (Result.Error (`Msg (Printf.sprintf "Archive %s file %s does not obey diagnostic_id/timestamp convention" filename first.Tar_lwt_unix.Header.file_name))) + end + ) (fun e -> + Lwt.return (Result.Error (`Msg (Printf.sprintf "Unexpected exception %s while processing %s" (Printexc.to_string e) filename))) + ) + >>= function + | Result.Error x -> + Block.disconnect block + >>= fun () -> + Lwt.return (Result.Error x) + | Result.Ok x -> + Lwt.return (Result.Ok x) + +let close t = + Block.disconnect t.block + +let analyse t = + Lwt_list.map_s + (fun problem -> + let key = t.diagnostic_id ^ "/" ^ t.timestamp ^ "/" ^ problem.Problem.in_file in + KV_RO.size t.kv_ro key + >>= function + | `Error _ -> + Logs.err (fun f -> f "Failed to find %s" key); + Lwt.return [] + | `Ok size -> + KV_RO.read t.kv_ro key 0 (Int64.to_int size) + >>= function + | `Error _ -> assert false + | `Ok buffers -> + let s = String.concat ~sep:"" (List.map Cstruct.to_string buffers) in + (* print_endline s; *) + try + let re = Re_perl.(compile @@ re ~opts:[`Multiline] problem.Problem.regexp) in + let groups = Re.exec re s in + (* At least one instance of the problem detected. Compute a context + substring for each one and then merge together (to avoid redundancy) *) + let rec loop i = match Re.Group.start groups i, Re.Group.stop groups i with + | start, stop -> (start, stop) :: (loop (i + 1)) + | exception Not_found -> [] in + let idxs = List.rev @@ loop 0 in + (* Aim to include at least [context_size] characters surrounding the match *) + let context_size = 1000 in + (* Snap to newlines and the edges of the string *) + let rec snap_to_newline delta from = + if from < 0 then 0 + else if from >= (String.length s) then String.length s - 1 + else if s.[from] = '\n' then from + else snap_to_newline delta (from + delta) in + let contexts = List.map (fun (start, stop) -> + snap_to_newline (-1) (start - context_size / 2), + snap_to_newline (+1) (stop + context_size / 2) + ) idxs in + (* Merge overlapping contexts to avoid redundant printing. *) + let merge = function + | [] -> [] + | at_least_one -> + let sorted = List.sort (fun a b -> compare (fst a) (fst b)) at_least_one in + let rec loop (a_start, a_end) = function + | [] -> [ a_start, a_end ] + | (b_start, b_end) :: next when b_start <= a_end -> loop (a_start, b_end) next + | b :: next -> (a_start, a_end) :: (loop b next) in + loop (List.hd sorted) (List.tl sorted) in + let contexts = merge contexts in + let contexts = List.map + (fun (context_start, context_end) -> + let context_length = context_end - context_start in + String.with_range ~first:context_start ~len:context_length s + ) contexts in + Lwt.return [ { Symptom.problem; archive = t; contexts } ] + with + | Not_found -> Lwt.return [] (* problem not detected *) + | Re_perl.Parse_error -> failwith (Printf.sprintf "Failed to parse perl-style regexp: %s" problem.Problem.regexp) + ) Problem.compiled_in + >>= fun x -> + Lwt.return (List.concat x) diff --git a/support/nurse/src/archive.mli b/support/nurse/src/archive.mli new file mode 100644 index 0000000000..87eb7e73dc --- /dev/null +++ b/support/nurse/src/archive.mli @@ -0,0 +1,32 @@ + +type t +(** A diagnostic archive *) + +val get_diagnostic_id: t -> string +(** The unique ID which represents the software installation *) + +val get_timestamp: t -> string +(** The time the diagnostics were generated *) + +type 'a error = ('a, [ `Msg of string]) Result.result + +val openarchive: string -> t error Lwt.t +(** [openarchive filename] opens the archive and prepares it for queries. *) + +val close: t -> unit Lwt.t +(** [close t] closes the archive, releasing associated resources. *) + +module Symptom: sig + type symptom = { + problem: Problem.t; (** the problem we found *) + archive: t; (** the archive we found it in *) + contexts: string list; (** the log file context(s) *) + } + (** Symptom(s) of a specific problem in a specific archive *) + + val to_markdown: symptom -> string + (** Render a report as markdown *) +end + +val analyse: t -> Symptom.symptom list Lwt.t +(** [analyse t] looks for symptoms of known problems in the archive. *) diff --git a/support/nurse/src/cache.ml b/support/nurse/src/cache.ml new file mode 100644 index 0000000000..a5e377fa5e --- /dev/null +++ b/support/nurse/src/cache.ml @@ -0,0 +1,107 @@ +open Lwt.Infix + +let pinata_fetch = "pinata:fetch" (* image used to fetch from S3 *) + +type 'a error = ('a, [ `Msg of string ]) Result.result + +type diagnostic_id = string + +type timestamp = string + +let (/) = Filename.concat + +let home = + try + Unix.getenv "HOME" + with Not_found -> + Logs.err (fun f -> f "Please set the HOME environment variable"); + exit 1 + +let default_cache_dir = + let dir = home / ".nurse" in + (try Unix.mkdir dir 0o0700 with Unix.Unix_error(Unix.EEXIST, _, _) -> ()); + dir + +let errorf fmt = + Printf.ksprintf (fun s -> + Result.Error (`Msg s) + ) fmt + +let path_of diagnostic_id timestamp = + default_cache_dir / diagnostic_id / timestamp / "decompressed.tar" + +let fetch diagnostic_id = + (* Check AWS credentials have been setup *) + let credentials_path = home / ".aws" / "credentials" in + if not(Sys.file_exists credentials_path) + then Lwt.return (errorf "Failed to find AWS credentials in %s. Please follow the instructions in the README.md" credentials_path) + else begin + Docker.inspect pinata_fetch + >>= function + | Result.Error (`Msg m) -> Lwt.return (Result.Error (`Msg m)) + | Result.Error (`Exit _n) -> Lwt.return (errorf "Failed to find image %s. Please build it using the instructions in the README.md" pinata_fetch) + | Result.Ok _ -> + let log_volume = default_cache_dir ^ ":/logs" in + let cred_volume = credentials_path ^ ":/root/.aws/credentials" in + Docker.run [ "-it"; "--rm"; "-v"; log_volume; "-v"; cred_volume; pinata_fetch; "mac"; diagnostic_id ] + >>= function + | Result.Error (`Msg m) -> Lwt.return (Result.Error (`Msg m)) + | Result.Error (`Exit _n) -> Lwt.return (errorf "Failed to fetch diagnostic ID %s. Please check for typos. Perhaps the upload never completed?" diagnostic_id) + | Result.Ok _ -> + (* Find and decompress *) + let timestamps = Array.to_list @@ Sys.readdir (default_cache_dir / diagnostic_id) in + Lwt_list.iter_s + (fun timestamp -> + let dir = default_cache_dir / diagnostic_id / timestamp in + if Sys.is_directory dir then begin + (* Freshly downloaded files are compressed, so fix the name *) + ( if Sys.file_exists (dir / "report.tar") then begin + Lwt_unix.rename (dir / "report.tar") (dir / "decompressed.tar.gz") + end else Lwt.return () ) + >>= fun () -> + (* Properly-named compressed files should be decompressed *) + ( if Sys.file_exists (dir / "decompressed.tar.gz") then begin + Command.run "/usr/bin/gunzip" [ dir / "decompressed.tar.gz" ] + >>= function + | Result.Error (`Exit 1) -> + (* For some reason the archives have trailing garbage. Is + this caused by an S3 upload block size? We consider it a + success if the decompressed file is created. *) + if Sys.file_exists (dir / "decompressed.tar") + then Lwt.return () + else begin + Logs.err (fun f -> f "Failed to gunzip %s" (dir / "decompressed.tar.gz")); + Lwt.return () + end + | Result.Error (`Exit _n) -> + Logs.err (fun f -> f "Failed to gunzip %s" (dir / "decompressed.tar.gz")); + Lwt.return () + | Result.Error (`Msg m) -> + Logs.err (fun f -> f "%s" m); + Lwt.return () + | Result.Ok _ -> + Lwt.return () + end else Lwt.return () ) + end else Lwt.return () + ) timestamps + >>= fun () -> + Lwt.return (Result.Ok ()) + end + +type path = string + +let get diagnostic_id timestamp = + let path = path_of diagnostic_id timestamp in + if Sys.file_exists path + then Lwt.return (Result.Ok path) + else begin + Logs.info (fun f -> f "Archive not found in cache, asking S3"); + fetch diagnostic_id + >>= function + | Result.Error (`Msg m) -> + Lwt.return (Result.Error (`Msg m)) + | Result.Ok () -> + if Sys.file_exists path + then Lwt.return (Result.Ok path) + else Lwt.return (errorf "Failed to find archive %s/%s on S3" diagnostic_id timestamp) + end diff --git a/support/nurse/src/cache.mli b/support/nurse/src/cache.mli new file mode 100644 index 0000000000..6c8a1aa8b6 --- /dev/null +++ b/support/nurse/src/cache.mli @@ -0,0 +1,12 @@ + + +type 'a error = ('a, [ `Msg of string ]) Result.result + +type diagnostic_id = string + +type timestamp = string + +type path = string + +val get: diagnostic_id -> timestamp -> path error Lwt.t +(** Return a local path of a decompressed archive *) diff --git a/support/nurse/src/command.ml b/support/nurse/src/command.ml new file mode 100644 index 0000000000..acb3a9a653 --- /dev/null +++ b/support/nurse/src/command.ml @@ -0,0 +1,23 @@ +open Lwt.Infix + +let errorf fmt = + Printf.ksprintf (fun s -> + Result.Error (`Msg s) + ) fmt + +let run path args = + let args = Filename.basename path :: args in + Logs.info (fun f -> f "%s" (String.concat " " args)); + Lwt_process.with_process_in (path, Array.of_list args) + (fun p -> + Lwt_io.read p#stdout + >>= fun output -> + p#status + >>= function + | Unix.WEXITED 0 -> + Lwt.return (Result.Ok output) + | Unix.WEXITED n -> + Lwt.return (Result.Error (`Exit n)) + | Unix.WSIGNALED n | Unix.WSTOPPED n -> + Lwt.return (errorf "Caught signal %d while running %s" n path) + ) diff --git a/support/nurse/src/command.mli b/support/nurse/src/command.mli new file mode 100644 index 0000000000..f970746704 --- /dev/null +++ b/support/nurse/src/command.mli @@ -0,0 +1,3 @@ + +val run: string -> string list -> (string, [ `Exit of int | `Msg of string ]) Result.result Lwt.t +(** [run path args] runs a process and returns stdout *) diff --git a/support/nurse/src/docker.ml b/support/nurse/src/docker.ml new file mode 100644 index 0000000000..71a41ca9ba --- /dev/null +++ b/support/nurse/src/docker.ml @@ -0,0 +1,7 @@ +open Lwt.Infix + +let docker = "/usr/local/bin/docker" + +let inspect name = Command.run docker [ "inspect"; name ] + +let run args = Command.run docker ("run" :: args) diff --git a/support/nurse/src/docker.mli b/support/nurse/src/docker.mli new file mode 100644 index 0000000000..52306195fe --- /dev/null +++ b/support/nurse/src/docker.mli @@ -0,0 +1,7 @@ + + +val inspect: string -> (string, [ `Exit of int | `Msg of string ]) Result.result Lwt.t +(** Same as the output of `docker inspect` *) + +val run: string list -> (string, [ `Exit of int | `Msg of string ]) Result.result Lwt.t +(** Same as the output of `docker run ` *) diff --git a/support/nurse/src/labels.ml b/support/nurse/src/labels.ml new file mode 100644 index 0000000000..dfcc791050 --- /dev/null +++ b/support/nurse/src/labels.ml @@ -0,0 +1,139 @@ +module ForMac = struct + type t = + | Stable1_12_0 + | Stable1_12_0_a + | Stable1_12_1 + | Beta21 + | Beta22 + | Beta23 + | Beta24 + | Beta25 + | Beta26 + | Beta26_1 + + let to_string = function + | Stable1_12_0 -> "version/1.12.0" + | Stable1_12_0_a -> "version/1.12.0-a" + | Stable1_12_1 -> "version/1.12.1" + | Beta21 -> "version/beta21" + | Beta22 -> "version/beta22" + | Beta23 -> "version/beta23" + | Beta24 -> "version/beta24" + | Beta25 -> "version/beta25" + | Beta26 -> "version/beta26" + | Beta26_1 -> "version/beta26.1" + + let of_string = function + | "1.12.0" -> Some Stable1_12_0 + | "1.12.0-a" -> Some Stable1_12_0_a + | "1.12.1" -> Some Stable1_12_1 + | "1.12.0-beta21" -> Some Beta21 + | "1.12.0-beta22" -> Some Beta22 + | "1.12.1-rc1-beta23" -> Some Beta23 + | "1.12.1-beta24" -> Some Beta24 + | "1.12.1-beta25" -> Some Beta25 + | "1.12.1-beta26" -> Some Beta26 + | "1.12.1-beta26.1" -> Some Beta26_1 + | _ -> None +end + +module Osx = struct + type t = + | Yosemite + | ElCapitan + | Sierra + + let to_string = function + | Yosemite -> "osx/10.10.x" + | ElCapitan -> "osx/10.11.x" + | Sierra -> "osx/10.12 (beta)" + + let of_string = function + | "10.10" -> Some Yosemite + | "10.11" -> Some ElCapitan + | "10.12" -> Some Sierra + | _ -> None +end + +module Status = struct + (* type t should be in Stage order (see Stage module below) *) + type t = + | NeedsTriage + | MoreInfoNeeded + | WontFix + | Acknowledged + | InProgress + | Fixed + | ReleasedBeta + | ReleasedStable + + module Set = Set.Make(struct + type nonrec t = t + + let compare = compare + end) + + module Stage = struct + type t = + | TriageStage + | WontFixStage + | AcknowledgedStage + | InProgressStage + | FixedStage + | ReleasedStage + + let of_status = function + | NeedsTriage | MoreInfoNeeded -> TriageStage + | WontFix -> WontFixStage + | Acknowledged -> AcknowledgedStage + | InProgress -> InProgressStage + | Fixed -> FixedStage + | ReleasedBeta | ReleasedStable -> ReleasedStage + + let to_string = function + | TriageStage -> "triage" + | WontFixStage -> "wontfix" + | AcknowledgedStage -> "acknowledged" + | InProgressStage -> "inprogress" + | FixedStage -> "fixed" + | ReleasedStage -> "released" + end + + let to_string = function + | NeedsTriage -> "status/0-triage" + | WontFix -> "status/0-wont-fix" + | MoreInfoNeeded -> "status/0-more-info-needed" + | Acknowledged -> "status/1-acknowledged" + | InProgress -> "status/2-in-progress" + | Fixed -> "status/3-fixed" + | ReleasedBeta -> "status/4-fix-released-beta" + | ReleasedStable -> "status/4-fix-released-stable" + + let of_string = function + | "status/0-triage" -> Some NeedsTriage + | "status/0-wont-fix" -> Some WontFix + | "status/0-more-info-needed" -> Some MoreInfoNeeded + | "status/1-acknowledged" -> Some Acknowledged + | "status/2-in-progress" -> Some InProgress + | "status/3-fixed" -> Some Fixed + | "status/4-fix-released-beta" -> Some ReleasedBeta + | "status/4-fix-released-stable" -> Some ReleasedStable + | _ -> None +end + +type t = { for_mac : ForMac.t option; osx : Osx.t option; status : Status.t } + +let to_string_list l = + let result = [Status.to_string l.status] in + let result = match l.for_mac with + | None -> result + | Some x -> result@ [ForMac.to_string x] + in + match l.osx with + | None -> result + | Some x -> result@ [Osx.to_string x] + +let to_string l = + let list_of_l = to_string_list l in + String.concat ", " list_of_l + diff --git a/support/nurse/src/nurse.ml b/support/nurse/src/nurse.ml new file mode 100644 index 0000000000..c578615152 --- /dev/null +++ b/support/nurse/src/nurse.ml @@ -0,0 +1,352 @@ +open Labels + +(* Triage *) + +let get_token t = + match t with + | None -> Github.Token.of_string "" + | Some t -> Github.Token.of_string t + +let version_number = Re.(seq [ + rep digit; + str "."; + rep digit; + group( seq [ + opt (str "."); + opt (rep digit); + ]; + ); + ]) + +let for_mac_version_re = + Re.(compile(seq [ + str "Docker for Mac: "; + opt (str "version: "); + group ( seq [ + version_number; + opt (str "-"); + opt (rep alnum); + opt (str "-"); + opt (rep alnum); + ]); + ])) + +let get_for_mac_label body = + let for_mac_matches = Re.exec_opt for_mac_version_re body in + match for_mac_matches with + | None -> None + | Some for_mac_matches -> begin + match + try Some (Re.get for_mac_matches 1) + with _ -> None + with + | None -> None; + | Some s -> ForMac.of_string s + end + +let osx_version_re = + Re.(compile(seq [ + alt [ + str "macOS: Version "; + str "OS X: version "; + ]; + group ( + version_number; + ); + ])) + +let get_osx_label body = + let osx_matches = Re.exec_opt osx_version_re body in + match osx_matches with + | None -> None + | Some osx_matches -> begin + match + try Some (Re.get osx_matches 1) + with _ -> None + with + | None -> None; + | Some version -> begin + (* Check if there is a patch version, if yes, strip it *) + match + try Some (Re.get osx_matches 2) + with _ -> None + with + | None -> Osx.of_string version + | Some patch -> begin + let stop = (String.length version) - (String.length patch) in + Osx.of_string @@ String.sub version 0 stop + end + end + end + +let should_skip l = + let lst = List.map (fun l -> + let name = Labels.Status.of_string l.Github_t.label_name in + match name with + | None -> false + | Some _status -> true + ) l in + List.fold_left (||) false lst + +module IntMap = Map.Make(struct type t = int let compare = compare end) + +let triage_issue acc issue = + let open Github_t in + (* Check for existing status labels *) + let labels = issue.issue_labels in + if should_skip labels + then Github.Monad.return acc + else + (* Check reported version *) + let for_mac_label = get_for_mac_label issue.issue_body in + let osx_label = get_osx_label issue.issue_body in + let computed_labels = { + for_mac = for_mac_label; + osx = osx_label; + (* Initial status is always NeedsTriage *) + status = Labels.Status.NeedsTriage; + } in + Logs.app (fun m -> + m "Issue %d: %s" issue.issue_number (to_string computed_labels)); + Github.Monad.return (IntMap.add issue.issue_number computed_labels acc) + +module Prompt = struct + type yes_no = [ + | `Yes + | `No + ] + + type response = [ + | yes_no + | `Other of string + ] + + let yes_no_of_input ~default = function + | "y" | "yes" -> Some `Yes + | "n" | "no" -> Some `No + | "" -> Some default + | _other -> None + + let rec yes_no ?(default=`Yes) s = + let options = match default with + | `Yes -> " [Y/n]" + | `No -> " [y/N]" + in + Printf.printf "%s%s %!" s options; + let r = String.(trim (lowercase (input_line stdin))) in + match yes_no_of_input ~default r with + | Some yes_no -> yes_no + | None -> + Printf.printf "I didn't understand that. Please enter 'y' or 'n'.\n%!"; + yes_no ~default s + +end + +let triage () token user repo = + let t = Github.(Monad.(run ( + API.set_token (get_token token) + >>= fun () -> + let issues = Issue.for_repo ~user ~repo () in + Stream.fold triage_issue IntMap.empty issues + >>= fun labels_to_apply -> + if IntMap.is_empty labels_to_apply + then embed Lwt.return_unit + else + match Prompt.yes_no "Would you like to apply these labels?" with + | `No -> embed Lwt.return_unit + | `Yes -> + IntMap.fold (fun num l m -> + m >>= fun () -> + let labels = Labels.to_string_list l in + Github.Issue.add_labels ~user ~repo ~num ~labels () + >>~ fun _labels -> embed Lwt.return_unit + ) labels_to_apply (embed Lwt.return_unit) + ))) in + Lwt_main.run t + +let clean () token user repo = + let t = Github.(Monad.(run ( + API.set_token (get_token token) + >>= fun () -> + let issues = Issue.for_repo ~user ~repo () in + Stream.fold Nurse_clean.compute_actions [] issues + >>= List.fold_left Nurse_clean.(fun m action -> + let open Github_t in + m >>= fun () -> + let issue = action.issue in + Printf.printf "Issue %d: %s\n%s\n" + issue.issue_number issue.issue_title issue.issue_html_url; + List.iter (fun change -> + Printf.printf "%s\n%!" (Nurse_clean.string_of_change change) + ) action.changes; + match Prompt.yes_no "Perform these changes?" with + | `No -> Printf.printf "Not this time\n%!"; return () + | `Yes -> clean_issue ~user ~repo action + ) (embed Lwt.return_unit) + ))) in + Lwt_main.run t + +let notify () token user repo label filename = + let t = + let open Lwt.Infix in + Lwt_unix.openfile filename [ Lwt_unix.O_RDONLY ] 0 + >>= fun fd -> + let ic = + Lwt_io.of_fd ~close:(fun () -> Lwt_unix.close fd) ~mode:Lwt_io.input fd + in + Lwt_io.read ic + >>= fun body -> + Lwt_io.close ic + >>= fun () -> + Printf.printf + "For issues with label '%s' I will append the following comment:\n%!" + label; + Printf.printf "\n%s\n%!" body; + + Github.(Monad.(run ( + let open Github_t in + API.set_token (get_token token) + >>= fun () -> + Stream.iter (fun issue -> + (* Check for existing status labels *) + let labels = + List.map (fun l -> l.Github_t.label_name) issue.issue_labels + in + if List.mem label labels then begin + Printf.printf "Issue %d: %s\n%s\n" + issue.issue_number issue.issue_title issue.issue_html_url; + match Prompt.yes_no "Append comment?" with + | `Yes -> + Printf.printf "OK, commenting\n%!"; + Issue.create_comment ~user ~repo ~num:issue.issue_number ~body () + >>~ fun comment -> + Printf.printf "%s\n" comment.issue_comment_html_url; + return () + | `No -> + Printf.printf "Not this time\n%!"; + return () + end else return () + ) (Issue.for_repo ~state:`All ~user ~repo ()) + ))) in + Lwt_main.run t + +let analyse () diagnostic_id timestamp = + let open Lwt.Infix in + let t = + Cache.get diagnostic_id timestamp + >>= function + | Result.Error (`Msg m) -> + Logs.err (fun f -> f "Failed to download archive from S3: %s" m); + Lwt.return () + | Result.Ok filename -> + Archive.openarchive filename + >>= function + | Result.Ok t -> + Logs.app (fun f -> f "Analysing report with diagnostic ID %s and timestamp %s" + diagnostic_id timestamp + ); + Archive.analyse t + >>= fun symptoms -> + Lwt_list.iter_s + (fun symptom -> + let md = Archive.Symptom.to_markdown symptom in + Lwt_io.write Lwt_io.stdout md + ) symptoms + >>= fun () -> + Archive.close t + | Result.Error (`Msg m) -> + Logs.err (fun f -> f "Failed to open archive: %s" m); + Lwt.return () in + Lwt_main.run t + +(* Command line interface *) + +open Cmdliner + +(* Logging *) + +let setup_log style_renderer level = + Fmt_tty.setup_std_outputs ?style_renderer (); + Logs.set_level level; + Logs.set_reporter (Logs_fmt.reporter ()); + () + +let setup_log = + Term.(const setup_log $ Fmt_cli.style_renderer () $ Logs_cli.level ()) + +(* Commands *) + +let github_token = + let doc = "Github API Token." in + Arg.(value & opt (some string) None & info ["token"] ~docv:"TOKEN" ~doc) + +let github_owner = + let doc = "Github Repository Owner." in + Arg.(required & pos 0 (some string) None & info [] ~docv:"OWNER" ~doc) + +let github_repo = + let doc = "Github Repository Name." in + Arg.(required & pos 1 (some string) None & info [] ~docv:"REPO" ~doc) + +let triage_cmd = + let doc = "Triage bugs" in + let man = [`S "DESCRIPTION"; `P "Triages bugs";] in + Term.(const triage $ setup_log $ github_token $ github_owner $ github_repo), + Term.info "triage" ~doc ~man + +let notify_cmd = + let label = + let doc = "Issue label" in + Arg.(required & pos 2 (some string) None & info [] ~docv:"LABEL" ~doc) in + let filename = + let doc = "File containing comment to add" in + Arg.(required & pos 3 (some file) None & info [] ~docv:"FILENAME" ~doc) in + let doc = "Append a comment to all issues with a specific tag" in + let man = [ + `S "DESCRIPTION"; + `P "Given a label and a filename, iterate over every issue (open and closed)\ + containing the tag and append the file's contents as a new comment."; + `P "You will br prompted to confirm every comment."; + ] in + Term.(const notify $ setup_log $ github_token $ github_owner $ github_repo $ label $ filename), + Term.info "notify" ~doc ~man + +let analyse_cmd = + let diagnostic_id = + let doc = "Diagnostic ID recorded in the tracker" in + Arg.(value & pos 0 string "F4466F9B-150B-4082-A6F5-6BA8FA0F085B" & info [] ~docv:"ID" ~doc) in + let timestamp = + let doc = "Timestamp of the report" in + Arg.(value & pos 1 string "20160802-140735" & info [] ~docv:"TIMESTAMP" ~doc) in + let doc = "Look for well-known errors in diagnostic tarballs" in + let man = [ + `S "DESCRIPTION"; + `P "This is a temporary command which should be merged with triage later" + ] in + Term.(const analyse $ setup_log $ diagnostic_id $ timestamp), + Term.info "analyse" ~doc ~man + +let clean_cmd = + let doc = "Clean bugs" in + let man = [`S "DESCRIPTION"; `P "Fixes labels and closes stale issues";] in + Term.(const clean $ setup_log $ github_token $ github_owner $ github_repo), + Term.info "clean" ~doc ~man + +let default_cmd = + let doc = "assists in the triaging and care of bugs" in + let man = [ `S "BUGS"; `P "Open an issue on GitHub";] in + Term.(ret (const (fun _ -> `Help (`Plain, None)) $ const ())), + Term.info "nurse" ~version:"0.0.1" ~doc ~man + +let cmds = [ + triage_cmd; + analyse_cmd; + notify_cmd; + clean_cmd; +] + +(* Entrypoint *) + +let () = match Term.eval_choice default_cmd cmds with + | `Error _ -> exit 1 + | _ -> exit (if Logs.err_count () > 0 then 1 else 0) diff --git a/support/nurse/src/nurse_clean.ml b/support/nurse/src/nurse_clean.ml new file mode 100644 index 0000000000..262a8e2437 --- /dev/null +++ b/support/nurse/src/nurse_clean.ml @@ -0,0 +1,94 @@ +type change = + | Close of string + | Remove_labels of string * string list + +type action = { + issue : Github_t.issue; + changes : change list; +} + +let string_of_change = function + | Close descr -> "Close because "^descr + | Remove_labels (descr, labels) -> + Printf.sprintf "Remove labels [%s] because %s" + (String.concat ", " labels) descr + +(* TODO: Offer to close issues (with a note) that are very old or + otherwise outdated. *) +let compute_actions list issue = + let open Github_t in + let open Labels in + + let statuses = List.fold_left (fun set label -> + match Status.of_string label.label_name with + | Some Status.MoreInfoNeeded (* MoreInfoNeeded is always allowed *) + | None -> set + | Some status -> Status.Set.add status set + ) Status.Set.empty issue.issue_labels in + + (* Compute status labels to remove due to later stage *) + let latest_status = + try Status.Set.max_elt statuses with Not_found -> Status.NeedsTriage + in + let latest_stage = Status.Stage.of_status latest_status in + let _to_keep, to_remove = Status.Set.partition (fun status -> + latest_stage = Status.Stage.of_status status + ) statuses in + let changes = + if Status.Set.is_empty to_remove then [] + else + let message = Printf.sprintf "issue is at stage %s" + (Status.Stage.to_string latest_stage) + in + let labels = + List.map Status.to_string (Status.Set.elements to_remove) + in + [ Remove_labels (message, labels) ] + in + + (* Decide whether to close issue due to completion *) + let changes = + if Status.(Set.mem ReleasedBeta) statuses + && Status.(Set.mem ReleasedStable) statuses + then + let message = "a fix has been released on beta and stable channels" in + (Close message)::changes + else changes + in + + (* Decide whether to close issue due to WontFix *) + let changes = + if Status.(Set.mem WontFix) statuses + then (Close "the issue has been marked WontFix")::changes + else changes + in + + Github.Monad.return (match changes with + | [] -> list + | changes -> { issue; changes; }::list + ) + +let clean_issue ~user ~repo action = + let open Github.Monad in + let num = action.issue.Github_t.issue_number in + List.fold_left (fun m -> function + | Close _ -> + m >>= fun () -> + let issue = Github_t.{ + update_issue_title = None; + update_issue_body = None; + update_issue_state = Some `Closed; + update_issue_assignee = None; + update_issue_milestone = None; + update_issue_labels = None; + } in + Github.Issue.update ~user ~repo ~num ~issue () + >>~ fun _updated_issue -> + return () + | Remove_labels (_, labels) -> List.fold_left (fun m name -> + m >>= fun () -> + Github.Issue.remove_label ~user ~repo ~num ~name () + >>~ fun _labels -> + return () + ) m labels + ) (return ()) action.changes diff --git a/support/nurse/src/problem.ml b/support/nurse/src/problem.ml new file mode 100644 index 0000000000..127112cb7a --- /dev/null +++ b/support/nurse/src/problem.ml @@ -0,0 +1,117 @@ +open Sexplib.Std + +type t = { + in_file: string; + regexp: string; + label: string; + description: string; + link_to_issues: (string * int) list; + link_from_issues: (string * int) list; +} [@@deriving sexp] + +let for_mac = "docker/for-mac" + +let transfused_crash = { + in_file = "docker-system.log"; + regexp = "Transport endpoint is not connected"; + label = "transfused-crash"; + description = "This error suggests that a component of the volume sharing system failed."; + link_to_issues = [ for_mac, 5 ]; + link_from_issues = []; +} + +let qcow2_corruption = { + in_file = "docker-system.log"; + regexp = "Invalid_argument\\(\"Cstruct.sub"; + label = "qcow2-corruption"; + description = "This error suggests that the .qcow2 file is corrupt and will need to be recreated."; + link_to_issues = [ for_mac, 11 ]; + link_from_issues = []; +} + +let var_lib_docker_corruption1 = { + in_file = "moby/var/log/docker.log"; + regexp = "invalid checksum digest format"; + label = "aufs-corruption"; + description = "The daemon cannot start because some files in /var/lib/docker are corrupt. \ + This can happen if the filesystem is unmounted uncleanly. The workaround is to reset to \ + factory defaults which will unfortunately delete /var/lib/docker: all containers and \ + images will need to be rebuilt."; + link_to_issues = [ for_mac, 20 ]; + link_from_issues = []; +} + +let var_lib_docker_corruption2 = { var_lib_docker_corruption1 with regexp = "invalid mount id value" } + +let docker_didnt_start = { + in_file = "moby/var/log/docker.log"; + regexp = "Error starting daemon:"; + label = "daemon-didnt-start"; + description = "The docker daemon inside the VM failed to start."; + link_to_issues = []; + link_from_issues = []; +} + +let communication_with_networking = { + in_file = "docker-system.log"; + regexp = "Communication with networking components failed"; + label = "vmnetd-comm-failure"; + description = "The UI failed to communicate with the networking helper process."; + link_to_issues = [ for_mac, 61 ]; + link_from_issues = []; +} + +let too_many_hyperkits = { + in_file = "ps-ax.log"; + regexp = "hyperkit -A -m(.|\n)+hyperkit -A -m"; + label = "more-than-one-hyperkit"; + description = "More than one hyperkit process has started and is accessing the \ + qcow2 disk at the same time. This will lead to corruption and Docker must be \ + reset to factory defaults."; + link_to_issues = [ for_mac, 71 ]; + link_from_issues = []; +} + +let virtualbox = { + in_file = "docker-system.log"; + regexp = "Please upgrade or uninstall Virtualbox"; + label = "uninstall-virtualbox"; + description = "Some parts of Virtualbox may still be installed. Please follow \ + https://www.virtualbox.org/manual/ch02.html#idm871 to completely uninstall \ + Virtualbox. Please note that dragging the application to the trash is not enough:\ + there are kernel modules installed as well."; + link_to_issues = [ for_mac, 78 ]; + link_from_issues = []; +} + +let moby_kernel = { + in_file = "docker-system.log"; + regexp = "rcu_sched self-detected stall"; + label = "moby-kernel-froze"; + description = "The kernel inside the VM stalled so Docker must be restarted."; + link_to_issues = [ for_mac, 87 ]; + link_from_issues = []; +} + +let invariants = { + in_file = "docker-system.log"; + regexp = "INVARIANT VIOLATED"; + label = "invariant-violated"; + description = "An invariant within the code of one of the components was \ + not held. Docker needs to be restarted."; + link_to_issues = [ for_mac, 89 ]; + link_from_issues = []; +} + +let compiled_in = [ + transfused_crash; + qcow2_corruption; + var_lib_docker_corruption1; + var_lib_docker_corruption2; + docker_didnt_start; + communication_with_networking; + too_many_hyperkits; + virtualbox; + moby_kernel; + invariants; +] diff --git a/support/nurse/src/problem.mli b/support/nurse/src/problem.mli new file mode 100644 index 0000000000..aeda683c80 --- /dev/null +++ b/support/nurse/src/problem.mli @@ -0,0 +1,13 @@ + +type t = { + in_file: string; (** Name of file within the diagnostic archive *) + regexp: string; (** If this regexp matches, the problem probably exists *) + label: string; (** A short label (e.g. "qcow2-corruption") *) + description: string; (** Description suitable for posting as a github comment *) + link_to_issues: (string * int) list; (** (repo, issue) we should link this to *) + link_from_issues: (string * int) list; (** (repo, issue) we should link to this *) +} [@@deriving sexp] +(** A well-known problem *) + +val compiled_in: t list +(** Problems compiled into this binary (as opposed to downloaded) *) diff --git a/support/s3/.dockerignore b/support/s3/.dockerignore new file mode 100644 index 0000000000..428006820e --- /dev/null +++ b/support/s3/.dockerignore @@ -0,0 +1,2 @@ +data/ +html/ diff --git a/support/s3/.gitignore b/support/s3/.gitignore new file mode 100644 index 0000000000..5a86f45b31 --- /dev/null +++ b/support/s3/.gitignore @@ -0,0 +1,4 @@ +data/ +html/ +.s3cfg +s3creds.env diff --git a/support/s3/Dockerfile.awscli b/support/s3/Dockerfile.awscli new file mode 100644 index 0000000000..e1e043db53 --- /dev/null +++ b/support/s3/Dockerfile.awscli @@ -0,0 +1,10 @@ +FROM alpine:latest +MAINTAINER Anil Madhavapeddy +RUN \ + mkdir -p /aws && \ + apk -Uuv add groff less python py-pip && \ + pip install awscli && \ + apk --purge -v del py-pip && \ + rm /var/cache/apk/* +WORKDIR /aws +ENTRYPOINT ["aws"] diff --git a/support/s3/Dockerfile.fetch b/support/s3/Dockerfile.fetch new file mode 100644 index 0000000000..0a1266e63a --- /dev/null +++ b/support/s3/Dockerfile.fetch @@ -0,0 +1,10 @@ +FROM python:2-alpine +MAINTAINER Dave Tucker + +RUN pip install awscli +ADD fetch.sh /usr/local/bin/fetch + +VOLUME /logs +VOLUME /root/.aws + +ENTRYPOINT ["fetch"] diff --git a/support/s3/Dockerfile.tohtml b/support/s3/Dockerfile.tohtml new file mode 100644 index 0000000000..18c0e93efc --- /dev/null +++ b/support/s3/Dockerfile.tohtml @@ -0,0 +1,4 @@ +FROM alpine +RUN apk update && apk add py-pygments py-pip bash tar coreutils +VOLUME /logs +VOLUME /html diff --git a/support/s3/README.md b/support/s3/README.md new file mode 100644 index 0000000000..869b031472 --- /dev/null +++ b/support/s3/README.md @@ -0,0 +1,63 @@ +Log formatter from S3 +===================== + +- `sync.sh` will populate the `data/` directory with the latest support logs +- `tohtml.sh` will process them into HTML in the `html/` directory. + +TODO: + +- classify log files so they can be processed with different containers (E.g. fuse) +- better index page +- add file sizes +- add infra for grep scripts to do autotriage +- plumb through a docker push to our private repo with the results + + +## Requesting Access To AWS + +To get the necessary account, please ask @dave-tucker, @balrajs, @ijc25 or anyone else who has admin access on AWS... + +- Your account needs to be created +- You need to be added to the `pinata` group. +- The `docker-pinata-support` bucket policy needs to have your username added to the whitelist + +## Using AWS + +You will be given a CSV with your username, password and login URL +Please login and change your password. You may also take this opportunity to set up 2FA. + +Once logged in, follow [these instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) to create an Access Key. This will let you use the AWS API and CLI tools. + +To create the `~/.aws/credentials` file, your best bet is to first install the AWS CLI. + +``` +# brew install pip if you don't have it already. +pip install awscli +``` + +Then you can `aws configure` and follow the instructions. + +If you have more than one set of AWS credentials, you may want to consider creating a different "profile" for each one. In this case: + +``` +aws configure --profile pinata +``` + +In order for your file to be bind-mountable as a `~/.aws/credentials` file, you may want to maintain a copy at `~/.aws/pinata-credentials` with your Pinata credentials as the `default` profile. + +# Fetch + +Fetch is a tool that fetches the logs for a provided platform/uuid and optionally a report +It connects to S3 and outputs the logs to the mounted `/logs` directory + +Example Usage: + +```bash +$ docker build -t pinata:fetch -f Dockerfile.fetch . + +$ docker run -it --rm \ + -v /my/logs/dir:/logs \ + -v /home/me/.aws/credentials:/root/.aws/credentials \ + pinata:fetch $platform $uuid $report +``` + diff --git a/support/s3/fetch.sh b/support/s3/fetch.sh new file mode 100755 index 0000000000..a9c0955d79 --- /dev/null +++ b/support/s3/fetch.sh @@ -0,0 +1,43 @@ +#!/bin/sh + +set -e + +# $1 is the platform (mac|win) +# $2 is the UUID from the bug report +# $3 is the optional report ID + +path="" +dst="" +bucket="docker-pinata-support" + +if [ -z "${1}" ]; then + echo "No platform provided" + exit 1 +fi + +case "${1}" in + win) + path="incoming/w1" + ;; + mac) + path="incoming/2" + ;; + *) + echo "${1} is not a valid platform" + exit 1 +esac + +if [ -z "${2}" ]; then + echo "No UUID provided" + exit 1 +fi + +path="${path}/${2}" +dst="/logs/${2}" + +if [ -n "${3}" ]; then + path="${path}/${3}" + dst="${dst}/${3}" +fi + +aws s3 cp "s3://${bucket}/${path}" "${dst}" --recursive --quiet diff --git a/support/s3/formatter/process-all-users.sh b/support/s3/formatter/process-all-users.sh new file mode 100755 index 0000000000..19530c1377 --- /dev/null +++ b/support/s3/formatter/process-all-users.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +set -ex + +IN=$1 +OUT=$2 + +if [ ! -d "$1" ]; then + echo INDIR not found + echo Usage: $0 INDIR OUTDIR + exit 1 +fi + +if [ ! -d "$2" ]; then + echo OUTDIR not found + echo Usage: $0 INDIR OUTDIR + exit 1 +fi + +mkdir -p ${OUT} +cat >${OUT}/index.html < + +

                                  Docker.App support logs

                                  +

                                  Upload By UUID

                                  +

                                    +Header-fragment + +cd ${IN} +for uuid in *; do + bash /formatter/process-one-user.sh ${IN}/$uuid ${OUT}/$uuid $uuid + echo "
                                  • $uuid
                                    • " >> ${OUT}/index.html +done + +cat >>${OUT}/index.html <

                                    + + +Footer-fragment diff --git a/support/s3/formatter/process-one-upload.sh b/support/s3/formatter/process-one-upload.sh new file mode 100755 index 0000000000..6b2359fd54 --- /dev/null +++ b/support/s3/formatter/process-one-upload.sh @@ -0,0 +1,66 @@ +#!/bin/sh -e + +IN=$1 +OUT=$2 + +if [ ! -d "$1" ]; then + echo Usage: $0 INDIR OUTDIR + exit 1 +fi + +if [ -z "$2" ]; then + echo Usage: $0 INDIR OUTDIR + exit 1 +fi + +d=$3 +dt=`echo $d | awk -F- '{print $1}'` +tm=`echo $d | awk -F- '{print $2}' | fold -w2 | paste -sd':' -` + +rm -rf ${OUT}/*.html +mkdir -p ${OUT} +cd ${OUT} + +tar --strip-components=2 -xf ${IN}/report.tar + +cat >index.html < + +

                                    User `cat ${OUT}/user-id`

                                    +

                                    back to date index

                                    +

                                    +Date:           $dt $tm
+Version:        `cat ${OUT}/version`
+`cat ${OUT}/sw_vers`
+

                                    +

                                    Logs

                                    +

                                      +Header-fragment + +FILES=`find . -type f` +echo ${FILES} +for i in `find . -type f`; do + file=`echo $i | sed -e 's,^\./,,g'` + case $i in + *DS_Store) + echo Skipping DS_Store + ;; + *.swp) + echo Ignoring dot files + ;; + *.html) + echo Skipping $i as HTML already + ;; + *) + echo Processing $i + pygmentize -f html -O full,encoding=latin1 -l text -o $i.html $i + echo "
                                    • $file
                                      • " >> index.html + ;; + esac +done + +cat >>index.html <

                                      + + +Footer-fragment diff --git a/support/s3/formatter/process-one-user.sh b/support/s3/formatter/process-one-user.sh new file mode 100755 index 0000000000..e02f0d9dba --- /dev/null +++ b/support/s3/formatter/process-one-user.sh @@ -0,0 +1,49 @@ +#!/bin/sh -e + +IN=$1 +OUT=$2 + +if [ ! -d "$1" ]; then + echo INDIR not found + echo Usage: $0 INDIR OUTDIR UUID + exit 1 +fi + +if [ -z "$2" ]; then + echo OUTDIR not specified + echo Usage: $0 INDIR OUTDIR UUID + exit 1 +fi + +if [ -z "$3" ]; then + echo UUID not specified + echo Usage: $0 INDIR OUTDIR UUID + exit 1 +fi + +user=$3 + +mkdir -p ${OUT} +cat >${OUT}/index.html < + +

                                      Docker.App logs for user $user

                                      +

                                      Uploads by Date

                                      +

                                        +Header-fragment + +cd ${IN} +for d in 20*; do + /formatter/process-one-upload.sh ${IN}/$d ${OUT}/$d $d + echo $d + dt=`echo $d | awk -F- '{print $1}'` + tm=`echo $d | awk -F- '{print $2}' | fold -w2 | paste -sd':' -` + echo "
                                      • $dt $tm
                                        • " >> ${OUT}/index.html +done + +cat >>${OUT}/index.html <

                                        + + +Footer-fragment + diff --git a/support/s3/s3creds.env.in b/support/s3/s3creds.env.in new file mode 100644 index 0000000000..57c51ca4a5 --- /dev/null +++ b/support/s3/s3creds.env.in @@ -0,0 +1,2 @@ +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= diff --git a/support/s3/sync.sh b/support/s3/sync.sh new file mode 100755 index 0000000000..e752fa409a --- /dev/null +++ b/support/s3/sync.sh @@ -0,0 +1,13 @@ +#!/bin/sh -ex + +VERSION=2 + +docker build -t aws -f Dockerfile.awscli . + +if [ ! -e "s3creds.env" ]; then + echo Need to initialise S3 config from s3creds.env.in + exit 1 +fi + +docker run -v `pwd`/data:/mnt --env-file s3creds.env \ + aws s3 sync --size-only s3://docker-pinata-support/incoming /mnt diff --git a/support/s3/tohtml.sh b/support/s3/tohtml.sh new file mode 100755 index 0000000000..215ea9156f --- /dev/null +++ b/support/s3/tohtml.sh @@ -0,0 +1,19 @@ +#!/bin/sh -ex + +docker build -q -f Dockerfile.tohtml -t tohtml . +docker run -v `pwd`/html:/html tohtml sh -c "rm -rf /html/*" + +VERSION=2 +ARGS="$ARGS -v `pwd`/formatter:/formatter" +ARGS="$ARGS -v `pwd`/html:/html" + +if [ -z "$1" ]; then + echo Processing all users + docker run $ARGS \ + -v `pwd`/data/$VERSION:/logs \ + -it tohtml /formatter/process-all-users.sh /logs /html +else + docker run $ARGS \ + -v `pwd`/data/$VERSION/$1:/logs \ + -it tohtml /formatter/process-one-user.sh /logs /html $1 +fi diff --git a/tests/.gitignore b/tests/.gitignore new file mode 100644 index 0000000000..9a48fb5167 --- /dev/null +++ b/tests/.gitignore @@ -0,0 +1,3 @@ +*.pyc +/_results +vm.json diff --git a/tests/LICENSE b/tests/LICENSE new file mode 100644 index 0000000000..616602c2f0 --- /dev/null +++ b/tests/LICENSE @@ -0,0 +1,176 @@ +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000000..1942a76745 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,451 @@ +# Regression tests + +This directory contains both a generic, cross-platform regression test +framework and regression tests specific to the Docker for Mac and +Windows products. + +The generic regression test framework is implemented in Python with +the majority of the code contained in the `./lib` sub-directory. This +directory should *not* contain any Docker for Mac and Windows specific +code. + +The Docker for Mac and Windows specific code and test cases are +implemented in the `./cases` sub-directory. + + +# Prerequisites + +On a Mac, this should pretty much just work out of the box. However, +for development you might want to consider installing `flake8`, a +python style checker. +``` +brew install python +pip install flake8 +``` + +On Windows, you need to have python installed as well as some form of +`bash`. The tests must be run from a `bash` shell and `bash.exe` must +be in your path. The simplest way is to install via +[chocolatey](https://chocolatey.org/): + +``` +choco install git python +``` + +See Section on Machine setup below for more details. + + +# Quickstart + +The regression test framework allows running tests on a local host (or +inside a VM) as well as against a suitably configured remote host. + +To run tests locally, simply execute the `rt-local run` command. It +will executed all the test cases in the `./cases` sub-directory. + +To list all current tests run `rt-local list`, or to get a one line +summary for each test use `rt-local info`. + +When running tests, by default a line per test is printed on the +console with a pass/fail indication. Detailed logs, by default, are +stored in `./_results//`. In that directory, `TESTS.log` +contains detailed logs of all tests, `TESTS.csv` contains a line per +test and `SUMMARY.csv` contains a one line summary of the all tests +run. The directory also contains a log file for each tests, with the +same contents as `TESTS.log`. + +If you prefer a bit more information in the log files use: +``` +rt-local -x run +``` +This executes the tests with `-x` and thus logs all commands executed. + +For a CI system, where the output is displayed on a web page use: +``` +rt-local -x -vvv run +``` +This prints the same information logged to the log file to the console. + + +To run test on a remote host use the `rt-host`. You must specify the type of the remote host (`osx` or `win`) as well as the address. Optionally, you can supply user name and password. +``` +./rt-host -t win -r 172.16.10.131 -- +``` +This: +- copies the test framework and tests from the local host to the remote host. +- executes the test on remote host +- copies the results back to the local host + + +# User's guide + +Tests are located in the top-level project directory (default +`./cases`). Within a project tests may be grouped by placing them in a +common sub-directory (forming a _Test Group_). The order in which +tests and test groups are executed is determined by alphabetical order +and one may prefix a directory with a number to force a order. Each +test is given a unique name based on the place in the directory +hierarchy where it is located. For the naming of tests, any +prefix-numbers, used to control the order of execution, are +stripped. For example, a test located in: +``` +cases/backend/compose/010_django_example/ +``` +will be named: +``` +backend.compose.django_example +``` + +While traversing the directory tree, directories starting with `_` are +ignored. + + +## Run control + +Apart from the simple command line examples given in the Quickstart +section above, the regression test framework provides fine grained +control over which tests are run. The primary tool for this are +_labels_. + +A test may define that a specific label "foobar" must be present for +it to be run or, by prefixing the label name with a '!', that the test +should not be run if a label is defined. The labels are defined with +`test.sh` (see below). + +The regression test framework is completely agnostic to which labels +are used for a given set of test cases, though it defines a number of +labels based on the OS it is executing on. See the output of `rt-local +list` on your host. + +A good strategy for using labels is to not define any labels for tests +which should always be executed (e.g. on every CI run). Then use +labels to control execution for tests which are specific for a given +platform (e.g. `osx` or `win` for OS X and Windows installer tests). +If a particualr test is known not to run on a given platform you can +use, e.g. `!win`, to indicate it. Finally, define separate labels, +e.g., long running tests or extensive tests which should only be run +on release candidates. + +You can control labels for executing tests by using the `-l` flag. +For example, if you have some long running tests which you do not wish +to execute on every CI run and have them marked with a label `long`, +then you can execute them with: +``` +./rt-local -l long run +``` + +You can see which tests would get executed using the `-l` flag for the +`list` command as well: +``` +./rt-local -l long list +``` + +In addition to control which tests are run via labels it is also +possible to specify an individual test or a group name on the command +line to just run these test (subject to labels). Here are two +examples: +``` +./rt-local run pinata.backend.volumes.hardlink +./rt-local run pinata.backend.volumes +``` +The first runs a single test, while the second is running all tests +within the volumes group. Note, that this is currently implemented as +a simple prefix match, so, if you have tests such as `foo.bar` and +`foo.bar_baz` and use `./rt-local run foo.bar`, it will execute both +`foo.bar` and `foo.bar_baz`. + + +## Writing tests + +Tests are simple scripts which return `0` on success and a non-zero +code on failure. A special return code (`253` or `RT_TEST_CANCEL`) +can be use to indicate that the test was cancelled (for whatever +reason). Each test must be located in its own sub-directory (together +with any files it may require). + +Currently, test is a simple shell script called `test.sh`. In the +future we will also support Python and Powershell (for Windows only +tests). + +There is a template `test.sh` in `./etc/templates/test.sh` which can be +used for writing tests. It contains a number of special comments +(`SUMMARY`, `LABELS`, `REPEAT`, and, `AUTHOR`) which are used by the +regression test framework. The `SUMMARY` line should contain a *short* +summary of what the test does. The `LABEL` is a (optional) list of +labels to control when a test should be executed. `AUTHOR` should +contain name and email address of the test author. If a test is from +multiple authors, multiple `AUTHOR` lines can be used. Finally, the +`REPEAT` line may contain a single number to indicate that a test +should be executed multiple times. The `REPEAT` line may also contain +`