Merge pull request #412 from mstanleyjones/troubleshooting_volume_errors.md

Add topic about troubleshooting volume problems

_data/toc.yaml

@@ -185,6 +185,10 @@ toc:
       title: Runtime metrics
     - path: /engine/admin/ambassador_pattern_linking/
       title: Link via an ambassador container
+  - sectiontitle: Troubleshoot Docker Engine
+    section:
+    - path: /engine/admin/troubleshooting_volume_errors.md
+      title: Troubleshoot volume problems
   - sectiontitle: Manage a swarm
     section:
     - path: /engine/swarm/

engine/admin/troubleshooting_volume_errors.md

@@ -0,0 +1,62 @@
+---
+description: Troubleshooting volume errors
+keywords:
+- cadvisor, troubleshooting, volumes, bind-mounts
+title: Troubleshoot volume errors
+---
+
+# Troubleshoot volume errors
+
+This topic discusses errors which may occur when you use Docker volumes or bind
+mounts.
+
+## `Error: Unable to remove filesystem`
+
+Some container-based utilities, such
+as [Google cAdvisor](https://github.com/google/cadvisor), mount Docker system
+directories, such as `/var/lib/docker/`, into a container.  For instance, the
+documentation for `cadvisor` instructs you to run the `cadvisor` container as
+follows:
+
+
+```bash
+$ sudo docker run \
+  --volume=/:/rootfs:ro \
+  --volume=/var/run:/var/run:rw \
+  --volume=/sys:/sys:ro \
+  --volume=/var/lib/docker/:/var/lib/docker:ro \
+  --publish=8080:8080 \
+  --detach=true \
+  --name=cadvisor \
+  google/cadvisor:latest
+```
+
+When you bind-mount `/var/lib/docker/`, this effectively mounts all resources of
+all other running containers as filesystems within the container which mounts
+`/var/lib/docker/`. When you attempt to remove any of these containers, the
+removal attempt may fail with an error like the following:
+
+```none
+Error: Unable to remove filesystem for
+74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515:
+remove /var/lib/docker/containers/74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515/shm:
+Device or resource busy
+```
+
+The problem occurs if the container which bind-mounts `/var/lib/docker/`
+uses `statfs` or `fstatfs` on filesystem handles within `/var/lib/docker/`
+and does not close them.
+
+Typically, we would advise against bind-mounting `/var/lib/docker` in this way.
+However, `cAdvisor` requires this bind-mount for core functionality.
+
+If you are unsure which process is causing the path mentioned in the error to
+be busy and preventing it from being removed, you can use the `lsof` command
+to find its process. For instance, for the error above:
+
+```bash
+$ sudo lsof /var/lib/docker/containers/74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515/shm
+```
+
+To work around this problem, stop the container which bind-mounts
+`/var/lib/docker` and try again to remove the other container.