From d400d938e6f1fc07a455ff6a9eb3c9216f166c52 Mon Sep 17 00:00:00 2001 From: Usha Mandya Date: Tue, 13 Oct 2020 11:30:29 +0100 Subject: [PATCH] Remove osxfs topics from Desktop docs Signed-off-by: Usha Mandya --- _data/toc.yaml | 4 - docker-for-mac/index.md | 2 + docker-for-mac/osxfs-caching.md | 237 -------------------- docker-for-mac/osxfs.md | 369 -------------------------------- 4 files changed, 2 insertions(+), 610 deletions(-) delete mode 100644 docker-for-mac/osxfs-caching.md delete mode 100644 docker-for-mac/osxfs.md diff --git a/_data/toc.yaml b/_data/toc.yaml index b65f83edd7..580fc48550 100644 --- a/_data/toc.yaml +++ b/_data/toc.yaml @@ -1160,10 +1160,6 @@ manuals: title: Leveraging Multi-CPU architecture support - path: /docker-for-mac/networking/ title: Networking - - path: /docker-for-mac/osxfs/ - title: File system sharing - - path: /docker-for-mac/osxfs-caching/ - title: Performance tuning for volume mounts (shared filesystems) - path: /docker-for-mac/space/ title: Disk utilization - path: /docker-for-mac/troubleshoot/ diff --git a/docker-for-mac/index.md b/docker-for-mac/index.md index 617351ed2d..e31e2a6d97 100644 --- a/docker-for-mac/index.md +++ b/docker-for-mac/index.md @@ -12,6 +12,8 @@ redirect_from: - /mac/started/ - /mackit/ - /mackit/getting-started/ +- /docker-for-mac/osxfs/ +- /docker-for-mac/osxfs-caching/ title: Docker Desktop for Mac user manual toc_min: 1 toc_max: 2 diff --git a/docker-for-mac/osxfs-caching.md b/docker-for-mac/osxfs-caching.md deleted file mode 100644 index bdae01909f..0000000000 --- a/docker-for-mac/osxfs-caching.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -description: Osxfs caching -keywords: mac, osxfs, volumes -title: Performance tuning for volume mounts (shared filesystems) ---- - -[Docker 17.04 CE Edge](https://github.com/docker/docker.github.io/blob/v17.03/edge/index.md#docker-ce-edge-new-features) adds support -for two new flags to the [docker run `-v`, `--volume`](../engine/reference/run.md#volume-shared-filesystems) -option, `cached` and `delegated`, that can significantly improve the performance -of mounted volume access on Docker Desktop for Mac. These options begin to solve some of -the challenges discussed in -[Performance issues, solutions, and roadmap](osxfs.md#performance-issues-solutions-and-roadmap). - -> **Tip:** Release notes for Docker CE Edge 17.04 are -[here](https://github.com/moby/moby/releases/tag/v17.04.0-ce), and the -associated pull request for the additional `docker run -v` flags is -[here](https://github.com/moby/moby/pull/31047). - -The following topics describe the challenges of bind-mounted volumes on `osxfs`, -and the caching options provided to optimize performance. - -This blog post on [Docker on Mac -Performance](https://stories.amazee.io/docker-on-mac-performance-docker-machine-vs-docker-for-mac-4c64c0afdf99) -gives a nice, quick summary. - -For information on how to configure these options in a Compose file, see -[Caching options for volume mounts](../compose/compose-file/index.md#caching-options-for-volume-mounts-docker-desktop-for-mac) -the Docker Compose topics. - -## Performance implications of host-container file system consistency - -With Docker distributions now available for an increasing number of -platforms, including macOS and Windows, generalizing mount semantics -during container run is a necessity to enable workload optimizations. - -The current implementations of mounts on Linux provide a consistent -view of a host directory tree inside a container: reads and writes -performed either on the host or in the container are immediately -reflected in the other environment, and file system events (`inotify`, -`FSEvents`) are consistently propagated in both directions. - -On Linux, these guarantees carry no overhead, since the underlying VFS is -shared directly between host and container. However, on macOS (and -other non-Linux platforms) there are significant overheads to -guaranteeing perfect consistency, since messages describing file system -actions must be passed synchronously between container and host. The -current implementation is sufficiently efficient for most tasks, but -with certain types of workloads the overhead of maintaining perfect -consistency can result in significantly worse performance than a -native (non-Docker) environment. For example, - - * running `go list ./...` in the bind-mounted `docker/docker` source tree - takes around 26 seconds - - * writing 100MB in 1k blocks into a bind-mounted directory takes - around 23 seconds - - * running `ember build` on a freshly created (empty) application - involves around 70000 sequential syscalls, each of which translates - into a request and response passed between container and host. - -Optimizations to reduce latency throughout the stack have brought -significant improvements to these workloads, and a few further -optimization opportunities remain. However, even when latency is -minimized, the constraints of maintaining consistency mean that these -workloads remain unacceptably slow for some use cases. - -## Tuning with consistent, cached, and delegated configurations - -**_Fortunately, in many cases where the performance degradation is most -severe, perfect consistency between container and host is unnecessary._** -In particular, in many cases there is no need for writes performed in a -container to be immediately reflected on the host. For example, while -interactive development requires that writes to a bind-mounted directory -on the host immediately generate file system events within a container, -there is no need for writes to build artifacts within the container to -be immediately reflected on the host file system. Distinguishing between -these two cases makes it possible to significantly improve performance. - -There are three broad scenarios to consider, based on which you can dial in the -level of consistency you need. In each case, the container has an -internally-consistent view of bind-mounted directories, but in two cases -temporary discrepancies are allowed between container and host. - - * `consistent`: perfect consistency - (host and container have an identical view of the mount at all times) - - * `cached`: the host's view is authoritative - (permit delays before updates on the host appear in the container) - - * `delegated`: the container's view is authoritative - (permit delays before updates on the container appear in the host) - -## Examples - -Each of these configurations (`consistent`, `cached`, `delegated`) can be -specified as a suffix to the -[`-v`](../engine/reference/run.md#volume-shared-filesystems) -option of [`docker run`](../engine/reference/commandline/run.md). For -example, to bind-mount `/Users/yallop/project` in a container under the path -`/project`, you might run the following command: - -```bash -docker run -v /Users/yallop/project:/project:cached alpine command -``` - -The caching configuration can be varied independently for each bind mount, -so you can mount each directory in a different mode: - -```bash -docker run -v /Users/yallop/project:/project:cached \ - -v /host/another-path:/mount/another-point:consistent - alpine command -``` - -## Semantics - -The semantics of each configuration is described as a set of guarantees -relating to the observable effects of file system operations. In this -specification, "host" refers to the file system of the user's Docker -client. - -### delegated - -The `delegated` configuration provides the weakest set of guarantees. -For directories mounted with `delegated` the container's view of the -file system is authoritative, and writes performed by containers may not -be immediately reflected on the host file system. In situations such as NFS -asynchronous mode, if a running container with a `delegated` bind mount -crashes, then writes may be lost. - -However, by relinquishing consistency, `delegated` mounts offer -significantly better performance than the other configurations. Where -the data written is ephemeral or readily reproducible, such as from scratch -space or build artifacts, `delegated` may be the right choice. - -A `delegated` mount offers the following guarantees, which are presented -as constraints on the container run-time: - -1. If the implementation offers file system events, the container state -as it relates to a specific event **_must_** reflect the host file system -state at the time the event was generated if no container modifications -pertain to related file system state. - -2. If flush or sync operations are performed, relevant data **_must_** be -written back to the host file system.Between flush or sync -operations containers **_may_** cache data written, metadata modifications, -and directory structure changes. - -3. All containers hosted by the same runtime **_must_** share a consistent -cache of the mount. - -4. When any container sharing a `delegated` mount terminates, changes -to the mount **_must_** be written back to the host file system. If this -writeback fails, the container's execution **_must_** fail via exit code -and/or Docker event channels. - -5. If a `delegated` mount is shared with a `cached` or a `consistent` -mount, those portions that overlap **_must_** obey `cached` or `consistent` -mount semantics, respectively. - - Besides these constraints, the `delegated` configuration offers the -container runtime a degree of flexibility: - -6. Containers **_may_** retain file data and metadata (including directory -structure, existence of nodes, etc) indefinitely and this cache **_may_** -desynchronize from the file system state of the host. Implementors should expire -caches when host file system changes occur, but this may be difficult to do on -a guaranteed timeframe due to platform limitations. - -7. If changes to the mount source directory are present on the host -file system, those changes **_may_** be lost when the `delegated` mount -synchronizes with the host source directory. - -8. Behaviors 6-7 **do not** apply to the file types of socket, pipe, or device. - -### cached - -The `cached` configuration provides all the guarantees of the `delegated` -configuration, and some additional guarantees around the visibility of writes -performed by containers. As such, `cached` typically improves the performance -of read-heavy workloads, at the cost of some temporary inconsistency between the -host and the container. - -For directories mounted with `cached`, the host's view of -the file system is authoritative; writes performed by containers are immediately -visible to the host, but there may be a delay before writes performed on the -host are visible within containers. - ->**Tip:** To learn more about `cached`, see the article on -[User-guided caching in Docker Desktop for Mac](https://blog.docker.com/2017/05/user-guided-caching-in-docker-for-mac/). - -1. Implementations **_must_** obey `delegated` Semantics 1-5. - -2. If the implementation offers file system events, the container state -as it relates to a specific event **_must_** reflect the host file system -state at the time the event was generated. - -3. Container mounts **_must_** perform metadata modifications, directory -structure changes, and data writes consistently with the host file -system, and **_must not_** cache data written, metadata modifications, or -directory structure changes. - -4. If a `cached` mount is shared with a `consistent` mount, those portions -that overlap **_must_** obey `consistent` mount semantics. - - Some of the flexibility of the `delegated` configuration is retained, -namely: - -5. Implementations **_may_** permit `delegated` Semantics 6. - -### consistent - -The `consistent` configuration places the most severe restrictions on -the container run-time. For directories mounted with `consistent` the -container and host views are always synchronized: writes performed -within the container are immediately visible on the host, and writes -performed on the host are immediately visible within the container. - -The `consistent` configuration most closely reflects the behavior of -bind mounts on Linux. However, the overheads of providing strong -consistency guarantees make it unsuitable for a few use cases, where -performance is a priority and maintaining perfect consistency has low -priority. - -1. Implementations **_must_** obey `cached` Semantics 1-4. - -2. Container mounts **_must_** reflect metadata modifications, directory -structure changes, and data writes on the host file system immediately. - -### default - -The `default` configuration is identical to the `consistent` -configuration except for its name. Crucially, this means that `cached` -Semantics 4 and `delegated` Semantics 5 that require strengthening -overlapping directories do not apply to `default` mounts. This is the -default configuration if no `state` flags are supplied. diff --git a/docker-for-mac/osxfs.md b/docker-for-mac/osxfs.md deleted file mode 100644 index 1778f267f4..0000000000 --- a/docker-for-mac/osxfs.md +++ /dev/null @@ -1,369 +0,0 @@ ---- -description: Osxfs -keywords: mac, osxfs -redirect_from: -- /mackit/osxfs/ -title: File system sharing (osxfs) ---- - -`osxfs` is a new shared file system solution, exclusive to Docker Desktop for Mac. -`osxfs` provides a close-to-native user experience for bind mounting macOS file -system trees into Docker containers. To this end, `osxfs` features a number of -unique capabilities as well as differences from a classical Linux file system. - -### Case sensitivity - -With Docker Desktop for Mac, file systems operate in containers in the same way as they -operate in macOS. If a file system on macOS is case-insensitive, that behavior -is shared by any bind mount from macOS into a container. - -On macOS Sierra and lower, the default file system is **HFS+**. On macOS High -Sierra, the default file system is **APFS**. Both are case-insensitive by -default but available in case-sensitive and case-insensitive variants. - -To get case-sensitive behavior, format the volume used in your bind mount as -HFS+ or APFS with case-sensitivity. See the -[APFS FAQ](https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html). - -Reformatting your root partition is not recommended as some Mac software relies -on case-insensitivity to function. - -### Access control - -`osxfs`, and therefore Docker, can access only those file system resources that -the Docker Desktop for Mac user has access to. `osxfs` does not run as `root`. If the macOS -user is an administrator, `osxfs` inherits those administrator privileges. We -are still evaluating which privileges to drop in the file system process to -balance security and ease-of-use. `osxfs` performs no additional permissions -checks and enforces no extra access control on accesses made through it. All -processes in containers can access the same objects in the same way as the -Docker user who started the containers. - -### Namespaces - -Much of the macOS file system that is accessible to the user is also available to -containers using the `-v` bind mount syntax. The following command runs a container -from an image called `r-base` and shares the macOS user's `~/Desktop/` directory as -`/Desktop` in the container. - -```bash -$ docker run -it -v ~/Desktop:/Desktop r-base bash -``` - -The user's `~/Desktop/` directory is now visible in the container as a directory -under `/`. - -``` -root@2h30fa0c600e:/# ls -Desktop boot etc lib lib64 media opt root sbin sys usr -bin dev home lib32 libx32 mnt proc run srv tmp var -``` - -By default, you can share files in `/Users/`, `/Volumes/`, `/private/`, and -`/tmp` directly. To add or remove directory trees that are exported to Docker, -use the **File sharing** tab in Docker preferences -![whale menu](images/whale-x.png){: .inline} -> **Preferences** -> -**File sharing**. (See [Preferences](index.md#file-sharing).) - -All other paths -used in `-v` bind mounts are sourced from the Moby Linux VM running the Docker -containers, so arguments such as `-v /var/run/docker.sock:/var/run/docker.sock` -should work as expected. If a macOS path is not shared and does not exist in the -VM, an attempt to bind mount it fails rather than create it in the VM. Paths -that already exist in the VM and contain files are reserved by Docker and cannot -be exported from macOS. - -> See **[Performance tuning for volume mounts (shared filesystems)](osxfs-caching.md)** -> to learn about new configuration options available with the Docker 17.04 CE Edge release. - -### Ownership - -Initially, any containerized process that requests ownership metadata of an -object is told that its `uid` and `gid` own the object. When any containerized -process changes the ownership of a shared file system object, such as by using -the `chown` command, the new ownership information is persisted in the -`com.docker.owner` extended attribute of the object. Subsequent requests for -ownership metadata return the previously set values. Ownership-based permissions -are only enforced at the macOS file system level with all accessing processes -behaving as the user running Docker. If the user does not have permission to -read extended attributes on an object (such as when that object's permissions -are `0000`), `osxfs` attempts to add an access control list (ACL) entry that -allows the user to read and write extended attributes. If this attempt fails, -the object appears to be owned by the process accessing it until the extended -attribute is readable again. - -### File system events - -Most `inotify` events are supported in bind mounts, and likely `dnotify` and -`fanotify` (though they have not been tested) are also supported. This means -that file system events from macOS are sent into containers and trigger any -listening processes there. - -The following are **supported file system events**: - -* Creation -* Modification -* Attribute changes -* Deletion -* Directory changes - -The following are **partially supported file system events**: - -* Move events trigger `IN_DELETE` on the source of the rename and - `IN_MODIFY` on the destination of the rename - -The following are **unsupported file system events**: - -* Open -* Access -* Close events -* Unmount events (see [Mounts](#mounts)) - -Some events may be delivered multiple times. These limitations do not apply to -events between containers, only to those events originating in macOS. - -### Mounts - -The macOS mount structure is not visible in the shared volume, but volume -contents are visible. Volume contents appear in the same file system as the rest -of the shared file system. Mounting/unmounting macOS volumes that are also bind -mounted into containers may result in unexpected behavior in those containers. -Unmount events are not supported. Mount export support is planned but is still -under development. - -### Symlinks - -Symlinks are shared unmodified. This may cause issues when symlinks contain -paths that rely on the default case-insensitivity of the default macOS file -system. - -### File types - -Symlinks, hardlinks, socket files, named pipes, regular files, and directories -are supported. Socket files and named pipes only transmit between containers and -between macOS processes -- no transmission across the hypervisor is supported, -yet. Character and block device files are not supported. - -### Extended attributes - -Extended attributes are not yet supported. - -### Technology - -`osxfs` does not use OSXFUSE. `osxfs` does not run under, inside, or -between macOS userspace processes and the macOS kernel. - -### SSH agent forwarding - -Docker Desktop for Mac allows you to use the host’s SSH agent inside a container. To do this: - -1. Bind mount the SSH agent socket by adding the following parameter to your `docker run` command: - - `--mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock` - -1. Add the `SSH_AUTH_SOCK` environment variable in your container: - - `-e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock"` - -To enable the SSH agent in Docker Compose, add the following flags to your service: - - ```yaml -services: - web: - image: nginx:alpine - volumes: - - type: bind - source: /run/host-services/ssh-auth.sock - target: /run/host-services/ssh-auth.sock - environment: - - SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock - ``` - -### Performance issues, solutions, and roadmap - -> See **[Performance tuning for volume mounts (shared filesystems)](osxfs-caching.md)** -> to learn about new configuration options available with the Docker 17.04 CE Edge release. - -With regard to reported performance issues ([GitHub issue 77: File access in -mounted volumes extremely slow](https://github.com/docker/for-mac/issues/77)), -and a similar thread on [Docker Desktop for Mac forums on topic: File access in mounted -volumes extremely -slow](https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/), -this topic provides an explanation of the issues, recent progress in addressing -them, how the community can help us, and what you can expect in the -future. This explanation derives from a [post about understanding -performance](https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/158) -by David Sheets (@dsheets) on the [Docker development -team](https://forums.docker.com/groups/Docker) to the forum topic just -mentioned. We want to surface it in the documentation for wider reach. - -#### Understanding performance - -Perhaps the most important thing to understand is that shared file system -performance is multi-dimensional. This means that, depending on your workload, -you may experience exceptional, adequate, or poor performance with `osxfs`, the -file system server in Docker Desktop for Mac. File system APIs are very wide (20-40 -message types) with many intricate semantics involving on-disk state, in-memory -cache state, and concurrent access by multiple processes. Additionally, `osxfs` -integrates a mapping between macOS's FSEvents API and Linux's `inotify` API -which is implemented inside of the file system itself, complicating matters -further (cache behavior in particular). - -At the highest level, there are two dimensions to file system performance: -throughput (read/write IO) and latency (roundtrip time). In a traditional file -system on a modern SSD, applications can generally expect throughput of a few -GB/s. With large sequential IO operations, `osxfs` can achieve throughput of -around 250 MB/s which, while not native speed, is not likely to be the bottleneck for -most applications which perform acceptably on HDDs. - -Latency is the time it takes for a file system call to complete. For instance, -the time between a thread issuing write in a container and resuming with the -number of bytes written. With a classical block-based file system, this latency -is typically under 10μs (microseconds). With `osxfs`, latency is presently -around 130μs for most operations or 13× slower. For workloads which demand many -sequential roundtrips, this results in significant observable slowdown. -Reducing the latency requires shortening the data path from a Linux system call to -macOS and back again. This requires tuning each component in the data path in -turn -- some of which require significant engineering effort. Even if we achieve -a huge latency reduction of 65μs/roundtrip, we still "only" see a doubling -of performance. This is typical of performance engineering, which requires -significant effort to analyze slowdowns and develop optimized components. We -know a number of approaches that may reduce the roundtrip time but we -haven't implemented all those improvements yet (more on this below in -[What you can do](osxfs.md#what-you-can-do)). - -A second approach to improving performance is to reduce the number of -roundtrips by caching data. Recent versions of Docker Desktop for Mac (17.04 onwards) -include caching support that brings significant (2-4×) improvements to many -applications. Much of the overhead of osxfs arises from the requirement to -keep the container's and the host's view of the file system consistent, but -full consistency is not necessary for all applications and relaxing the -constraint opens up a number of opportunities for improved performance. - -At present there is support for read caching, with which the container's view -of the file system can temporarily drift apart from the authoritative view on -the host. Further caching developments, including support for write caching, -are planned. -A [detailed description of the behavior in various caching configurations](osxfs-caching.md) -is available. - -#### What we are doing - -We continue to actively work on increasing caching and on reducing the -file system data path latency. This requires significant analysis of file -system traces and speculative development of system improvements to try to -address specific performance issues. Perhaps surprisingly, application -workload can have a huge effect on performance. As an example, here are two -different use cases contributed on the -[forum topic](https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/) -and how their performance differs and suffers due to latency, caching, and -coherence: - -1. A rake example (see below) appears to attempt to access 37000+ -different files that don't exist on the shared volume. Even with a 2× speedup -via latency reduction this use case still seems "slow". -With caching enabled the performance increases around 3.5×, as described in -the [user-guided caching post](https://blog.docker.com/2017/05/user-guided-caching-in-docker-for-mac/). -We expect to see further performance improvements for rake with a "negative dcache" that -keeps track of, in the Linux kernel itself, the files that do not exist. -However, even this is not sufficient for the first time rake is run on a -shared directory. To handle that case, we actually need to develop a Linux -kernel patch which negatively caches all directory entries not in a -specified set -- and this cache must be kept up-to-date in real-time with the macOS -file system state even in the presence of missing macOS FSEvents messages and -so must be invalidated if macOS ever reports an event delivery failure. - -2. Running `ember build` in a shared file system results in ember creating many -different temporary directories and performing lots of intermediate activity -within them. An empty ember project is over 300MB. This usage pattern does not -require coherence between Linux and macOS, and is significantly improved by -write caching. - -These two examples come from performance use cases contributed by users and they -are incredibly helpful in prioritizing aspects of file system performance to -improve. We are developing statistical file system trace analysis tools -to characterize slow-performing workloads more easily to decide what to -work on next. - -Under development, we have: - -1. A growing performance test suite of real world use cases (more on this below -in What you can do) - -2. Further caching improvements, including negative, structural, and write-back -caching, and lazy cache invalidation. - -3. A Linux kernel patch to reduce data path latency by 2/7 copies and 2/5 -context switches - -4. Increased macOS integration to reduce the latency between the hypervisor and -the file system server - -#### What you can do - -When you report shared file system performance issues, it is most helpful to -include a minimal Real World reproduction test case that demonstrates poor -performance. - -Without a reproduction, it is very difficult for us to analyze your use case and -determine what improvements would speed it up. When you don't provide a -reproduction, one of us needs to figure out the specific software -you are using and guess and hope that we have configured it in a typical way or -a way that has poor performance. That usually takes 1-4 hours depending on your -use case and once it is done, we must then determine what regular performance is -like and what kind of slow-down your use case is experiencing. In some cases, it -is not obvious what operation is even slow in your specific development -workflow. The additional set-up to reproduce the problem means we have less time -to fix bugs, develop analysis tools, or improve performance. So, include -simple, immediate performance issue reproduction test cases. The [rake -reproduction -case](https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/103) -by @hirowatari shown in the forums thread is a great example. - -This example originally provided: - -1. A version-controlled repository so any changes/improvements to the test case -can be easily tracked. - -2. A Dockerfile which constructs the exact image to run - -3. A command-line invocation of how to start the container - -4. A straight-forward way to measure the performance of the use case - -5. A clear explanation (README) of how to run the test case - -#### What you can expect - -We continue to work toward an optimized shared file system implementation -on the Edge channel of Docker Desktop for Mac. - -You can expect some of the performance improvement work mentioned above to reach -the Edge channel in the coming release cycles. - -We plan to eventually open source all of our shared file system components. At -that time, we would be very happy to collaborate with you on improving the -implementation of `osxfs` and related software. - -We also plan to write up and publish further details of shared file system -performance analysis and improvement on the Docker blog. Look for or nudge -@dsheets about those articles, which should serve as a jumping off point for -understanding the system, measuring it, or contributing to it. - -#### Wrapping Up - -We hope this gives you a rough idea of where `osxfs` performance is and where -it's going. We are treating good performance as a top priority feature of the -file system sharing component and we are actively working on improving it -through a number of different avenues. The osxfs project started in December -2015. Since the first integration into Docker Desktop for Mac in February 2016, we've -improved performance by 50x or more for many workloads while achieving nearly -complete POSIX compliance and without compromising coherence (it is shared and -not simply synced). Of course, in the beginning there was lots of low-hanging -fruit and now many of the remaining performance improvements require significant -engineering work on custom low-level components. - -We appreciate your understanding as we continue development of the product and -work on all dimensions of performance. We want to continue to work with the -community on this, so continue to report issues as you find them. We look -forward to collaborating with you on ideas and on the source code itself.