diff --git a/_data/toc.yaml b/_data/toc.yaml index fd834bfa11..e9cef0d3f1 100644 --- a/_data/toc.yaml +++ b/_data/toc.yaml @@ -325,12 +325,16 @@ guides: title: View a container's logs - path: /config/containers/logging/configure/ title: Configure logging drivers + - path: /config/containers/logging/dual-logging/ + title: Use docker logs with a logging driver - path: /config/containers/logging/plugins/ title: Use a logging driver plugin - path: /config/containers/logging/log_tags/ title: Customize log driver output - sectiontitle: Logging driver details section: + - path: /config/containers/logging/local/ + title: Local file logging driver - path: /config/containers/logging/logentries/ title: Logentries logging driver - path: /config/containers/logging/json-file/ diff --git a/config/containers/logging/configure.md b/config/containers/logging/configure.md index f281ed2140..4d44f688c4 100644 --- a/config/containers/logging/configure.md +++ b/config/containers/logging/configure.md @@ -19,7 +19,6 @@ unless you configure it to use a different logging driver. In addition to using the logging drivers included with Docker, you can also implement and use [logging driver plugins](/engine/admin/logging/plugins.md). - ## Configure the default logging driver To configure the Docker daemon to default to a specific logging driver, set the @@ -60,7 +59,7 @@ the default output for commands such as `docker inspect ` is JSON. To find the current default logging driver for the Docker daemon, run `docker info` and search for `Logging Driver`. You can use the following -command: +command on Linux, macOS, or PowerShell on Windows: {% raw %} ```bash @@ -146,8 +145,8 @@ see more options. | Driver | Description | |:------------------------------|:--------------------------------------------------------------------------------------------------------------| | `none` | No logs are available for the container and `docker logs` does not return any output. | +| [`local`](local.md) | Logs are stored in a custom format designed for minimal overhead. | | [`json-file`](json-file.md) | The logs are formatted as JSON. The default logging driver for Docker. | -| [`local`](local.md) | Writes logs messages to local filesystem in binary files using Protobuf. | | [`syslog`](syslog.md) | Writes logging messages to the `syslog` facility. The `syslog` daemon must be running on the host machine. | | [`journald`](journald.md) | Writes log messages to `journald`. The `journald` daemon must be running on the host machine. | | [`gelf`](gelf.md) | Writes log messages to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash. | @@ -160,6 +159,25 @@ see more options. ## Limitations of logging drivers -The `docker logs` command is not available for drivers other than `json-file` -and `journald`. +- Users of Docker Enterprise can make use of "dual logging", which enables you to use the `docker logs` +command for any logging driver. Refer to +[Reading logs when using remote logging drivers](/config/containers/logging/dual-logging/) for information about +using `docker logs` to read container logs locally for many third party logging solutions, including: + - syslog + - gelf + - fluentd + - awslogs + - splunk + - etwlogs + - gcplogs + - Logentries + +- When using Docker Community Engine, the `docker logs` command is only available on the following drivers: + + - `local` + - `json-file` + - `journald` + +- Reading log information requires decompressing rotated log files, which causes a temporary increase in disk usage (until the log entries from the rotated files are read) and an increased CPU usage while decompressing. +- The capacity of the host storage where docker’s data directory resides determines the maximum size of the log file information. diff --git a/config/containers/logging/dual-logging.md b/config/containers/logging/dual-logging.md new file mode 100644 index 0000000000..2f3d395582 --- /dev/null +++ b/config/containers/logging/dual-logging.md @@ -0,0 +1,114 @@ +--- +description: Learn how to read container logs locally when using a third party logging solution. +keywords: docker, logging, driver +title: Using docker logs to read container logs for remote logging drivers +--- + +## Overview + +Prior to Docker Engine Enterprise 18.03, the `jsonfile` and `journald` log drivers supported reading +container logs using `docker logs`. However, many third party logging drivers had no +support for locally reading logs using `docker logs`, including: + +- syslog +- gelf +- fluentd +- awslogs +- splunk +- etwlogs +- gcplogs +- Logentries + +This created multiple problems, especially with UCP, when attempting to gather log data in an +automated and standard way. Log information could only be accessed and viewed through the +third-party solution in the format specified by that third-party tool. + +Starting with Docker Engine Enterprise 18.03.1-ee-1, you can use `docker logs` to read container +logs regardless of the configured logging driver or plugin. This capability, sometimes referred to +as dual logging, allows you to use `docker logs` to read container logs locally in a consistent format, +regardless of the remote log driver used, because the engine is configured to log information to the “local” +logging driver. Refer to [Configure the default logging driver](/configure) for additional information. + +## Prerequisites + +- Docker Enterprise - Dual logging is only supported for Docker Enterprise, and is enabled by default starting with +Engine Enterprise 18.03.1-ee-1. + +## Usage +Dual logging is enabled by default. You must configure either the docker daemon or the container with remote logging driver. + +The following example shows the results of running a `docker logs` command with and without dual logging availability: + +### Without dual logging capability: +When a container or `dockerd` was configured with a remote logging driver such as splunk, an error was +displayed when attempting to read container logs locally: + +- Step 1: Configure Docker daemon + + ``` + $ cat /etc/docker/daemon.json + { + "log-driver": "splunk", + "log-opts": { + ... + } + } + ``` + +- Step 2: Start the container + + ``` + $ docker run -d busybox --name testlog top + ``` + +- Step 3: Read the container logs + ``` + $ docker logs 7d6ac83a89a0 + The docker logs command was not available for drivers other than json-file and journald. + ``` + +### With dual logging capability: +To configure a container or docker with a remote logging driver such as splunk: + +- Step 1: Configure Docker daemon + ``` + $ cat /etc/docker/daemon.json + { + "log-driver": "splunk", + "log-opts": { + ... + } + } + ``` + +- Step 2: Start the container + ``` + $ docker run -d busybox --name testlog top + ``` + +- Step 3: Read the container logs + ``` + $ docker logs 7d6ac83a89a0 + 2019-02-04T19:48:15.423Z [INFO] core: marked as sealed + 2019-02-04T19:48:15.423Z [INFO] core: pre-seal teardown starting + 2019-02-04T19:48:15.423Z [INFO] core: stopping cluster listeners + 2019-02-04T19:48:15.423Z [INFO] core: shutting down forwarding rpc listeners + 2019-02-04T19:48:15.423Z [INFO] core: forwarding rpc listeners stopped + 2019-02-04T19:48:15.599Z [INFO] core: rpc listeners successfully shut down + 2019-02-04T19:48:15.599Z [INFO] core: cluster listeners successfully shut down + ``` + +Note: +For a local driver, such as json-file and journald, there is no difference in functionality +before or after the dual logging capability became available. The log is locally visible in both scenarios. + + +## Limitations + +- You cannot specify more than one log driver. +- If a container using a logging driver or plugin that sends logs remotely suddenly has a "network" issue, +no ‘write’ to the local cache occurs. +- If a write to `logdriver` fails for any reason (file system full, write permissions removed), +the cache write fails and is logged in the daemon log. The log entry to the cache is not retried. +- Some logs might be lost from the cache in the default configuration because a ring buffer is used to +prevent blocking the stdio of the container in case of slow file writes. An admin must repair these while the daemon is shut down. diff --git a/config/containers/logging/json-file.md b/config/containers/logging/json-file.md index 913f08d305..c05825f476 100644 --- a/config/containers/logging/json-file.md +++ b/config/containers/logging/json-file.md @@ -13,10 +13,6 @@ and writes them in files using the JSON format. The JSON format annotates each l origin (`stdout` or `stderr`) and its timestamp. Each log file contains information about only one container. -```json -{"log":"Log line is here\n","stream":"stdout","time":"2019-01-01T11:11:11.111111111Z"} -``` - ## Usage To use the `json-file` driver as the default logging driver, set the `log-driver` @@ -26,22 +22,20 @@ located in `/etc/docker/` on Linux hosts or configuring Docker using `daemon.json`, see [daemon.json](/engine/reference/commandline/dockerd.md#daemon-configuration-file). -The following example sets the log driver to `json-file` and sets the `max-size` -and `max-file` options. +The following example sets the log driver to `json-file` and sets the `max-size` and 'max-file' options. ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", - "max-file": "3" + "max-file": "3" } } ``` - -> **Note**: `log-opt` configuration options in the `daemon.json` configuration -> file must be provided as strings. Boolean and numeric values (such as the value -> for `max-file` in the example above) must therefore be enclosed in quotes (`"`). +**Note**: `log-opt` configuration options in the `daemon.json` configuration +file must be provided as strings. Boolean and numeric values (such as the value +for `max-file` in the example above) must therefore be enclosed in quotes (`"`). Restart Docker for the changes to take effect for newly created containers. Existing containers do not use the new logging configuration. @@ -65,6 +59,7 @@ The `json-file` logging driver supports the following logging options: | `labels` | Applies when starting the Docker daemon. A comma-separated list of logging-related labels this daemon accepts. Used for advanced [log tag options](log_tags.md). | `--log-opt labels=production_status,geo` | | `env` | Applies when starting the Docker daemon. A comma-separated list of logging-related environment variables this daemon accepts. Used for advanced [log tag options](log_tags.md). | `--log-opt env=os,customer` | | `env-regex` | Similar to and compatible with `env`. A regular expression to match logging-related environment variables. Used for advanced [log tag options](log_tags.md). | `--log-opt env-regex=^(os|customer).` | +| `compress` | Toggles compression for rotated logs. Default is `disabled`. | `--log-opt compress=true` | ### Examples diff --git a/config/containers/logging/local.md b/config/containers/logging/local.md index 708e4a3ee8..dbd9d9974d 100644 --- a/config/containers/logging/local.md +++ b/config/containers/logging/local.md @@ -1,46 +1,53 @@ --- -description: Describes how to use the local binary (Protobuf) logging driver. -keywords: local, protobuf, docker, logging, driver +description: Describes how to use the local logging driver. +keywords: local, docker, logging, driver redirect_from: - /engine/reference/logging/local/ - /engine/admin/logging/local/ -title: local binary file Protobuf logging driver +title: Local File logging driver --- -This `log-driver` writes to `local` binary files using Protobuf [Protocol Buffers](https://en.wikipedia.org/wiki/Protocol_Buffers) +The `local` logging driver captures output from container's stdout/stderr and +writes them to an internal storage that is optimized for performance and disk +use. + +By default the `local` driver preserves 100MB of log messages per container and +uses automatic compression to reduce the size on disk. + +> *Note*: the `local` logging driver currently uses file-based storage. The +> file-format and storage mechanism are designed to be exclusively accessed by +> the Docker daemon, and should not be used by external tools as the +> implementation may change in future releases. ## Usage To use the `local` driver as the default logging driver, set the `log-driver` and `log-opt` keys to appropriate values in the `daemon.json` file, which is located in `/etc/docker/` on Linux hosts or -`C:\ProgramData\docker\config\daemon.json` on Windows Server. For more information about +`C:\ProgramData\docker\config\daemon.json` on Windows Server. For more about configuring Docker using `daemon.json`, see [daemon.json](/engine/reference/commandline/dockerd.md#daemon-configuration-file). -The following example sets the log driver to `local`. +The following example sets the log driver to `local` and sets the `max-size` +option. ```json { "log-driver": "local", - "log-opts": {} + "log-opts": { + "max-size": "10m" + } } ``` -> **Note**: `log-opt` configuration options in the `daemon.json` configuration -> file must be provided as strings. Boolean and numeric values (such as the value -> for `max-file` in the example above) must therefore be enclosed in quotes (`"`). - -Restart Docker for the changes to take effect for newly created containers. - -Existing containers will not use the new logging configuration. +Restart Docker for the changes to take effect for newly created containers. Existing containers do not use the new logging configuration. You can set the logging driver for a specific container by using the `--log-driver` flag to `docker container create` or `docker run`: ```bash $ docker run \ - --log-driver local --log-opt compress="false" \ + --log-driver local --log-opt max-size=10m \ alpine echo hello world ``` @@ -50,6 +57,15 @@ The `local` logging driver supports the following logging options: | Option | Description | Example value | |:------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------| -| `max-size` | The maximum size of each binary log file before rotation. A positive integer plus a modifier representing the unit of measure (`k`, `m`, or `g`). Defaults to `20m`. | `--log-opt max-size=10m` | -| `max-file` | The maximum number of binary log files. If rotating the logs creates an excess file, the oldest file is removed. **Only effective when `max-size` is also set.** A positive integer. Defaults to `5`. | `--log-opt max-file=5` | -| `compress` | Whether or not the binary files should be compressed. Defaults to `true` | `--log-opt compress=true` | +| `max-size` | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (`k`, `m`, or `g`). Defaults to 20m. | `--log-opt max-size=10m` | +| `max-file` | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. **Only effective when `max-size` is also set.** A positive integer. Defaults to 5. | `--log-opt max-file=3` | +| `compress` | Toggle compression of rotated log files. Enabled by default. | `--log-opt compress=false` | + +### Examples + +This example starts an `alpine` container which can have a maximum of 3 log +files no larger than 10 megabytes each. + +```bash +$ docker run -it --log-opt max-size=10m --log-opt max-file=3 alpine ash +``` diff --git a/ee/ucp/interlock/deploy/production.md b/ee/ucp/interlock/deploy/production.md index 0a353b4e8c..61ebc16e20 100644 --- a/ee/ucp/interlock/deploy/production.md +++ b/ee/ucp/interlock/deploy/production.md @@ -128,4 +128,4 @@ to provide more bandwidth for the user services. ## Next steps - [Configure Interlock](../config/index.md) -- [Deploy applications](../usage.index.md) +- [Deploy applications](./index.md) diff --git a/ee/ucp/interlock/usage/index.md b/ee/ucp/interlock/usage/index.md index 0a488ccf26..ccdf7bcb33 100644 --- a/ee/ucp/interlock/usage/index.md +++ b/ee/ucp/interlock/usage/index.md @@ -151,13 +151,12 @@ able to start using the service from your browser. ## Next steps - [Publish a service as a canary instance](./canary.md) -- [Usie context or path-based routing](./context.md) +- [Use context or path-based routing](./context.md) - [Publish a default host service](./interlock-vip-mode.md) - [Specify a routing mode](./interlock-vip-mode.md) - [Use routing labels](./labels-reference.md) - [Implement redirects](./redirects.md) - [Implement a service cluster](./service-clusters.md) - [Implement persistent (sticky) sessions](./sessions.md) -- [Implement SSL](./ssl.md) - [Secure services with TLS](./tls.md) - [Configure websockets](./websockets.md) diff --git a/registry/index.md b/registry/index.md index 89f6abf4c8..8d48990954 100644 --- a/registry/index.md +++ b/registry/index.md @@ -38,7 +38,7 @@ free-to-use, hosted Registry, plus additional features (organization accounts, automated builds, and more). Users looking for a commercially supported version of the Registry should look -into [Docker Trusted Registry](/datacenter/dtr/2.1/guides/index.md). +into [Docker Trusted Registry](/ee/dtr/). ## Requirements diff --git a/storage/storagedriver/zfs-driver.md b/storage/storagedriver/zfs-driver.md index 2f3faa159a..e4fb127ded 100644 --- a/storage/storagedriver/zfs-driver.md +++ b/storage/storagedriver/zfs-driver.md @@ -263,7 +263,7 @@ There are several factors that influence the performance of Docker using the filesystems like ZFS. ZFS mitigates this by using a small block size of 128k. The ZFS intent log (ZIL) and the coalescing of writes (delayed writes) also help to reduce fragmentation. You can monitor fragmentation using - `zfs status`. However, there is no way to defragment ZFS without reformatting + `zpool status`. However, there is no way to defragment ZFS without reformatting and restoring the filesystem. - **Use the native ZFS driver for Linux**: The ZFS FUSE implementation is not