docker
/
docs
mirror of https://github.com/docker/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #12139 from thaJeztah/20.10_logging_changes 

Rewrite logging section for dual logging (engine 20.10), and update post-install
This commit is contained in:
Usha Mandya 2021-01-20 16:09:53 +00:00 committed by GitHub
parent 8489de14df 6b1bae466f
commit 6cd2915d61
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 148 additions and 121 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

92
config/containers/logging/configure.md
View File

@ -11,26 +11,46 @@ title: Configure logging drivers
Docker includes multiple logging mechanisms to help you
[get information from running containers and services](index.md).
These mechanisms are called logging drivers.
These mechanisms are called logging drivers. Each Docker daemon has a default
logging driver, which each container uses unless you configure it to use a
different logging driver, or "log-driver" for short.
Each Docker daemon has a default logging driver, which each container uses
unless you configure it to use a different logging driver.
As a default, Docker uses the [`json-file` logging driver](json-file.md), which
caches container logs as JSON internally. In addition to using the logging drivers
included with Docker, you can also implement and use [logging driver plugins](plugins.md).
In addition to using the logging drivers included with Docker, you can also
implement and use [logging driver plugins](plugins.md).
> **Tip: use the "local" logging driver to prevent disk-exhaustion**
>
> By default, no log-rotation is performed. As a result, log-files stored by the
> default [`json-file` logging driver](json-file.md) logging driver can cause
> a significant amount of disk space to be used for containers that generate much
> output, which can lead to disk space exhaustion.
>
> Docker keeps the json-file logging driver (without log-rotation) as a default
> to remain backward compatibility with older versions of Docker, and for situations
> where Docker is used as runtime for Kubernetes.
>
> For other situations, the "local" logging driver is recommended as it performs
> log-rotation by default, and uses a more efficient file format. Refer to the
> [Configure the default logging driver](configure-the-default-logging-driver)
> section below to learn how to configure the "local" logging driver as a default,
> and the [local file logging driver](local.md) page for more details about the
> "local" logging driver.
## Configure the default logging driver
To configure the Docker daemon to default to a specific logging driver, set the
value of `log-driver` to the name of the logging driver in the `daemon.json`
file, which is located in `/etc/docker/` on Linux hosts or
`C:\ProgramData\docker\config\` on Windows server hosts. Note that you should create `daemon.json`
file, if the file does not exist.
The default logging driver is `json-file`. The following example explicitly sets the default logging driver to `syslog`:
configuration file. Refer to the "daemon configuration file" section in the
[`dockerd` reference manual](/engine/reference/commandline/dockerd/#daemon-configuration-file)
for details.
The default logging driver is `json-file`. The following example sets the default
logging driver to the [`local` log driver](local.md):
```json
{
"log-driver": "syslog"
"log-driver": "local"
}
```
@ -50,6 +70,9 @@ example sets two configurable options on the `json-file` logging driver:
}
```
Restart Docker for the changes to take effect for newly created containers.
Existing containers do not use the new logging configuration.
> **Note**
>
> `log-opts` configuration options in the `daemon.json` configuration file must
@ -62,13 +85,24 @@ To find the current default logging driver for the Docker daemon, run
command on Linux, macOS, or PowerShell on Windows:
{% raw %}
```bash
```console
$ docker info --format '{{.LoggingDriver}}'
json-file
```
{% endraw %}
> **Note**
>
> Changing the default logging driver or logging driver options in the daemon
> configuration only affects containers that are created after the configuration
> is changed. Existing containers retain the logging driver options that were
> used when they were created. To update the logging driver for a container, the
> container has to be re-created with the desired options.
> Refer to the [configure the logging driver for a container](#configure-the-logging-driver-for-a-container)
> section below to learn how to find the logging-driver configuration of a
> container.
## Configure the logging driver for a container
When you start a container, you can configure it to use a different logging
@ -79,7 +113,7 @@ default logging driver, it can use different configurable options.
The following example starts an Alpine container with the `none` logging driver.
```bash
```console
$ docker run -it --log-driver none alpine ash
```
@ -88,7 +122,7 @@ is using the `json-file` logging driver, run the following `docker inspect`
command, substituting the container name or ID for `<CONTAINER>`:
{% raw %}
```bash
```console
$ docker inspect -f '{{.HostConfig.LogConfig.Type}}' <CONTAINER>
json-file
@ -108,7 +142,8 @@ The `non-blocking` message delivery mode prevents applications from blocking due
to logging back pressure. Applications are likely to fail in unexpected ways when
STDERR or STDOUT streams block.
> **WARNING**
> **Warning**
>
> When the buffer is full and a new message is enqueued, the oldest message in
> memory is dropped. Dropping messages is often preferred to blocking the
> log-writing process of an application.
@ -124,7 +159,7 @@ defaults to 1 megabyte.
The following example starts an Alpine container with log output in non-blocking
mode and a 4 megabyte buffer:
```bash
```console
$ docker run -it --log-opt mode=non-blocking --log-opt max-buffer-size=4m alpine ping 127.0.0.1
```
@ -135,7 +170,7 @@ flags to the container's logs. This example starts a container using the Docker
daemon's default logging driver (let's assume `json-file`) but sets the
environment variable `os=ubuntu`.
```bash
```console
$ docker run -dit --label production_status=testing -e os=ubuntu alpine sh
```
@ -168,25 +203,16 @@ see more options.
| [`gcplogs`](gcplogs.md) | Writes log messages to Google Cloud Platform (GCP) Logging. |
| [`logentries`](logentries.md) | Writes log messages to Rapid7 Logentries. |
> **Note**
>
> When using Docker Engine 19.03 or older, the [`docker logs` command](../../../engine/reference/commandline/logs.md)
> is only functional for the `local`, `json-file` and `journald` logging drivers.
> Docker 20.10 and up introduces "dual logging", which uses a local buffer that
> allows you to use the `docker logs` command for any logging driver. Refer to
> [reading logs when using remote logging drivers](dual-logging.md) for details.
## Limitations of logging drivers
- Users of Docker Enterprise can make use of "dual logging", which enables you
to use the `docker logs` command for any logging driver. Refer to
[reading logs when using remote logging drivers](dual-logging.md) for
information about using `docker logs` to read container logs locally for many
third party logging solutions, including:
- `syslog`
- `gelf`
- `fluentd`
- `awslogs`
- `splunk`
- `etwlogs`
- `gcplogs`
- `Logentries`
- When using Docker Community Engine, the `docker logs` command is only available on the following drivers:
- `local`
- `json-file`
- `journald`
- Reading log information requires decompressing rotated log files, which causes
a temporary increase in disk usage (until the log entries from the rotated
files are read) and an increased CPU usage while decompressing.

61
config/containers/logging/dual-logging.md
View File

@ -6,46 +6,32 @@ title: Use docker logs to read container logs for remote logging drivers
## Overview
Prior to Docker Engine Enterprise 18.03, the `jsonfile` and `journald` log
drivers supported reading container logs using `docker logs`. However, many
third party logging drivers had no support for locally reading logs using
`docker logs`, including:
- `syslog`
- `gelf`
- `fluentd`
- `awslogs`
- `splunk`
- `etwlogs`
- `gcplogs`
- `logentries`
Prior to Docker Engine 20.10, the [`docker logs` command](../../../engine/reference/commandline/logs.md)
could only be used with logging drivers that supported for containers using the
`local`, `json-file`, or `journald` log drivers. However, many third party logging
drivers had no support for locally reading logs using `docker logs`
This created multiple problems when attempting to gather log data in an
automated and standard way. Log information could only be accessed and viewed
through the third-party solution in the format specified by that
third-party tool.
Starting with Docker Engine Enterprise 18.03.1-ee-1, you can use `docker logs`
to read container logs regardless of the configured logging driver or plugin.
This capability, sometimes referred to as dual logging, allows you to use
`docker logs` to read container logs locally in a consistent format, regardless
of the remote log driver used, because the engine is configured to log
information to the “local” logging driver. Refer to
[Configure the default logging driver](/config/containers/logging/configure) for
additional information.
Starting with Docker Engine 20.10, you can use `docker logs` to read container
logs regardless of the configured logging driver or plugin. This capability,
referred to as "dual logging", allows you to use `docker logs` to read container
logs locally in a consistent format, regardless of the log driver used, because
the engine is configured to log information to the “local” logging driver. Refer
to [Configure the default logging driver](/config/containers/logging/configure)
for additional information.
## Prerequisites
Dual logging is only supported for Docker Enterprise, and is enabled by default
starting with Engine Enterprise 18.03.1-ee-1.
No configuration changes are needed to use dual logging. Docker Engine 20.10 and
up automatically enable dual logging if the configured logging driver does not
support reading logs.
## Usage
Dual logging is enabled by default. You must configure either the docker daemon
or the container with remote logging driver.
The following example shows the results of running a `docker logs` command with
The following examples show the result of running a `docker logs` command with
and without dual logging availability:
### Without dual logging capability:
@ -56,7 +42,7 @@ locally:
- Step 1: Configure Docker daemon
```bash
```console
$ cat /etc/docker/daemon.json
{
"log-driver": "splunk",
@ -68,23 +54,24 @@ locally:
- Step 2: Start the container
```bash
```console
$ docker run -d busybox --name testlog top
```
- Step 3: Read the container logs
```bash
```console
$ docker logs 7d6ac83a89a0
The docker logs command was not available for drivers other than json-file and journald.
Error response from daemon: configured logging driver does not support reading
```
### With dual logging capability:
To configure a container or docker with a remote logging driver such as splunk:
- Step 1: Configure Docker daemon
```bash
```console
$ cat /etc/docker/daemon.json
{
"log-driver": "splunk",
@ -96,13 +83,13 @@ To configure a container or docker with a remote logging driver such as splunk:
- Step 2: Start the container
```bash
```console
$ docker run -d busybox --name testlog top
```
- Step 3: Read the container logs
```bash
```console
$ docker logs 7d6ac83a89a0
2019-02-04T19:48:15.423Z [INFO] core: marked as sealed
2019-02-04T19:48:15.423Z [INFO] core: pre-seal teardown starting
@ -115,7 +102,7 @@ To configure a container or docker with a remote logging driver such as splunk:
> **Note**
>
> For a local driver, such as json-file and journald, there is no difference in
> For a local driver, such as `json-file` and `journald`, there is no difference in
> functionality before or after the dual logging capability became available.
> The log is locally visible in both scenarios.

10
config/containers/logging/json-file.md
View File

@ -17,6 +17,13 @@ only one container.
{"log":"Log line is here\n","stream":"stdout","time":"2019-01-01T11:11:11.111111111Z"}
```
> *Warning*
>
> The `json-file` logging driver uses file-based storage. These files are designed
> to be exclusively accessed by the Docker daemon. Interacting with these files
> with external tools may interfere with Docker's logging system and result in
> unexpected behavior, and should be avoided.
## Usage
To use the `json-file` driver as the default logging driver, set the `log-driver`
@ -26,7 +33,8 @@ located in `/etc/docker/` on Linux hosts or
configuring Docker using `daemon.json`, see
[daemon.json](../../../engine/reference/commandline/dockerd.md#daemon-configuration-file).
The following example sets the log driver to `json-file` and sets the `max-size` and `max-file` options.
The following example sets the log driver to `json-file` and sets the `max-size`
and `max-file` options to enable automatic log-rotation.
```json
{

106
engine/install/linux-postinstall.md
View File

@ -39,13 +39,13 @@ To create the `docker` group and add your user:
1. Create the `docker` group.
```bash
```console
$ sudo groupadd docker
```
2. Add your user to the `docker` group.
```bash
```console
$ sudo usermod -aG docker $USER
```
@ -57,13 +57,13 @@ To create the `docker` group and add your user:
On Linux, you can also run the following command to activate the changes to groups:
```bash
```console
$ newgrp docker
```
4. Verify that you can run `docker` commands without `sudo`.
```bash
```console
$ docker run hello-world
```
@ -85,48 +85,35 @@ To create the `docker` group and add your user:
are lost), or change its ownership and permissions using the
following commands:
```bash
```console
$ sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
$ sudo chmod g+rwx "$HOME/.docker" -R
```
## Configure Docker to start on boot
Most current Linux distributions (RHEL, CentOS, Fedora, Ubuntu 16.04 and higher)
use [`systemd`](#systemd) to manage which services start when the system boots.
Ubuntu 14.10 and below use [`upstart`](#upstart).
Most current Linux distributions (RHEL, CentOS, Fedora, Debian, Ubuntu 16.04 and
higher) use [`systemd`](#systemd) to manage which services start when the system
boots. On Debian and Ubuntu, the Docker service is configured to start on boot
by default. To automatically start Docker and Containerd on boot for other
distros, use the commands below:
### `systemd`
```bash
$ sudo systemctl enable docker
```console
$ sudo systemctl enable docker.service
$ sudo systemctl enable containerd.service
```
To disable this behavior, use `disable` instead.
```bash
$ sudo systemctl disable docker
```console
$ sudo systemctl disable docker.service
$ sudo systemctl disable containerd.service
```
If you need to add an HTTP Proxy, set a different directory or partition for the
Docker runtime files, or make other customizations, see
[customize your systemd Docker daemon options](../../config/daemon/systemd.md).
### `upstart`
Docker is automatically configured to start on boot using
`upstart`. To disable this behavior, use the following command:
```bash
$ echo manual | sudo tee /etc/init/docker.override
```
### `chkconfig`
```bash
$ sudo chkconfig docker on
```
## Use a different storage engine
For information about the different storage engines, see
@ -136,11 +123,27 @@ your host's Linux distribution and available kernel drivers.
## Configure default logging driver
Docker provides the [capability](../../config/containers/logging/index.md) to collect and view log data from all containers running on a host via a series of logging drivers. The default logging driver, `json-file`, writes log data to JSON-formatted files on the host filesystem. Over time, these log files expand in size, leading to potential exhaustion of disk resources. To alleviate such issues, either configure an alternative logging driver such as Splunk or Syslog, or [set up log rotation](/config/containers/logging/configure/#configure-the-default-logging-driver) for the default driver. If you configure an alternative logging driver, see [Use `docker logs` to read container logs for remote logging drivers](/config/containers/logging/dual-logging/).
Docker provides the [capability](../../config/containers/logging/index.md) to
collect and view log data from all containers running on a host via a series of
logging drivers. The default logging driver, `json-file`, writes log data to
JSON-formatted files on the host filesystem. Over time, these log files expand
in size, leading to potential exhaustion of disk resources.
To alleviate such issues, either configure the `json-file` logging driver to
enable [log rotation](../../config/containers/logging/json-file.md), use an
[alternative logging driver](../../config/containers/logging/configure.md#configure-the-default-logging-driver)
such as the ["local" logging driver](../../config/containers/logging/local.md)
that performs log rotation by default, or use a logging driver that sends
logs to a remote logging aggregator.
## Configure where the Docker daemon listens for connections
By default, the Docker daemon listens for connections on a UNIX socket to accept requests from local clients. It is possible to allow Docker to accept requests from remote hosts by configuring it to listen on an IP address and port as well as the UNIX socket. For more detailed information on this configuration option take a look at "Bind Docker to another host/port or a unix socket" section of the [Docker CLI Reference](https://docs.docker.com/engine/reference/commandline/dockerd/) article.
By default, the Docker daemon listens for connections on a UNIX socket to accept
requests from local clients. It is possible to allow Docker to accept requests
from remote hosts by configuring it to listen on an IP address and port as well
as the UNIX socket. For more detailed information on this configuration option
take a look at "Bind Docker to another host/port or a unix socket" section of
the [Docker CLI Reference](/engine/reference/commandline/dockerd/) article.
> Secure your connection
>
@ -148,10 +151,13 @@ By default, the Docker daemon listens for connections on a UNIX socket to accept
> understand the security implications of opening docker to the network. If steps are not taken to secure the connection,
> it is possible for remote non-root users to gain root access on the host. For more information on how to use TLS
> certificates to secure this connection, check this article on
> [how to protect the Docker daemon socket](https://docs.docker.com/engine/security/https/).
> [how to protect the Docker daemon socket](../security/https.md).
{: .warning}
Configuring Docker to accept remote connections can be done with the `docker.service` systemd unit file for Linux distributions using systemd, such as recent versions of RedHat, CentOS, Ubuntu and SLES, or with the `daemon.json` file which is recommended for Linux distributions that do not use systemd.
Configuring Docker to accept remote connections can be done with the `docker.service`
systemd unit file for Linux distributions using systemd, such as recent versions
of RedHat, CentOS, Ubuntu and SLES, or with the `daemon.json` file which is
recommended for Linux distributions that do not use systemd.
> systemd vs daemon.json
>
@ -174,19 +180,19 @@ Configuring Docker to accept remote connections can be done with the `docker.ser
4. Reload the `systemctl` configuration.
```bash
```console
$ sudo systemctl daemon-reload
```
5. Restart Docker.
```bash
```console
$ sudo systemctl restart docker.service
```
6. Check to see whether the change was honored by reviewing the output of `netstat` to confirm `dockerd` is listening on the configured port.
```bash
```console
$ sudo netstat -lntp | grep dockerd
tcp 0 0 127.0.0.1:2375 0.0.0.0:* LISTEN 3758/dockerd
```
@ -197,7 +203,7 @@ Configuring Docker to accept remote connections can be done with the `docker.ser
```json
{
"hosts": ["unix:///var/run/docker.sock", "tcp://127.0.0.1:2375"]
"hosts": ["unix:///var/run/docker.sock", "tcp://127.0.0.1:2375"]
}
```
@ -205,7 +211,7 @@ Configuring Docker to accept remote connections can be done with the `docker.ser
3. Check to see whether the change was honored by reviewing the output of `netstat` to confirm `dockerd` is listening on the configured port.
```bash
```console
$ sudo netstat -lntp | grep dockerd
tcp 0 0 127.0.0.1:2375 0.0.0.0:* LISTEN 3758/dockerd
```
@ -224,7 +230,7 @@ is missing some modules. To check kernel compatibility, you can download and
run the [`check-config.sh`](https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh)
script.
```bash
```console
$ curl https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh > check-config.sh
$ bash ./check-config.sh
@ -245,7 +251,7 @@ Cannot connect to the Docker daemon. Is 'docker daemon' running on this host?
To see which host your client is configured to connect to, check the value of
the `DOCKER_HOST` variable in your environment.
```bash
```console
$ env | grep DOCKER_HOST
```
@ -254,7 +260,7 @@ Docker daemon running on that host. If it is unset, the Docker client is set to
connect to the Docker daemon running on the local host. If it is set in error,
use the following command to unset it:
```bash
```console
$ unset DOCKER_HOST
```
@ -315,7 +321,7 @@ can't use it. Using default external servers : [8.8.8.8 8.8.4.4]
If you see this warning, first check to see if you use `dnsmasq`:
```bash
```console
$ ps aux |grep dnsmasq
```
@ -340,7 +346,7 @@ at `/etc/docker/daemon.json`.
`/etc/docker/daemon.json` file, which controls the Docker daemon
configuration.
```bash
```console
$ sudo nano /etc/docker/daemon.json
```
@ -349,7 +355,7 @@ at `/etc/docker/daemon.json`.
```json
{
"dns": ["8.8.8.8", "8.8.4.4"]
"dns": ["8.8.8.8", "8.8.4.4"]
}
```
@ -361,21 +367,21 @@ at `/etc/docker/daemon.json`.
3. Restart the Docker daemon.
```bash
```console
$ sudo service docker restart
```
4. Verify that Docker can resolve external IP addresses by trying to pull an
image:
```bash
```console
$ docker pull hello-world
```
5. If necessary, verify that Docker containers can resolve an internal hostname
by pinging it.
```bash
```console
$ docker run --rm -it alpine ping -c4 <my_internal_host>
PING google.com (192.168.1.2): 56 data bytes
@ -406,7 +412,7 @@ IP address, follow these instructions to disable `dnsmasq` in NetworkManager.
4. Restart both NetworkManager and Docker. As an alternative, you can reboot
your system.
```bash
```console
$ sudo restart network-manager
$ sudo restart docker
```
@ -417,7 +423,7 @@ To disable `dnsmasq` on RHEL, CentOS, or Fedora:
1. Disable the `dnsmasq` service:
```bash
```console
$ sudo service dnsmasq stop
$ sudo systemctl disable dnsmasq
@ -485,7 +491,7 @@ and a 10% overall performance degradation, even if Docker is not running.
3. Update GRUB.
```bash
```console
$ sudo update-grub
```