diff --git a/docs/tasks/debug-application-cluster/logging-stackdriver.md b/docs/tasks/debug-application-cluster/logging-stackdriver.md index ca80aed743..5faab6b84f 100644 --- a/docs/tasks/debug-application-cluster/logging-stackdriver.md +++ b/docs/tasks/debug-application-cluster/logging-stackdriver.md @@ -10,7 +10,7 @@ with the [overview of logging in Kubernetes](/docs/concepts/cluster-administrati **Note:** By default, Stackdriver logging collects only your container's standard output and standard error streams. To collect any logs your application writes to a file (for example), -see the [sidecar approach](/docs/concepts/cluster-administration/logging/#using-a-sidecar-container-with-the-logging-agent) +see the [sidecar approach](/docs/concepts/cluster-administration/logging#using-a-sidecar-container-with-the-logging-agent) in the Kubernetes logging overview. ## Deploying @@ -80,16 +80,16 @@ Stackdriver Logging agents to the running cluster. 1. Deploy a `ConfigMap` with the logging agent configuration by running the following command: ```shell - kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml + kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/fluentd-gcp-configmap.yaml ``` - The command creates the `ConfigMap` in the `kube-system` namespace. You can download the file + The command creates the `ConfigMap` in the `default` namespace. You can download the file manually and change it before creating the `ConfigMap` object. 1. Deploy the logging agent `DaemonSet` by running the following command: ```shell - kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml + kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/fluentd-gcp-ds.yaml ``` You can download and edit this file before using it as well. @@ -231,3 +231,98 @@ or to [BigQuery](https://cloud.google.com/bigquery/) to run further analysis. Stackdriver Logging offers the concept of sinks, where you can specify the destination of log entries. More information is available on the Stackdriver [Exporting Logs page](https://cloud.google.com/logging/docs/export/configure_export_v2). + +## Configuring Stackdriver Logging Agents + +Sometimes the default installation of Stackdriver Logging may not suite your needs, for example: + +* You may want to add more resources because default performance doesn't suite your needs. +* You may want to introduce additional parsing to extract more metadata from your log messages, +like severity or source code reference. +* You may want to send logs not only to Stackdriver or send it to Stackdriver only partially. + +In this case you need to be able to change the parameters of `DaemonSet` and `ConfigMap`. + +### Prerequisites + +If you're on GKE and Stackdriver Logging is enabled in your cluster, you cannot change its +parameters. Likewise, if you're not on GKE, but Stackdriver Logging is installed as an addon, +you won't be able to change deployment parameters using Kubernetes API. To make it possible +to change parameters of Stackdriver Logging agents, you should switch to the API object +deployment, when Stackdriver Logging is installed into a running cluster that didn't have any +cluster logging solutions installed before that. + +You can find notes on how to install Stackdriver Logging agents into a running cluster in the +[Deploying section](#deploying). + +### Changing `DaemonSet` parameters + +When you have the Stackdriver Logging `DaemonSet` in your cluster, you can just modify the +`template` field in its spec, daemonset controller will update the pods for you. For example, +let's assume you've just installed the Stackdriver Logging as described above. Now you want to +change the memory limit to give fluentd more memory to safely process more logs. + +Get the spec of `DaemonSet` running in your cluster: + +```shell +kubectl get ds fluentd-gcp-v2.0 --namespace kube-system -o yaml > fluentd-gcp-ds.yaml +``` + +Then edit resource requirements in the spec file and update the `DaemonSet` object +in the apiserver using the following command: + +```shell +kubectl replace -f fluentd-gcp-ds.yaml +``` + +After some time, Stackdriver Logging agent pods will be restarted with the new configuration. + +### Changing fluentd parameters + +Fluentd configuration is stored in the `ConfigMap` object. It is effectively a set of configuration +files that are merged together. You can learn about fluentd configuration on the [official +site](http://docs.fluentd.org). + +Imagine you want to add a new parsing logic to the configuration, so that fluentd can understand +default Python logging format. An appropriate fluentd filter looks similar to this: + +``` + + type parser + format /^(?\w):(?\w):(?.*)/ + reserve_data true + suppress_parse_error_log true + key_name log + +``` + +Now you have to put it in the configuration and make Stackdriver Logging agents pick it up. +Get the current version of the Stackdriver Logging `ConfigMap` in your cluster +by running the following command: + +```shell +kubectl get ds fluentd-gcp-v2.0 --namespace kube-system -o yaml > fluentd-gcp-ds.yaml +``` + +Then in the value for the key `containers.input.conf` insert a new filter right after +the `source` section. **Note:** order is important. + +Updating `ConfigMap` in the apiserver is more complicated than updating `DaemonSet`. It's better +to consider `ConfigMap` to be immutable. Then, in order to update the configuration, you should +create `ConfigMap` with a new name and then change `DaemonSet` to point to it +using [guide above](#changing-daemonset-parameters). + +### Adding fluentd plugins + +Fluentd is written in Ruby and allows to extend its capabilities using +[plugins](http://www.fluentd.org/plugins). If you want to use a plugin, which is not included +in the default Stackdriver Logging container image, you have to build a custom image. Imagine +you want to add Kafka sink for messages from a particular container for additional processing. +You can re-use the default [container image sources](https://github.com/kubernetes/contrib/tree/master/fluentd/fluentd-gcp-image) +with minor changes: + +* Change Makefile to point to your container repository, e.g. `PREFIX=gcr.io/`. +* Add your dependency to the Gemfile, for example `gem 'fluent-plugin-kafka'`. + +Then run `make build push` from this directory. After updating `DaemonSet` to pick up the +new image, you can use the plugin you installed in the fluentd configuration.