diff --git a/serving/fluentd/README.md b/serving/fluentd/README.md new file mode 100644 index 000000000..da62bfb5f --- /dev/null +++ b/serving/fluentd/README.md @@ -0,0 +1,47 @@ +# Fluentd Docker Image on Knative Serving + +Knative Serving uses a [Fluentd](https://www.fluentd.org/) docker image to +collect logs. Operators can customize their own Fluentd docker image and +configuration to define logging output. + +## Requirements + +Knative requires the customized Fluentd docker image with the following plugins +installed: + +* [fluentd](https://github.com/fluent/fluentd) >= v0.14.0 +* [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) >= + 1.0.0 AND < 2.1.0: To enrich log entries with Kubernetes metadata. +* [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions) >= + 0.0.9: To combine multi-line exception stack traces logs into one log entry. +* [fluent-plugin-multi-format-parser](https://github.com/repeatedly/fluent-plugin-multi-format-parser) >= + 1.0.0: To detect log format as Json or plain text. + +## Sample images + +Operators can use any Docker image which meets the requirements above and +includes the desired output plugin. Two examples below: + +### Send logs to Elasticsearch + +Operators can use +[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image) +which includes +[fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) +that allows sending logs to a Elasticsearch service. + +### Send logs to Stackdriver + +This sample [Dockerfile](stackdriver/Dockerfile) is based on +[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch). +It additionally adds one more plugin - +[fluent-plugin-google-cloud](https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud) +which allows sending logs to Stackdriver. + +Operators can build this image and push it to a container registry which their +Kubernetes cluster has access to. See [Setting Up A Logging Plugin](/serving/setting-up-a-logging-plugin.md) +for details. **NOTE**: Operators need to add credentials +file the stackdriver agent needs to the docker image if their Knative Serving is +not built on a GCP based cluster or they want to send logs to another GCP +project. See [here](https://cloud.google.com/logging/docs/agent/authorization) +for more information. diff --git a/serving/fluentd/stackdriver/Dockerfile b/serving/fluentd/stackdriver/Dockerfile new file mode 100644 index 000000000..f6132768b --- /dev/null +++ b/serving/fluentd/stackdriver/Dockerfile @@ -0,0 +1,15 @@ +FROM k8s.gcr.io/fluentd-elasticsearch:v2.0.4 + +RUN BUILD_DEPS="make gcc g++ libc6-dev ruby-dev libffi-dev" \ + && clean-install $BUILD_DEPS \ + ca-certificates \ + libjemalloc1 \ + liblz4-1 \ + ruby \ + && echo 'gem: --no-document' >> /etc/gemrc \ + && gem install fluent-plugin-google-cloud -v 0.6.19 \ + && rm -rf /tmp/* \ + /var/lib/apt/lists/* \ + /usr/lib/ruby/gems/*/cache/*.gem \ + /var/log/* \ + /var/tmp/* diff --git a/serving/installing-logging-metrics-traces.md b/serving/installing-logging-metrics-traces.md index 633218bd2..690512426 100644 --- a/serving/installing-logging-metrics-traces.md +++ b/serving/installing-logging-metrics-traces.md @@ -7,58 +7,80 @@ or You can install only one of these two setups and side-by-side installation of these two are not supported. +## Before you begin + +The following instructions assume that you cloned the Knative Serving repository. +To clone the repository, run the following commands: + + ```shell + git clone https://github.com/knative/serving knative-serving + cd knative-serving + git checkout v0.1.1 + ``` + ## Elasticsearch, Kibana, Prometheus & Grafana Setup If you installed the [full Knative release](../install/README.md#installing-knative), -skip this step and continue to -[Create Elasticsearch Indices](#create-elasticsearch-indices) +the monitoring component is already installed and you can skip down to the +[Create Elasticsearch Indices](#create-elasticsearch-indices) section. -- Install Knative monitoring components from the root of the [Serving repository](https://github.com/knative/serving): +To configure and setup monitoring: - ```shell - kubectl apply --recursive --filename config/monitoring/100-common \ +1. Choose a container image that meets the + [Fluentd image requirements](fluentd/README.md#requirements). For example, you can use the + public image [k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image). + Or you can create a custom one and upload the image to a container registry + which your cluster has read access to. +1. Follow the instructions in + ["Setting up a logging plugin"](setting-up-a-logging-plugin.md#Configuring) + to configure the Elasticsearch components settings. +1. Install Knative monitoring components by running the following command from the root directory of + [knative/serving](https://github.com/knative/serving) repository: + + ```shell + kubectl apply --recursive --filename config/monitoring/100-common \ --filename config/monitoring/150-elasticsearch \ --filename third_party/config/monitoring/common \ --filename third_party/config/monitoring/elasticsearch \ --filename config/monitoring/200-common \ --filename config/monitoring/200-common/100-istio.yaml - ``` + ``` -- The installation is complete when logging & monitoring components are all - reported `Running` or `Completed`: + The installation is complete when logging & monitoring components are all + reported `Running` or `Completed`: - ```shell - kubectl get pods --namespace monitoring --watch - ``` + ```shell + kubectl get pods --namespace monitoring --watch + ``` - ``` - NAME READY STATUS RESTARTS AGE - elasticsearch-logging-0 1/1 Running 0 2d - elasticsearch-logging-1 1/1 Running 0 2d - fluentd-ds-5kc85 1/1 Running 0 2d - fluentd-ds-vhrcq 1/1 Running 0 2d - fluentd-ds-xghk9 1/1 Running 0 2d - grafana-798cf569ff-v4q74 1/1 Running 0 2d - kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d - kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d - node-exporter-cr6bh 2/2 Running 0 2d - node-exporter-mf6k7 2/2 Running 0 2d - node-exporter-rhzr7 2/2 Running 0 2d - prometheus-system-0 1/1 Running 0 2d - prometheus-system-1 1/1 Running 0 2d - ``` + ``` + NAME READY STATUS RESTARTS AGE + elasticsearch-logging-0 1/1 Running 0 2d + elasticsearch-logging-1 1/1 Running 0 2d + fluentd-ds-5kc85 1/1 Running 0 2d + fluentd-ds-vhrcq 1/1 Running 0 2d + fluentd-ds-xghk9 1/1 Running 0 2d + grafana-798cf569ff-v4q74 1/1 Running 0 2d + kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d + kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d + node-exporter-cr6bh 2/2 Running 0 2d + node-exporter-mf6k7 2/2 Running 0 2d + node-exporter-rhzr7 2/2 Running 0 2d + prometheus-system-0 1/1 Running 0 2d + prometheus-system-1 1/1 Running 0 2d + ``` CTRL+C to exit watch. ### Create Elasticsearch Indices -To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will create two indices in Elasticsearch using `Logstash` for application logs and `Zipkin` +To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will +create two indices in Elasticsearch using `Logstash` for application logs and `Zipkin` for request traces. -- To open the Kibana UI (the visualization tool for - [Elasticsearch](https://info.elastic.co)), start a local proxy with the - following command: +- To open the Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co)), + you must start a local proxy by running the following command: ```shell kubectl proxy @@ -84,25 +106,23 @@ for request traces. ## Stackdriver, Prometheus & Grafana Setup -If your Knative Serving is not built on a Google Cloud Platform (GCP) based -cluster or you want to send logs to another GCP project, you need to build your -own Fluentd image and modify the configuration first. See +You must configure and build your own Fluentd image if either of the following are true: -1. Install - [Fluentd image on Knative Serving](https://github.com/knative/serving/blob/master/image/fluentd/README.md). -2. [Set up a logging plugin](setting-up-a-logging-plugin.md). -3. Install Knative monitoring components: + * Your Knative Serving component is not hosted on a Google Cloud Platform (GCP) based cluster. + * You want to send logs to another GCP project. - a. Clone the Knative Serving repository: - - ```shell - git clone https://github.com/knative/serving knative-serving - cd knative-serving - git checkout v0.1.1 - ``` - - b. Apply the monitoring manifests: +To configure and setup monitoring: +1. Choose a container image that meets the + [Fluentd image requirements](fluentd/README.md#requirements). For example, you can use a + public image. Or you can create a custom one and upload the image to a + container registry which your cluster has read access to. +2. Follow the instructions in + ["Setting up a logging plugin"](setting-up-a-logging-plugin.md#Configuring) + to configure the stackdriver components settings. +3. Install Knative monitoring components by running the following command from the root directory of + [knative/serving](https://github.com/knative/serving) repository: + ```shell kubectl apply --recursive --filename config/monitoring/100-common \ --filename config/monitoring/150-stackdriver-prod \ @@ -111,6 +131,31 @@ own Fluentd image and modify the configuration first. See --filename config/monitoring/200-common/100-istio.yaml ``` + The installation is complete when logging & monitoring components are all + reported `Running` or `Completed`: + + ```shell + kubectl get pods --namespace monitoring --watch + ``` + + ``` + NAME READY STATUS RESTARTS AGE + elasticsearch-logging-0 1/1 Running 0 2d + elasticsearch-logging-1 1/1 Running 0 2d + fluentd-ds-5kc85 1/1 Running 0 2d + fluentd-ds-vhrcq 1/1 Running 0 2d + fluentd-ds-xghk9 1/1 Running 0 2d + grafana-798cf569ff-v4q74 1/1 Running 0 2d + kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d + kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d + node-exporter-cr6bh 2/2 Running 0 2d + node-exporter-mf6k7 2/2 Running 0 2d + node-exporter-rhzr7 2/2 Running 0 2d + prometheus-system-0 1/1 Running 0 2d + prometheus-system-1 1/1 Running 0 2d + ``` + + CTRL+C to exit watch. ## Learn More - Learn more about accessing logs, metrics, and traces: diff --git a/serving/setting-up-a-logging-plugin.md b/serving/setting-up-a-logging-plugin.md index 7ab4b3888..4137b8ab3 100644 --- a/serving/setting-up-a-logging-plugin.md +++ b/serving/setting-up-a-logging-plugin.md @@ -11,6 +11,10 @@ a new process will be required. In order to minimize the effort for a future migration, we recommend only changing the output configuration of Fluentd and leaving the rest intact. +**NOTE**: All the files mentioned below are in +[knative/serving](https://github.com/knative/serving) repository. You run the +commands mentioned below from the root directory of `knative/serving`. + ## Configuring ### Configure the DaemonSet for stdout/stderr logs @@ -19,15 +23,14 @@ Operators can do the following steps to configure the Fluentd DaemonSet for collecting `stdout/stderr` logs from the containers: 1. Replace `900.output.conf` part in - [fluentd-configmap.yaml](/config/monitoring/fluentd-configmap.yaml) with the - desired output configuration. Knative provides samples for sending logs to - Elasticsearch or Stackdriver. Developers can simply choose one of `150-*` - from [/config/monitoring](/config/monitoring) or override any with other - configuration. -1. Replace the `image` field of `fluentd-ds` container - in [fluentd-ds.yaml](/third_party/config/monitoring/common/fluentd/fluentd-ds.yaml) + [100-fluentd-configmap.yaml](https://https://github.com/knative/serving/blob/master/config/monitoring/150-elasticsearch/100-fluentd-configmap.yaml) with the + desired output configuration. Knative provides a sample for sending logs to + Elasticsearch or Stackdriver. Developers can simply use `100-fluentd-configmap.yaml` + or override any with other configuration. +2. Replace the `image` field of `fluentd-ds` container + in [fluentd-ds.yaml](https://github.com/knative/serving/blob/master/third_party/config/monitoring/common/kubernetes/fluentd/fluentd-ds.yaml) with the Fluentd image including the desired Fluentd output plugin. - See [here](/image/fluentd/README.md) for the requirements of Flunetd image + See [here](image/fluentd/README.md) for the requirements of Flunetd image on Knative. ### Configure the Sidecar for log files under /var/log @@ -38,15 +41,15 @@ collecting log files under `/var/log`. An is in process to get rid of the sidecar. The steps to configure are: 1. Replace `logging.fluentd-sidecar-output-config` flag in - [config-observability](/config/config-observability.yaml) with the + [config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml) with the desired output configuration. **NOTE**: The Fluentd DaemonSet is in `monitoring` namespace while the Fluentd sidecar is in the namespace same with the app. There may be small differences between the configuration for DaemonSet and sidecar even though the desired backends are the same. 1. Replace `logging.fluentd-sidecar-image` flag in - [config-observability](/config/config-observability.yaml) with the Fluentd image including the - desired Fluentd output plugin. In theory, this is the same - with the one for Fluentd DaemonSet. + [config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml) + with the Fluentd image including the desired Fluentd output plugin. In theory, + this is the same with the one for Fluentd DaemonSet. ## Deploying @@ -78,9 +81,20 @@ the Elasticsearch and Kibana services. Knative provides this sample: kubectl apply --recursive --filename third_party/config/monitoring/elasticsearch ``` -See [here](/config/monitoring/README.md) for deploying the whole Knative +See [here](/serving/installing-logging-metrics-traces.md) for deploying the whole Knative monitoring components. +## Uninstalling + +To uninstall a logging plugin, run: + +```shell +kubectl delete -f \ + -f third_party/config/monitoring/common/kubernetes/fluentd/fluentd-ds.yaml \ + -f config/monitoring/200-common/100-fluentd.yaml + -f config/monitoring/200-common/100-istio.yaml +``` + --- Except as otherwise noted, the content of this page is licensed under the