knative
/
docs
mirror of https://github.com/knative/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Move the document and fluent image sample from serving repo to docs repo (#259) 

* add doccument

* add reasons for plugins

* Formatted file

* add more details

* fix links

* Remove typo and duplicate steps +

+ made a few edits for clarity and consistency
This commit is contained in:
Yanwei Guo 2018-09-18 15:23:19 -07:00 committed by Knative Prow Robot
parent d630a8fa81
commit 12e3364b2d
4 changed files with 181 additions and 60 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
serving/fluentd/README.md Normal file
View File

@ -0,0 +1,47 @@
# Fluentd Docker Image on Knative Serving
Knative Serving uses a [Fluentd](https://www.fluentd.org/) docker image to
collect logs. Operators can customize their own Fluentd docker image and
configuration to define logging output.
## Requirements
Knative requires the customized Fluentd docker image with the following plugins
installed:
* [fluentd](https://github.com/fluent/fluentd) >= v0.14.0
* [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) >=
1.0.0 AND < 2.1.0: To enrich log entries with Kubernetes metadata.
* [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions) >=
0.0.9: To combine multi-line exception stack traces logs into one log entry.
* [fluent-plugin-multi-format-parser](https://github.com/repeatedly/fluent-plugin-multi-format-parser) >=
1.0.0: To detect log format as Json or plain text.
## Sample images
Operators can use any Docker image which meets the requirements above and
includes the desired output plugin. Two examples below:
### Send logs to Elasticsearch
Operators can use
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image)
which includes
[fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch)
that allows sending logs to a Elasticsearch service.
### Send logs to Stackdriver
This sample [Dockerfile](stackdriver/Dockerfile) is based on
[k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch).
It additionally adds one more plugin -
[fluent-plugin-google-cloud](https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud)
which allows sending logs to Stackdriver.
Operators can build this image and push it to a container registry which their
Kubernetes cluster has access to. See [Setting Up A Logging Plugin](/serving/setting-up-a-logging-plugin.md)
for details. **NOTE**: Operators need to add credentials
file the stackdriver agent needs to the docker image if their Knative Serving is
not built on a GCP based cluster or they want to send logs to another GCP
project. See [here](https://cloud.google.com/logging/docs/agent/authorization)
for more information.

15
serving/fluentd/stackdriver/Dockerfile Normal file
View File

@ -0,0 +1,15 @@
FROM k8s.gcr.io/fluentd-elasticsearch:v2.0.4
RUN BUILD_DEPS="make gcc g++ libc6-dev ruby-dev libffi-dev" \
&& clean-install $BUILD_DEPS \
ca-certificates \
libjemalloc1 \
liblz4-1 \
ruby \
&& echo 'gem: --no-document' >> /etc/gemrc \
&& gem install fluent-plugin-google-cloud -v 0.6.19 \
&& rm -rf /tmp/* \
/var/lib/apt/lists/* \
/usr/lib/ruby/gems/*/cache/*.gem \
/var/log/* \
/var/tmp/*

139
serving/installing-logging-metrics-traces.md
View File

@ -7,58 +7,80 @@ or
You can install only one of these two setups and side-by-side installation of
these two are not supported.
## Before you begin
The following instructions assume that you cloned the Knative Serving repository.
To clone the repository, run the following commands:
```shell
git clone https://github.com/knative/serving knative-serving
cd knative-serving
git checkout v0.1.1
```
## Elasticsearch, Kibana, Prometheus & Grafana Setup
If you installed the
[full Knative release](../install/README.md#installing-knative),
skip this step and continue to
[Create Elasticsearch Indices](#create-elasticsearch-indices)
the monitoring component is already installed and you can skip down to the
[Create Elasticsearch Indices](#create-elasticsearch-indices) section.
- Install Knative monitoring components from the root of the [Serving repository](https://github.com/knative/serving):
To configure and setup monitoring:
```shell
kubectl apply --recursive --filename config/monitoring/100-common \
1. Choose a container image that meets the
[Fluentd image requirements](fluentd/README.md#requirements). For example, you can use the
public image [k8s.gcr.io/fluentd-elasticsearch:v2.0.4](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image).
Or you can create a custom one and upload the image to a container registry
which your cluster has read access to.
1. Follow the instructions in
["Setting up a logging plugin"](setting-up-a-logging-plugin.md#Configuring)
to configure the Elasticsearch components settings.
1. Install Knative monitoring components by running the following command from the root directory of
[knative/serving](https://github.com/knative/serving) repository:
```shell
kubectl apply --recursive --filename config/monitoring/100-common \
--filename config/monitoring/150-elasticsearch \
--filename third_party/config/monitoring/common \
--filename third_party/config/monitoring/elasticsearch \
--filename config/monitoring/200-common \
--filename config/monitoring/200-common/100-istio.yaml
```
```
- The installation is complete when logging & monitoring components are all
reported `Running` or `Completed`:
The installation is complete when logging & monitoring components are all
reported `Running` or `Completed`:
```shell
kubectl get pods --namespace monitoring --watch
```
```shell
kubectl get pods --namespace monitoring --watch
```
```
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-0 1/1 Running 0 2d
elasticsearch-logging-1 1/1 Running 0 2d
fluentd-ds-5kc85 1/1 Running 0 2d
fluentd-ds-vhrcq 1/1 Running 0 2d
fluentd-ds-xghk9 1/1 Running 0 2d
grafana-798cf569ff-v4q74 1/1 Running 0 2d
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
node-exporter-cr6bh 2/2 Running 0 2d
node-exporter-mf6k7 2/2 Running 0 2d
node-exporter-rhzr7 2/2 Running 0 2d
prometheus-system-0 1/1 Running 0 2d
prometheus-system-1 1/1 Running 0 2d
```
```
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-0 1/1 Running 0 2d
elasticsearch-logging-1 1/1 Running 0 2d
fluentd-ds-5kc85 1/1 Running 0 2d
fluentd-ds-vhrcq 1/1 Running 0 2d
fluentd-ds-xghk9 1/1 Running 0 2d
grafana-798cf569ff-v4q74 1/1 Running 0 2d
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
node-exporter-cr6bh 2/2 Running 0 2d
node-exporter-mf6k7 2/2 Running 0 2d
node-exporter-rhzr7 2/2 Running 0 2d
prometheus-system-0 1/1 Running 0 2d
prometheus-system-1 1/1 Running 0 2d
```
CTRL+C to exit watch.
### Create Elasticsearch Indices
To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will create two indices in Elasticsearch using `Logstash` for application logs and `Zipkin`
To visualize logs with Kibana, you need to set which Elasticsearch indices to explore. We will
create two indices in Elasticsearch using `Logstash` for application logs and `Zipkin`
for request traces.
- To open the Kibana UI (the visualization tool for
[Elasticsearch](https://info.elastic.co)), start a local proxy with the
following command:
- To open the Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co)),
you must start a local proxy by running the following command:
```shell
kubectl proxy
@ -84,25 +106,23 @@ for request traces.
## Stackdriver, Prometheus & Grafana Setup
If your Knative Serving is not built on a Google Cloud Platform (GCP) based
cluster or you want to send logs to another GCP project, you need to build your
own Fluentd image and modify the configuration first. See
You must configure and build your own Fluentd image if either of the following are true:
1. Install
[Fluentd image on Knative Serving](https://github.com/knative/serving/blob/master/image/fluentd/README.md).
2. [Set up a logging plugin](setting-up-a-logging-plugin.md).
3. Install Knative monitoring components:
* Your Knative Serving component is not hosted on a Google Cloud Platform (GCP) based cluster.
* You want to send logs to another GCP project.
a. Clone the Knative Serving repository:
```shell
git clone https://github.com/knative/serving knative-serving
cd knative-serving
git checkout v0.1.1
```
b. Apply the monitoring manifests:
To configure and setup monitoring:
1. Choose a container image that meets the
[Fluentd image requirements](fluentd/README.md#requirements). For example, you can use a
public image. Or you can create a custom one and upload the image to a
container registry which your cluster has read access to.
2. Follow the instructions in
["Setting up a logging plugin"](setting-up-a-logging-plugin.md#Configuring)
to configure the stackdriver components settings.
3. Install Knative monitoring components by running the following command from the root directory of
[knative/serving](https://github.com/knative/serving) repository:
```shell
kubectl apply --recursive --filename config/monitoring/100-common \
--filename config/monitoring/150-stackdriver-prod \
@ -111,6 +131,31 @@ own Fluentd image and modify the configuration first. See
--filename config/monitoring/200-common/100-istio.yaml
```
The installation is complete when logging & monitoring components are all
reported `Running` or `Completed`:
```shell
kubectl get pods --namespace monitoring --watch
```
```
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-0 1/1 Running 0 2d
elasticsearch-logging-1 1/1 Running 0 2d
fluentd-ds-5kc85 1/1 Running 0 2d
fluentd-ds-vhrcq 1/1 Running 0 2d
fluentd-ds-xghk9 1/1 Running 0 2d
grafana-798cf569ff-v4q74 1/1 Running 0 2d
kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d
node-exporter-cr6bh 2/2 Running 0 2d
node-exporter-mf6k7 2/2 Running 0 2d
node-exporter-rhzr7 2/2 Running 0 2d
prometheus-system-0 1/1 Running 0 2d
prometheus-system-1 1/1 Running 0 2d
```
CTRL+C to exit watch.
## Learn More
- Learn more about accessing logs, metrics, and traces:

40
serving/setting-up-a-logging-plugin.md
View File

@ -11,6 +11,10 @@ a new process will be required. In order to minimize the effort for a future
migration, we recommend only changing the output configuration of Fluentd and
leaving the rest intact.
**NOTE**: All the files mentioned below are in
[knative/serving](https://github.com/knative/serving) repository. You run the
commands mentioned below from the root directory of `knative/serving`.
## Configuring
### Configure the DaemonSet for stdout/stderr logs
@ -19,15 +23,14 @@ Operators can do the following steps to configure the Fluentd DaemonSet for
collecting `stdout/stderr` logs from the containers:
1. Replace `900.output.conf` part in
[fluentd-configmap.yaml](/config/monitoring/fluentd-configmap.yaml) with the
desired output configuration. Knative provides samples for sending logs to
Elasticsearch or Stackdriver. Developers can simply choose one of `150-*`
from [/config/monitoring](/config/monitoring) or override any with other
configuration.
1. Replace the `image` field of `fluentd-ds` container
in [fluentd-ds.yaml](/third_party/config/monitoring/common/fluentd/fluentd-ds.yaml)
[100-fluentd-configmap.yaml](https://https://github.com/knative/serving/blob/master/config/monitoring/150-elasticsearch/100-fluentd-configmap.yaml) with the
desired output configuration. Knative provides a sample for sending logs to
Elasticsearch or Stackdriver. Developers can simply use `100-fluentd-configmap.yaml`
or override any with other configuration.
2. Replace the `image` field of `fluentd-ds` container
in [fluentd-ds.yaml](https://github.com/knative/serving/blob/master/third_party/config/monitoring/common/kubernetes/fluentd/fluentd-ds.yaml)
with the Fluentd image including the desired Fluentd output plugin.
See [here](/image/fluentd/README.md) for the requirements of Flunetd image
See [here](image/fluentd/README.md) for the requirements of Flunetd image
on Knative.
### Configure the Sidecar for log files under /var/log
@ -38,15 +41,15 @@ collecting log files under `/var/log`. An
is in process to get rid of the sidecar. The steps to configure are:
1. Replace `logging.fluentd-sidecar-output-config` flag in
[config-observability](/config/config-observability.yaml) with the
[config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml) with the
desired output configuration. **NOTE**: The Fluentd DaemonSet is in
`monitoring` namespace while the Fluentd sidecar is in the namespace same with
the app. There may be small differences between the configuration for DaemonSet
and sidecar even though the desired backends are the same.
1. Replace `logging.fluentd-sidecar-image` flag in
[config-observability](/config/config-observability.yaml) with the Fluentd image including the
desired Fluentd output plugin. In theory, this is the same
with the one for Fluentd DaemonSet.
[config-observability](https://github.com/knative/serving/blob/master/config/config-observability.yaml)
with the Fluentd image including the desired Fluentd output plugin. In theory,
this is the same with the one for Fluentd DaemonSet.
## Deploying
@ -78,9 +81,20 @@ the Elasticsearch and Kibana services. Knative provides this sample:
kubectl apply --recursive --filename third_party/config/monitoring/elasticsearch
```
See [here](/config/monitoring/README.md) for deploying the whole Knative
See [here](/serving/installing-logging-metrics-traces.md) for deploying the whole Knative
monitoring components.
## Uninstalling
To uninstall a logging plugin, run:
```shell
kubectl delete -f <the-fluentd-config-for-daemonset> \
-f third_party/config/monitoring/common/kubernetes/fluentd/fluentd-ds.yaml \
-f config/monitoring/200-common/100-fluentd.yaml
-f config/monitoring/200-common/100-istio.yaml
```
---
Except as otherwise noted, the content of this page is licensed under the