来自2022 Top 100全球软件案例研究峰会的分享。
+主要内容包括：
+来自云原生社区可观测性直播分享。
+主要内容包括：
+来自datafun talk的分享。
+网易从2015年就开始了云原生的探索与实践，作为可观测性的重要一环，日志平台也经历了从主机到容器的演进，支撑了集团内各业务部门的大规模云原生化改造。
+本文会讲述在这个过程中我们遇到的问题，如何演进和改造，并从中沉淀了哪些经验与最佳实践。
主要内容包括：
+参考：
+ +time.after()。使用time.NewTicker()代替，并及时清理tickerlog.Info()、log.Error()。不允许使用fmt.Printf()、fmt.Println()等go内置fmt方法。项目日志系统采用 zerologPANIC和FATAL区别
+PANIC最终调用的是panic(msg)
+FATAL最终调用的是os.Exit(1)
+yaml:"cleanDataTimeout,omitempty"，代替yaml:"clean_data_timeout,omitempty"Copyright-text
+/*
+Copyright 2022 Loggie Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+推荐参考已有的组件开发一个新的组件
+// 生命周期接口
+type Lifecycle interface {
+ // 初始化，例如初始化Kafka连接
+ Init(context Context)
+ // 启动运行，例如开始消费Kafka
+ Start()
+ // 停止
+ Stop()
+}
+
+// 描述接口
+type Describable interface {
+ // 类别，例如source
+ Category() Category
+ // 类型，例如kafka
+ Type() Type
+ // 自定义描述
+ String() string
+}
+
+// 配置获取接口
+type Config interface {
+ // 获取配置
+ Config() interface{}
+}
+
+// 组件接口
+type Component interface {
+ // 生命周期管理
+ Lifecycle
+ // 描述管理
+ Describable
+ // 配置管理
+ Config
+}
+source组件对接数据源输入，开发一个新的source插件需要实现如下接口 +
// source组件接口
+type Source interface {
+ Component
+ Producer
+ // 提交接口，确认sink端成功然后提交
+ Commit(events []Event)
+}
+
+// 生产接口，source组件需要实现
+type Producer interface {
+ // 对接数据源
+ ProductLoop(productFunc ProductFunc)
+}
+sink组件对接输出端，开发一个新的sink插件需要实现如下接口 +
// sink组件接口
+type Sink interface {
+ Component
+ Consumer
+}
+
+// 消费接口，sink组件需要实现
+type Consumer interface {
+ // 对接输出端
+ Consume(batch Batch) Result
+}
+interceptor组件对事件进行拦截处理，开发一个新的interceptor插件需要实现如下接口 +
// interceptor组件接口
+type Interceptor interface {
+ Component
+ // 拦截处理
+ Intercept(invoker Invoker, invocation Invocation) api.Result
+}
+Note
+请注意新增的组件需要放到pkg/include/include.go的import当中注册
+访问https://github.com/loggie-io/loggie，点击右上角的Fork，fork到自己的github仓库中。
+将fork后的工程clone到本地： +
cd ${go-workspace}
+
+git clone git@github.com:${yourAccountId}/loggie.git
+在本地工程中，添加upstream remote：
+git remote add upstream https://github.com/loggie-io/loggie.git
+git remote set-url --push upstream no_push
+可以使用git remote -v查看，origin为你个人账号fork的git地址，upstream为Loggie官方仓库地址，uptream仅为本地同步代码使用，正常无push权限。
基于最新的main分支，创建出你的分支，通常以feat/fix/chore等开头。
+git checkout -b feat-mywork
+在该分支开发完后，待提交前，建议保持代码和上游一致：
+git pull upstream -r
+git commit
+
+git push origin feat-mywork
+Commit规范
+通常我们使用<type>(<scope>): <subject>的形式：
<type>:（必须）
<scope>:（可选）
<subject>: （必须）即commit的内容描述
访问你的个人github账号里fork的Loggie项目，并Open pull request。
+尽量描述清楚PR的背景、目的和改动点。
+如果有相关Issues，需关联。
提交前请检查：
+你可以在OWNERS找人，或者默认添加loggie-robot账号作为你PR的Reviewers或Assigneees。
+ + + + + + +本文介绍如何在本地开发和调试Loggie工程。
+Loggie使用Golang编写，请确保本地有Golang开发环境。
+go version
+请使用自己的github账号fork Loggie工程。然后git clone到本地。
+git clone git@github.com:<Account>/loggie.git
+正常情况下，本地无特殊依赖。
+针对不同的功能场景，需要根据具体情况安装或者使用外部的依赖。
+建议本地运行Kubernetes进行部署和管理。
+搭建Kubernetes
+推荐使用Kind。
+部署依赖组件
+推荐使用Helm来部署。相关的组件可使用公开的Helm仓库来添加和部署。
+本地Loggie连接Kubernetes
+如果需要使用LogConfig等CRD，或者调试Kubernetes下的使用方式，需要在系统配置中，enable Kubernetes Discovery。
+loggie.yml
+ discovery:
+ enabled: true
+ kubernetes:
+ kubeconfig: ${home}/.kube/config
+指定kubeconfig连接本地的Kubernetes APIServer。
+另外，Agent形态的Loggie，会只监听本节点的Kubernetes Pod事件，需要在Loggie启动参数中添加-meta.nodeName来模拟所在的节点。
例如，可以使用kubectl get node查看所有节点名称：
NAME STATUS ROLES AGE VERSION
+kind-control-plane Ready master 2y50d v1.21.0
+-meta.nodeName=kind-control-plane，可以使Loggie启动后被认为部署在该节点。
+Loggie默认启动后，输出的日志格式为JSON，如果你不习惯，可在启动参数中添加-log.jsonFormat=false来关闭。
本地尽量使用模拟的方式，合理利用dev source和dev sink。比如：
+示例：在pipelines中，使用dev sink，查看最终输出发送至下游的数据。
+pipelines.yml
+ sink:
+ type: "dev"
+ printEvents: true
+ codec:
+ pretty: true
+release-vA.B，例如release-v1.1，这里只使用版本前两位v1.1.0v1.1.1确认版本需要的所有功能已经合入，main分支为最新提交：
+git checkout main
+git pull upstream main -r
+基于main分支创建新的release分支，例如release-v1.1：
+
git checkout -b release-v${A.B}
+基于该分支填写CHANGELOG，提交至工程中合入
+git add .
+git commit -m'Release: add v${A.B.C} changelog'
+git push upstream release-v${A.B.C}
+release Loggie-installationation部署脚本工程，和上述1，2步骤相同
+请注意：
+基于release分支进行回归测试。
+release分支push至Github后，会触发action进行镜像构建，使用该镜像进行回归测试。
+这里的测试会使用loggie-installationation部署脚本里相应的release分支，同时回归验证部署脚本正确性。
+如果测试发现Bug，提交PR合入release分支，重新进行回归。
测试通过后，基于该release分支打上相应的版本tag，比如v1.1.0
+
git pull upstream release-v${A.B.C}
+git tag v${A.B.C}
+git push v${A.B.C}
+loggie-v${A.B.C}.tgz helm chart包。helm package ./helm-chart即可。提PR将release分支合入main，可等稳定一段时间后再合入，注意Loggie工程和Loggie-installation部署脚本均需要。
+请确保本地有kubectl和helm可执行命令。
+ +VERSION=v1.4.0
+helm pull https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz && tar xvzf loggie-$VERSION.tgz
+<VERSION>替换成具体的版本号。比如v1.4.0。 可从release tag里找。
+进入chart目录： +
cd installation/helm-chart
+查看values.yml，并根据实际情况进行修改。
+目前可配置的参数如下所示：
+image: loggieio/loggie:main
+resources:
+ limits:
+ cpu: 2
+ memory: 2Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+extraArgs: {}
+extraArgs:
+ log.level: debug
+ log.jsonFormat: false
+extraVolumeMounts:
+ - mountPath: /var/log/pods
+ name: podlogs
+ - mountPath: /var/lib/kubelet/pods
+ name: kubelet
+ - mountPath: /var/lib/docker
+ name: docker
+
+
+extraVolumes:
+ - hostPath:
+ path: /var/log/pods
+ type: DirectoryOrCreate
+ name: podlogs
+ - hostPath:
+ path: /var/lib/kubelet/pods
+ type: DirectoryOrCreate
+ name: kubelet
+ - hostPath:
+ path: /var/lib/docker
+ type: DirectoryOrCreate
+ name: docker
+这里简单列举一下采集什么样的日志需要挂载什么路径：
+采集 stdout 标准输出：Loggie会从/var/log/pods下采集，所以Loggie需要挂载：
+volumeMounts:
+- mountPath: /var/log/pods
+ name: podlogs
+- mountPath: /var/lib/docker
+ name: docker
+
+volumes:
+- hostPath:
+ path: /var/log/pods
+ type: DirectoryOrCreate
+name: podlogs
+- hostPath:
+ path: /var/lib/docker
+ type: DirectoryOrCreate
+name: docker
+但是有可能/var/log/pods下这些日志文件会软链接到docker的root路径下，默认为/var/lib/docker，这个时候，需要同样挂载/var/lib/docker这个路径到Loggie中：
volumeMounts:
+- mountPath: /var/log/pods
+ name: podlogs
+
+volumes:
+- hostPath:
+ path: /var/log/pods
+ type: DirectoryOrCreate
+name: podlogs
+另外，如果非docker运行时，比如使用containerd，无需挂载/var/lib/docker，Loggie会从/var/log/pods中寻找实际的标准输出路径。
采集业务Pod使用 HostPath 挂载的日志：比如业务统一将日志挂载到了节点的/data/logs路径下，则需要挂载挂载该路径:
volumeMounts:
+ - mountPath: /data/logs
+ name: logs
+
+ volumes:
+ - hostPath:
+ path: /data/logs
+ type: DirectoryOrCreate
+ name: logs
+/var/lib/kubelet/pods路径下，所以需要Loggie挂载该路径。如果环境的kubelet修改了该路径配置，这里需要同步修改:volumeMounts:
+ - mountPath: /var/lib/kubelet/pods
+ name: kubelet
+
+ volumes:
+ - hostPath:
+ path: /var/lib/kubelet/pods
+ type: DirectoryOrCreate
+ name: kubelet
+采集业务Pod 无挂载 ，同时设置了rootFsCollectionEnabled: true，Loggie会自动从docker的rootfs里找到容器里的实际路径，此时需要挂载docker的root路径：
volumeMounts:
+- mountPath: /var/lib/docker
+ name: docker
+
+volumes:
+- hostPath:
+ path: /var/lib/docker
+ type: DirectoryOrCreate
+name: docker
+/data/docker，则挂载为：
+volumeMounts:
+- mountPath: /data/docker
+ name: docker
+
+volumes:
+- hostPath:
+ path: /data/docker
+ type: DirectoryOrCreate
+name: docker
+其他：
+/data/loggie--{{ template "loggie.name" . }}目录。nodeSelector: {}
+
+affinity: {}
+# podAntiAffinity:
+# requiredDuringSchedulingIgnoredDuringExecution:
+# - labelSelector:
+# matchExpressions:
+# - key: app
+# operator: In
+# values:
+# - loggie
+# topologyKey: "kubernetes.io/hostname"
+tolerations: []
+# - effect: NoExecute
+# operator: Exists
+# - effect: NoSchedule
+# operator: Exists
+updateStrategy:
+ type: RollingUpdate
+RollingUpdate或者OnDelete。
+config:
+ loggie:
+ reload:
+ enabled: true
+ period: 10s
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ sink: ~
+ discovery:
+ enabled: true
+ kubernetes:
+ containerRuntime: containerd
+ fields:
+ container.name: containername
+ logConfig: logconfig
+ namespace: namespace
+ node.name: nodename
+ pod.name: podname
+ http:
+ enabled: true
+ port: 9196
+containerRuntime: containerd，指定容器运行时。
+如果Loggie希望接收其他服务发送的数据，需要将自身的服务通过service暴露出来。
+正常情况下，使用Agent模式的Loggie只需要暴露自身管理端口。
+servicePorts:
+ - name: monitor
+ port: 9196
+ targetPort: 9196
+初次部署，我们指定部署在loggie namespace下，并让helm自动创建该namespace。
+
helm install loggie ./ -nloggie --create-namespace
+如果你的环境中已经创建了loggie namespace，可以忽略其中的-nloggie和--create-namespace参数。当然，你也可以使用自己的namespace，将其中loggie替换即可。
Kubernetes版本问题
+failed to install CRD crds/crds.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1"
+rm loggie/crds/crds.yaml，重新install。
+执行完后，通过使用helm命令来查看部署状态： +
helm list -nloggie
+NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
+loggie loggie 1 2021-11-30 18:06:16.976334232 +0800 CST deployed loggie-v0.1.0 v0.1.0
+同时也可以通过kubectl命令查看Pod是否已经被创建。
+
kubectl -nloggie get po
+loggie-sxxwh 1/1 Running 0 5m21s 10.244.0.5 kind-control-plane <none> <none>
+部署Aggregator基本和Agent一致，在helm chart中我们提供了aggregator config部分，只需改成enabled: true即可。
helm chart中提供了statefulSet的部署方式，你也可以根据需求修改成deployment等方式。
+同时，请注意在values.yaml中根据情况增加：
+servicePorts:
+- name: grpc
+ port: 6066
+ targetPort: 6066
+config:
+ loggie:
+ discovery:
+ enabled: true
+ kubernetes:
+ cluster: aggregator
+执行部署命令参考： +
helm install loggie-aggregator ./ -nloggie-aggregator --create-namespace
+Note
+Loggie中转机同样可以使用Deployment或者StatefulSet来部署，请参考DaemonSet自行修改helm chart。
+Loggie使用Golang编译成二进制，可根据自身需求对接各类部署系统。
+这里我们提供一个使用systemd部署Loggie的参考。
目前release仅包含GOOS=linux GOARCH=amd64生成的二进制可执行文件。其他系统和架构，请自行基于源码交叉编译。
+VERSION=v1.4.0
+mkdir /opt/loggie && curl https://github.com/loggie-io/loggie/releases/download/$VERSION/loggie -o /opt/loggie/loggie && chmod +x /opt/loggie/loggie
+请将以上的<VERSION>替换成具体的版本号。
请根据实际需求创建配置，以下为参考：
+loggie.yml
+cat << EOF > /opt/loggie/loggie.yml
+loggie:
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ sink: ~
+
+ reload:
+ enabled: true
+ period: 10s
+
+ http:
+ enabled: true
+ port: 9196
+EOF
+pipelines.yml
+cat << EOF > /opt/loggie/pipelines.yml
+pipelines:
+ - name: local
+ sources:
+ - type: file
+ name: demo
+ paths:
+ - /tmp/log/*.log
+ sink:
+ type: dev
+ printEvents: true
+ codec:
+ pretty: true
+EOF
+cat << EOF > /lib/systemd/system/loggie.service
+[Unit]
+Description=Loggie
+Documentation=https://loggie-io.github.io/docs/getting-started/install/node/
+
+[Service]
+MemoryMax=200M
+ExecStart=/opt/loggie/loggie -config.system=/opt/loggie/loggie.yml -config.pipeline=/opt/loggie/pipelines.yml
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+EOF
+首先生效配置： +
systemctl daemon-reload
+然后设置为开机启动： +
systemctl enable loggie
+接着就可以正式启动Loggie了：
+systemctl start loggie
+启动后，你可以随时查看进程状态： +
systemctl status loggie
+Loggie是一个基于Golang的轻量级、高性能、云原生日志采集Agent和中转处理Aggregator，支持多Pipeline和组件热插拔，提供了：
+
一栈式日志解决方案：同时支持日志中转、过滤、解析、切分、日志报警等
云原生的日志形态：快速便捷的容器日志采集方式，原生的Kubernetes动态配置下发
生产级的特性：Loggie吸收了我们长期的大规模运维经验，形成了全方位的可观测性、快速排障、异常预警、自动化运维能力
欢迎来到Loggie的世界。
了解¶想知道什么是Loggie？了解一下「核心概念」。
+如果你希望快速体验Loggie，请参考「快速上手」。
+有更多兴趣？想知道我们为什么研发Loggie、Loggie有什么优势、和同类项目的对比？请戳这里。
+
使用¶如果你希望尝试一下Loggie，但不知道如何开始？
+可以先从整体思考一下：如何采集业务的日志，如何形成一个完善并适合实际需求的日志架构？ +这里有一个参考。
+接着，你可以选择适合的业务「场景」，查看Loggie在Kubernetes下的最佳实践。
+这里还涉及到如何根据实际情况「部署」Loggie。
+另外，想查看具体组件的使用方式，请参考「配置」。
+
解答¶在使用Loggie的时候遇到问题？ +请提issues或者联系我们。
+微信扫码加入Loggie讨论群：（老的微信已满，未通过的请重新扫码）
+
参与¶如果你对Loggie的具体实现感兴趣，想参与Loggie开源的研发？想自研插件？请看「开发手册」。
+ + + + + + +下面将带你演示在一个Kubernetes集群中，通过创建LogConfig CRD快速采集Pod的日志。
+可以使用现有Kubernetes集群，或者部署Kubernetes。本地推荐使用Kind搭建Kubernetes集群。
+本文的操作需要在本地使用:
+ +请确保本地有kubectl和helm可执行命令。
+你可以在 installation 页面查看所有发布的部署chart。
+可以选择：
+VERSION=v1.4.0
+helm pull https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz && tar xvzf loggie-$VERSION.tgz
+<VERSION>替换成具体的版本号。
+然后部署安装：
+helm install loggie ./loggie -nloggie --create-namespace
+当然你也可以：
+helm install loggie -nloggie --create-namespace https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz
+<VERSION>替换成具体的版本号。
+想使用其他版本镜像？
+为了方便体验最新的Fix和特性，我们提供了main分支每次合并后的镜像版本，可通过 这里 进行选择。
+同时你可以在helm install命令中增加--set image=loggieio/loggie:vX.Y.Z来指定具体的Loggie镜像。
部署有问题？
+如果尝试部署后出现问题，或者在你的环境中以下演示操作未成功，请参考Kubernetes下部署Loggie，修改相关配置。
+Loggie定义了Kubernetes CRD LogConfig，一个LogConfig表示采集一类Pods的日志采集任务。
+我们先创建一个Pod用于被采集日志的对象。 +
kubectl create deploy nginx --image=nginx
+接着，我们创建一个Loggie定义的CRD Sink实例，表明日志发送的后端。
+为了方便演示，这里我们将日志发送至Loggie Agent自身的日志中并打印。
cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: default
+spec:
+ sink: |
+ type: dev
+ printEvents: true
+EOF
+可以通过kubectl get sink查看到已创建的Sink。
Loggie定义CRD LogConfig，表示一个日志采集任务。我们创建一个LogConfig示例如下所示：
+cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: nginx
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+ pipeline:
+ sources: |
+ - type: file
+ name: mylog
+ paths:
+ - stdout
+ sinkRef: default
+EOF
+可以看到，上面使用了sinkRef引用了刚才创建的sink default CR。当然，我们还可以直接在Logconfig中使用sink字段，示例如下：
cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: nginx
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+ pipeline:
+ sources: |
+ - type: file
+ name: mylog
+ paths:
+ - stdout
+ sink: |
+ type: dev
+ printEvents: true
+ codec:
+ type: json
+ pretty: true
+EOF
+创建完之后，我们可以使用kubectl get lgc查看到创建的CRD实例。
同时，我们还可以通过kubectl describe lgc nginx查看LogConfig的事件，以获取最新的状态。
Events:
+ Type Reason Age From Message
+ ---- ------ ---- ---- -------
+ Normal syncSuccess 52s loggie/kind-control-plane Sync type pod [nginx-6799fc88d8-5cb67] success
+上面的nginx LogConfig通过其中的spec.selector来匹配采集哪些Pod的日志，这里我们使用app: nginx选择了刚才创建的nginx Pod。
+spec.pipeline则表示Loggie的Pipeline配置，我们只采集容器标准输出的日志，所以在paths中填写stdout即可。
首先找到所在的nginx pod节点： +
kubectl get po -owide -l app=nginx
+然后我们找到该节点的Loggie： +
kubectl -nloggie get po -owide |grep ${node}
+kubectl -nloggie logs -f ${logge-pod}
+上文只是一个简单的快速演示，部署出现问题或者想了解更多Kubernetes下Loggie如何使用？
+下面我们将演示一个最简单的采集节点日志文件的场景。
+请找一台Linux服务器节点，下载Loggie二进制可执行文件 +
VERSION=v1.4.0
+curl -LJ https://github.com/loggie-io/loggie/releases/download/$VERSION/loggie -o loggie
+请将以上的<VERSION>替换成具体的版本号。
我们先使用dev sink将file source采集的日志文件打印到标准输出，复制以下内容为pipelines.yml文件：
+cat << EOF > pipelines.yml
+pipelines:
+ - name: demo
+ sources:
+ - type: file
+ name: mylog
+ paths:
+ - "/var/log/*.log"
+ sink:
+ type: dev
+ printEvents: true
+EOF
+/var/log目录下满足*.log匹配规则的日志文件。文件采集后，文件会被发送至dev sink输出源，该sink仅仅将采集的文件打印到标准输出。
+pipeline文件表示我们想要的输入、输出等业务相关的配置，除了pipeline配置文件外，Loggie还需要有一个全局的配置文件。
+// loggie.yml
+cat << EOF > loggie.yml
+loggie:
+ reload:
+ enabled: true
+ period: 10s
+EOF
+这里我们只展示了一个比较简单的配置，表示打开loggie的动态配置reload功能，同时间隔检查时间为10s。
+在节点上增加以上两个配置文件后，我们就可以开始启动Loggie了。
+./loggie -config.system=./loggie.yml -config.pipeline=./pipelines.yml -log.jsonFormat=false
+启动参数里，填入上面的loggie.yml和pipelines.yml的文件路径。
+看到正常的启动日志后，表明Loggie就开始正常的工作了。同时节点/var/log/*.log下的日志文件，都会被打印到标准输出。
+ + + + + + +Loggie是一个日志采集Agent和中转处理的Aggregator，包含多个Pipeline管道，每个Pipeline又由Source输入、Interceptor拦截处理和Sink输出组成。
+基于这种插件式设计，Loggie并不局限在日志采集，通过配置不同的Source/Interceptor/Sink，Loggie可以组合实现各种不同的功能。
+简单起见，这里我们从采集日志开始。
+选择你的部署环境
：
+ 可扩展、热插拔
+配置不同的Source/Interceptor/Sink，获得中转、过滤、解析、切分等能力，可快速自研插件
+
+ 强隔离
+多Pipeline设计，减少互相干扰，支持同时发送多个不同数据源
+
+ 轻量级、高性能
+基于Golang，极少的资源占用，强大的吞吐性能，满足各类场景需求
+
+ 可靠性保障
+完善的日志可观测性，原生Prometheus metrics支持，还有限流、背压等Interceptor
+
+ 云原生
+配置中心集成Kubernetes，创建CRD实例即可采集容器或节点日志
+
+ 不仅仅是日志
+数据流基于Source/Interceptor/Sink模型，可采集各种可观测性事件，扩展更多的可能性
+GET /api/v1/help/log
+查询该Loggie Agent的日志采集状态
+示例：
+
+ /api/v1/help/log?pipeline=test&status=pending
+| 参数名称 | +说明 | +参数类型 | +备注 | +
|---|---|---|---|
| fdStatus | +文件句柄状态 | ++ | + |
| fdStatus.activeFdCount | +活跃的fd个数 | +int | ++ |
| fdStatus.inActiveFdCount | +不活跃的fd个数 | +int | ++ |
| fileStatus | +文件采集状态 | ++ | + |
fileStatus.pipeline.<name> |
+管道状态，对应配置中的pipeline name，参考下面的pipeline参数 | +map | ++ |
| 参数名称 | +说明 | +参数类型 | +备注 | +
|---|---|---|---|
source.<name> |
+pipeline中source的状态，参考下面的source参数 | +map | ++ |
| 参数名称 | +说明 | +参数类型 | +备注 | +
|---|---|---|---|
| paths | +配置文件source中定义的path | +string数组 | ++ |
| detail | +pipeline中source的状态 | +数组 | ++ |
| detail[n].filename | +文件名称 | +string | ++ |
| detail[n].offset | +采集进度offset | +int | ++ |
| detail[n].size | +文件大小 | +int | ++ |
| detail[n].modify | +文件最近的更新时间 | +int | +unix milliseconds | +
| detail[n].ignored | +文件是否被忽略（由file source中的ignoreOlder配置决定） | +bool | ++ |
Example
+{
+ "fdStatus": {
+ "activeFdCount": 0,
+ "inActiveFdCount": 1
+ },
+ "fileStatus": {
+ "pipeline": {
+ "local": {
+ "source": {
+ "demo": {
+ "paths": [
+ "/tmp/log/*.log"
+ ],
+ "detail": [
+ {
+ "filename": "/tmp/log/access.log",
+ "offset": 469,
+ "size": 469,
+ "modify": 1673436846523,
+ "ignored": false
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+}
+Cluster级别CRD，可用于：
+Example
+apiVersion: loggie.io/v1beta1
+kind: ClusterLogConfig
+metadata:
+ name: test
+spec:
+ selector:
+ type: node
+ nodeSelector:
+ nodepool: test
+ pipeline:
+ sources: |
+ - type: file
+ name: messages
+ paths:
+ - /var/log/messages
+ sinkRef: default
+表示Pipeline配置适用的范围
+通过Pipeline配置选择一批Pod进行日志采集
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| labelSelector | +map | +必填 | ++ | 通过该label来匹配Pods，支持使用*来匹配所有的value，比如app: '*' |
+
Example
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+app: nginx的所有Pod的日志。
+Warning
+在type: pod时，下面的Pipeline只能使用file source，此时的场景只能是采集日志。
下发Pipeline配置至该批节点。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| nodeSelector | +map | +必填 | ++ | 通过label选择下发配置的node | +
Example
+spec:
+ selector:
+ type: node
+ nodeSelector:
+ nodepool: test
+nodepool: test的所有node上。
+下发Pipeline配置至某个Loggie集群，通常需要配合cluster字段指定集群名使用。
Example
+spec:
+ selector:
+ cluster: aggregator
+ type: cluster
+cluster为aggregator的Loggie集群。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cluster | +string | +非必填 | +"" | +表示配置指定下发的Loggie集群。当部署多套Loggie时，和全局系统配置discovery.kubernetes.cluster配套使用 |
+
配置和LogConfig一致。
+ + + + + + +表示一个interceptor组。用于在LogConfig/ClusterLogConfig中被引用。
+Example
+apiVersion: loggie.io/v1beta1
+kind: Interceptor
+metadata:
+ name: default
+spec:
+ interceptors: |
+ - type: rateLimit
+ qps: 90000
+使用"|"符号表示一整段interceptors配置列表，和Pipelines里的配置一致。
+ + + + + + +namespace级别CRD，表示一个日志采集任务，用于采集Pod容器日志。
+Example
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: tomcat
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: tomcat
+ pipeline:
+ sources: |
+ - type: file
+ name: common
+ paths:
+ - stdout
+ sink: |
+ type: dev
+ printEvents: false
+ interceptors: |
+ - type: rateLimit
+ qps: 90000
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: nginx
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+ pipeline:
+ sources: |
+ - type: file
+ name: mylog
+ paths:
+ - stdout
+ sinkRef: default
+ interceptorRef: default
+表示Pipeline配置适用的范围，可以选择采集一批Pods的日志
+采集Pods日志
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| labelSelector | +map | +必填 | ++ | 通过该label来匹配Pods，支持使用*来匹配所有的value，比如app: '*' |
+
Example
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+app: nginx的所有Pods日志。
+Warning
+在type: pod时，下面的Pipeline只支持使用file source，此时的场景只能是采集日志。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cluster | +string | +非必填 | +"" | +表示配置指定下发的Loggie集群。当部署多套Loggie时，和全局系统配置discovery.kubernetes.cluster配套使用 |
+
表示一个Pipeline，不支持填写多个Pipeline。
+和在配置文件中Pipelines的区别在：
+｜表示保留换行符在LogConfig中，如果type: pod，file source新增几个专门针对容器化的参数：
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| containerName | +string | +非必填 | ++ | 表示指定采集的容器名称，建议在Pod里包含多个容器时填写 | +
| excludeContainerPatterns | +string数组 | +非必填 | ++ | 排除的容器名称，使用正则表达式形式 | +
非必填, 将Pod中的信息加入到Fields中
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| labelKey | +string数组 | +非必填 | ++ | 指定增加的Pod上的Label Key值，比如Pod上包含Label: app: demo，此处填写labelKey: app，此时会将Pod上的app: demo label增加到file source fields中，采集到的日志会被加上该label信息。适用于匹配的Pod的label存在不一致的场景。支持配置为"*"的方式获取所有的label |
+
| annotationKey | +string数组 | +非必填 | ++ | 和上面labelKey类似，注入的为Pod Annoatation的值，支持配置为"*"的方式获取所有的annotation | +
| env | +string数组 | +非必填 | ++ | 和上面labelKey类似，注入的为Pod Env环境变量的值，支持配置为"*"的方式获取所有的env | +
| reformatKeys | ++ | 非必填 | ++ | 重新格式化key | +
| reformatKeys.label | +fmt参数数组 | +非必填 | ++ | 重新格式化label key | +
| reformatKeys.annotation | +fmt参数数组 | +非必填 | ++ | 重新格式化annotation key | +
| reformatKeys.env | +fmt参数数组 | +非必填 | ++ | 重新格式化env key | +
fmt参数
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| regex | +string | +非必填 | ++ | 匹配的正则表达式 | +
| replace | +string | +非必填 | ++ | 重新渲染的格式 | +
reformatKeys
+假设pod labels为aa.bb/foo=bar
+配置reformatKeys如下：
+
matchFields:
+ reformatKeys:
+ label:
+ - regex: aa.bb/(.*)
+ replace: pre-${1}
+pre-foo=bar
+Example
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: nginx
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: nginx
+ pipeline:
+ sources: |
+ - type: file
+ name: mylog
+ containerName: nginx
+ matchFields:
+ labelKey: ["app"]
+ paths:
+ - stdout
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| interceptors | +string | +非必填 | ++ | 表示该Pipeline的interceptor，使用方式和以上sources类似 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sink | +string | +非必填 | ++ | 表示该Pipeline的sink，使用方式和以上的sources类似 | +
如果你希望sink和interceptor可以在不同的ClusterLogConfig/LogConfig间复用，则可以使用以下ref的方式：
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sinkRef | +string | +非必填 | ++ | 表示该Pipeline引用的Sink CR | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| interceptorRef | +string | +非必填 | ++ | 表示该Pipeline引用的Interceptor CR | +
表示一个sink配置。用于在LogConfig/ClusterLogConfig中被引用。
+Example
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: default
+spec:
+ sink: |
+ type: elasticsearch
+ index: "loggie"
+ hosts: ["elasticsearch-master.default.svc:9200"]
+使用"|"符号表示一个sink配置，和Pipelines里的配置一致。
+ + + + + + +-config.from: 默认为file，即默认使用文件的配置方式。可选：env，此时会从环境变量中读取配置（此时不支持reload）。-config.system: 默认为loggie.yml，表示指定Loggie系统配置的路径和文件名称。-config.from=env，则为system配置的环境变量名称）-config.pipeline: 默认为pipelines.yml，表示Pipeline配置文件所在的路径，需要填写符合glob匹配的路径，比如具体的路径和文件名/etc/loggie/pipelines.yml，或者glob匹配的方式，比如/etc/loggie/*.yml。-config.from=env，则为pipeline配置的环境变量名称）Warning
+值得注意的是，如果config.pipeline=/etc/loggie，glob匹配会认为/etc/loggie为/etc目录下的loggie文件，而不是匹配/etc/loggie目录下的文件，请避免类似的设置方式
-meta.nodeName：默认情况下会使用系统的hostname，在Kubernetes部署中会使用Downward API来注入nodeName。一般情况下不需要单独配置-log.level: 日志级别，默认为info，可配置为debug、info、warn和error-log.jsonFormat: 是否将日志输出为json格式，默认为false-log.enableStdout: 是否输出标准输出日志，默认为true-log.enableFile: 是否输出日志文件，默认为false，即不输出日志文件，默认打印到标准输出-log.directory: 日志文件的路径，默认为/var/log，当log.enableFile=true时生效-log.filename: 日志文件的名称，默认为loggie.log，一般同log.directory搭配使用-log.maxSize: 日志轮转的时候，最大的文件大小，默认为1024MB-log.maxBackups: 日志轮转最多保留的文件个数，默认为3-log.maxAge: 日志轮转最大保留的天数，默认为7-log.timeFormat: 每行日志输出的时间格式，默认格式为2006-01-02 15:04:05-log.noColor: 输出是否有颜色美化，默认为false，即有颜色美化，对于从标准输出查看日志比较方便。如果输出到日志文件中，建议设置为true，避免引入额外的格式。Info
+Loggie的日志轮转使用lumberjack库
+defaults用于设置Pipelines配置中的默认值。当Pipeline中没有设置值时生效，或者用于覆盖默认的参数。
+Example
+defaults:
+ sources:
+ - type: file
+ watcher:
+ cleanFiles:
+ maxHistory: 10
+ sink:
+ type: dev
+ printEvents: true
+和Pipeline中的source一致。当Pipelines配置了相同type的source时，会覆盖其中未填写字段的默认值。
+比如: +
sources:
+ - type: file
+ watcher:
+ cleanFiles:
+ maxHistory: 10
+和Pipeline中的sink一致，如果集群只需要设置一个全局的sink输出源，则只需要在这里配置一次，避免在每个Pipeline中填写。
+defaults中配置的interceptors会添加到pipeline中定义的interceptors中，但pipeline中的interceptor会覆盖defaults中的同一个type的interceptor。
+如果不希望覆盖相同类型的interceptor，而是添加相同type的interceptor，依次进行处理，需要额外填写name字段，进行唯一性标识。
在defaults中已经定义normalize interceptor如下:
+defaults:
+ interceptors:
+ - type: normalize
+ processor:
+ - addMeta: ~
+如果在pipelines中定义如下normalize interceptor：
+pipelines:
+ interceptors:
+ - type: normalize
+ processor:
+ - drop:
+ ...
+此时defaults中的normalize interceptor会被覆盖，不会生效。
+如果我们希望先执行defaults中的normalize interceptor，接着再执行pipeline中的normalize interceptor，可以在defaults中改为：
+defaults:
+ interceptors:
+ - type: normalize
+ name: global # 用于区分pipelines中的normalize
+ order: 500 # 默认normalize的order值为900，这里定义一个相对较小值，可控制先执行defaults中的normalize
+ processor:
+ - addMeta: ~
+Loggie会默认设置metric、maxbytes、retry3个系统内置interceptors。
+如果需要添加其他的默认interceptors，会覆盖掉以上的内置interceptors，所以强烈建议此时将内置interceptors加上，除非你确认不需要以上系统内置的interceptors。
默认为channel queue。
+ + + + + + +服务发现和配置下发相关的配置。目前主要为Kubernetes相关全局配置。
+Example
+discovery:
+ enabled: true
+ kubernetes:
+ # Choose: docker or containerd
+ containerRuntime: containerd
+ # Collect log files inside the container from the root filesystem of the container, no need to mount the volume
+ rootFsCollectionEnabled: false
+ # Automatically parse and convert the wrapped container standard output format into the original log content
+ parseStdout: false
+ # If set to true, it means that the pipeline configuration generated does not contain specific Pod paths and meta information,
+ # and these data will be dynamically obtained by the file source, thereby reducing the number of configuration changes and reloads.
+ dynamicContainerLog: false
+ # Automatically add fields when selector.type is pod in logconfig/clusterlogconfig
+ typePodFields:
+ logconfig: "${_k8s.logconfig}"
+ namespace: "${_k8s.pod.namespace}"
+ nodename: "${_k8s.node.name}"
+ podname: "${_k8s.pod.name}"
+ containername: "${_k8s.pod.container.name}"
+ typeNodeFields:
+ nodename: "${_k8s.node.name}"
+ clusterlogconfig: "${_k8s.clusterlogconfig}"
+ os: "${_k8s.node.nodeInfo.osImage}"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +false | +是否开启服务发现配置下发模块 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cluster | +string | +非必填 | +"" | +标识Loggie集群名称。Loggie支持在一个Kubernetes集群中部署多套Loggie，可以通过在LogConfig CRD中指定selector.cluster，指定配置下发的Loggie集群 |
+
| kubeconfig | +string | +非必填 | ++ | 指定请求Kubernetes集群API的kubeconfig文件。通常在Loggie部署到Kubernetes集群中无需填写，此时为inCluster模式。如果Loggie部署在Kubernetes集群外（例如本地调试时），需要指定该kubeconfig文件。 | +
| master | +string | +非必填 | ++ | 指定请求Kubernetes集群API的master地址，inCluster模式一般无需填写 | +
| containerRuntime | +string | +非必填 | +docker | +容器运行时，可选docker、containerd、none |
+
| rootFsCollectionEnabled | +bool | +非必填 | +false | +是否开启采集root filesystem里的日志，用于不挂载日志volume的场景 | +
| parseStdout | +bool | +非必填 | +false | +是否开启自动提取容器标准输出原始内容 | +
| dynamicContainerLog | +bool | +非必填 | +false | +是否开启动态容器日志配置，打开后配置文件不会渲染具体的path和动态fields字段，可以有效避免大规模容器化场景里Pod变动从而导致配置的频繁渲染，显著减少reload次数，特别是在单节点的Pod个数较多和使用clusterlogconfig匹配大量的Pod时。一般建议设置为true。 | +
| kubeletRootDir | +string | +非必填 | +/var/lib/kubelet | +kubelet的root路径 | +
| podLogDirPrefix | +string | +非必填 | +/var/log/pods | +kubernetes默认放置的pod标准输出路径 | +
| typePodFields | +map | +非必填 | ++ | 当logconfig/clusterlogconfig里selector为type: pod时，自动添加的kubernetes相关元信息，key即为添加的元信息key，value请使用${_k8s.XX}的方式指定，同时支持填写固定值的key:value字段 |
+
| typeNodeFields | +map | +非必填 | ++ | 当logconfig/clusterlogconfig里selector为type: node时，自动添加的kubernetes相关元信息，key即为添加的元信息key，value请使用${_k8s.XX}的方式指定，同时支持填写固定值的key:value字段 |
+
"${_k8s.XX}"的方式可填写以下参数：
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| ${_k8s.logconfig} | +string | +非必填 | ++ | 添加logConfig name作为元信息 | +
| ${_k8s.node.name} | +string | +非必填 | ++ | 添加所在节点node name作为元信息 | +
| ${_k8s.node.ip} | +string | +非必填 | ++ | 添加所在节点node ip作为元信息 | +
| ${_k8s.pod.namespace} | +string | +非必填 | ++ | 添加namespace作为元信息 | +
| ${_k8s.pod.name} | +string | +非必填 | ++ | 添加pod name作为元信息 | +
| ${_k8s.pod.ip} | +string | +非必填 | ++ | 添加pod ip作为元信息 | +
| ${_k8s.pod.uid} | +string | +非必填 | ++ | 添加pod uid作为元信息 | +
| ${_k8s.pod.container.name} | +string | +非必填 | ++ | 添加container name作为元信息 | +
| ${_k8s.pod.container.id} | +string | +非必填 | ++ | 添加container id作为元信息 | +
| ${_k8s.pod.container.image} | +string | +非必填 | ++ | 添加container image作为元信息 | +
| ${_k8s.workload.kind} | +string | +非必填 | ++ | 添加pod归属的Deployment/Statefulset/DaemonSet/Job作为元信息 | +
| ${_k8s.workload.name} | +string | +非必填 | ++ | 添加pod归属的Deployment/Statefulset/DaemonSet/Job名称作为元信息 | +
"${_k8s.XX}"的方式可填写以下参数：
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| ${_k8s.clusterlogconfig} | +string | +非必填 | ++ | 添加clusterlogconfig name作为元信息 | +
| ${_k8s.node.name} | +string | +非必填 | ++ | 添加所在节点node name作为元信息 | +
| ${_k8s.node.addresses.InternalIP} | +string | +非必填 | ++ | 添加node InternalIP 作为元信息 | +
| ${_k8s.node.addresses.Hostname} | +string | +非必填 | ++ | 添加node Hostname作为元信息 | +
| ${_k8s.node.nodeInfo.kernelVersion} | +string | +非必填 | ++ | 添加node kernelVersion作为元信息 | +
| ${_k8s.node.nodeInfo.osImage} | +string | +非必填 | ++ | 添加node osImage作为元信息 | +
| ${_k8s.node.nodeInfo.containerRuntimeVersion} | +string | +非必填 | ++ | 添加node containerRuntimeVersion作为元信息 | +
| ${_k8s.node.nodeInfo.kubeletVersion} | +string | +非必填 | ++ | 添加node kubeletVersion作为元信息 | +
| ${_k8s.node.nodeInfo.kubeProxyVersion} | +string | +非必填 | ++ | 添加node kubeProxyVersion作为元信息 | +
| ${_k8s.node.nodeInfo.operatingSystem} | +string | +非必填 | ++ | 添加node operatingSystem作为元信息 | +
| ${_k8s.node.nodeInfo.architecture} | +string | +非必填 | ++ | 添加node architecture作为元信息 | +
${_k8s.node.labels.<key>} |
+string | +非必填 | ++ | 添加node的某个label作为元信息，其中的<key>请替换成具体的label key |
+
${_k8s.node.annotations.<key>} |
+string | +非必填 | ++ | 添加node的某个annotation作为元信息，其中的<key>请替换成具体的annotation key |
+
Loggie自身提供的Http端口，包含监控metrics，内部运维等接口。
+Example
+http:
+ enabled: true
+ host: "0.0.0.0"
+ port: 9196
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +false | +是否开启http | +
| host | +string | +非必填 | +0.0.0.0 | +http监听的host | +
| port | +http | +非必填 | +9196 | +http监控的端口 | +
Tips
+一般推荐打开http端口，但是如果Kubernetes或者容器部署时，使用hostNetwork请注意端口冲突，以及监听host是否暴露给公网、是否有安全隐患。
+reload会定时检查启动参数-config.pipeline指定的配置文件，如果检测到文件内容发生变动，会重启有变动配置的Pipeline，未被修改的Pipeline不会受影响。
Example
+ reload:
+ enabled: true
+ period: 10s
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +false | +是否开启reload | +
| period | +time.Duration | +非必填 | +10s | +reload检测配置的时间间隔，不建议设置过短，否则可能会增加CPU的消耗 | +
在很多场景下，我们往往需要动态的获取event里的某个字段。 +比如：
+${a.b}的方式来取值。以下event为例:
+
{
+ "fields": {
+ "svc": "test",
+ }
+}
+kafka sink topic配置为：log-${fields.svc}，则最终渲染生成的为log-test。
Caution
+一般可以使用.点号来表示嵌套的字段。但是，如果字段本身就包括.号，则需要使用[]包围起来，避免误认为是一个嵌套的字段。
+比如：
+
{
+ "fields": {
+ "a.b": "demo",
+ }
+}
+${fields.[a.b]}来表示字段a.b。
+Loggie的配置主要分为两类：
+全局的系统配置，启动参数中使用-config.system指定，包含如下：
# loggie.yml
+loggie:
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ sink: ~
+
+ discovery:
+ enabled: false
+
+ reload:
+ enabled: true
+ period: 10s
+
+ defaults:
+ sink:
+ type: dev
+ sources:
+ - type: file
+ watcher:
+ cleanFiles:
+ maxHistory: 1
+ http:
+ enabled: true
+ port: 9196
+Pipeline的配置，通过启动参数-config.pipeline指定。表示队列使用的Source、Sink、Queue和Interceptor。
pipelines:
+- name: demo # pipeline name必填
+ sources:
+ - type: ${sourceType}
+ name: access # source中name必填
+ ...
+ interceptors:
+ - type: ${interceptorType}
+ ...
+ sink:
+ type: ${sinkType}
+ ...
+Loggie定义了以下几个CRD用于在Kubernetes集群环境里下发配置：
+LogConfig：namespace级别，表示一个Pipeline配置，可用于采集Pods的容器日志。
+ClusterLogConfig：cluster级别，表示一个Pipeline配置，包括集群级别的跨Namespace采集Pod容器日志，采集Node节点上的日志，以及为某个Loggie集群下发通用的pipeline配置。
+Sink：cluster级别，表示一个sink配置，可以在LogConfig/ClusterLogConfig中引用该Sink。
+Interceptors：cluster级别，表示一个interceptors组，可以在LogConfig中引用该interceptors组。
+Note
+ClusterLogConfig/LogConfig中的pipeline可以定义sink和interceptor，用于该pipeline的sink/interceptor。
+如果你希望在多个ClusterLogConfig/LogConfig中复用sink或者interceptor，可以创建Sink/Interceptor CR，在ClusterLogConfig/LogConfig中使用sinkRef/interceptorRef进行引用。
实时文件采集的监控，表示当前日志采集的进度与状态，包括文件的名称、采集进度、QPS等。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +10s | +listener消费处理数据的时间间隔 | +
| fieldsRef | +string数组 | +非必填 | ++ | 支持从source fields配置当中获取key value加入到metrics中为label | +
# HELP file size
+# TYPE loggie_filesource_file_size gauge
+loggie_filesource_file_size{pipeline="xxx", source="access", filename="/var/log/a.log"} 2048
+# HELP file offset
+# TYPE loggie_filesource_file_offset gauge
+loggie_filesource_file_offset{pipeline="xxx", source="access", filename="/var/log/a.log"} 1024
+# HELP current read line number
+# TYPE loggie_filesource_line_number gauge
+loggie_filesource_line_number{pipeline="xxx", source="access", filename="/var/log/a.log"} 20
+# HELP current read line qps
+# TYPE loggie_filesource_line_qps gauge
+loggie_filesource_line_qps{pipeline="xxx", source="access", filename="/var/log/a.log"} 48
+对文件采集情况的定时检查并暴露指标，包括文件名称、ackOffset、修改时间、大小等。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +5m | +定时检查间隔时间 | +
| checkUnFinishedTimeout | +time.Duration | +非必填 | +24h | +检查文件是否采集完毕的超时时间，如果检测到文件的最近修改时间为checkUnFinishedTimeout之前，同时文件的并未采集完毕，则会在metrics中被标记为unfinished状态，可用于检查是否有长时间未被采集的日志文件 |
+
# HELP file count total
+# TYPE loggie_filewatcher_total_file_count gauge
+loggie_filewatcher_total_file_count{} 20
+# HELP inactive file count
+# TYPE loggie_filewatcher_inactive_file_count gauge
+loggie_filewatcher_inactive_file_count{} 20
+文件级别包括了以下prometheus labels:
+checkUnFinishedTimeout时间ignore_older时间由于定时扫描的时间间隔period默认为5min，以下指标可能存在一定程度的延迟。
# HELP file size
+# TYPE loggie_filewatcher_file_size gauge
+loggie_filewatcher_file_size{pipeline="xxx", source="access", filename="/var/log/a.log", status="pending"} 2048
+# HELP file ack offset
+# TYPE loggie_filewatcher_file_ack_offset gauge
+loggie_filewatcher_file_ack_offset{pipeline="xxx", source="access", filename="/var/log/a.log", status="pending"} 1024
+# HELP file last modify timestamp
+# TYPE loggie_filewatcher_file_last_modify gauge
+loggie_filewatcher_file_last_modify{pipeline="xxx", source="access", filename="/var/log/a.log", status="pending"} 2343214422
+展示Loggie本身的一些信息。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +10s | +暴露指标的时间间隔 | +
# HELP loggie_info_stat Loggie info
+# TYPE loggie_info_stat gauge
+loggie_info_stat{version=v1.4} 1
+用于日志报警的发送。 +使用示例请参考日志报警。
+Example
+logAlert:
+ addr: [ "http://127.0.0.1:8080/loggie" ]
+ bufferSize: 100
+ batchTimeout: 10s
+ batchSize: 1
+ lineLimit: 10
+ template: |
+ *****
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addr | +string数组 | +必填 | ++ | 发送alert的http地址 | +
| bufferSize | +int | +非必填 | +100 | +日志报警发送的buffer大小，单位为报警事件个数 | +
| batchTimeout | +time.Duration | +非必填 | +10s | +每个报警发送batch的最大发送时间 | +
| batchSize | +int | +非必填 | +10 | +每个报警发送batch的最大包含报警请求个数 | +
| template | +string | +非必填 | ++ | 渲染发送的alert结构体的go template模板 | +
| timeout | +time.Duration | +非必填 | +30s | +发送alert的http timeout | +
| headers | +map | +非必填 | ++ | 发送alert的http header | +
| method | +string | +非必填 | +POST | +发送alert的http method, 如果不填put(不区分大小写)，都认为是POST | +
| lineLimit | +int | +非必填 | +10 | +多行日志采集情况下，每个alert中包含的最大日志行数 | +
监控事件总线，所有的组件都可以发出自己的metrics指标数据，由listeners消费处理。
+Example
+monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ sink: ~
+Loggie支持将metrics指标输出到日志中，可以通过logger配置。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| logger.enabled | +bool | +非必填 | +false | +是否开启 | +
| logger.period | +time.Duration | +非必填 | +10s | +指标打印的时间间隔，数据量较大时建议将间隔延长，如30s、5m | +
| logger.pretty | +bool | +非必填 | +false | +打印的指标json是否需要友好展示 | +
| logger.additionLogEnabled | +bool | +非必填 | +false | +是否需要将打印的指标单独输出到另外的日志文件中，在数据量比较多的情况下，如果我们配置的打印时间间隔较短，可以打开该开关，避免太多的metrics日志干扰 | +
| logger.additionLogConfig | ++ | 非必填 | ++ | 额外输出的日志配置参数 | +
| logger.additionLogConfig.directory | +bool | +非必填 | +/data/loggie/log | +额外输出的日志目录 | +
| logger.additionLogConfig.maxBackups | +int | +非必填 | +metrics.log | +日志轮转最多保留的文件个数，默认为3 | +
| logger.additionLogConfig.maxSize | +int | +非必填 | +1024 | +日志轮转的时候，最大的文件大小，单位为MB | +
| logger.additionLogConfig.maxAge | +int | +非必填 | +14 | +日志轮转最大保留的天数 | +
| logger.additionLogConfig.timeFormat | +string | +非必填 | +2006-01-02 15:04:05 | +每行日志输出的时间格式 | +
表示具体启动的listeners。
+配置不填写即为关闭，不启动该Listener，相关的的指标也不会被处理和暴露。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +10s | +listener消费处理数据的时间间隔 | +
# HELP queue capacity
+# TYPE loggie_queue_capacity gauge
+loggie_queue_capacity{pipeline="xxx", type="channel"} 2048
+# HELP queue size
+# TYPE loggie_queue_size gauge
+loggie_queue_size{pipeline="xxx", type="channel"} 2048
+# HELP how full is queue
+# TYPE loggie_queue_fill_percentage gauge
+loggie_queue_fill_percentage{pipeline="xxx", type="channel"} 50
+sink发送端的监控指标处理，包括发送成功的event数量、发送失败的event数量、event qps等
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +10s | +listener消费处理数据的时间间隔 | +
# HELP send event success count
+# TYPE loggie_sink_success_event gauge
+loggie_sink_success_event{pipeline="xxx", source="access"} 2048
+period时间段内，发送成功的event个数# HELP send event failed count
+# TYPE loggie_sink_failed_event gauge
+loggie_sink_failed_event{pipeline="xxx", source="access"} 2048
+period时间段内，发送失败的event个数# HELP send success event failed count
+# TYPE loggie_sink_event_qps gauge
+loggie_sink_event_qps{pipeline="xxx", source="access"} 2048
+period时间段内，发送的event QPSLoggie本身的CPU和Memory指标
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| period | +time.Duration | +非必填 | +10s | +listener消费处理数据的时间间隔 | +
# HELP loggie_sys_cpu_percent Loggie cpu percent
+# TYPE loggie_sys_cpu_percent gauge
+loggie_sys_cpu_percent 0.37
+# HELP loggie_sys_mem_rss Loggie memory rss bytes
+# TYPE loggie_sys_mem_rss gauge
+loggie_sys_mem_rss 2.5853952e+07
+用于从event中的某些字段（比如日志文件的路径中），获取到：
+pod.uidnamespace与pod.namecontainer.id以上3种任意其一的索引信息，此时Loggie可根据该索引查询到具体的Pod，并添加额外的kubernetes${node.name}、${namespace}、${pod.uid}、${pod.name}等元信息作加入到event中，用于后续的分析处理。
+属于source interceptor。
Example
+interceptors:
+- type: addK8sMeta
+ pattern: "/var/log/${pod.uid}/${pod.name}/"
+ addFields:
+ nodename: "${node.name}"
+ namespace: "${namespace}"
+ podname: "${pod.name}"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| pattern | +string | +必填 | ++ | 提取字段的匹配模型 | +
必须包含有：
+pod.uidnamespace与pod.namecontainer.id其中之一。
+比如：/var/log/${pod.uid}/${pod.name}/
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| patternFields | +string | +非必填 | +默认会从event中获取系统字段里的filename，此时需要使用file source | +从event中用于提取的pattern的字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fieldsName | +string | +非必填 | +kubernetes | +添加元信息的字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addFields | +map | +非必填 | ++ | 需要添加的元信息 | +
目前支持添加的元信息字段有：
+${cluster}：集群信息，为系统配置中discovery.kubernetes.cluster字段。${node.name}${namespace}${workload.kind}：Deployment/StatefulSet/DaemonSet/Job等${workload.name}：工作负载的名称${pod.uid}${pod.name}用于日志报警检测。
+属于source interceptor。
+使用示例请参考日志报警。
Example
+interceptors:
+- type: logAlert
+ matcher:
+ contains: ["error", "err"]
+ regexp: ['.*example.*']
+ ignore: ['.*INFO.*']
+ sendOnlyMatched: true
+ additions:
+ module: "loggie"
+ alertname: "alert-test"
+ cluster: "local-cluster"
+ namespace: "default"
+ advanced:
+ enabled: true
+ mode: [ "noData","regexp" ]
+ duration: 6h
+ matchType: "any"
+ rules:
+ - regexp: '(?<date>.*?) (?<time>[\S|\\.]+) (<status>[\S|\\.]+) (?<u>.*?) --- (?<thread>\[*?\]) (?<pkg>.*) : (?<message>(.|\n|\t)*)'
+ matchType: "any"
+ groups:
+ - key: status
+ operator: "eq"
+ value: WARN
+ - key: thread
+ operator: "eq"
+ value: 200
+ - regexp: '(?<date>.*?) (?<time>[\S|\\.]+) (?<status>[\S|\\.]+) (?<u>.*?) --- (?<thread>\[.*?\]) (?<pkg>.*) : (?<message>(.|\n|\t)*)'
+ matchType: "any"
+ groups:
+ - key: status
+ operator: "eq"
+ value: ERROR
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| matcher.contains | +string数组 | +非必填 | ++ | 日志数据包含字符串检测 | +
| matcher.regexp | +string数组 | +非必填 | ++ | 日志数据正则检测 | +
| matcher.target | +string | +非必填 | +body | +根据日志数据的该字段进行检测，如果进行日志切分或者drop body字段，请填写所需字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| ignore | +string数组 | +非必填 | ++ | 正则表达式，若匹配，则忽略这条日志，向下传递，可用于告警时排除某些日志 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| additions | +map | +非必填 | ++ | 发送alert时，额外添加的字段，会放在_additions字段中，可用作渲染。 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sendOnlyMatched | +bool | +非必填 | +false | +是否仅将匹配成功的数据发送至sink | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +false | +是否开启高级匹配模式 | +
| mode | +string列表 | +非必填 | ++ | 匹配模式 支持regexp和noData两种，可同时生效。 |
+
| duration | +time.Duration | +非必填 | ++ | noData模式必填，在一定时间内，没有日志会发出告警。 | +
| matchType | +string | +非必填 | ++ | regexp模式必填，可选any或者all，表示匹配任意或者全部规则rule | +
| rules | +Rule列表 | +非必填 | ++ | regexp模式必填，匹配规则列表 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| regexp | +string | +必填 | ++ | 正则分组表达式 | +
| matchType | +string | +必填 | ++ | 可选any或者all，表示匹配任意或者全部匹配组group | +
| groups | +group列表 | +必填 | ++ | 匹配组列表 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| key | +string | +必填 | ++ | 分组匹配之后的键值 | +
| operator | +string | +必填 | ++ | 操作符，目前支持eq，gt，lt | +
| value | +string | +必填 | ++ | 目标值 | +
对数据传输进行采样metics，提供给monitor eventbus中listener使用。
+系统内置，默认加载，属于sink interceptor。
Example
+interceptors:
+- type: metric
+（该interceptor后续不再维护，建议使用transformer替换）
+用于日志切分处理。
+属于source interceptor。可指定只被某些source使用。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| processors | +数组 | +必填 | +无 | +所有的处理processor列表 | +
配置的processor将按照顺序依次执行。
+Tips
+Loggie支持使用a.b的形式引用嵌套的字段。
+比如数据为:
+
{
+ "fields": {
+ "hello": "world"
+ }
+}
+fields.hello指定嵌套在fields里的hello: world。
+默认情况下，Loggie不会添加任何的系统内部信息到原始数据中。
+可通过addMeta添加系统内置字段发送给下游。
Note
+请注意，在pipeline中配置addMeta，只会影响该pipeline发送的所有数据，如果需要全局生效，请在defaults中配置normalize.addMeta。
+loggie:
+ defaults:
+ interceptors:
+ - type: normalize
+ name: global
+ processors:
+ - addMeta: ~
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| target | +string | +非必填 | +meta | +系统内置字段添加到event中的字段名 | +
将指定字段进行正则提取。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| regex.pattern | +string | +必填 | +无 | +正则解析规则 | +
| regex.target | +string | +非必填 | +body | +正则解析的目标字段 | +
| regex.ignoreError | +bool | +非必填 | +false | +是否忽略错误 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - regex:
+ pattern: '(?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)'
+使用以上的正则表达式，可以将以下示例的日志： +
10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] "GET / HTTP/1.1" 404 683
+"ip": "10.244.0.1",
+"id": "-",
+"u": "-",
+"time": "[13/Dec/2021:12:40:48 +0000]",
+"url": "\"GET / HTTP/1.1\"",
+"status": "404",
+"size": "683"
+具体配置的时候，建议先使用一些正则调试工具 (https://regex101.com/) 验证是否可以匹配。
+将指定字段json解析提取。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| jsonDecode.target | +string | +非必填 | +body | +json decode的目标字段 | +
| jsonDecode.ignoreError | +bool | +非必填 | +false | +是否忽略错误 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - jsonDecode: ~
+将指定字段通过分隔符进行提取。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| split.target | +string | +非必填 | +body | +split的目标字段 | +
| split.separator | +string | +必填 | +无 | +分隔符 | +
| split.max | +int | +非必填 | +-1 | +通过分割符分割后得到的最多的字段数 | +
| split.keys | +string数组 | +必填 | +无 | +分割后字段对应的key | +
| split.ignoreError | +bool | +非必填 | +false | +是否忽略错误 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - split:
+ separator: '|'
+ keys: ["time", "order", "service", "price"]
+2021-08-08|U12345|storeCenter|13.14
+"time": "2021-08-08"
+"order": "U12345"
+"service": "storeCenter"
+"price": 13.14
+interceptors:
+- type: normalize
+ processors:
+ - split:
+ separator: ' '
+ max: 2
+ keys: ["time", "content"]
+max参数，可以控制最多分割的字段。2021-08-08 U12345 storeCenter 13.14
+"time": "2021-08-08"
+"content": "U12345 storeCenter 13.14"
+丢弃指定字段。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| drop.targets | +string数组 | +必填 | +无 | +drop的字段 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - drop:
+ targets: ["id", "body"]
+重命名指定字段。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| rename.convert | +数组 | +必填 | +无 | ++ |
| rename.convert[n].from | +string | +必填 | +无 | +rename的目标 | +
| rename.convert[n].to | +string | +必填 | +无 | +rename后的名称 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - rename:
+ convert:
+ - from: "hello"
+ to: "world"
+新增字段。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| add.fields | +map | +必填 | +无 | +新增的key:value值 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - add:
+ fields:
+ hello: world
+字段类型转换。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| convert.convert | +数组 | +必填 | +无 | ++ |
| convert.convert[n].from | +string | +必填 | +无 | +需要转换的字段名 | +
| convert.convert[n].to | +string | +必填 | +无 | +转换后的类型，可为："bool", "integer", "float" | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - convert:
+ convert:
+ - from: count
+ to: float
+字段复制。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| copy.convert | +数组 | +必填 | +无 | ++ |
| copy.convert[n].from | +string | +必填 | +无 | +需要复制的字段名 | +
| copy.convert[n].to | +string | +必填 | +无 | +复制后的字段名 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - copy:
+ convert:
+ - from: hello
+ to: world
+将字段中的所有key:value放到event最外层。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| underRoot.keys | +string数组 | +必填 | +无 | +需要underRoot的字段名 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - underRoot:
+ keys: ["fields"]
+转换时间格式。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timestamp.convert | +数组 | +必填 | +无 | ++ |
| timestamp.convert[n].from | +string | +必填 | +无 | +指定转换时间格式的字段 | +
| timestamp.convert[n].fromLayout | +string | +必填 | +无 | +指定字段的时间格式(golang形式) | +
| timestamp.convert[n].toLayout | +string | +必填 | +无 | +转换后的时间格式(golang形式)，另外可为unix和unix_ms |
+
| timestamp.convert[n].toType | +string | +非必填 | +无 | +转换后的时间字段类型 | +
| timestamp.convert[n].local | +bool | +非必填 | +false | +是否将解析的时间转成当前时区 | +
Example
+interceptors:
+- type: normalize
+ processors:
+ - timestamp:
+ convert:
+ - from: logtime
+ fromLayout: "2006-01-02T15:04:05Z07:00"
+ toLayout: "unix"
+以上的layout参数需要填写golang形式，可参考： +
const (
+ Layout = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
+ ANSIC = "Mon Jan _2 15:04:05 2006"
+ UnixDate = "Mon Jan _2 15:04:05 MST 2006"
+ RubyDate = "Mon Jan 02 15:04:05 -0700 2006"
+ RFC822 = "02 Jan 06 15:04 MST"
+ RFC822Z = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
+ RFC850 = "Monday, 02-Jan-06 15:04:05 MST"
+ RFC1123 = "Mon, 02 Jan 2006 15:04:05 MST"
+ RFC1123Z = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
+ RFC3339 = "2006-01-02T15:04:05Z07:00"
+ RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
+ Kitchen = "3:04PM"
+ // Handy time stamps.
+ Stamp = "Jan _2 15:04:05"
+ StampMilli = "Jan _2 15:04:05.000"
+ StampMicro = "Jan _2 15:04:05.000000"
+ StampNano = "Jan _2 15:04:05.000000000"
+)
+字段内容重新格式化。可根据其他字段内容进行组合和格式化。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fmt.fields | +map | +必填 | +无 | +key表示需要格式化的字段名称，value为需要格式化的内容。可使用${}的方式表示取值某个字段 |
+
Example
+interceptors:
+- type: normalize
+ processors:
+ - fmt:
+ fields:
+ d: new-${a.b}-${c}
+interceptors字段为数组，一个Pipeline中可填写多个interceptor组件配置。
+目前，interceptor分为两种类型：
+source -> source interceptor -> queue。queue -> sink interceptor -> sink。一个interceptor只属于其中一种。大部分组件为source interceptor类型，可支持配置belongTo被部分source使用。少数通用性质的比如retry interceptor为sink interceptor类型。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +true | +表示是否开启该interceptor | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| name | +string | +非必填 | ++ | 表示interceptor的名称。当pipeline里配置相同type interceptor的情况下，必填，用于区分标识 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| belongTo | +string数组 | +非必填 | ++ | 仅source interceptor可用，用于指定该interceptor仅被哪些source使用 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| order | +int | +非必填 | ++ | interceptor的排列顺序权重 | +
专注于日志格式转换与适配的interceptor。
+属于source interceptor。
对于大部分日志对接的场景中，我们要求的日志格式可能有一些差异，这些差异主要体现在时间字段、body字段等。
+默认情况下，Loggie只会把source采集或者接收到的原始数据放到body字段中，以最简单的方式发送：
+
{
+ "body": "this is raw data"
+}
+但是，一些场景下我们需要：
+body等字段Example
+interceptors:
+ - type: schema
+ addMeta:
+ timestamp:
+ key: "@timestamp"
+ remap:
+ body:
+ key: message
+转换后的event: +
{
+ "message": "this is raw data"
+ "@timestamp": "2022-08-30T06:58:49.545Z",
+}
+interceptors:
+ - type: schema
+ addMeta:
+ timestamp:
+ key: "_timestamp_"
+ location: Local
+ layout: 2006-01-02T15:04:05Z07:00
+ remap:
+ body:
+ key: _log_
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addMeta | ++ | 非必填 | ++ | 增加系统元信息字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addMeta.timestamp | ++ | 非必填 | ++ | 增加系统时间字段（source采集到数据的时间） | +
| addMeta.timestamp.key | +string | +必填 | ++ | 系统时间的key | +
| addMeta.timestamp.location | +string | +非必填 | +默认为空，即为UTC时间 | +增加的时间时区，另外还支持Local |
+
| addMeta.timestamp.layout | +string | +非必填 | +"2006-01-02T15:04:05.000Z" | +golang的时间类型layout，可参考https://go.dev/src/time/format.go | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addMeta.pipelineName | ++ | 非必填 | ++ | 将pipelineName加入到event中 | +
| addMeta.pipelineName.key | +string | +必填 | ++ | 增加后的字段的key | +
Example
+interceptors:
+ - type: schema
+ addMeta:
+ pipelineName:
+ key: pipeline
+{
+ "pipeline": "demo"
+ ...
+}
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addMeta.sourceName | ++ | 非必填 | ++ | 将sourceName加入到event中 | +
| addMeta.sourceName.key | +string | +必填 | ++ | 增加后的字段的key | +
Example
+interceptors:
+ - type: schema
+ addMeta:
+ sourceName:
+ key: source
+{
+ "source": "local"
+ ...
+}
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| remap | +map | +非必填 | ++ | 对字段进行转换，目前支持重命名 | +
| remap.[originKey] | +string | +非必填 | ++ | 原始字段key | +
| remap.[originKey].key | +string | +非必填 | ++ | 转换后的字段key | +
Example
+interceptors:
+ - type: schema
+ remap:
+ body:
+ key: msg
+ state:
+ key: meta
+{
+ "body": "this is log"
+ "state": "ok",
+}
+转换后的event: +
{
+ "msg": "this is log"
+ "meta": "ok",
+}
+带有条件判断的函数式数据处理interceptor。
+属于source interceptor。
示例参考。
+transformer会按照配置的actions里顺序执行所有的action。action类似函数的方式，可以写入参数，参数一般为event里的字段。
+同时，每个action里还可能包括额外的控制字段。比如下面regex(body)，body即为regex的参数，pattern为额外的字段。
interceptors:
+ - type: transformer
+ actions:
+ - action: regex(body)
+ pattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$
+ - action: add(topic, common)
+另外，action还支持条件判断if-then-else的方式：
- if: <condition>
+ then:
+ - action: funcA()
+ else:
+ - action: funcB()
+其中，condition条件判断也为函数的形式。
+interceptors:
+ - type: transformer
+ actions:
+ - if: equal(status, 404)
+ then:
+ - action: add(topic, not_found)
+ - action: return()
+Example
+- type: transformer
+ actions:
+ - action: regex(body)
+ pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
+ ignoreError: true
+表示如果出现错误，直接丢弃该条event。
+Example
+- type: transformer
+ actions:
+ - action: regex(body)
+ pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
+ dropIfError: true
+给event添加额外的key:value。
+Example
+- action: add(topic, loggie)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "topic": "loggie"
+}
+复制event里的字段。
+参数：
+Example
+- action: copy(foo, bar)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "loggie"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "loggie",
+ "bar": "loggie"
+}
+移动/重命名字段。
+参数：
+Example
+- action: move(foo, bar)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "loggie"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "bar": "loggie"
+}
+更新字段key的值为value。
+参数：
+Example
+- action: set(foo, test)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "loggie"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "test"
+}
+删除字段。可填写多个字段key。
+Example
+- action: del(foo)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "foo": "loggie"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+}
+将嵌套的字段放在根部（最外层）。
+Example
+- action: underRoot(state)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "state": {
+ "node": "127.0.0.1",
+ "phase": "running"
+ }
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "node": "127.0.0.1",
+ "phase": "running"
+}
+将某个字段的值重新渲染，可根据其他的字段值组成一个值。如果key不存在则会新增该字段。
+额外字段：
+${state.node}-${state.phase}。如果pattern为固定值，则类似set(key, value)。Example
+- action: fmt(status)
+ pattern: ${state.node} is ${state.phase}
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "state": {
+ "node": "127.0.0.1",
+ "phase": "running"
+ }
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "state": {
+ "node": "127.0.0.1",
+ "phase": "running"
+ }，
+ "status": "127.0.0.1 is running"
+}
+字段的时间格式转换。
+额外字段：
+unix和unix_msUTC或者Local，如果为空，则为UTCunix和unix_msUTC或者Local，如果为空，则为UTCExample
+- action: timestamp(time)
+ fromLayout: "2006-01-02 15:04:05"
+ fromLocation: Asia/Shanghai
+ toLayout: unix_ms
+ toLocation: Local
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "time": "2022-06-28 11:24:35"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "time": 1656386675000
+}
+以上的layout参数需要填写golang形式，可参考： +
const (
+ Layout = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
+ ANSIC = "Mon Jan _2 15:04:05 2006"
+ UnixDate = "Mon Jan _2 15:04:05 MST 2006"
+ RubyDate = "Mon Jan 02 15:04:05 -0700 2006"
+ RFC822 = "02 Jan 06 15:04 MST"
+ RFC822Z = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
+ RFC850 = "Monday, 02-Jan-06 15:04:05 MST"
+ RFC1123 = "Mon, 02 Jan 2006 15:04:05 MST"
+ RFC1123Z = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
+ RFC3339 = "2006-01-02T15:04:05Z07:00"
+ RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
+ Kitchen = "3:04PM"
+ // Handy time stamps.
+ Stamp = "Jan _2 15:04:05"
+ StampMilli = "Jan _2 15:04:05.000"
+ StampMicro = "Jan _2 15:04:05.000000"
+ StampNano = "Jan _2 15:04:05.000000000"
+)
+使用正则的方式切分日志，提取字段。 +另外也可以为regex(key, to)。
+参数：
+额外字段：
+Example
+- action: regex(body)
+ pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
+input: +
{
+ "body": "10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] 'GET / HTTP/1.1' 404 683",
+}
+output: +
{
+ "ip": "10.244.0.1",
+ "id": "-",
+ "u": "-",
+ "time": "[13/Dec/2021:12:40:48 +0000]",
+ "url": "GET / HTTP/1.1",
+ "status": "404",
+ "size": "683",
+}
+使用grok的方式切分日志，提取字段。 +另外也可以为grok(key, to)。
+参数：
+额外字段：
+Example
+- action: grok(body)
+ match: "^%{DATESTAMP:datetime} %{FILE:file}:%{INT:line}: %{IPV4:ip} %{PATH:path} %{UUID:uuid}(?P<space>[a-zA-Z]?)"
+ pattern:
+ FILE: "[a-zA-Z0-9._-]+"
+input: +
{
+ "body": "2022/05/28 01:32:01 logTest.go:66: 192.168.0.1 /var/log/test.log 54ce5d87-b94c-c40a-74a7-9cd375289334",
+}
+output: +
"datetime": "2022/05/28 01:32:01",
+ "line": "66",
+ "ip": "192.168.0.1",
+ "path": "/var/log/test.log",
+ "uuid": "54ce5d87-b94c-c40a-74a7-9cd375289334",
+将json文本反序列化。 +也可以为jsonDecode(key, to)。
+参数：
+Example
+- action: jsonDecode(body)
+input: +
{
+ "body": `{"log":"I0610 08:29:07.698664 Waiting for caches to sync", "stream":"stderr", "time":"2021-06-10T08:29:07.698731204Z"}`,
+}
+output: +
{
+ "log": "I0610 08:29:07.698664 Waiting for caches to sync",
+ "stream": "stderr",
+ "time": "2021-06-10T08:29:07.698731204Z"
+}
+将多个字段序列化成json string形式。 +也可以为jsonEncode(key, to)。
+参数：
+Example
+interceptors:
+ - type: transformer
+ actions:
+ - action: jsonEncode(fields)
+input: +
{
+ "body": "this is test",
+ "fields":
+ "topic": "loggie",
+ "foo": "bar"
+}
+output: +
{
+ "fields": "{\"topic\":\"loggie\",\"foo\":\"bar\"}",
+ "body": "this is test"
+}
+将一行日志根据某种分割符切分。
+参数：
+额外字段：
+Example
+ interceptors:
+ - type: transformer
+ actions:
+ - action: split(body)
+ separator: "|"
+ keys: ["time", "order", "service", "price"]
+input: +
"body": `2021-08-08|U12345|storeCenter|13.14`,
+output: +
"time": "2021-08-08"
+ "order": "U12345"
+ "service": "storeCenter"
+ "price: "13.14"
+字段值类型转换。
+参数：
+bool, int, floatExample
+- action: strconv(code, int)
+input: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "code": "200"
+}
+output: +
{
+ "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
+ "code": 200
+}
+打印event。一般用于调试阶段使用。
+控制类型函数，执行到return()后返回，不再继续执行下面的action。
+控制类型函数，执行到dropEvent()后会将该event直接丢弃。这意味着该条数据会丢失，也不会继续被后续的interceptor或者sink处理消费。
+Example
+interceptors:
+ - type: transformer
+ actions:
+ - action: regex(body)
+ pattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$
+ - if: equal(level, DEBUG)
+ then:
+ - action: dropEvent()
+2021-02-16T09:21:20.545525544Z DEBUG this is log body，则满足level字段为DEBUG，会直接丢弃该条日志。
+条件判断类函数。
+和。Example
+interceptors:
+ - type: transformer
+ actions:
+ - if: equal(level, DEBUG) AND equal(code, 200)
+ then:
+ - action: dropEvent()
+或。Example
+interceptors:
+ - type: transformer
+ actions:
+ - if: equal(level, DEBUG) OR equal(level, INFO)
+ then:
+ - action: dropEvent()
+反。Example
+interceptors:
+ - type: transformer
+ actions:
+ - if: NOT equal(level, DEBUG)
+ then:
+ - action: dropEvent()
+目标字段值是否和参数值target相等。
+字段值是否包含参数值target。
+Caution
+target请直接使用字符串，无需添加双引号。
+比如contain(body, error)，而不是contain(body, "error")。contain(body, "error")会被当作"error"来匹配。
目标字段是否存在或者是否为空。
+目标字段的值是否大于参数值value。
+目标字段的值是否小于参数值value。
+目标字段的值是否包含prefix前缀。
+目标字段的值是否和参数regex正则匹配。
+目标字段的值是否是参数值value1...其中之一。
+ + + + + + +channel queue,是基于go chan实现的内存缓冲queue。
+Example
+queue:
+ type: channel
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchSize | +int | +不必填 | +2048 | +一个批次包含的event数量 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchBytes | +int64 | +不必填 | +33554432(32MB) | +一个批次包含的数据最大字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchAggTimeout | +time.Duration | +不必填 | +1s | +组装聚合多个event成一个batch等待的超时时间 | +
memory queue,是基于go-disruptor实现的内存缓冲queue。
+Note
+由于go-disruptor版本还未release，memory queue还属于实验阶段。生产环境不建议使用！
+Example
+queue:
+ type: memory
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchSize | +int | +不必填 | +2048 | +一个批次包含的event数量 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchBufferFactor | +int | +不必填 | +2 | +queue缓冲区的大小(channel的容量)=batchSize*batchBufferFactor | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchBytes | +int64 | +不必填 | +33554432(32MB) | +一个批次包含的数据最大字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchAggTimeout | +time.Duration | +不必填 | +1s | +组装聚合多个event成一个batch等待的超时时间 | +
dev sink将日志数据打印到控制台，一般可以用于debug或者排查问题。
+配置dev sink后，可以设置printEvents=true，查看在Loggie中发送至sink的日志数据，该数据除了source接收或者采集的原始日志，一般还包含其他元信息。
Example
+sink:
+ type: dev
+ printEvents: true
+ codec:
+ type: json
+ pretty: true
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| printEvents | +bool | +非必填 | +false | +是否打印采集的日志 | +
默认情况下Loggie的日志打印为json格式，可以配置启动参数-log.jsonFormat=false，便于在Loggie日志上查看输出结果。
使用Elasticsearch sink发送数据至Elasticsearch集群。
+Example
+sink:
+ type: elasticsearch
+ hosts: ["elasticsearch1:9200", "elasticsearch2:9200", "elasticsearch3:9200"]
+ index: "log-${fields.service}-${+YYYY.MM.DD}"
+Caution
+如果elasticsearch版本为v6.x，请加上以下etype: _doc参数。
sink:
+ type: elasticsearch
+ etype: _doc
+ ...
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| hosts | +string数组 | +必填 | +无 | +发送日志至Elasticsearch的地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| index | +string | +必填 | +无 | +发送至Elasticsearch后，日志数据存储的index | +
可以使用${a.b}的方式获取日志数据里的字段，或者加上${+YYYY.MM.DD.hh}时间戳等方式来动态生成index。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| username | +string | +非必填 | +无 | +如果Elasticsearch配置了用户名密码验证，需要填写请求的用户名 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| password | +string | +非必填 | +无 | +如果Elasticsearch配置了用户名密码验证，需要填写请求的密码 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| schema | +string | +非必填 | +http | +client sniffing时使用 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sniff | +bool | +非必填 | +false | +是否开启sniffer | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| gzip | +bool | +非必填 | +false | +发送数据是否开启gzip压缩 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| documentId | +string | +非必填 | ++ | 发送至elasticsearch的id值，可使用${}的方式取某个字段 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| opType | +string | +非必填 | +index | +参考官方文档, 如果目标为datastream，则需要设置为create | +
将接收到的数据以文件的形式写入到本地。
+Example
+sink:
+ type: file
+ workerCount: 1024
+ baseDirs:
+ - /data0
+ - /data1
+ - /data2
+ dirHashKey: ${namespace}-${deployName}
+ filename: /${namespace}/${deployName}/${podName}/${filename}
+ maxSize: 500
+ maxAge: 7
+ maxBackups: 50
+ compress: true
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| workerCount | +int | +非必填 | +1 | +写文件的并发数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| baseDirs | +string数组 | +非必填 | ++ | 文件的基础目录，可以按某个key做哈希，然后存储到对应的基础目录上 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| dirHashKey | +string | +非必填 | ++ | 按指定Key做哈希，支持变量 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| filename | +string | +必填 | ++ | 文件名，支持变量 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxSize | +int | +非必填 | +1 | +文件大小，单位为MiB | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxAge | +int | +非必填 | ++ | 旧文件保留天数，单位「天」，默认不删除 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxBackups | +int | +非必填 | +1 | +最大保留的备份文件数，默认不删除（如果maxAge配置了，那么文件依旧会被删除） | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| localTime | +bool | +非必填 | +false | +是否用本地时间格式化备份文件，默认使用UTC时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| compress | +bool | +非必填 | +false | +是否压缩，使用gzip，默认不压缩 | +
使用franz-go kafka库将日志数据发送至下游Kafka，并且能比较好的支持kerberos认证。
+（本sink和kafka sink的区别一般只在于使用的kafka golang库不同，提供给对franz kafka库有偏好的用户使用）
Example
+sink:
+ type: franzKafka
+ brokers: ["127.0.0.1:6400"]
+ topic: "log-${fields.topic}"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| brokers | +string数组 | +必填 | +无 | +发送日志至Kafka的brokers地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| topic | +string | +非必填 | +loggie | +发送日志至Kafka的topic | +
可使用${a.b}的方式，获取event里的字段值作为具体的topic名称。
比如，一个event为：
+{
+ "topic": "loggie",
+ "hello": "world"
+}
+topic: ${topic}，此时该event发送到Kafka的topic为"loggie"。
+同时支持嵌套的选择方式：
+{
+ "fields": {
+ "topic": "loggie"
+ },
+ "hello": "world"
+}
+topic: ${fields.topic}，同样也会发送到topic "loggie"。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| balance | +string | +非必填 | +roundRobin | +负载均衡策略，可填roundRobin、range、sticky、cooperativeSticky |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| compression | +string | +非必填 | +gzip | +日志发送至Kafka的压缩策略，可填gzip、snappy、lz4、zstd |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchSize | +int | +非必填 | +100 | +发送时每个batch最多包含的数据个数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchBytes | +int | +非必填 | +1048576 | +每个发送请求包含的最大字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| writeTimeout | +time.Duration | +非必填 | +10s | +写入超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tls.enabled | +bool | +非必填 | +false | +是否启用 | +
| tls.caCertFiles | +string | +非必填 | ++ | 证书文件路径 | +
| tls.clientCertFile | +string | +必填 | ++ | SASL类型，可为：客户端cert文件 |
+
| tls.clientKeyFile | +string | +必填 | ++ | SASL类型，可为：客户端key文件 |
+
| tls.endpIdentAlgo | +bool | +type=scram时必填 | ++ | 客户端是否验证服务端的证书名字 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sasl | ++ | 非必填 | ++ | SASL authentication | +
| sasl.enabled | +bool | +非必填 | +false | +是否启用 | +
| sasl.mechanism | +string | +必填 | ++ | SASL类型，可为：PLAIN、SCRAM-SHA-256、SCRAM-SHA-512、GSSAPI |
+
| sasl.username | +string | +必填 | ++ | 用户名 | +
| sasl.password | +string | +必填 | ++ | 密码 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sasl.gssapi | ++ | 非必填 | ++ | SASL authentication | +
| sasl.gssapi.authType | +string | +必填 | ++ | SASL类型，可为：1 使用账号密码、2 使用keytab | +
| sasl.gssapi.keyTabPath | +string | +必填 | ++ | keytab 文件路径 | +
| sasl.gssapi.kerberosConfigPath | +string | +必填 | ++ | kerbeos 文件路径 | +
| sasl.gssapi.serviceName | +string | +必填 | ++ | 服务名称 | +
| sasl.gssapi.userName | +string | +必填 | ++ | 用户名 | +
| sasl.gssapi.password | +string | +必填 | ++ | 密码 | +
| sasl.gssapi.realm | +string | +必填 | ++ | 领域 | +
| sasl.gssapi.disablePAFXFAST | +bool | +type=scram时必填 | ++ | DisablePAFXFAST 用于将客户端配置为不使用 PA_FX_FAST | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| security | +string | +非必填 | ++ | java格式的安全认证内容，可以自动转化成为franz-go适配的格式 | +
案例: +
pipelines:
+ - name: local
+ sources:
+ - type: file
+ name: demo
+ paths:
+ - /tmp/log/*.log
+ sink:
+ type: franzKafka
+ brokers:
+ - "hadoop74.axrzpt.com:9092"
+ topic: loggie
+ writeTimeout: 5s
+ sasl:
+ gssapi:
+ kerberosConfigPath: /etc/krb5-conf/krb5.conf
+ security:
+ security.protocol: "SASL_PLAINTEXT"
+ sasl.mechanism: "GSSAPI"
+ sasl.jaas.config: "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true keyTab=\"/shylock/kerberos/zork.keytab\" principal=\"zork@AXRZPT.COM\";"
+ sasl.kerberos.service.name: "kafka"
+Kubernetes 挂载keytab二进制证书，请参考官方文档。
+ + + + + + +使用grpc sink发送至下游，需要下游支持相同格式的grpc协议。可以使用该sink发送至grpc source的Loggie中转机。
+Example
+sink:
+ type: grpc
+ host: "loggie-aggregator.loggie-aggregator:6166"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| host | +string | +必填 | +无 | +发送至下游的host地址，多个ip使用,逗号分隔 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| loadBalance | +string | +非必填 | +round_robin | +grpc负载均衡策略 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +30s | +发送超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| grpcHeaderKey | +string | +非必填 | +无 | ++ |
使用sink kafka将日志数据发送至下游Kafka。
+Example
+sink:
+ type: kafka
+ brokers: ["127.0.0.1:6400"]
+ topic: "log-${fields.topic}"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| brokers | +string数组 | +必填 | +无 | +发送日志至Kafka的brokers地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| topic | +string | +非必填 | +loggie | +发送日志至Kafka的topic | +
可使用${a.b}的方式，获取event里的字段值作为具体的topic名称。
比如，一个event为：
+{
+ "topic": "loggie",
+ "hello": "world"
+}
+topic: ${topic}，此时该event发送到Kafka的topic为"loggie"。
+同时支持嵌套的选择方式：
+{
+ "fields": {
+ "topic": "loggie"
+ },
+ "hello": "world"
+}
+topic: ${fields.topic}，同样也会发送到topic "loggie"。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| balance | +string | +非必填 | +roundRobin | +负载均衡策略，可填hash、roundRobin、leastBytes |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| compression | +string | +非必填 | +gzip | +日志发送至Kafka的压缩策略，可填gzip、snappy、lz4、zstd |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxAttempts | +int | +非必填 | +10 | +发送最多重试次数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchSize | +int | +非必填 | +100 | +发送时每个batch最多包含的数据个数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchBytes | +int | +非必填 | +1048576 | +每个发送请求包含的最大字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchTimeout | +time.Duration | +非必填 | +1s | +形成每个发送batch的最长时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readTimeout | +time.Duration | +非必填 | +10s | +读取超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| writeTimeout | +time.Duration | +非必填 | +10s | +写入超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| requiredAcks | +int | +非必填 | +0 | +等待ack参数，可为0、1、-1 |
+
0: 不要求ack1: 等待leader partition ack-1: 等待ISR中所有replica ack字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sasl | ++ | 非必填 | ++ | SASL authentication | +
| sasl.type | +string | +必填 | ++ | SASL类型，可为：plain、scram |
+
| sasl.userName | +string | +必填 | ++ | 用户名 | +
| sasl.password | +string | +必填 | ++ | 密码 | +
| sasl.algorithm | +string | +type=scram时必填 | ++ | type=scram时使用的算法，可选sha256、sha512 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| partitionKey | +string | +非必填 | +无 | +控制发送至topic下哪个分区 | +
与topic相似，可使用${a.b}的方式，获取event里的字段值作为具体的topic名称。
loki sink用于发送数据至Loki存储。Loki文档可参考这里。
+Example
+sink:
+ type: loki
+ url: "http://localhost:3100/loki/api/v1/push"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| url | +string | +必填 | ++ | push loki的api | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tenantId | +string | +非必填 | ++ | 发送使用的租户名称 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +30s | +发送的超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| entryLine | +string | +非必填 | ++ | 发送至Loki的日志内容，默认为loggie event的body | +
Loki的日志数据结构大概分为label和主体数据，loggie会默认将header里的元信息字段，转成以下划线_连接的label。
+另外需要注意的是，由于loki的labels key不支持., /, -，这里会自动将header里包含这些符号的key转成_的形式。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| insecureSkipVerify | +bool | +非必填 | +false | +是否忽略证书认证 | +
一个Pipeline对应一个Sink。
+Concurrency相关配置使用可参照自适应sink流量控制。
+Example
+sink:
+ type: "dev"
+ codec:
+ type: json
+ pretty: true
+ parallelism: 16
+ concurrency:
+ enabled: true
+ rtt:
+ blockJudgeThreshold: 120%
+ newRttWeigh: 0.4
+ goroutine:
+ initThreshold: 8
+ maxGoroutine: 20
+ unstableTolerate: 3
+ channelLenOfCap: 0.4
+ ratio:
+ multi: 2
+ linear: 2
+ linearWhenBlocked: 4
+ duration:
+ unstable: 15
+ stable: 30
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| parallelism | +int | +非必填 | +1 | +sink客户端的并发度，可同时启动多个增大发送吞吐量，可设置的最大值为100 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| codec | ++ | 非必填 | ++ | sink发送数据给下游时，数据使用的格式 | +
| codec.type | +string | +非必填 | +json | +codec类型 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| codec.pretty | ++ | 非必填 | +false | +是否进行json格式美化 | +
| codec.beatsFormat | ++ | 非必填 | +false | +日志转成类filebeats格式：增加@timestamp字段，同时body字段命名为message |
+
用于发送采集的原始body数据。
+Example
+sink:
+ type: dev
+ codec:
+ type: raw
+Example
+sink:
+ type: kafka
+ concurrency:
+ enabled: true
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +false | +是否开启sink自适应并发度控制 | +
注：默认此功能不开启。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| initThreshold | +int | +非必填 | +16 | +初始阈值，低于该值，协程数指数增长（快启动阶段），高于该值，线性增长 | +
| maxGoroutine | +int | +非必填 | +30 | +最大协程数 | +
| unstableTolerate | +int | +非必填 | +3 | +进入平稳阶段后，对网络波动的容忍，协程数需减少的情况包括rtt增大，请求失败，协程数需增大的情况包括网络平稳且channel饱和，相同情况出现3次才会触发协程数改变，若出现第四次，将会追加，直到相反的情况触发协程数改变。 | +
| channelLenOfCap | +float | +非必填 | +0.4 | +channel饱和阈值，超过该值认为协程数需增大，仅在rtt稳定情况下才会计算该值 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| blockJudgeThreshold | +string | +非必填 | +120% | +判断rtt增大阈值，当新rtt超过当前平均rtt的到达一定程度，认为协程数需减少 | +
| newRttWeigh | +float | +非必填 | +0.5 | +计算新的平均rtt时，新rtt的权重 | +
注：blockJudgeThreshold（b）支持百分比和浮点数两种。
+若为百分比，则判断是否 (新rtt/平均rtt)>b 。
+若为浮点数，则判断是否 (新rtt-平均rtt)>b 。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| multi | +int | +非必填 | +2 | +快启动阶段，协程数指数增长速率 | +
| linear | +int | +非必填 | +2 | +（快启动之后）协程数线性增长或减少速率 | +
| linearWhenBlocked | +int | +非必填 | +4 | +channel满时(上游阻塞)，协程数线性增长速率 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| unstable | +int | +非必填 | +15 | +非平稳阶段，收集数据计算协程数的时间间隔，单位秒 | +
| stable | +int | +非必填 | +30 | +平稳阶段，收集数据计算协程数的时间间隔，单位秒 | +
pulsar sink用于发送数据至pulsar存储。
+该sink为beta试用状态，请谨慎使用于生产环境。
Example
+sink:
+ type: pulsar
+ url: pulsar://localhost:6650
+ topic: my-topic
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| url | +string | +必填 | +无 | +日志发送端pulsar连接地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| topic | +string | +必填 | +无 | +发送日志至pulsar的topic | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| producerName | +string | +非必填 | +无 | +specifies a name for the producer | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| producerName | +map | +非必填 | +无 | +Properties specifies a set of application defined properties for the producer | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| operationTimeoutSeconds | +time.Duration | +非必填 | +30s | +Producer-create, subscribe and unsubscribe operations will be retried until this interval, after which the operation will be marked as failed | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| connectionTimeout | +time.Duration | +非必填 | +5s | +Timeout for the establishment of a TCP connection | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sendTimeout | +time.Duration | +非必填 | +30s | +SendTimeout set the timeout for a message that is not acknowledged by the server 30s | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sendTimeout | +time.Duration | +非必填 | +无 | +MaxPendingMessages specifies the max size of the queue holding the messages pending to receive an acknowledgment from the broker | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| hashingSchema | +int | +非必填 | +0 | +HashingScheme is used to define the partition on where to publish a particular message. 0:JavaStringHash，1:Murmur3_32Hash | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| compressionType | +int | +非必填 | +0 | +0:NoCompression, 1:LZ4, 2:ZLIB, 3:ZSTD | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| compressionLevel | +int | +非必填 | +0 | +0:Default, 1:Faster, 2:Better | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| logLevel | +string | +非必填 | +0 | +日志级别: "info","debug", "error" | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchingMaxSize | +int | +非必填 | +2048(KB) | +BatchingMaxSize specifies the maximum number of bytes permitted in a batch | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchingMaxMessages | +int | +非必填 | +1000 | +BatchingMaxMessages specifies the maximum number of messages permitted in a batch | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| batchingMaxPublishDelay | +time.Duration | +非必填 | +10ms | +BatchingMaxPublishDelay specifies the time period within which the messages sent will be batched | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| useTLS | +bool | +非必填 | +false | +是否使用TLS认证 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tlsTrustCertsFilePath | +string | +非必填 | +无 | +the path to the trusted TLS certificate file | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tlsAllowInsecureConnection | +bool | +非必填 | +false | +Configure whether the Pulsar client accept untrusted TLS certificate from broker | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| certificatePath | +string | +非必填 | +无 | +TLS证书路径 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| privateKeyPath | +string | +非必填 | +无 | +TLS privateKey路径 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| token | +string | +非必填 | +无 | +如果使用token认证鉴权pulsar，请填写此项 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tokenFilePath | +string | +非必填 | +无 | +auth token from a file | +
sls sink用于将日志发送至阿里云可观测统一存储SLS。
+Example
+sink:
+ type: sls
+ name: demo
+ endpoint: cn-hangzhou.log.aliyuncs.com
+ accessKeyId: ${id}
+ accessKeySecret: ${secret}
+ project: test
+ logstore: test1
+ topic: myservice
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| endpoint | +string | +必填 | ++ | SLS存储的访问域名 | +
你可以在具体project页面的项目概览中查看到。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| accessKeyId | +string | +必填 | ++ | 访问的accessKeyId，请查看阿里云账号的访问凭证管理 | +
建议使用阿里云的子账号，子账号需要有对应project、logstore的权限。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| accessKeySecret | +string | +必填 | ++ | 访问的accessKeySecret，请查看阿里云账号的访问凭证管理 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| project | +string | +必填 | ++ | SLS存储的project名称 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| logstore | +string | +必填 | ++ | SLS存储的logstore名称 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| topic | +string | +非必填 | ++ | 日志主题（Topic）是日志服务的基础管理单元。您可在采集日志时指定Topic，用于区分日志 | +
alertWebhook sink将日志数据发送至http接收方。 +使用示例请参考日志报警
+Example
+sink:
+ type: alertWebhook
+ addr: http://localhost:8080/loggie
+ headers:
+ api: test1
+ lineLimit: 10
+ template: |
+ ******
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addr | +string | +非必填 | ++ | 发送alert的http地址，若为空，则不会发送 | +
| template | +string | +非必填 | ++ | 用来渲染的模板 | +
| timeout | +time.Duration | +非必填 | +30s | +发送alert的http timeout | +
| headers | +map | +非必填 | ++ | 发送alert的http header | +
| method | +string | +非必填 | +POST | +发送alert的http method, 如果不填put(不区分大小写)，都认为是POST | +
| lineLimit | +int | +非必填 | +10 | +多行日志采集情况下，每个alert中包含的最大日志行数 | +
zinc sink用于发送数据至zinc存储。
+Example
+sink:
+ type: zinc
+ host: "http://127.0.0.1:4080"
+ username: admin
+ password: Complexpass#123
+ index: "demo"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| host | +string | +非必填 | +http://127.0.0.1:4080 | +zinc的url地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| username | +string | +非必填 | ++ | 发送至zinc的用户名 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| password | +string | +非必填 | ++ | 发送至zinc的密码 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| index | +string | +非必填 | +default | +发送至zinc的index | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| skipSSLVerify | +bool | +非必填 | +true | +是否忽略SSL校验 | +
用于开发或者压测场景下，自动生成数据。
+Example
+sources:
+- type: dev
+ name: benchmark
+ qps: 100
+ byteSize: 1024
+ eventsTotal: 10000
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| qps | +int | +非必填 | +1000 | +生成event的QPS | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| byteSize | +int | +非必填 | +1024 | +单条event的字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| eventsTotal | +int | +非必填 | +-1，无限个 | +发送的所有event总数，之后会停止发送 | +
消费elasticsearch的数据。
+Example
+pipelines:
+ - name: local
+ sources:
+ - type: elasticsearch
+ name: elastic
+ hosts: ["localhost:9200"]
+ indices: ["blog*"]
+ size: 10 # data size per fetch
+ interval: 30s # pull data frequency
+pipelines:
+ - name: local
+ sources:
+ - type: elasticsearch
+ name: elastic
+ hosts:
+ - "localhost:9200"
+ - "localhost:9201"
+ indices: ["blog*"]
+ username: "bob"
+ password: "bob"
+ schema: ""
+ sniff: false
+ gzip: true
+ includeFields: # pull selected field
+ - Title
+ - Content
+ - Author
+ excludeFields: # exclude selected field
+ - Content
+ query: | # elastic query phrases
+ {
+ "match": {"Title": "bob"}
+ }
+ size: 10 # data size per fetch
+ interval: 30s # pull data frequency
+ timeout: 5s # pull timeout
+ db:
+ flushTimeout: 2s # persistent the elastic pull location frequency
+ cleanInactiveTimeout: 24h # delete the db record after the time
+ cleanScanInterval: 1h # check the expired db record frequency
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| hosts | +string数组 | +必填 | ++ | 消费的elasticsearch url地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| index | +string数组 | +必填 | ++ | 查询elasticsearch的index名称 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| username | +string | +非必填 | ++ | 消费elasticsearch的用户名 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| password | +string | +必填 | ++ | 消费elasticsearch的密码 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| schema | +string | +非必填 | +http | +HTTP scheme(http/https)，sniff的时候使用 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| gzip | +bool | +非必填 | +false | +是否开启gzip压缩 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| includeFields | +string数组 | +非必填 | ++ | 只返回指定的_source字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| excludeFields | +string数组 | +非必填 | ++ | 排除指定的_source字段 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| query | +string | +非必填 | ++ | 查询elasticsearch的表达式 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| size | +int | +非必填 | +100 | +每次请求得到hits返回的个数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| interval | +time.Duration | +非必填 | +30s | +定时请求elasticsearch的时间间隔 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +5s | +请求的超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| db | ++ | 非必填 | ++ | 持久化记录查询elasticsearch请求的进度，会存储至elasticsearch中，避免Loggie重启后重复消费数据 | +
| db.indexPrefix | +string | +非必填 | +.loggie-db | +默认情况下，loggie会将持久化的数据定时写入格式为${indexPrefix}-${pipelineName}-${sourceName}的index中 |
+
| db.flushTimeout | +time.Duration | +非必填 | +2s | +持久化数据写入的间隔时间 | +
| db.cleanInactiveTimeout | +time.Duration | +非必填 | +504h (21day) | +清理过期的持久化数据超时时间 | +
| db.cleanScanInterval | +time.Duration | +非必填 | +1h | +检查过期时间间隔 | +
file source用于日志采集。
+Example
+sources:
+- type: file
+ name: accesslog
+Tips
+如果你使用logconfig/clusterlogconfig采集容器日志，file source里还增加了额外的字段，请参考这里。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| paths | +string数组 | +必填 | +无 | +采集的path路径，使用glob表达式来匹配。支持glob扩展表达式Brace Expansion和Glob Star |
+
Example
+需要采集的目标文件： +
/tmp/loggie/service/order/access.log
+/tmp/loggie/service/order/access.log.2022-04-11
+/tmp/loggie/service/pay/access.log
+/tmp/loggie/service/pay/access.log.2022-04-11
+对应配置： +
sources:
+- type: file
+ paths:
+ - /tmp/loggie/**/access.log{,.[2-9][0-9][0-9][0-9]-[01][0-9]-[0123][0-9]}
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| excludeFiles | +string数组 | +非必填 | +无 | +排除采集的文件正则表达式 | +
Example
+sources:
+- type: file
+ paths:
+ - /tmp/*.log
+ excludeFiles:
+ - \.gz$
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| ignoreOlder | +time.Duration | +非必填 | +无 | +例如48h，表示忽略更新时间在2天之前的文件，无需进行采集 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| ignoreSymlink | +bool | +非必填 | +false | +是否忽略符号链接（软链接）的文件 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| addonMeta | +bool | +非必填 | +false | +是否添加默认的日志采集state元信息 | +
event示例
+{
+ "body": "this is test",
+ "state": {
+ "pipeline": "local",
+ "source": "demo",
+ "filename": "/var/log/a.log",
+ "timestamp": "2006-01-02T15:04:05.000Z",
+ "offset": 1024,
+ "bytes": 4096,
+ "hostname": "node-1"
+ }
+}
+state含义解释：
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| workerCount | +int | +非必填 | +1 | +读取文件内容的工作线程(goroutine)数。单节点超过100个文件的时候考虑提高 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readBufferSize | +int | +非必填 | +65536 | +单次读取文件的数据量。默认64K=65536 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxContinueRead | +int | +非必填 | +16 | +连续读取同一个文件内容的次数，达到这个次数将强制切换到下个文件读取。主要作用是用来避免活跃文件一直占据读取资源，非活跃文件长时间得不到读取采集 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxContinueReadTimeout | +time.Duration | +非必填 | +3s | +同一个文件最长读取时间，超过这个时间将强制切换下个文件读取。作用与maxContinueRead类似 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| inactiveTimeout | +time.Duration | +非必填 | +3s | +如果当文件从上一次采集到现在超过inactiveTimeout的话，则认为文件进入不活跃状态(即最后一条日志已经写入完成)，则可以安全的采集最后一行日志 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| firstNBytesForIdentifier | +int | +非必填 | +128 | +使用采集目标文件的前n个字符来生成文件唯一code。如果文件的大小小于n，则该文件暂时不会采集。用途主要是，结合文件inode信息，用来精确标识一个文件。辅助判断文件是否删除或者是改名 | +
编码转换，用于将不同的编码转换为utf8，当下支持的编码转换格式.
+Example
+ sources:
+ - type: file
+ name: demo
+ paths:
+ - /tmp/log/*.log
+ fields:
+ topic: "loggie"
+ charset: "gbk"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| charset | +string | +否 | +utf-8 | +提取字段的匹配模型 | +
当前支持的转换为utf-8的编码格式有
+nopplainutf-8gbkbig5euc-jpiso2022-jpshift-jiseuc-kriso8859-6eiso8859-6iiso8859-8eiso8859-8iiso8859-1iso8859-2iso8859-3iso8859-4iso8859-5iso8859-6iso8859-7iso8859-8iso8859-9iso8859-10iso8859-13iso8859-14iso8859-15iso8859-16cp437cp850cp852cp855cp858cp860cp862cp863cp865cp866ebcdic-037ebcdic-1040ebcdic-1047koi8rkoi8umacintoshmacintosh-cyrillicwindows1250windows1251windows1252windows1253windows1254windows1255windows1256windows1257windows1258windows874utf-16be-bomutf-16le-bom换行符相关配置
+Example
+sources:
+ - type: file
+ name: demo
+ lineDelimiter:
+ type: carriage_return_line_feed
+ value: "\r\n"
+ charset: gbk
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| type | +bool | +非必填 | +auto | +只有在type是custome时候value才会有效 | +
当前支持的type有
+autoline_feedvertical_tabform_feedcarriage_returncarriage_return_line_feednext_lineline_separatorparagraph_separatornull_terminator对应的换行符为：
+ ```
+ auto: {'\u000A'},
+ line_feed: {'\u000A'},
+ vertical_tab: {'\u000B'},
+ form_feed: {'\u000C'},
+ carriage_return: {'\u000D'},
+ carriage_return_line_feed: []byte("\u000D\u000A"),
+ next_line: {'\u0085'},
+ line_separator: []byte("\u2028"),
+ paragraph_separator: []byte("\u2029"),
+ null_terminator: {'\u0000'},
+ ```
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| value | +string | +非必填 | +\n | +换行符的内容 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| charset | +string | +非必填 | +utf-8 | +换行符编码 | +
多行采集相关配置
+Example
+sources:
+- type: file
+ name: accesslog
+ multi:
+ active: true
+ pattern: '^\d{4}-\d{2}-\d{2}'
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| active | +bool | +非必填 | +false | +是否开启多行采集模式 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| pattern | +string | +当multi.active=true的时候必填 | +false | +判断为一条全新日志的正则表达式。例如配置为'^\['，则认为行首以[开头才是一条新日志，否则将这行内容合入上一条日志作为上一条日志的一部分 |
+
Example
+假设有多行日志如下所示：
+2023-05-11 14:30:15 ERROR Exception in thread "main" java.lang.NullPointerException
+ at com.example.MyClass.myMethod(MyClass.java:25)
+ at com.example.MyClass.main(MyClass.java:10)
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxLines | +int | +非必填 | +500 | +1条日志最多包含几行内容。默认500行，超过上限将强制发送当前日志，超出部分作为新的一条日志 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxBytes | +int64 | +非必填 | +131072 | +1条日志最多包含几个字节。默认128K，超过上限将强制发送当前日志，超出部分作为新的一条日志 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +5s | +1条日志最多等待多久采集为完整的1条日志。默认5s，超过上限将强制发送当前日志，超出部分作为新的一条日志 | +
source的确认机制相关配置。如果需确保要at least once，需要开启ack机制，但是会有一定性能顺耗
Caution
+该配置只能配置在defaults中
+Example
+defaults:
+ sources:
+ - type: file
+ ack:
+ enable: true
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enable | +bool | +非必填 | +true | +是否开启确认机制 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maintenanceInterval | +time.Duration | +非必填 | +20h | +维护周期。用来定时清理过期的确认文件数据(例如不再采集的文件的ack信息) | +
使用sqlite3作为数据库。保存采集过程中的文件名称、文件inode、文件采集的offset等信息。用来在loggie reload或者重启后恢复上一次的采集进度
Caution
+该配置只能配置在defaults中
+Example
+defaults:
+ sources:
+ - type: file
+ db:
+ file: "./data/loggie.db"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| file | +string | +非必填 | +./data/loggie.db | +数据库文件路径 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| tableName | +string | +非必填 | +registry | +数据库表名称 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| flushTimeout | +time.Duration | +非必填 | +2s | +定时将采集信息写入到数据库 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| bufferSize | +int | +非必填 | +2048 | +输入数据库的采集信息的缓冲区大小 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cleanInactiveTimeout | +time.Duration | +非必填 | +504h | +清理数据库中的过期数据。如果数据的更新时间超过配置值，将会删除该条数据。默认保留21天 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cleanScanInterval | +time.Duration | +非必填 | +1h | +周期性的检查数据库中的过期数据。默认每隔1小时检查一次 | +
监控文件变化的相关配置
+Caution
+该配置只能配置在defaults中
+Example
+defaults:
+ sources:
+ - type: file
+ watcher:
+ enableOsWatch: true
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enableOsWatch | +bool | +非必填 | +true | +是否启用OS的监控通知机制。例如linux的inotify指令 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| scanTimeInterval | +time.Duration | +非必填 | +10s | +周期性的检查文件的状态变更（例如文件的新建、删除等）。默认每隔10s检查一次 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maintenanceInterval | +time.Duration | +非必填 | +5m | +周期性的维护工作（例如上报采集统计信息、清理文件等） | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fdHoldTimeoutWhenInactive | +time.Duration | +非必填 | +5m | +当文件的上次采集到现在的时间超过限制（文件长时间没有写入，认为大概率不会再写入内容），将会释放该文件的文件句柄以释放系统资源 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fdHoldTimeoutWhenRemove | +time.Duration | +非必填 | +5m | +当文件被删除且未采集完成，会等待的最大时间来采集完成。超过限制不管文件最终是否采集完成，都会直接释放文件句柄不再采集 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxOpenFds | +int | +非必填 | +1024 | +最大打开的文件句柄数量，超出后的文件将暂时不会采集 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxEofCount | +int | +非必填 | +3 | +最大连续读取文件遇到eof的次数。超过限制认为文件暂时不活跃，将进入“僵尸”队列等待更新事件被激活 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| cleanWhenRemoved | +bool | +非必填 | +true | +当文件被删除后，是否同步删除db中的采集相关信息 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readFromTail | +bool | +非必填 | +false | +是否从文件的最新一行开始采集，而不管历史写入到文件的内容。适用于采集系统的迁移等场景 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| taskStopTimeout | +time.Duration | +非必填 | +30s | +采集任务退出的超时时间。是一个兜底方案，放在采集任务假死导致无法reload | +
清理文件相关配置。过期且已经采集完成的文件将会直接从磁盘删除以释放磁盘空间
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxHistoryDays | +int | +非必填 | +无 | +（采集完成后的）文件最多保留的天数。如果超出限制，将会把文件直接从磁盘中删除。不配置则永远不会删除文件 | +
Grpc source用于接收Loggie Grpc格式的数据请求。
+一般用在中转机场景，接收其他Loggie集群发送的日志。
Example
+sources:
+- type: grpc
+ name: aggre
+ port: 6066
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| bind | +string | +非必填 | +0.0.0.0 | +提供server绑定的host | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| port | +string | +非必填 | +6066 | +提供服务的端口号 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +20s | +超时时间 | +
Kafka source用于接收Kafka数据。
+Example
+sources:
+- type: kafka
+ brokers: ["kafka1.kafka.svc:9092"]
+ topic: log-*
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| brokers | +string数组 | +必填 | +无 | +Kafka broker地址 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| topic | +string | +必填 | +无 | +接收的topics，可使用正则来匹配多个topic | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| groupId | +string | +非必填 | +loggie | +Loggie消费kafka的groupId | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| queueCapacity | +int | +非必填 | +100 | +内部发送的队列容量 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| minAcceptedBytes | +int | +非必填 | +1 | +最小接收的batch字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxAcceptedBytes | +int | +非必填 | +1e6（1MB) | +最大接收的消息字节数，如果超过会被truncate，可以设置为一个能容忍的较大的值 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readMaxAttempts | +int | +非必填 | +3 | +最大的重试次数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxPollWait | +time.Duration | +非必填 | +10s | +接收的最长等待时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readBackoffMin | +time.Duration | +非必填 | +100ms | +在接收新的消息前，最小的时间间隔 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| readBackoffMax | +time.Duration | +非必填 | +1s | +在接收新的消息前，最大的时间间隔 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enableAutoCommit | +bool | +非必填 | +false | +是否开启autoCommit | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| autoCommitInterval | +time.Duration | +非必填 | +1s | +autoCommit的间隔时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| autoOffsetReset | +string | +非必填 | +latest | +没有offset时，初始的offset采用方式，可为earliest和latest |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| sasl | ++ | 非必填 | ++ | SASL authentication | +
| sasl.type | +string | +必填 | ++ | SASL类型，可为：plain、scram |
+
| sasl.userName | +string | +必填 | ++ | 用户名 | +
| sasl.password | +string | +必填 | ++ | 密码 | +
| sasl.algorithm | +string | +type=scram时必填 | ++ | type=scram时使用的算法，可选sha256、sha512 |
+
接收Kubernetes events的source。
+使用方式可参考采集Kubernetes Events。
+Example
+sources:
+- type: kubeEvent
+ name: event
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| kubeconfig | +string | +非必填 | ++ | 请求Kubernetes的kubeconfig文件，当Loggie部署在Kubernetes集群中时，无需填写，会进入in cluster模式 |
+
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| master | +string | +非必填 | ++ | 请求Kubernetes的master地址，当Loggie部署在Kubernetes集群中时，无需填写 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| bufferSize | +int | +非必填 | +1000 | +监听的队列大小，最小为1 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| watchLatestEvents | +bool | +非必填 | +false | +是否只监听最新的events | +
由于Loggie重启后会重新list所有的events，会导致重复发送，如果不希望重复发送，可以设置为true，当然可能导致重启时间段内新产生的events丢失。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| blackListNamespaces | +string数组 | +非必填 | ++ | 不接收其中定义的namespaces中产生的events | +
sources字段为数组，一个Pipeline中可填写多个source组件配置。
+因此，请注意所有的source中name必填，作为pipeline中source的唯一标识。
所有Source均可以使用以下配置。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| enabled | +bool | +非必填 | +true | +表示是否开启该source | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| name | +string | +必填 | ++ | 表示source的名称，建议填写有标识意义的词 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fields | +map | +非必填 | ++ | 自定义额外添加到event中的字段 | +
比如如下配置:
+Example
+sources:
+- type: file
+ name: access
+ paths:
+ - /var/log/*.log
+ fields:
+ service: demo
+会给采集的所有日志上，都加上service: demo字段。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fieldsFromEnv | +map | +非必填 | ++ | 额外添加到event中的字段，value为env环境变量的key | +
比如如下配置:
+Example
+sources:
+- type: file
+ name: access
+ paths:
+ - /var/log/*.log
+ fieldsFromEnv:
+ service: SVC_NAME
+会从Loggie所在的环境变量中，获取SVC_NAME的值${SVC_NAME}，然后给所有的日志event上添加字段：service: ${SVC_NAME}。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fieldsFromPath | +map | +非必填 | ++ | 额外添加到event中的字段，value为path指定文件里的内容 | +
比如如下配置:
+Example
+sources:
+- type: file
+ name: access
+ paths:
+ - /var/log/*.log
+ fieldsFromPath:
+ test: /tmp/foo
+假设文件/tmp/foo的内容为bar：
+
cat /tmp/foo
+---
+bar
+Loggie会给所有的日志event上添加字段：test: bar。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fieldsUnderRoot | +bool | +非必填 | +false | +额外添加的fields是否放在event的根部 | +
比如，默认情况下，输出的日志格式为：
+{
+ "body": "hello world",
+ "fields": {
+ "service": "demo"
+ }
+}
+如果设置fieldsUnderRoot=true，输出的日志格式为：
+{
+ "body": "hello world",
+ "service": "demo"
+}
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| fieldsUnderKey | +string | +非必填 | +fields | +当fieldsUnderRoot=false时，字段的名称 |
+
比如可以修改默认的字段fields为tag，输出的日志为：
{
+ "body": "hello world",
+ "tag": {
+ "service": "demo"
+ }
+}
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| codec | ++ | 非必填 | ++ | source接收到数据的时候用于解析预处理 | +
| codec.type | +string | +非必填 | +无 | ++ |
请注意：目前仅file source支持source codec。
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| codec.bodyFields | ++ | 必填 | ++ | 使用解析读取到的json数据中的该字段作为body |
+
配置示例：
+type: json
+ sources:
+ - type: file
+ name: nginx
+ paths:
+ - /var/log/*.log
+ codec:
+ type: json
+ bodyFields: log
+如果采集到的日志为： +
{"log":"I0610 08:29:07.698664 Waiting for caches to sync\n", "stream":"stderr", "time:"2021-06-10T08:29:07.698731204Z"}
+body: "I0610 08:29:07.698664 Waiting for caches to sync"
+请注意：目前非bodyFields的字段均会被丢弃。
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| codec.pattern | ++ | 必填 | ++ | 正则表达式 | +
| codec.bodyFields | ++ | 必填 | ++ | 使用正则提取到的该字段作为body |
+
配置示例：
+type: regex
+ sources:
+ - type: file
+ name: nginx
+ paths:
+ - /var/log/*.log
+ codec:
+ type: regex
+ pattern: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$
+ bodyFields: log
+如果采集到的日志为： +
2021-12-01T03:13:58.298476921Z stderr F INFO [main] Starting service [Catalina]
+body: "INFO [main] Starting service [Catalina]"
+请注意：目前非bodyFields的字段均会被丢弃。
+ + + + + + +采集Prometheus Metrics的指标数据。
+Example
+sources:
+- type: prometheusExporter
+ name: metric
+ endpoints:
+ - "http://127.0.0.1:9196/metrics"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| endpoints | +string数组 | +必填 | ++ | 抓取的远端exporter地址，请注意Loggie不会默认在请求路径中添加/metrics | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| interval | +time.Duration | +非必填 | +30s | +定时抓取远端exporter的时间间隔 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +5s | +抓取请求的超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| toJson | +bool | +非必填 | +false | +是否将抓取到的prometheus原生指标，转换成JSON格式 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| labels | +map | +非必填 | ++ | 给所有metrics指标增加额外的label | +
labels支持配置${_env.XX}的方式获取环境变量。 +例如，配置：
+labels
+ sources:
+ - type: prometheusExporter
+ name: metric
+ endpoints:
+ - "http://127.0.0.1:9196/metrics"
+ labels:
+ svc: ${_env.SVC}
+假设环境变量SVC=test，会将所有的metrics加上svc=test的label。
+ + + + + + +通过unix socket接收数据。
+Example
+sources:
+- type: unix
+ name: demo
+ path: "/tmp/loggie.sock"
+字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| path | +string | +必填 | ++ | 接收的路径名 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxBytes | +int | +非必填 | +40960 | +接收的最大字节数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| maxConnections | +int | +非必填 | +512 | +同时保持最多的连接数 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| timeout | +time.Duration | +非必填 | +5m | +连接超时时间 | +
字段 |
+类型 |
+是否必填 |
+默认值 |
+含义 |
+
|---|---|---|---|---|
| mode | +string | +非必填 | +0755 | ++ |
ethfoo 2022-12-17 \u00a0 15 min read
\u6765\u81ea2022 Top 100\u5168\u7403\u8f6f\u4ef6\u6848\u4f8b\u7814\u7a76\u5cf0\u4f1a\u7684\u5206\u4eab\u3002
\u4e3b\u8981\u5185\u5bb9\u5305\u62ec\uff1a
Continue reading
"},{"location":"blog/#loggie_1","title":"\u4f7f\u7528Loggie\u5feb\u901f\u6784\u5efa\u53ef\u6269\u5c55\u7684\u4e91\u539f\u751f\u65e5\u5fd7\u67b6\u6784","text":"ethfoo 2022-10-26 \u00a0 10 min read
\u6765\u81ea\u4e91\u539f\u751f\u793e\u533a\u53ef\u89c2\u6d4b\u6027\u76f4\u64ad\u5206\u4eab\u3002
\u4e3b\u8981\u5185\u5bb9\u5305\u62ec\uff1a
Continue reading
"},{"location":"blog/#_1","title":"\u7f51\u6613\u6570\u5e06\u4e91\u539f\u751f\u65e5\u5fd7\u5e73\u53f0\u67b6\u6784\u5b9e\u8df5","text":"ethfoo 2022-03-11 \u00a0 8 min read
\u6765\u81eadatafun talk\u7684\u5206\u4eab\u3002
\u7f51\u6613\u4ece2015\u5e74\u5c31\u5f00\u59cb\u4e86\u4e91\u539f\u751f\u7684\u63a2\u7d22\u4e0e\u5b9e\u8df5\uff0c\u4f5c\u4e3a\u53ef\u89c2\u6d4b\u6027\u7684\u91cd\u8981\u4e00\u73af\uff0c\u65e5\u5fd7\u5e73\u53f0\u4e5f\u7ecf\u5386\u4e86\u4ece\u4e3b\u673a\u5230\u5bb9\u5668\u7684\u6f14\u8fdb\uff0c\u652f\u6491\u4e86\u96c6\u56e2\u5185\u5404\u4e1a\u52a1\u90e8\u95e8\u7684\u5927\u89c4\u6a21\u4e91\u539f\u751f\u5316\u6539\u9020\u3002 \u672c\u6587\u4f1a\u8bb2\u8ff0\u5728\u8fd9\u4e2a\u8fc7\u7a0b\u4e2d\u6211\u4eec\u9047\u5230\u7684\u95ee\u9898\uff0c\u5982\u4f55\u6f14\u8fdb\u548c\u6539\u9020\uff0c\u5e76\u4ece\u4e2d\u6c89\u6dc0\u4e86\u54ea\u4e9b\u7ecf\u9a8c\u4e0e\u6700\u4f73\u5b9e\u8df5\u3002
\u4e3b\u8981\u5185\u5bb9\u5305\u62ec\uff1a
Continue reading
PPT
"},{"location":"developer-guide/contributing/","title":"\u4ee3\u7801\u8d21\u732e","text":"Note
\u5982\u679c\u4f60\uff1a
\u5efa\u8bae\u5148\u63d0Issues\uff0c\u63cf\u8ff0\u4f60\u7684\u76ee\u7684\u6216\u8005\u95ee\u9898\u3002
"},{"location":"developer-guide/contributing/#_2","title":"\u4ee3\u7801/\u6587\u6863\u8d21\u732e\u6d41\u7a0b","text":""},{"location":"developer-guide/contributing/#1","title":"1. \u521b\u5efa\u672c\u5730\u5de5\u7a0b","text":""},{"location":"developer-guide/contributing/#fork","title":"fork\u4ee3\u7801","text":"\u8bbf\u95eehttps://github.com/loggie-io/loggie\uff0c\u70b9\u51fb\u53f3\u4e0a\u89d2\u7684Fork\uff0cfork\u5230\u81ea\u5df1\u7684github\u4ed3\u5e93\u4e2d\u3002
"},{"location":"developer-guide/contributing/#clone","title":"clone\u5230\u672c\u5730","text":"\u5c06fork\u540e\u7684\u5de5\u7a0bclone\u5230\u672c\u5730\uff1a
cd ${go-workspace}\n\ngit clone git@github.com:${yourAccountId}/loggie.git\n\u5728\u672c\u5730\u5de5\u7a0b\u4e2d\uff0c\u6dfb\u52a0upstream remote\uff1a
git remote add upstream https://github.com/loggie-io/loggie.git\ngit remote set-url --push upstream no_push\n\u53ef\u4ee5\u4f7f\u7528git remote -v\u67e5\u770b\uff0corigin\u4e3a\u4f60\u4e2a\u4eba\u8d26\u53f7fork\u7684git\u5730\u5740\uff0cupstream\u4e3aLoggie\u5b98\u65b9\u4ed3\u5e93\u5730\u5740\uff0cuptream\u4ec5\u4e3a\u672c\u5730\u540c\u6b65\u4ee3\u7801\u4f7f\u7528\uff0c\u6b63\u5e38\u65e0push\u6743\u9650\u3002
\u57fa\u4e8e\u6700\u65b0\u7684main\u5206\u652f\uff0c\u521b\u5efa\u51fa\u4f60\u7684\u5206\u652f\uff0c\u901a\u5e38\u4ee5feat/fix/chore\u7b49\u5f00\u5934\u3002
git checkout -b feat-mywork\n\u5728\u8be5\u5206\u652f\u5f00\u53d1\u5b8c\u540e\uff0c\u5f85\u63d0\u4ea4\u524d\uff0c\u5efa\u8bae\u4fdd\u6301\u4ee3\u7801\u548c\u4e0a\u6e38\u4e00\u81f4\uff1a
git pull upstream -r\ngit commit\n\ngit push origin feat-mywork\nCommit\u89c4\u8303
\u901a\u5e38\u6211\u4eec\u4f7f\u7528<type>(<scope>): <subject>\u7684\u5f62\u5f0f\uff1a
<type>:\uff08\u5fc5\u987b\uff09
<scope>:\uff08\u53ef\u9009\uff09
<subject>: \uff08\u5fc5\u987b\uff09\u5373commit\u7684\u5185\u5bb9\u63cf\u8ff0
\u8bbf\u95ee\u4f60\u7684\u4e2a\u4ebagithub\u8d26\u53f7\u91ccfork\u7684Loggie\u9879\u76ee\uff0c\u5e76Open pull request\u3002 \u5c3d\u91cf\u63cf\u8ff0\u6e05\u695aPR\u7684\u80cc\u666f\u3001\u76ee\u7684\u548c\u6539\u52a8\u70b9\u3002 \u5982\u679c\u6709\u76f8\u5173Issues\uff0c\u9700\u5173\u8054\u3002
\u63d0\u4ea4\u524d\u8bf7\u68c0\u67e5\uff1a
\u4f60\u53ef\u4ee5\u5728OWNERS\u627e\u4eba\uff0c\u6216\u8005\u9ed8\u8ba4\u6dfb\u52a0loggie-robot\u8d26\u53f7\u4f5c\u4e3a\u4f60PR\u7684Reviewers\u6216Assigneees\u3002
"},{"location":"developer-guide/development/","title":"\u672c\u5730\u5f00\u53d1","text":"\u672c\u6587\u4ecb\u7ecd\u5982\u4f55\u5728\u672c\u5730\u5f00\u53d1\u548c\u8c03\u8bd5Loggie\u5de5\u7a0b\u3002
"},{"location":"developer-guide/development/#_2","title":"\u672c\u5730\u5de5\u7a0b","text":""},{"location":"developer-guide/development/#golang","title":"golang\u5f00\u53d1\u73af\u5883","text":"Loggie\u4f7f\u7528Golang\u7f16\u5199\uff0c\u8bf7\u786e\u4fdd\u672c\u5730\u6709Golang\u5f00\u53d1\u73af\u5883\u3002
go version\n\u8bf7\u4f7f\u7528\u81ea\u5df1\u7684github\u8d26\u53f7fork Loggie\u5de5\u7a0b\u3002\u7136\u540egit clone\u5230\u672c\u5730\u3002
git clone git@github.com:<Account>/loggie.git\n\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u672c\u5730\u65e0\u7279\u6b8a\u4f9d\u8d56\u3002
\u9488\u5bf9\u4e0d\u540c\u7684\u529f\u80fd\u573a\u666f\uff0c\u9700\u8981\u6839\u636e\u5177\u4f53\u60c5\u51b5\u5b89\u88c5\u6216\u8005\u4f7f\u7528\u5916\u90e8\u7684\u4f9d\u8d56\u3002
\u5efa\u8bae\u672c\u5730\u8fd0\u884cKubernetes\u8fdb\u884c\u90e8\u7f72\u548c\u7ba1\u7406\u3002
"},{"location":"developer-guide/development/#kubernetes","title":"Kubernetes","text":"\u642d\u5efaKubernetes
\u63a8\u8350\u4f7f\u7528Kind\u3002
\u90e8\u7f72\u4f9d\u8d56\u7ec4\u4ef6
\u63a8\u8350\u4f7f\u7528Helm\u6765\u90e8\u7f72\u3002\u76f8\u5173\u7684\u7ec4\u4ef6\u53ef\u4f7f\u7528\u516c\u5f00\u7684Helm\u4ed3\u5e93\u6765\u6dfb\u52a0\u548c\u90e8\u7f72\u3002
\u672c\u5730Loggie\u8fde\u63a5Kubernetes
\u5982\u679c\u9700\u8981\u4f7f\u7528LogConfig\u7b49CRD\uff0c\u6216\u8005\u8c03\u8bd5Kubernetes\u4e0b\u7684\u4f7f\u7528\u65b9\u5f0f\uff0c\u9700\u8981\u5728\u7cfb\u7edf\u914d\u7f6e\u4e2d\uff0cenable Kubernetes Discovery\u3002
loggie.yml
discovery:\nenabled: true\nkubernetes:\nkubeconfig: ${home}/.kube/config\n\u6307\u5b9akubeconfig\u8fde\u63a5\u672c\u5730\u7684Kubernetes APIServer\u3002
\u53e6\u5916\uff0cAgent\u5f62\u6001\u7684Loggie\uff0c\u4f1a\u53ea\u76d1\u542c\u672c\u8282\u70b9\u7684Kubernetes Pod\u4e8b\u4ef6\uff0c\u9700\u8981\u5728Loggie\u542f\u52a8\u53c2\u6570\u4e2d\u6dfb\u52a0-meta.nodeName\u6765\u6a21\u62df\u6240\u5728\u7684\u8282\u70b9\u3002
\u4f8b\u5982\uff0c\u53ef\u4ee5\u4f7f\u7528kubectl get node\u67e5\u770b\u6240\u6709\u8282\u70b9\u540d\u79f0\uff1a
NAME STATUS ROLES AGE VERSION\nkind-control-plane Ready master 2y50d v1.21.0\n-meta.nodeName=kind-control-plane\uff0c\u53ef\u4ee5\u4f7fLoggie\u542f\u52a8\u540e\u88ab\u8ba4\u4e3a\u90e8\u7f72\u5728\u8be5\u8282\u70b9\u3002"},{"location":"developer-guide/development/#_5","title":"\u9a8c\u8bc1\u4e0e\u8c03\u8bd5","text":"Loggie\u9ed8\u8ba4\u542f\u52a8\u540e\uff0c\u8f93\u51fa\u7684\u65e5\u5fd7\u683c\u5f0f\u4e3aJSON\uff0c\u5982\u679c\u4f60\u4e0d\u4e60\u60ef\uff0c\u53ef\u5728\u542f\u52a8\u53c2\u6570\u4e2d\u6dfb\u52a0-log.jsonFormat=false\u6765\u5173\u95ed\u3002
\u672c\u5730\u5c3d\u91cf\u4f7f\u7528\u6a21\u62df\u7684\u65b9\u5f0f\uff0c\u5408\u7406\u5229\u7528dev source\u548cdev sink\u3002\u6bd4\u5982\uff1a
\u793a\u4f8b\uff1a\u5728pipelines\u4e2d\uff0c\u4f7f\u7528dev sink\uff0c\u67e5\u770b\u6700\u7ec8\u8f93\u51fa\u53d1\u9001\u81f3\u4e0b\u6e38\u7684\u6570\u636e\u3002
pipelines.yml
sink:\ntype: \"dev\"\nprintEvents: true\ncodec:\npretty: true\nrelease-vA.B\uff0c\u4f8b\u5982release-v1.1\uff0c\u8fd9\u91cc\u53ea\u4f7f\u7528\u7248\u672c\u524d\u4e24\u4f4dv1.1.0v1.1.1\u786e\u8ba4\u7248\u672c\u9700\u8981\u7684\u6240\u6709\u529f\u80fd\u5df2\u7ecf\u5408\u5165\uff0cmain\u5206\u652f\u4e3a\u6700\u65b0\u63d0\u4ea4\uff1a
git checkout main\ngit pull upstream main -r\n\u57fa\u4e8emain\u5206\u652f\u521b\u5efa\u65b0\u7684release\u5206\u652f\uff0c\u4f8b\u5982release-v1.1\uff1a
git checkout -b release-v${A.B}\n\u57fa\u4e8e\u8be5\u5206\u652f\u586b\u5199CHANGELOG\uff0c\u63d0\u4ea4\u81f3\u5de5\u7a0b\u4e2d\u5408\u5165
git add .\ngit commit -m'Release: add v${A.B.C} changelog'\ngit push upstream release-v${A.B.C}\nrelease Loggie-installationation\u90e8\u7f72\u811a\u672c\u5de5\u7a0b\uff0c\u548c\u4e0a\u8ff01\uff0c2\u6b65\u9aa4\u76f8\u540c
\u8bf7\u6ce8\u610f\uff1a
\u57fa\u4e8erelease\u5206\u652f\u8fdb\u884c\u56de\u5f52\u6d4b\u8bd5\u3002 release\u5206\u652fpush\u81f3Github\u540e\uff0c\u4f1a\u89e6\u53d1action\u8fdb\u884c\u955c\u50cf\u6784\u5efa\uff0c\u4f7f\u7528\u8be5\u955c\u50cf\u8fdb\u884c\u56de\u5f52\u6d4b\u8bd5\u3002 \u8fd9\u91cc\u7684\u6d4b\u8bd5\u4f1a\u4f7f\u7528loggie-installationation\u90e8\u7f72\u811a\u672c\u91cc\u76f8\u5e94\u7684release\u5206\u652f\uff0c\u540c\u65f6\u56de\u5f52\u9a8c\u8bc1\u90e8\u7f72\u811a\u672c\u6b63\u786e\u6027\u3002 \u5982\u679c\u6d4b\u8bd5\u53d1\u73b0Bug\uff0c\u63d0\u4ea4PR\u5408\u5165release\u5206\u652f\uff0c\u91cd\u65b0\u8fdb\u884c\u56de\u5f52\u3002
"},{"location":"developer-guide/release/#5-tag","title":"5. \u6253\u4e0a\u7248\u672ctag","text":"\u6d4b\u8bd5\u901a\u8fc7\u540e\uff0c\u57fa\u4e8e\u8be5release\u5206\u652f\u6253\u4e0a\u76f8\u5e94\u7684\u7248\u672ctag\uff0c\u6bd4\u5982v1.1.0
git pull upstream release-v${A.B.C}\ngit tag v${A.B.C}\ngit push v${A.B.C}\nloggie-v${A.B.C}.tgz helm chart\u5305\u3002helm package ./helm-chart\u5373\u53ef\u3002\u63d0PR\u5c06release\u5206\u652f\u5408\u5165main\uff0c\u53ef\u7b49\u7a33\u5b9a\u4e00\u6bb5\u65f6\u95f4\u540e\u518d\u5408\u5165\uff0c\u6ce8\u610fLoggie\u5de5\u7a0b\u548cLoggie-installation\u90e8\u7f72\u811a\u672c\u5747\u9700\u8981\u3002
"},{"location":"developer-guide/release/#bugfix","title":"\u6709BugFix","text":"\u53c2\u8003\uff1a
time.after()\u3002\u4f7f\u7528time.NewTicker()\u4ee3\u66ff\uff0c\u5e76\u53ca\u65f6\u6e05\u7406tickerlog.Info()\u3001log.Error()\u3002\u4e0d\u5141\u8bb8\u4f7f\u7528fmt.Printf()\u3001fmt.Println()\u7b49go\u5185\u7f6efmt\u65b9\u6cd5\u3002\u9879\u76ee\u65e5\u5fd7\u7cfb\u7edf\u91c7\u7528 zerologPANIC\u548cFATAL\u533a\u522b
PANIC\u6700\u7ec8\u8c03\u7528\u7684\u662fpanic(msg)
FATAL\u6700\u7ec8\u8c03\u7528\u7684\u662fos.Exit(1)
"},{"location":"developer-guide/code/coding-guide/#_5","title":"\u76d1\u63a7","text":"yaml:\"cleanDataTimeout,omitempty\"\uff0c\u4ee3\u66ffyaml:\"clean_data_timeout,omitempty\"Copyright-text
/*\nCopyright 2022 Loggie Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n*/\n\u63a8\u8350\u53c2\u8003\u5df2\u6709\u7684\u7ec4\u4ef6\u5f00\u53d1\u4e00\u4e2a\u65b0\u7684\u7ec4\u4ef6
"},{"location":"developer-guide/component/component-guide/#_1","title":"\u516c\u5171\u63a5\u53e3","text":"// \u751f\u547d\u5468\u671f\u63a5\u53e3\ntype Lifecycle interface {\n// \u521d\u59cb\u5316\uff0c\u4f8b\u5982\u521d\u59cb\u5316Kafka\u8fde\u63a5\nInit(context Context)\n// \u542f\u52a8\u8fd0\u884c\uff0c\u4f8b\u5982\u5f00\u59cb\u6d88\u8d39Kafka\nStart()\n// \u505c\u6b62\nStop()\n}\n\n// \u63cf\u8ff0\u63a5\u53e3\ntype Describable interface {\n// \u7c7b\u522b\uff0c\u4f8b\u5982source\nCategory() Category\n// \u7c7b\u578b\uff0c\u4f8b\u5982kafka\nType() Type\n// \u81ea\u5b9a\u4e49\u63cf\u8ff0\nString() string\n}\n\n// \u914d\u7f6e\u83b7\u53d6\u63a5\u53e3\ntype Config interface {\n// \u83b7\u53d6\u914d\u7f6e\nConfig() interface{}\n}\n\n// \u7ec4\u4ef6\u63a5\u53e3\ntype Component interface {\n// \u751f\u547d\u5468\u671f\u7ba1\u7406\nLifecycle\n// \u63cf\u8ff0\u7ba1\u7406\nDescribable\n// \u914d\u7f6e\u7ba1\u7406\nConfig\n}\nsource\u7ec4\u4ef6\u5bf9\u63a5\u6570\u636e\u6e90\u8f93\u5165\uff0c\u5f00\u53d1\u4e00\u4e2a\u65b0\u7684source\u63d2\u4ef6\u9700\u8981\u5b9e\u73b0\u5982\u4e0b\u63a5\u53e3
// source\u7ec4\u4ef6\u63a5\u53e3\ntype Source interface {\nComponent\nProducer\n// \u63d0\u4ea4\u63a5\u53e3\uff0c\u786e\u8ba4sink\u7aef\u6210\u529f\u7136\u540e\u63d0\u4ea4\nCommit(events []Event)\n}\n\n// \u751f\u4ea7\u63a5\u53e3\uff0csource\u7ec4\u4ef6\u9700\u8981\u5b9e\u73b0\ntype Producer interface {\n// \u5bf9\u63a5\u6570\u636e\u6e90\nProductLoop(productFunc ProductFunc)\n}\nsink\u7ec4\u4ef6\u5bf9\u63a5\u8f93\u51fa\u7aef\uff0c\u5f00\u53d1\u4e00\u4e2a\u65b0\u7684sink\u63d2\u4ef6\u9700\u8981\u5b9e\u73b0\u5982\u4e0b\u63a5\u53e3
// sink\u7ec4\u4ef6\u63a5\u53e3\ntype Sink interface {\nComponent\nConsumer\n}\n\n// \u6d88\u8d39\u63a5\u53e3\uff0csink\u7ec4\u4ef6\u9700\u8981\u5b9e\u73b0\ntype Consumer interface {\n// \u5bf9\u63a5\u8f93\u51fa\u7aef\nConsume(batch Batch) Result\n}\ninterceptor\u7ec4\u4ef6\u5bf9\u4e8b\u4ef6\u8fdb\u884c\u62e6\u622a\u5904\u7406\uff0c\u5f00\u53d1\u4e00\u4e2a\u65b0\u7684interceptor\u63d2\u4ef6\u9700\u8981\u5b9e\u73b0\u5982\u4e0b\u63a5\u53e3
// interceptor\u7ec4\u4ef6\u63a5\u53e3\ntype Interceptor interface {\nComponent\n// \u62e6\u622a\u5904\u7406\nIntercept(invoker Invoker, invocation Invocation) api.Result\n}\nNote
\u8bf7\u6ce8\u610f\u65b0\u589e\u7684\u7ec4\u4ef6\u9700\u8981\u653e\u5230pkg/include/include.go\u7684import\u5f53\u4e2d\u6ce8\u518c
"},{"location":"getting-started/overview/","title":"Loggie\u6f2b\u6e38\u6307\u5357","text":"\u6b22\u8fce\u6765\u5230Loggie\u7684\u4e16\u754c\u3002
"},{"location":"getting-started/overview/#_1","title":"\u4e86\u89e3","text":"\u60f3\u77e5\u9053\u4ec0\u4e48\u662fLoggie\uff1f\u4e86\u89e3\u4e00\u4e0b\u300c\u6838\u5fc3\u6982\u5ff5\u300d\u3002
\u5982\u679c\u4f60\u5e0c\u671b\u5feb\u901f\u4f53\u9a8cLoggie\uff0c\u8bf7\u53c2\u8003\u300c\u5feb\u901f\u4e0a\u624b\u300d\u3002
\u6709\u66f4\u591a\u5174\u8da3\uff1f\u60f3\u77e5\u9053\u6211\u4eec\u4e3a\u4ec0\u4e48\u7814\u53d1Loggie\u3001Loggie\u6709\u4ec0\u4e48\u4f18\u52bf\u3001\u548c\u540c\u7c7b\u9879\u76ee\u7684\u5bf9\u6bd4\uff1f\u8bf7\u6233\u8fd9\u91cc\u3002
"},{"location":"getting-started/overview/#_2","title":"\u4f7f\u7528","text":"\u5982\u679c\u4f60\u5e0c\u671b\u5c1d\u8bd5\u4e00\u4e0bLoggie\uff0c\u4f46\u4e0d\u77e5\u9053\u5982\u4f55\u5f00\u59cb\uff1f
\u53ef\u4ee5\u5148\u4ece\u6574\u4f53\u601d\u8003\u4e00\u4e0b\uff1a\u5982\u4f55\u91c7\u96c6\u4e1a\u52a1\u7684\u65e5\u5fd7\uff0c\u5982\u4f55\u5f62\u6210\u4e00\u4e2a\u5b8c\u5584\u5e76\u9002\u5408\u5b9e\u9645\u9700\u6c42\u7684\u65e5\u5fd7\u67b6\u6784\uff1f \u8fd9\u91cc\u6709\u4e00\u4e2a\u53c2\u8003\u3002
\u63a5\u7740\uff0c\u4f60\u53ef\u4ee5\u9009\u62e9\u9002\u5408\u7684\u4e1a\u52a1\u300c\u573a\u666f\u300d\uff0c\u67e5\u770bLoggie\u5728Kubernetes\u4e0b\u7684\u6700\u4f73\u5b9e\u8df5\u3002
\u8fd9\u91cc\u8fd8\u6d89\u53ca\u5230\u5982\u4f55\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u300c\u90e8\u7f72\u300dLoggie\u3002
\u53e6\u5916\uff0c\u60f3\u67e5\u770b\u5177\u4f53\u7ec4\u4ef6\u7684\u4f7f\u7528\u65b9\u5f0f\uff0c\u8bf7\u53c2\u8003\u300c\u914d\u7f6e\u300d\u3002
"},{"location":"getting-started/overview/#_3","title":"\u89e3\u7b54","text":"\u5728\u4f7f\u7528Loggie\u7684\u65f6\u5019\u9047\u5230\u95ee\u9898\uff1f \u8bf7\u63d0issues\u6216\u8005\u8054\u7cfb\u6211\u4eec\u3002
\u5fae\u4fe1\u626b\u7801\u52a0\u5165Loggie\u8ba8\u8bba\u7fa4\uff1a\uff08\u8001\u7684\u5fae\u4fe1\u5df2\u6ee1\uff0c\u672a\u901a\u8fc7\u7684\u8bf7\u91cd\u65b0\u626b\u7801\uff09
"},{"location":"getting-started/overview/#_4","title":"\u53c2\u4e0e","text":"\u5982\u679c\u4f60\u5bf9Loggie\u7684\u5177\u4f53\u5b9e\u73b0\u611f\u5174\u8da3\uff0c\u60f3\u53c2\u4e0eLoggie\u5f00\u6e90\u7684\u7814\u53d1\uff1f\u60f3\u81ea\u7814\u63d2\u4ef6\uff1f\u8bf7\u770b\u300c\u5f00\u53d1\u624b\u518c\u300d\u3002
"},{"location":"getting-started/install/kubernetes/","title":"Kubernetes\u90e8\u7f72","text":""},{"location":"getting-started/install/kubernetes/#loggie-daemonset","title":"\u90e8\u7f72Loggie DaemonSet","text":"\u8bf7\u786e\u4fdd\u672c\u5730\u6709kubectl\u548chelm\u53ef\u6267\u884c\u547d\u4ee4\u3002
VERSION=v1.4.0\nhelm pull https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz && tar xvzf loggie-$VERSION.tgz\n<VERSION>\u66ff\u6362\u6210\u5177\u4f53\u7684\u7248\u672c\u53f7\u3002\u6bd4\u5982v1.4.0\u3002 \u53ef\u4ecerelease tag\u91cc\u627e\u3002"},{"location":"getting-started/install/kubernetes/#_1","title":"\u4fee\u6539\u914d\u7f6e","text":"\u8fdb\u5165chart\u76ee\u5f55\uff1a
cd installation/helm-chart\n\u67e5\u770bvalues.yml\uff0c\u5e76\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u8fdb\u884c\u4fee\u6539\u3002
\u76ee\u524d\u53ef\u914d\u7f6e\u7684\u53c2\u6570\u5982\u4e0b\u6240\u793a\uff1a
"},{"location":"getting-started/install/kubernetes/#_2","title":"\u955c\u50cf","text":"image: loggieio/loggie:main\nresources:\nlimits:\ncpu: 2\nmemory: 2Gi\nrequests:\ncpu: 100m\nmemory: 100Mi\nextraArgs: {}\nextraArgs:\nlog.level: debug\nlog.jsonFormat: false\nextraVolumeMounts:\n- mountPath: /var/log/pods\nname: podlogs\n- mountPath: /var/lib/kubelet/pods\nname: kubelet\n- mountPath: /var/lib/docker\nname: docker\n\n\nextraVolumes:\n- hostPath:\npath: /var/log/pods\ntype: DirectoryOrCreate\nname: podlogs\n- hostPath:\npath: /var/lib/kubelet/pods\ntype: DirectoryOrCreate\nname: kubelet\n- hostPath:\npath: /var/lib/docker\ntype: DirectoryOrCreate\nname: docker\n\u8fd9\u91cc\u7b80\u5355\u5217\u4e3e\u4e00\u4e0b\u91c7\u96c6\u4ec0\u4e48\u6837\u7684\u65e5\u5fd7\u9700\u8981\u6302\u8f7d\u4ec0\u4e48\u8def\u5f84\uff1a
\u91c7\u96c6 stdout \u6807\u51c6\u8f93\u51fa\uff1aLoggie\u4f1a\u4ece/var/log/pods\u4e0b\u91c7\u96c6\uff0c\u6240\u4ee5Loggie\u9700\u8981\u6302\u8f7d\uff1a
volumeMounts:\n- mountPath: /var/log/pods\nname: podlogs\n- mountPath: /var/lib/docker\nname: docker volumes:\n- hostPath:\npath: /var/log/pods\ntype: DirectoryOrCreate\nname: podlogs\n- hostPath:\npath: /var/lib/docker\ntype: DirectoryOrCreate\nname: docker\n\u4f46\u662f\u6709\u53ef\u80fd/var/log/pods\u4e0b\u8fd9\u4e9b\u65e5\u5fd7\u6587\u4ef6\u4f1a\u8f6f\u94fe\u63a5\u5230docker\u7684root\u8def\u5f84\u4e0b\uff0c\u9ed8\u8ba4\u4e3a/var/lib/docker\uff0c\u8fd9\u4e2a\u65f6\u5019\uff0c\u9700\u8981\u540c\u6837\u6302\u8f7d/var/lib/docker\u8fd9\u4e2a\u8def\u5f84\u5230Loggie\u4e2d\uff1a
volumeMounts:\n- mountPath: /var/log/pods\nname: podlogs\n\nvolumes:\n- hostPath:\npath: /var/log/pods\ntype: DirectoryOrCreate\nname: podlogs\n\u53e6\u5916\uff0c\u5982\u679c\u975edocker\u8fd0\u884c\u65f6\uff0c\u6bd4\u5982\u4f7f\u7528containerd\uff0c\u65e0\u9700\u6302\u8f7d/var/lib/docker\uff0cLoggie\u4f1a\u4ece/var/log/pods\u4e2d\u5bfb\u627e\u5b9e\u9645\u7684\u6807\u51c6\u8f93\u51fa\u8def\u5f84\u3002
\u91c7\u96c6\u4e1a\u52a1Pod\u4f7f\u7528 HostPath \u6302\u8f7d\u7684\u65e5\u5fd7\uff1a\u6bd4\u5982\u4e1a\u52a1\u7edf\u4e00\u5c06\u65e5\u5fd7\u6302\u8f7d\u5230\u4e86\u8282\u70b9\u7684/data/logs\u8def\u5f84\u4e0b\uff0c\u5219\u9700\u8981\u6302\u8f7d\u6302\u8f7d\u8be5\u8def\u5f84:
volumeMounts:\n- mountPath: /data/logs\nname: logs\n\nvolumes:\n- hostPath:\npath: /data/logs\ntype: DirectoryOrCreate\nname: logs\n/var/lib/kubelet/pods\u8def\u5f84\u4e0b\uff0c\u6240\u4ee5\u9700\u8981Loggie\u6302\u8f7d\u8be5\u8def\u5f84\u3002\u5982\u679c\u73af\u5883\u7684kubelet\u4fee\u6539\u4e86\u8be5\u8def\u5f84\u914d\u7f6e\uff0c\u8fd9\u91cc\u9700\u8981\u540c\u6b65\u4fee\u6539:volumeMounts:\n- mountPath: /var/lib/kubelet/pods\nname: kubelet\n\nvolumes:\n- hostPath:\npath: /var/lib/kubelet/pods\ntype: DirectoryOrCreate\nname: kubelet\n\u91c7\u96c6\u4e1a\u52a1Pod \u65e0\u6302\u8f7d \uff0c\u540c\u65f6\u8bbe\u7f6e\u4e86rootFsCollectionEnabled: true\uff0cLoggie\u4f1a\u81ea\u52a8\u4ecedocker\u7684rootfs\u91cc\u627e\u5230\u5bb9\u5668\u91cc\u7684\u5b9e\u9645\u8def\u5f84\uff0c\u6b64\u65f6\u9700\u8981\u6302\u8f7ddocker\u7684root\u8def\u5f84\uff1a
volumeMounts:\n- mountPath: /var/lib/docker\nname: docker volumes:\n- hostPath:\npath: /var/lib/docker\ntype: DirectoryOrCreate\nname: docker\n/data/docker\uff0c\u5219\u6302\u8f7d\u4e3a\uff1a volumeMounts:\n- mountPath: /data/docker\nname: docker volumes:\n- hostPath:\npath: /data/docker\ntype: DirectoryOrCreate\nname: docker\n\u5176\u4ed6\uff1a
/data/loggie--{{ template \"loggie.name\" . }}\u76ee\u5f55\u3002nodeSelector: {}\n\naffinity: {}\n# podAntiAffinity:\n# requiredDuringSchedulingIgnoredDuringExecution:\n# - labelSelector:\n# matchExpressions:\n# - key: app\n# operator: In\n# values:\n# - loggie\n# topologyKey: \"kubernetes.io/hostname\"\ntolerations: []\n# - effect: NoExecute\n# operator: Exists\n# - effect: NoSchedule\n# operator: Exists\nupdateStrategy:\ntype: RollingUpdate\nRollingUpdate\u6216\u8005OnDelete\u3002 "},{"location":"getting-started/install/kubernetes/#_8","title":"\u5168\u5c40\u914d\u7f6e","text":"config:\nloggie:\nreload:\nenabled: true\nperiod: 10s\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~\ndiscovery:\nenabled: true\nkubernetes:\ncontainerRuntime: containerd\nfields:\ncontainer.name: containername\nlogConfig: logconfig\nnamespace: namespace\nnode.name: nodename\npod.name: podname\nhttp:\nenabled: true\nport: 9196\ncontainerRuntime: containerd\uff0c\u6307\u5b9a\u5bb9\u5668\u8fd0\u884c\u65f6\u3002 "},{"location":"getting-started/install/kubernetes/#service","title":"service","text":"\u5982\u679cLoggie\u5e0c\u671b\u63a5\u6536\u5176\u4ed6\u670d\u52a1\u53d1\u9001\u7684\u6570\u636e\uff0c\u9700\u8981\u5c06\u81ea\u8eab\u7684\u670d\u52a1\u901a\u8fc7service\u66b4\u9732\u51fa\u6765\u3002
\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u4f7f\u7528Agent\u6a21\u5f0f\u7684Loggie\u53ea\u9700\u8981\u66b4\u9732\u81ea\u8eab\u7ba1\u7406\u7aef\u53e3\u3002
servicePorts:\n- name: monitor\nport: 9196\ntargetPort: 9196\n\u521d\u6b21\u90e8\u7f72\uff0c\u6211\u4eec\u6307\u5b9a\u90e8\u7f72\u5728loggie namespace\u4e0b\uff0c\u5e76\u8ba9helm\u81ea\u52a8\u521b\u5efa\u8be5namespace\u3002
helm install loggie ./ -nloggie --create-namespace\n\u5982\u679c\u4f60\u7684\u73af\u5883\u4e2d\u5df2\u7ecf\u521b\u5efa\u4e86loggie namespace\uff0c\u53ef\u4ee5\u5ffd\u7565\u5176\u4e2d\u7684-nloggie\u548c--create-namespace\u53c2\u6570\u3002\u5f53\u7136\uff0c\u4f60\u4e5f\u53ef\u4ee5\u4f7f\u7528\u81ea\u5df1\u7684namespace\uff0c\u5c06\u5176\u4e2dloggie\u66ff\u6362\u5373\u53ef\u3002
Kubernetes\u7248\u672c\u95ee\u9898
failed to install CRD crds/crds.yaml: unable to recognize \"\": no matches for kind \"CustomResourceDefinition\" in version \"apiextensions.k8s.io/v1\"\nrm loggie/crds/crds.yaml\uff0c\u91cd\u65b0install\u3002"},{"location":"getting-started/install/kubernetes/#_10","title":"\u67e5\u770b\u90e8\u7f72\u72b6\u6001","text":"\u6267\u884c\u5b8c\u540e\uff0c\u901a\u8fc7\u4f7f\u7528helm\u547d\u4ee4\u6765\u67e5\u770b\u90e8\u7f72\u72b6\u6001\uff1a
helm list -nloggie\nNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION\nloggie loggie 1 2021-11-30 18:06:16.976334232 +0800 CST deployed loggie-v0.1.0 v0.1.0\n\u540c\u65f6\u4e5f\u53ef\u4ee5\u901a\u8fc7kubectl\u547d\u4ee4\u67e5\u770bPod\u662f\u5426\u5df2\u7ecf\u88ab\u521b\u5efa\u3002
kubectl -nloggie get po\nloggie-sxxwh 1/1 Running 0 5m21s 10.244.0.5 kind-control-plane <none> <none>\n\u90e8\u7f72Aggregator\u57fa\u672c\u548cAgent\u4e00\u81f4\uff0c\u5728helm chart\u4e2d\u6211\u4eec\u63d0\u4f9b\u4e86aggregator config\u90e8\u5206\uff0c\u53ea\u9700\u6539\u6210enabled: true\u5373\u53ef\u3002
helm chart\u4e2d\u63d0\u4f9b\u4e86statefulSet\u7684\u90e8\u7f72\u65b9\u5f0f\uff0c\u4f60\u4e5f\u53ef\u4ee5\u6839\u636e\u9700\u6c42\u4fee\u6539\u6210deployment\u7b49\u65b9\u5f0f\u3002
\u540c\u65f6\uff0c\u8bf7\u6ce8\u610f\u5728values.yaml\u4e2d\u6839\u636e\u60c5\u51b5\u589e\u52a0\uff1a
servicePorts:\n- name: grpc\nport: 6066\ntargetPort: 6066\nconfig:\nloggie:\ndiscovery:\nenabled: true\nkubernetes:\ncluster: aggregator\n\u6267\u884c\u90e8\u7f72\u547d\u4ee4\u53c2\u8003\uff1a
helm install loggie-aggregator ./ -nloggie-aggregator --create-namespace\nNote
Loggie\u4e2d\u8f6c\u673a\u540c\u6837\u53ef\u4ee5\u4f7f\u7528Deployment\u6216\u8005StatefulSet\u6765\u90e8\u7f72\uff0c\u8bf7\u53c2\u8003DaemonSet\u81ea\u884c\u4fee\u6539helm chart\u3002
"},{"location":"getting-started/install/node/","title":"\u4e3b\u673a\u90e8\u7f72","text":"Loggie\u4f7f\u7528Golang\u7f16\u8bd1\u6210\u4e8c\u8fdb\u5236\uff0c\u53ef\u6839\u636e\u81ea\u8eab\u9700\u6c42\u5bf9\u63a5\u5404\u7c7b\u90e8\u7f72\u7cfb\u7edf\u3002 \u8fd9\u91cc\u6211\u4eec\u63d0\u4f9b\u4e00\u4e2a\u4f7f\u7528systemd\u90e8\u7f72Loggie\u7684\u53c2\u8003\u3002
"},{"location":"getting-started/install/node/#_2","title":"\u524d\u7f6e\u68c0\u67e5","text":"\u76ee\u524drelease\u4ec5\u5305\u542bGOOS=linux GOARCH=amd64\u751f\u6210\u7684\u4e8c\u8fdb\u5236\u53ef\u6267\u884c\u6587\u4ef6\u3002\u5176\u4ed6\u7cfb\u7edf\u548c\u67b6\u6784\uff0c\u8bf7\u81ea\u884c\u57fa\u4e8e\u6e90\u7801\u4ea4\u53c9\u7f16\u8bd1\u3002
"},{"location":"getting-started/install/node/#_3","title":"\u4e0b\u8f7d\u4e8c\u8fdb\u5236","text":"VERSION=v1.4.0\nmkdir /opt/loggie && curl https://github.com/loggie-io/loggie/releases/download/$VERSION/loggie -o /opt/loggie/loggie && chmod +x /opt/loggie/loggie\n\u8bf7\u5c06\u4ee5\u4e0a\u7684<VERSION>\u66ff\u6362\u6210\u5177\u4f53\u7684\u7248\u672c\u53f7\u3002
\u8bf7\u6839\u636e\u5b9e\u9645\u9700\u6c42\u521b\u5efa\u914d\u7f6e\uff0c\u4ee5\u4e0b\u4e3a\u53c2\u8003\uff1a
"},{"location":"getting-started/install/node/#loggieyml","title":"\u521b\u5efaloggie.yml","text":"loggie.yml
cat << EOF > /opt/loggie/loggie.yml\nloggie:\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~\n\nreload:\nenabled: true\nperiod: 10s\n\nhttp:\nenabled: true\nport: 9196\nEOF\npipelines.yml
cat << EOF > /opt/loggie/pipelines.yml\npipelines:\n- name: local\nsources:\n- type: file\nname: demo\npaths:\n- /tmp/log/*.log\nsink:\ntype: dev\nprintEvents: true\ncodec:\npretty: true\nEOF\ncat << EOF > /lib/systemd/system/loggie.service\n[Unit]\nDescription=Loggie\nDocumentation=https://loggie-io.github.io/docs/getting-started/install/node/\n\n[Service]\nMemoryMax=200M\nExecStart=/opt/loggie/loggie -config.system=/opt/loggie/loggie.yml -config.pipeline=/opt/loggie/pipelines.yml\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\u9996\u5148\u751f\u6548\u914d\u7f6e\uff1a
systemctl daemon-reload\n\u7136\u540e\u8bbe\u7f6e\u4e3a\u5f00\u673a\u542f\u52a8\uff1a
systemctl enable loggie\n\u63a5\u7740\u5c31\u53ef\u4ee5\u6b63\u5f0f\u542f\u52a8Loggie\u4e86\uff1a
systemctl start loggie\n\u542f\u52a8\u540e\uff0c\u4f60\u53ef\u4ee5\u968f\u65f6\u67e5\u770b\u8fdb\u7a0b\u72b6\u6001\uff1a
systemctl status loggie\nLoggie\u662f\u4e00\u4e2a\u57fa\u4e8eGolang\u7684\u8f7b\u91cf\u7ea7\u3001\u9ad8\u6027\u80fd\u3001\u4e91\u539f\u751f\u65e5\u5fd7\u91c7\u96c6Agent\u548c\u4e2d\u8f6c\u5904\u7406Aggregator\uff0c\u652f\u6301\u591aPipeline\u548c\u7ec4\u4ef6\u70ed\u63d2\u62d4\uff0c\u63d0\u4f9b\u4e86\uff1a
\u4e0b\u9762\u5c06\u5e26\u4f60\u6f14\u793a\u5728\u4e00\u4e2aKubernetes\u96c6\u7fa4\u4e2d\uff0c\u901a\u8fc7\u521b\u5efaLogConfig CRD\u5feb\u901f\u91c7\u96c6Pod\u7684\u65e5\u5fd7\u3002
"},{"location":"getting-started/quick-start/kubernetes/#1-kubernetes","title":"1. \u51c6\u5907Kubernetes\u73af\u5883","text":"\u53ef\u4ee5\u4f7f\u7528\u73b0\u6709Kubernetes\u96c6\u7fa4\uff0c\u6216\u8005\u90e8\u7f72Kubernetes\u3002\u672c\u5730\u63a8\u8350\u4f7f\u7528Kind\u642d\u5efaKubernetes\u96c6\u7fa4\u3002
\u672c\u6587\u7684\u64cd\u4f5c\u9700\u8981\u5728\u672c\u5730\u4f7f\u7528:
\u8bf7\u786e\u4fdd\u672c\u5730\u6709kubectl\u548chelm\u53ef\u6267\u884c\u547d\u4ee4\u3002
"},{"location":"getting-started/quick-start/kubernetes/#2-loggie-daemonset","title":"2. \u90e8\u7f72Loggie DaemonSet","text":"\u4f60\u53ef\u4ee5\u5728 installation \u9875\u9762\u67e5\u770b\u6240\u6709\u53d1\u5e03\u7684\u90e8\u7f72chart\u3002
\u53ef\u4ee5\u9009\u62e9\uff1a
"},{"location":"getting-started/quick-start/kubernetes/#chart","title":"\u4e0b\u8f7dchart\u518d\u90e8\u7f72","text":"VERSION=v1.4.0\nhelm pull https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz && tar xvzf loggie-$VERSION.tgz\n<VERSION>\u66ff\u6362\u6210\u5177\u4f53\u7684\u7248\u672c\u53f7\u3002 \u7136\u540e\u90e8\u7f72\u5b89\u88c5\uff1a
helm install loggie ./loggie -nloggie --create-namespace\n\u5f53\u7136\u4f60\u4e5f\u53ef\u4ee5\uff1a
"},{"location":"getting-started/quick-start/kubernetes/#_1","title":"\u76f4\u63a5\u90e8\u7f72\uff1a","text":"helm install loggie -nloggie --create-namespace https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz\n<VERSION>\u66ff\u6362\u6210\u5177\u4f53\u7684\u7248\u672c\u53f7\u3002 \u60f3\u4f7f\u7528\u5176\u4ed6\u7248\u672c\u955c\u50cf\uff1f
\u4e3a\u4e86\u65b9\u4fbf\u4f53\u9a8c\u6700\u65b0\u7684Fix\u548c\u7279\u6027\uff0c\u6211\u4eec\u63d0\u4f9b\u4e86main\u5206\u652f\u6bcf\u6b21\u5408\u5e76\u540e\u7684\u955c\u50cf\u7248\u672c\uff0c\u53ef\u901a\u8fc7 \u8fd9\u91cc \u8fdb\u884c\u9009\u62e9\u3002 \u540c\u65f6\u4f60\u53ef\u4ee5\u5728helm install\u547d\u4ee4\u4e2d\u589e\u52a0--set image=loggieio/loggie:vX.Y.Z\u6765\u6307\u5b9a\u5177\u4f53\u7684Loggie\u955c\u50cf\u3002
\u90e8\u7f72\u6709\u95ee\u9898\uff1f
\u5982\u679c\u5c1d\u8bd5\u90e8\u7f72\u540e\u51fa\u73b0\u95ee\u9898\uff0c\u6216\u8005\u5728\u4f60\u7684\u73af\u5883\u4e2d\u4ee5\u4e0b\u6f14\u793a\u64cd\u4f5c\u672a\u6210\u529f\uff0c\u8bf7\u53c2\u8003Kubernetes\u4e0b\u90e8\u7f72Loggie\uff0c\u4fee\u6539\u76f8\u5173\u914d\u7f6e\u3002
"},{"location":"getting-started/quick-start/kubernetes/#3","title":"3. \u91c7\u96c6\u65e5\u5fd7","text":"Loggie\u5b9a\u4e49\u4e86Kubernetes CRD LogConfig\uff0c\u4e00\u4e2aLogConfig\u8868\u793a\u91c7\u96c6\u4e00\u7c7bPods\u7684\u65e5\u5fd7\u91c7\u96c6\u4efb\u52a1\u3002
"},{"location":"getting-started/quick-start/kubernetes/#31-pods","title":"3.1 \u521b\u5efa\u88ab\u91c7\u96c6\u7684Pods","text":"\u6211\u4eec\u5148\u521b\u5efa\u4e00\u4e2aPod\u7528\u4e8e\u88ab\u91c7\u96c6\u65e5\u5fd7\u7684\u5bf9\u8c61\u3002
kubectl create deploy nginx --image=nginx\n\u63a5\u7740\uff0c\u6211\u4eec\u521b\u5efa\u4e00\u4e2aLoggie\u5b9a\u4e49\u7684CRD Sink\u5b9e\u4f8b\uff0c\u8868\u660e\u65e5\u5fd7\u53d1\u9001\u7684\u540e\u7aef\u3002 \u4e3a\u4e86\u65b9\u4fbf\u6f14\u793a\uff0c\u8fd9\u91cc\u6211\u4eec\u5c06\u65e5\u5fd7\u53d1\u9001\u81f3Loggie Agent\u81ea\u8eab\u7684\u65e5\u5fd7\u4e2d\u5e76\u6253\u5370\u3002
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: default\nspec:\nsink: |\ntype: dev\nprintEvents: true\nEOF\n\u53ef\u4ee5\u901a\u8fc7kubectl get sink\u67e5\u770b\u5230\u5df2\u521b\u5efa\u7684Sink\u3002
Loggie\u5b9a\u4e49CRD LogConfig\uff0c\u8868\u793a\u4e00\u4e2a\u65e5\u5fd7\u91c7\u96c6\u4efb\u52a1\u3002\u6211\u4eec\u521b\u5efa\u4e00\u4e2aLogConfig\u793a\u4f8b\u5982\u4e0b\u6240\u793a\uff1a
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: nginx\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: nginx\npipeline:\nsources: |\n- type: file\nname: mylog\npaths:\n- stdout\nsinkRef: default\nEOF\n\u53ef\u4ee5\u770b\u5230\uff0c\u4e0a\u9762\u4f7f\u7528\u4e86sinkRef\u5f15\u7528\u4e86\u521a\u624d\u521b\u5efa\u7684sink default CR\u3002\u5f53\u7136\uff0c\u6211\u4eec\u8fd8\u53ef\u4ee5\u76f4\u63a5\u5728Logconfig\u4e2d\u4f7f\u7528sink\u5b57\u6bb5\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: nginx\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: nginx\npipeline:\nsources: |\n- type: file\nname: mylog\npaths:\n- stdout\nsink: |\ntype: dev\nprintEvents: true\ncodec:\ntype: json\npretty: true\nEOF\n\u521b\u5efa\u5b8c\u4e4b\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u4f7f\u7528kubectl get lgc\u67e5\u770b\u5230\u521b\u5efa\u7684CRD\u5b9e\u4f8b\u3002
\u540c\u65f6\uff0c\u6211\u4eec\u8fd8\u53ef\u4ee5\u901a\u8fc7kubectl describe lgc nginx\u67e5\u770bLogConfig\u7684\u4e8b\u4ef6\uff0c\u4ee5\u83b7\u53d6\u6700\u65b0\u7684\u72b6\u6001\u3002
Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal syncSuccess 52s loggie/kind-control-plane Sync type pod [nginx-6799fc88d8-5cb67] success\n\u4e0a\u9762\u7684nginx LogConfig\u901a\u8fc7\u5176\u4e2d\u7684spec.selector\u6765\u5339\u914d\u91c7\u96c6\u54ea\u4e9bPod\u7684\u65e5\u5fd7\uff0c\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528app: nginx\u9009\u62e9\u4e86\u521a\u624d\u521b\u5efa\u7684nginx Pod\u3002 spec.pipeline\u5219\u8868\u793aLoggie\u7684Pipeline\u914d\u7f6e\uff0c\u6211\u4eec\u53ea\u91c7\u96c6\u5bb9\u5668\u6807\u51c6\u8f93\u51fa\u7684\u65e5\u5fd7\uff0c\u6240\u4ee5\u5728paths\u4e2d\u586b\u5199stdout\u5373\u53ef\u3002
\u9996\u5148\u627e\u5230\u6240\u5728\u7684nginx pod\u8282\u70b9\uff1a
kubectl get po -owide -l app=nginx\n\u7136\u540e\u6211\u4eec\u627e\u5230\u8be5\u8282\u70b9\u7684Loggie\uff1a
kubectl -nloggie get po -owide |grep ${node}\nkubectl -nloggie logs -f ${logge-pod}\n\u4e0a\u6587\u53ea\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5feb\u901f\u6f14\u793a\uff0c\u90e8\u7f72\u51fa\u73b0\u95ee\u9898\u6216\u8005\u60f3\u4e86\u89e3\u66f4\u591aKubernetes\u4e0bLoggie\u5982\u4f55\u4f7f\u7528\uff1f
\u4e0b\u9762\u6211\u4eec\u5c06\u6f14\u793a\u4e00\u4e2a\u6700\u7b80\u5355\u7684\u91c7\u96c6\u8282\u70b9\u65e5\u5fd7\u6587\u4ef6\u7684\u573a\u666f\u3002
"},{"location":"getting-started/quick-start/node/#1","title":"1. \u4e0b\u8f7d\u53ef\u6267\u884c\u6587\u4ef6","text":"\u8bf7\u627e\u4e00\u53f0Linux\u670d\u52a1\u5668\u8282\u70b9\uff0c\u4e0b\u8f7dLoggie\u4e8c\u8fdb\u5236\u53ef\u6267\u884c\u6587\u4ef6
VERSION=v1.4.0\ncurl -LJ https://github.com/loggie-io/loggie/releases/download/$VERSION/loggie -o loggie\n\u8bf7\u5c06\u4ee5\u4e0a\u7684<VERSION>\u66ff\u6362\u6210\u5177\u4f53\u7684\u7248\u672c\u53f7\u3002
\u6211\u4eec\u5148\u4f7f\u7528dev sink\u5c06file source\u91c7\u96c6\u7684\u65e5\u5fd7\u6587\u4ef6\u6253\u5370\u5230\u6807\u51c6\u8f93\u51fa\uff0c\u590d\u5236\u4ee5\u4e0b\u5185\u5bb9\u4e3apipelines.yml\u6587\u4ef6\uff1a
pipelines.ymlcat << EOF > pipelines.yml\npipelines:\n- name: demo\nsources:\n- type: file\nname: mylog\npaths:\n- \"/var/log/*.log\"\nsink:\ntype: dev\nprintEvents: true\nEOF\n/var/log\u76ee\u5f55\u4e0b\u6ee1\u8db3*.log\u5339\u914d\u89c4\u5219\u7684\u65e5\u5fd7\u6587\u4ef6\u3002\u6587\u4ef6\u91c7\u96c6\u540e\uff0c\u6587\u4ef6\u4f1a\u88ab\u53d1\u9001\u81f3dev sink\u8f93\u51fa\u6e90\uff0c\u8be5sink\u4ec5\u4ec5\u5c06\u91c7\u96c6\u7684\u6587\u4ef6\u6253\u5370\u5230\u6807\u51c6\u8f93\u51fa\u3002 pipeline\u6587\u4ef6\u8868\u793a\u6211\u4eec\u60f3\u8981\u7684\u8f93\u5165\u3001\u8f93\u51fa\u7b49\u4e1a\u52a1\u76f8\u5173\u7684\u914d\u7f6e\uff0c\u9664\u4e86pipeline\u914d\u7f6e\u6587\u4ef6\u5916\uff0cLoggie\u8fd8\u9700\u8981\u6709\u4e00\u4e2a\u5168\u5c40\u7684\u914d\u7f6e\u6587\u4ef6\u3002
// loggie.yml\ncat << EOF > loggie.yml\nloggie:\nreload:\nenabled: true\nperiod: 10s\nEOF\n\u8fd9\u91cc\u6211\u4eec\u53ea\u5c55\u793a\u4e86\u4e00\u4e2a\u6bd4\u8f83\u7b80\u5355\u7684\u914d\u7f6e\uff0c\u8868\u793a\u6253\u5f00loggie\u7684\u52a8\u6001\u914d\u7f6ereload\u529f\u80fd\uff0c\u540c\u65f6\u95f4\u9694\u68c0\u67e5\u65f6\u95f4\u4e3a10s\u3002
\u5728\u8282\u70b9\u4e0a\u589e\u52a0\u4ee5\u4e0a\u4e24\u4e2a\u914d\u7f6e\u6587\u4ef6\u540e\uff0c\u6211\u4eec\u5c31\u53ef\u4ee5\u5f00\u59cb\u542f\u52a8Loggie\u4e86\u3002
"},{"location":"getting-started/quick-start/node/#3","title":"3. \u8fd0\u884c","text":"./loggie -config.system=./loggie.yml -config.pipeline=./pipelines.yml -log.jsonFormat=false\n\u542f\u52a8\u53c2\u6570\u91cc\uff0c\u586b\u5165\u4e0a\u9762\u7684loggie.yml\u548cpipelines.yml\u7684\u6587\u4ef6\u8def\u5f84\u3002
\u770b\u5230\u6b63\u5e38\u7684\u542f\u52a8\u65e5\u5fd7\u540e\uff0c\u8868\u660eLoggie\u5c31\u5f00\u59cb\u6b63\u5e38\u7684\u5de5\u4f5c\u4e86\u3002\u540c\u65f6\u8282\u70b9/var/log/*.log\u4e0b\u7684\u65e5\u5fd7\u6587\u4ef6\uff0c\u90fd\u4f1a\u88ab\u6253\u5370\u5230\u6807\u51c6\u8f93\u51fa\u3002
"},{"location":"getting-started/quick-start/quick-start/","title":"\u5feb\u901f\u4e0a\u624b","text":"Loggie\u662f\u4e00\u4e2a\u65e5\u5fd7\u91c7\u96c6Agent\u548c\u4e2d\u8f6c\u5904\u7406\u7684Aggregator\uff0c\u5305\u542b\u591a\u4e2aPipeline\u7ba1\u9053\uff0c\u6bcf\u4e2aPipeline\u53c8\u7531Source\u8f93\u5165\u3001Interceptor\u62e6\u622a\u5904\u7406\u548cSink\u8f93\u51fa\u7ec4\u6210\u3002
\u57fa\u4e8e\u8fd9\u79cd\u63d2\u4ef6\u5f0f\u8bbe\u8ba1\uff0cLoggie\u5e76\u4e0d\u5c40\u9650\u5728\u65e5\u5fd7\u91c7\u96c6\uff0c\u901a\u8fc7\u914d\u7f6e\u4e0d\u540c\u7684Source/Interceptor/Sink\uff0cLoggie\u53ef\u4ee5\u7ec4\u5408\u5b9e\u73b0\u5404\u79cd\u4e0d\u540c\u7684\u529f\u80fd\u3002
\u7b80\u5355\u8d77\u89c1\uff0c\u8fd9\u91cc\u6211\u4eec\u4ece\u91c7\u96c6\u65e5\u5fd7\u5f00\u59cb\u3002
\u9009\u62e9\u4f60\u7684\u90e8\u7f72\u73af\u5883\uff1a
"},{"location":"getting-started/quick-start/quick-start/#kubernetes","title":"Kubernetes","text":""},{"location":"getting-started/quick-start/quick-start/#_2","title":"\u4e3b\u673a","text":""},{"location":"getting-started/roadmap/roadmap-2023/","title":"2023 Loggie RoadMap","text":""},{"location":"getting-started/roadmap/roadmap-2023/#_1","title":"\u66f4\u591a\u7ec4\u4ef6\u4e0e\u529f\u80fd\u6269\u5c55","text":"Loggie\u7684\u914d\u7f6e\u4e3b\u8981\u5206\u4e3a\u4e24\u7c7b\uff1a
"},{"location":"reference/#_2","title":"\u7cfb\u7edf\u914d\u7f6e","text":"\u5168\u5c40\u7684\u7cfb\u7edf\u914d\u7f6e\uff0c\u542f\u52a8\u53c2\u6570\u4e2d\u4f7f\u7528-config.system\u6307\u5b9a\uff0c\u5305\u542b\u5982\u4e0b\uff1a
# loggie.yml\nloggie:\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~\n\ndiscovery:\nenabled: false\n\nreload:\nenabled: true\nperiod: 10s\n\ndefaults:\nsink:\ntype: dev\nsources:\n- type: file\nwatcher:\ncleanFiles:\nmaxHistory: 1\nhttp:\nenabled: true\nport: 9196\nPipeline\u7684\u914d\u7f6e\uff0c\u901a\u8fc7\u542f\u52a8\u53c2\u6570-config.pipeline\u6307\u5b9a\u3002\u8868\u793a\u961f\u5217\u4f7f\u7528\u7684Source\u3001Sink\u3001Queue\u548cInterceptor\u3002
pipelines:\n- name: demo # pipeline name\u5fc5\u586b\nsources:\n- type: ${sourceType}\nname: access # source\u4e2dname\u5fc5\u586b\n...\ninterceptors:\n- type: ${interceptorType}\n...\nsink:\ntype: ${sinkType}\n...\nLoggie\u5b9a\u4e49\u4e86\u4ee5\u4e0b\u51e0\u4e2aCRD\u7528\u4e8e\u5728Kubernetes\u96c6\u7fa4\u73af\u5883\u91cc\u4e0b\u53d1\u914d\u7f6e\uff1a
LogConfig\uff1anamespace\u7ea7\u522b\uff0c\u8868\u793a\u4e00\u4e2aPipeline\u914d\u7f6e\uff0c\u53ef\u7528\u4e8e\u91c7\u96c6Pods\u7684\u5bb9\u5668\u65e5\u5fd7\u3002
ClusterLogConfig\uff1acluster\u7ea7\u522b\uff0c\u8868\u793a\u4e00\u4e2aPipeline\u914d\u7f6e\uff0c\u5305\u62ec\u96c6\u7fa4\u7ea7\u522b\u7684\u8de8Namespace\u91c7\u96c6Pod\u5bb9\u5668\u65e5\u5fd7\uff0c\u91c7\u96c6Node\u8282\u70b9\u4e0a\u7684\u65e5\u5fd7\uff0c\u4ee5\u53ca\u4e3a\u67d0\u4e2aLoggie\u96c6\u7fa4\u4e0b\u53d1\u901a\u7528\u7684pipeline\u914d\u7f6e\u3002
Sink\uff1acluster\u7ea7\u522b\uff0c\u8868\u793a\u4e00\u4e2asink\u914d\u7f6e\uff0c\u53ef\u4ee5\u5728LogConfig/ClusterLogConfig\u4e2d\u5f15\u7528\u8be5Sink\u3002
Interceptors\uff1acluster\u7ea7\u522b\uff0c\u8868\u793a\u4e00\u4e2ainterceptors\u7ec4\uff0c\u53ef\u4ee5\u5728LogConfig\u4e2d\u5f15\u7528\u8be5interceptors\u7ec4\u3002
Note
ClusterLogConfig/LogConfig\u4e2d\u7684pipeline\u53ef\u4ee5\u5b9a\u4e49sink\u548cinterceptor\uff0c\u7528\u4e8e\u8be5pipeline\u7684sink/interceptor\u3002 \u5982\u679c\u4f60\u5e0c\u671b\u5728\u591a\u4e2aClusterLogConfig/LogConfig\u4e2d\u590d\u7528sink\u6216\u8005interceptor\uff0c\u53ef\u4ee5\u521b\u5efaSink/Interceptor CR\uff0c\u5728ClusterLogConfig/LogConfig\u4e2d\u4f7f\u7528sinkRef/interceptorRef\u8fdb\u884c\u5f15\u7528\u3002
"},{"location":"reference/apis/ops/","title":"\u8fd0\u7ef4\u7c7b\u63a5\u53e3","text":""},{"location":"reference/apis/ops/#_2","title":"\u67e5\u8be2\u65e5\u5fd7\u91c7\u96c6\u72b6\u6001","text":""},{"location":"reference/apis/ops/#url","title":"URL","text":"GET /api/v1/help/log
"},{"location":"reference/apis/ops/#_3","title":"\u63cf\u8ff0","text":"\u67e5\u8be2\u8be5Loggie Agent\u7684\u65e5\u5fd7\u91c7\u96c6\u72b6\u6001
"},{"location":"reference/apis/ops/#_4","title":"\u8bf7\u6c42\u53c2\u6570","text":"\u793a\u4f8b\uff1a
/api/v1/help/log?pipeline=test&status=pending\n<name> \u7ba1\u9053\u72b6\u6001\uff0c\u5bf9\u5e94\u914d\u7f6e\u4e2d\u7684pipeline name\uff0c\u53c2\u8003\u4e0b\u9762\u7684pipeline\u53c2\u6570 map"},{"location":"reference/apis/ops/#pipeline","title":"pipeline","text":"\u53c2\u6570\u540d\u79f0 \u8bf4\u660e \u53c2\u6570\u7c7b\u578b \u5907\u6ce8 source.<name> pipeline\u4e2dsource\u7684\u72b6\u6001\uff0c\u53c2\u8003\u4e0b\u9762\u7684source\u53c2\u6570 map"},{"location":"reference/apis/ops/#source","title":"source","text":"\u53c2\u6570\u540d\u79f0 \u8bf4\u660e \u53c2\u6570\u7c7b\u578b \u5907\u6ce8 paths \u914d\u7f6e\u6587\u4ef6source\u4e2d\u5b9a\u4e49\u7684path string\u6570\u7ec4 detail pipeline\u4e2dsource\u7684\u72b6\u6001 \u6570\u7ec4 detail[n].filename \u6587\u4ef6\u540d\u79f0 string detail[n].offset \u91c7\u96c6\u8fdb\u5ea6offset int detail[n].size \u6587\u4ef6\u5927\u5c0f int detail[n].modify \u6587\u4ef6\u6700\u8fd1\u7684\u66f4\u65b0\u65f6\u95f4 int unix milliseconds detail[n].ignored \u6587\u4ef6\u662f\u5426\u88ab\u5ffd\u7565\uff08\u7531file source\u4e2d\u7684ignoreOlder\u914d\u7f6e\u51b3\u5b9a\uff09 bool Example
{\n\"fdStatus\": {\n\"activeFdCount\": 0,\n\"inActiveFdCount\": 1\n},\n\"fileStatus\": {\n\"pipeline\": {\n\"local\": {\n\"source\": {\n\"demo\": {\n\"paths\": [\n\"/tmp/log/*.log\"\n],\n\"detail\": [\n{\n\"filename\": \"/tmp/log/access.log\",\n\"offset\": 469,\n\"size\": 469,\n\"modify\": 1673436846523,\n\"ignored\": false\n}\n]\n}\n}\n}\n}\n}\n}\nCluster\u7ea7\u522bCRD\uff0c\u53ef\u7528\u4e8e\uff1a
Example
apiVersion: loggie.io/v1beta1\nkind: ClusterLogConfig\nmetadata:\nname: test\nspec:\nselector:\ntype: node\nnodeSelector:\nnodepool: test\npipeline:\nsources: |\n- type: file\nname: messages\npaths:\n- /var/log/messages\nsinkRef: default\n\u8868\u793aPipeline\u914d\u7f6e\u9002\u7528\u7684\u8303\u56f4
"},{"location":"reference/discovery/kubernetes/clusterlogconfig/#type-pod","title":"type: pod","text":"\u901a\u8fc7Pipeline\u914d\u7f6e\u9009\u62e9\u4e00\u6279Pod\u8fdb\u884c\u65e5\u5fd7\u91c7\u96c6
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 labelSelector map \u5fc5\u586b \u901a\u8fc7\u8be5label\u6765\u5339\u914dPods\uff0c\u652f\u6301\u4f7f\u7528*\u6765\u5339\u914d\u6240\u6709\u7684value\uff0c\u6bd4\u5982app: '*' Example
spec: selector:\ntype: pod\nlabelSelector:\napp: nginx\napp: nginx\u7684\u6240\u6709Pod\u7684\u65e5\u5fd7\u3002 Warning
\u5728type: pod\u65f6\uff0c\u4e0b\u9762\u7684Pipeline\u53ea\u80fd\u4f7f\u7528file source\uff0c\u6b64\u65f6\u7684\u573a\u666f\u53ea\u80fd\u662f\u91c7\u96c6\u65e5\u5fd7\u3002
\u4e0b\u53d1Pipeline\u914d\u7f6e\u81f3\u8be5\u6279\u8282\u70b9\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 nodeSelector map \u5fc5\u586b \u901a\u8fc7label\u9009\u62e9\u4e0b\u53d1\u914d\u7f6e\u7684node Example
spec: selector:\ntype: node\nnodeSelector:\nnodepool: test\nnodepool: test\u7684\u6240\u6709node\u4e0a\u3002"},{"location":"reference/discovery/kubernetes/clusterlogconfig/#type-cluster","title":"type: cluster","text":"\u4e0b\u53d1Pipeline\u914d\u7f6e\u81f3\u67d0\u4e2aLoggie\u96c6\u7fa4\uff0c\u901a\u5e38\u9700\u8981\u914d\u5408cluster\u5b57\u6bb5\u6307\u5b9a\u96c6\u7fa4\u540d\u4f7f\u7528\u3002
Example
spec:\nselector:\ncluster: aggregator\ntype: cluster\ncluster\u4e3aaggregator\u7684Loggie\u96c6\u7fa4\u3002"},{"location":"reference/discovery/kubernetes/clusterlogconfig/#cluster","title":"cluster","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cluster string \u975e\u5fc5\u586b \"\" \u8868\u793a\u914d\u7f6e\u6307\u5b9a\u4e0b\u53d1\u7684Loggie\u96c6\u7fa4\u3002\u5f53\u90e8\u7f72\u591a\u5957Loggie\u65f6\uff0c\u548c\u5168\u5c40\u7cfb\u7edf\u914d\u7f6ediscovery.kubernetes.cluster\u914d\u5957\u4f7f\u7528"},{"location":"reference/discovery/kubernetes/clusterlogconfig/#specpipeline","title":"spec.pipeline","text":"\u914d\u7f6e\u548cLogConfig\u4e00\u81f4\u3002
"},{"location":"reference/discovery/kubernetes/interceptors/","title":"Interceptor","text":"\u8868\u793a\u4e00\u4e2ainterceptor\u7ec4\u3002\u7528\u4e8e\u5728LogConfig/ClusterLogConfig\u4e2d\u88ab\u5f15\u7528\u3002
Example
apiVersion: loggie.io/v1beta1\nkind: Interceptor\nmetadata:\nname: default\nspec:\ninterceptors: |\n- type: rateLimit\nqps: 90000\n\u4f7f\u7528\"|\"\u7b26\u53f7\u8868\u793a\u4e00\u6574\u6bb5interceptors\u914d\u7f6e\u5217\u8868\uff0c\u548cPipelines\u91cc\u7684\u914d\u7f6e\u4e00\u81f4\u3002
"},{"location":"reference/discovery/kubernetes/logconfig/","title":"Logconfig","text":"namespace\u7ea7\u522bCRD\uff0c\u8868\u793a\u4e00\u4e2a\u65e5\u5fd7\u91c7\u96c6\u4efb\u52a1\uff0c\u7528\u4e8e\u91c7\u96c6Pod\u5bb9\u5668\u65e5\u5fd7\u3002
Example
\u76f4\u63a5\u5b9a\u4e49sink/interceptor\u65b9\u5f0fapiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: tomcat\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: tomcat\npipeline:\nsources: |\n- type: file\nname: common\npaths:\n- stdout\nsink: |\ntype: dev\nprintEvents: false\ninterceptors: |\n- type: rateLimit\nqps: 90000\napiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: nginx\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: nginx\npipeline:\nsources: |\n- type: file\nname: mylog\npaths:\n- stdout\nsinkRef: default\ninterceptorRef: default \u8868\u793aPipeline\u914d\u7f6e\u9002\u7528\u7684\u8303\u56f4\uff0c\u53ef\u4ee5\u9009\u62e9\u91c7\u96c6\u4e00\u6279Pods\u7684\u65e5\u5fd7
"},{"location":"reference/discovery/kubernetes/logconfig/#type-pod","title":"type: pod","text":"\u91c7\u96c6Pods\u65e5\u5fd7
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 labelSelector map \u5fc5\u586b \u901a\u8fc7\u8be5label\u6765\u5339\u914dPods\uff0c\u652f\u6301\u4f7f\u7528*\u6765\u5339\u914d\u6240\u6709\u7684value\uff0c\u6bd4\u5982app: '*' Example
spec: selector:\ntype: pod\nlabelSelector:\napp: nginx\napp: nginx\u7684\u6240\u6709Pods\u65e5\u5fd7\u3002 Warning
\u5728type: pod\u65f6\uff0c\u4e0b\u9762\u7684Pipeline\u53ea\u652f\u6301\u4f7f\u7528file source\uff0c\u6b64\u65f6\u7684\u573a\u666f\u53ea\u80fd\u662f\u91c7\u96c6\u65e5\u5fd7\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cluster string \u975e\u5fc5\u586b \"\" \u8868\u793a\u914d\u7f6e\u6307\u5b9a\u4e0b\u53d1\u7684Loggie\u96c6\u7fa4\u3002\u5f53\u90e8\u7f72\u591a\u5957Loggie\u65f6\uff0c\u548c\u5168\u5c40\u7cfb\u7edf\u914d\u7f6ediscovery.kubernetes.cluster\u914d\u5957\u4f7f\u7528"},{"location":"reference/discovery/kubernetes/logconfig/#specpipeline","title":"spec.pipeline","text":"\u8868\u793a\u4e00\u4e2aPipeline\uff0c\u4e0d\u652f\u6301\u586b\u5199\u591a\u4e2aPipeline\u3002
\u548c\u5728\u914d\u7f6e\u6587\u4ef6\u4e2dPipelines\u7684\u533a\u522b\u5728\uff1a
\uff5c\u8868\u793a\u4fdd\u7559\u6362\u884c\u7b26\u5728LogConfig\u4e2d\uff0c\u5982\u679ctype: pod\uff0cfile source\u65b0\u589e\u51e0\u4e2a\u4e13\u95e8\u9488\u5bf9\u5bb9\u5668\u5316\u7684\u53c2\u6570\uff1a
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 containerName string \u975e\u5fc5\u586b \u8868\u793a\u6307\u5b9a\u91c7\u96c6\u7684\u5bb9\u5668\u540d\u79f0\uff0c\u5efa\u8bae\u5728Pod\u91cc\u5305\u542b\u591a\u4e2a\u5bb9\u5668\u65f6\u586b\u5199 excludeContainerPatterns string\u6570\u7ec4 \u975e\u5fc5\u586b \u6392\u9664\u7684\u5bb9\u5668\u540d\u79f0\uff0c\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f\u5f62\u5f0f"},{"location":"reference/discovery/kubernetes/logconfig/#sourcesmatchfields","title":"sources.matchFields","text":"\u975e\u5fc5\u586b, \u5c06Pod\u4e2d\u7684\u4fe1\u606f\u52a0\u5165\u5230Fields\u4e2d
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 labelKey string\u6570\u7ec4 \u975e\u5fc5\u586b \u6307\u5b9a\u589e\u52a0\u7684Pod\u4e0a\u7684Label Key\u503c\uff0c\u6bd4\u5982Pod\u4e0a\u5305\u542bLabel: app: demo\uff0c\u6b64\u5904\u586b\u5199labelKey: app\uff0c\u6b64\u65f6\u4f1a\u5c06Pod\u4e0a\u7684app: demo label\u589e\u52a0\u5230file source fields\u4e2d\uff0c\u91c7\u96c6\u5230\u7684\u65e5\u5fd7\u4f1a\u88ab\u52a0\u4e0a\u8be5label\u4fe1\u606f\u3002\u9002\u7528\u4e8e\u5339\u914d\u7684Pod\u7684label\u5b58\u5728\u4e0d\u4e00\u81f4\u7684\u573a\u666f\u3002\u652f\u6301\u914d\u7f6e\u4e3a\"*\"\u7684\u65b9\u5f0f\u83b7\u53d6\u6240\u6709\u7684label annotationKey string\u6570\u7ec4 \u975e\u5fc5\u586b \u548c\u4e0a\u9762labelKey\u7c7b\u4f3c\uff0c\u6ce8\u5165\u7684\u4e3aPod Annoatation\u7684\u503c\uff0c\u652f\u6301\u914d\u7f6e\u4e3a\"*\"\u7684\u65b9\u5f0f\u83b7\u53d6\u6240\u6709\u7684annotation env string\u6570\u7ec4 \u975e\u5fc5\u586b \u548c\u4e0a\u9762labelKey\u7c7b\u4f3c\uff0c\u6ce8\u5165\u7684\u4e3aPod Env\u73af\u5883\u53d8\u91cf\u7684\u503c\uff0c\u652f\u6301\u914d\u7f6e\u4e3a\"*\"\u7684\u65b9\u5f0f\u83b7\u53d6\u6240\u6709\u7684env reformatKeys \u975e\u5fc5\u586b \u91cd\u65b0\u683c\u5f0f\u5316key reformatKeys.label fmt\u53c2\u6570\u6570\u7ec4 \u975e\u5fc5\u586b \u91cd\u65b0\u683c\u5f0f\u5316label key reformatKeys.annotation fmt\u53c2\u6570\u6570\u7ec4 \u975e\u5fc5\u586b \u91cd\u65b0\u683c\u5f0f\u5316annotation key reformatKeys.env fmt\u53c2\u6570\u6570\u7ec4 \u975e\u5fc5\u586b \u91cd\u65b0\u683c\u5f0f\u5316env key fmt\u53c2\u6570
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 regex string \u975e\u5fc5\u586b \u5339\u914d\u7684\u6b63\u5219\u8868\u8fbe\u5f0f replace string \u975e\u5fc5\u586b \u91cd\u65b0\u6e32\u67d3\u7684\u683c\u5f0f reformatKeys
\u5047\u8bbepod labels\u4e3aaa.bb/foo=bar \u914d\u7f6ereformatKeys\u5982\u4e0b\uff1a
matchFields:\n reformatKeys:\n label:\n - regex: aa.bb/(.*)\n replace: pre-${1}\npre-foo=bar Example
apiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: nginx\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: nginx\npipeline:\nsources: |\n- type: file\nname: mylog\ncontainerName: nginx\nmatchFields:\nlabelKey: [\"app\"]\npaths:\n- stdout\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 interceptors string \u975e\u5fc5\u586b \u8868\u793a\u8be5Pipeline\u7684interceptor\uff0c\u4f7f\u7528\u65b9\u5f0f\u548c\u4ee5\u4e0asources\u7c7b\u4f3c"},{"location":"reference/discovery/kubernetes/logconfig/#sink","title":"sink","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sink string \u975e\u5fc5\u586b \u8868\u793a\u8be5Pipeline\u7684sink\uff0c\u4f7f\u7528\u65b9\u5f0f\u548c\u4ee5\u4e0a\u7684sources\u7c7b\u4f3c \u5982\u679c\u4f60\u5e0c\u671bsink\u548cinterceptor\u53ef\u4ee5\u5728\u4e0d\u540c\u7684ClusterLogConfig/LogConfig\u95f4\u590d\u7528\uff0c\u5219\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0bref\u7684\u65b9\u5f0f\uff1a
"},{"location":"reference/discovery/kubernetes/logconfig/#sinkref","title":"sinkRef","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sinkRef string \u975e\u5fc5\u586b \u8868\u793a\u8be5Pipeline\u5f15\u7528\u7684Sink CR"},{"location":"reference/discovery/kubernetes/logconfig/#interceptorref","title":"interceptorRef","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 interceptorRef string \u975e\u5fc5\u586b \u8868\u793a\u8be5Pipeline\u5f15\u7528\u7684Interceptor CR"},{"location":"reference/discovery/kubernetes/sink/","title":"Sink","text":"\u8868\u793a\u4e00\u4e2asink\u914d\u7f6e\u3002\u7528\u4e8e\u5728LogConfig/ClusterLogConfig\u4e2d\u88ab\u5f15\u7528\u3002
Example
apiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: default\nspec:\nsink: |\ntype: elasticsearch\nindex: \"loggie\"\nhosts: [\"elasticsearch-master.default.svc:9200\"]\n\u4f7f\u7528\"|\"\u7b26\u53f7\u8868\u793a\u4e00\u4e2asink\u914d\u7f6e\uff0c\u548cPipelines\u91cc\u7684\u914d\u7f6e\u4e00\u81f4\u3002
"},{"location":"reference/global/args/","title":"\u542f\u52a8\u53c2\u6570","text":""},{"location":"reference/global/args/#_1","title":"\u7cfb\u7edf\u53c2\u6570","text":"-config.from: \u9ed8\u8ba4\u4e3afile\uff0c\u5373\u9ed8\u8ba4\u4f7f\u7528\u6587\u4ef6\u7684\u914d\u7f6e\u65b9\u5f0f\u3002\u53ef\u9009\uff1aenv\uff0c\u6b64\u65f6\u4f1a\u4ece\u73af\u5883\u53d8\u91cf\u4e2d\u8bfb\u53d6\u914d\u7f6e\uff08\u6b64\u65f6\u4e0d\u652f\u6301reload\uff09\u3002-config.system: \u9ed8\u8ba4\u4e3aloggie.yml\uff0c\u8868\u793a\u6307\u5b9aLoggie\u7cfb\u7edf\u914d\u7f6e\u7684\u8def\u5f84\u548c\u6587\u4ef6\u540d\u79f0\u3002 \uff08\u5982\u679c-config.from=env\uff0c\u5219\u4e3asystem\u914d\u7f6e\u7684\u73af\u5883\u53d8\u91cf\u540d\u79f0\uff09-config.pipeline: \u9ed8\u8ba4\u4e3apipelines.yml\uff0c\u8868\u793aPipeline\u914d\u7f6e\u6587\u4ef6\u6240\u5728\u7684\u8def\u5f84\uff0c\u9700\u8981\u586b\u5199\u7b26\u5408glob\u5339\u914d\u7684\u8def\u5f84\uff0c\u6bd4\u5982\u5177\u4f53\u7684\u8def\u5f84\u548c\u6587\u4ef6\u540d/etc/loggie/pipelines.yml\uff0c\u6216\u8005glob\u5339\u914d\u7684\u65b9\u5f0f\uff0c\u6bd4\u5982/etc/loggie/*.yml\u3002 \uff08\u5982\u679c-config.from=env\uff0c\u5219\u4e3apipeline\u914d\u7f6e\u7684\u73af\u5883\u53d8\u91cf\u540d\u79f0\uff09Warning
\u503c\u5f97\u6ce8\u610f\u7684\u662f\uff0c\u5982\u679cconfig.pipeline=/etc/loggie\uff0cglob\u5339\u914d\u4f1a\u8ba4\u4e3a/etc/loggie\u4e3a/etc\u76ee\u5f55\u4e0b\u7684loggie\u6587\u4ef6\uff0c\u800c\u4e0d\u662f\u5339\u914d/etc/loggie\u76ee\u5f55\u4e0b\u7684\u6587\u4ef6\uff0c\u8bf7\u907f\u514d\u7c7b\u4f3c\u7684\u8bbe\u7f6e\u65b9\u5f0f
-meta.nodeName\uff1a\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u4f1a\u4f7f\u7528\u7cfb\u7edf\u7684hostname\uff0c\u5728Kubernetes\u90e8\u7f72\u4e2d\u4f1a\u4f7f\u7528Downward API\u6765\u6ce8\u5165nodeName\u3002\u4e00\u822c\u60c5\u51b5\u4e0b\u4e0d\u9700\u8981\u5355\u72ec\u914d\u7f6e-log.level: \u65e5\u5fd7\u7ea7\u522b\uff0c\u9ed8\u8ba4\u4e3ainfo\uff0c\u53ef\u914d\u7f6e\u4e3adebug\u3001info\u3001warn\u548cerror-log.jsonFormat: \u662f\u5426\u5c06\u65e5\u5fd7\u8f93\u51fa\u4e3ajson\u683c\u5f0f\uff0c\u9ed8\u8ba4\u4e3afalse-log.enableStdout: \u662f\u5426\u8f93\u51fa\u6807\u51c6\u8f93\u51fa\u65e5\u5fd7\uff0c\u9ed8\u8ba4\u4e3atrue-log.enableFile: \u662f\u5426\u8f93\u51fa\u65e5\u5fd7\u6587\u4ef6\uff0c\u9ed8\u8ba4\u4e3afalse\uff0c\u5373\u4e0d\u8f93\u51fa\u65e5\u5fd7\u6587\u4ef6\uff0c\u9ed8\u8ba4\u6253\u5370\u5230\u6807\u51c6\u8f93\u51fa-log.directory: \u65e5\u5fd7\u6587\u4ef6\u7684\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3a/var/log\uff0c\u5f53log.enableFile=true\u65f6\u751f\u6548-log.filename: \u65e5\u5fd7\u6587\u4ef6\u7684\u540d\u79f0\uff0c\u9ed8\u8ba4\u4e3aloggie.log\uff0c\u4e00\u822c\u540clog.directory\u642d\u914d\u4f7f\u7528-log.maxSize: \u65e5\u5fd7\u8f6e\u8f6c\u7684\u65f6\u5019\uff0c\u6700\u5927\u7684\u6587\u4ef6\u5927\u5c0f\uff0c\u9ed8\u8ba4\u4e3a1024MB-log.maxBackups: \u65e5\u5fd7\u8f6e\u8f6c\u6700\u591a\u4fdd\u7559\u7684\u6587\u4ef6\u4e2a\u6570\uff0c\u9ed8\u8ba4\u4e3a3-log.maxAge: \u65e5\u5fd7\u8f6e\u8f6c\u6700\u5927\u4fdd\u7559\u7684\u5929\u6570\uff0c\u9ed8\u8ba4\u4e3a7-log.timeFormat: \u6bcf\u884c\u65e5\u5fd7\u8f93\u51fa\u7684\u65f6\u95f4\u683c\u5f0f\uff0c\u9ed8\u8ba4\u683c\u5f0f\u4e3a2006-01-02 15:04:05-log.noColor: \u8f93\u51fa\u662f\u5426\u6709\u989c\u8272\u7f8e\u5316\uff0c\u9ed8\u8ba4\u4e3afalse\uff0c\u5373\u6709\u989c\u8272\u7f8e\u5316\uff0c\u5bf9\u4e8e\u4ece\u6807\u51c6\u8f93\u51fa\u67e5\u770b\u65e5\u5fd7\u6bd4\u8f83\u65b9\u4fbf\u3002\u5982\u679c\u8f93\u51fa\u5230\u65e5\u5fd7\u6587\u4ef6\u4e2d\uff0c\u5efa\u8bae\u8bbe\u7f6e\u4e3atrue\uff0c\u907f\u514d\u5f15\u5165\u989d\u5916\u7684\u683c\u5f0f\u3002Info
Loggie\u7684\u65e5\u5fd7\u8f6e\u8f6c\u4f7f\u7528lumberjack\u5e93
"},{"location":"reference/global/defaults/","title":"Defaults","text":"defaults\u7528\u4e8e\u8bbe\u7f6ePipelines\u914d\u7f6e\u4e2d\u7684\u9ed8\u8ba4\u503c\u3002\u5f53Pipeline\u4e2d\u6ca1\u6709\u8bbe\u7f6e\u503c\u65f6\u751f\u6548\uff0c\u6216\u8005\u7528\u4e8e\u8986\u76d6\u9ed8\u8ba4\u7684\u53c2\u6570\u3002
Example
defaults:\nsources:\n- type: file\nwatcher:\ncleanFiles:\nmaxHistory: 10\nsink:\ntype: dev\nprintEvents: true\n\u548cPipeline\u4e2d\u7684source\u4e00\u81f4\u3002\u5f53Pipelines\u914d\u7f6e\u4e86\u76f8\u540ctype\u7684source\u65f6\uff0c\u4f1a\u8986\u76d6\u5176\u4e2d\u672a\u586b\u5199\u5b57\u6bb5\u7684\u9ed8\u8ba4\u503c\u3002
\u6bd4\u5982:
sources:\n- type: file\nwatcher:\ncleanFiles:\nmaxHistory: 10\n\u548cPipeline\u4e2d\u7684sink\u4e00\u81f4\uff0c\u5982\u679c\u96c6\u7fa4\u53ea\u9700\u8981\u8bbe\u7f6e\u4e00\u4e2a\u5168\u5c40\u7684sink\u8f93\u51fa\u6e90\uff0c\u5219\u53ea\u9700\u8981\u5728\u8fd9\u91cc\u914d\u7f6e\u4e00\u6b21\uff0c\u907f\u514d\u5728\u6bcf\u4e2aPipeline\u4e2d\u586b\u5199\u3002
"},{"location":"reference/global/defaults/#interceptors","title":"interceptors","text":"defaults\u4e2d\u914d\u7f6e\u7684interceptors\u4f1a\u6dfb\u52a0\u5230pipeline\u4e2d\u5b9a\u4e49\u7684interceptors\u4e2d\uff0c\u4f46pipeline\u4e2d\u7684interceptor\u4f1a\u8986\u76d6defaults\u4e2d\u7684\u540c\u4e00\u4e2atype\u7684interceptor\u3002 \u5982\u679c\u4e0d\u5e0c\u671b\u8986\u76d6\u76f8\u540c\u7c7b\u578b\u7684interceptor\uff0c\u800c\u662f\u6dfb\u52a0\u76f8\u540ctype\u7684interceptor\uff0c\u4f9d\u6b21\u8fdb\u884c\u5904\u7406\uff0c\u9700\u8981\u989d\u5916\u586b\u5199name\u5b57\u6bb5\uff0c\u8fdb\u884c\u552f\u4e00\u6027\u6807\u8bc6\u3002
\u5728defaults\u4e2d\u5df2\u7ecf\u5b9a\u4e49normalize interceptor\u5982\u4e0b:
defaults:\ninterceptors:\n- type: normalize\nprocessor:\n- addMeta: ~\n\u5982\u679c\u5728pipelines\u4e2d\u5b9a\u4e49\u5982\u4e0bnormalize interceptor\uff1a
pipelines:\ninterceptors:\n- type: normalize\nprocessor:\n- drop:\n...\n\u6b64\u65f6defaults\u4e2d\u7684normalize interceptor\u4f1a\u88ab\u8986\u76d6\uff0c\u4e0d\u4f1a\u751f\u6548\u3002
\u5982\u679c\u6211\u4eec\u5e0c\u671b\u5148\u6267\u884cdefaults\u4e2d\u7684normalize interceptor\uff0c\u63a5\u7740\u518d\u6267\u884cpipeline\u4e2d\u7684normalize interceptor\uff0c\u53ef\u4ee5\u5728defaults\u4e2d\u6539\u4e3a\uff1a
defaults:\ninterceptors:\n- type: normalize\nname: global # \u7528\u4e8e\u533a\u5206pipelines\u4e2d\u7684normalize\norder: 500 # \u9ed8\u8ba4normalize\u7684order\u503c\u4e3a900\uff0c\u8fd9\u91cc\u5b9a\u4e49\u4e00\u4e2a\u76f8\u5bf9\u8f83\u5c0f\u503c\uff0c\u53ef\u63a7\u5236\u5148\u6267\u884cdefaults\u4e2d\u7684normalize\nprocessor:\n- addMeta: ~\nLoggie\u4f1a\u9ed8\u8ba4\u8bbe\u7f6emetric\u3001maxbytes\u3001retry3\u4e2a\u7cfb\u7edf\u5185\u7f6einterceptors\u3002 \u5982\u679c\u9700\u8981\u6dfb\u52a0\u5176\u4ed6\u7684\u9ed8\u8ba4interceptors\uff0c\u4f1a\u8986\u76d6\u6389\u4ee5\u4e0a\u7684\u5185\u7f6einterceptors\uff0c\u6240\u4ee5\u5f3a\u70c8\u5efa\u8bae\u6b64\u65f6\u5c06\u5185\u7f6einterceptors\u52a0\u4e0a\uff0c\u9664\u975e\u4f60\u786e\u8ba4\u4e0d\u9700\u8981\u4ee5\u4e0a\u7cfb\u7edf\u5185\u7f6e\u7684interceptors\u3002
\u9ed8\u8ba4\u4e3achannel queue\u3002
"},{"location":"reference/global/discovery/","title":"Discovery","text":"\u670d\u52a1\u53d1\u73b0\u548c\u914d\u7f6e\u4e0b\u53d1\u76f8\u5173\u7684\u914d\u7f6e\u3002\u76ee\u524d\u4e3b\u8981\u4e3aKubernetes\u76f8\u5173\u5168\u5c40\u914d\u7f6e\u3002
Example
discovery:\nenabled: true\nkubernetes:\n# Choose: docker or containerd\ncontainerRuntime: containerd\n# Collect log files inside the container from the root filesystem of the container, no need to mount the volume\nrootFsCollectionEnabled: false\n# Automatically parse and convert the wrapped container standard output format into the original log content\nparseStdout: false\n# If set to true, it means that the pipeline configuration generated does not contain specific Pod paths and meta information,\n# and these data will be dynamically obtained by the file source, thereby reducing the number of configuration changes and reloads.\ndynamicContainerLog: false\n# Automatically add fields when selector.type is pod in logconfig/clusterlogconfig\ntypePodFields:\nlogconfig: \"${_k8s.logconfig}\"\nnamespace: \"${_k8s.pod.namespace}\"\nnodename: \"${_k8s.node.name}\"\npodname: \"${_k8s.pod.name}\"\ncontainername: \"${_k8s.pod.container.name}\"\ntypeNodeFields:\nnodename: \"${_k8s.node.name}\"\nclusterlogconfig: \"${_k8s.clusterlogconfig}\"\nos: \"${_k8s.node.nodeInfo.osImage}\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u670d\u52a1\u53d1\u73b0\u914d\u7f6e\u4e0b\u53d1\u6a21\u5757"},{"location":"reference/global/discovery/#kubernetes","title":"Kubernetes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cluster string \u975e\u5fc5\u586b \"\" \u6807\u8bc6Loggie\u96c6\u7fa4\u540d\u79f0\u3002Loggie\u652f\u6301\u5728\u4e00\u4e2aKubernetes\u96c6\u7fa4\u4e2d\u90e8\u7f72\u591a\u5957Loggie\uff0c\u53ef\u4ee5\u901a\u8fc7\u5728LogConfig CRD\u4e2d\u6307\u5b9aselector.cluster\uff0c\u6307\u5b9a\u914d\u7f6e\u4e0b\u53d1\u7684Loggie\u96c6\u7fa4 kubeconfig string \u975e\u5fc5\u586b \u6307\u5b9a\u8bf7\u6c42Kubernetes\u96c6\u7fa4API\u7684kubeconfig\u6587\u4ef6\u3002\u901a\u5e38\u5728Loggie\u90e8\u7f72\u5230Kubernetes\u96c6\u7fa4\u4e2d\u65e0\u9700\u586b\u5199\uff0c\u6b64\u65f6\u4e3ainCluster\u6a21\u5f0f\u3002\u5982\u679cLoggie\u90e8\u7f72\u5728Kubernetes\u96c6\u7fa4\u5916\uff08\u4f8b\u5982\u672c\u5730\u8c03\u8bd5\u65f6\uff09\uff0c\u9700\u8981\u6307\u5b9a\u8be5kubeconfig\u6587\u4ef6\u3002 master string \u975e\u5fc5\u586b \u6307\u5b9a\u8bf7\u6c42Kubernetes\u96c6\u7fa4API\u7684master\u5730\u5740\uff0cinCluster\u6a21\u5f0f\u4e00\u822c\u65e0\u9700\u586b\u5199 containerRuntime string \u975e\u5fc5\u586b docker \u5bb9\u5668\u8fd0\u884c\u65f6\uff0c\u53ef\u9009docker\u3001containerd\u3001none rootFsCollectionEnabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u91c7\u96c6root filesystem\u91cc\u7684\u65e5\u5fd7\uff0c\u7528\u4e8e\u4e0d\u6302\u8f7d\u65e5\u5fd7volume\u7684\u573a\u666f parseStdout bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u81ea\u52a8\u63d0\u53d6\u5bb9\u5668\u6807\u51c6\u8f93\u51fa\u539f\u59cb\u5185\u5bb9 dynamicContainerLog bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u52a8\u6001\u5bb9\u5668\u65e5\u5fd7\u914d\u7f6e\uff0c\u6253\u5f00\u540e\u914d\u7f6e\u6587\u4ef6\u4e0d\u4f1a\u6e32\u67d3\u5177\u4f53\u7684path\u548c\u52a8\u6001fields\u5b57\u6bb5\uff0c\u53ef\u4ee5\u6709\u6548\u907f\u514d\u5927\u89c4\u6a21\u5bb9\u5668\u5316\u573a\u666f\u91ccPod\u53d8\u52a8\u4ece\u800c\u5bfc\u81f4\u914d\u7f6e\u7684\u9891\u7e41\u6e32\u67d3\uff0c\u663e\u8457\u51cf\u5c11reload\u6b21\u6570\uff0c\u7279\u522b\u662f\u5728\u5355\u8282\u70b9\u7684Pod\u4e2a\u6570\u8f83\u591a\u548c\u4f7f\u7528clusterlogconfig\u5339\u914d\u5927\u91cf\u7684Pod\u65f6\u3002\u4e00\u822c\u5efa\u8bae\u8bbe\u7f6e\u4e3atrue\u3002 kubeletRootDir string \u975e\u5fc5\u586b /var/lib/kubelet kubelet\u7684root\u8def\u5f84 podLogDirPrefix string \u975e\u5fc5\u586b /var/log/pods kubernetes\u9ed8\u8ba4\u653e\u7f6e\u7684pod\u6807\u51c6\u8f93\u51fa\u8def\u5f84 typePodFields map \u975e\u5fc5\u586b \u5f53logconfig/clusterlogconfig\u91ccselector\u4e3atype: pod\u65f6\uff0c\u81ea\u52a8\u6dfb\u52a0\u7684kubernetes\u76f8\u5173\u5143\u4fe1\u606f\uff0ckey\u5373\u4e3a\u6dfb\u52a0\u7684\u5143\u4fe1\u606fkey\uff0cvalue\u8bf7\u4f7f\u7528${_k8s.XX}\u7684\u65b9\u5f0f\u6307\u5b9a\uff0c\u540c\u65f6\u652f\u6301\u586b\u5199\u56fa\u5b9a\u503c\u7684key:value\u5b57\u6bb5 typeNodeFields map \u975e\u5fc5\u586b \u5f53logconfig/clusterlogconfig\u91ccselector\u4e3atype: node\u65f6\uff0c\u81ea\u52a8\u6dfb\u52a0\u7684kubernetes\u76f8\u5173\u5143\u4fe1\u606f\uff0ckey\u5373\u4e3a\u6dfb\u52a0\u7684\u5143\u4fe1\u606fkey\uff0cvalue\u8bf7\u4f7f\u7528${_k8s.XX}\u7684\u65b9\u5f0f\u6307\u5b9a\uff0c\u540c\u65f6\u652f\u6301\u586b\u5199\u56fa\u5b9a\u503c\u7684key:value\u5b57\u6bb5"},{"location":"reference/global/discovery/#typepodfields","title":"typePodFields\u652f\u6301\u7684\u53d8\u91cf","text":"\"${_k8s.XX}\"\u7684\u65b9\u5f0f\u53ef\u586b\u5199\u4ee5\u4e0b\u53c2\u6570\uff1a
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 ${_k8s.logconfig} string \u975e\u5fc5\u586b \u6dfb\u52a0logConfig name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.name} string \u975e\u5fc5\u586b \u6dfb\u52a0\u6240\u5728\u8282\u70b9node name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.ip} string \u975e\u5fc5\u586b \u6dfb\u52a0\u6240\u5728\u8282\u70b9node ip\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.namespace} string \u975e\u5fc5\u586b \u6dfb\u52a0namespace\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.name} string \u975e\u5fc5\u586b \u6dfb\u52a0pod name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.ip} string \u975e\u5fc5\u586b \u6dfb\u52a0pod ip\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.uid} string \u975e\u5fc5\u586b \u6dfb\u52a0pod uid\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.container.name} string \u975e\u5fc5\u586b \u6dfb\u52a0container name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.container.id} string \u975e\u5fc5\u586b \u6dfb\u52a0container id\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.pod.container.image} string \u975e\u5fc5\u586b \u6dfb\u52a0container image\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.workload.kind} string \u975e\u5fc5\u586b \u6dfb\u52a0pod\u5f52\u5c5e\u7684Deployment/Statefulset/DaemonSet/Job\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.workload.name} string \u975e\u5fc5\u586b \u6dfb\u52a0pod\u5f52\u5c5e\u7684Deployment/Statefulset/DaemonSet/Job\u540d\u79f0\u4f5c\u4e3a\u5143\u4fe1\u606f"},{"location":"reference/global/discovery/#typenodefields","title":"typeNodeFields\u652f\u6301\u7684\u53d8\u91cf","text":"\"${_k8s.XX}\"\u7684\u65b9\u5f0f\u53ef\u586b\u5199\u4ee5\u4e0b\u53c2\u6570\uff1a
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 ${_k8s.clusterlogconfig} string \u975e\u5fc5\u586b \u6dfb\u52a0clusterlogconfig name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.name} string \u975e\u5fc5\u586b \u6dfb\u52a0\u6240\u5728\u8282\u70b9node name\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.addresses.InternalIP} string \u975e\u5fc5\u586b \u6dfb\u52a0node InternalIP \u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.addresses.Hostname} string \u975e\u5fc5\u586b \u6dfb\u52a0node Hostname\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.kernelVersion} string \u975e\u5fc5\u586b \u6dfb\u52a0node kernelVersion\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.osImage} string \u975e\u5fc5\u586b \u6dfb\u52a0node osImage\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.containerRuntimeVersion} string \u975e\u5fc5\u586b \u6dfb\u52a0node containerRuntimeVersion\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.kubeletVersion} string \u975e\u5fc5\u586b \u6dfb\u52a0node kubeletVersion\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.kubeProxyVersion} string \u975e\u5fc5\u586b \u6dfb\u52a0node kubeProxyVersion\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.operatingSystem} string \u975e\u5fc5\u586b \u6dfb\u52a0node operatingSystem\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.nodeInfo.architecture} string \u975e\u5fc5\u586b \u6dfb\u52a0node architecture\u4f5c\u4e3a\u5143\u4fe1\u606f ${_k8s.node.labels.<key>} string \u975e\u5fc5\u586b \u6dfb\u52a0node\u7684\u67d0\u4e2alabel\u4f5c\u4e3a\u5143\u4fe1\u606f\uff0c\u5176\u4e2d\u7684<key>\u8bf7\u66ff\u6362\u6210\u5177\u4f53\u7684label key ${_k8s.node.annotations.<key>} string \u975e\u5fc5\u586b \u6dfb\u52a0node\u7684\u67d0\u4e2aannotation\u4f5c\u4e3a\u5143\u4fe1\u606f\uff0c\u5176\u4e2d\u7684<key>\u8bf7\u66ff\u6362\u6210\u5177\u4f53\u7684annotation key"},{"location":"reference/global/http/","title":"Http","text":"Loggie\u81ea\u8eab\u63d0\u4f9b\u7684Http\u7aef\u53e3\uff0c\u5305\u542b\u76d1\u63a7metrics\uff0c\u5185\u90e8\u8fd0\u7ef4\u7b49\u63a5\u53e3\u3002
Example
http:\nenabled: true\nhost: \"0.0.0.0\"\nport: 9196\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542fhttp host string \u975e\u5fc5\u586b 0.0.0.0 http\u76d1\u542c\u7684host port http \u975e\u5fc5\u586b 9196 http\u76d1\u63a7\u7684\u7aef\u53e3 Tips
\u4e00\u822c\u63a8\u8350\u6253\u5f00http\u7aef\u53e3\uff0c\u4f46\u662f\u5982\u679cKubernetes\u6216\u8005\u5bb9\u5668\u90e8\u7f72\u65f6\uff0c\u4f7f\u7528hostNetwork\u8bf7\u6ce8\u610f\u7aef\u53e3\u51b2\u7a81\uff0c\u4ee5\u53ca\u76d1\u542chost\u662f\u5426\u66b4\u9732\u7ed9\u516c\u7f51\u3001\u662f\u5426\u6709\u5b89\u5168\u9690\u60a3\u3002
"},{"location":"reference/global/monitor/","title":"Monitor","text":"\u76d1\u63a7\u4e8b\u4ef6\u603b\u7ebf\uff0c\u6240\u6709\u7684\u7ec4\u4ef6\u90fd\u53ef\u4ee5\u53d1\u51fa\u81ea\u5df1\u7684metrics\u6307\u6807\u6570\u636e\uff0c\u7531listeners\u6d88\u8d39\u5904\u7406\u3002
\u5177\u4f53\u8bf7\u53c2\u8003\u8fd9\u91cc\u3002
Example
monitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~ reload\u4f1a\u5b9a\u65f6\u68c0\u67e5\u542f\u52a8\u53c2\u6570-config.pipeline\u6307\u5b9a\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u5982\u679c\u68c0\u6d4b\u5230\u6587\u4ef6\u5185\u5bb9\u53d1\u751f\u53d8\u52a8\uff0c\u4f1a\u91cd\u542f\u6709\u53d8\u52a8\u914d\u7f6e\u7684Pipeline\uff0c\u672a\u88ab\u4fee\u6539\u7684Pipeline\u4e0d\u4f1a\u53d7\u5f71\u54cd\u3002
Example
reload:\nenabled: true\nperiod: 10s\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542freload period time.Duration \u975e\u5fc5\u586b 10s reload\u68c0\u6d4b\u914d\u7f6e\u7684\u65f6\u95f4\u95f4\u9694\uff0c\u4e0d\u5efa\u8bae\u8bbe\u7f6e\u8fc7\u77ed\uff0c\u5426\u5219\u53ef\u80fd\u4f1a\u589e\u52a0CPU\u7684\u6d88\u8017"},{"location":"reference/global/var/","title":"\u5b57\u6bb5\u52a8\u6001\u53d8\u91cf","text":"\u5728\u5f88\u591a\u573a\u666f\u4e0b\uff0c\u6211\u4eec\u5f80\u5f80\u9700\u8981\u52a8\u6001\u7684\u83b7\u53d6event\u91cc\u7684\u67d0\u4e2a\u5b57\u6bb5\u3002 \u6bd4\u5982\uff1a
${a.b}\u7684\u65b9\u5f0f\u6765\u53d6\u503c\u3002\u4ee5\u4e0bevent\u4e3a\u4f8b:
{\n\"fields\": {\n\"svc\": \"test\",\n}\n}\nkafka sink topic\u914d\u7f6e\u4e3a\uff1alog-${fields.svc}\uff0c\u5219\u6700\u7ec8\u6e32\u67d3\u751f\u6210\u7684\u4e3alog-test\u3002
Caution
\u4e00\u822c\u53ef\u4ee5\u4f7f\u7528.\u70b9\u53f7\u6765\u8868\u793a\u5d4c\u5957\u7684\u5b57\u6bb5\u3002\u4f46\u662f\uff0c\u5982\u679c\u5b57\u6bb5\u672c\u8eab\u5c31\u5305\u62ec.\u53f7\uff0c\u5219\u9700\u8981\u4f7f\u7528[]\u5305\u56f4\u8d77\u6765\uff0c\u907f\u514d\u8bef\u8ba4\u4e3a\u662f\u4e00\u4e2a\u5d4c\u5957\u7684\u5b57\u6bb5\u3002 \u6bd4\u5982\uff1a
{\n\"fields\": {\n\"a.b\": \"demo\",\n}\n}\n${fields.[a.b]}\u6765\u8868\u793a\u5b57\u6bb5a.b\u3002"},{"location":"reference/monitor/filesource/","title":"filesource listener","text":"\u5b9e\u65f6\u6587\u4ef6\u91c7\u96c6\u7684\u76d1\u63a7\uff0c\u8868\u793a\u5f53\u524d\u65e5\u5fd7\u91c7\u96c6\u7684\u8fdb\u5ea6\u4e0e\u72b6\u6001\uff0c\u5305\u62ec\u6587\u4ef6\u7684\u540d\u79f0\u3001\u91c7\u96c6\u8fdb\u5ea6\u3001QPS\u7b49\u3002
"},{"location":"reference/monitor/filesource/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 10s listener\u6d88\u8d39\u5904\u7406\u6570\u636e\u7684\u65f6\u95f4\u95f4\u9694 fieldsRef string\u6570\u7ec4 \u975e\u5fc5\u586b \u652f\u6301\u4ecesource fields\u914d\u7f6e\u5f53\u4e2d\u83b7\u53d6key value\u52a0\u5165\u5230metrics\u4e2d\u4e3alabel"},{"location":"reference/monitor/filesource/#metrics","title":"Metrics","text":"# HELP file size\n# TYPE loggie_filesource_file_size gauge\nloggie_filesource_file_size{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\"} 2048\n# HELP file offset\n# TYPE loggie_filesource_file_offset gauge\nloggie_filesource_file_offset{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\"} 1024\n# HELP current read line number\n# TYPE loggie_filesource_line_number gauge\nloggie_filesource_line_number{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\"} 20\n# HELP current read line qps\n# TYPE loggie_filesource_line_qps gauge\nloggie_filesource_line_qps{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\"} 48\n\u5bf9\u6587\u4ef6\u91c7\u96c6\u60c5\u51b5\u7684\u5b9a\u65f6\u68c0\u67e5\u5e76\u66b4\u9732\u6307\u6807\uff0c\u5305\u62ec\u6587\u4ef6\u540d\u79f0\u3001ackOffset\u3001\u4fee\u6539\u65f6\u95f4\u3001\u5927\u5c0f\u7b49\u3002
"},{"location":"reference/monitor/filewatcher/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 5m \u5b9a\u65f6\u68c0\u67e5\u95f4\u9694\u65f6\u95f4 checkUnFinishedTimeout time.Duration \u975e\u5fc5\u586b 24h \u68c0\u67e5\u6587\u4ef6\u662f\u5426\u91c7\u96c6\u5b8c\u6bd5\u7684\u8d85\u65f6\u65f6\u95f4\uff0c\u5982\u679c\u68c0\u6d4b\u5230\u6587\u4ef6\u7684\u6700\u8fd1\u4fee\u6539\u65f6\u95f4\u4e3acheckUnFinishedTimeout\u4e4b\u524d\uff0c\u540c\u65f6\u6587\u4ef6\u7684\u5e76\u672a\u91c7\u96c6\u5b8c\u6bd5\uff0c\u5219\u4f1a\u5728metrics\u4e2d\u88ab\u6807\u8bb0\u4e3aunfinished\u72b6\u6001\uff0c\u53ef\u7528\u4e8e\u68c0\u67e5\u662f\u5426\u6709\u957f\u65f6\u95f4\u672a\u88ab\u91c7\u96c6\u7684\u65e5\u5fd7\u6587\u4ef6"},{"location":"reference/monitor/filewatcher/#metrics","title":"Metrics","text":""},{"location":"reference/monitor/filewatcher/#_2","title":"\u5168\u5c40\u7ea7\u522b","text":""},{"location":"reference/monitor/filewatcher/#total_file_count","title":"total_file_count","text":"# HELP file count total\n# TYPE loggie_filewatcher_total_file_count gauge\nloggie_filewatcher_total_file_count{} 20\n# HELP inactive file count\n# TYPE loggie_filewatcher_inactive_file_count gauge\nloggie_filewatcher_inactive_file_count{} 20\n\u6587\u4ef6\u7ea7\u522b\u5305\u62ec\u4e86\u4ee5\u4e0bprometheus labels:
checkUnFinishedTimeout\u65f6\u95f4ignore_older\u65f6\u95f4\u7531\u4e8e\u5b9a\u65f6\u626b\u63cf\u7684\u65f6\u95f4\u95f4\u9694period\u9ed8\u8ba4\u4e3a5min\uff0c\u4ee5\u4e0b\u6307\u6807\u53ef\u80fd\u5b58\u5728\u4e00\u5b9a\u7a0b\u5ea6\u7684\u5ef6\u8fdf\u3002
# HELP file size\n# TYPE loggie_filewatcher_file_size gauge\nloggie_filewatcher_file_size{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\", status=\"pending\"} 2048\n# HELP file ack offset\n# TYPE loggie_filewatcher_file_ack_offset gauge\nloggie_filewatcher_file_ack_offset{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\", status=\"pending\"} 1024\n# HELP file last modify timestamp\n# TYPE loggie_filewatcher_file_last_modify gauge\nloggie_filewatcher_file_last_modify{pipeline=\"xxx\", source=\"access\", filename=\"/var/log/a.log\", status=\"pending\"} 2343214422\n\u5c55\u793aLoggie\u672c\u8eab\u7684\u4e00\u4e9b\u4fe1\u606f\u3002
"},{"location":"reference/monitor/info/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 10s \u66b4\u9732\u6307\u6807\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/monitor/info/#metrics","title":"Metrics","text":""},{"location":"reference/monitor/info/#loggie_info_stat","title":"loggie_info_stat","text":"# HELP loggie_info_stat Loggie info\n# TYPE loggie_info_stat gauge\nloggie_info_stat{version=v1.4} 1\n\u7528\u4e8e\u65e5\u5fd7\u62a5\u8b66\u7684\u53d1\u9001\u3002 \u4f7f\u7528\u793a\u4f8b\u8bf7\u53c2\u8003\u65e5\u5fd7\u62a5\u8b66\u3002
Example
logAlert:\naddr: [ \"http://127.0.0.1:8080/loggie\" ]\nbufferSize: 100\nbatchTimeout: 10s\nbatchSize: 1\nlineLimit: 10\ntemplate: |\n*****\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addr string\u6570\u7ec4 \u5fc5\u586b \u53d1\u9001alert\u7684http\u5730\u5740 bufferSize int \u975e\u5fc5\u586b 100 \u65e5\u5fd7\u62a5\u8b66\u53d1\u9001\u7684buffer\u5927\u5c0f\uff0c\u5355\u4f4d\u4e3a\u62a5\u8b66\u4e8b\u4ef6\u4e2a\u6570 batchTimeout time.Duration \u975e\u5fc5\u586b 10s \u6bcf\u4e2a\u62a5\u8b66\u53d1\u9001batch\u7684\u6700\u5927\u53d1\u9001\u65f6\u95f4 batchSize int \u975e\u5fc5\u586b 10 \u6bcf\u4e2a\u62a5\u8b66\u53d1\u9001batch\u7684\u6700\u5927\u5305\u542b\u62a5\u8b66\u8bf7\u6c42\u4e2a\u6570 template string \u975e\u5fc5\u586b \u6e32\u67d3\u53d1\u9001\u7684alert\u7ed3\u6784\u4f53\u7684go template\u6a21\u677f timeout time.Duration \u975e\u5fc5\u586b 30s \u53d1\u9001alert\u7684http timeout headers map \u975e\u5fc5\u586b \u53d1\u9001alert\u7684http header method string \u975e\u5fc5\u586b POST \u53d1\u9001alert\u7684http method, \u5982\u679c\u4e0d\u586bput(\u4e0d\u533a\u5206\u5927\u5c0f\u5199)\uff0c\u90fd\u8ba4\u4e3a\u662fPOST lineLimit int \u975e\u5fc5\u586b 10 \u591a\u884c\u65e5\u5fd7\u91c7\u96c6\u60c5\u51b5\u4e0b\uff0c\u6bcf\u4e2aalert\u4e2d\u5305\u542b\u7684\u6700\u5927\u65e5\u5fd7\u884c\u6570"},{"location":"reference/monitor/overview/","title":"Monitor","text":"\u76d1\u63a7\u4e8b\u4ef6\u603b\u7ebf\uff0c\u6240\u6709\u7684\u7ec4\u4ef6\u90fd\u53ef\u4ee5\u53d1\u51fa\u81ea\u5df1\u7684metrics\u6307\u6807\u6570\u636e\uff0c\u7531listeners\u6d88\u8d39\u5904\u7406\u3002
Example
monitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~ Loggie\u652f\u6301\u5c06metrics\u6307\u6807\u8f93\u51fa\u5230\u65e5\u5fd7\u4e2d\uff0c\u53ef\u4ee5\u901a\u8fc7logger\u914d\u7f6e\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 logger.enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f logger.period time.Duration \u975e\u5fc5\u586b 10s \u6307\u6807\u6253\u5370\u7684\u65f6\u95f4\u95f4\u9694\uff0c\u6570\u636e\u91cf\u8f83\u5927\u65f6\u5efa\u8bae\u5c06\u95f4\u9694\u5ef6\u957f\uff0c\u598230s\u30015m logger.pretty bool \u975e\u5fc5\u586b false \u6253\u5370\u7684\u6307\u6807json\u662f\u5426\u9700\u8981\u53cb\u597d\u5c55\u793a logger.additionLogEnabled bool \u975e\u5fc5\u586b false \u662f\u5426\u9700\u8981\u5c06\u6253\u5370\u7684\u6307\u6807\u5355\u72ec\u8f93\u51fa\u5230\u53e6\u5916\u7684\u65e5\u5fd7\u6587\u4ef6\u4e2d\uff0c\u5728\u6570\u636e\u91cf\u6bd4\u8f83\u591a\u7684\u60c5\u51b5\u4e0b\uff0c\u5982\u679c\u6211\u4eec\u914d\u7f6e\u7684\u6253\u5370\u65f6\u95f4\u95f4\u9694\u8f83\u77ed\uff0c\u53ef\u4ee5\u6253\u5f00\u8be5\u5f00\u5173\uff0c\u907f\u514d\u592a\u591a\u7684metrics\u65e5\u5fd7\u5e72\u6270 logger.additionLogConfig \u975e\u5fc5\u586b \u989d\u5916\u8f93\u51fa\u7684\u65e5\u5fd7\u914d\u7f6e\u53c2\u6570 logger.additionLogConfig.directory bool \u975e\u5fc5\u586b /data/loggie/log \u989d\u5916\u8f93\u51fa\u7684\u65e5\u5fd7\u76ee\u5f55 logger.additionLogConfig.maxBackups int \u975e\u5fc5\u586b metrics.log \u65e5\u5fd7\u8f6e\u8f6c\u6700\u591a\u4fdd\u7559\u7684\u6587\u4ef6\u4e2a\u6570\uff0c\u9ed8\u8ba4\u4e3a3 logger.additionLogConfig.maxSize int \u975e\u5fc5\u586b 1024 \u65e5\u5fd7\u8f6e\u8f6c\u7684\u65f6\u5019\uff0c\u6700\u5927\u7684\u6587\u4ef6\u5927\u5c0f\uff0c\u5355\u4f4d\u4e3aMB logger.additionLogConfig.maxAge int \u975e\u5fc5\u586b 14 \u65e5\u5fd7\u8f6e\u8f6c\u6700\u5927\u4fdd\u7559\u7684\u5929\u6570 logger.additionLogConfig.timeFormat string \u975e\u5fc5\u586b 2006-01-02 15:04:05 \u6bcf\u884c\u65e5\u5fd7\u8f93\u51fa\u7684\u65f6\u95f4\u683c\u5f0f"},{"location":"reference/monitor/overview/#listeners","title":"listeners","text":"\u8868\u793a\u5177\u4f53\u542f\u52a8\u7684listeners\u3002 \u914d\u7f6e\u4e0d\u586b\u5199\u5373\u4e3a\u5173\u95ed\uff0c\u4e0d\u542f\u52a8\u8be5Listener\uff0c\u76f8\u5173\u7684\u7684\u6307\u6807\u4e5f\u4e0d\u4f1a\u88ab\u5904\u7406\u548c\u66b4\u9732\u3002
"},{"location":"reference/monitor/queue/","title":"queue listener","text":""},{"location":"reference/monitor/queue/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 10s listener\u6d88\u8d39\u5904\u7406\u6570\u636e\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/monitor/queue/#metrics","title":"Metrics","text":"# HELP queue capacity\n# TYPE loggie_queue_capacity gauge\nloggie_queue_capacity{pipeline=\"xxx\", type=\"channel\"} 2048\n# HELP queue size\n# TYPE loggie_queue_size gauge\nloggie_queue_size{pipeline=\"xxx\", type=\"channel\"} 2048\n# HELP how full is queue\n# TYPE loggie_queue_fill_percentage gauge\nloggie_queue_fill_percentage{pipeline=\"xxx\", type=\"channel\"} 50\nreload\u7684\u6307\u6807\uff0c\u603b\u7684reload\u6b21\u6570\u3002
"},{"location":"reference/monitor/reload/#metrics","title":"Metrics","text":""},{"location":"reference/monitor/reload/#total","title":"total","text":"# HELP Loggie reload total count\n# TYPE loggie_reload_total gauge\nloggie_reload_total{} 10\nsink\u53d1\u9001\u7aef\u7684\u76d1\u63a7\u6307\u6807\u5904\u7406\uff0c\u5305\u62ec\u53d1\u9001\u6210\u529f\u7684event\u6570\u91cf\u3001\u53d1\u9001\u5931\u8d25\u7684event\u6570\u91cf\u3001event qps\u7b49
"},{"location":"reference/monitor/sink/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 10s listener\u6d88\u8d39\u5904\u7406\u6570\u636e\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/monitor/sink/#metrics","title":"Metrics","text":"# HELP send event success count\n# TYPE loggie_sink_success_event gauge\nloggie_sink_success_event{pipeline=\"xxx\", source=\"access\"} 2048\nperiod\u65f6\u95f4\u6bb5\u5185\uff0c\u53d1\u9001\u6210\u529f\u7684event\u4e2a\u6570# HELP send event failed count\n# TYPE loggie_sink_failed_event gauge\nloggie_sink_failed_event{pipeline=\"xxx\", source=\"access\"} 2048\nperiod\u65f6\u95f4\u6bb5\u5185\uff0c\u53d1\u9001\u5931\u8d25\u7684event\u4e2a\u6570# HELP send success event failed count\n# TYPE loggie_sink_event_qps gauge\nloggie_sink_event_qps{pipeline=\"xxx\", source=\"access\"} 2048\nperiod\u65f6\u95f4\u6bb5\u5185\uff0c\u53d1\u9001\u7684event QPSLoggie\u672c\u8eab\u7684CPU\u548cMemory\u6307\u6807
"},{"location":"reference/monitor/sys/#_1","title":"\u914d\u7f6e","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 period time.Duration \u975e\u5fc5\u586b 10s listener\u6d88\u8d39\u5904\u7406\u6570\u636e\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/monitor/sys/#metrics","title":"Metrics","text":""},{"location":"reference/monitor/sys/#cpu_percent","title":"cpu_percent","text":"# HELP loggie_sys_cpu_percent Loggie cpu percent\n# TYPE loggie_sys_cpu_percent gauge\nloggie_sys_cpu_percent 0.37\n# HELP loggie_sys_mem_rss Loggie memory rss bytes\n# TYPE loggie_sys_mem_rss gauge\nloggie_sys_mem_rss 2.5853952e+07\n\u7528\u4e8e\u4eceevent\u4e2d\u7684\u67d0\u4e9b\u5b57\u6bb5\uff08\u6bd4\u5982\u65e5\u5fd7\u6587\u4ef6\u7684\u8def\u5f84\u4e2d\uff09\uff0c\u83b7\u53d6\u5230\uff1a
pod.uidnamespace\u4e0epod.namecontainer.id\u4ee5\u4e0a3\u79cd\u4efb\u610f\u5176\u4e00\u7684\u7d22\u5f15\u4fe1\u606f\uff0c\u6b64\u65f6Loggie\u53ef\u6839\u636e\u8be5\u7d22\u5f15\u67e5\u8be2\u5230\u5177\u4f53\u7684Pod\uff0c\u5e76\u6dfb\u52a0\u989d\u5916\u7684kubernetes${node.name}\u3001${namespace}\u3001${pod.uid}\u3001${pod.name}\u7b49\u5143\u4fe1\u606f\u4f5c\u52a0\u5165\u5230event\u4e2d\uff0c\u7528\u4e8e\u540e\u7eed\u7684\u5206\u6790\u5904\u7406\u3002 \u5c5e\u4e8esource interceptor\u3002
Example
interceptors:\n- type: addK8sMeta\npattern: \"/var/log/${pod.uid}/${pod.name}/\"\naddFields:\nnodename: \"${node.name}\"\nnamespace: \"${namespace}\"\npodname: \"${pod.name}\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 pattern string \u5fc5\u586b \u63d0\u53d6\u5b57\u6bb5\u7684\u5339\u914d\u6a21\u578b \u5fc5\u987b\u5305\u542b\u6709\uff1a
pod.uidnamespace\u4e0epod.namecontainer.id\u5176\u4e2d\u4e4b\u4e00\u3002
\u6bd4\u5982\uff1a/var/log/${pod.uid}/${pod.name}/
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 patternFields string \u975e\u5fc5\u586b \u9ed8\u8ba4\u4f1a\u4eceevent\u4e2d\u83b7\u53d6\u7cfb\u7edf\u5b57\u6bb5\u91cc\u7684filename\uff0c\u6b64\u65f6\u9700\u8981\u4f7f\u7528file source \u4eceevent\u4e2d\u7528\u4e8e\u63d0\u53d6\u7684pattern\u7684\u5b57\u6bb5"},{"location":"reference/pipelines/interceptor/addk8smeta/#fieldsname","title":"fieldsName","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fieldsName string \u975e\u5fc5\u586b kubernetes \u6dfb\u52a0\u5143\u4fe1\u606f\u7684\u5b57\u6bb5"},{"location":"reference/pipelines/interceptor/addk8smeta/#addfields","title":"addFields","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addFields map \u975e\u5fc5\u586b \u9700\u8981\u6dfb\u52a0\u7684\u5143\u4fe1\u606f \u76ee\u524d\u652f\u6301\u6dfb\u52a0\u7684\u5143\u4fe1\u606f\u5b57\u6bb5\u6709\uff1a
${cluster}\uff1a\u96c6\u7fa4\u4fe1\u606f\uff0c\u4e3a\u7cfb\u7edf\u914d\u7f6e\u4e2ddiscovery.kubernetes.cluster\u5b57\u6bb5\u3002${node.name}${namespace}${workload.kind}\uff1aDeployment/StatefulSet/DaemonSet/Job\u7b49${workload.name}\uff1a\u5de5\u4f5c\u8d1f\u8f7d\u7684\u540d\u79f0${pod.uid}${pod.name}\u7528\u4e8e\u65e5\u5fd7\u9650\u6d41\u3002 \u5c5e\u4e8esource interceptor\u3002
Example
interceptors:\n- type: rateLimit\nqps: 4000\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 qps int \u975e\u5fc5\u586b 2048 \u9650\u6d41qps"},{"location":"reference/pipelines/interceptor/logalert/","title":"logAlert","text":"\u7528\u4e8e\u65e5\u5fd7\u62a5\u8b66\u68c0\u6d4b\u3002 \u5c5e\u4e8esource interceptor\u3002 \u4f7f\u7528\u793a\u4f8b\u8bf7\u53c2\u8003\u65e5\u5fd7\u62a5\u8b66\u3002
Example
interceptors:\n- type: logAlert\nmatcher:\ncontains: [\"error\", \"err\"]\nregexp: ['.*example.*']\nignore: ['.*INFO.*']\nsendOnlyMatched: true\nadditions:\nmodule: \"loggie\"\nalertname: \"alert-test\"\ncluster: \"local-cluster\"\nnamespace: \"default\"\nadvanced:\nenabled: true\nmode: [ \"noData\",\"regexp\" ]\nduration: 6h\nmatchType: \"any\"\nrules:\n- regexp: '(?<date>.*?) (?<time>[\\S|\\\\.]+) (<status>[\\S|\\\\.]+) (?<u>.*?) --- (?<thread>\\[*?\\]) (?<pkg>.*) : (?<message>(.|\\n|\\t)*)'\nmatchType: \"any\"\ngroups:\n- key: status\noperator: \"eq\"\nvalue: WARN\n- key: thread\noperator: \"eq\"\nvalue: 200\n- regexp: '(?<date>.*?) (?<time>[\\S|\\\\.]+) (?<status>[\\S|\\\\.]+) (?<u>.*?) --- (?<thread>\\[.*?\\]) (?<pkg>.*) : (?<message>(.|\\n|\\t)*)'\nmatchType: \"any\"\ngroups:\n- key: status\noperator: \"eq\"\nvalue: ERROR\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 matcher.contains string\u6570\u7ec4 \u975e\u5fc5\u586b \u65e5\u5fd7\u6570\u636e\u5305\u542b\u5b57\u7b26\u4e32\u68c0\u6d4b matcher.regexp string\u6570\u7ec4 \u975e\u5fc5\u586b \u65e5\u5fd7\u6570\u636e\u6b63\u5219\u68c0\u6d4b matcher.target string \u975e\u5fc5\u586b body \u6839\u636e\u65e5\u5fd7\u6570\u636e\u7684\u8be5\u5b57\u6bb5\u8fdb\u884c\u68c0\u6d4b\uff0c\u5982\u679c\u8fdb\u884c\u65e5\u5fd7\u5207\u5206\u6216\u8005drop body\u5b57\u6bb5\uff0c\u8bf7\u586b\u5199\u6240\u9700\u5b57\u6bb5"},{"location":"reference/pipelines/interceptor/logalert/#ignore","title":"ignore","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 ignore string\u6570\u7ec4 \u975e\u5fc5\u586b \u6b63\u5219\u8868\u8fbe\u5f0f\uff0c\u82e5\u5339\u914d\uff0c\u5219\u5ffd\u7565\u8fd9\u6761\u65e5\u5fd7\uff0c\u5411\u4e0b\u4f20\u9012\uff0c\u53ef\u7528\u4e8e\u544a\u8b66\u65f6\u6392\u9664\u67d0\u4e9b\u65e5\u5fd7"},{"location":"reference/pipelines/interceptor/logalert/#additions","title":"additions","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 additions map \u975e\u5fc5\u586b \u53d1\u9001alert\u65f6\uff0c\u989d\u5916\u6dfb\u52a0\u7684\u5b57\u6bb5\uff0c\u4f1a\u653e\u5728_additions\u5b57\u6bb5\u4e2d\uff0c\u53ef\u7528\u4f5c\u6e32\u67d3\u3002"},{"location":"reference/pipelines/interceptor/logalert/#sendonlymatched","title":"sendOnlyMatched","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sendOnlyMatched bool \u975e\u5fc5\u586b false \u662f\u5426\u4ec5\u5c06\u5339\u914d\u6210\u529f\u7684\u6570\u636e\u53d1\u9001\u81f3sink"},{"location":"reference/pipelines/interceptor/logalert/#advanced","title":"advanced","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u9ad8\u7ea7\u5339\u914d\u6a21\u5f0f mode string\u5217\u8868 \u975e\u5fc5\u586b \u5339\u914d\u6a21\u5f0f \u652f\u6301regexp\u548cnoData\u4e24\u79cd\uff0c\u53ef\u540c\u65f6\u751f\u6548\u3002 duration time.Duration \u975e\u5fc5\u586b noData\u6a21\u5f0f\u5fc5\u586b\uff0c\u5728\u4e00\u5b9a\u65f6\u95f4\u5185\uff0c\u6ca1\u6709\u65e5\u5fd7\u4f1a\u53d1\u51fa\u544a\u8b66\u3002 matchType string \u975e\u5fc5\u586b regexp\u6a21\u5f0f\u5fc5\u586b\uff0c\u53ef\u9009any\u6216\u8005all\uff0c\u8868\u793a\u5339\u914d\u4efb\u610f\u6216\u8005\u5168\u90e8\u89c4\u5219rule rules Rule\u5217\u8868 \u975e\u5fc5\u586b regexp\u6a21\u5f0f\u5fc5\u586b\uff0c\u5339\u914d\u89c4\u5219\u5217\u8868"},{"location":"reference/pipelines/interceptor/logalert/#advancedrule","title":"advanced.rule","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 regexp string \u5fc5\u586b \u6b63\u5219\u5206\u7ec4\u8868\u8fbe\u5f0f matchType string \u5fc5\u586b \u53ef\u9009any\u6216\u8005all\uff0c\u8868\u793a\u5339\u914d\u4efb\u610f\u6216\u8005\u5168\u90e8\u5339\u914d\u7ec4group groups group\u5217\u8868 \u5fc5\u586b \u5339\u914d\u7ec4\u5217\u8868"},{"location":"reference/pipelines/interceptor/logalert/#advancedrulegroup","title":"advanced.rule.group","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 key string \u5fc5\u586b \u5206\u7ec4\u5339\u914d\u4e4b\u540e\u7684\u952e\u503c operator string \u5fc5\u586b \u64cd\u4f5c\u7b26\uff0c\u76ee\u524d\u652f\u6301eq\uff0cgt\uff0clt value string \u5fc5\u586b \u76ee\u6807\u503c"},{"location":"reference/pipelines/interceptor/maxbytes/","title":"maxbytes","text":"\u5bf9\u539f\u59cb\u5355\u884c\u65e5\u5fd7\u7684\u5927\u5c0f\u8fdb\u884c\u9650\u5236\uff0c\u907f\u514d\u5355\u884c\u65e5\u5fd7\u6570\u636e\u91cf\u592a\u5927\u5f71\u54cdLoggie\u5185\u5b58\u548c\u7a33\u5b9a\u6027\u3002 \u7cfb\u7edf\u5185\u7f6e\uff0c\u9ed8\u8ba4\u52a0\u8f7d\uff0c\u5c5e\u4e8esource interceptor\u3002
Example
interceptors:\n- type: maxbytes\nmaxBytes: 102400\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxBytes int \u975e\u5fc5\u586b 131072(128KB) \u5355\u884c\u6700\u5927\u5b57\u8282\u4e2a\u6570\uff0c\u8d85\u51fa\u7684\u90e8\u5206\u5c06\u4f1a\u88ab\u4e22\u5f03"},{"location":"reference/pipelines/interceptor/metrics/","title":"metric","text":"\u5bf9\u6570\u636e\u4f20\u8f93\u8fdb\u884c\u91c7\u6837metics\uff0c\u63d0\u4f9b\u7ed9monitor eventbus\u4e2dlistener\u4f7f\u7528\u3002 \u7cfb\u7edf\u5185\u7f6e\uff0c\u9ed8\u8ba4\u52a0\u8f7d\uff0c\u5c5e\u4e8esink interceptor\u3002
Example
interceptors:\n- type: metric\n\uff08\u8be5interceptor\u540e\u7eed\u4e0d\u518d\u7ef4\u62a4\uff0c\u5efa\u8bae\u4f7f\u7528transformer\u66ff\u6362\uff09
\u7528\u4e8e\u65e5\u5fd7\u5207\u5206\u5904\u7406\u3002 \u5c5e\u4e8esource interceptor\u3002\u53ef\u6307\u5b9a\u53ea\u88ab\u67d0\u4e9bsource\u4f7f\u7528\u3002
"},{"location":"reference/pipelines/interceptor/normalize/#processors","title":"processors","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 processors \u6570\u7ec4 \u5fc5\u586b \u65e0 \u6240\u6709\u7684\u5904\u7406processor\u5217\u8868 \u914d\u7f6e\u7684processor\u5c06\u6309\u7167\u987a\u5e8f\u4f9d\u6b21\u6267\u884c\u3002
Tips
Loggie\u652f\u6301\u4f7f\u7528a.b\u7684\u5f62\u5f0f\u5f15\u7528\u5d4c\u5957\u7684\u5b57\u6bb5\u3002 \u6bd4\u5982\u6570\u636e\u4e3a:
{\n\"fields\": {\n\"hello\": \"world\"\n}\n}\nfields.hello\u6307\u5b9a\u5d4c\u5957\u5728fields\u91cc\u7684hello: world\u3002"},{"location":"reference/pipelines/interceptor/normalize/#addmeta","title":"addMeta","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0cLoggie\u4e0d\u4f1a\u6dfb\u52a0\u4efb\u4f55\u7684\u7cfb\u7edf\u5185\u90e8\u4fe1\u606f\u5230\u539f\u59cb\u6570\u636e\u4e2d\u3002 \u53ef\u901a\u8fc7addMeta\u6dfb\u52a0\u7cfb\u7edf\u5185\u7f6e\u5b57\u6bb5\u53d1\u9001\u7ed9\u4e0b\u6e38\u3002
Note
\u8bf7\u6ce8\u610f\uff0c\u5728pipeline\u4e2d\u914d\u7f6eaddMeta\uff0c\u53ea\u4f1a\u5f71\u54cd\u8be5pipeline\u53d1\u9001\u7684\u6240\u6709\u6570\u636e\uff0c\u5982\u679c\u9700\u8981\u5168\u5c40\u751f\u6548\uff0c\u8bf7\u5728defaults\u4e2d\u914d\u7f6enormalize.addMeta\u3002
loggie:\n defaults:\n interceptors:\n - type: normalize\n name: global\n processors:\n - addMeta: ~\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 target string \u975e\u5fc5\u586b meta \u7cfb\u7edf\u5185\u7f6e\u5b57\u6bb5\u6dfb\u52a0\u5230event\u4e2d\u7684\u5b57\u6bb5\u540d"},{"location":"reference/pipelines/interceptor/normalize/#regex","title":"regex","text":"\u5c06\u6307\u5b9a\u5b57\u6bb5\u8fdb\u884c\u6b63\u5219\u63d0\u53d6\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 regex.pattern string \u5fc5\u586b \u65e0 \u6b63\u5219\u89e3\u6790\u89c4\u5219 regex.target string \u975e\u5fc5\u586b body \u6b63\u5219\u89e3\u6790\u7684\u76ee\u6807\u5b57\u6bb5 regex.ignoreError bool \u975e\u5fc5\u586b false \u662f\u5426\u5ffd\u7565\u9519\u8bef Example
interceptors:\n- type: normalize\nprocessors:\n- regex:\npattern: '(?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)'\n\u4f7f\u7528\u4ee5\u4e0a\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\uff0c\u53ef\u4ee5\u5c06\u4ee5\u4e0b\u793a\u4f8b\u7684\u65e5\u5fd7\uff1a
10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] \"GET / HTTP/1.1\" 404 683\n\"ip\": \"10.244.0.1\",\n\"id\": \"-\",\n\"u\": \"-\",\n\"time\": \"[13/Dec/2021:12:40:48 +0000]\",\n\"url\": \"\\\"GET / HTTP/1.1\\\"\",\n\"status\": \"404\",\n\"size\": \"683\"\n\u5177\u4f53\u914d\u7f6e\u7684\u65f6\u5019\uff0c\u5efa\u8bae\u5148\u4f7f\u7528\u4e00\u4e9b\u6b63\u5219\u8c03\u8bd5\u5de5\u5177 (https://regex101.com/) \u9a8c\u8bc1\u662f\u5426\u53ef\u4ee5\u5339\u914d\u3002
"},{"location":"reference/pipelines/interceptor/normalize/#jsondecode","title":"jsonDecode","text":"\u5c06\u6307\u5b9a\u5b57\u6bb5json\u89e3\u6790\u63d0\u53d6\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 jsonDecode.target string \u975e\u5fc5\u586b body json decode\u7684\u76ee\u6807\u5b57\u6bb5 jsonDecode.ignoreError bool \u975e\u5fc5\u586b false \u662f\u5426\u5ffd\u7565\u9519\u8bef Example
interceptors:\n- type: normalize\nprocessors:\n- jsonDecode: ~\n\u5c06\u6307\u5b9a\u5b57\u6bb5\u901a\u8fc7\u5206\u9694\u7b26\u8fdb\u884c\u63d0\u53d6\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 split.target string \u975e\u5fc5\u586b body split\u7684\u76ee\u6807\u5b57\u6bb5 split.separator string \u5fc5\u586b \u65e0 \u5206\u9694\u7b26 split.max int \u975e\u5fc5\u586b -1 \u901a\u8fc7\u5206\u5272\u7b26\u5206\u5272\u540e\u5f97\u5230\u7684\u6700\u591a\u7684\u5b57\u6bb5\u6570 split.keys string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u5206\u5272\u540e\u5b57\u6bb5\u5bf9\u5e94\u7684key split.ignoreError bool \u975e\u5fc5\u586b false \u662f\u5426\u5ffd\u7565\u9519\u8bef Example
baseinterceptors:\n- type: normalize\nprocessors:\n- split:\nseparator: '|'\nkeys: [\"time\", \"order\", \"service\", \"price\"]\n2021-08-08|U12345|storeCenter|13.14\n\"time\": \"2021-08-08\"\n\"order\": \"U12345\"\n\"service\": \"storeCenter\"\n\"price\": 13.14\ninterceptors:\n- type: normalize\nprocessors:\n- split:\nseparator: ' '\nmax: 2\nkeys: [\"time\", \"content\"]\nmax\u53c2\u6570\uff0c\u53ef\u4ee5\u63a7\u5236\u6700\u591a\u5206\u5272\u7684\u5b57\u6bb5\u3002 \u6bd4\u5982\u4ee5\u4e0b\u65e5\u5fd7: 2021-08-08 U12345 storeCenter 13.14\n\"time\": \"2021-08-08\"\n\"content\": \"U12345 storeCenter 13.14\"\n\u4e22\u5f03\u6307\u5b9a\u5b57\u6bb5\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 drop.targets string\u6570\u7ec4 \u5fc5\u586b \u65e0 drop\u7684\u5b57\u6bb5 Example
interceptors:\n- type: normalize\nprocessors:\n- drop:\ntargets: [\"id\", \"body\"]\n\u91cd\u547d\u540d\u6307\u5b9a\u5b57\u6bb5\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 rename.convert \u6570\u7ec4 \u5fc5\u586b \u65e0 rename.convert[n].from string \u5fc5\u586b \u65e0 rename\u7684\u76ee\u6807 rename.convert[n].to string \u5fc5\u586b \u65e0 rename\u540e\u7684\u540d\u79f0 Example
interceptors:\n- type: normalize\nprocessors:\n- rename:\nconvert:\n- from: \"hello\"\nto: \"world\"\n\u65b0\u589e\u5b57\u6bb5\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 add.fields map \u5fc5\u586b \u65e0 \u65b0\u589e\u7684key:value\u503c Example
interceptors:\n- type: normalize\nprocessors:\n- add:\nfields:\nhello: world\n\u5b57\u6bb5\u7c7b\u578b\u8f6c\u6362\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 convert.convert \u6570\u7ec4 \u5fc5\u586b \u65e0 convert.convert[n].from string \u5fc5\u586b \u65e0 \u9700\u8981\u8f6c\u6362\u7684\u5b57\u6bb5\u540d convert.convert[n].to string \u5fc5\u586b \u65e0 \u8f6c\u6362\u540e\u7684\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1a\"bool\", \"integer\", \"float\" Example
interceptors:\n- type: normalize\nprocessors:\n- convert:\nconvert:\n- from: count\nto: float\n\u5b57\u6bb5\u590d\u5236\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 copy.convert \u6570\u7ec4 \u5fc5\u586b \u65e0 copy.convert[n].from string \u5fc5\u586b \u65e0 \u9700\u8981\u590d\u5236\u7684\u5b57\u6bb5\u540d copy.convert[n].to string \u5fc5\u586b \u65e0 \u590d\u5236\u540e\u7684\u5b57\u6bb5\u540d Example
interceptors:\n- type: normalize\nprocessors:\n- copy:\nconvert:\n- from: hello\nto: world\n\u5c06\u5b57\u6bb5\u4e2d\u7684\u6240\u6709key:value\u653e\u5230event\u6700\u5916\u5c42\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 underRoot.keys string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u9700\u8981underRoot\u7684\u5b57\u6bb5\u540d Example
interceptors:\n- type: normalize\nprocessors:\n- underRoot:\nkeys: [\"fields\"]\n\u8f6c\u6362\u65f6\u95f4\u683c\u5f0f\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timestamp.convert \u6570\u7ec4 \u5fc5\u586b \u65e0 timestamp.convert[n].from string \u5fc5\u586b \u65e0 \u6307\u5b9a\u8f6c\u6362\u65f6\u95f4\u683c\u5f0f\u7684\u5b57\u6bb5 timestamp.convert[n].fromLayout string \u5fc5\u586b \u65e0 \u6307\u5b9a\u5b57\u6bb5\u7684\u65f6\u95f4\u683c\u5f0f(golang\u5f62\u5f0f) timestamp.convert[n].toLayout string \u5fc5\u586b \u65e0 \u8f6c\u6362\u540e\u7684\u65f6\u95f4\u683c\u5f0f(golang\u5f62\u5f0f)\uff0c\u53e6\u5916\u53ef\u4e3aunix\u548cunix_ms timestamp.convert[n].toType string \u975e\u5fc5\u586b \u65e0 \u8f6c\u6362\u540e\u7684\u65f6\u95f4\u5b57\u6bb5\u7c7b\u578b timestamp.convert[n].local bool \u975e\u5fc5\u586b false \u662f\u5426\u5c06\u89e3\u6790\u7684\u65f6\u95f4\u8f6c\u6210\u5f53\u524d\u65f6\u533a Example
interceptors:\n- type: normalize\nprocessors:\n- timestamp:\nconvert:\n- from: logtime\nfromLayout: \"2006-01-02T15:04:05Z07:00\"\ntoLayout: \"unix\"\n\u4ee5\u4e0a\u7684layout\u53c2\u6570\u9700\u8981\u586b\u5199golang\u5f62\u5f0f\uff0c\u53ef\u53c2\u8003\uff1a
const (\n Layout = \"01/02 03:04:05PM '06 -0700\" // The reference time, in numerical order.\n ANSIC = \"Mon Jan _2 15:04:05 2006\"\n UnixDate = \"Mon Jan _2 15:04:05 MST 2006\"\n RubyDate = \"Mon Jan 02 15:04:05 -0700 2006\"\n RFC822 = \"02 Jan 06 15:04 MST\"\n RFC822Z = \"02 Jan 06 15:04 -0700\" // RFC822 with numeric zone\n RFC850 = \"Monday, 02-Jan-06 15:04:05 MST\"\n RFC1123 = \"Mon, 02 Jan 2006 15:04:05 MST\"\n RFC1123Z = \"Mon, 02 Jan 2006 15:04:05 -0700\" // RFC1123 with numeric zone\n RFC3339 = \"2006-01-02T15:04:05Z07:00\"\n RFC3339Nano = \"2006-01-02T15:04:05.999999999Z07:00\"\n Kitchen = \"3:04PM\"\n // Handy time stamps.\n Stamp = \"Jan _2 15:04:05\"\n StampMilli = \"Jan _2 15:04:05.000\"\n StampMicro = \"Jan _2 15:04:05.000000\"\n StampNano = \"Jan _2 15:04:05.000000000\"\n)\n\u5b57\u6bb5\u5185\u5bb9\u91cd\u65b0\u683c\u5f0f\u5316\u3002\u53ef\u6839\u636e\u5176\u4ed6\u5b57\u6bb5\u5185\u5bb9\u8fdb\u884c\u7ec4\u5408\u548c\u683c\u5f0f\u5316\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fmt.fields map \u5fc5\u586b \u65e0 key\u8868\u793a\u9700\u8981\u683c\u5f0f\u5316\u7684\u5b57\u6bb5\u540d\u79f0\uff0cvalue\u4e3a\u9700\u8981\u683c\u5f0f\u5316\u7684\u5185\u5bb9\u3002\u53ef\u4f7f\u7528${}\u7684\u65b9\u5f0f\u8868\u793a\u53d6\u503c\u67d0\u4e2a\u5b57\u6bb5 Example
interceptors:\n- type: normalize\nprocessors:\n- fmt:\nfields:\nd: new-${a.b}-${c}\ninterceptors\u5b57\u6bb5\u4e3a\u6570\u7ec4\uff0c\u4e00\u4e2aPipeline\u4e2d\u53ef\u586b\u5199\u591a\u4e2ainterceptor\u7ec4\u4ef6\u914d\u7f6e\u3002
\u76ee\u524d\uff0cinterceptor\u5206\u4e3a\u4e24\u79cd\u7c7b\u578b\uff1a
source -> source interceptor -> queue\u3002queue -> sink interceptor -> sink\u3002\u4e00\u4e2ainterceptor\u53ea\u5c5e\u4e8e\u5176\u4e2d\u4e00\u79cd\u3002\u5927\u90e8\u5206\u7ec4\u4ef6\u4e3asource interceptor\u7c7b\u578b\uff0c\u53ef\u652f\u6301\u914d\u7f6ebelongTo\u88ab\u90e8\u5206source\u4f7f\u7528\u3002\u5c11\u6570\u901a\u7528\u6027\u8d28\u7684\u6bd4\u5982retry interceptor\u4e3asink interceptor\u7c7b\u578b\u3002
"},{"location":"reference/pipelines/interceptor/overview/#interceptor","title":"Interceptor\u901a\u7528\u914d\u7f6e","text":""},{"location":"reference/pipelines/interceptor/overview/#enabled","title":"enabled","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b true \u8868\u793a\u662f\u5426\u5f00\u542f\u8be5interceptor"},{"location":"reference/pipelines/interceptor/overview/#name","title":"name","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 name string \u975e\u5fc5\u586b \u8868\u793ainterceptor\u7684\u540d\u79f0\u3002\u5f53pipeline\u91cc\u914d\u7f6e\u76f8\u540ctype interceptor\u7684\u60c5\u51b5\u4e0b\uff0c\u5fc5\u586b\uff0c\u7528\u4e8e\u533a\u5206\u6807\u8bc6"},{"location":"reference/pipelines/interceptor/overview/#belongto","title":"belongTo","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 belongTo string\u6570\u7ec4 \u975e\u5fc5\u586b \u4ec5source interceptor\u53ef\u7528\uff0c\u7528\u4e8e\u6307\u5b9a\u8be5interceptor\u4ec5\u88ab\u54ea\u4e9bsource\u4f7f\u7528"},{"location":"reference/pipelines/interceptor/overview/#order","title":"order","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 order int \u975e\u5fc5\u586b interceptor\u7684\u6392\u5217\u987a\u5e8f\u6743\u91cd"},{"location":"reference/pipelines/interceptor/retry/","title":"retry","text":"\u7528\u4e8e\u7ed9\u4e0b\u6e38\u53d1\u9001\u5931\u8d25\u65f6\u91cd\u8bd5\u3002 \u7cfb\u7edf\u5185\u7f6e\uff0c\u9ed8\u8ba4\u52a0\u8f7d\uff0c\u5c5e\u4e8esink interceptor\u3002
Example
interceptors:\n- type: retry\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 retryMaxCount int \u975e\u5fc5\u586b 0 \u6700\u5927\u7684\u91cd\u8bd5\u6b21\u6570"},{"location":"reference/pipelines/interceptor/schema/","title":"schema","text":"\u4e13\u6ce8\u4e8e\u65e5\u5fd7\u683c\u5f0f\u8f6c\u6362\u4e0e\u9002\u914d\u7684interceptor\u3002 \u5c5e\u4e8esource interceptor\u3002
"},{"location":"reference/pipelines/interceptor/schema/#_1","title":"\u4f7f\u7528\u573a\u666f","text":"\u5bf9\u4e8e\u5927\u90e8\u5206\u65e5\u5fd7\u5bf9\u63a5\u7684\u573a\u666f\u4e2d\uff0c\u6211\u4eec\u8981\u6c42\u7684\u65e5\u5fd7\u683c\u5f0f\u53ef\u80fd\u6709\u4e00\u4e9b\u5dee\u5f02\uff0c\u8fd9\u4e9b\u5dee\u5f02\u4e3b\u8981\u4f53\u73b0\u5728\u65f6\u95f4\u5b57\u6bb5\u3001body\u5b57\u6bb5\u7b49\u3002 \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0cLoggie\u53ea\u4f1a\u628asource\u91c7\u96c6\u6216\u8005\u63a5\u6536\u5230\u7684\u539f\u59cb\u6570\u636e\u653e\u5230body\u5b57\u6bb5\u4e2d\uff0c\u4ee5\u6700\u7b80\u5355\u7684\u65b9\u5f0f\u53d1\u9001\uff1a
{\n\"body\": \"this is raw data\"\n}\n\u4f46\u662f\uff0c\u4e00\u4e9b\u573a\u666f\u4e0b\u6211\u4eec\u9700\u8981\uff1a
body\u7b49\u5b57\u6bb5Example
\u589e\u52a0@timestamp\uff0cbody\u4fee\u6539\u4e3amessageinterceptors:\n- type: schema\naddMeta:\ntimestamp:\nkey: \"@timestamp\"\nremap:\nbody:\nkey: message\n\u8f6c\u6362\u540e\u7684event:
{\n\"message\": \"this is raw data\"\n\"@timestamp\": \"2022-08-30T06:58:49.545Z\",\n}\ninterceptors:\n- type: schema\naddMeta:\ntimestamp:\nkey: \"_timestamp_\"\nlocation: Local\nlayout: 2006-01-02T15:04:05Z07:00\nremap:\nbody:\nkey: _log_\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addMeta \u975e\u5fc5\u586b \u589e\u52a0\u7cfb\u7edf\u5143\u4fe1\u606f\u5b57\u6bb5"},{"location":"reference/pipelines/interceptor/schema/#timestamp","title":"timestamp","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addMeta.timestamp \u975e\u5fc5\u586b \u589e\u52a0\u7cfb\u7edf\u65f6\u95f4\u5b57\u6bb5\uff08source\u91c7\u96c6\u5230\u6570\u636e\u7684\u65f6\u95f4\uff09 addMeta.timestamp.key string \u5fc5\u586b \u7cfb\u7edf\u65f6\u95f4\u7684key addMeta.timestamp.location string \u975e\u5fc5\u586b \u9ed8\u8ba4\u4e3a\u7a7a\uff0c\u5373\u4e3aUTC\u65f6\u95f4 \u589e\u52a0\u7684\u65f6\u95f4\u65f6\u533a\uff0c\u53e6\u5916\u8fd8\u652f\u6301Local addMeta.timestamp.layout string \u975e\u5fc5\u586b \"2006-01-02T15:04:05.000Z\" golang\u7684\u65f6\u95f4\u7c7b\u578blayout\uff0c\u53ef\u53c2\u8003https://go.dev/src/time/format.go"},{"location":"reference/pipelines/interceptor/schema/#pipelinename","title":"pipelineName","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addMeta.pipelineName \u975e\u5fc5\u586b \u5c06pipelineName\u52a0\u5165\u5230event\u4e2d addMeta.pipelineName.key string \u5fc5\u586b \u589e\u52a0\u540e\u7684\u5b57\u6bb5\u7684key Example
interceptors:\n- type: schema\naddMeta:\npipelineName:\nkey: pipeline\n{\n\"pipeline\": \"demo\"\n...\n}\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addMeta.sourceName \u975e\u5fc5\u586b \u5c06sourceName\u52a0\u5165\u5230event\u4e2d addMeta.sourceName.key string \u5fc5\u586b \u589e\u52a0\u540e\u7684\u5b57\u6bb5\u7684key Example
interceptors:\n- type: schema\naddMeta:\nsourceName:\nkey: source\n{\n\"source\": \"local\"\n...\n}\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 remap map \u975e\u5fc5\u586b \u5bf9\u5b57\u6bb5\u8fdb\u884c\u8f6c\u6362\uff0c\u76ee\u524d\u652f\u6301\u91cd\u547d\u540d remap.[originKey] string \u975e\u5fc5\u586b \u539f\u59cb\u5b57\u6bb5key remap.[originKey].key string \u975e\u5fc5\u586b \u8f6c\u6362\u540e\u7684\u5b57\u6bb5key Example
interceptors:\n- type: schema\nremap:\nbody:\nkey: msg\nstate:\nkey: meta\n{\n\"body\": \"this is log\"\n\"state\": \"ok\",\n}\n\u8f6c\u6362\u540e\u7684event:
{\n\"msg\": \"this is log\"\n\"meta\": \"ok\",\n}\n\u5e26\u6709\u6761\u4ef6\u5224\u65ad\u7684\u51fd\u6570\u5f0f\u6570\u636e\u5904\u7406interceptor\u3002 \u5c5e\u4e8esource interceptor\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#_1","title":"\u4f7f\u7528\u573a\u666f","text":"\u793a\u4f8b\u53c2\u8003\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#_2","title":"\u4f7f\u7528\u65b9\u5f0f","text":"transformer\u4f1a\u6309\u7167\u914d\u7f6e\u7684actions\u91cc\u987a\u5e8f\u6267\u884c\u6240\u6709\u7684action\u3002action\u7c7b\u4f3c\u51fd\u6570\u7684\u65b9\u5f0f\uff0c\u53ef\u4ee5\u5199\u5165\u53c2\u6570\uff0c\u53c2\u6570\u4e00\u822c\u4e3aevent\u91cc\u7684\u5b57\u6bb5\u3002 \u540c\u65f6\uff0c\u6bcf\u4e2aaction\u91cc\u8fd8\u53ef\u80fd\u5305\u62ec\u989d\u5916\u7684\u63a7\u5236\u5b57\u6bb5\u3002\u6bd4\u5982\u4e0b\u9762regex(body)\uff0cbody\u5373\u4e3aregex\u7684\u53c2\u6570\uff0cpattern\u4e3a\u989d\u5916\u7684\u5b57\u6bb5\u3002
interceptors:\n- type: transformer\nactions:\n- action: regex(body)\npattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$\n- action: add(topic, common)\n\u53e6\u5916\uff0caction\u8fd8\u652f\u6301\u6761\u4ef6\u5224\u65adif-then-else\u7684\u65b9\u5f0f\uff1a
- if: <condition>\nthen:\n- action: funcA()\nelse:\n- action: funcB()\n\u5176\u4e2d\uff0ccondition\u6761\u4ef6\u5224\u65ad\u4e5f\u4e3a\u51fd\u6570\u7684\u5f62\u5f0f\u3002
interceptors:\n- type: transformer\nactions:\n- if: equal(status, 404)\nthen:\n- action: add(topic, not_found)\n- action: return()\nExample
- type: transformer\nactions:\n- action: regex(body)\npattern: (?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)\nignoreError: true\n\u8868\u793a\u5982\u679c\u51fa\u73b0\u9519\u8bef\uff0c\u76f4\u63a5\u4e22\u5f03\u8be5\u6761event\u3002
Example
- type: transformer\nactions:\n- action: regex(body)\npattern: (?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)\ndropIfError: true\n\u7ed9event\u6dfb\u52a0\u989d\u5916\u7684key:value\u3002
Example
- action: add(topic, loggie)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"topic\": \"loggie\"\n}\n\u590d\u5236event\u91cc\u7684\u5b57\u6bb5\u3002
\u53c2\u6570\uff1a
Example
- action: copy(foo, bar)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"loggie\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"loggie\",\n\"bar\": \"loggie\"\n}\n\u79fb\u52a8/\u91cd\u547d\u540d\u5b57\u6bb5\u3002
\u53c2\u6570\uff1a
Example
- action: move(foo, bar)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"loggie\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"bar\": \"loggie\"\n}\n\u66f4\u65b0\u5b57\u6bb5key\u7684\u503c\u4e3avalue\u3002
\u53c2\u6570\uff1a
Example
- action: set(foo, test)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"loggie\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"test\"\n}\n\u5220\u9664\u5b57\u6bb5\u3002\u53ef\u586b\u5199\u591a\u4e2a\u5b57\u6bb5key\u3002
Example
- action: del(foo)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"foo\": \"loggie\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n}\n\u5c06\u5d4c\u5957\u7684\u5b57\u6bb5\u653e\u5728\u6839\u90e8\uff08\u6700\u5916\u5c42\uff09\u3002
Example
- action: underRoot(state)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"state\": {\n\"node\": \"127.0.0.1\",\n\"phase\": \"running\"\n}\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"node\": \"127.0.0.1\",\n\"phase\": \"running\"\n}\n\u5c06\u67d0\u4e2a\u5b57\u6bb5\u7684\u503c\u91cd\u65b0\u6e32\u67d3\uff0c\u53ef\u6839\u636e\u5176\u4ed6\u7684\u5b57\u6bb5\u503c\u7ec4\u6210\u4e00\u4e2a\u503c\u3002\u5982\u679ckey\u4e0d\u5b58\u5728\u5219\u4f1a\u65b0\u589e\u8be5\u5b57\u6bb5\u3002
\u989d\u5916\u5b57\u6bb5\uff1a
${state.node}-${state.phase}\u3002\u5982\u679cpattern\u4e3a\u56fa\u5b9a\u503c\uff0c\u5219\u7c7b\u4f3cset(key, value)\u3002Example
- action: fmt(status)\npattern: ${state.node} is ${state.phase}\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"state\": {\n\"node\": \"127.0.0.1\",\n\"phase\": \"running\"\n}\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"state\": {\n\"node\": \"127.0.0.1\",\n\"phase\": \"running\"\n}\uff0c\n\"status\": \"127.0.0.1 is running\"\n}\n\u5b57\u6bb5\u7684\u65f6\u95f4\u683c\u5f0f\u8f6c\u6362\u3002
\u989d\u5916\u5b57\u6bb5\uff1a
unix\u548cunix_msUTC\u6216\u8005Local\uff0c\u5982\u679c\u4e3a\u7a7a\uff0c\u5219\u4e3aUTCunix\u548cunix_msUTC\u6216\u8005Local\uff0c\u5982\u679c\u4e3a\u7a7a\uff0c\u5219\u4e3aUTCExample
- action: timestamp(time)\nfromLayout: \"2006-01-02 15:04:05\"\nfromLocation: Asia/Shanghai\ntoLayout: unix_ms\ntoLocation: Local\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"time\": \"2022-06-28 11:24:35\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"time\": 1656386675000\n}\n\u4ee5\u4e0a\u7684layout\u53c2\u6570\u9700\u8981\u586b\u5199golang\u5f62\u5f0f\uff0c\u53ef\u53c2\u8003\uff1a
const (\n Layout = \"01/02 03:04:05PM '06 -0700\" // The reference time, in numerical order.\n ANSIC = \"Mon Jan _2 15:04:05 2006\"\n UnixDate = \"Mon Jan _2 15:04:05 MST 2006\"\n RubyDate = \"Mon Jan 02 15:04:05 -0700 2006\"\n RFC822 = \"02 Jan 06 15:04 MST\"\n RFC822Z = \"02 Jan 06 15:04 -0700\" // RFC822 with numeric zone\n RFC850 = \"Monday, 02-Jan-06 15:04:05 MST\"\n RFC1123 = \"Mon, 02 Jan 2006 15:04:05 MST\"\n RFC1123Z = \"Mon, 02 Jan 2006 15:04:05 -0700\" // RFC1123 with numeric zone\n RFC3339 = \"2006-01-02T15:04:05Z07:00\"\n RFC3339Nano = \"2006-01-02T15:04:05.999999999Z07:00\"\n Kitchen = \"3:04PM\"\n // Handy time stamps.\n Stamp = \"Jan _2 15:04:05\"\n StampMilli = \"Jan _2 15:04:05.000\"\n StampMicro = \"Jan _2 15:04:05.000000\"\n StampNano = \"Jan _2 15:04:05.000000000\"\n)\n\u4f7f\u7528\u6b63\u5219\u7684\u65b9\u5f0f\u5207\u5206\u65e5\u5fd7\uff0c\u63d0\u53d6\u5b57\u6bb5\u3002 \u53e6\u5916\u4e5f\u53ef\u4ee5\u4e3aregex(key, to)\u3002
\u53c2\u6570\uff1a
\u989d\u5916\u5b57\u6bb5\uff1a
Example
- action: regex(body)\npattern: (?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)\ninput:
{\n\"body\": \"10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] 'GET / HTTP/1.1' 404 683\",\n}\noutput:
{\n\"ip\": \"10.244.0.1\",\n\"id\": \"-\",\n\"u\": \"-\",\n\"time\": \"[13/Dec/2021:12:40:48 +0000]\",\n\"url\": \"GET / HTTP/1.1\",\n\"status\": \"404\",\n\"size\": \"683\",\n}\n\u4f7f\u7528grok\u7684\u65b9\u5f0f\u5207\u5206\u65e5\u5fd7\uff0c\u63d0\u53d6\u5b57\u6bb5\u3002 \u53e6\u5916\u4e5f\u53ef\u4ee5\u4e3agrok(key, to)\u3002
\u53c2\u6570\uff1a
\u989d\u5916\u5b57\u6bb5\uff1a
Example
- action: grok(body)\nmatch: \"^%{DATESTAMP:datetime} %{FILE:file}:%{INT:line}: %{IPV4:ip} %{PATH:path} %{UUID:uuid}(?P<space>[a-zA-Z]?)\"\npattern: FILE: \"[a-zA-Z0-9._-]+\"\ninput:
{\n\"body\": \"2022/05/28 01:32:01 logTest.go:66: 192.168.0.1 /var/log/test.log 54ce5d87-b94c-c40a-74a7-9cd375289334\",\n}\noutput:
\"datetime\": \"2022/05/28 01:32:01\",\n \"line\": \"66\",\n \"ip\": \"192.168.0.1\",\n \"path\": \"/var/log/test.log\",\n \"uuid\": \"54ce5d87-b94c-c40a-74a7-9cd375289334\",\n\u5c06json\u6587\u672c\u53cd\u5e8f\u5217\u5316\u3002 \u4e5f\u53ef\u4ee5\u4e3ajsonDecode(key, to)\u3002
\u53c2\u6570\uff1a
Example
- action: jsonDecode(body)\ninput:
{\n\"body\": `{\"log\":\"I0610 08:29:07.698664 Waiting for caches to sync\", \"stream\":\"stderr\", \"time\":\"2021-06-10T08:29:07.698731204Z\"}`,\n}\noutput:
{\n\"log\": \"I0610 08:29:07.698664 Waiting for caches to sync\",\n\"stream\": \"stderr\", \"time\": \"2021-06-10T08:29:07.698731204Z\"\n}\n\u5c06\u591a\u4e2a\u5b57\u6bb5\u5e8f\u5217\u5316\u6210json string\u5f62\u5f0f\u3002 \u4e5f\u53ef\u4ee5\u4e3ajsonEncode(key, to)\u3002
\u53c2\u6570\uff1a
Example
interceptors:\n- type: transformer\nactions:\n- action: jsonEncode(fields)\ninput:
{\n\"body\": \"this is test\",\n\"fields\":\n\"topic\": \"loggie\",\n\"foo\": \"bar\"\n}\noutput:
{\n\"fields\": \"{\\\"topic\\\":\\\"loggie\\\",\\\"foo\\\":\\\"bar\\\"}\",\n\"body\": \"this is test\"\n}\n\u5c06\u4e00\u884c\u65e5\u5fd7\u6839\u636e\u67d0\u79cd\u5206\u5272\u7b26\u5207\u5206\u3002
\u53c2\u6570\uff1a
\u989d\u5916\u5b57\u6bb5\uff1a
Example
interceptors:\n- type: transformer\nactions:\n- action: split(body)\nseparator: \"|\"\nkeys: [\"time\", \"order\", \"service\", \"price\"]\ninput:
\"body\": `2021-08-08|U12345|storeCenter|13.14`,\noutput:
\"time\": \"2021-08-08\"\n \"order\": \"U12345\"\n \"service\": \"storeCenter\"\n \"price: \"13.14\"\n\u5b57\u6bb5\u503c\u7c7b\u578b\u8f6c\u6362\u3002
\u53c2\u6570\uff1a
bool, int, floatExample
- action: strconv(code, int)\ninput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"code\": \"200\"\n}\noutput:
{\n\"body\": \"2021-02-16T09:21:20.545525544Z DEBUG this is log body\",\n\"code\": 200\n}\n\u6253\u5370event\u3002\u4e00\u822c\u7528\u4e8e\u8c03\u8bd5\u9636\u6bb5\u4f7f\u7528\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#return","title":"return()","text":"\u63a7\u5236\u7c7b\u578b\u51fd\u6570\uff0c\u6267\u884c\u5230return()\u540e\u8fd4\u56de\uff0c\u4e0d\u518d\u7ee7\u7eed\u6267\u884c\u4e0b\u9762\u7684action\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#dropevent","title":"dropEvent()","text":"\u63a7\u5236\u7c7b\u578b\u51fd\u6570\uff0c\u6267\u884c\u5230dropEvent()\u540e\u4f1a\u5c06\u8be5event\u76f4\u63a5\u4e22\u5f03\u3002\u8fd9\u610f\u5473\u7740\u8be5\u6761\u6570\u636e\u4f1a\u4e22\u5931\uff0c\u4e5f\u4e0d\u4f1a\u7ee7\u7eed\u88ab\u540e\u7eed\u7684interceptor\u6216\u8005sink\u5904\u7406\u6d88\u8d39\u3002
Example
interceptors:\n- type: transformer\nactions:\n- action: regex(body)\npattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$\n- if: equal(level, DEBUG)\nthen:\n- action: dropEvent()\n2021-02-16T09:21:20.545525544Z DEBUG this is log body\uff0c\u5219\u6ee1\u8db3level\u5b57\u6bb5\u4e3aDEBUG\uff0c\u4f1a\u76f4\u63a5\u4e22\u5f03\u8be5\u6761\u65e5\u5fd7\u3002"},{"location":"reference/pipelines/interceptor/transformer/#condition","title":"condition","text":"\u6761\u4ef6\u5224\u65ad\u7c7b\u51fd\u6570\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#_4","title":"\u64cd\u4f5c\u7b26","text":"\u548c\u3002Example
interceptors:\n- type: transformer\nactions:\n- if: equal(level, DEBUG) AND equal(code, 200)\nthen:\n- action: dropEvent()\n\u6216\u3002Example
interceptors:\n- type: transformer\nactions:\n- if: equal(level, DEBUG) OR equal(level, INFO)\nthen:\n- action: dropEvent()\n\u53cd\u3002Example
interceptors:\n- type: transformer\nactions:\n- if: NOT equal(level, DEBUG)\nthen:\n- action: dropEvent()\n\u76ee\u6807\u5b57\u6bb5\u503c\u662f\u5426\u548c\u53c2\u6570\u503ctarget\u76f8\u7b49\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#containkey-target","title":"contain(key, target)","text":"\u5b57\u6bb5\u503c\u662f\u5426\u5305\u542b\u53c2\u6570\u503ctarget\u3002
Caution
target\u8bf7\u76f4\u63a5\u4f7f\u7528\u5b57\u7b26\u4e32\uff0c\u65e0\u9700\u6dfb\u52a0\u53cc\u5f15\u53f7\u3002 \u6bd4\u5982contain(body, error)\uff0c\u800c\u4e0d\u662fcontain(body, \"error\")\u3002contain(body, \"error\")\u4f1a\u88ab\u5f53\u4f5c\"error\"\u6765\u5339\u914d\u3002
\u76ee\u6807\u5b57\u6bb5\u662f\u5426\u5b58\u5728\u6216\u8005\u662f\u5426\u4e3a\u7a7a\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#greaterkey-value","title":"greater(key, value)","text":"\u76ee\u6807\u5b57\u6bb5\u7684\u503c\u662f\u5426\u5927\u4e8e\u53c2\u6570\u503cvalue\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#lesskey-value","title":"less(key, value)","text":"\u76ee\u6807\u5b57\u6bb5\u7684\u503c\u662f\u5426\u5c0f\u4e8e\u53c2\u6570\u503cvalue\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#hasprefixkey-prefix","title":"hasPrefix(key, prefix)","text":"\u76ee\u6807\u5b57\u6bb5\u7684\u503c\u662f\u5426\u5305\u542bprefix\u524d\u7f00\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#matchkey-regex","title":"match(key, regex)","text":"\u76ee\u6807\u5b57\u6bb5\u7684\u503c\u662f\u5426\u548c\u53c2\u6570regex\u6b63\u5219\u5339\u914d\u3002
"},{"location":"reference/pipelines/interceptor/transformer/#oneofkey-value1-value2","title":"oneOf(key, value1, value2...)","text":"\u76ee\u6807\u5b57\u6bb5\u7684\u503c\u662f\u5426\u662f\u53c2\u6570\u503cvalue1...\u5176\u4e2d\u4e4b\u4e00\u3002
"},{"location":"reference/pipelines/queue/channel/","title":"channel","text":"channel queue,\u662f\u57fa\u4e8ego chan\u5b9e\u73b0\u7684\u5185\u5b58\u7f13\u51b2queue\u3002
Example
queue:\ntype: channel\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchSize int \u4e0d\u5fc5\u586b 2048 \u4e00\u4e2a\u6279\u6b21\u5305\u542b\u7684event\u6570\u91cf"},{"location":"reference/pipelines/queue/channel/#batchbytes","title":"batchBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchBytes int64 \u4e0d\u5fc5\u586b 33554432(32MB) \u4e00\u4e2a\u6279\u6b21\u5305\u542b\u7684\u6570\u636e\u6700\u5927\u5b57\u8282\u6570"},{"location":"reference/pipelines/queue/channel/#batchaggtimeout","title":"batchAggTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchAggTimeout time.Duration \u4e0d\u5fc5\u586b 1s \u7ec4\u88c5\u805a\u5408\u591a\u4e2aevent\u6210\u4e00\u4e2abatch\u7b49\u5f85\u7684\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/queue/memory/","title":"memory","text":"memory queue,\u662f\u57fa\u4e8ego-disruptor\u5b9e\u73b0\u7684\u5185\u5b58\u7f13\u51b2queue\u3002
Note
\u7531\u4e8ego-disruptor\u7248\u672c\u8fd8\u672arelease\uff0cmemory queue\u8fd8\u5c5e\u4e8e\u5b9e\u9a8c\u9636\u6bb5\u3002\u751f\u4ea7\u73af\u5883\u4e0d\u5efa\u8bae\u4f7f\u7528\uff01
Example
queue:\ntype: memory\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchSize int \u4e0d\u5fc5\u586b 2048 \u4e00\u4e2a\u6279\u6b21\u5305\u542b\u7684event\u6570\u91cf"},{"location":"reference/pipelines/queue/memory/#batchbufferfactor","title":"batchBufferFactor","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchBufferFactor int \u4e0d\u5fc5\u586b 2 queue\u7f13\u51b2\u533a\u7684\u5927\u5c0f(channel\u7684\u5bb9\u91cf)=batchSize*batchBufferFactor"},{"location":"reference/pipelines/queue/memory/#batchbytes","title":"batchBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchBytes int64 \u4e0d\u5fc5\u586b 33554432(32MB) \u4e00\u4e2a\u6279\u6b21\u5305\u542b\u7684\u6570\u636e\u6700\u5927\u5b57\u8282\u6570"},{"location":"reference/pipelines/queue/memory/#batchaggtimeout","title":"batchAggTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchAggTimeout time.Duration \u4e0d\u5fc5\u586b 1s \u7ec4\u88c5\u805a\u5408\u591a\u4e2aevent\u6210\u4e00\u4e2abatch\u7b49\u5f85\u7684\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/dev/","title":"dev","text":"dev sink\u5c06\u65e5\u5fd7\u6570\u636e\u6253\u5370\u5230\u63a7\u5236\u53f0\uff0c\u4e00\u822c\u53ef\u4ee5\u7528\u4e8edebug\u6216\u8005\u6392\u67e5\u95ee\u9898\u3002 \u914d\u7f6edev sink\u540e\uff0c\u53ef\u4ee5\u8bbe\u7f6eprintEvents=true\uff0c\u67e5\u770b\u5728Loggie\u4e2d\u53d1\u9001\u81f3sink\u7684\u65e5\u5fd7\u6570\u636e\uff0c\u8be5\u6570\u636e\u9664\u4e86source\u63a5\u6536\u6216\u8005\u91c7\u96c6\u7684\u539f\u59cb\u65e5\u5fd7\uff0c\u4e00\u822c\u8fd8\u5305\u542b\u5176\u4ed6\u5143\u4fe1\u606f\u3002
Example
sink:\ntype: dev\nprintEvents: true\ncodec:\ntype: json\npretty: true\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 printEvents bool \u975e\u5fc5\u586b false \u662f\u5426\u6253\u5370\u91c7\u96c6\u7684\u65e5\u5fd7 \u9ed8\u8ba4\u60c5\u51b5\u4e0bLoggie\u7684\u65e5\u5fd7\u6253\u5370\u4e3ajson\u683c\u5f0f\uff0c\u53ef\u4ee5\u914d\u7f6e\u542f\u52a8\u53c2\u6570-log.jsonFormat=false\uff0c\u4fbf\u4e8e\u5728Loggie\u65e5\u5fd7\u4e0a\u67e5\u770b\u8f93\u51fa\u7ed3\u679c\u3002
\u4f7f\u7528Elasticsearch sink\u53d1\u9001\u6570\u636e\u81f3Elasticsearch\u96c6\u7fa4\u3002
Example
sink:\ntype: elasticsearch\nhosts: [\"elasticsearch1:9200\", \"elasticsearch2:9200\", \"elasticsearch3:9200\"]\nindex: \"log-${fields.service}-${+YYYY.MM.DD}\"\nCaution
\u5982\u679celasticsearch\u7248\u672c\u4e3av6.x\uff0c\u8bf7\u52a0\u4e0a\u4ee5\u4e0betype: _doc\u53c2\u6570\u3002
sink:\ntype: elasticsearch\netype: _doc\n...\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 hosts string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u53d1\u9001\u65e5\u5fd7\u81f3Elasticsearch\u7684\u5730\u5740"},{"location":"reference/pipelines/sink/elasticsearch/#index","title":"index","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 index string \u5fc5\u586b \u65e0 \u53d1\u9001\u81f3Elasticsearch\u540e\uff0c\u65e5\u5fd7\u6570\u636e\u5b58\u50a8\u7684index \u53ef\u4ee5\u4f7f\u7528${a.b}\u7684\u65b9\u5f0f\u83b7\u53d6\u65e5\u5fd7\u6570\u636e\u91cc\u7684\u5b57\u6bb5\uff0c\u6216\u8005\u52a0\u4e0a${+YYYY.MM.DD.hh}\u65f6\u95f4\u6233\u7b49\u65b9\u5f0f\u6765\u52a8\u6001\u751f\u6210index\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 username string \u975e\u5fc5\u586b \u65e0 \u5982\u679cElasticsearch\u914d\u7f6e\u4e86\u7528\u6237\u540d\u5bc6\u7801\u9a8c\u8bc1\uff0c\u9700\u8981\u586b\u5199\u8bf7\u6c42\u7684\u7528\u6237\u540d"},{"location":"reference/pipelines/sink/elasticsearch/#password","title":"password","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 password string \u975e\u5fc5\u586b \u65e0 \u5982\u679cElasticsearch\u914d\u7f6e\u4e86\u7528\u6237\u540d\u5bc6\u7801\u9a8c\u8bc1\uff0c\u9700\u8981\u586b\u5199\u8bf7\u6c42\u7684\u5bc6\u7801"},{"location":"reference/pipelines/sink/elasticsearch/#schema","title":"schema","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 schema string \u975e\u5fc5\u586b http client sniffing\u65f6\u4f7f\u7528"},{"location":"reference/pipelines/sink/elasticsearch/#sniff","title":"sniff","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sniff bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542fsniffer"},{"location":"reference/pipelines/sink/elasticsearch/#gzip","title":"gzip","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 gzip bool \u975e\u5fc5\u586b false \u53d1\u9001\u6570\u636e\u662f\u5426\u5f00\u542fgzip\u538b\u7f29"},{"location":"reference/pipelines/sink/elasticsearch/#documentid","title":"documentId","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 documentId string \u975e\u5fc5\u586b \u53d1\u9001\u81f3elasticsearch\u7684id\u503c\uff0c\u53ef\u4f7f\u7528${}\u7684\u65b9\u5f0f\u53d6\u67d0\u4e2a\u5b57\u6bb5"},{"location":"reference/pipelines/sink/elasticsearch/#optype","title":"opType","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 opType string \u975e\u5fc5\u586b index \u53c2\u8003\u5b98\u65b9\u6587\u6863, \u5982\u679c\u76ee\u6807\u4e3adatastream\uff0c\u5219\u9700\u8981\u8bbe\u7f6e\u4e3acreate"},{"location":"reference/pipelines/sink/file/","title":"file","text":"\u5c06\u63a5\u6536\u5230\u7684\u6570\u636e\u4ee5\u6587\u4ef6\u7684\u5f62\u5f0f\u5199\u5165\u5230\u672c\u5730\u3002
Example
sink:\ntype: file\nworkerCount: 1024\nbaseDirs:\n- /data0\n- /data1\n- /data2\ndirHashKey: ${namespace}-${deployName}\nfilename: /${namespace}/${deployName}/${podName}/${filename}\nmaxSize: 500\nmaxAge: 7\nmaxBackups: 50\ncompress: true\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 workerCount int \u975e\u5fc5\u586b 1 \u5199\u6587\u4ef6\u7684\u5e76\u53d1\u6570"},{"location":"reference/pipelines/sink/file/#basedirs","title":"baseDirs","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 baseDirs string\u6570\u7ec4 \u975e\u5fc5\u586b \u6587\u4ef6\u7684\u57fa\u7840\u76ee\u5f55\uff0c\u53ef\u4ee5\u6309\u67d0\u4e2akey\u505a\u54c8\u5e0c\uff0c\u7136\u540e\u5b58\u50a8\u5230\u5bf9\u5e94\u7684\u57fa\u7840\u76ee\u5f55\u4e0a"},{"location":"reference/pipelines/sink/file/#dirhashkey","title":"dirHashKey","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 dirHashKey string \u975e\u5fc5\u586b \u6309\u6307\u5b9aKey\u505a\u54c8\u5e0c\uff0c\u652f\u6301\u53d8\u91cf"},{"location":"reference/pipelines/sink/file/#filename","title":"filename","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 filename string \u5fc5\u586b \u6587\u4ef6\u540d\uff0c\u652f\u6301\u53d8\u91cf"},{"location":"reference/pipelines/sink/file/#maxsize","title":"maxSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxSize int \u975e\u5fc5\u586b 1 \u6587\u4ef6\u5927\u5c0f\uff0c\u5355\u4f4d\u4e3aMiB"},{"location":"reference/pipelines/sink/file/#maxage","title":"maxAge","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxAge int \u975e\u5fc5\u586b \u65e7\u6587\u4ef6\u4fdd\u7559\u5929\u6570\uff0c\u5355\u4f4d\u300c\u5929\u300d\uff0c\u9ed8\u8ba4\u4e0d\u5220\u9664"},{"location":"reference/pipelines/sink/file/#maxbackups","title":"maxBackups","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxBackups int \u975e\u5fc5\u586b 1 \u6700\u5927\u4fdd\u7559\u7684\u5907\u4efd\u6587\u4ef6\u6570\uff0c\u9ed8\u8ba4\u4e0d\u5220\u9664\uff08\u5982\u679cmaxAge\u914d\u7f6e\u4e86\uff0c\u90a3\u4e48\u6587\u4ef6\u4f9d\u65e7\u4f1a\u88ab\u5220\u9664\uff09"},{"location":"reference/pipelines/sink/file/#localtime","title":"localTime","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 localTime bool \u975e\u5fc5\u586b false \u662f\u5426\u7528\u672c\u5730\u65f6\u95f4\u683c\u5f0f\u5316\u5907\u4efd\u6587\u4ef6\uff0c\u9ed8\u8ba4\u4f7f\u7528UTC\u65f6\u95f4"},{"location":"reference/pipelines/sink/file/#compress","title":"compress","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 compress bool \u975e\u5fc5\u586b false \u662f\u5426\u538b\u7f29\uff0c\u4f7f\u7528gzip\uff0c\u9ed8\u8ba4\u4e0d\u538b\u7f29"},{"location":"reference/pipelines/sink/franzkafka/","title":"franz kafka","text":"\u4f7f\u7528franz-go kafka\u5e93\u5c06\u65e5\u5fd7\u6570\u636e\u53d1\u9001\u81f3\u4e0b\u6e38Kafka\uff0c\u5e76\u4e14\u80fd\u6bd4\u8f83\u597d\u7684\u652f\u6301kerberos\u8ba4\u8bc1\u3002 \uff08\u672csink\u548ckafka sink\u7684\u533a\u522b\u4e00\u822c\u53ea\u5728\u4e8e\u4f7f\u7528\u7684kafka golang\u5e93\u4e0d\u540c\uff0c\u63d0\u4f9b\u7ed9\u5bf9franz kafka\u5e93\u6709\u504f\u597d\u7684\u7528\u6237\u4f7f\u7528\uff09
Example
sink:\ntype: franzKafka\nbrokers: [\"127.0.0.1:6400\"]\ntopic: \"log-${fields.topic}\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 brokers string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u53d1\u9001\u65e5\u5fd7\u81f3Kafka\u7684brokers\u5730\u5740"},{"location":"reference/pipelines/sink/franzkafka/#topic","title":"topic","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 topic string \u975e\u5fc5\u586b loggie \u53d1\u9001\u65e5\u5fd7\u81f3Kafka\u7684topic \u53ef\u4f7f\u7528${a.b}\u7684\u65b9\u5f0f\uff0c\u83b7\u53d6event\u91cc\u7684\u5b57\u6bb5\u503c\u4f5c\u4e3a\u5177\u4f53\u7684topic\u540d\u79f0\u3002
\u6bd4\u5982\uff0c\u4e00\u4e2aevent\u4e3a\uff1a
{\n\"topic\": \"loggie\",\n\"hello\": \"world\"\n}\ntopic: ${topic}\uff0c\u6b64\u65f6\u8be5event\u53d1\u9001\u5230Kafka\u7684topic\u4e3a\"loggie\"\u3002 \u540c\u65f6\u652f\u6301\u5d4c\u5957\u7684\u9009\u62e9\u65b9\u5f0f\uff1a
{\n\"fields\": {\n\"topic\": \"loggie\"\n},\n\"hello\": \"world\"\n}\ntopic: ${fields.topic}\uff0c\u540c\u6837\u4e5f\u4f1a\u53d1\u9001\u5230topic \"loggie\"\u3002"},{"location":"reference/pipelines/sink/franzkafka/#balance","title":"balance","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 balance string \u975e\u5fc5\u586b roundRobin \u8d1f\u8f7d\u5747\u8861\u7b56\u7565\uff0c\u53ef\u586broundRobin\u3001range\u3001sticky\u3001cooperativeSticky"},{"location":"reference/pipelines/sink/franzkafka/#compression","title":"compression","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 compression string \u975e\u5fc5\u586b gzip \u65e5\u5fd7\u53d1\u9001\u81f3Kafka\u7684\u538b\u7f29\u7b56\u7565\uff0c\u53ef\u586bgzip\u3001snappy\u3001lz4\u3001zstd"},{"location":"reference/pipelines/sink/franzkafka/#batchsize","title":"batchSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchSize int \u975e\u5fc5\u586b 100 \u53d1\u9001\u65f6\u6bcf\u4e2abatch\u6700\u591a\u5305\u542b\u7684\u6570\u636e\u4e2a\u6570"},{"location":"reference/pipelines/sink/franzkafka/#batchbytes","title":"batchBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchBytes int \u975e\u5fc5\u586b 1048576 \u6bcf\u4e2a\u53d1\u9001\u8bf7\u6c42\u5305\u542b\u7684\u6700\u5927\u5b57\u8282\u6570"},{"location":"reference/pipelines/sink/franzkafka/#writetimeout","title":"writeTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 writeTimeout time.Duration \u975e\u5fc5\u586b 10s \u5199\u5165\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/franzkafka/#tls","title":"tls","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tls.enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u542f\u7528 tls.caCertFiles string \u975e\u5fc5\u586b \u8bc1\u4e66\u6587\u4ef6\u8def\u5f84 tls.clientCertFile string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1a\u5ba2\u6237\u7aefcert\u6587\u4ef6 tls.clientKeyFile string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1a\u5ba2\u6237\u7aefkey\u6587\u4ef6 tls.endpIdentAlgo bool type=scram\u65f6\u5fc5\u586b \u5ba2\u6237\u7aef\u662f\u5426\u9a8c\u8bc1\u670d\u52a1\u7aef\u7684\u8bc1\u4e66\u540d\u5b57"},{"location":"reference/pipelines/sink/franzkafka/#sasl","title":"sasl","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sasl \u975e\u5fc5\u586b SASL authentication sasl.enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u542f\u7528 sasl.mechanism string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1aPLAIN\u3001SCRAM-SHA-256\u3001SCRAM-SHA-512\u3001GSSAPI sasl.username string \u5fc5\u586b \u7528\u6237\u540d sasl.password string \u5fc5\u586b \u5bc6\u7801"},{"location":"reference/pipelines/sink/franzkafka/#gssapi","title":"gssapi","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sasl.gssapi \u975e\u5fc5\u586b SASL authentication sasl.gssapi.authType string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1a1 \u4f7f\u7528\u8d26\u53f7\u5bc6\u7801\u30012 \u4f7f\u7528keytab sasl.gssapi.keyTabPath string \u5fc5\u586b keytab \u6587\u4ef6\u8def\u5f84 sasl.gssapi.kerberosConfigPath string \u5fc5\u586b kerbeos \u6587\u4ef6\u8def\u5f84 sasl.gssapi.serviceName string \u5fc5\u586b \u670d\u52a1\u540d\u79f0 sasl.gssapi.userName string \u5fc5\u586b \u7528\u6237\u540d sasl.gssapi.password string \u5fc5\u586b \u5bc6\u7801 sasl.gssapi.realm string \u5fc5\u586b \u9886\u57df sasl.gssapi.disablePAFXFAST bool type=scram\u65f6\u5fc5\u586b DisablePAFXFAST \u7528\u4e8e\u5c06\u5ba2\u6237\u7aef\u914d\u7f6e\u4e3a\u4e0d\u4f7f\u7528 PA_FX_FAST"},{"location":"reference/pipelines/sink/franzkafka/#security","title":"security","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 security string \u975e\u5fc5\u586b java\u683c\u5f0f\u7684\u5b89\u5168\u8ba4\u8bc1\u5185\u5bb9\uff0c\u53ef\u4ee5\u81ea\u52a8\u8f6c\u5316\u6210\u4e3afranz-go\u9002\u914d\u7684\u683c\u5f0f \u6848\u4f8b:
pipelines:\n - name: local\n sources:\n - type: file\n name: demo\n paths:\n - /tmp/log/*.log\n sink:\n type: franzKafka\n brokers:\n - \"hadoop74.axrzpt.com:9092\"\n topic: loggie\n writeTimeout: 5s\n sasl:\n gssapi:\n kerberosConfigPath: /etc/krb5-conf/krb5.conf\n security:\n security.protocol: \"SASL_PLAINTEXT\"\n sasl.mechanism: \"GSSAPI\"\n sasl.jaas.config: \"com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true keyTab=\\\"/shylock/kerberos/zork.keytab\\\" principal=\\\"zork@AXRZPT.COM\\\";\"\n sasl.kerberos.service.name: \"kafka\"\nKubernetes \u6302\u8f7dkeytab\u4e8c\u8fdb\u5236\u8bc1\u4e66\uff0c\u8bf7\u53c2\u8003\u5b98\u65b9\u6587\u6863\u3002
"},{"location":"reference/pipelines/sink/grpc/","title":"grpc","text":"\u4f7f\u7528grpc sink\u53d1\u9001\u81f3\u4e0b\u6e38\uff0c\u9700\u8981\u4e0b\u6e38\u652f\u6301\u76f8\u540c\u683c\u5f0f\u7684grpc\u534f\u8bae\u3002\u53ef\u4ee5\u4f7f\u7528\u8be5sink\u53d1\u9001\u81f3grpc source\u7684Loggie\u4e2d\u8f6c\u673a\u3002
Example
sink:\ntype: grpc\nhost: \"loggie-aggregator.loggie-aggregator:6166\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 host string \u5fc5\u586b \u65e0 \u53d1\u9001\u81f3\u4e0b\u6e38\u7684host\u5730\u5740\uff0c\u591a\u4e2aip\u4f7f\u7528,\u9017\u53f7\u5206\u9694"},{"location":"reference/pipelines/sink/grpc/#loadbalance","title":"loadBalance","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 loadBalance string \u975e\u5fc5\u586b round_robin grpc\u8d1f\u8f7d\u5747\u8861\u7b56\u7565"},{"location":"reference/pipelines/sink/grpc/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 30s \u53d1\u9001\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/grpc/#grpcheaderkey","title":"grpcHeaderKey","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 grpcHeaderKey string \u975e\u5fc5\u586b \u65e0"},{"location":"reference/pipelines/sink/kafka/","title":"kafka","text":"\u4f7f\u7528sink kafka\u5c06\u65e5\u5fd7\u6570\u636e\u53d1\u9001\u81f3\u4e0b\u6e38Kafka\u3002
Example
sink:\ntype: kafka\nbrokers: [\"127.0.0.1:6400\"]\ntopic: \"log-${fields.topic}\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 brokers string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u53d1\u9001\u65e5\u5fd7\u81f3Kafka\u7684brokers\u5730\u5740"},{"location":"reference/pipelines/sink/kafka/#topic","title":"topic","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 topic string \u975e\u5fc5\u586b loggie \u53d1\u9001\u65e5\u5fd7\u81f3Kafka\u7684topic \u53ef\u4f7f\u7528${a.b}\u7684\u65b9\u5f0f\uff0c\u83b7\u53d6event\u91cc\u7684\u5b57\u6bb5\u503c\u4f5c\u4e3a\u5177\u4f53\u7684topic\u540d\u79f0\u3002
\u6bd4\u5982\uff0c\u4e00\u4e2aevent\u4e3a\uff1a
{\n\"topic\": \"loggie\",\n\"hello\": \"world\"\n}\ntopic: ${topic}\uff0c\u6b64\u65f6\u8be5event\u53d1\u9001\u5230Kafka\u7684topic\u4e3a\"loggie\"\u3002 \u540c\u65f6\u652f\u6301\u5d4c\u5957\u7684\u9009\u62e9\u65b9\u5f0f\uff1a
{\n\"fields\": {\n\"topic\": \"loggie\"\n},\n\"hello\": \"world\"\n}\ntopic: ${fields.topic}\uff0c\u540c\u6837\u4e5f\u4f1a\u53d1\u9001\u5230topic \"loggie\"\u3002"},{"location":"reference/pipelines/sink/kafka/#balance","title":"balance","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 balance string \u975e\u5fc5\u586b roundRobin \u8d1f\u8f7d\u5747\u8861\u7b56\u7565\uff0c\u53ef\u586bhash\u3001roundRobin\u3001leastBytes"},{"location":"reference/pipelines/sink/kafka/#compression","title":"compression","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 compression string \u975e\u5fc5\u586b gzip \u65e5\u5fd7\u53d1\u9001\u81f3Kafka\u7684\u538b\u7f29\u7b56\u7565\uff0c\u53ef\u586bgzip\u3001snappy\u3001lz4\u3001zstd"},{"location":"reference/pipelines/sink/kafka/#maxattempts","title":"maxAttempts","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxAttempts int \u975e\u5fc5\u586b 10 \u53d1\u9001\u6700\u591a\u91cd\u8bd5\u6b21\u6570"},{"location":"reference/pipelines/sink/kafka/#batchsize","title":"batchSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchSize int \u975e\u5fc5\u586b 100 \u53d1\u9001\u65f6\u6bcf\u4e2abatch\u6700\u591a\u5305\u542b\u7684\u6570\u636e\u4e2a\u6570"},{"location":"reference/pipelines/sink/kafka/#batchbytes","title":"batchBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchBytes int \u975e\u5fc5\u586b 1048576 \u6bcf\u4e2a\u53d1\u9001\u8bf7\u6c42\u5305\u542b\u7684\u6700\u5927\u5b57\u8282\u6570"},{"location":"reference/pipelines/sink/kafka/#batchtimeout","title":"batchTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchTimeout time.Duration \u975e\u5fc5\u586b 1s \u5f62\u6210\u6bcf\u4e2a\u53d1\u9001batch\u7684\u6700\u957f\u65f6\u95f4"},{"location":"reference/pipelines/sink/kafka/#readtimeout","title":"readTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readTimeout time.Duration \u975e\u5fc5\u586b 10s \u8bfb\u53d6\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/kafka/#writetimeout","title":"writeTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 writeTimeout time.Duration \u975e\u5fc5\u586b 10s \u5199\u5165\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/kafka/#requiredacks","title":"requiredAcks","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 requiredAcks int \u975e\u5fc5\u586b 0 \u7b49\u5f85ack\u53c2\u6570\uff0c\u53ef\u4e3a0\u30011\u3001-1 0: \u4e0d\u8981\u6c42ack1: \u7b49\u5f85leader partition ack-1: \u7b49\u5f85ISR\u4e2d\u6240\u6709replica ack\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sasl \u975e\u5fc5\u586b SASL authentication sasl.type string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1aplain\u3001scram sasl.userName string \u5fc5\u586b \u7528\u6237\u540d sasl.password string \u5fc5\u586b \u5bc6\u7801 sasl.algorithm string type=scram\u65f6\u5fc5\u586b type=scram\u65f6\u4f7f\u7528\u7684\u7b97\u6cd5\uff0c\u53ef\u9009sha256\u3001sha512"},{"location":"reference/pipelines/sink/kafka/#partitionkey","title":"partitionKey","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 partitionKey string \u975e\u5fc5\u586b \u65e0 \u63a7\u5236\u53d1\u9001\u81f3topic\u4e0b\u54ea\u4e2a\u5206\u533a \u4e0etopic\u76f8\u4f3c\uff0c\u53ef\u4f7f\u7528${a.b}\u7684\u65b9\u5f0f\uff0c\u83b7\u53d6event\u91cc\u7684\u5b57\u6bb5\u503c\u4f5c\u4e3a\u5177\u4f53\u7684topic\u540d\u79f0\u3002
loki sink\u7528\u4e8e\u53d1\u9001\u6570\u636e\u81f3Loki\u5b58\u50a8\u3002Loki\u6587\u6863\u53ef\u53c2\u8003\u8fd9\u91cc\u3002
Example
sink:\ntype: loki\nurl: \"http://localhost:3100/loki/api/v1/push\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 url string \u5fc5\u586b push loki\u7684api"},{"location":"reference/pipelines/sink/loki/#tenantid","title":"tenantId","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tenantId string \u975e\u5fc5\u586b \u53d1\u9001\u4f7f\u7528\u7684\u79df\u6237\u540d\u79f0"},{"location":"reference/pipelines/sink/loki/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 30s \u53d1\u9001\u7684\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/sink/loki/#entryline","title":"entryLine","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 entryLine string \u975e\u5fc5\u586b \u53d1\u9001\u81f3Loki\u7684\u65e5\u5fd7\u5185\u5bb9\uff0c\u9ed8\u8ba4\u4e3aloggie event\u7684body Loki\u7684\u65e5\u5fd7\u6570\u636e\u7ed3\u6784\u5927\u6982\u5206\u4e3alabel\u548c\u4e3b\u4f53\u6570\u636e\uff0cloggie\u4f1a\u9ed8\u8ba4\u5c06header\u91cc\u7684\u5143\u4fe1\u606f\u5b57\u6bb5\uff0c\u8f6c\u6210\u4ee5\u4e0b\u5212\u7ebf_\u8fde\u63a5\u7684label\u3002 \u53e6\u5916\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c\u7531\u4e8eloki\u7684labels key\u4e0d\u652f\u6301., /, -\uff0c\u8fd9\u91cc\u4f1a\u81ea\u52a8\u5c06header\u91cc\u5305\u542b\u8fd9\u4e9b\u7b26\u53f7\u7684key\u8f6c\u6210_\u7684\u5f62\u5f0f\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 insecureSkipVerify bool \u975e\u5fc5\u586b false \u662f\u5426\u5ffd\u7565\u8bc1\u4e66\u8ba4\u8bc1"},{"location":"reference/pipelines/sink/overview/","title":"Overview","text":"\u4e00\u4e2aPipeline\u5bf9\u5e94\u4e00\u4e2aSink\u3002
Concurrency\u76f8\u5173\u914d\u7f6e\u4f7f\u7528\u53ef\u53c2\u7167\u81ea\u9002\u5e94sink\u6d41\u91cf\u63a7\u5236\u3002
"},{"location":"reference/pipelines/sink/overview/#sink","title":"Sink\u901a\u7528\u914d\u7f6e","text":"Example
sink:\ntype: \"dev\"\ncodec:\ntype: json\npretty: true\nparallelism: 16\nconcurrency:\nenabled: true\nrtt:\nblockJudgeThreshold: 120%\nnewRttWeigh: 0.4\ngoroutine:\ninitThreshold: 8\nmaxGoroutine: 20\nunstableTolerate: 3\nchannelLenOfCap: 0.4\nratio:\nmulti: 2\nlinear: 2\nlinearWhenBlocked: 4\nduration:\nunstable: 15\nstable: 30\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 parallelism int \u975e\u5fc5\u586b 1 sink\u5ba2\u6237\u7aef\u7684\u5e76\u53d1\u5ea6\uff0c\u53ef\u540c\u65f6\u542f\u52a8\u591a\u4e2a\u589e\u5927\u53d1\u9001\u541e\u5410\u91cf\uff0c\u53ef\u8bbe\u7f6e\u7684\u6700\u5927\u503c\u4e3a100"},{"location":"reference/pipelines/sink/overview/#codec","title":"codec","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 codec \u975e\u5fc5\u586b sink\u53d1\u9001\u6570\u636e\u7ed9\u4e0b\u6e38\u65f6\uff0c\u6570\u636e\u4f7f\u7528\u7684\u683c\u5f0f codec.type string \u975e\u5fc5\u586b json codec\u7c7b\u578b"},{"location":"reference/pipelines/sink/overview/#type-json","title":"type: json","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 codec.pretty \u975e\u5fc5\u586b false \u662f\u5426\u8fdb\u884cjson\u683c\u5f0f\u7f8e\u5316 codec.beatsFormat \u975e\u5fc5\u586b false \u65e5\u5fd7\u8f6c\u6210\u7c7bfilebeats\u683c\u5f0f\uff1a\u589e\u52a0@timestamp\u5b57\u6bb5\uff0c\u540c\u65f6body\u5b57\u6bb5\u547d\u540d\u4e3amessage"},{"location":"reference/pipelines/sink/overview/#type-raw","title":"type: raw","text":"\u7528\u4e8e\u53d1\u9001\u91c7\u96c6\u7684\u539f\u59cbbody\u6570\u636e\u3002
Example
sink:\ntype: dev\ncodec:\ntype: raw\nExample
sink:\ntype: kafka\nconcurrency:\nenabled: true\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542fsink\u81ea\u9002\u5e94\u5e76\u53d1\u5ea6\u63a7\u5236 \u6ce8\uff1a\u9ed8\u8ba4\u6b64\u529f\u80fd\u4e0d\u5f00\u542f\u3002
"},{"location":"reference/pipelines/sink/overview/#concurrencygoroutine","title":"concurrency.goroutine","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 initThreshold int \u975e\u5fc5\u586b 16 \u521d\u59cb\u9608\u503c\uff0c\u4f4e\u4e8e\u8be5\u503c\uff0c\u534f\u7a0b\u6570\u6307\u6570\u589e\u957f\uff08\u5feb\u542f\u52a8\u9636\u6bb5\uff09\uff0c\u9ad8\u4e8e\u8be5\u503c\uff0c\u7ebf\u6027\u589e\u957f maxGoroutine int \u975e\u5fc5\u586b 30 \u6700\u5927\u534f\u7a0b\u6570 unstableTolerate int \u975e\u5fc5\u586b 3 \u8fdb\u5165\u5e73\u7a33\u9636\u6bb5\u540e\uff0c\u5bf9\u7f51\u7edc\u6ce2\u52a8\u7684\u5bb9\u5fcd\uff0c\u534f\u7a0b\u6570\u9700\u51cf\u5c11\u7684\u60c5\u51b5\u5305\u62ecrtt\u589e\u5927\uff0c\u8bf7\u6c42\u5931\u8d25\uff0c\u534f\u7a0b\u6570\u9700\u589e\u5927\u7684\u60c5\u51b5\u5305\u62ec\u7f51\u7edc\u5e73\u7a33\u4e14channel\u9971\u548c\uff0c\u76f8\u540c\u60c5\u51b5\u51fa\u73b03\u6b21\u624d\u4f1a\u89e6\u53d1\u534f\u7a0b\u6570\u6539\u53d8\uff0c\u82e5\u51fa\u73b0\u7b2c\u56db\u6b21\uff0c\u5c06\u4f1a\u8ffd\u52a0\uff0c\u76f4\u5230\u76f8\u53cd\u7684\u60c5\u51b5\u89e6\u53d1\u534f\u7a0b\u6570\u6539\u53d8\u3002 channelLenOfCap float \u975e\u5fc5\u586b 0.4 channel\u9971\u548c\u9608\u503c\uff0c\u8d85\u8fc7\u8be5\u503c\u8ba4\u4e3a\u534f\u7a0b\u6570\u9700\u589e\u5927\uff0c\u4ec5\u5728rtt\u7a33\u5b9a\u60c5\u51b5\u4e0b\u624d\u4f1a\u8ba1\u7b97\u8be5\u503c"},{"location":"reference/pipelines/sink/overview/#concurrencyrtt","title":"concurrency.rtt","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 blockJudgeThreshold string \u975e\u5fc5\u586b 120% \u5224\u65adrtt\u589e\u5927\u9608\u503c\uff0c\u5f53\u65b0rtt\u8d85\u8fc7\u5f53\u524d\u5e73\u5747rtt\u7684\u5230\u8fbe\u4e00\u5b9a\u7a0b\u5ea6\uff0c\u8ba4\u4e3a\u534f\u7a0b\u6570\u9700\u51cf\u5c11 newRttWeigh float \u975e\u5fc5\u586b 0.5 \u8ba1\u7b97\u65b0\u7684\u5e73\u5747rtt\u65f6\uff0c\u65b0rtt\u7684\u6743\u91cd \u6ce8\uff1ablockJudgeThreshold\uff08b\uff09\u652f\u6301\u767e\u5206\u6bd4\u548c\u6d6e\u70b9\u6570\u4e24\u79cd\u3002
\u82e5\u4e3a\u767e\u5206\u6bd4\uff0c\u5219\u5224\u65ad\u662f\u5426 (\u65b0rtt/\u5e73\u5747rtt)>b \u3002
\u82e5\u4e3a\u6d6e\u70b9\u6570\uff0c\u5219\u5224\u65ad\u662f\u5426 (\u65b0rtt-\u5e73\u5747rtt)>b \u3002
"},{"location":"reference/pipelines/sink/overview/#concurrencyratio","title":"concurrency.ratio","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 multi int \u975e\u5fc5\u586b 2 \u5feb\u542f\u52a8\u9636\u6bb5\uff0c\u534f\u7a0b\u6570\u6307\u6570\u589e\u957f\u901f\u7387 linear int \u975e\u5fc5\u586b 2 \uff08\u5feb\u542f\u52a8\u4e4b\u540e\uff09\u534f\u7a0b\u6570\u7ebf\u6027\u589e\u957f\u6216\u51cf\u5c11\u901f\u7387 linearWhenBlocked int \u975e\u5fc5\u586b 4 channel\u6ee1\u65f6(\u4e0a\u6e38\u963b\u585e)\uff0c\u534f\u7a0b\u6570\u7ebf\u6027\u589e\u957f\u901f\u7387"},{"location":"reference/pipelines/sink/overview/#concurrencyduration","title":"concurrency.duration","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 unstable int \u975e\u5fc5\u586b 15 \u975e\u5e73\u7a33\u9636\u6bb5\uff0c\u6536\u96c6\u6570\u636e\u8ba1\u7b97\u534f\u7a0b\u6570\u7684\u65f6\u95f4\u95f4\u9694\uff0c\u5355\u4f4d\u79d2 stable int \u975e\u5fc5\u586b 30 \u5e73\u7a33\u9636\u6bb5\uff0c\u6536\u96c6\u6570\u636e\u8ba1\u7b97\u534f\u7a0b\u6570\u7684\u65f6\u95f4\u95f4\u9694\uff0c\u5355\u4f4d\u79d2"},{"location":"reference/pipelines/sink/pulsar/","title":"pulsar","text":"pulsar sink\u7528\u4e8e\u53d1\u9001\u6570\u636e\u81f3pulsar\u5b58\u50a8\u3002 \u8be5sink\u4e3abeta\u8bd5\u7528\u72b6\u6001\uff0c\u8bf7\u8c28\u614e\u4f7f\u7528\u4e8e\u751f\u4ea7\u73af\u5883\u3002
Example
sink:\ntype: pulsar\nurl: pulsar://localhost:6650\ntopic: my-topic\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 url string \u5fc5\u586b \u65e0 \u65e5\u5fd7\u53d1\u9001\u7aefpulsar\u8fde\u63a5\u5730\u5740"},{"location":"reference/pipelines/sink/pulsar/#topic","title":"topic","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 topic string \u5fc5\u586b \u65e0 \u53d1\u9001\u65e5\u5fd7\u81f3pulsar\u7684topic"},{"location":"reference/pipelines/sink/pulsar/#producername","title":"producerName","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 producerName string \u975e\u5fc5\u586b \u65e0 specifies a name for the producer"},{"location":"reference/pipelines/sink/pulsar/#properties","title":"properties","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 producerName map \u975e\u5fc5\u586b \u65e0 Properties specifies a set of application defined properties for the producer"},{"location":"reference/pipelines/sink/pulsar/#operationtimeoutseconds","title":"operationTimeoutSeconds","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 operationTimeoutSeconds time.Duration \u975e\u5fc5\u586b 30s Producer-create, subscribe and unsubscribe operations will be retried until this interval, after which the operation will be marked as failed"},{"location":"reference/pipelines/sink/pulsar/#connectiontimeout","title":"connectionTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 connectionTimeout time.Duration \u975e\u5fc5\u586b 5s Timeout for the establishment of a TCP connection"},{"location":"reference/pipelines/sink/pulsar/#sendtimeout","title":"sendTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sendTimeout time.Duration \u975e\u5fc5\u586b 30s SendTimeout set the timeout for a message that is not acknowledged by the server 30s"},{"location":"reference/pipelines/sink/pulsar/#maxpendingmessages","title":"maxPendingMessages","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sendTimeout time.Duration \u975e\u5fc5\u586b \u65e0 MaxPendingMessages specifies the max size of the queue holding the messages pending to receive an acknowledgment from the broker"},{"location":"reference/pipelines/sink/pulsar/#hashingschema","title":"hashingSchema","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 hashingSchema int \u975e\u5fc5\u586b 0 HashingScheme is used to define the partition on where to publish a particular message. 0:JavaStringHash\uff0c1:Murmur3_32Hash"},{"location":"reference/pipelines/sink/pulsar/#compressiontype","title":"compressionType","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 compressionType int \u975e\u5fc5\u586b 0 0:NoCompression, 1:LZ4, 2:ZLIB, 3:ZSTD"},{"location":"reference/pipelines/sink/pulsar/#compressionlevel","title":"compressionLevel","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 compressionLevel int \u975e\u5fc5\u586b 0 0:Default, 1:Faster, 2:Better"},{"location":"reference/pipelines/sink/pulsar/#loglevel","title":"logLevel","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 logLevel string \u975e\u5fc5\u586b 0 \u65e5\u5fd7\u7ea7\u522b: \"info\",\"debug\", \"error\""},{"location":"reference/pipelines/sink/pulsar/#batchingmaxsize","title":"batchingMaxSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchingMaxSize int \u975e\u5fc5\u586b 2048(KB) BatchingMaxSize specifies the maximum number of bytes permitted in a batch"},{"location":"reference/pipelines/sink/pulsar/#batchingmaxmessages","title":"batchingMaxMessages","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchingMaxMessages int \u975e\u5fc5\u586b 1000 BatchingMaxMessages specifies the maximum number of messages permitted in a batch"},{"location":"reference/pipelines/sink/pulsar/#batchingmaxpublishdelay","title":"batchingMaxPublishDelay","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 batchingMaxPublishDelay time.Duration \u975e\u5fc5\u586b 10ms BatchingMaxPublishDelay specifies the time period within which the messages sent will be batched"},{"location":"reference/pipelines/sink/pulsar/#usetls","title":"useTLS","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 useTLS bool \u975e\u5fc5\u586b false \u662f\u5426\u4f7f\u7528TLS\u8ba4\u8bc1"},{"location":"reference/pipelines/sink/pulsar/#tlstrustcertsfilepath","title":"tlsTrustCertsFilePath","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tlsTrustCertsFilePath string \u975e\u5fc5\u586b \u65e0 the path to the trusted TLS certificate file"},{"location":"reference/pipelines/sink/pulsar/#tlsallowinsecureconnection","title":"tlsAllowInsecureConnection","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tlsAllowInsecureConnection bool \u975e\u5fc5\u586b false Configure whether the Pulsar client accept untrusted TLS certificate from broker"},{"location":"reference/pipelines/sink/pulsar/#certificatepath","title":"certificatePath","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 certificatePath string \u975e\u5fc5\u586b \u65e0 TLS\u8bc1\u4e66\u8def\u5f84"},{"location":"reference/pipelines/sink/pulsar/#privatekeypath","title":"privateKeyPath","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 privateKeyPath string \u975e\u5fc5\u586b \u65e0 TLS privateKey\u8def\u5f84"},{"location":"reference/pipelines/sink/pulsar/#token","title":"token","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 token string \u975e\u5fc5\u586b \u65e0 \u5982\u679c\u4f7f\u7528token\u8ba4\u8bc1\u9274\u6743pulsar\uff0c\u8bf7\u586b\u5199\u6b64\u9879"},{"location":"reference/pipelines/sink/pulsar/#tokenfilepath","title":"tokenFilePath","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tokenFilePath string \u975e\u5fc5\u586b \u65e0 auth token from a file"},{"location":"reference/pipelines/sink/sls/","title":"sls","text":"sls sink\u7528\u4e8e\u5c06\u65e5\u5fd7\u53d1\u9001\u81f3\u963f\u91cc\u4e91\u53ef\u89c2\u6d4b\u7edf\u4e00\u5b58\u50a8SLS\u3002
Example
sink:\ntype: sls\nname: demo\nendpoint: cn-hangzhou.log.aliyuncs.com\naccessKeyId: ${id}\naccessKeySecret: ${secret}\nproject: test\nlogstore: test1\ntopic: myservice\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 endpoint string \u5fc5\u586b SLS\u5b58\u50a8\u7684\u8bbf\u95ee\u57df\u540d \u4f60\u53ef\u4ee5\u5728\u5177\u4f53project\u9875\u9762\u7684\u9879\u76ee\u6982\u89c8\u4e2d\u67e5\u770b\u5230\u3002
"},{"location":"reference/pipelines/sink/sls/#accesskeyid","title":"accessKeyId","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 accessKeyId string \u5fc5\u586b \u8bbf\u95ee\u7684accessKeyId\uff0c\u8bf7\u67e5\u770b\u963f\u91cc\u4e91\u8d26\u53f7\u7684\u8bbf\u95ee\u51ed\u8bc1\u7ba1\u7406 \u5efa\u8bae\u4f7f\u7528\u963f\u91cc\u4e91\u7684\u5b50\u8d26\u53f7\uff0c\u5b50\u8d26\u53f7\u9700\u8981\u6709\u5bf9\u5e94project\u3001logstore\u7684\u6743\u9650\u3002
"},{"location":"reference/pipelines/sink/sls/#accesskeysecret","title":"accessKeySecret","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 accessKeySecret string \u5fc5\u586b \u8bbf\u95ee\u7684accessKeySecret\uff0c\u8bf7\u67e5\u770b\u963f\u91cc\u4e91\u8d26\u53f7\u7684\u8bbf\u95ee\u51ed\u8bc1\u7ba1\u7406"},{"location":"reference/pipelines/sink/sls/#project","title":"project","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 project string \u5fc5\u586b SLS\u5b58\u50a8\u7684project\u540d\u79f0"},{"location":"reference/pipelines/sink/sls/#logstore","title":"logstore","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 logstore string \u5fc5\u586b SLS\u5b58\u50a8\u7684logstore\u540d\u79f0"},{"location":"reference/pipelines/sink/sls/#topic","title":"topic","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 topic string \u975e\u5fc5\u586b \u65e5\u5fd7\u4e3b\u9898\uff08Topic\uff09\u662f\u65e5\u5fd7\u670d\u52a1\u7684\u57fa\u7840\u7ba1\u7406\u5355\u5143\u3002\u60a8\u53ef\u5728\u91c7\u96c6\u65e5\u5fd7\u65f6\u6307\u5b9aTopic\uff0c\u7528\u4e8e\u533a\u5206\u65e5\u5fd7"},{"location":"reference/pipelines/sink/webhook/","title":"alertWebhook","text":"alertWebhook sink\u5c06\u65e5\u5fd7\u6570\u636e\u53d1\u9001\u81f3http\u63a5\u6536\u65b9\u3002 \u4f7f\u7528\u793a\u4f8b\u8bf7\u53c2\u8003\u65e5\u5fd7\u62a5\u8b66
Example
sink:\ntype: alertWebhook\naddr: http://localhost:8080/loggie\nheaders:\napi: test1\nlineLimit: 10\ntemplate: |\n******\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addr string \u975e\u5fc5\u586b \u53d1\u9001alert\u7684http\u5730\u5740\uff0c\u82e5\u4e3a\u7a7a\uff0c\u5219\u4e0d\u4f1a\u53d1\u9001 template string \u975e\u5fc5\u586b \u7528\u6765\u6e32\u67d3\u7684\u6a21\u677f timeout time.Duration \u975e\u5fc5\u586b 30s \u53d1\u9001alert\u7684http timeout headers map \u975e\u5fc5\u586b \u53d1\u9001alert\u7684http header method string \u975e\u5fc5\u586b POST \u53d1\u9001alert\u7684http method, \u5982\u679c\u4e0d\u586bput(\u4e0d\u533a\u5206\u5927\u5c0f\u5199)\uff0c\u90fd\u8ba4\u4e3a\u662fPOST lineLimit int \u975e\u5fc5\u586b 10 \u591a\u884c\u65e5\u5fd7\u91c7\u96c6\u60c5\u51b5\u4e0b\uff0c\u6bcf\u4e2aalert\u4e2d\u5305\u542b\u7684\u6700\u5927\u65e5\u5fd7\u884c\u6570"},{"location":"reference/pipelines/sink/zinc/","title":"zinc","text":"zinc sink\u7528\u4e8e\u53d1\u9001\u6570\u636e\u81f3zinc\u5b58\u50a8\u3002
Example
sink:\ntype: zinc\nhost: \"http://127.0.0.1:4080\"\nusername: admin\npassword: Complexpass#123\nindex: \"demo\" \u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 host string \u975e\u5fc5\u586b http://127.0.0.1:4080 zinc\u7684url\u5730\u5740"},{"location":"reference/pipelines/sink/zinc/#username","title":"username","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 username string \u975e\u5fc5\u586b \u53d1\u9001\u81f3zinc\u7684\u7528\u6237\u540d"},{"location":"reference/pipelines/sink/zinc/#password","title":"password","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 password string \u975e\u5fc5\u586b \u53d1\u9001\u81f3zinc\u7684\u5bc6\u7801"},{"location":"reference/pipelines/sink/zinc/#index","title":"index","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 index string \u975e\u5fc5\u586b default \u53d1\u9001\u81f3zinc\u7684index"},{"location":"reference/pipelines/sink/zinc/#skipsslverify","title":"skipSSLVerify","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 skipSSLVerify bool \u975e\u5fc5\u586b true \u662f\u5426\u5ffd\u7565SSL\u6821\u9a8c"},{"location":"reference/pipelines/source/dev/","title":"dev","text":"\u7528\u4e8e\u5f00\u53d1\u6216\u8005\u538b\u6d4b\u573a\u666f\u4e0b\uff0c\u81ea\u52a8\u751f\u6210\u6570\u636e\u3002
Example
sources:\n- type: dev\nname: benchmark\nqps: 100\nbyteSize: 1024\neventsTotal: 10000\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 qps int \u975e\u5fc5\u586b 1000 \u751f\u6210event\u7684QPS"},{"location":"reference/pipelines/source/dev/#bytesize","title":"byteSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 byteSize int \u975e\u5fc5\u586b 1024 \u5355\u6761event\u7684\u5b57\u8282\u6570"},{"location":"reference/pipelines/source/dev/#eventstotal","title":"eventsTotal","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 eventsTotal int \u975e\u5fc5\u586b -1\uff0c\u65e0\u9650\u4e2a \u53d1\u9001\u7684\u6240\u6709event\u603b\u6570\uff0c\u4e4b\u540e\u4f1a\u505c\u6b62\u53d1\u9001"},{"location":"reference/pipelines/source/elasticsearch/","title":"elasticsearch","text":"\u6d88\u8d39elasticsearch\u7684\u6570\u636e\u3002
Example
\u5e38\u89c4pipelines:\n- name: local\nsources:\n- type: elasticsearch\nname: elastic\nhosts: [\"localhost:9200\"]\nindices: [\"blog*\"]\nsize: 10 # data size per fetch\ninterval: 30s # pull data frequency\npipelines:\n- name: local\nsources:\n- type: elasticsearch\nname: elastic\nhosts:\n- \"localhost:9200\"\n- \"localhost:9201\"\nindices: [\"blog*\"]\nusername: \"bob\"\npassword: \"bob\"\nschema: \"\"\nsniff: false\ngzip: true\nincludeFields: # pull selected field\n- Title\n- Content\n- Author\nexcludeFields: # exclude selected field\n- Content\nquery: | # elastic query phrases\n{\n\"match\": {\"Title\": \"bob\"}\n}\nsize: 10 # data size per fetch\ninterval: 30s # pull data frequency\ntimeout: 5s # pull timeout\ndb: flushTimeout: 2s # persistent the elastic pull location frequency\ncleanInactiveTimeout: 24h # delete the db record after the time\ncleanScanInterval: 1h # check the expired db record frequency\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 hosts string\u6570\u7ec4 \u5fc5\u586b \u6d88\u8d39\u7684elasticsearch url\u5730\u5740"},{"location":"reference/pipelines/source/elasticsearch/#indices","title":"indices","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 index string\u6570\u7ec4 \u5fc5\u586b \u67e5\u8be2elasticsearch\u7684index\u540d\u79f0"},{"location":"reference/pipelines/source/elasticsearch/#username","title":"username","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 username string \u975e\u5fc5\u586b \u6d88\u8d39elasticsearch\u7684\u7528\u6237\u540d"},{"location":"reference/pipelines/source/elasticsearch/#password","title":"password","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 password string \u5fc5\u586b \u6d88\u8d39elasticsearch\u7684\u5bc6\u7801"},{"location":"reference/pipelines/source/elasticsearch/#schema","title":"schema","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 schema string \u975e\u5fc5\u586b http HTTP scheme(http/https)\uff0csniff\u7684\u65f6\u5019\u4f7f\u7528"},{"location":"reference/pipelines/source/elasticsearch/#gzip","title":"gzip","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 gzip bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542fgzip\u538b\u7f29"},{"location":"reference/pipelines/source/elasticsearch/#includefields","title":"includeFields","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 includeFields string\u6570\u7ec4 \u975e\u5fc5\u586b \u53ea\u8fd4\u56de\u6307\u5b9a\u7684_source\u5b57\u6bb5"},{"location":"reference/pipelines/source/elasticsearch/#excludefields","title":"excludeFields","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 excludeFields string\u6570\u7ec4 \u975e\u5fc5\u586b \u6392\u9664\u6307\u5b9a\u7684_source\u5b57\u6bb5"},{"location":"reference/pipelines/source/elasticsearch/#query","title":"query","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 query string \u975e\u5fc5\u586b \u67e5\u8be2elasticsearch\u7684\u8868\u8fbe\u5f0f"},{"location":"reference/pipelines/source/elasticsearch/#size","title":"size","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 size int \u975e\u5fc5\u586b 100 \u6bcf\u6b21\u8bf7\u6c42\u5f97\u5230hits\u8fd4\u56de\u7684\u4e2a\u6570"},{"location":"reference/pipelines/source/elasticsearch/#interval","title":"interval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 interval time.Duration \u975e\u5fc5\u586b 30s \u5b9a\u65f6\u8bf7\u6c42elasticsearch\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/pipelines/source/elasticsearch/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 5s \u8bf7\u6c42\u7684\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/source/elasticsearch/#db","title":"db","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 db \u975e\u5fc5\u586b \u6301\u4e45\u5316\u8bb0\u5f55\u67e5\u8be2elasticsearch\u8bf7\u6c42\u7684\u8fdb\u5ea6\uff0c\u4f1a\u5b58\u50a8\u81f3elasticsearch\u4e2d\uff0c\u907f\u514dLoggie\u91cd\u542f\u540e\u91cd\u590d\u6d88\u8d39\u6570\u636e db.indexPrefix string \u975e\u5fc5\u586b .loggie-db \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0cloggie\u4f1a\u5c06\u6301\u4e45\u5316\u7684\u6570\u636e\u5b9a\u65f6\u5199\u5165\u683c\u5f0f\u4e3a${indexPrefix}-${pipelineName}-${sourceName}\u7684index\u4e2d db.flushTimeout time.Duration \u975e\u5fc5\u586b 2s \u6301\u4e45\u5316\u6570\u636e\u5199\u5165\u7684\u95f4\u9694\u65f6\u95f4 db.cleanInactiveTimeout time.Duration \u975e\u5fc5\u586b 504h (21day) \u6e05\u7406\u8fc7\u671f\u7684\u6301\u4e45\u5316\u6570\u636e\u8d85\u65f6\u65f6\u95f4 db.cleanScanInterval time.Duration \u975e\u5fc5\u586b 1h \u68c0\u67e5\u8fc7\u671f\u65f6\u95f4\u95f4\u9694"},{"location":"reference/pipelines/source/file/","title":"file","text":"file source\u7528\u4e8e\u65e5\u5fd7\u91c7\u96c6\u3002
Example
sources:\n- type: file\nname: accesslog\nTips
\u5982\u679c\u4f60\u4f7f\u7528logconfig/clusterlogconfig\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0cfile source\u91cc\u8fd8\u589e\u52a0\u4e86\u989d\u5916\u7684\u5b57\u6bb5\uff0c\u8bf7\u53c2\u8003\u8fd9\u91cc\u3002
"},{"location":"reference/pipelines/source/file/#paths","title":"paths","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 paths string\u6570\u7ec4 \u5fc5\u586b \u65e0 \u91c7\u96c6\u7684path\u8def\u5f84\uff0c\u4f7f\u7528glob\u8868\u8fbe\u5f0f\u6765\u5339\u914d\u3002\u652f\u6301glob\u6269\u5c55\u8868\u8fbe\u5f0fBrace Expansion\u548cGlob Star Example
\u9700\u8981\u91c7\u96c6\u7684\u76ee\u6807\u6587\u4ef6\uff1a
/tmp/loggie/service/order/access.log\n/tmp/loggie/service/order/access.log.2022-04-11\n/tmp/loggie/service/pay/access.log\n/tmp/loggie/service/pay/access.log.2022-04-11\n\u5bf9\u5e94\u914d\u7f6e\uff1a
sources:\n- type: file\npaths:\n- /tmp/loggie/**/access.log{,.[2-9][0-9][0-9][0-9]-[01][0-9]-[0123][0-9]}\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 excludeFiles string\u6570\u7ec4 \u975e\u5fc5\u586b \u65e0 \u6392\u9664\u91c7\u96c6\u7684\u6587\u4ef6\u6b63\u5219\u8868\u8fbe\u5f0f Example
sources:\n- type: file\npaths:\n- /tmp/*.log\nexcludeFiles:\n- \\.gz$\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 ignoreOlder time.Duration \u975e\u5fc5\u586b \u65e0 \u4f8b\u598248h\uff0c\u8868\u793a\u5ffd\u7565\u66f4\u65b0\u65f6\u95f4\u57282\u5929\u4e4b\u524d\u7684\u6587\u4ef6\uff0c\u65e0\u9700\u8fdb\u884c\u91c7\u96c6"},{"location":"reference/pipelines/source/file/#ignoresymlink","title":"ignoreSymlink","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 ignoreSymlink bool \u975e\u5fc5\u586b false \u662f\u5426\u5ffd\u7565\u7b26\u53f7\u94fe\u63a5\uff08\u8f6f\u94fe\u63a5\uff09\u7684\u6587\u4ef6"},{"location":"reference/pipelines/source/file/#addonmeta","title":"addonMeta","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 addonMeta bool \u975e\u5fc5\u586b false \u662f\u5426\u6dfb\u52a0\u9ed8\u8ba4\u7684\u65e5\u5fd7\u91c7\u96c6state\u5143\u4fe1\u606f event\u793a\u4f8b
{\n\"body\": \"this is test\",\n\"state\": {\n\"pipeline\": \"local\",\n\"source\": \"demo\",\n\"filename\": \"/var/log/a.log\",\n\"timestamp\": \"2006-01-02T15:04:05.000Z\",\n\"offset\": 1024,\n\"bytes\": 4096,\n\"hostname\": \"node-1\"\n}\n}\nstate\u542b\u4e49\u89e3\u91ca\uff1a
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 workerCount int \u975e\u5fc5\u586b 1 \u8bfb\u53d6\u6587\u4ef6\u5185\u5bb9\u7684\u5de5\u4f5c\u7ebf\u7a0b(goroutine)\u6570\u3002\u5355\u8282\u70b9\u8d85\u8fc7100\u4e2a\u6587\u4ef6\u7684\u65f6\u5019\u8003\u8651\u63d0\u9ad8"},{"location":"reference/pipelines/source/file/#readbuffersize","title":"readBufferSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readBufferSize int \u975e\u5fc5\u586b 65536 \u5355\u6b21\u8bfb\u53d6\u6587\u4ef6\u7684\u6570\u636e\u91cf\u3002\u9ed8\u8ba464K=65536"},{"location":"reference/pipelines/source/file/#maxcontinueread","title":"maxContinueRead","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxContinueRead int \u975e\u5fc5\u586b 16 \u8fde\u7eed\u8bfb\u53d6\u540c\u4e00\u4e2a\u6587\u4ef6\u5185\u5bb9\u7684\u6b21\u6570\uff0c\u8fbe\u5230\u8fd9\u4e2a\u6b21\u6570\u5c06\u5f3a\u5236\u5207\u6362\u5230\u4e0b\u4e2a\u6587\u4ef6\u8bfb\u53d6\u3002\u4e3b\u8981\u4f5c\u7528\u662f\u7528\u6765\u907f\u514d\u6d3b\u8dc3\u6587\u4ef6\u4e00\u76f4\u5360\u636e\u8bfb\u53d6\u8d44\u6e90\uff0c\u975e\u6d3b\u8dc3\u6587\u4ef6\u957f\u65f6\u95f4\u5f97\u4e0d\u5230\u8bfb\u53d6\u91c7\u96c6"},{"location":"reference/pipelines/source/file/#maxcontinuereadtimeout","title":"maxContinueReadTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxContinueReadTimeout time.Duration \u975e\u5fc5\u586b 3s \u540c\u4e00\u4e2a\u6587\u4ef6\u6700\u957f\u8bfb\u53d6\u65f6\u95f4\uff0c\u8d85\u8fc7\u8fd9\u4e2a\u65f6\u95f4\u5c06\u5f3a\u5236\u5207\u6362\u4e0b\u4e2a\u6587\u4ef6\u8bfb\u53d6\u3002\u4f5c\u7528\u4e0emaxContinueRead\u7c7b\u4f3c"},{"location":"reference/pipelines/source/file/#inactivetimeout","title":"inactiveTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 inactiveTimeout time.Duration \u975e\u5fc5\u586b 3s \u5982\u679c\u5f53\u6587\u4ef6\u4ece\u4e0a\u4e00\u6b21\u91c7\u96c6\u5230\u73b0\u5728\u8d85\u8fc7inactiveTimeout\u7684\u8bdd\uff0c\u5219\u8ba4\u4e3a\u6587\u4ef6\u8fdb\u5165\u4e0d\u6d3b\u8dc3\u72b6\u6001(\u5373\u6700\u540e\u4e00\u6761\u65e5\u5fd7\u5df2\u7ecf\u5199\u5165\u5b8c\u6210)\uff0c\u5219\u53ef\u4ee5\u5b89\u5168\u7684\u91c7\u96c6\u6700\u540e\u4e00\u884c\u65e5\u5fd7"},{"location":"reference/pipelines/source/file/#firstnbytesforidentifier","title":"firstNBytesForIdentifier","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 firstNBytesForIdentifier int \u975e\u5fc5\u586b 128 \u4f7f\u7528\u91c7\u96c6\u76ee\u6807\u6587\u4ef6\u7684\u524dn\u4e2a\u5b57\u7b26\u6765\u751f\u6210\u6587\u4ef6\u552f\u4e00code\u3002\u5982\u679c\u6587\u4ef6\u7684\u5927\u5c0f\u5c0f\u4e8en\uff0c\u5219\u8be5\u6587\u4ef6\u6682\u65f6\u4e0d\u4f1a\u91c7\u96c6\u3002\u7528\u9014\u4e3b\u8981\u662f\uff0c\u7ed3\u5408\u6587\u4ef6inode\u4fe1\u606f\uff0c\u7528\u6765\u7cbe\u786e\u6807\u8bc6\u4e00\u4e2a\u6587\u4ef6\u3002\u8f85\u52a9\u5224\u65ad\u6587\u4ef6\u662f\u5426\u5220\u9664\u6216\u8005\u662f\u6539\u540d"},{"location":"reference/pipelines/source/file/#charset","title":"charset","text":"\u7f16\u7801\u8f6c\u6362\uff0c\u7528\u4e8e\u5c06\u4e0d\u540c\u7684\u7f16\u7801\u8f6c\u6362\u4e3autf8\uff0c\u5f53\u4e0b\u652f\u6301\u7684\u7f16\u7801\u8f6c\u6362\u683c\u5f0f.
Example
sources:\n- type: file\nname: demo\npaths:\n- /tmp/log/*.log\nfields:\ntopic: \"loggie\"\ncharset: \"gbk\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 charset string \u5426 utf-8 \u63d0\u53d6\u5b57\u6bb5\u7684\u5339\u914d\u6a21\u578b \u5f53\u524d\u652f\u6301\u7684\u8f6c\u6362\u4e3autf-8\u7684\u7f16\u7801\u683c\u5f0f\u6709
nopplainutf-8gbkbig5euc-jpiso2022-jpshift-jiseuc-kriso8859-6eiso8859-6iiso8859-8eiso8859-8iiso8859-1iso8859-2iso8859-3iso8859-4iso8859-5iso8859-6iso8859-7iso8859-8iso8859-9iso8859-10iso8859-13iso8859-14iso8859-15iso8859-16cp437cp850cp852cp855cp858cp860cp862cp863cp865cp866ebcdic-037ebcdic-1040ebcdic-1047koi8rkoi8umacintoshmacintosh-cyrillicwindows1250windows1251windows1252windows1253windows1254windows1255windows1256windows1257windows1258windows874utf-16be-bomutf-16le-bom\u6362\u884c\u7b26\u76f8\u5173\u914d\u7f6e
Example
sources:\n- type: file\nname: demo\nlineDelimiter:\ntype: carriage_return_line_feed\nvalue: \"\\r\\n\"\ncharset: gbk\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 type bool \u975e\u5fc5\u586b auto \u53ea\u6709\u5728type\u662fcustome\u65f6\u5019value\u624d\u4f1a\u6709\u6548 \u5f53\u524d\u652f\u6301\u7684type\u6709
autoline_feedvertical_tabform_feedcarriage_returncarriage_return_line_feednext_lineline_separatorparagraph_separatornull_terminator\u5bf9\u5e94\u7684\u6362\u884c\u7b26\u4e3a\uff1a
```\n auto: {'\\u000A'},\n line_feed: {'\\u000A'},\n vertical_tab: {'\\u000B'},\n form_feed: {'\\u000C'},\n carriage_return: {'\\u000D'},\n carriage_return_line_feed: []byte(\"\\u000D\\u000A\"),\n next_line: {'\\u0085'},\n line_separator: []byte(\"\\u2028\"),\n paragraph_separator: []byte(\"\\u2029\"),\n null_terminator: {'\\u0000'},\n ```\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 value string \u975e\u5fc5\u586b \\n \u6362\u884c\u7b26\u7684\u5185\u5bb9"},{"location":"reference/pipelines/source/file/#charset_1","title":"charset","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 charset string \u975e\u5fc5\u586b utf-8 \u6362\u884c\u7b26\u7f16\u7801"},{"location":"reference/pipelines/source/file/#multi","title":"multi","text":"\u591a\u884c\u91c7\u96c6\u76f8\u5173\u914d\u7f6e
Example
sources:\n- type: file\nname: accesslog\nmulti:\nactive: true\npattern: '^\\d{4}-\\d{2}-\\d{2}'\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 active bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542f\u591a\u884c\u91c7\u96c6\u6a21\u5f0f"},{"location":"reference/pipelines/source/file/#pattern","title":"pattern","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 pattern string \u5f53multi.active=true\u7684\u65f6\u5019\u5fc5\u586b false \u5224\u65ad\u4e3a\u4e00\u6761\u5168\u65b0\u65e5\u5fd7\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\u3002\u4f8b\u5982\u914d\u7f6e\u4e3a'^\\['\uff0c\u5219\u8ba4\u4e3a\u884c\u9996\u4ee5[\u5f00\u5934\u624d\u662f\u4e00\u6761\u65b0\u65e5\u5fd7\uff0c\u5426\u5219\u5c06\u8fd9\u884c\u5185\u5bb9\u5408\u5165\u4e0a\u4e00\u6761\u65e5\u5fd7\u4f5c\u4e3a\u4e0a\u4e00\u6761\u65e5\u5fd7\u7684\u4e00\u90e8\u5206 Example
\u5047\u8bbe\u6709\u591a\u884c\u65e5\u5fd7\u5982\u4e0b\u6240\u793a\uff1a
2023-05-11 14:30:15 ERROR Exception in thread \"main\" java.lang.NullPointerException\n at com.example.MyClass.myMethod(MyClass.java:25)\n at com.example.MyClass.main(MyClass.java:10)\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxLines int \u975e\u5fc5\u586b 500 1\u6761\u65e5\u5fd7\u6700\u591a\u5305\u542b\u51e0\u884c\u5185\u5bb9\u3002\u9ed8\u8ba4500\u884c\uff0c\u8d85\u8fc7\u4e0a\u9650\u5c06\u5f3a\u5236\u53d1\u9001\u5f53\u524d\u65e5\u5fd7\uff0c\u8d85\u51fa\u90e8\u5206\u4f5c\u4e3a\u65b0\u7684\u4e00\u6761\u65e5\u5fd7"},{"location":"reference/pipelines/source/file/#maxbytes","title":"maxBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxBytes int64 \u975e\u5fc5\u586b 131072 1\u6761\u65e5\u5fd7\u6700\u591a\u5305\u542b\u51e0\u4e2a\u5b57\u8282\u3002\u9ed8\u8ba4128K\uff0c\u8d85\u8fc7\u4e0a\u9650\u5c06\u5f3a\u5236\u53d1\u9001\u5f53\u524d\u65e5\u5fd7\uff0c\u8d85\u51fa\u90e8\u5206\u4f5c\u4e3a\u65b0\u7684\u4e00\u6761\u65e5\u5fd7"},{"location":"reference/pipelines/source/file/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 5s 1\u6761\u65e5\u5fd7\u6700\u591a\u7b49\u5f85\u591a\u4e45\u91c7\u96c6\u4e3a\u5b8c\u6574\u76841\u6761\u65e5\u5fd7\u3002\u9ed8\u8ba45s\uff0c\u8d85\u8fc7\u4e0a\u9650\u5c06\u5f3a\u5236\u53d1\u9001\u5f53\u524d\u65e5\u5fd7\uff0c\u8d85\u51fa\u90e8\u5206\u4f5c\u4e3a\u65b0\u7684\u4e00\u6761\u65e5\u5fd7"},{"location":"reference/pipelines/source/file/#ack","title":"ack","text":"source\u7684\u786e\u8ba4\u673a\u5236\u76f8\u5173\u914d\u7f6e\u3002\u5982\u679c\u9700\u786e\u4fdd\u8981at least once\uff0c\u9700\u8981\u5f00\u542fack\u673a\u5236\uff0c\u4f46\u662f\u4f1a\u6709\u4e00\u5b9a\u6027\u80fd\u987a\u8017
Caution
\u8be5\u914d\u7f6e\u53ea\u80fd\u914d\u7f6e\u5728defaults\u4e2d
Example
defaults:\nsources:\n- type: file\nack:\nenable: true\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enable bool \u975e\u5fc5\u586b true \u662f\u5426\u5f00\u542f\u786e\u8ba4\u673a\u5236"},{"location":"reference/pipelines/source/file/#maintenanceinterval","title":"maintenanceInterval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maintenanceInterval time.Duration \u975e\u5fc5\u586b 20h \u7ef4\u62a4\u5468\u671f\u3002\u7528\u6765\u5b9a\u65f6\u6e05\u7406\u8fc7\u671f\u7684\u786e\u8ba4\u6587\u4ef6\u6570\u636e(\u4f8b\u5982\u4e0d\u518d\u91c7\u96c6\u7684\u6587\u4ef6\u7684ack\u4fe1\u606f)"},{"location":"reference/pipelines/source/file/#db","title":"db","text":"\u4f7f\u7528sqlite3\u4f5c\u4e3a\u6570\u636e\u5e93\u3002\u4fdd\u5b58\u91c7\u96c6\u8fc7\u7a0b\u4e2d\u7684\u6587\u4ef6\u540d\u79f0\u3001\u6587\u4ef6inode\u3001\u6587\u4ef6\u91c7\u96c6\u7684offset\u7b49\u4fe1\u606f\u3002\u7528\u6765\u5728loggie reload\u6216\u8005\u91cd\u542f\u540e\u6062\u590d\u4e0a\u4e00\u6b21\u7684\u91c7\u96c6\u8fdb\u5ea6
Caution
\u8be5\u914d\u7f6e\u53ea\u80fd\u914d\u7f6e\u5728defaults\u4e2d
Example
defaults:\nsources:\n- type: file\ndb:\nfile: \"./data/loggie.db\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 file string \u975e\u5fc5\u586b ./data/loggie.db \u6570\u636e\u5e93\u6587\u4ef6\u8def\u5f84"},{"location":"reference/pipelines/source/file/#tablename","title":"tableName","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 tableName string \u975e\u5fc5\u586b registry \u6570\u636e\u5e93\u8868\u540d\u79f0"},{"location":"reference/pipelines/source/file/#flushtimeout","title":"flushTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 flushTimeout time.Duration \u975e\u5fc5\u586b 2s \u5b9a\u65f6\u5c06\u91c7\u96c6\u4fe1\u606f\u5199\u5165\u5230\u6570\u636e\u5e93"},{"location":"reference/pipelines/source/file/#buffersize","title":"bufferSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 bufferSize int \u975e\u5fc5\u586b 2048 \u8f93\u5165\u6570\u636e\u5e93\u7684\u91c7\u96c6\u4fe1\u606f\u7684\u7f13\u51b2\u533a\u5927\u5c0f"},{"location":"reference/pipelines/source/file/#cleaninactivetimeout","title":"cleanInactiveTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cleanInactiveTimeout time.Duration \u975e\u5fc5\u586b 504h \u6e05\u7406\u6570\u636e\u5e93\u4e2d\u7684\u8fc7\u671f\u6570\u636e\u3002\u5982\u679c\u6570\u636e\u7684\u66f4\u65b0\u65f6\u95f4\u8d85\u8fc7\u914d\u7f6e\u503c\uff0c\u5c06\u4f1a\u5220\u9664\u8be5\u6761\u6570\u636e\u3002\u9ed8\u8ba4\u4fdd\u755921\u5929"},{"location":"reference/pipelines/source/file/#cleanscaninterval","title":"cleanScanInterval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cleanScanInterval time.Duration \u975e\u5fc5\u586b 1h \u5468\u671f\u6027\u7684\u68c0\u67e5\u6570\u636e\u5e93\u4e2d\u7684\u8fc7\u671f\u6570\u636e\u3002\u9ed8\u8ba4\u6bcf\u96941\u5c0f\u65f6\u68c0\u67e5\u4e00\u6b21"},{"location":"reference/pipelines/source/file/#watcher","title":"watcher","text":"\u76d1\u63a7\u6587\u4ef6\u53d8\u5316\u7684\u76f8\u5173\u914d\u7f6e
Caution
\u8be5\u914d\u7f6e\u53ea\u80fd\u914d\u7f6e\u5728defaults\u4e2d
Example
defaults:\nsources:\n- type: file\nwatcher:\nenableOsWatch: true\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enableOsWatch bool \u975e\u5fc5\u586b true \u662f\u5426\u542f\u7528OS\u7684\u76d1\u63a7\u901a\u77e5\u673a\u5236\u3002\u4f8b\u5982linux\u7684inotify\u6307\u4ee4"},{"location":"reference/pipelines/source/file/#scantimeinterval","title":"scanTimeInterval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 scanTimeInterval time.Duration \u975e\u5fc5\u586b 10s \u5468\u671f\u6027\u7684\u68c0\u67e5\u6587\u4ef6\u7684\u72b6\u6001\u53d8\u66f4\uff08\u4f8b\u5982\u6587\u4ef6\u7684\u65b0\u5efa\u3001\u5220\u9664\u7b49\uff09\u3002\u9ed8\u8ba4\u6bcf\u969410s\u68c0\u67e5\u4e00\u6b21"},{"location":"reference/pipelines/source/file/#maintenanceinterval_1","title":"maintenanceInterval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maintenanceInterval time.Duration \u975e\u5fc5\u586b 5m \u5468\u671f\u6027\u7684\u7ef4\u62a4\u5de5\u4f5c\uff08\u4f8b\u5982\u4e0a\u62a5\u91c7\u96c6\u7edf\u8ba1\u4fe1\u606f\u3001\u6e05\u7406\u6587\u4ef6\u7b49\uff09"},{"location":"reference/pipelines/source/file/#fdholdtimeoutwheninactive","title":"fdHoldTimeoutWhenInactive","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fdHoldTimeoutWhenInactive time.Duration \u975e\u5fc5\u586b 5m \u5f53\u6587\u4ef6\u7684\u4e0a\u6b21\u91c7\u96c6\u5230\u73b0\u5728\u7684\u65f6\u95f4\u8d85\u8fc7\u9650\u5236\uff08\u6587\u4ef6\u957f\u65f6\u95f4\u6ca1\u6709\u5199\u5165\uff0c\u8ba4\u4e3a\u5927\u6982\u7387\u4e0d\u4f1a\u518d\u5199\u5165\u5185\u5bb9\uff09\uff0c\u5c06\u4f1a\u91ca\u653e\u8be5\u6587\u4ef6\u7684\u6587\u4ef6\u53e5\u67c4\u4ee5\u91ca\u653e\u7cfb\u7edf\u8d44\u6e90"},{"location":"reference/pipelines/source/file/#fdholdtimeoutwhenremove","title":"fdHoldTimeoutWhenRemove","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fdHoldTimeoutWhenRemove time.Duration \u975e\u5fc5\u586b 5m \u5f53\u6587\u4ef6\u88ab\u5220\u9664\u4e14\u672a\u91c7\u96c6\u5b8c\u6210\uff0c\u4f1a\u7b49\u5f85\u7684\u6700\u5927\u65f6\u95f4\u6765\u91c7\u96c6\u5b8c\u6210\u3002\u8d85\u8fc7\u9650\u5236\u4e0d\u7ba1\u6587\u4ef6\u6700\u7ec8\u662f\u5426\u91c7\u96c6\u5b8c\u6210\uff0c\u90fd\u4f1a\u76f4\u63a5\u91ca\u653e\u6587\u4ef6\u53e5\u67c4\u4e0d\u518d\u91c7\u96c6"},{"location":"reference/pipelines/source/file/#maxopenfds","title":"maxOpenFds","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxOpenFds int \u975e\u5fc5\u586b 1024 \u6700\u5927\u6253\u5f00\u7684\u6587\u4ef6\u53e5\u67c4\u6570\u91cf\uff0c\u8d85\u51fa\u540e\u7684\u6587\u4ef6\u5c06\u6682\u65f6\u4e0d\u4f1a\u91c7\u96c6"},{"location":"reference/pipelines/source/file/#maxeofcount","title":"maxEofCount","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxEofCount int \u975e\u5fc5\u586b 3 \u6700\u5927\u8fde\u7eed\u8bfb\u53d6\u6587\u4ef6\u9047\u5230eof\u7684\u6b21\u6570\u3002\u8d85\u8fc7\u9650\u5236\u8ba4\u4e3a\u6587\u4ef6\u6682\u65f6\u4e0d\u6d3b\u8dc3\uff0c\u5c06\u8fdb\u5165\u201c\u50f5\u5c38\u201d\u961f\u5217\u7b49\u5f85\u66f4\u65b0\u4e8b\u4ef6\u88ab\u6fc0\u6d3b"},{"location":"reference/pipelines/source/file/#cleanwhenremoved","title":"cleanWhenRemoved","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 cleanWhenRemoved bool \u975e\u5fc5\u586b true \u5f53\u6587\u4ef6\u88ab\u5220\u9664\u540e\uff0c\u662f\u5426\u540c\u6b65\u5220\u9664db\u4e2d\u7684\u91c7\u96c6\u76f8\u5173\u4fe1\u606f"},{"location":"reference/pipelines/source/file/#readfromtail","title":"readFromTail","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readFromTail bool \u975e\u5fc5\u586b false \u662f\u5426\u4ece\u6587\u4ef6\u7684\u6700\u65b0\u4e00\u884c\u5f00\u59cb\u91c7\u96c6\uff0c\u800c\u4e0d\u7ba1\u5386\u53f2\u5199\u5165\u5230\u6587\u4ef6\u7684\u5185\u5bb9\u3002\u9002\u7528\u4e8e\u91c7\u96c6\u7cfb\u7edf\u7684\u8fc1\u79fb\u7b49\u573a\u666f"},{"location":"reference/pipelines/source/file/#taskstoptimeout","title":"taskStopTimeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 taskStopTimeout time.Duration \u975e\u5fc5\u586b 30s \u91c7\u96c6\u4efb\u52a1\u9000\u51fa\u7684\u8d85\u65f6\u65f6\u95f4\u3002\u662f\u4e00\u4e2a\u515c\u5e95\u65b9\u6848\uff0c\u653e\u5728\u91c7\u96c6\u4efb\u52a1\u5047\u6b7b\u5bfc\u81f4\u65e0\u6cd5reload"},{"location":"reference/pipelines/source/file/#cleanfiles","title":"cleanFiles","text":"\u6e05\u7406\u6587\u4ef6\u76f8\u5173\u914d\u7f6e\u3002\u8fc7\u671f\u4e14\u5df2\u7ecf\u91c7\u96c6\u5b8c\u6210\u7684\u6587\u4ef6\u5c06\u4f1a\u76f4\u63a5\u4ece\u78c1\u76d8\u5220\u9664\u4ee5\u91ca\u653e\u78c1\u76d8\u7a7a\u95f4
"},{"location":"reference/pipelines/source/file/#maxhistorydays","title":"maxHistoryDays","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxHistoryDays int \u975e\u5fc5\u586b \u65e0 \uff08\u91c7\u96c6\u5b8c\u6210\u540e\u7684\uff09\u6587\u4ef6\u6700\u591a\u4fdd\u7559\u7684\u5929\u6570\u3002\u5982\u679c\u8d85\u51fa\u9650\u5236\uff0c\u5c06\u4f1a\u628a\u6587\u4ef6\u76f4\u63a5\u4ece\u78c1\u76d8\u4e2d\u5220\u9664\u3002\u4e0d\u914d\u7f6e\u5219\u6c38\u8fdc\u4e0d\u4f1a\u5220\u9664\u6587\u4ef6"},{"location":"reference/pipelines/source/grpc/","title":"grpc","text":"Grpc source\u7528\u4e8e\u63a5\u6536Loggie Grpc\u683c\u5f0f\u7684\u6570\u636e\u8bf7\u6c42\u3002 \u4e00\u822c\u7528\u5728\u4e2d\u8f6c\u673a\u573a\u666f\uff0c\u63a5\u6536\u5176\u4ed6Loggie\u96c6\u7fa4\u53d1\u9001\u7684\u65e5\u5fd7\u3002
Example
sources:\n- type: grpc\nname: aggre\nport: 6066\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 bind string \u975e\u5fc5\u586b 0.0.0.0 \u63d0\u4f9bserver\u7ed1\u5b9a\u7684host"},{"location":"reference/pipelines/source/grpc/#port","title":"port","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 port string \u975e\u5fc5\u586b 6066 \u63d0\u4f9b\u670d\u52a1\u7684\u7aef\u53e3\u53f7"},{"location":"reference/pipelines/source/grpc/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 20s \u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/source/kafka/","title":"kafka","text":"Kafka source\u7528\u4e8e\u63a5\u6536Kafka\u6570\u636e\u3002
Example
sources:\n- type: kafka\nbrokers: [\"kafka1.kafka.svc:9092\"]\ntopic: log-*\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 brokers string\u6570\u7ec4 \u5fc5\u586b \u65e0 Kafka broker\u5730\u5740"},{"location":"reference/pipelines/source/kafka/#topic","title":"topic","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 topic string \u5fc5\u586b \u65e0 \u63a5\u6536\u7684topics\uff0c\u53ef\u4f7f\u7528\u6b63\u5219\u6765\u5339\u914d\u591a\u4e2atopic"},{"location":"reference/pipelines/source/kafka/#groupid","title":"groupId","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 groupId string \u975e\u5fc5\u586b loggie Loggie\u6d88\u8d39kafka\u7684groupId"},{"location":"reference/pipelines/source/kafka/#queuecapacity","title":"queueCapacity","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 queueCapacity int \u975e\u5fc5\u586b 100 \u5185\u90e8\u53d1\u9001\u7684\u961f\u5217\u5bb9\u91cf"},{"location":"reference/pipelines/source/kafka/#minacceptedbytes","title":"minAcceptedBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 minAcceptedBytes int \u975e\u5fc5\u586b 1 \u6700\u5c0f\u63a5\u6536\u7684batch\u5b57\u8282\u6570"},{"location":"reference/pipelines/source/kafka/#maxacceptedbytes","title":"maxAcceptedBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxAcceptedBytes int \u975e\u5fc5\u586b 1e6\uff081MB) \u6700\u5927\u63a5\u6536\u7684\u6d88\u606f\u5b57\u8282\u6570\uff0c\u5982\u679c\u8d85\u8fc7\u4f1a\u88abtruncate\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u4e3a\u4e00\u4e2a\u80fd\u5bb9\u5fcd\u7684\u8f83\u5927\u7684\u503c"},{"location":"reference/pipelines/source/kafka/#readmaxattempts","title":"readMaxAttempts","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readMaxAttempts int \u975e\u5fc5\u586b 3 \u6700\u5927\u7684\u91cd\u8bd5\u6b21\u6570"},{"location":"reference/pipelines/source/kafka/#maxpollwait","title":"maxPollWait","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxPollWait time.Duration \u975e\u5fc5\u586b 10s \u63a5\u6536\u7684\u6700\u957f\u7b49\u5f85\u65f6\u95f4"},{"location":"reference/pipelines/source/kafka/#readbackoffmin","title":"readBackoffMin","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readBackoffMin time.Duration \u975e\u5fc5\u586b 100ms \u5728\u63a5\u6536\u65b0\u7684\u6d88\u606f\u524d\uff0c\u6700\u5c0f\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/pipelines/source/kafka/#readbackoffmax","title":"readBackoffMax","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 readBackoffMax time.Duration \u975e\u5fc5\u586b 1s \u5728\u63a5\u6536\u65b0\u7684\u6d88\u606f\u524d\uff0c\u6700\u5927\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/pipelines/source/kafka/#enableautocommit","title":"enableAutoCommit","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enableAutoCommit bool \u975e\u5fc5\u586b false \u662f\u5426\u5f00\u542fautoCommit"},{"location":"reference/pipelines/source/kafka/#autocommitinterval","title":"autoCommitInterval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 autoCommitInterval time.Duration \u975e\u5fc5\u586b 1s autoCommit\u7684\u95f4\u9694\u65f6\u95f4"},{"location":"reference/pipelines/source/kafka/#autooffsetreset","title":"autoOffsetReset","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 autoOffsetReset string \u975e\u5fc5\u586b latest \u6ca1\u6709offset\u65f6\uff0c\u521d\u59cb\u7684offset\u91c7\u7528\u65b9\u5f0f\uff0c\u53ef\u4e3aearliest\u548clatest"},{"location":"reference/pipelines/source/kafka/#sasl","title":"sasl","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 sasl \u975e\u5fc5\u586b SASL authentication sasl.type string \u5fc5\u586b SASL\u7c7b\u578b\uff0c\u53ef\u4e3a\uff1aplain\u3001scram sasl.userName string \u5fc5\u586b \u7528\u6237\u540d sasl.password string \u5fc5\u586b \u5bc6\u7801 sasl.algorithm string type=scram\u65f6\u5fc5\u586b type=scram\u65f6\u4f7f\u7528\u7684\u7b97\u6cd5\uff0c\u53ef\u9009sha256\u3001sha512"},{"location":"reference/pipelines/source/kube-event/","title":"kubeEvent","text":"\u63a5\u6536Kubernetes events\u7684source\u3002
\u4f7f\u7528\u65b9\u5f0f\u53ef\u53c2\u8003\u91c7\u96c6Kubernetes Events\u3002
Example
sources:\n- type: kubeEvent\nname: event\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 kubeconfig string \u975e\u5fc5\u586b \u8bf7\u6c42Kubernetes\u7684kubeconfig\u6587\u4ef6\uff0c\u5f53Loggie\u90e8\u7f72\u5728Kubernetes\u96c6\u7fa4\u4e2d\u65f6\uff0c\u65e0\u9700\u586b\u5199\uff0c\u4f1a\u8fdb\u5165in cluster\u6a21\u5f0f"},{"location":"reference/pipelines/source/kube-event/#master","title":"master","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 master string \u975e\u5fc5\u586b \u8bf7\u6c42Kubernetes\u7684master\u5730\u5740\uff0c\u5f53Loggie\u90e8\u7f72\u5728Kubernetes\u96c6\u7fa4\u4e2d\u65f6\uff0c\u65e0\u9700\u586b\u5199"},{"location":"reference/pipelines/source/kube-event/#buffersize","title":"bufferSize","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 bufferSize int \u975e\u5fc5\u586b 1000 \u76d1\u542c\u7684\u961f\u5217\u5927\u5c0f\uff0c\u6700\u5c0f\u4e3a1"},{"location":"reference/pipelines/source/kube-event/#watchlatestevents","title":"watchLatestEvents","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 watchLatestEvents bool \u975e\u5fc5\u586b false \u662f\u5426\u53ea\u76d1\u542c\u6700\u65b0\u7684events \u7531\u4e8eLoggie\u91cd\u542f\u540e\u4f1a\u91cd\u65b0list\u6240\u6709\u7684events\uff0c\u4f1a\u5bfc\u81f4\u91cd\u590d\u53d1\u9001\uff0c\u5982\u679c\u4e0d\u5e0c\u671b\u91cd\u590d\u53d1\u9001\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u4e3atrue\uff0c\u5f53\u7136\u53ef\u80fd\u5bfc\u81f4\u91cd\u542f\u65f6\u95f4\u6bb5\u5185\u65b0\u4ea7\u751f\u7684events\u4e22\u5931\u3002
"},{"location":"reference/pipelines/source/kube-event/#blacklistnamespaces","title":"blackListNamespaces","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 blackListNamespaces string\u6570\u7ec4 \u975e\u5fc5\u586b \u4e0d\u63a5\u6536\u5176\u4e2d\u5b9a\u4e49\u7684namespaces\u4e2d\u4ea7\u751f\u7684events"},{"location":"reference/pipelines/source/overview/","title":"Overview","text":"sources\u5b57\u6bb5\u4e3a\u6570\u7ec4\uff0c\u4e00\u4e2aPipeline\u4e2d\u53ef\u586b\u5199\u591a\u4e2asource\u7ec4\u4ef6\u914d\u7f6e\u3002
\u56e0\u6b64\uff0c\u8bf7\u6ce8\u610f\u6240\u6709\u7684source\u4e2dname\u5fc5\u586b\uff0c\u4f5c\u4e3apipeline\u4e2dsource\u7684\u552f\u4e00\u6807\u8bc6\u3002
\u6240\u6709Source\u5747\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u914d\u7f6e\u3002
"},{"location":"reference/pipelines/source/overview/#enabled","title":"enabled","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 enabled bool \u975e\u5fc5\u586b true \u8868\u793a\u662f\u5426\u5f00\u542f\u8be5source"},{"location":"reference/pipelines/source/overview/#name","title":"name","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 name string \u5fc5\u586b \u8868\u793asource\u7684\u540d\u79f0\uff0c\u5efa\u8bae\u586b\u5199\u6709\u6807\u8bc6\u610f\u4e49\u7684\u8bcd"},{"location":"reference/pipelines/source/overview/#fields","title":"fields","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fields map \u975e\u5fc5\u586b \u81ea\u5b9a\u4e49\u989d\u5916\u6dfb\u52a0\u5230event\u4e2d\u7684\u5b57\u6bb5 \u6bd4\u5982\u5982\u4e0b\u914d\u7f6e:
Example
sources:\n- type: file\n name: access\n paths:\n - /var/log/*.log\n fields:\n service: demo\n\u4f1a\u7ed9\u91c7\u96c6\u7684\u6240\u6709\u65e5\u5fd7\u4e0a\uff0c\u90fd\u52a0\u4e0aservice: demo\u5b57\u6bb5\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fieldsFromEnv map \u975e\u5fc5\u586b \u989d\u5916\u6dfb\u52a0\u5230event\u4e2d\u7684\u5b57\u6bb5\uff0cvalue\u4e3aenv\u73af\u5883\u53d8\u91cf\u7684key \u6bd4\u5982\u5982\u4e0b\u914d\u7f6e:
Example
sources:\n- type: file\n name: access\n paths:\n - /var/log/*.log\n fieldsFromEnv:\n service: SVC_NAME\n\u4f1a\u4eceLoggie\u6240\u5728\u7684\u73af\u5883\u53d8\u91cf\u4e2d\uff0c\u83b7\u53d6SVC_NAME\u7684\u503c${SVC_NAME}\uff0c\u7136\u540e\u7ed9\u6240\u6709\u7684\u65e5\u5fd7event\u4e0a\u6dfb\u52a0\u5b57\u6bb5\uff1aservice: ${SVC_NAME}\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fieldsFromPath map \u975e\u5fc5\u586b \u989d\u5916\u6dfb\u52a0\u5230event\u4e2d\u7684\u5b57\u6bb5\uff0cvalue\u4e3apath\u6307\u5b9a\u6587\u4ef6\u91cc\u7684\u5185\u5bb9 \u6bd4\u5982\u5982\u4e0b\u914d\u7f6e:
Example
sources:\n- type: file\nname: access\npaths:\n- /var/log/*.log\nfieldsFromPath:\ntest: /tmp/foo\n\u5047\u8bbe\u6587\u4ef6/tmp/foo\u7684\u5185\u5bb9\u4e3abar\uff1a
cat /tmp/foo\n---\nbar\nLoggie\u4f1a\u7ed9\u6240\u6709\u7684\u65e5\u5fd7event\u4e0a\u6dfb\u52a0\u5b57\u6bb5\uff1atest: bar\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fieldsUnderRoot bool \u975e\u5fc5\u586b false \u989d\u5916\u6dfb\u52a0\u7684fields\u662f\u5426\u653e\u5728event\u7684\u6839\u90e8 \u6bd4\u5982\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u8f93\u51fa\u7684\u65e5\u5fd7\u683c\u5f0f\u4e3a\uff1a
{\n\"body\": \"hello world\",\n\"fields\": {\n\"service\": \"demo\"\n}\n}\n\u5982\u679c\u8bbe\u7f6efieldsUnderRoot=true\uff0c\u8f93\u51fa\u7684\u65e5\u5fd7\u683c\u5f0f\u4e3a\uff1a
{\n\"body\": \"hello world\",\n\"service\": \"demo\"\n}\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 fieldsUnderKey string \u975e\u5fc5\u586b fields \u5f53fieldsUnderRoot=false\u65f6\uff0c\u5b57\u6bb5\u7684\u540d\u79f0 \u6bd4\u5982\u53ef\u4ee5\u4fee\u6539\u9ed8\u8ba4\u7684\u5b57\u6bb5fields\u4e3atag\uff0c\u8f93\u51fa\u7684\u65e5\u5fd7\u4e3a\uff1a
{\n\"body\": \"hello world\",\n\"tag\": {\n\"service\": \"demo\"\n}\n}\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 codec \u975e\u5fc5\u586b source\u63a5\u6536\u5230\u6570\u636e\u7684\u65f6\u5019\u7528\u4e8e\u89e3\u6790\u9884\u5904\u7406 codec.type string \u975e\u5fc5\u586b \u65e0 \u8bf7\u6ce8\u610f\uff1a\u76ee\u524d\u4ec5file source\u652f\u6301source codec\u3002
\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 codec.bodyFields \u5fc5\u586b \u4f7f\u7528\u89e3\u6790\u8bfb\u53d6\u5230\u7684json\u6570\u636e\u4e2d\u7684\u8be5\u5b57\u6bb5\u4f5c\u4e3abody \u914d\u7f6e\u793a\u4f8b\uff1a
type: json
sources:\n- type: file\nname: nginx\npaths:\n- /var/log/*.log\ncodec:\ntype: json\nbodyFields: log\n\u5982\u679c\u91c7\u96c6\u5230\u7684\u65e5\u5fd7\u4e3a\uff1a
{\"log\":\"I0610 08:29:07.698664 Waiting for caches to sync\\n\", \"stream\":\"stderr\", \"time:\"2021-06-10T08:29:07.698731204Z\"}\nbody: \"I0610 08:29:07.698664 Waiting for caches to sync\"\n\u8bf7\u6ce8\u610f\uff1a\u76ee\u524d\u975ebodyFields\u7684\u5b57\u6bb5\u5747\u4f1a\u88ab\u4e22\u5f03\u3002
"},{"location":"reference/pipelines/source/overview/#type-regex","title":"type: regex","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 codec.pattern \u5fc5\u586b \u6b63\u5219\u8868\u8fbe\u5f0f codec.bodyFields \u5fc5\u586b \u4f7f\u7528\u6b63\u5219\u63d0\u53d6\u5230\u7684\u8be5\u5b57\u6bb5\u4f5c\u4e3abody \u914d\u7f6e\u793a\u4f8b\uff1a
type: regex
sources:\n- type: file\nname: nginx\npaths:\n- /var/log/*.log\ncodec:\ntype: regex\npattern: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$\nbodyFields: log\n\u5982\u679c\u91c7\u96c6\u5230\u7684\u65e5\u5fd7\u4e3a\uff1a
2021-12-01T03:13:58.298476921Z stderr F INFO [main] Starting service [Catalina]\nbody: \"INFO [main] Starting service [Catalina]\"\n\u8bf7\u6ce8\u610f\uff1a\u76ee\u524d\u975ebodyFields\u7684\u5b57\u6bb5\u5747\u4f1a\u88ab\u4e22\u5f03\u3002
"},{"location":"reference/pipelines/source/prometheus-exporter/","title":"prometheusExporter","text":"\u91c7\u96c6Prometheus Metrics\u7684\u6307\u6807\u6570\u636e\u3002
Example
sources:\n- type: prometheusExporter\nname: metric\nendpoints:\n- \"http://127.0.0.1:9196/metrics\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 endpoints string\u6570\u7ec4 \u5fc5\u586b \u6293\u53d6\u7684\u8fdc\u7aefexporter\u5730\u5740\uff0c\u8bf7\u6ce8\u610fLoggie\u4e0d\u4f1a\u9ed8\u8ba4\u5728\u8bf7\u6c42\u8def\u5f84\u4e2d\u6dfb\u52a0/metrics"},{"location":"reference/pipelines/source/prometheus-exporter/#interval","title":"interval","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 interval time.Duration \u975e\u5fc5\u586b 30s \u5b9a\u65f6\u6293\u53d6\u8fdc\u7aefexporter\u7684\u65f6\u95f4\u95f4\u9694"},{"location":"reference/pipelines/source/prometheus-exporter/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 5s \u6293\u53d6\u8bf7\u6c42\u7684\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/source/prometheus-exporter/#tojson","title":"toJson","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 toJson bool \u975e\u5fc5\u586b false \u662f\u5426\u5c06\u6293\u53d6\u5230\u7684prometheus\u539f\u751f\u6307\u6807\uff0c\u8f6c\u6362\u6210JSON\u683c\u5f0f"},{"location":"reference/pipelines/source/prometheus-exporter/#labels","title":"labels","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 labels map \u975e\u5fc5\u586b \u7ed9\u6240\u6709metrics\u6307\u6807\u589e\u52a0\u989d\u5916\u7684label labels\u652f\u6301\u914d\u7f6e${_env.XX}\u7684\u65b9\u5f0f\u83b7\u53d6\u73af\u5883\u53d8\u91cf\u3002 \u4f8b\u5982\uff0c\u914d\u7f6e\uff1a
labels
sources:\n- type: prometheusExporter\nname: metric\nendpoints:\n- \"http://127.0.0.1:9196/metrics\"\nlabels:\nsvc: ${_env.SVC}\n\u5047\u8bbe\u73af\u5883\u53d8\u91cfSVC=test\uff0c\u4f1a\u5c06\u6240\u6709\u7684metrics\u52a0\u4e0asvc=test\u7684label\u3002
"},{"location":"reference/pipelines/source/unix/","title":"unix","text":"\u901a\u8fc7unix socket\u63a5\u6536\u6570\u636e\u3002
Example
sources:\n- type: unix\nname: demo\npath: \"/tmp/loggie.sock\"\n\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 path string \u5fc5\u586b \u63a5\u6536\u7684\u8def\u5f84\u540d"},{"location":"reference/pipelines/source/unix/#maxbytes","title":"maxBytes","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxBytes int \u975e\u5fc5\u586b 40960 \u63a5\u6536\u7684\u6700\u5927\u5b57\u8282\u6570"},{"location":"reference/pipelines/source/unix/#maxconnections","title":"maxConnections","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 maxConnections int \u975e\u5fc5\u586b 512 \u540c\u65f6\u4fdd\u6301\u6700\u591a\u7684\u8fde\u63a5\u6570"},{"location":"reference/pipelines/source/unix/#timeout","title":"timeout","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 timeout time.Duration \u975e\u5fc5\u586b 5m \u8fde\u63a5\u8d85\u65f6\u65f6\u95f4"},{"location":"reference/pipelines/source/unix/#mode","title":"mode","text":"\u5b57\u6bb5 \u7c7b\u578b \u662f\u5426\u5fc5\u586b \u9ed8\u8ba4\u503c \u542b\u4e49 mode string \u975e\u5fc5\u586b 0755"},{"location":"user-guide/","title":"\u7528\u6237\u6307\u5357","text":"\u5728\u672c\u300c\u7528\u6237\u6307\u5357\u300d\u90e8\u5206\uff0c\u6211\u4eec\u4e3b\u8981\u4ecb\u7ecdLoggie\u7684\u4e3b\u8981\u529f\u80fd\u548c\u7279\u6027\uff0c\u540c\u65f6\u5c55\u793a\u5728\u5404\u79cd\u4e0d\u540c\u7684\u573a\u666f\u4e0b\uff0c\u5982\u4f55\u4f7f\u7528Loggie\u6ee1\u8db3\u5404\u7c7b\u9700\u6c42\u3002
\u5982\u679c\u5e0c\u671b\u67e5\u8be2\u5177\u4f53\u7ec4\u4ef6\u7684\u4f7f\u7528\u4e0e\u914d\u7f6e\u65b9\u5f0f\uff0c\u8bf7\u6233\u300c\u7ec4\u4ef6\u914d\u7f6e\u300d\u3002
"},{"location":"user-guide/#_2","title":"\u843d\u5730\u4e00\u5957\u65e5\u5fd7\u7cfb\u7edf\u4f1a\u9047\u5230\u54ea\u4e9b\u95ee\u9898\uff1f","text":"\u5728\u4f01\u4e1a\u4e2d\uff0c\u6211\u4eec\u9700\u8981\u600e\u4e48\u53bb\u6784\u5efa\u4e00\u5957\u5b8c\u6574\u7684\u65e5\u5fd7\u7cfb\u7edf\uff1f\u5982\u4f55\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u9009\u578b\uff0c\u5176\u4e2d\u53c8\u4f1a\u78b0\u5230\u54ea\u4e9b\u95ee\u9898\uff1f \u5728\u4e0d\u540c\u7684\u4e1a\u52a1\u7c7b\u578b\u3001\u4e0d\u540c\u7684\u4f7f\u7528\u573a\u666f\u3001\u4e0d\u540c\u7684\u65e5\u5fd7\u89c4\u6a21\u4e0b\uff0c\u6211\u4eec\u53ef\u4ee5\u91c7\u7528\u54ea\u4e9b\u65e5\u5fd7\u7cfb\u7edf\u67b6\u6784\uff1f
\u8bf7\u770b\u300c\u4f01\u4e1a\u5b9e\u6218\u300d\u3002
\u5728\u6211\u4eec\u5bf9\u843d\u5730\u4e00\u5957\u529f\u80fd\u5b8c\u5584\u3001\u67b6\u6784\u5b8c\u6574\u7684\u65e5\u5fd7\u7cfb\u7edf\u6709\u521d\u6b65\u4e86\u89e3\u540e\uff0c\u60f3\u8981\u77e5\u9053\uff1a
"},{"location":"user-guide/#loggie","title":"\u4e3a\u4ec0\u4e48Loggie\u80fd\u89e3\u51b3\u8fd9\u4e9b\u95ee\u9898\uff1f","text":"\u5e02\u9762\u4e0a\u5f00\u6e90\u7684\u65e5\u5fd7\u91c7\u96c6Agent\u5df2\u7ecf\u5f88\u591a\u4e86\uff0c\u6211\u4eec\u4e3a\u4ec0\u4e48\u53c8\u9009\u62e9\u7814\u53d1Loggie\u5462\uff1f Loggie\u7684\u8bde\u751f\u662f\u4e3a\u4e86\u89e3\u51b3\u4ec0\u4e48\u95ee\u9898\u5462\uff1f Loggie\u548c\u5176\u4ed6\u7684\u5f00\u6e90\u65e5\u5fd7Agent\u533a\u522b\u662f\u4ec0\u4e48\uff1f
\u8bf7\u770b\u300c\u67b6\u6784\u4e0e\u7279\u6027\u300d\u3002
"},{"location":"user-guide/#loggie_1","title":"\u5982\u4f55\u4f7f\u7528Loggie\u89e3\u51b3\u8fd9\u4e9b\u95ee\u9898\uff1f","text":"\u5982\u4f55\u5728Kubernetes\u96c6\u7fa4\u4e0b\u4f7f\u7528Loggie\uff1f \u8bf7\u770b\u300cKubernetes\u300d\u3002
\u5982\u4f55\u914d\u7f6e\u5404\u79cd\u4e0d\u540c\u7684\u65e5\u5fd7\u67b6\u6784\uff1f\u5982\u4f55\u63a5\u5165\u73b0\u6709\u7684\u670d\u52a1\uff1f\u5982\u4f55\u914d\u7f6e\u65e5\u5fd7\u7684\u5904\u7406\u548c\u5207\u5206\uff1fLoggie\u8fd8\u63d0\u4f9b\u4e86\u54ea\u4e9b\u6709\u7528\u53c8\u597d\u7528\u7684\u529f\u80fd\uff1f \u8bf7\u770b\u300c\u6700\u4f73\u5b9e\u8df5\u300d\u3002
\u5982\u4f55\u914d\u7f6e\u6574\u4f53\u7684\u76d1\u63a7\u548c\u62a5\u8b66\uff0c\u4fdd\u8bc1Loggie\u6b63\u5e38\u8fd0\u884c\uff1f\u5982\u4f55\u76d1\u63a7\u662f\u5426\u91c7\u96c6\u5230ERROR\u65e5\u5fd7\uff1f \u8bf7\u770b\u300c\u76d1\u63a7\u62a5\u8b66\u300d\u3002
"},{"location":"user-guide/architecture/advantages/","title":"Loggie\u7684\u4f18\u52bf\u4e0e\u7279\u6027","text":"Loggie\u652f\u6301\u591a\u4e2aPipeline\uff0c\u6bcf\u4e2aPipeline\u90fd\u57fa\u4e8e\u7b80\u5355\u76f4\u89c2\u7684source->interceptor->sink\u7684\u67b6\u6784\u3002 \u8fd9\u6837\u8bbe\u8ba1\u5e26\u6765\u7684\u597d\u5904\u6709\uff1a
"},{"location":"user-guide/architecture/advantages/#_1","title":"\u5f3a\u9694\u79bb","text":"\u591aPipeline\u8bbe\u8ba1\uff0c\u51cf\u5c11\u4e92\u76f8\u5e72\u6270\u3002\u6bd4\u5982\u6211\u4eec\u53ef\u4ee5\u5c06\u91cd\u8981\u7684\u4e1a\u52a1\u65e5\u5fd7\u653e\u5728\u4e00\u4e2aPipeline\u4e2d\uff0c\u5176\u4ed6\u7684\u4e0d\u91cd\u8981\u7684\u65e5\u5fd7\u914d\u7f6e\u4e3a\u53e6\u5916\u7684Pipeline\uff0c\u4e0d\u91cd\u8981\u7684\u65e5\u5fd7\u914d\u7f6e\u53d8\u52a8\u3001\u53d1\u751f\u4e0b\u6e38\u5835\u585e\u65f6\uff0c\u4e0d\u4f1a\u5f71\u54cd\u91cd\u8981\u65e5\u5fd7\u7684\u91c7\u96c6\u548c\u53d1\u9001\u3002
"},{"location":"user-guide/architecture/advantages/#_2","title":"\u901a\u7528\u6027\u66f4\u597d","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u6211\u4eec\u53ef\u80fd\u4f1a\u5c06\u4e0d\u540c\u7c7b\u578b\u7684\u670d\u52a1\u6df7\u5408\u90e8\u7f72\u5728\u4e00\u4e2a\u8282\u70b9\u4e0a\uff0c\u5f88\u53ef\u80fd\u4ed6\u4eec\u7684\u65e5\u5fd7\u4f1a\u53d1\u9001\u5230\u4e0d\u540c\u7684Kafka\u96c6\u7fa4\u4e2d\uff0c\u5982\u679c\u53ea\u6709\u4e00\u4e2a\u5168\u5c40\u7684\u8f93\u51fa\u6e90\uff0c\u9700\u8981\u5728\u8282\u70b9\u4e0a\u90e8\u7f72\u4e24\u4e2aAgent\uff0c\u5982\u679c\u4f7f\u7528Loggie\u5219\u53ea\u9700\u8981\u4f7f\u7528\u4e0d\u540c\u7684Pipeline\u5373\u53ef\uff0c\u6bcf\u4e2aPipeline\u914d\u7f6e\u4e0d\u540c\u7684Sink\uff0c\u51cf\u5c11\u90e8\u7f72\u6210\u672c\u3002 \u6211\u4eec\u751a\u81f3\u53ef\u4ee5\u91c7\u96c6\u76f8\u540c\u7684\u65e5\u5fd7\uff0c\u53d1\u9001\u5230\u4e0d\u540c\u7684\u540e\u7aef\u8f93\u51fa\u6e90\uff0c\u6839\u636e\u5b9e\u9645\u9700\u6c42\u7075\u6d3b\u914d\u7f6e\u3002
"},{"location":"user-guide/architecture/advantages/#_3","title":"\u7075\u6d3b\u3001\u70ed\u63d2\u62d4\u3001\u53ef\u6269\u5c55","text":"\u672c\u8d28\u4e0asource->interceptor->sink\u67b6\u6784\u662f\u4e00\u4e2a\u6570\u636e\u6d41\u5f0f\u7684\u8bbe\u8ba1\uff0c\u4e0d\u540c\u7c7b\u578b\u7684source/interceptor/sink\u7684\u6392\u5217\u7ec4\u5408\uff0c\u53ef\u4ee5\u6ee1\u8db3\u65e5\u5fd7\u7684\u4e0d\u540c\u9700\u6c42\uff0c Loggie\u5e76\u6ca1\u6709\u5c06interceptor\u66f4\u7ec6\u5316\u7684\u5206\u7c7b\u6210\u6bd4\u5982Filter/Formater\u7b49\u7c7b\u578b\uff0cinterceptor\u627f\u62c5\u4e86\u9664\u4e86source\u8bfb\u53d6\uff0csink\u53d1\u9001\u4e4b\u5916\u7684\u5927\u90e8\u5206\u5de5\u4f5c\uff0c\u53ea\u9700\u8981\u914d\u7f6e\u4e0d\u540c\u7684interceptor\u5c31\u53ef\u4ee5\u62e5\u6709\u4e2d\u8f6c\u3001\u8fc7\u6ee4\u3001\u89e3\u6790\u3001\u5207\u5206\u3001\u65e5\u5fd7\u62a5\u8b66\u7b49\u80fd\u529b\u3002 \u4e8e\u662f\uff0cLoggie\u53ef\u4ee5\uff1a
\u5728\u90e8\u7f72\u548c\u7ef4\u62a4\u5c42\u9762\u5e26\u6765\u7684\u597d\u5904\u662f\uff1a \u5982\u679c\u4e4b\u524d\u91c7\u7528\u5e38\u89c4\u7684ELK\u67b6\u6784\uff0c\u4f7f\u7528Filebeat\u91c7\u96c6\u65e5\u5fd7\u3001Logstash\u4e2d\u8f6c\u548c\u89e3\u6790\u65e5\u5fd7\uff0c\u7531\u4e8eFilebeat\u548cLogstash\u662f\u4e24\u4e2a\u4e0d\u540c\u7684\u8bed\u8a00\u6808\uff0c\u5e26\u6765\u6392\u67e5\u95ee\u9898\u548c\u6269\u5c55\u5f00\u53d1\u6210\u672c\u5747\u8f83\u9ad8\u3002 \u6539\u7528Loggie\u540e\u6211\u4eec\u65e0\u9700\u7ef4\u62a4\u4e24\u4e2a\u9879\u76ee\uff0c\u751a\u81f3\u5982\u679c\u6ca1\u6709\u4e2d\u8f6c\u673a\u7684\u9700\u6c42\uff0c\u53ef\u4ee5\u9009\u62e9\u5728Agent\u7aef\u914d\u7f6e\u65e5\u5fd7\u89e3\u6790\u3002
\u53e6\u5916\uff0c\u76ee\u524d\u9488\u5bf9\u65e5\u5fd7\u62a5\u8b66\uff0c\u5f00\u6e90\u7684\u65b9\u6848\u4e00\u822c\u4e3a\u4f7f\u7528elastAlert\uff0c\u4f46\u662felastAlert\u65e0\u6cd5\u76f4\u63a5\u5bf9\u63a5AlertManager\uff0c\u5e76\u4e14\u5728\u9ad8\u53ef\u7528\u7b49\u65b9\u9762\u5b58\u5728\u95ee\u9898\uff0c\u540c\u65f6\u4f1a\u5f3a\u4f9d\u8d56Elasticsearch\u3002\u6240\u4ee5\u5982\u679c\u4f7f\u7528Loggie\uff0c\u53ef\u4ee5\u65e0\u9700\u5f15\u5165\u989d\u5916\u7684\u7ec4\u4ef6\uff0c\u76f4\u63a5\u4f7f\u7528Loggie\u6765\u68c0\u6d4b\u5f02\u5e38\u65e5\u5fd7\u5e76\u63a5\u5165\u62a5\u8b66\u3002
"},{"location":"user-guide/architecture/advantages/#_4","title":"\u53ef\u5feb\u901f\u65b9\u4fbf\u7684\u5199\u4e00\u4e2a\u7ec4\u4ef6","text":"Loggie\u57fa\u4e8e\u5fae\u5185\u6838\u7684\u67b6\u6784\uff0c\u6240\u6709\u7684source/interceptor/sink/queue\u90fd\u88ab\u62bd\u8c61\u6210component\uff0c\u53ea\u9700\u8981\u5728\u4ee3\u7801\u4e2d\u5b9e\u73b0Golang\u63a5\u53e3\uff0c\u5373\u53ef\u65b9\u4fbf\u7684\u7814\u53d1\u4e00\u4e2acomponent\u3002 \u5982\u679cLoggie\u5728\u67d0\u4e9b\u573a\u666f\u4e0b\uff0c\u65e0\u6cd5\u6ee1\u8db3\u4f60\u7684\u9700\u6c42\uff0c\u53ef\u4ee5\u5c1d\u8bd5\u5199\u4e00\u4e2a\u81ea\u5df1\u7684component\u3002 \u6bd4\u5982\u9700\u8981Loggie\u8f6c\u6362\u6210\u7279\u5b9a\u7684\u65e5\u5fd7\u683c\u5f0f\uff0c\u53ef\u4ee5\u5199\u4e00\u4e2ainterceptor\u53bb\u5904\u7406\uff1b\u9700\u8981Loggie\u5c06\u91c7\u96c6\u7684\u65e5\u5fd7\u53d1\u9001\u81f3\u5c1a\u672a\u652f\u6301\u7684\u670d\u52a1\uff0c\u53ef\u4ee5\u5199\u4e00\u4e2asink\u3002 \u5f53\u7136\uff0cLoggie\u4f7f\u7528Golang\u7f16\u5199\uff0c\u6240\u4ee5\u4f60\u76ee\u524d\u9700\u8981\u7528Golang\u6765\u5199component\u3002Golang\u548cJava\u6216\u8005C/C++\u76f8\u6bd4\uff0c\u5728\u6027\u80fd\u548c\u7814\u53d1\u6548\u7387\u4e0a\u6709\u4e00\u4e2a\u6298\u4e2d\uff0c\u66f4\u9002\u5408\u7c7b\u4f3c\u65e5\u5fd7Agent\u7684\u573a\u666f\u3002
"},{"location":"user-guide/architecture/advantages/#kubernetes","title":"\u66f4\u65b9\u4fbf\u7684Kubernetes\u5bb9\u5668\u65e5\u5fd7\u91c7\u96c6\uff0c\u66f4\u597d\u7684\u4e91\u539f\u751f\u652f\u6301","text":"\u5982\u679c\u4f60\u5c1d\u8bd5\u8fc7\u5728Kubernetes\u73af\u5883\u4e0b\u91c7\u96c6\u65e5\u5fd7\uff0c\u5e94\u8be5\u9047\u5230\u8fc7\u8fd9\u4e9b\u95ee\u9898\uff1a
\u5982\u679c\u4f60\u4f7f\u7528Loggie\uff0c\u4f60\u53ef\u4ee5\uff1a
\u5728\u5b9e\u9645\u7684\u751f\u4ea7\u73af\u5883\u4e2d\uff0c\u65e5\u5fd7Agent\u672c\u8eab\u7684\u7a33\u5b9a\u6027\u5f88\u91cd\u8981\uff0c\u540c\u65f6\u4e0d\u5f71\u54cd\u4e1a\u52a1\u4e5f\u5f88\u91cd\u8981\u3002 Loggie\u53ef\u914d\u7f6e\u9650\u6d41interceptor\uff0c\u5728\u65e5\u5fd7\u91cf\u592a\u5927\u65f6\uff0c\u53ef\u4ee5\u907f\u514d\u53d1\u9001\u65e5\u5fd7\u6570\u636e\u5360\u636e\u4e86\u592a\u591a\u7f51\u7edc\u5e26\u5bbd\u3002 Loggie\u6709\u5408\u7406\u7684\u6587\u4ef6\u53e5\u67c4\u5904\u7406\u673a\u5236\uff0c\u907f\u514dfd\u88ab\u5360\u7528\u7684\u5404\u79cd\u5f02\u5e38\u573a\u666f\u5bfc\u81f4\u8282\u70b9\u4e0d\u7a33\u5b9a\u3002
\u53e6\u5916\uff0cLoggie\u7ed3\u5408\u6211\u4eec\u5728\u5404\u79cd\u73af\u5883\u4e2d\u9047\u5230\u7684\u5404\u79cd\u95ee\u9898\uff0c\u9488\u5bf9\u6027\u7684\u68c0\u6d4b\u66b4\u9732\u51fa\u76f8\u5e94\u7684\u6307\u6807\u3002 \u6bd4\u5982\u6307\u6807\u652f\u6301\u91c7\u96c6\u548c\u53d1\u9001\u5ef6\u8fdf\u68c0\u6d4b\u3002\u6bd4\u5982\u9488\u5bf9\u6587\u4ef6size\u589e\u957f\u592a\u5feb\uff0c\u6216\u8005\u6587\u4ef6size\u592a\u5927\u7b49\u573a\u666f\uff0c\u652f\u6301\u8be5\u7c7bmetric\u4e0a\u62a5\u3002
\u540c\u65f6Loggie\u652f\u6301\u539f\u751fPrometheus metric\uff0c\u53ef\u907f\u514d\u989d\u5916\u90e8\u7f72exporter\u5e26\u6765\u7684\u90e8\u7f72\u6210\u672c\u548c\u8d44\u6e90\u6d88\u8017\u3002Loggie\u8fd8\u63d0\u4f9b\u4e86\u5b8c\u5584\u7684Grafana\u76d1\u63a7\u56fe\u8868\uff0c\u53ef\u4ee5\u65b9\u4fbf\u5feb\u901f\u63a5\u5165\u4f7f\u7528\u3002
"},{"location":"user-guide/architecture/advantages/#_6","title":"\u66f4\u4f4e\u7684\u8d44\u6e90\u5360\u7528\uff0c\u66f4\u597d\u7684\u6027\u80fd","text":"Loggie\u57fa\u4e8eGolang\u7f16\u5199\uff0c\u5728\u4ee3\u7801\u5c42\u9762\u6211\u4eec\u6709\u5f88\u591a\u4f18\u5316\uff0c\u5728\u8f83\u5c11\u8d44\u6e90\u5360\u7528\u7684\u540c\u65f6\uff0c\u8fd8\u53ef\u63d0\u4f9b\u5f3a\u5927\u7684\u541e\u5410\u6027\u80fd\u3002
"},{"location":"user-guide/architecture/background/","title":"Loggie\u7684\u8bde\u751f\u80cc\u666f","text":"\u4e3a\u4ec0\u4e48\u6211\u4eec\u4f1a\u9009\u62e9\u7814\u53d1Loggie\uff0c\u5f53\u65f6\u7684\u80cc\u666f\u548c\u539f\u56e0\u662f\u4ec0\u4e48\uff1f
"},{"location":"user-guide/architecture/background/#_1","title":"\u4e00\u3001\u90a3\u4e9b\u5e74\u6211\u4eec\u9047\u5230\u7684\u95ee\u9898","text":"\u5728\u7814\u53d1Loggie\u4e4b\u524d\uff0c\u6211\u4eec\u7684\u65e5\u5fd7\u670d\u52a1\u91c7\u96c6\u7aef\u90fd\u662f\u4f7f\u7528Filebeat\uff0c\u5f53\u65f6\u4e3a\u4ec0\u4e48\u9009\u62e9Filebeat\u5462\uff1f
Filebeat\u662fElastic\u516c\u53f8\u7684\u4ea7\u54c1\uff0c\u4e3b\u6253\u8f7b\u91cf\u7ea7\uff0c\u7528\u4e8e\u66ff\u6362\u539f\u6709Logstash\u53bb\u5b9e\u73b0\u65e5\u5fd7\u91c7\u96c6\u7684\u5de5\u4f5c\uff0c\u76f8\u6bd4\u4ed6\u4eec\u81ea\u5bb6\u57fa\u4e8eJRuby\u8bed\u8a00\u7684Logstash\uff0cFilebeat\u786e\u5b9e\u662f\u76f8\u5bf9\u8f7b\u91cf\u3001\u8d44\u6e90\u5360\u7528\u5c11\u3002\u548c\u5176\u4ed6\u7684\u5f00\u6e90\u65e5\u5fd7\u91c7\u96c6Agent\u76f8\u6bd4\uff0c\u57fa\u4e8eGolang\u5f00\u53d1\u7684Filebeat\uff0c\u4e5f\u6709\u5f88\u591a\u4f18\u52bf\u3002 \u4f8b\u5982\uff0c\u76f8\u6bd4\u57fa\u4e8eJava\u7684Flume\uff0c\u6027\u80fd\u66f4\u597d\uff0c\u8d44\u6e90\u5360\u7528\u66f4\u5c11\uff1b\u76f8\u6bd4\u57fa\u4e8eRuby\u7684Fluentd\uff0c\u6027\u80fd\u66f4\u597d\uff0c\u4e8c\u6b21\u5f00\u53d1\u66f4\u65b9\u4fbf\uff1b\u76f8\u6bd4\u57fa\u4e8eC\u7684Fluentd-bit\uff0c\u529f\u80fd\u66f4\u5b8c\u5584\uff0c\u5f00\u53d1\u66f4\u53cb\u597d\u3002 \u6240\u4ee5\uff0c\u603b\u4f53\u6765\u8bf4\uff0cFilebeat\u662f\u4e00\u4e2a\u76f8\u5bf9\u6bd4\u8f83\u5e73\u8861\u7684\u65e5\u5fd7\u91c7\u96c6Agent\uff0c\u8fd9\u4e5f\u662f\u6211\u4eec\u5f53\u521d\u9009\u62e9Filebeat\u4f5c\u4e3a\u9ed8\u8ba4\u65e5\u5fd7\u91c7\u96c6Agent\u7684\u539f\u56e0\u3002
\u4f46\u662f\u968f\u7740\u6211\u4eec\u5bf9Filebeat\u66f4\u52a0\u6df1\u5ea6\u7684\u4f7f\u7528\uff0c\u5728\u516c\u53f8\u96c6\u56e2\u5185\u90e8\u5b9e\u8df5\u548c\u5916\u90e8\u5ba2\u6237\u7684\u4ea4\u4ed8\u4e2d\uff0c\u4e5f\u78b0\u5230\u4e86\u4e00\u4e9b\u95ee\u9898\u3002
\u7531\u4e8eFilebeat\u8bbe\u8ba1\u5f53\u521d\u8bbe\u8ba1\u65f6\uff0c\u4e3a\u4e86\u533a\u5206\u4e8eLogstash\uff0c\u7a81\u51fa\u8f7b\u91cf\u7ea7\uff0c\u727a\u7272\u4e86\u5f88\u591a\u53ef\u6269\u5c55\u6027\u7684\u8bbe\u8ba1\u3002 \u6700\u660e\u663e\u7684\u5c31\u662f\uff0cFilebeat\u53ea\u6709\u4e00\u4e2aQueue\u548c\u4e00\u4e2aOutput\u3002\u8fd9\u6837\u4e5f\u5bfc\u81f4\u4e86\uff1a
"},{"location":"user-guide/architecture/background/#_2","title":"\u9694\u79bb\u6027\u5f31","text":"\u7531\u4e8e\u6240\u6709\u7684\u670d\u52a1\u65e5\u5fd7\u90fd\u4f1a\u53d1\u9001\u5230\u5168\u5c40\u552f\u4e00\u7684Queue\u91cc\uff0c\u5bfc\u81f4\u670d\u52a1\u65e5\u5fd7\u6570\u636e\u6df7\u5728\u4e00\u8d77\uff0c\u5728\u5f02\u5e38\u573a\u666f\u53d1\u751f\u65f6\uff0c\u65e0\u6cd5\u6709\u9694\u79bb\u6027\u7684\u4fdd\u969c\u3002 \u6bd4\u5982Filebeat\u5168\u5c40\u7684Queue\u5806\u79ef\uff0c\u4f1a\u5bfc\u81f4\u8282\u70b9\u7684\u6240\u6709\u670d\u52a1\u65e5\u5fd7\u5747\u65e0\u6cd5\u53d1\u9001\uff0c\u5982\u679c\u6211\u4eec\u5bf9\u4e0d\u540c\u670d\u52a1\u7684\u65e5\u5fd7\u7684\u7ea7\u522b\u548c\u8981\u6c42\u4e0d\u4e00\u6837\uff0c\u4e0d\u7ba1\u91cd\u8981\u8fd8\u662f\u4e0d\u91cd\u8981\u7684\u65e5\u5fd7\u90fd\u4f1a\u53d7\u5230\u5f71\u54cd\u3002
"},{"location":"user-guide/architecture/background/#output","title":"\u4e0d\u652f\u6301\u591a\u4e2aOutput","text":"\u6709\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u6211\u4eec\u53ef\u80fd\u9700\u8981\u5c06\u4e0d\u540c\u670d\u52a1\u7684\u4e0d\u540c\u7c7b\u578b\u65e5\u5fd7\u53d1\u9001\u81f3\u4e0d\u540c\u7684\u540e\u7aef\uff0c\u4f46Filebeat\u65e0\u6cd5\u4f7f\u7528\u540c\u4e00\u4e2aAgent\u53bb\u53d1\u9001\u5230\u4e0d\u540c\u7684Kafka\u96c6\u7fa4\uff0c\u53ea\u80fd\u5728\u8282\u70b9\u4e0a\u90e8\u7f72\u591a\u4e2aAgent\uff0c\u5bfc\u81f4\u7ef4\u62a4\u548c\u8d44\u6e90\u6210\u672c\u4e0a\u5347\u3002
"},{"location":"user-guide/architecture/background/#_3","title":"\u53ef\u6269\u5c55\u6027\u6709\u9650","text":"\u76f8\u6bd4Logstash/Flume\u7b49\uff0cFilebeat\u5e76\u975e\u4f7f\u7528\u7c7b\u4f3c\u7684input->queue->output\u7684\u7075\u6d3b\u591a\u4e2apipeline\u8bbe\u8ba1\uff0c\u5728\u5bf9\u4e8e\u65e5\u5fd7\u6570\u636e\u7684\u5904\u7406/\u8fc7\u6ee4/\u589e\u5f3a\u4e0a\uff0c\u4f9d\u8d56\u7684\u662fFilebeat\u6709\u9650\u7684\u4e00\u4e9bprocessor\uff0c\u53ef\u6269\u5c55\u6027\u4e0d\u8db3\u3002 \u540c\u65f6Filebeat\u4e5f\u65e0\u6cd5\u4f5c\u4e3a\u4e2d\u8f6c\u805a\u5408\u4f7f\u7528\uff0c\u5728\u4f7f\u7528\u573a\u666f\u4e0b\u5927\u5927\u53d7\u9650\uff0c\u9700\u8981\u989d\u5916\u5f15\u5165\u5176\u4ed6\u7ec4\u4ef6\u3002\u53e6\u5916\u7c7b\u4f3c\u65e5\u5fd7\u62a5\u8b66\u7b49\u573a\u666f\uff0cFilebeat\u4e5f\u65e0\u6cd5\u6ee1\u8db3\u3002 \u6211\u4eec\u4e5f\u5c1d\u8bd5\u8fc7\u5b9a\u5236\u5316\u5f00\u53d1\uff0c\u4f46Filebeat\u672c\u8eab\u7684\u67b6\u6784\u8bbe\u8ba1\u4e0a\u96be\u4ee5\u5b9e\u73b0\u66f4\u591a\u7684\u6269\u5c55\u80fd\u529b\uff0c\u5e76\u4e14\u957f\u671f\u4f1a\u5e26\u6765\u5347\u7ea7\u4e0e\u4e0a\u6e38\u793e\u533a\u4ee3\u7801\u540c\u6b65\u7684\u95ee\u9898\u3002
"},{"location":"user-guide/architecture/background/#_4","title":"\u65e5\u5fd7\u6392\u969c\u8fd0\u7ef4\u56f0\u5883","text":"Filebeat\u7684metrics\u6bd4\u8f83\u6709\u9650\uff0c\u5f88\u591a\u65f6\u5019\u6211\u4eec\u60f3\u8981\u6392\u67e5\u8bf8\u5982\u5e38\u89c1\u7684\u65e5\u5fd7\u662f\u5426\u6709\u91c7\u96c6\u3001\u91c7\u96c6\u7684\u65e5\u5fd7\u662f\u5426\u5b8c\u6574\u3001\u53d1\u9001\u662f\u5426\u6709\u5ef6\u8fdf\u7b49\u7b49\u6392\u969c\u573a\u666f\uff0cFilebeat\u6ca1\u6709\u63d0\u4f9b\u76f8\u5e94\u7684\u529f\u80fd\uff0c\u5341\u5206\u5f71\u54cd\u7ebf\u4e0a\u7684\u95ee\u9898\u6392\u67e5\u6548\u7387\u3002\u800c\u4e14Filebeat\u672a\u63d0\u4f9bPrometheus\u683c\u5f0f\u7684\u76d1\u63a7\u6307\u6807\uff0c\u9700\u8981\u989d\u5916\u6ce8\u5165exporter\u3002
"},{"location":"user-guide/architecture/background/#_5","title":"\u6027\u80fd\u4e0d\u591f","text":"\u867d\u7136Filebeat\u6027\u80fd\u5c1a\u53ef\uff0c\u4f46\u662f\u5728\u6211\u4eec\u7684\u5b9e\u9645\u4f7f\u7528\u65f6\uff0c\u9047\u5230\u65e5\u5fd7\u573a\u666f\u590d\u6742\u3001\u65e5\u5fd7\u91cf\u5927\u7684\u60c5\u51b5\u65f6\uff0c\u5b58\u5728\u541e\u5410\u91cf\u7684\u74f6\u9888\uff0c\u65e0\u6cd5\u6ee1\u8db3\u5b9e\u65f6\u6027\u7684\u9700\u6c42\u3002
"},{"location":"user-guide/architecture/background/#_6","title":"\u4e8c\u3001\u4e3a\u4ec0\u4e48\u73b0\u6709\u7684\u5176\u4ed6\u5f00\u6e90\u65e5\u5fd7\u9879\u76ee\u4e0d\u80fd\u6ee1\u8db3\u9700\u6c42\uff1f","text":"Fluentd/Fluent-bit
Fluentd\u57fa\u4e8eRuby\u6027\u80fd\u4e00\u822c\uff0c\u5355\u7ebf\u7a0b\uff1bFluent-bit\u57fa\u4e8eC\uff0c\u5bf9\u4e8e\u6211\u4eec\u7684\u6280\u672f\u6808\u6765\u8bf4\uff0cRuby\u548cC\u7684\u7ef4\u62a4\u548c\u4e8c\u6b21\u5f00\u53d1\u6210\u672c\u6bd4\u8f83\u5927\u3002
Logstash
Logstash\u6027\u80fd\u8f83\u5dee\uff0c\u57fa\u4e8eJRuby\u7684\u8d44\u6e90\u5360\u7528\u548c\u635f\u8017\u90fd\u6bd4\u8f83\u5927\u3002
Flume
\u8d44\u6e90\u5360\u7528\u8f83\u5927\uff0c\u6027\u80fd\u4e00\u822c\uff0c\u4e4b\u524d\u6709\u5185\u90e8\u90e8\u95e8\u4f7f\u7528\u8fc7Flume\uff0c\u5bf9\u6bd4\u7684\u538b\u6d4b\u7ed3\u8bba\u8bc1\u5b9e\u786e\u5b9e\u4e0d\u5982Filebeat\u3002
\u6700\u91cd\u8981\u7684\u662f\uff0c\u76ee\u524d\u6240\u6709\u7684\u5f00\u6e90Agent\uff0c\u5747\u6ca1\u6709\u5bf9K8s\u6709\u5f88\u597d\u7684\u539f\u751f\u652f\u6301\uff0c\u4e2a\u522b\u652f\u6301\u7684\u4e5f\u53ea\u80fd\u91c7\u96c6stdout\u7684\u65e5\u5fd7\u3002 \u6b63\u662f\u7531\u4e8e\u76ee\u524d\u5f00\u6e90\u7684Agent\u5b58\u5728\u4e00\u4e9b\u95ee\u9898\uff0c\u4e0d\u80fd\u6ee1\u8db3\u957f\u671f\u7684\u9700\u6c42\uff0c\u6240\u4ee5\u6211\u4eec\u51b3\u5b9a\u5f00\u59cb\u81ea\u7814\u3002
"},{"location":"user-guide/architecture/background/#agent","title":"\u4e09\u3001\u4ec0\u4e48\u624d\u662f\u6211\u4eec\u7406\u60f3\u4e2d\u7684\u65e5\u5fd7Agent\uff1f","text":"\u6574\u4f53\u7684\u76ee\u6807\uff1a \u9ad8\u6027\u80fd\u3001\u8d44\u6e90\u5360\u7528\u4f4e\u3001\u9ad8\u53ef\u7528\u3001\u7a33\u5b9a\u6027\u5f3a\u3001\u53ef\u6269\u5c55\u6027\u5f3a\u3001\u66f4\u9002\u5408\u4e91\u539f\u751f\u3002
\u6027\u80fd\u4e0e\u8d44\u6e90\uff1a \u6027\u80fd\u76f8\u540c\u7684\u60c5\u51b5\u4e0b\uff0cCPU\u6bd4\u793e\u533aFilebeat\u5927\u5927\u964d\u4f4e\uff0c\u541e\u5410\u91cf\u4e0a\u9650\u8981\u8fdc\u9ad8\u4e8eFilebeat\u3002
\u9ad8\u53ef\u7528\u3001\u7a33\u5b9a\u6027\u5f3a\uff1a \u8d44\u6e90\u9694\u79bb\uff0c\u4f5c\u4e3a\u57fa\u7840\u8bbe\u65bd\u4e00\u5b9a\u8981\u7a33\u5b9a\u53ef\u9760\uff0c\u540c\u65f6\u9ed8\u8ba4\u652f\u6301\u5927\u91cf\u76d1\u63a7\u6307\u6807\uff0c\u5bf9\u5e38\u89c1\u7684\u8fd0\u7ef4\u7c7b\u95ee\u9898\u6709\u826f\u597d\u7684\u652f\u6491\uff0c\u51cf\u5c11\u8fd0\u7ef4\u8d1f\u62c5\u3002
\u53ef\u6269\u5c55\u6027\uff1a \u6574\u4f53\u8bbe\u8ba1\u4e0a\uff0c\u65b9\u4fbf\u7528\u6237\u6269\u5c55\uff0c\u5b9e\u73b0\u8fc7\u6ee4\u3001\u8def\u7531\u3001\u7f16\u7801\u7b49\u80fd\u529b\u3002\u6bd4\u5982\u53ef\u4ee5\u5f88\u5feb\u901f\u7684\u5199\u4e00\u4e2a\u5904\u7406\u903b\u8f91\uff0c\u5c31\u53ef\u4ee5\u8fdb\u884c\u6570\u636e\u5904\u7406\u3002
\u603b\u7ed3\u4e00\u4e0b\uff0c\u6211\u4eec\u7406\u60f3\u4e2d\u7684\u65e5\u5fd7Agent\u662f\u4e00\u4e2a\uff1a
\u6d4b\u8bd5\u73af\u5883\uff1a
\u6d4b\u8bd5\u76ee\u7684\uff1a
Filebeat\u548cLoggie\u7684\u6027\u80fd\u5bf9\u6bd4
\u6d4b\u8bd5\u601d\u8def\uff1a
Filebeat\u548cLoggie\uff0c\u5747\u91c7\u96c6\u65e5\u5fd7\u53d1\u9001\u81f3Kafka\uff0c\u89c2\u5bdf\u76f8\u5e94\u7684\u8d44\u6e90\u5360\u7528\u548c\u53d1\u9001\u541e\u5410\u91cf
\u6d4b\u8bd5\u8be6\u60c5\uff1a
\u5355\u6587\u4ef6\u81ea\u52a8\u751f\u62105000000\u884c\u65e5\u5fd7\uff0c\u6bcf\u884c\u5185\u5bb9\u5982\u4e0b\u6240\u793a\uff1a
[13/May/2021:10:20:29 +0800] 0.015 10.200.170.107 \"GET /static/3tJHS3Ubrf.html?activity_channel_id=22=1_00000&fromMiniapp=1&miniapp_uuid=uEd93lG2eG8Qj5fRXuiJwNt4bmiylkmg HTTP/1.1\" 200 138957 \"110.183.45.54, 10.200.151.37\" act.you.163.com \"\" \"Mozilla/5.0 (Linux; Android 8.1.0; PADM00Build/O11019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 XWEB/568 MMWEBSDK/190102 Mobile Safari/537.36 MMWEBID/6881 MicroMessenger/7.0.3.1400(0x2700033B) Process/appbrand0 NetType/WIFI Language/zh_CN miniProgram\" \"\" [127.0.0.1:8990] [0.014] [] [] immsg={\"st\":1553307293614,\"sb\":138963,\"rc\":200,\"cf\":{\"sr\":1},\"if\":\"default\",\"ut\":14,\"sv\":\"static\",\"pd\":\"activity\",\"qb\":764}\n\u914d\u7f6eFilebeat\u548cLoggie\u91c7\u96c6\u65e5\u5fd7\uff0c\u5e76\u53d1\u9001\u81f3Kafka\u67d0\u4e2aTopic\uff0c\u4e0d\u4f7f\u7528\u5ba2\u6237\u7aef\u538b\u7f29\uff0cKafka Topic\u914d\u7f6ePartition\u4e3a3\u3002
\u5728\u4fdd\u8bc1Agent\u89c4\u683c\u8d44\u6e90\u5145\u8db3\u7684\u60c5\u51b5\u4e0b\uff0c\u4fee\u6539\u91c7\u96c6\u7684\u6587\u4ef6\u4e2a\u6570\u3001\u53d1\u9001\u5ba2\u6237\u7aef\u5e76\u53d1\u5ea6\uff08\u914d\u7f6eFilebeat worker\u548cLoggie parallelism)\uff0c\u89c2\u5bdf\u5404\u81ea\u7684CPU\u3001Memory\u548cPod\u7f51\u5361\u53d1\u9001\u901f\u7387\u3002
\u6d4b\u8bd5\u5f97\u5230\u5982\u4e0b\u6570\u636e\uff1a
Agent \u6587\u4ef6\u5927\u5c0f \u65e5\u5fd7\u6587\u4ef6\u6570 \u53d1\u9001\u5e76\u53d1\u5ea6 CPU MEM (rss) \u7f51\u5361\u53d1\u5305\u901f\u7387 Filebeat 3.2G 1 3 7.5~8.5c 63.8MiB 75.9MiB/s Filebeat 3.2G 1 8 10c 65MiB 70MiB/s Filebeat 3.2G 10 8 11c 65MiB 80MiB/s Loggie 3.2G 1 3 2.1c 60MiB 120MiB/s Loggie 3.2G 1 8 2.4c 68.7MiB 120MiB/s Loggie 3.2G 10 8 3.5c 70MiB 210MiB/s\u6d4b\u8bd5\u7ed3\u8bba\uff1a
\u76f8\u540c\u538b\u6d4b\u6761\u4ef6\u548c\u573a\u666f\u4e0b\uff1a
Loggie\u548cFilebeat\u6d88\u8017\u7684CPU\u76f8\u6bd4\uff0c\u5927\u6982\u4ec5\u4e3a\u540e\u8005\u7684\u00bc\uff0c\u540c\u65f6\u53d1\u9001\u541e\u5410\u91cf\u4e3a\u540e\u8005\u76841.6\uff5e2.6\u500d\u3002
Memory\u76f8\u5f53\uff0c\u5747\u5904\u4e8e\u8f83\u4f4e\u7684\u6c34\u51c6\u3002
Filebeat\u7684\u6781\u9650\u541e\u5410\u91cf\u5b58\u5728\u74f6\u9888\uff0c80MB/s\u540e\u5f88\u96be\u63d0\u5347\uff0c\u800cLoggie\u5219\u53ef\u4ee5\u8fbe\u5230200MiB/s\u4ee5\u4e0a\u3002
Loggie\u8be6\u7ec6\u7684\u8bbe\u8ba1\u5982\u4e0b\u56fe\u6240\u793a\uff1a
\u6570\u636e\u94fe\u8def:
\u63a7\u5236\u94fe\u8def:
\u672c\u8d28\u4e0aLoggie\u662f\u4e00\u4e2a\u6570\u636e\u4f20\u8f93\u6d41\u5f0f\u8bbe\u8ba1\uff0c\u6240\u4ee5\u6211\u4eec\u53ef\u4ee5\u7075\u6d3b\u7684\u4f7f\u7528Loggie\u7684Pipeline\u3002\u4ece\u4f7f\u7528\u5f62\u6001\u4e0a\u53ef\u5206\u4e3a\uff1a
\u9664\u4e86\u65e5\u5fd7\u91c7\u96c6\uff0c\u5f88\u591a\u6d89\u53ca\u5230\u6570\u636e\u4f20\u8f93\u3001\u8f6c\u6362\u7684\u573a\u666f\uff0c\u90fd\u53ef\u4ee5\u8003\u8651\u4f7f\u7528Loggie\uff0c\u5373\u4f7fLoggie\u6ca1\u6709\u4f60\u60f3\u8981\u7684\u7ec4\u4ef6\uff0c\u4f60\u4e5f\u53ef\u4ee5\u5feb\u901f\u5f00\u53d1\u4e00\u4e2asource\u3001sink\u6216interceptor\uff0c\u540c\u65f6\u590d\u7528Loggie\u7684\u5f88\u591a\u80fd\u529b\uff0c\u907f\u514d\u91cd\u590d\u7684\u5f00\u53d1\u5de5\u4f5c\uff0c\u6bd4\u5982\uff1a
Info
\u4e86\u89e3Loggie\u5185\u90e8\u6570\u636e\u683c\u5f0f\u7684\u8bbe\u8ba1\uff0c\u80fd\u5e2e\u52a9\u6211\u4eec\u914d\u7f6e\u5408\u9002\u7684\u65e5\u5fd7\u5904\u7406\u548c\u65e5\u5fd7\u683c\u5f0f\u8f6c\u6362
"},{"location":"user-guide/architecture/schema/#_2","title":"\u7ed3\u6784\u8bbe\u8ba1","text":"\u5728Loggie\u5185\u90e8\u7684\u65e5\u5fd7\u6570\u636e\uff0c\u5305\u62ec\uff1a
\u5982\u679c\u4ee5\u4e0a\u7684\u683c\u5f0f\u4e0d\u6ee1\u8db3\u9700\u6c42\uff0c\u53ef\u4ee5\u53c2\u8003\uff1a
\u5bf9\u4e8e\u539f\u59cb\u65e5\u5fd7\u6570\u636e\u7684\u5207\u5206\u4e0e\u5904\u7406\uff0c\u8bf7\u4f7f\u7528 normalize interceptor\uff0c\u8bf7\u53c2\u8003\uff1a
Loggie\u53ef\u4ee5\u90e8\u7f72\u4e3aAgent\uff0c\u540c\u65f6\u652f\u6301\u72ec\u7acb\u90e8\u7f72\uff0c\u8fdb\u884c\u805a\u5408\u3001\u8f6c\u53d1\u548c\u5904\u7406\u3002
"},{"location":"user-guide/best-practice/aggregator/#_1","title":"\u51c6\u5907\uff1a\u9009\u62e9\u67b6\u6784","text":"\u4f7f\u7528\u4e2d\u8f6c\u673a\u67b6\u6784\u7684\u65b9\u5f0f\u4e00\u822c\u6709\u591a\u79cd\uff0c\u5e38\u89c1\u7684\u6709\uff1a
Agent -> Aggregator: Agent\u76f4\u63a5\u53d1\u9001\u81f3Aggregator\uff0cAggregator\u518d\u53d1\u9001\u81f3\u540e\u7aef\u5b58\u50a8\u3002 Agent -> MQ -> Aggregator: Agent\u53d1\u9001\u81f3\u6d88\u606f\u961f\u5217\uff0c\u6bd4\u5982Kafka\uff0c\u7136\u540eAggregator\u518d\u6d88\u8d39Kafka\u6d88\u606f\u53d1\u9001\u81f3\u540e\u7aef\u3002 \u662f\u5426\u5f15\u5165Kafka\u7b49\u6d88\u606f\u961f\u5217\uff0c\u4e3b\u8981\u53d6\u51b3\u4e8e\u81ea\u8eab\u7684\u573a\u666f\u9700\u6c42\u548c\u6570\u636e\u7684\u91cf\u7ea7\u3002
"},{"location":"user-guide/best-practice/aggregator/#_2","title":"\u51c6\u5907\uff1a\u90e8\u7f72","text":"\u90e8\u7f72Agent
\u90e8\u7f72Aggregator
\u90e8\u7f72\u4e3aAggregator\u7c7b\u578b\u65f6\uff0c\u8bf7\u52a1\u5fc5\u5728Kubernetes\u914d\u7f6e\u4e2d\u6307\u5b9acluster\u96c6\u7fa4\u540d\u79f0\u3002
\u91c7\u96c6\u65e5\u5fd7\u7684LogConfig\u914d\u7f6e\u65e0\u533a\u522b\uff0c\u53ea\u9700\u8981\u4fee\u6539sink\u53d1\u9001\u81f3Loggie Aggregator\u6216\u8005Kafka\u5373\u53ef\u3002
\u88ab\u91c7\u96c6\u7684\u5bb9\u5668\u521b\u5efa\u4ee5\u53ca\u5339\u914d\u7684LogConfig\uff0c\u8bf7\u53c2\u8003Loggie\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u3002
\u8fd9\u91cc\u6211\u4eec\u5c06\u5176\u4e2d\u7684sink\u4fee\u6539\u4e3a\u4ee5\u4e0b\u793a\u4f8b\uff1a
Example
AggregatorapiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: aggregator\nspec:\nsink: |\ntype: grpc\nhost: \"loggie-aggregator.loggie-aggregator:6066\"\napiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: kafka\nspec:\nsink: |\ntype: kafka\nbrokers: [\"127.0.0.1:6400\"]\ntopic: \"log-${fields.topic}\"\n\u914d\u7f6eLogConfig\u4e0b\u53d1\u81f3Aggregator\u672c\u8d28\u4e0a\u548cAgent\u4fa7\u65e0\u533a\u522b\uff0c\u53ea\u9700\u8981\u4fee\u6539\u5176\u4e2dselector\u90e8\u5206\u3002
\u7c7b\u4f3c\u5982\u4e0b\uff1a
Example
apiVersion: loggie.io/v1beta1\nkind: ClusterLogConfig\nmetadata:\nname: aggre\nspec:\nselector:\ntype: cluster\ncluster: aggregator\npipeline:\nsources: |\n- type: grpc\nname: rec1\nport: 6066\nsinkRef: dev\napiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: dev\nspec:\nsink: |\ntype: dev\nprintEvents: true\ncodec:\ntype: json\npretty: true\ntype: cluster\u8868\u793a\u9009\u62e9\u4e0b\u53d1\u914d\u7f6e\u5230cluster\u6307\u5b9a\u7684Loggie\u96c6\u7fa4\uff0c\u5373\u6211\u4eec\u521a\u90e8\u7f72\u7684\u4e2d\u8f6c\u673a\u96c6\u7fa4\uff0c\u5982\u679c\u4e0d\u586b\u5199\u4f1a\u5c06\u914d\u7f6e\u6307\u5b9a\u5230\u9ed8\u8ba4\u7684Agent\u96c6\u7fa4\uff0c\u5bfc\u81f4\u65e0\u6cd5\u751f\u6548\u3002
\u8fd9\u91cc\u7684source\u4e3aGrpc\uff0c\u63a5\u6536Agent Grpc sink\u53d1\u51fa\u7684\u6570\u636e\uff0c\u7136\u540e\u8f6c\u53d1\u81f3\u81ea\u8eabsinkRef\u6307\u5b9a\u7684sink\u4e2d\u3002\u8fd9\u91cc\u6211\u4eec\u521b\u5efa\u4e86\u4e00\u4e2adev sink\u7528\u4e8e\u67e5\u770b\u4e2d\u8f6c\u673a\u8f93\u51fa\u7684\u6570\u636e\u3002
"},{"location":"user-guide/best-practice/aggregator/#_4","title":"\u67e5\u770b\u65e5\u5fd7","text":"\u901a\u8fc7kubectl -nloggie-aggregator logs -f <podName> --tail=200\u547d\u4ee4\u3002 \u53ef\u4ee5\u5728\u4e2d\u8f6c\u673a\u8282\u70b9\u4e0a\u67e5\u770b\u5230\u7c7b\u4f3c\u5982\u4e0b\u65e5\u5fd7\uff1a
events
2021-12-20 09:58:50 INF go/src/loggie.io/loggie/pkg/sink/dev/sink.go:98 > event: {\n\"body\": \"14-Dec-2021 06:19:58.306 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [141] milliseconds\",\n\"fields\": {\n\"podname\": \"tomcat-684c698b66-gkrfs\",\n\"containername\": \"tomcat\",\n\"logconfig\": \"tomcat\",\n\"namespace\": \"default\",\n\"nodename\": \"kind-control-plane\"\n},\n}\n\u5728\u65e5\u5fd7\u88ab\u53d1\u9001\u81f3\u4e0b\u6e38\u670d\u52a1\u65f6\uff0c\u5e76\u53d1\u5ea6\u53ef\u4ee5\u88ab\u63a7\u5236\u3002\u4f46\u8d77\u7801\u6709\u4ee5\u4e0b\u4e24\u70b9\u4f1a\u5bf9\u5b9e\u9645\u60c5\u51b5\u9020\u6210\u5f71\u54cd\uff1a\u4e0b\u6e38\u670d\u52a1\u5668\u662f\u5426\u6709\u538b\u529b\uff0c\u6570\u636e\u53d1\u9001\u662f\u5426\u6709\u538b\u529b\u3002\u8fd9\u4e24\u70b9\u662f\u5b9e\u65f6\u53d8\u5316\u7684\uff0c\u4e00\u4e2a\u5b9a\u503c\u5f88\u96be\u957f\u671f\u6ee1\u8db3\u5b9e\u9645\u9700\u6c42\u3002
\u81ea\u9002\u5e94sink\u6d41\u91cf\u63a7\u5236\u529f\u80fd\u53ef\u4ee5\uff1a
\u6839\u636e\u4e0b\u6e38\u6570\u636e\u54cd\u5e94\u7684\u5b9e\u9645\u60c5\u51b5\uff0c\u81ea\u52a8\u8c03\u6574\u4e0b\u6e38\u6570\u636e\u53d1\u9001\u5e76\u884c\u6570\uff0c\u8981\u505a\u5230\u5c3d\u91cf\u53d1\u6325\u4e0b\u6e38\u670d\u52a1\u7aef\u7684\u6027\u80fd\uff0c\u4e14\u4e0d\u5f71\u54cd\u5176\u6027\u80fd\u3002
\u5728\u4e0a\u6e38\u6570\u636e\u6536\u96c6\u88ab\u963b\u585e\u65f6\uff0c\u9002\u5f53\u8c03\u6574\u4e0b\u6e38\u6570\u636e\u53d1\u9001\u901f\u5ea6\uff0c\u7f13\u89e3\u4e0a\u6e38\u963b\u585e\u3002
\u914d\u7f6e\u7ec6\u8282\u53ef\u53c2\u8003sink concurrency\u3002
"},{"location":"user-guide/best-practice/concurrency/#_1","title":"\u5173\u952e\u8bcd","text":"\u6838\u5fc3\u662f\u6a21\u4efftcp\u6d41\u91cf\u63a7\u5236\uff0c\u5e76\u6839\u636eloggie\u81ea\u8eab\u505a\u8c03\u6574\u3002
\u5feb\u901f\u542f\u52a8\u9636\u6bb5
\u5feb\u901f\u542f\u52a8\u7ed3\u675f
\u5e73\u7a33\u9636\u6bb5
\u5176\u4ed6\u7ec6\u8282
\u4e0b\u6e38\u989d\u5916\u642d\u5efa\u4e86\u672c\u5730\u670d\u52a1\u5668\uff0c\u53ef\u4ee5\u81ea\u7531\u8c03\u6574rtt\u503c\uff0c\u7528\u6765\u6a21\u62df\u7f51\u7edc\u6ce2\u52a8\u3002
\u52a0\u5165\u4e86\u968f\u673a\u7684\u5931\u8d25\u8fd4\u56de\uff0c\u6982\u7387\u53ef\u4ee5\u8bbe\u5b9a\u3002
\u963b\u585e\u60c5\u51b5\u7528\u5927\u5e45\u589e\u5927rtt\u60c5\u51b5\u6765\u6a21\u62df\uff0c\u7531\u4e8erttT\u7684\u4e0d\u662f\u56fa\u5b9a\u7684\u503c\uff0c\u6240\u4ee5\u5f53rtt\u7a33\u5b9a\u540e\uff0c\u4e0d\u4f1a\u5bf9\u5224\u65ad\u903b\u8f91\u6709\u5f71\u54cd\u3002
"},{"location":"user-guide/best-practice/concurrency/#_5","title":"\u914d\u7f6e","text":"Config
concurrency:\nenable: true\ngoroutine:\ninitThreshold: 16\nmaxGoroutine: 30\nunstableTolerate: 3\nchannelLenOfCap: 0.4\nrtt:\nblockJudgeThreshold: 120%\nnewRttWeigh: 0.5\nratio:\nmulti: 2\nlinear: 2\nlinearWhenBlocked: 4\nduration:\nunstable: 15\nstable: 30\n\u6a21\u62df\u4e0b\u6e38\u670d\u52a1\u6ca1\u6709\u8fd4\u56de\u9519\u8bef\u60c5\u51b5\uff0c\u4ec5\u8c03\u6574rtt\uff0c\u6d4b\u8bd5\u7b97\u6cd5\u5bf9\u7f51\u7edc\u5ef6\u8fdf\u7684\u54cd\u5e94\u3002
\u89e3\u8bfb
\u5728\u6848\u4f8b\u4e00\u7684\u57fa\u7840\u4e0a\uff0c\u6a21\u62df\u4e0b\u6e38\u6709\u9519\u8bef\u8fd4\u56de\u7684\u60c5\u51b5\uff0c\u6982\u7387\u8bbe\u5b9a\u4e3a0.15%\u3002
\u89e3\u8bfb
Info
\u6b64\u529f\u80fd\u5728\u8bd5\u9a8c\u9636\u6bb5\uff0c\u9ed8\u8ba4\u4e3a\u5173\u95ed\uff0c\u6b22\u8fce\u4ea4\u6d41\u8ba8\u8bba\u3002
"},{"location":"user-guide/best-practice/log-enrich/","title":"\u65e5\u5fd7\u683c\u5f0f\u4e0e\u5143\u4fe1\u606f\u5b57\u6bb5","text":"\u5efa\u8bae\u5148\u4e86\u89e3Loggie\u5185\u90e8\u65e5\u5fd7\u6570\u636eschema\u8bbe\u8ba1\u3002
Loggie\u90e8\u7f72\u5728\u4e0d\u540c\u7684\u73af\u5883\u4e2d\uff0c\u5982\u679c\u9700\u8981\u5728\u539f\u59cb\u7684\u65e5\u5fd7\u6570\u636e\u91cc\uff0c\u589e\u52a0\u4e00\u4e9b\u5143\u4fe1\u606f\uff0c\u540c\u65f6\u517c\u5bb9\u5df2\u6709\u7684\u683c\u5f0f\uff0c\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\u7684\u529e\u6cd5\u3002
"},{"location":"user-guide/best-practice/log-enrich/#_2","title":"\u5b57\u6bb5\u683c\u5f0f\u8f6c\u6362","text":""},{"location":"user-guide/best-practice/log-enrich/#schema-interceptor","title":"\u4f7f\u7528schema interceptor","text":"\u4f7f\u7528schema interceptor\u53ef\u4ee5\u589e\u52a0\u65f6\u95f4\u5b57\u6bb5\uff0c\u4ee5\u53capipelineName\u4e0esourceName\u5b57\u6bb5\u3002\u53e6\u5916\u8fd8\u53ef\u4ee5\u5bf9\u5b57\u6bb5\u8fdb\u884c\u91cd\u547d\u540d\uff0c\u6bd4\u5982\u4fee\u6539body\u4e3amessage\u3002 \u8bf7\u53c2\u8003schema interceptor\u3002
\u7531\u4e8e\u5927\u90e8\u5206\u60c5\u51b5\u4e0b\uff0c\u6211\u4eec\u9700\u8981\u5168\u5c40\u751f\u6548\uff0c\u800c\u4e0d\u662f\u4ec5\u4ec5\u53ea\u5728\u67d0\u4e2apipeline\u91cc\u6dfb\u52a0\u8be5interceptor\uff0c\u6240\u4ee5\u5efa\u8bae\u5728\u7cfb\u7edf\u914d\u7f6e\u7684defaults\u4e2d\u6dfb\u52a0schema interceptor\uff0c \u8fd9\u6837\u53ef\u4ee5\u907f\u514d\u6bcf\u4e2apipeline\u5747\u9700\u914d\u7f6e\u8be5interceptor\u3002
loggie.yml
loggie:\ndefaults:\ninterceptors:\n- type: schema\nname: global\norder: 700\naddMeta:\ntimestamp:\nkey: \"@timestamp\"\nremap:\nbody:\nkey: message\n\u8fd9\u91cc\u7684name\u662f\u4e3a\u4e86\u589e\u52a0\u6807\u8bc6\uff0c\u907f\u514d\u5982\u679c\u5728pipeline\u4e2d\u53c8\u65b0\u589eschema interceptor\u4f1a\u5bfc\u81f4\u6821\u9a8c\u4e0d\u901a\u8fc7\u3002\u53e6\u5916\u589e\u52a0order\u5b57\u6bb5\u4e3a\u4e00\u4e2a\u8f83\u5c0f\u7684\u503c\uff08\u9ed8\u8ba4\u4e3a900)\uff0c\u8fd9\u6837default\u91cc\u7684interceptor\u4f1a\u4f18\u5148\u4e8epipeline\u91cc\u5b9a\u4e49\u7684\u5176\u4ed6interceptor\u6267\u884c\u3002
"},{"location":"user-guide/best-practice/log-enrich/#transformer-interceptor","title":"\u4f7f\u7528transformer interceptor","text":"tranformer\u63d0\u4f9b\u4e86\u66f4\u4e30\u5bcc\u7684\u529f\u80fd\uff0c\u53ef\u4ee5\u5e94\u5bf9\u590d\u6742\u65e5\u5fd7\u7684\u573a\u666f\u3002 \u5177\u4f53\u8bf7\u53c2\u8003transformer interceptor\u3002
"},{"location":"user-guide/best-practice/log-enrich/#_3","title":"\u6dfb\u52a0\u5143\u4fe1\u606f","text":""},{"location":"user-guide/best-practice/log-enrich/#fields","title":"\u6dfb\u52a0fields\u81ea\u5b9a\u4e49\u5143\u4fe1\u606f","text":"\u5982\u679c\u6211\u4eec\u5728source\u4e0a\u914d\u7f6e\u4e86\u4e00\u4e9b\u81ea\u5b9a\u4e49\u7684fields\u3002
pipelines.yml
pipelines:\n- name: local\nsources:\n- type: file\nname: demo\npaths:\n- /tmp/log/*.log\nfields:\ntopic: \"loggie\"\n\nsink:\ntype: dev\nprintEvents: true\ncodec:\npretty: true\n\u90a3\u4e48sink\u8f93\u51fa\u7684\u4e3a\uff1a
{\n\"fields\": {\n\"topic\": \"loggie\",\n},\n\"body\": \"01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]\"\n}\n\u5f53\u7136\u6211\u4eec\u4e5f\u53ef\u4ee5\u914d\u7f6efieldsUnderRoot: true\uff0c\u8ba9fields\u91cc\u7684key:value\u548cbody\u540c\u4e00\u5c42\u7ea7\u3002
pipelines.yml
pipelines:\n- name: local\nsources:\n- type: file\nfields:\ntopic: \"loggie\"\nfieldsUnderRoot: true\n...\n{\n\"topic\": \"loggie\",\n\"body\": \"01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]\"\n}\n\u5728\u6211\u4eec\u4f7f\u7528file source\u65f6\uff0c\u53ef\u80fd\u5e0c\u671b\u81ea\u52a8\u5728\u65e5\u5fd7\u539f\u59cb\u6570\u636e\u91cc\uff0c\u589e\u52a0\u4e00\u4e9b\u65e5\u5fd7\u91c7\u96c6\u7684\u72b6\u6001\uff0c\u6bd4\u5982\u91c7\u96c6\u7684\u6587\u4ef6\u540d\u79f0\u3001\u91c7\u96c6\u7684\u6587\u4ef6offsest\u7b49\uff0cfile source\u63d0\u4f9b\u4e86\u4e00\u4e2aaddonMeta\u914d\u7f6e\uff0c\u53ef\u5feb\u901fenable\u3002
\u793a\u4f8b\uff1a\u6dfb\u52a0\u5982\u4e0baddonMeta\uff0c\u5e76\u8bbe\u7f6e\u4e3atrue\u3002
file source
sources:\n- type: file\npaths:\n- /var/log/*.log\naddonMeta: true\n\u6b64\u65f6\uff0c\u91c7\u96c6\u7684event\u4f1a\u53d8\u6210\u7c7b\u4f3c\u5982\u4e0b\uff1a
Example
{\n\"body\": \"this is test\",\n\"state\": {\n\"pipeline\": \"local\",\n\"source\": \"demo\",\n\"filename\": \"/var/log/a.log\",\n\"timestamp\": \"2006-01-02T15:04:05.000Z\",\n\"offset\": 1024,\n\"bytes\": 4096,\n\"hostname\": \"node-1\"\n}\n}\n\u5177\u4f53\u5b57\u6bb5\u542b\u4e49\u53ef\u53c2\u8003file source
"},{"location":"user-guide/best-practice/log-enrich/#kubernetes","title":"\u589e\u52a0Kubernetes\u5143\u4fe1\u606f","text":"\u5728Kubernetes\u7684\u573a\u666f\u4e2d\uff0c\u91c7\u96c6\u7684\u5bb9\u5668\u65e5\u5fd7\uff0c\u4e3a\u4e86\u5728\u67e5\u8be2\u7684\u65f6\u5019\uff0c\u4f7f\u7528namespace/podName\u7b49\u4fe1\u606f\u8fdb\u884c\u68c0\u7d22\uff0c\u5f80\u5f80\u9700\u8981\u589e\u52a0\u76f8\u5173\u7684\u5143\u6570\u636e\u3002
\u6211\u4eec\u53ef\u4ee5\u5728\u7cfb\u7edf\u914d\u7f6e\u7684discovery.kubernetes\u4e2d\uff0c\u914d\u7f6e\u989d\u5916\u7684k8s fields\u5b57\u6bb5\u3002
\u53ef\u53c2\u8003discovery\u3002
"},{"location":"user-guide/best-practice/log-enrich/#meta","title":"\u6dfb\u52a0meta\u7cfb\u7edf\u5185\u7f6e\u5143\u4fe1\u606f","text":"\u6709\u4e00\u4e9bLoggie\u7cfb\u7edf\u5185\u7f6e\u7684\u5143\u4fe1\u606f\uff0c\u6211\u4eec\u4e5f\u5e0c\u671b\u53d1\u9001\u7ed9\u4e0b\u6e38\uff0c\u8fd9\u4e2a\u65f6\u5019\uff0c\u9700\u8981\u4f7f\u7528normalize interceptor\u4e2d\u7684addMeta processors\u3002 \uff08\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c\u8be5\u64cd\u4f5c\u4f1a\u5bf9\u91c7\u96c6\u4f20\u8f93\u6027\u80fd\u6709\u4e00\u5b9a\u5f71\u54cd\uff0c\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u5e76\u4e0d\u5efa\u8bae\u4f7f\u7528\u8be5\u65b9\u5f0f\uff09
pipelines.yml
pipelines:\n- name: local\nsources:\n- type: file\nname: demo\npaths:\n- /tmp/log/*.log\nfields:\ntopic: \"loggie\"\ninterceptors:\n- type: normalize\nprocessors:\n- addMeta: ~\n\nsink:\ntype: dev\nprintEvents: true\ncodec:\npretty: true\n\u914d\u7f6e\u4e86addMeta processor\u4e4b\u540e\uff0c\u9ed8\u8ba4\u4f1a\u628a\u6240\u6709\u7684\u7cfb\u7edf\u5185\u7f6e\u5143\u4fe1\u606f\u8f93\u51fa\u3002
\u9ed8\u8ba4Json\u683c\u5f0f\u8f93\u51fa\u793a\u4f8b\u5982\u4e0b\uff1a
Example
{\n\"fields\": {\n\"topic\": \"loggie\"\n},\n\"meta\": {\n\"systemState\": {\n\"nextOffset\": 720,\n\"filename\": \"/tmp/log/a.log\",\n\"collectTime\": \"2022-03-08T11:33:47.369813+08:00\",\n\"contentBytes\": 90,\n\"jobUid\": \"43772050-16777231\",\n\"lineNumber\": 8,\n\"offset\": 630\n},\n\"systemProductTime\": \"2022-03-08T11:33:47.370166+08:00\",\n\"systemPipelineName\": \"local\",\n\"systemSourceName\": \"demo\"\n},\n\"body\": \"01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]\"\n}\n\u5f53\u7136\uff0c\u6211\u4eec\u53ef\u80fd\u4f1a\u89c9\u5f97\u8fd9\u4e9b\u6570\u636e\u592a\u591a\u4e86\uff0c\u6216\u8005\u60f3\u5bf9\u5b57\u6bb5\u8fdb\u884c\u4fee\u6539\u3002\u6211\u4eec\u5c31\u53ef\u4ee5\u4f7f\u7528transformer interceptor\u91cc\u7684action\u8fdb\u884c\u64cd\u4f5c\u3002
"},{"location":"user-guide/best-practice/log-process/","title":"\u65e5\u5fd7\u5207\u5206\u5904\u7406","text":"Loggie\u53ef\u4f7f\u7528transformer interceptor\u6765\u8fdb\u884c\u65e5\u5fd7\u7684\u5207\u5206\u548c\u5904\u7406\uff0c\u5c06\u65e5\u5fd7\u6570\u636e\u8fdb\u884c\u7ed3\u6784\u5316\u7684\u63d0\u53d6\uff0c\u540c\u65f6\u53ef\u4ee5\u5bf9\u63d0\u53d6\u540e\u7684\u5b57\u6bb5\u8fdb\u884c\u5904\u7406\u3002 \u5efa\u8bae\u5148\u4e86\u89e3Loggie\u5185\u90e8\u65e5\u5fd7\u6570\u636eschema\u8bbe\u8ba1\u3002
"},{"location":"user-guide/best-practice/log-process/#_2","title":"\u9700\u6c42\u573a\u666f","text":"\u6700\u4e3b\u8981\u7684\u662f\u5bf9\u65e5\u5fd7\u8fdb\u884c\u5207\u5206\u89e3\u6790\u63d0\u53d6\u548c\u5904\u7406\u3002
\u6bd4\u5982\u4ee5\u4e0b\u65e5\u5fd7\uff1a
01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]\n\u6211\u4eec\u53ef\u80fd\u4f1a\u9700\u8981\u5c06\u5176\u4e2d\u7684\u65e5\u671f\u3001\u65e5\u5fd7\u7ea7\u522b\u89e3\u6790\u51fa\u6765\uff0c\u6700\u7ec8\u5f62\u6210\uff1a
{\n\"time\": \"01-Dec-2021 03:13:58.298\",\n\"level\": \"INFO\",\n\"message\": \"[main] Starting service [Catalina]\"\n}\n\u8fd9\u79cd\u7ed3\u6784\u5316\u7684\u6570\u636e\uff0c\u5b58\u50a8\u7684\u65f6\u5019\u4fbf\u4e8e\u8fc7\u6ee4\u67e5\u8be2\uff0c\u6216\u8005\u6839\u636e\u65e5\u5fd7\u91cc\u7684\u65f6\u95f4\u6765\u6392\u5e8f\uff0c\u800c\u4e0d\u662f\u91c7\u96c6\u7684\u65f6\u95f4\u6233\uff0c\u6216\u8005\u6839\u636e\u65e5\u5fd7\u7ea7\u522b\u8fdb\u884c\u4e00\u4e9b\u8fc7\u6ee4\uff0c\u53ef\u4ee5\u65b9\u4fbf\u67e5\u8be2\u5230ERROR\u7ea7\u522b\u7684\u65e5\u5fd7\u7b49\u7b49\u3002 \u5f53\u7136\u4e0d\u4ec5\u4ec5\u662f\u50cf\u4ee5\u4e0atomcat\u7684\u8fd0\u7ef4\u7c7b\u65e5\u5fd7\uff0c\u8fd8\u6709\u8bf8\u5982\u4e1a\u52a1\u7684\u4e00\u4e9b\u8ba2\u5355\u7b49\u7b49\u65e5\u5fd7\uff0c\u90fd\u6709\u7c7b\u4f3c\u7684\u9700\u6c42\u548c\u4f7f\u7528\u573a\u666f\u3002
\u5173\u4e8estdout\u65e5\u5fd7\u7684\u89e3\u6790\u63d0\u53d6
\u4ee5\u4e0b\u793a\u4f8b\u4ec5\u63d0\u4f9b\u65e5\u5fd7\u5207\u5206\u5904\u7406\u7684\u53c2\u8003\u601d\u8def\uff0c\u5982\u679c\u4f60\u9700\u8981\u63d0\u53d6\u5bb9\u5668\u6807\u51c6\u8f93\u51fa\u7684\u539f\u59cb\u65e5\u5fd7\uff0c\u8bf7\u53c2\u8003\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u3002
"},{"location":"user-guide/best-practice/log-process/#_3","title":"\u914d\u7f6e\u793a\u4f8b","text":"\u65e5\u5fd7\u5207\u5206\u5904\u7406\u5728Loggie Agent\u7aef\u6216\u8005Loggie\u4e2d\u8f6c\u673a\u4fa7\u5747\u53ef\uff0c\u53d6\u51b3\u4e8e\u6211\u4eec\u662f\u5426\u9700\u8981\u4e2d\u8f6c\u673a\uff0c\u4ee5\u53ca\u5e0c\u671b\u65e5\u5fd7\u5904\u7406\u8fd9\u79cdCPU\u5bc6\u96c6\u578b\u7684\u8ba1\u7b97\u662f\u5206\u5e03\u5728Agent\u4e0a\uff0c\u7531\u5404\u4e2a\u8282\u70b9\u627f\u62c5\uff0c\u8fd8\u662f\u5e0c\u671b\u5728\u4e2d\u8f6c\u673a\u96c6\u7fa4\u4e2d\u96c6\u4e2d\u8fdb\u884c\u3002
\u4e0b\u9762\u4ee5\u91c7\u96c6tomcat\u670d\u52a1\u7684access\u65e5\u5fd7\u4e3a\u4f8b\uff0c\u5c55\u793a\u5982\u4f55\u5bf9access\u65e5\u5fd7\u8fdb\u884c\u5b57\u6bb5\u5207\u5206\u3002
\u7b80\u5355\u8d77\u89c1\uff0c\u793a\u4f8b\u4f7f\u7528CRD\u5b9e\u4f8b\u914d\u7f6e\u4e0b\u53d1\u5728Agent\uff0c\u540c\u65f6\u4f7f\u7528dev sink\u76f4\u63a5\u8f93\u51fa\u5904\u7406\u7ed3\u679c\u5c55\u793a\u3002
"},{"location":"user-guide/best-practice/log-process/#tomcat-deployment","title":"\u521b\u5efatomcat deployment","text":"\u8bf7\u53c2\u8003
"},{"location":"user-guide/best-practice/log-process/#logconfig","title":"\u521b\u5efalogconfig","text":"\u914d\u7f6elogconfig\u5982\u4e0b\u6240\u793a\uff1a
Example
apiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: tomcat\nnamespace: default\nspec:\nselector:\nlabelSelector:\napp: tomcat\ntype: pod\npipeline:\nsources: |\n- type: file\nname: access\npaths:\n- /usr/local/tomcat/logs/localhost_access_log.*.txt\n\ninterceptors: |\n- type: transformer\nactions:\n- action: regex(body)\npattern: (?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)\n\nsink: |\ntype: dev\nprintEvents: true\ncodec:\ntype: json\npretty: true\n\u8fd9\u91cc\u6211\u4eec\u5728transformer interceptors\u91cc\uff0c\u914d\u7f6e\u4e86regex action\uff0c\u9488\u5bf9access\u65e5\u5fd7\u8fdb\u884c\u6b63\u5219\u63d0\u53d6\u3002
\u539f\u59cb\u7684access\u65e5\u5fd7\u5927\u6982\u5982\u4e0b\u6240\u793a\uff1a
10.244.0.1 - - [31/Aug/2022:03:13:40 +0000] \"GET / HTTP/1.1\" 404 683\n\u7ecf\u8fc7transformer\u5904\u7406\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u901a\u8fc7kubectl -nloggie logs -f <loggie-pod-name> --tail=100\u6765\u67e5\u770b\u8f93\u51fa\u7684\u65e5\u5fd7\u3002
\u8f6c\u6362\u540e\u7684event\u793a\u4f8b\u5982\u4e0b\uff1a
{\n\"status\": \"404\",\n\"size\": \"683\",\n\"fields\": {\n\"logconfig\": \"tomcat\",\n\"namespace\": \"test1\",\n\"nodename\": \"kind-control-plane\",\n\"podname\": \"tomcat-85c84988d8-frs4n\",\n\"containername\": \"tomcat\"\n},\n\"ip\": \"10.244.0.1\",\n\"id\": \"-\",\n\"u\": \"-\",\n\"time\": \"[31/Aug/2022:03:13:40 +0000]\",\n\"url\": \"\\\"GET / HTTP/1.1\\\"\"\n}\n\u6211\u4eec\u53ef\u4ee5\u6253\u9020\u4ee5Loggie\u4e3a\u6838\u5fc3\u7684\u4e91\u539f\u751f\u53ef\u6269\u5c55\u7684\u5168\u94fe\u8def\u6570\u636e\u5e73\u53f0\uff0cLoggie\u652f\u6301\u4f7f\u7528\u4e0d\u540c\u7684\u6280\u672f\u9009\u578b\u3002
\u5728\u4e0d\u540c\u7684\u4e1a\u52a1\u7c7b\u578b\u3001\u4e0d\u540c\u7684\u4f7f\u7528\u573a\u666f\u3001\u4e0d\u540c\u7684\u65e5\u5fd7\u89c4\u6a21\u4e0b\uff0c\u6211\u4eec\u53ef\u80fd\u4f1a\u91c7\u7528\u4e0d\u540c\u7684\u65e5\u5fd7\u7cfb\u7edf\u67b6\u6784\uff0c\u67b6\u6784\u4e0d\u5b58\u5728\u597d\u574f\uff0c\u53ea\u6709\u5408\u4e0d\u5408\u9002\u3002\u4e00\u4e2a\u7b80\u5355\u7684\u573a\u666f\u4e0b\uff0c\u4f7f\u7528\u590d\u6742\u7684\u67b6\u6784\u642d\u5efa\u51fa\u6765\u7684\u65e5\u5fd7\u7cfb\u7edf\uff0c\u5927\u6982\u4f1a\u5e26\u6765\u8fd0\u7ef4\u707e\u96be\u3002
\u8fd9\u91cc\u901a\u8fc7\u89c4\u6a21\u6f14\u8fdb\u7684\u89c6\u89d2\uff0c\u603b\u7ed3\u4e00\u4e0b\u5e38\u89c1\u7684\u65e5\u5fd7\u7cfb\u7edf\u67b6\u6784\uff0c\u5f53\u7136\u5b9e\u9645\u7684\u6280\u672f\u9009\u578b\u53ca\u53d8\u79cd\u6709\u5f88\u591a\uff0c\u6211\u4eec\u65e0\u6cd5\u4e00\u4e00\u5217\u51fa\uff0c\u76f8\u4fe1\u4f60\u53ef\u4ee5\u901a\u8fc7\u53c2\u8003\u4e0b\u6587\uff0c\u642d\u5efa\u9002\u5408\u81ea\u5df1\u4e1a\u52a1\u7684\u67b6\u6784\u3002
"},{"location":"user-guide/enterprise-practice/architecture-and-evolution/#_2","title":"\u67b6\u6784\u6f14\u8fdb","text":"\u9700\u8981\u63d0\u524d\u8bf4\u660e\u7684\u662f:
\u6bcf\u5929\u7684\u65e5\u5fd7\u89c4\u6a21\u8f83\u5c0f\uff0c\u6bd4\u5982\u53ea\u6709\u51e0\u767eG\uff08\u9884\u4f30500G\u4ee5\u4e0b\uff09\u5de6\u53f3\uff0c\u65e5\u5fd7\u7684\u4f7f\u7528\u573a\u666f\u4ec5\u4ec5\u7528\u4e8e\u65e5\u5e38\u8fd0\u7ef4\u6392\u67e5\u95ee\u9898\uff0c\u53ef\u4ee5\u91c7\u7528Loggie\u76f4\u63a5\u53d1\u9001\u81f3Elasticsearch\u96c6\u7fa4\u7684\u65b9\u5f0f\u3002 \u67b6\u6784\u56fe\u5982\u4e0b\u6240\u793a\uff1a
\u4f18\u70b9\uff1a
\u7f3a\u70b9\uff1a
\u53d8\u79cd\uff1a \u56e0\u4e3a\u4e00\u76f4\u4ee5\u6765ELK\u67b6\u6784\u7684\u6d41\u884c\uff0cElasticsearch\u662f\u6700\u5e38\u7528\u7684\u65e5\u5fd7\u5b58\u50a8\u3002 \u5982\u679c\u6709\u5176\u4ed6\u670d\u52a1\u5bf9Elasticsearch\u7684\u4f9d\u8d56\uff0c\u6216\u8005\u6709Elasticsearch\u7684\u8fd0\u7ef4\u7ecf\u9a8c\uff0cElasticsearch\u662f\u4e00\u4e2a\u8fd8\u4e0d\u9519\u7684\u9009\u62e9\u3002 \u4f46\u662f\uff0cElasticsearch\u5bf9\u8d44\u6e90\u548c\u8fd0\u7ef4\u6709\u4e00\u5b9a\u7684\u8981\u6c42\uff0c\u5728\u67d0\u4e9b\u8f7b\u91cf\u7ea7\u548c\u8d44\u6e90\u654f\u611f\u7684\u73af\u5883\u4e0b\uff0c\u53ef\u4ee5\u8003\u8651\uff1a
\u5728\u6bcf\u5929\u7684\u65e5\u5fd7\u91cf\u7ea7\u7a0d\u5927\uff0c\u6bd4\u5982\u5728500G\u81f31T\u7684\u89c4\u6a21\uff0c\u67b6\u6784\u548c\u4e1a\u52a1\u4f7f\u7528\u4e0a\u6709\u6269\u5c55\u6027\u7684\u8003\u8651\uff0c\u53ef\u8003\u8651\u5f15\u5165Loggie\u4e2d\u8f6c\u96c6\u7fa4\u3002
\u4f18\u70b9\uff1a
\u7f3a\u70b9\uff1a
\u5982\u679c\u65e5\u5fd7\u91cf\u8f83\u5927\uff0c\u6bd4\u59821T\u4ee5\u4e0a\u573a\u666f\uff0c\u5bf9\u6027\u80fd\u4e0e\u7a33\u5b9a\u6027\u8981\u6c42\u6bd4\u8f83\u9ad8\uff0c\u53ef\u8003\u8651\u4f7f\u7528Kafka\u7b49\u6d88\u606f\u961f\u5217\u96c6\u7fa4\u3002
\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0cKafka\u672c\u8eab\u5e76\u4e0d\u80fd\u76f4\u63a5\u53d1\u9001\u81f3\u540e\u7aef\uff0c\u6240\u4ee5\u8fd9\u91cc\u9700\u8981\u8003\u8651\u5982\u4f55\u5c06Kafka\u7684\u6570\u636e\u5b9e\u65f6\u5bfc\u5165\u5230\u540e\u7aef\u5b58\u50a8\u4e2d\u3002 \u8fd9\u65f6\u5019\uff0c\u6211\u4eec\u53ef\u4ee5\u9009\u62e9\u4e00\u4e9b\u7ec4\u4ef6\u6d88\u8d39Kafka\uff0c\u53d1\u9001\u81f3\u540e\u7aef\uff0c\u6bd4\u5982Loggie/Logstash/Kafka connect/Flink\u7b49\u3002 \u4f46\u662fFlink\u9002\u5408\u6709\u81ea\u5df1\u7684\u5b9e\u65f6\u6d41\u5e73\u53f0\u6216\u8005\u8fd0\u7ef4\u80fd\u529b\u7684\u4f01\u4e1a\uff0c\u5426\u5219\u53ef\u80fd\u5f15\u5165\u66f4\u591a\u8fd0\u7ef4\u6210\u672c\u3002
\u4f18\u70b9\uff1a
\u51e0\u5341TB\u81f3PB\u7ea7\uff0c\u76f8\u6bd4\u4e0a\u9762\u5927\u89c4\u6a21\u573a\u666f\uff0c\u96c6\u7fa4\u6570\u91cf\u591a\uff0c\u673a\u623f\u67b6\u6784\u590d\u6742\uff0c\u53ef\u4ee5\u6839\u636e\u4ee5\u4e0a\u67b6\u6784\u589e\u52a0\u66f4\u591a\u7075\u6d3b\u7684\u6269\u5c55\u3002
\u6bd4\u5982\uff1a
\u6700\u7ec8\u6211\u4eec\u53ef\u4ee5\u57fa\u4e8eLoggie\uff0c\u642d\u5efa\u4e00\u5957\u751f\u4ea7\u7ea7\u522b\u7684\u5168\u94fe\u8def\u65e5\u5fd7\u6570\u636e\u5e73\u53f0\u3002
"},{"location":"user-guide/enterprise-practice/architecture-and-evolution/#_7","title":"\u66f4\u591a","text":"\u5b9e\u9645\u5728\u843d\u5730\u4e00\u5957\u5b8c\u5584\u7684\u65e5\u5fd7\u67b6\u6784\u548c\u5e73\u53f0\uff0c\u8fd8\u9700\u8981\u8003\u8651\uff1a
\u4e0d\u7ba1\u4f60\u7684\u670d\u52a1\u90e8\u7f72\u5728\u963f\u91cc\u4e91\u8fd8\u662f\u4f7f\u7528\u6df7\u5408\u4e91\u7684\u67b6\u6784\uff0cLoggie\u540c\u6837\u652f\u6301\u91c7\u96c6\u65e5\u5fd7\u6570\u636e\u81f3\u963f\u91cc\u4e91\u53ef\u89c2\u6d4b\u7edf\u4e00\u5b58\u50a8SLS\u3002 \u800c\u4f60\u53ea\u9700\u8981\u4f7f\u7528sls sink\u5373\u53ef\u3002
\u963f\u91cc\u4e91\u5b98\u65b9\u6587\u6863\u4e2d\u5173\u4e8eLoggie\u7684\u4f7f\u7528\u8bf7\u53c2\u8003\uff1a\u4f7f\u7528Loggie\u4e0a\u4f20\u65e5\u5fd7
"},{"location":"user-guide/enterprise-practice/sls/#sls-project","title":"\u51c6\u5907\uff1a\u521b\u5efaSLS Project","text":"\u4e3a\u4e86\u4f7f\u7528\u963f\u91cc\u4e91\u53ef\u89c2\u6d4b\u7edf\u4e00\u5b58\u50a8SLS\uff0c\u6211\u4eec\u9700\u8981\u5148\u521b\u5efa\u4e00\u4e2a\u5bf9\u5e94\u7684Project\u3002 \u5982\u4e0b\u6240\u793a\uff0c\u5728SLS\u9875\u9762\u4e2d\uff0c\u70b9\u51fb\u521b\u5efaProject\uff0c\u586b\u5165project\u540d\u79f0\u548c\u5bf9\u5e94\u7684\u5730\u57df\uff0c\u521b\u5efa\u5b8c\u4e4b\u540e\u540c\u6b65\u521b\u5efa\u76f8\u5e94\u7684Logstore\u3002
\u7136\u540e\u6211\u4eec\u53ef\u4ee5\u51c6\u5907\u4e00\u4e0bsls sink\u7684\u914d\u7f6e\uff0c\u793a\u4f8b\u5982\u4e0b\u6240\u793a\uff0c\u5177\u4f53\u53ef\u53c2\u8003sls sink \u914d\u7f6e\u53c2\u6570\uff1a
sls sink
sink:\ntype: sls\nname: demo\nendpoint: cn-hangzhou.log.aliyuncs.com\naccessKeyId: xxxx\naccessKeySecret: xxxx\nproject: test\nlogstore: test1\ntopic: myservice\n\u53c2\u8003\u4e3b\u673a\u90e8\u7f72\u7684\u65b9\u5f0f\uff0c\u6211\u4eec\u53ef\u4ee5\u4fee\u6539pipeline.yml\uff0c\u52a0\u4e0asls sink\u7684\u914d\u7f6e\u5373\u53ef\u3002
Pipeline\u914d\u7f6e\u53c2\u8003\u793a\u4f8b\u5982\u4e0b\uff1a
sls pipeline
pipelines:\n- name: test\nsources:\n- type: file\nname: demo\naddonMeta: true\npaths:\n- /tmp/log/*.log\nsink:\ntype: sls\nendpoint: cn-hangzhou.log.aliyuncs.com\naccessKeyId: ${accessKeyId}\naccessKeySecret: ${accessKeySecret}\nproject: loggietest\nlogstore: demo1\ntopic: myservice\n\u548c\u81ea\u5efaKubernetes\u96c6\u7fa4\u4e00\u6837\uff0cLoggie\u540c\u6837\u53ef\u4ee5\u90e8\u7f72\u5728\u963f\u91cc\u4e91ACK\u7684Kubernetes\u96c6\u7fa4\u4e2d\u3002 \u6211\u4eec\u53ef\u4ee5\u6839\u636eACK\u63d0\u4f9b\u7684kubeconfig\u6587\u4ef6\uff0c\u4f7f\u7528kubectl\u6216\u8005helm\u6765\u90e8\u7f72\u3002\u53ef\u53c2\u8003Kubernetes\u90e8\u7f72\u3002
\u8bf7\u6ce8\u610f\uff1a
discovery.kubernetes.rootFsCollectionEnabled\u8bbe\u7f6e\u4e3atrue\u3002discovery.kubernetes.containerRuntime\u4e3a\u5339\u914d\u7684Kubernetes\u96c6\u7fa4\u7684\u5bb9\u5668\u8fd0\u884c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u4e3acontainerd\u3002\u90e8\u7f72\u5b8c\u4e4b\u540e\uff0c\u548c\u5e38\u89c4\u7684Kubernetes\u96c6\u7fa4\u4e0b\u7684\u4f7f\u7528\u59ff\u52bf\u7c7b\u4f3c\uff0c\u540c\u6837\u53ef\u4ee5\u4f7f\u7528ClusterLogConfig/LogConfig/Interceptor/Sink CR\u7684\u65b9\u5f0f\u3002
ACK\u63d0\u4f9b\u4e86CRD\u7684\u4f7f\u7528\u9875\u9762\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5728\u8be5\u9875\u9762\u4e2d\u8fdb\u884c\u521b\u5efa\u548c\u4fee\u6539\u3002
\u91c7\u96c6\u67d0\u4e2a\u670d\u52a1\u7684\u65e5\u5fd7\u793a\u4f8b\u53c2\u8003\u5982\u4e0b\uff0c\u53ea\u9700\u4f7f\u7528sls sink\u5373\u53ef\u3002\u5f53\u7136\u4e5f\u53ef\u4ee5\u4f7f\u7528\u5355\u72ec\u521b\u5efa\u4e00\u4e2asink CR\u5b58\u50a8sls sink\u914d\u7f6e\uff0c\u5728LogConfig\u4e2d\u901a\u8fc7sinkRef\u5f15\u7528\u3002
logconfig
apiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: tomcat\nnamespace: default\nspec:\nselector:\nlabelSelector:\napp: tomcat\ntype: pod\n\npipeline:\nsources: |\n- type: file\nname: stdout\npaths:\n- stdout\n- type: file\nname: access\nignoreOlder: 3d\npaths:\n- /usr/local/tomcat/logs/*.log\n\nsink: |\ntype: sls\nendpoint: cn-hangzhou.log.aliyuncs.com\naccessKeyId: ${accessKeyId}\naccessKeySecret: ${accessKeySecret} \nproject: loggietest\nlogstore: demo1\ntopic: myservice\n\u6211\u4eec\u53ef\u4ee5\u5728SLS\u5177\u4f53\u7684Project\u4e2d\u67e5\u770b\u91c7\u96c6\u7684\u65e5\u5fd7\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a
\u5177\u4f53\u7684\u5143\u4fe1\u606f\u7684\u6dfb\u52a0\u548c\u65e5\u5fd7\u5b57\u6bb5\u4fee\u6539\uff0c\u53ef\u53c2\u8003\u65e5\u5fd7\u5207\u5206\u5904\u7406\u4ee5\u53ca\u65e5\u5fd7\u589e\u52a0\u5143\u4fe1\u606f\u3002
"},{"location":"user-guide/monitor/loggie-monitor/","title":"Loggie\u7684\u76d1\u63a7\u4e0e\u62a5\u8b66","text":"Loggie\u7684monitor eventbus\u88ab\u8bbe\u8ba1\u4e3a\u53d1\u5e03\u548c\u8ba2\u9605\u6a21\u5f0f\uff0c\u5404\u4e2a\u7ec4\u4ef6\u53d1\u9001metrics\u5230\u6307\u5b9a\u7684topic\u4e2d\uff0c\u7531\u72ec\u7acb\u7684listener\u6765\u6d88\u8d39\u5904\u7406\u3002
\u6bd4\u5982file source\u4f1a\u5c06\u91c7\u96c6\u65e5\u5fd7\u7684\u4e00\u4e9b\u6307\u6807\u6570\u636e\uff0c\u53d1\u9001\u81f3filesource topic\uff0c\u7531filesource listener\u6765\u6d88\u8d39\uff0cfilesource listener\u4f1a\u5c06\u6570\u636e\u805a\u5408\u8ba1\u7b97\u4e4b\u540e\uff0c\u6253\u5370\u81f3\u65e5\u5fd7\uff0c\u66b4\u9732\u51faPrometheus\u6307\u6807\u3002
\u7ec4\u4ef6\u548ctopic\u4ee5\u53calistener\u4e4b\u95f4\u662f\u677e\u8026\u5408\u5173\u7cfb\uff0c\u6bd4\u5982file source\u8fd8\u4f1a\u5b9a\u65f6\u5c06\u5168\u91cf\u5339\u914d\u7684\u65e5\u5fd7\u6587\u4ef6\u6307\u6807\u53d1\u9001\u81f3filewatcher topic\uff0cfilewatcher listener\u4f1a\u5904\u7406\u548c\u66b4\u9732\u6307\u6807\u3002
monitor eventbus\u914d\u7f6e\u5728\u5168\u5c40\u7684\u7cfb\u7edf\u914d\u7f6e\u4e2d\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a
Config
loggie:\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nqueue: ~\nsink: ~\nhttp:\nenabled: true\nport: 9196\n\u5176\u4e2dlogger\u63a7\u5236\u6240\u6709\u7684metrics\u6307\u6807\u7684\u65e5\u5fd7\u6253\u5370\uff0c\u4f1a\u5728period\u65f6\u95f4\u95f4\u9694\u5c06\u914d\u7f6elisteners\u4ea7\u751f\u7684metrics\u805a\u5408\u6253\u5370\u5728Loggie\u65e5\u5fd7\u4e2d\uff0c\u4fbf\u4e8e\u56de\u6eaf\u548c\u6392\u67e5\u95ee\u9898\u3002
listeners\u5219\u7528\u4e8e\u914d\u7f6e\u76f8\u5173\u7684listener\u662f\u5426\u5f00\u542f\u3002
Prometheus\u683c\u5f0fmetrics\u9ed8\u8ba4\u901a\u8fc7http.port\u7aef\u53e3\u66b4\u9732\u5728/metrics\u3002\u53ef\u4ee5\u901a\u8fc7curl <podIp>:9196/metrics\u6765\u67e5\u770b\u5f53\u524d\u7684metrics\u6307\u6807\u3002
\u76ee\u524d\u6709\u4ee5\u4e0b\u7684listener\uff0c\u4e3b\u8981\u5305\u542b\uff1a
\u53ef\u4ee5\u4f7f\u7528\u73af\u5883\u5df2\u6709\u7684Prometheus\u6216\u8005Grafana\uff0c\u5982\u679c\u9700\u8981\u65b0\u90e8\u7f72\uff0c\u53ef\u53c2\u8003\uff1ahttps://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack\u3002
\u4f7f\u7528Helm\u90e8\u7f72:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo update\nhelm install prometheus prometheus-community/kube-prometheus-stack -nprometheus --create-namespace\nNote
\u7531\u4e8e\u4f17\u6240\u5468\u77e5\u7684\u539f\u56e0\uff0c\u4f60\u7684\u73af\u5883\u91cc\u53ef\u80fd\u65e0\u6cd5\u4e0b\u8f7d\u5176\u4e2d\u7684\u67d0\u4e9bk8s.gcr.io\u955c\u50cf\uff0c\u53ef\u4ee5\u8003\u8651\u4e0b\u8f7dchart\u5305\u66ff\u6362\u540e\u518d\u91cd\u65b0\u90e8\u7f72\u3002
\u786e\u8ba4Pod\u6b63\u5e38running\u540e\uff0c\u53ef\u4ee5\u6839\u636e\u5b9e\u9645\u73af\u5883\u8bbf\u95eegrafana\u3002\u901a\u8fc7\u4ee3\u7406\u8bbf\u95eegrafana\u7684\u65b9\u5f0f\u53ef\u53c2\u8003\uff1a
kubectl -nprometheus port-forward --address 0.0.0.0 service/prometheus-grafana 8181:80\nGrafana\u7528\u6237\u540d\u548c\u5bc6\u7801\u53ef\u5728prometheus-grafana sercret\u4e2d\uff0c\u901a\u8fc7base64 -d\u6765\u67e5\u770b\u3002
\u5728\u90e8\u7f72\u4e86Loggie\u7684Kubernetes\u96c6\u7fa4\u4e2d\uff0c\u9700\u8981\u521b\u5efa\u5982\u4e0b\u7684ServiceMonitor\u6765\u8ba9Prometheus\u91c7\u96c6Loggie Agent\u6307\u6807\u3002
apiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\nlabels:\napp: loggie\nrelease: prometheus\nname: loggie-agent\nnamespace: prometheus\nspec:\nnamespaceSelector:\nmatchNames:\n- loggie\nendpoints:\n- port: monitor\nselector:\nmatchLabels:\napp: loggie\ninstance: loggie\n\u540c\u65f6\uff0c\u6211\u4eec\u9700\u8981\u5728Grafana\u4e2d\u6dfb\u52a0install\u5de5\u7a0b\u4e2d\u7684json\u6765\u5c55\u793aLoggie\u7684\u76d1\u63a7\u63a7\u5236\u53f0\u3002
Note
Kubernetes\u7248\u672c\u548cGrafana\u7248\u672c\u4e0d\u540c\uff0c\u53ef\u80fd\u5bfc\u81f4\u56fe\u8868\u5c55\u793a\u4e0d\u517c\u5bb9\uff0c\u9700\u8981\u6839\u636e\u60c5\u51b5\u8fdb\u884c\u4fee\u6539\u3002
\u5bfc\u5165\u7684Grafana\u56fe\u8868\u76ee\u524d\u5305\u542b\u76d1\u63a7\u5927\u76d8\u548c\u8282\u70b9Agent\u7684\u76d1\u63a7\u56fe\u8868\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a
"},{"location":"user-guide/monitor/service-log-alarm/","title":"\u4e1a\u52a1\u65e5\u5fd7\u62a5\u8b66","text":"
\u9664\u4e86Loggie\u672c\u8eab\u7684\u62a5\u8b66\uff0c\u4e1a\u52a1\u65e5\u5fd7\u672c\u8eab\u7684\u76d1\u63a7\u62a5\u8b66\u4e5f\u662f\u4e00\u4e2a\u5e38\u7528\u7684\u529f\u80fd\uff0c\u6bd4\u5982\u5728\u65e5\u5fd7\u4e2d\u5305\u542b\u4e86ERROR\u65e5\u5fd7\uff0c\u53ef\u4ee5\u53d1\u9001\u62a5\u8b66\uff0c\u8fd9\u79cd\u62a5\u8b66\u4f1a\u66f4\u8d34\u8fd1\u4e1a\u52a1\u672c\u8eab\uff0c\u662f\u57fa\u4e8emetrics\u62a5\u8b66\u7684\u4e00\u79cd\u5f88\u597d\u7684\u8865\u5145\u3002
"},{"location":"user-guide/monitor/service-log-alarm/#_2","title":"\u4f7f\u7528\u65b9\u5f0f","text":"\u6709\u4ee5\u4e0b\u4e24\u79cd\u65b9\u5f0f\u53ef\u4ee5\u9009\u62e9\uff1a
\u91c7\u96c6\u94fe\u8def\u4e0d\u9700\u8981\u72ec\u7acb\u90e8\u7f72Loggie\uff0c\u4f46\u662f\u7531\u4e8e\u5728\u91c7\u96c6\u7684\u6570\u636e\u94fe\u8def\u4e0a\u8fdb\u884c\u5339\u914d\uff0c\u7406\u8bba\u4e0a\u4f1a\u5bf9\u4f20\u8f93\u6027\u80fd\u9020\u6210\u4e00\u5b9a\u5f71\u54cd\uff0c\u4f46\u80dc\u5728\u65b9\u4fbf\u7b80\u5355\u3002
logAlert interceptor\u7528\u4e8e\u5728\u65e5\u5fd7\u4f20\u8f93\u7684\u65f6\u5019\u68c0\u6d4b\u5f02\u5e38\u65e5\u5fd7\uff0c\u5f02\u5e38\u65e5\u5fd7\u4f1a\u88ab\u5c01\u88c5\u6210\u62a5\u8b66\u7684\u4e8b\u4ef6\u53d1\u9001\u81f3monitor eventbus\u7684logAlert topic\uff0c\u7531logAlert listener\u6765\u6d88\u8d39\u3002logAlert listener\u652f\u6301\u53d1\u9001\u81f3\u4efb\u610fhttp\u540e\u7aef\uff08\u53ef\u4ee5\u591a\u4e2a\uff09\u3002 \u53d1\u9001\u4f53\u6839\u636e\u81ea\u5b9a\u4e49\u6a21\u677f\u8fdb\u884c\u6e32\u67d3\uff0c\u82e5\u6a21\u677f\u672a\u5b9a\u4e49\uff0c\u5219\u4f1a\u53d1\u9001\u539f\u59cb\u6570\u636e\u3002\u5728\u914d\u7f6e\u6a21\u677f\u524d\uff0c\u53ef\u4ee5\u5148\u89c2\u5bdf\u539f\u59cb\u6570\u636e\uff08\u8bbe\u7f6edebug\u6a21\u5f0f\u542f\u52a8\uff09\uff0c\u518d\u8fdb\u884c\u6a21\u677f\u914d\u7f6e\uff0c\u539f\u59cb\u6570\u636e\u53ef\u80fd\u4f1a\u6839\u636epipeline\u914d\u7f6e\u88ab\u5176\u4ed6interceptor\u6539\u52a8\u800c\u4e0e\u793a\u4f8b\u4e0d\u540c\u3002
\u914d\u7f6e\u65b0\u589elogAlert listener\u53d1\u9001\u544a\u8b66\u914d\u7f6e\u3002\u7528\u4e8e\u5728\u68c0\u6d4b\u5230\u5339\u914d\u7684\u65e5\u5fd7\u540e\uff0c\u5c06\u65e5\u5fd7\u62a5\u8b66\u53d1\u9001\u81f3\u8bf8\u5982alertManager\u7b49\u540e\u7aef\uff0c\u8be6\u7ec6\u914d\u7f6e\u53ef\u53c2\u8003logAlert listener\u3002
\u5168\u5c40Config\u6587\u4ef6
loggie:\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nlogAlert:\naddr: [\"http://127.0.0.1:8080/loggie\"]\nbufferSize: 100\nbatchTimeout: 10s\nbatchSize: 10\nlinelimit: 10\ntemplate: |\n{\n\"alerts\":\n[\n{{$first := true}}\n{{range .Alerts}}\n{{if $first}}{{$first = false}}{{else}},{{end}}\n{\n\"labels\": {\n\"topic\": \"{{.fields.topic}}\"\n},\n\"annotations\": {\n\"message\": \"\\nNew alert: \\nbody:\\n{{range .body}}{{.}}\\n{{end}}\\ncontainerid: {{._meta.pipelineName}}\\nsource: {{._meta.sourceName}}\\ncontainername: {{.fields.containername}}\\nlogconfig: {{.fields.logconfig}}\\nname: {{.fields.name}}\\nnamespace: {{.fields.namespace}}\\nnodename: {{.fields.nodename}}\\npodname: {{.fields.podname}}\\nfilename: {{.state.filename}}\\n\",\n\"reason\": \"{{.reason}}\"\n},\n\"startsAt\": \"{{._meta.timestamp}}\",\n\"endsAt\": \"{{._meta.timestamp}}\"\n}\n{{end}}\n],\n{{$first := true}}\n{{range .Alerts}}\n{{if $first}}{{$first = false}}{{else}}\n\"commonLabels\": {\n\"module\": \"{{._additions.module}}\",\n\"alertname\": \"{{._additions.alertname}}\",\n\"cluster\": \"{{._additions.cluster}}\"\n}\n{{end}}\n{{end}}\n}\nfilesource: ~\nfilewatcher: ~\nreload: ~\nqueue: ~\nsink: ~\nhttp:\nenabled: true\nport: 9196\n\u4e0a\u9762\u7684template\u6a21\u7248\u8868\u793a\u53d1\u9001\u7684\u544a\u8b66\u5185\u5bb9\u683c\u5f0f\uff0c\u4f7f\u7528go template\u683c\u5f0f\u3002\u53ef\u53c2\u8003GO Template\uff0c\u6216\u8005\u8bf7\u81ea\u884c\u641c\u7d22go template\u4f7f\u7528\u6559\u7a0b\u3002
\u5176\u4e2d\u53ef\u4f7f\u7528\u7c7b\u4f3c{{._meta.timestamp}}\u7b49\u5f62\u5f0f\u52a8\u6001\u6e32\u67d3\u539f\u59cbalert\u6570\u636e\u4e2d\u7684\u5b57\u6bb5\u3002
alert\u5b57\u6bb5\u89e3\u91ca\uff1a
\u5b57\u6bb5 \u662f\u5426\u5185\u7f6e \u542b\u4e49 _meta \u662f alert\u5143\u6570\u636e _meta.pipelineName \u8868\u793apipeline\u540d\u79f0 _meta.sourceName \u8868\u793asource\u540d\u79f0 _meta.timestamp \u8868\u793a\u65e5\u5fd7\u65f6\u95f4\u6233 body \u662f logBody reason \u662f \u5339\u914d\u6210\u529f\u539f\u56e0 fields \u5426 field\u5b57\u6bb5\uff0c\u7531\u5176\u4f59\u914d\u7f6e\u6dfb\u52a0 state \u5426 \u91c7\u96c6\u4fe1\u606f\uff0c\u9700\u8981\u5728file source\u914d\u7f6eaddonMeta: true _additions \u5426 \u7531\u914d\u7f6e\u6307\u5b9a \u539f\u59cbalert\u6570\u636e\u4e3a\u4e00\u4e2ajson\uff0c\u5176\u4e2dAlerts\u4e3a\u56fa\u5b9a\u7684key\u3002
\u539f\u59cbalert\u6570\u636e\u793a\u4f8b
{\n\"Alerts\": [\n{\n\"_meta\": {\n\"pipelineName\": \"default/spring\",\n\"sourceName\": \"loggie-source-756fd6bb94-4skqv/loggie-alert/common\",\n\"timestamp\": \"2022-10-28T13:12:30.528824+08:00\"\n},\n\"body\": [\n\"2022-10-28 01:48:07.093 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/]. [dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\",\n\"\",\n\"java.lang.ArithmeticException: / by zero\"\n],\n\"fields\": {\n\"containerid\": \"0dc5f07983bfdf7709ee4fce752679983c4184e94c70dab5fe6df5843d5cbb68\",\n\"containername\": \"loggie-alert\",\n\"logconfig\": \"spring\",\n\"name\": \"loggie-source\",\n\"namespace\": \"default\",\n\"nodename\": \"docker-desktop\",\n\"podname\": \"loggie-source-756fd6bb94-4skqv\",\n\"topic\": \"loggie\"\n},\n\"reason\": \"matches some rules\",\n\"state\": {\n\"bytes\": 6913,\n\"filename\": \"/var/log/pods/ default_loggie-source-756fd6bb94-4skqv_9da3e440-e749-4930-8e4d-41e0d5b66417/ loggie-alert/1.log\",\n\"hostname\": \"docker-desktop\",\n\"offset\": 3836,\n\"pipeline\": \"default/spring\",\n\"source\": \"loggie-source-756fd6bb94-4skqv/loggie-alert/common\",\n\"timestamp\": \"2022-10-28T13:12:30.527Z\"\n},\n\"_additions\": {\n\"namespace\": \"default\",\n\"cluster\": \"local\",\n\"alertname\": \"loggie-test\",\n\"module\": \"loggie\"\n}\n}\n]\n}\nPipeline\u4e2d\u589e\u52a0logAlert interceptor\u7528\u4e8e\u5728\u91c7\u96c6\u7684\u65f6\u5019\u68c0\u6d4b\u65e5\u5fd7\u5e76\u5339\u914d\u65e5\u5fd7\u62a5\u8b66\u89c4\u5219\uff0c\u53ef\u5728ClusterLogConfig/LogConfig\u4e2d\u5f15\u7528\u3002\u5176\u4e2dadditions\u4e3a\u7ed9alert\u989d\u5916\u6dfb\u52a0\u7684\u5b57\u6bb5\uff0c\u4f1a\u653e\u5165alert\u539f\u59cb\u6570\u636e\u7684_addtions\u5b57\u6bb5\u4e2d\uff0c\u53ef\u7528\u505a\u6a21\u677f\u6e32\u67d3\u3002
\u5efa\u8bae\u5148\u4f7f\u7528debug\u6a21\u5f0f\uff08-log.level=debug\uff09\u89c2\u5bdf\u539f\u59cbalert\u6570\u636e\u683c\u5f0f\uff0c\u518d\u914d\u7f6e\u6a21\u677f\u8fdb\u884c\u6e32\u67d3\uff0c\u539f\u59cb\u6570\u636e\u4f1a\u53d7\u5230\u5176\u4ed6\u914d\u7f6e\u7684\u5f71\u54cd\uff0c\u8fd9\u91cc\u4ec5\u5c55\u793a\u4e00\u4e2a\u793a\u4f8b\u3002
\u8be6\u7ec6\u914d\u7f6e\u53ef\u53c2\u8003logAlert interceptor\u3002
Config
apiVersion: loggie.io/v1beta1\nkind: Interceptor\nmetadata:\nname: logalert\nspec:\ninterceptors: |\n- type: logAlert\nmatcher:\ncontains: [\"ERROR\"]\nadditions:\nmodule: \"loggie\"\nalertname: \"loggie-test\"\ncluster: \"local\"\n\u5339\u914d\u5230\u65e5\u5fd7\u544a\u8b66\u89c4\u5219\u4e4b\u540e\uff0c\u544a\u8b66\u540e\u7aef\u53ef\u63a5\u6536\u5230\u7c7b\u4f3c\u7684\u6570\u636e\u5982\u4e0b\u6240\u793a\uff1a
Example
{\n\"alerts\": [\n{\n\"labels\": {\n\"topic\": \"loggie\"\n},\n\"annotations\": {\n\"message\": \"\\nNew alert: \\nbody:\\n2022-10-28 01:48:07.093 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\\n\\njava.lang.ArithmeticException: / by zero\\ncontainerid: 0dc5f07983bfdf7709ee4fce752679983c4184e94c70dab5fe6df5843d5cbb68\\nsource: loggie-source-756fd6bb94-4skqv/loggie-alert/common\\ncontainername: loggie-alert\\nlogconfig: spring\\nname: loggie-source\\nnamespace: default\\nnodename: docker-desktop\\npodname: loggie-source-756fd6bb94-4skqv\\nfilename: /var/log/pods/default_loggie-source-756fd6bb94-4skqv_9da3e440-e749-4930-8e4d-41e0d5b66417/loggie-alert/1.log\\n\",\n\"reason\": \"matches some rules\"\n},\n\"startsAt\": \"2022-10-28T13:12:30.527Z\",\n\"endsAt\": \"2022-10-28T13:12:30.527Z\"\n}\n],\n\"commonLabels\": {\n\"module\": \"loggie\",\n\"alertname\": \"loggie-test\",\n\"cluster\": \"local\"\n}\n}\nLoggie\u914d\u7f6esource\u91c7\u96c6\u65e5\u5fd7\uff0c\u7ecf\u8fc7logAlert interceptor\u5339\u914d\u65f6\uff0c\u53ef\u914d\u7f6esendOnlyMatched\u4ec5\u5c06\u5339\u914d\u6210\u529f\u7684\u65e5\u5fd7\u53d1\u9001\u81f3alertWebhook sink\uff0c\u5339\u914d\u5931\u8d25\u7684\u65e5\u5fd7\u770b\u4f5c\u6b63\u5e38\u65e5\u5fd7\u88ab\u5ffd\u7565\u3002\u5efa\u8bae\u5728\u4f7f\u7528alertWebhook sink\u65f6\uff0c\u540c\u65f6\u5f00\u542flogAlert interceptor, \u8bbe\u7f6esendOnlyMatched\u4e3atrue\u642d\u914d\u4f7f\u7528\u3002
\u914d\u7f6e\u65b0\u589ealertWebhook sink\u3002\u8be6\u7ec6\u914d\u7f6e\u53ef\u53c2\u8003alertWebhook Sink\u3002
Config
sink:\ntype: alertWebhook\naddr: http://localhost:8080/loggie\nlinelimit: 10\ntemplate: |\n{\n\"alerts\":\n[\n{{$first := true}}\n{{range .Alerts}}\n{{if $first}}{{$first = false}}{{else}},{{end}}\n{\n\"labels\": {\n\"topic\": \"{{.fields.topic}}\"\n},\n\"annotations\": {\n\"message\": \"\\nNew alert: \\nbody:\\n{{range .body}}{{.}}\\n {{end}}\\ncontainerid: {{._meta.pipelineName}}\\nsource: {{. _meta.sourceName}}\\ncontainername: {{.fields. containername}}\\nlogconfig: {{.fields.logconfig}}\\nname: {{.fields.name}}\\nnamespace: {{.fields.namespace}} \\nnodename: {{.fields.nodename}}\\npodname: {{.fields. podname}}\\nfilename: {{.state.filename}}\\n\",\n\"reason\": \"{{.reason}}\"\n},\n\"startsAt\": \"{{._meta.timestamp}}\",\n\"endsAt\": \"{{._meta.timestamp}}\"\n}\n{{end}}\n],\n{{$first := true}}\n{{range .Alerts}}\n{{if $first}}{{$first = false}}{{else}}\n\"commonLabels\": {\n\"namespace\": \"{{._additions.namespace}}\",\n\"module\": \"{{._additions.module}}\",\n\"alertname\": \"{{._additions.alertname}}\",\n\"cluster\": \"{{._additions.cluster}}\"\n}\n{{end}}\n{{end}}\n}\nlogAlert Interceptor\u914d\u7f6e\u548c\u63a5\u6536\u65b9\u6536\u5230\u7684\u62a5\u8b66\u4e0e\u91c7\u96c6\u94fe\u8def\u68c0\u6d4b\u62a5\u8b66\u7c7b\u4f3c\u3002
\u4e3a\u4ec0\u4e48\u6211\u7684\u65e5\u5fd7\u6ca1\u6709\u91c7\u96c6\uff1f
\u65e5\u5fd7\u91c7\u96c6\u4e2d\uff0c\u6700\u4e3a\u5173\u952e\u6838\u5fc3\u7684\u95ee\u9898\u662f\uff0c\u65e5\u5fd7\u6709\u6ca1\u6709\u88ab\u91c7\u96c6\u5230\uff0c\u4e3a\u4ec0\u4e48\u6211\u914d\u7f6e\u7684\u65e5\u5fd7\u6ca1\u6709\u53d1\u9001\u8fc7\u53bb\uff1f \u4e0b\u9762\u63d0\u4f9b\u4e86\u6838\u5fc3\u7684\u6392\u67e5\u601d\u8def\u548c\u624b\u6bb5\u4f9b\u53c2\u8003\u3002 \u53e6\u5916\uff0c\u6700\u91cd\u8981\u7684\u662f\uff0c\u5728\u73af\u5883\u91cc\u914d\u7f6eLoggie\u7684Prometheus\u76d1\u63a7\u548cGrafana\u56fe\u8868\uff0c\u53ef\u4ee5\u5feb\u901f\u53d1\u73b0\u95ee\u9898\u3002
"},{"location":"user-guide/troubleshot/log-collection/#_2","title":"\u65e5\u5fd7\u91c7\u96c6\u6838\u5fc3\u673a\u5236","text":"\u4e86\u89e3\u5b9e\u73b0\u673a\u5236\u662f\u6392\u969c\u7684\u57fa\u7840\uff1a
\uff08\u9488\u5bf9\u975eKubernetes\u7684\u4e3b\u673a\u573a\u666f\uff0c\u53ea\u662f\u5c11\u4e86LogConfig CRD\u914d\u7f6e\u4e0b\u53d1\u7684\u6b65\u9aa4\uff0c\u5176\u4f59\u7c7b\u4f3c\uff09
"},{"location":"user-guide/troubleshot/log-collection/#loggie-dashboard","title":"Loggie dashboard","text":"\u5728Kubernetes\u7684\u573a\u666f\u4e0b\uff0c\u76ee\u524dLoggie\u63d0\u4f9b\u4e86\u4e00\u4e2a\u57fa\u4e8eterminal\u7684\u53ef\u4ea4\u4e92\u5f0fdashboard\uff0c\u53ef\u4ee5\u66f4\u597d\u7684\u5e2e\u52a9\u6211\u4eec\u4fbf\u6377\u7684\u6392\u67e5\u95ee\u9898\u3002
\u8fdb\u5165dashboard
\u627e\u5230\u4efb\u610f\u4e00\u4e2aLoggie Pod
kubectl -nloggie get po -owide\n\u8fdb\u5165\u5176\u4e2d\u4e00\u4e2aLoggie Pod
kubectl -nloggie exec -it ${podName} bash\n\u8fd0\u884cdashboard
./loggie inspect\n\u4f7f\u7528terminal
terminal\u9996\u9875\u5c55\u793a\u793a\u4f8b\u5982\u4e0b\u6240\u793a\uff1a
\u5177\u4f53\u8bf7\u53c2\u8003\u4f7f\u7528\u8bf4\u660e\u89c6\u9891\u3002
Tips
Loggie terminal\u529f\u80fd\u4ec5\u4ecev1.4\u7248\u672c\u5f00\u59cb\u63d0\u4f9b\uff0c\u5982\u679c\u4ece\u4f4e\u7248\u672c\u5347\u7ea7\uff0c\u9700\u589e\u52a0clusterrole\u914d\u7f6e\uff0c\u8bf7\u53c2\u8003\u8fd9\u91cc\u3002
"},{"location":"user-guide/troubleshot/log-collection/#_3","title":"\u624b\u52a8\u6392\u67e5\u6b65\u9aa4","text":"\u6392\u67e5\u95ee\u9898\u5173\u952e\u5148\u8981\u786e\u5b9a\u662f\u54ea\u4e00\u6b65\u51fa\u73b0\u4e86\u95ee\u9898\u3002
"},{"location":"user-guide/troubleshot/log-collection/#_4","title":"\u65e5\u5fd7\u91c7\u96c6\u4efb\u52a1\u6392\u67e5","text":"\u67e5\u770b\u6211\u4eec\u8981\u6392\u67e5\u7684\u65e5\u5fd7\u91c7\u96c6\u4efb\u52a1LogConfig/ClusterLogConfig\u7684Events\u4e8b\u4ef6\uff1a
kubectl -n ${namespace} describe lgc ${name}\n\u5982\u679c\u6ca1\u6709events\uff0c\u5219\u53ef\u80fd\u4e3a\uff1a
labelSelector\u6307\u5b9a\u7684label\u672a\u548c\u6211\u4eec\u671f\u671b\u7684Pod\u5339\u914d\u3002\u901a\u8fc7\u4ee5\u4e0b\u547d\u4ee4\u67e5\u770b kubectl -n ${namespace} get po -owide -l ${labels}\nkubectl -n ns1 get po -owide -l app=tomcat,service=web \u6765\u5224\u65ad\u4e00\u4e0b\u662f\u5426\u6709\u5339\u914d\u7684Pod\u3002\u5982\u679c\u6ca1\u6709\u7c7b\u4f3csync success\u7684events\uff0c\u53ef\u6839\u636eevents\u540c\u65f6\u7ed3\u5408Loggie\u65e5\u5fd7\u6392\u67e5\u95ee\u9898\uff1a
kubectl -n ${loggie-namespace} logs -f ${loggie-pod-name} \u2014-tail=${N}\nkubectl -nloggie logs -f loggie-5x6vf --tail=100\u3002 \u67e5\u770b\u5bf9\u5e94\u8282\u70b9\u7684Loggie\u65e5\u5fd7\uff0c\u6839\u636e\u65e5\u5fd7\u60c5\u51b5\u8fdb\u884c\u5904\u7406\u3002 \u5e38\u89c1\u7684\u5f02\u5e38\u6709\uff1a
/var/log/*.log\u3002\u6700\u5feb\u901f\u7684\u505a\u6cd5\u662f\uff0c\u5728\u9700\u8981\u91c7\u96c6\u7684\u4e1a\u52a1\u7684Pod\u91cc\uff0c\u6267\u884cls <path>\uff0c\u56e0\u4e3als\u4e5f\u662f\u4f7f\u7528\u7684glob\u8868\u8fbe\u5f0f\u53bb\u5339\u914d\u65e5\u5fd7\u6587\u4ef6\u3002\u53e6\u5916\u4e5f\u9700\u989d\u5916\u6ce8\u610f\u662f\u5426\u914d\u7f6e\u4e86ignoreOlder/excludeFiles\u7b49\u53c2\u6570\uff0c\u5ffd\u7565\u6216\u8005\u6392\u9664\u4e86\u6211\u4eec\u5e0c\u671b\u91c7\u96c6\u7684\u65e5\u5fd7\u6587\u4ef6\u3002\u6839\u636e\u5728logConfig\u7684labelSelector\u627e\u5230\u4e00\u4e2a\u5339\u914d\u7684\u4e1a\u52a1pod\uff1a
kubectl -n ${namespace} get po -owide -l ${labels}\n${node-name}\uff0c\u7136\u540e\u901a\u8fc7\uff1a kubectl -n ${loggie-namespace} get po -owide |grep ${node-name}\n\u68c0\u67e5\u662f\u5426\u6709\u5f02\u5e38\uff0c\u5982\u679c\u6709\u5f02\u5e38\uff0c\u5219\u9700\u6839\u636e\u5f02\u5e38\u65e5\u5fd7\u518d\u505a\u5206\u6790\u5224\u65ad
kubectl -n ${loggie-namespace} logs -f ${loggie-pod-name} \u2014-tail=${N}\n\u8c03\u7528\u81ea\u52a8\u6392\u969c\u63a5\u53e3
\u9488\u5bf9\u5bf9\u5e94\u7684Agent\uff0c\u8c03\u7528help\u63a5\u53e3\uff1a
curl <ip>:9196/api/v1/help\n\u63a5\u53e3\u8fd4\u56de\u91cc\u6700\u5f00\u5934\u5305\u62ec\u4e86\u4e00\u4e2a\u4f7f\u7528\u63d0\u793a\uff1a
--------- Usage: -----------------------\n|--- view details: /api/v1/help?detail=<module>, module is one of: all/pipeline/log\n|--- query by pipeline name: /api/v1/help?pipeline=<name>\n|--- query by source name: /api/v1/help?source=<name>\ncurl <ip>:9196/api/v1/help?detail=all\u6765\u67e5\u8be2\u6240\u6709\u7684\u7ec6\u8282\u8be6\u60c5\uff0c\u4e5f\u53ef\u4ee5\u4f7f\u7528\u6839\u636epipeline\u6216\u8005source\u7684\u540d\u79f0\u6765\u641c\u7d22\u3002 \u76ee\u524d\u8fd4\u56de\u4e3b\u8981\u5305\u62ec\u4e24\u90e8\u5206\uff1a
\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u8be5\u63a5\u53e3\u8986\u76d6\u4e86\u4ee5\u4e0b\u64cd\u4f5c\u5f97\u5230\u7684\u5185\u5bb9\uff0c\u65e0\u9700\u7ee7\u7eed\u8fdb\u884c\u4ee5\u4e0b\u64cd\u4f5c\uff0c\u4e0b\u6587\u6b65\u9aa4\u4ec5\u4f9b\u53c2\u8003\u3002
\u67e5\u770b\u8282\u70b9Loggie\u6e32\u67d3\u751f\u6210\u7684\u914d\u7f6e
\u8fdb\u5165\u5230\u5bb9\u5668\u4e2d\uff1a
kubectl -n ${loggie-namespace} exec -it ${loggie-pod-name} bash\nls /opt/loggie/pipeline/\ncurl ${loggie-pod-ip}:9196/api/v1/reload/config\n\u786e\u8ba4\u65e5\u5fd7\u91c7\u96c6\u914d\u7f6e
cat /opt/loggie/pipeline/\n\u67e5\u770b\u65e5\u5fd7\u91c7\u96c6\u6301\u4e45\u5316\u72b6\u6001
Loggie\u4f1a\u8bb0\u5f55\u6bcf\u4e2a\u65e5\u5fd7\u6587\u4ef6\u7684\u91c7\u96c6\u72b6\u6001\uff0c\u8fd9\u6837\u5373\u4f7fLoggie\u91cd\u542f\u540e\uff0c\u4e5f\u53ef\u4ee5\u7ee7\u7eed\u4fdd\u6301\u4e0a\u4e00\u6b21\u7684\u91c7\u96c6\u8fdb\u5ea6\uff0c\u907f\u514d\u91cd\u65b0\u91c7\u96c6\u65e5\u5fd7\u6587\u4ef6\u3002 \u53ef\u4ee5\u901a\u8fc7\u8c03\u7528\u63a5\u53e3\u6765\u67e5\u770b\uff1a
curl ${loggie-pod-ip}:9196/api/v1/source/file/registry?format=text | grep XXX\n {\n \"id\": 85,\n \"pipelineName\": \"default/tomcat\",\n \"sourceName\": \"tomcat-7d64c4f6c9-cm8jm/tomcat/common\",\n \"filename\": \"/var/lib/kubelet/pods/9397b8be-8927-44ba-8b94-73e5a4459377/volumes/kubernetes.io~empty-dir/log/catalina.2022-06-02.log\",\n \"jobUid\": \"3670030-65025\",\n \"offset\": 4960,\n \"collectTime\": \"2022-06-06 12:44:12.861\",\n \"version\": \"0.0.1\"\n },\nfilename\u4e3aLoggie\u8f6c\u6362\u540e\u7684\u5b9e\u9645\u8282\u70b9\u4e0a\u7684\u65e5\u5fd7\u8def\u5f84\uff0cjobUid\u7ec4\u6210\u4e3a\u6587\u4ef6\u7684inode-deviceId\uff0coffset\u4e3asink\u53d1\u9001\u6210\u529f\u63a5\u6536\u5230ack\u540e\u7684offset\u3002 \u53e6\u5916\uff0c\u6211\u4eec\u53ef\u4ee5\u901a\u8fc7\u5728Loggie\u5bb9\u5668\u4e2d\u6267\u884c\uff1a stat ${filename}\nstat
File: /var/lib/kubelet/pods/9397b8be-8927-44ba-8b94-73e5a4459377/volumes/kubernetes.io~empty-dir/log/catalina.2022-06-02.log\nSize: 4960 Blocks: 16 IO Block: 4096 regular file\nDevice: fe01h/65025d Inode: 3670030 Links: 1\nAccess: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)\nAccess: 2022-06-06 12:44:12.859236003 +0000\nModify: 2022-06-02 08:54:33.177240007 +0000\nChange: 2022-06-02 08:54:33.177240007 +0000\n\u53ef\u901a\u8fc7\u6bd4\u8f83size\u548coffset\u6765\u5224\u65ad\u91c7\u96c6\u8fdb\u5ea6\u3002\u5982\u679csize=offset\uff0c\u5219\u8bf4\u660e\u5df2\u7ecf\u6587\u4ef6\u5df2\u7ecf\u5168\u90e8\u91c7\u96c6\u5e76\u53d1\u9001\u6210\u529f\u3002
"},{"location":"user-guide/troubleshot/problems/","title":"\u95ee\u9898\u6848\u4f8b","text":""},{"location":"user-guide/troubleshot/problems/#netcgolookupip-segmentation-violation","title":"net.cgoLookupIP \u5bfc\u81f4segmentation violation","text":"\u73b0\u8c61
\u53ef\u80fd\u5728\u4f7f\u7528\u4e86\u6bd4\u5982unix sock source\u7684\u65f6\u5019\uff0cLoggie\u542f\u52a8\u540ecrash\uff0c\u5e76\u6709\u5982\u4e0b\u7684\u65e5\u5fd7\uff1a
fatal error: unexpected signal during runtime execution\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f59b4528360]\n\nruntime stack:\nruntime.throw({0x213b1a1, 0x7f59b423b640})\n /usr/local/go/src/runtime/panic.go:1198 +0x71\nruntime.sigpanic()\n /usr/local/go/src/runtime/signal_unix.go:719 +0x396\n\ngoroutine 86 [syscall]:\nruntime.cgocall(0x1a39d30, 0xc000510d90)\n /usr/local/go/src/runtime/cgocall.go:156 +0x5c fp=0xc000510d68 sp=0xc000510d30 pc=0x40565c\nnet._C2func_getaddrinfo(0xc00030bc40, 0x0, 0xc000724fc0, 0xc0005bd990)\n _cgo_gotypes.go:91 +0x56 fp=0xc000510d90 sp=0xc000510d68 pc=0x5c7bb6\nnet.cgoLookupIPCNAME.func1({0xc00030bc40, 0xc0004d29c0, 0x4}, 0xc00030bb80, 0xc000510e50)\n /usr/local/go/src/net/cgo_unix.go:163 +0x9f fp=0xc000510de8 sp=0xc000510d90 pc=0x5c98ff\nnet.cgoLookupIPCNAME({0x20f751c, 0x3}, {0xc00030bb80, 0xc00022b5e0})\n /usr/local/go/src/net/cgo_unix.go:163 +0x16d fp=0xc000510f38 sp=0xc000510de8 pc=0x5c914d\nnet.cgoIPLookup(0x358aed0, {0x20f751c, 0xc00030bbc0}, {0xc00030bb80, 0xc000510fb8})\n /usr/local/go/src/net/cgo_unix.go:220 +0x3b fp=0xc000510fa8 sp=0xc000510f38 pc=0x5c99bb\nnet.cgoLookupIP\u00b7dwrap\u00b725()\n /usr/local/go/src/net/cgo_unix.go:230 +0x36 fp=0xc000510fe0 sp=0xc000510fa8 pc=0x5c9e36\nruntime.goexit()\n /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000510fe8 sp=0xc000510fe0 pc=0x46ae81\ncreated by net.cgoLookupIP\n /usr/local/go/src/net/cgo_unix.go:230 +0x125\n\u539f\u56e0
\u5177\u4f53\u539f\u56e0\u8bf7\u53c2\u8003\uff1ago net\u3002
Name Resolution\n\nThe method for resolving domain names, whether indirectly with functions like Dial\nor directly with functions like LookupHost and LookupAddr, varies by operating system.\n\nOn Unix systems, the resolver has two options for resolving names.\nIt can use a pure Go resolver that sends DNS requests directly to the servers\nlisted in /etc/resolv.conf, or it can use a cgo-based resolver that calls C\nlibrary routines such as getaddrinfo and getnameinfo.\n\nBy default the pure Go resolver is used, because a blocked DNS request consumes\nonly a goroutine, while a blocked C call consumes an operating system thread.\nWhen cgo is available, the cgo-based resolver is used instead under a variety of\nconditions: on systems that do not let programs make direct DNS requests (OS X),\nwhen the LOCALDOMAIN environment variable is present (even if empty),\nwhen the RES_OPTIONS or HOSTALIASES environment variable is non-empty,\nwhen the ASR_CONFIG environment variable is non-empty (OpenBSD only),\nwhen /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the\nGo resolver does not implement, and when the name being looked up ends in .local\nor is an mDNS name.\n\nThe resolver decision can be overridden by setting the netdns value of the\nGODEBUG environment variable (see package runtime) to go or cgo, as in:\n\n export GODEBUG=netdns=go # force pure Go resolver\n export GODEBUG=netdns=cgo # force cgo resolver\n\nThe decision can also be forced while building the Go source tree\nby setting the netgo or netcgo build tag.\n\u89e3\u51b3\u529e\u6cd5
\u5728Loggie \u90e8\u7f72\u811a\u672c\u91cc\u589e\u52a0\u73af\u5883\u53d8\u91cf\uff1a
env:\n- name: GODEBUG\nvalue: netdns=go\n\u9605\u8bfb\u672c\u6587\u4e4b\u524d\uff0c\u5efa\u8bae\u5148\u53c2\u8003Kubernetes\u4e0b\u65e5\u5fd7\u91c7\u96c6\u95ee\u9898\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#loggie_1","title":"Loggie\u5982\u4f55\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff1f","text":"\u7531\u4e8eKubernetes\u826f\u597d\u7684\u53ef\u6269\u5c55\u6027\uff0cKubernetes\u8bbe\u8ba1\u4e86\u4e00\u79cd\u81ea\u5b9a\u4e49\u8d44\u6e90CRD\u7684\u6982\u5ff5\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5df1\u5b9a\u4e49CRD\u8868\u793a\u81ea\u5df1\u7684\u671f\u671b\u72b6\u6001\uff0c\u5e76\u501f\u52a9\u4e00\u4e9bframework\u5f00\u53d1Controller\uff0c\u4f7f\u7528Controller\u5c06\u6211\u4eec\u7684\u671f\u671b\u53d8\u6210\u73b0\u5b9e\u3002
\u57fa\u4e8e\u8fd9\u4e2a\u601d\u8def\uff0c\u4e00\u4e2a\u670d\u52a1\u9700\u8981\u91c7\u96c6\u54ea\u4e9b\u65e5\u5fd7\uff0c\u9700\u8981\u4ec0\u4e48\u6837\u7684\u65e5\u5fd7\u914d\u7f6e\uff0c\u662f\u7528\u6237\u7684\u671f\u671b\uff0c\u800c\u8fd9\u5c31\u9700\u8981\u6211\u4eec\u5f00\u53d1\u4e00\u4e2a\u65e5\u5fd7\u91c7\u96c6\u7684Controller\u53bb\u5b9e\u73b0\u3002
\u6240\u4ee5\uff0c\u7528\u6237\u53ea\u9700\u8981\u5728\u6211\u4eec\u5b9a\u4e49\u7684CRD LogConfig\u4e2d\uff0c\u586b\u5199\u9700\u8981\u91c7\u96c6\u54ea\u4e9bPods\u7684\u65e5\u5fd7\uff0c\u5728Pod\u4e2d\u7684\u65e5\u5fd7\u8def\u5f84\u662f\u4ec0\u4e48\u3002
\u6838\u5fc3\u67b6\u6784\u5982\u4e0b\u56fe\u6240\u793a\uff1a
Loggie\u4f1a\u611f\u77e5\u5230Pod\u548cCRD\u7684\u4e8b\u4ef6\uff0c\u8fdb\u884c\u914d\u7f6e\u7684\u52a8\u6001\u66f4\u65b0\u3002\u540c\u65f6\uff0cLoggie\u53ef\u4ee5\u6839\u636e\u65e5\u5fd7\u6587\u4ef6\u8def\u5f84\u6302\u8f7d\u7684Volume\uff0c\u627e\u5230\u76f8\u5e94\u5728\u8282\u70b9\u7684\u6587\u4ef6\u8fdb\u884c\u91c7\u96c6\u3002\u53e6\u5916\u8fd8\u53ef\u4ee5\u6839\u636e\u914d\u7f6e\uff0c\u81ea\u52a8\u5c06Pod\u4e0a\u7684Env/Annotation/Label\u52a0\u5165\u5230\u65e5\u5fd7\u91cc\u4f5c\u4e3a\u5143\u4fe1\u606f\u3002
\u540c\u65f6\u76f8\u6bd4\u7c97\u66b4\u7684\u6240\u6709\u8282\u70b9\u6302\u8f7d\u76f8\u540c\u8def\u5f84\u8fdb\u884c\u901a\u914d\u91c7\u96c6\u7684\u65b9\u5f0f\uff0c\u4e5f\u89e3\u51b3\u4e86\u6ca1\u6cd5\u9488\u5bf9\u5355\u4e2a\u670d\u52a1\u7cbe\u7ec6\u5316\u914d\u7f6e\u3001\u91c7\u96c6\u65e0\u5173\u65e5\u5fd7\u7684\u95ee\u9898\u3002
\u5f53\u7136\u5e26\u6765\u7684\u597d\u5904\u4e0d\u4ec5\u4ec5\u8fd9\u4e9b\uff0cLoggie\u5728\u52a8\u6001\u914d\u7f6e\u4e0b\u53d1\u3001\u76d1\u63a7\u6307\u6807\u7b49\u65b9\u9762\u90fd\u53ef\u4ee5\u57fa\u4e8eKubernetes\u8fdb\u884c\u76f8\u5e94\u7684\u9002\u914d\u548c\u652f\u6301\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#crd","title":"CRD\u4f7f\u7528\u8bf4\u660e","text":"Loggie\u76ee\u524d\u6709\u4ee5\u4e0b\u51e0\u79cdCRD\uff1a
\u4f7f\u7528CRD\u7684\u6d41\u7a0b\u67b6\u6784\u5982\u4e0b\u6240\u793a\uff1a
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#_1","title":"\u51c6\u5907\u5de5\u4f5c","text":""},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#_2","title":"\u90e8\u7f72\u67b6\u6784","text":"\u6574\u4f53\u7684\u90e8\u7f72\u67b6\u6784\u662f\u4ec0\u4e48\uff1f
\u672c\u6587\u4ec5\u5173\u6ce8\u91c7\u96c6\u7aef\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u90e8\u7f72Loggie Aggregator\uff0c\u8bf7\u53c2\u8003Loggie\u4e2d\u8f6c\u673a\u3002
\u5728\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u4e4b\u524d\uff0c\u8bf7\u786e\u4fdd\u5df2\u7ecf\u5728Kubernetes\u4e2d\u90e8\u7f72\u4e86Loggie DaemonSet\u3002\u5982\u4f55\u5728Kubernetes\u4e2d\u90e8\u7f72Loggie\uff1f
Info
\u6211\u4eec\u63a8\u8350\u4f7f\u7528DaemonSet\u7684\u65b9\u5f0f\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0c\u540e\u7eedLoggie\u8ba1\u5212\u652f\u6301\u81ea\u52a8\u6ce8\u5165Loggie Sidecar\u7684\u65b9\u5f0f\u91c7\u96c6\u65e5\u5fd7\u3002RoadMap
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#_3","title":"\u4f7f\u7528\u65b9\u5f0f","text":"\u4e1a\u52a1Pod\u4f7f\u7528\u4ec0\u4e48\u65b9\u5f0f\u6302\u8f7d\u65e5\u5fd7\uff1f \u5efa\u8bae\u5982\u679c\u5bf9\u65e5\u5fd7\u53ef\u80fd\u6027\u7684\u4e22\u5931\u4e0d\u654f\u611f\uff0c\u6bd4\u5982\u8fd0\u7ef4\u7c7b\u7684\u65e5\u5fd7\uff0c\u53ef\u4ee5\u4f7f\u7528emptyDir\u3002\u5982\u679c\u65e5\u5fd7\u6bd4\u8f83\u91cd\u8981\uff0c\u4e0d\u5141\u8bb8\u4e22\u5931\uff0c\u8bf7\u4f7f\u7528hostPath\uff0c\u5e76\u5728volumeMount\u914d\u7f6esubPathExpr\u5b9e\u73b0\u8def\u5f84\u9694\u79bb\u3002
Example
tomcat-emptydir.ymlapiVersion: apps/v1\nkind: Deployment\nmetadata:\nlabels:\napp: tomcat\nname: tomcat\nnamespace: default\nspec:\nreplicas: 1\nselector:\nmatchLabels:\napp: tomcat\ntemplate:\nmetadata:\nlabels:\napp: tomcat\nspec:\ncontainers:\n- name: tomcat\nimage: tomcat\nvolumeMounts:\n- mountPath: /usr/local/tomcat/logs\nname: log\nvolumes:\n- emptyDir: {}\nname: log apiVersion: apps/v1\nkind: Deployment\nmetadata:\nlabels:\napp: tomcat\nname: tomcat\nnamespace: default\nspec:\nreplicas: 1\nselector:\nmatchLabels:\napp: tomcat\ntemplate:\nmetadata:\nlabels:\napp: tomcat\nspec:\ncontainers:\n- env:\n- name: POD_NAME\nvalueFrom:\nfieldRef:\napiVersion: v1\nfieldPath: metadata.name\n- name: NAMESPACE\nvalueFrom:\nfieldRef:\napiVersion: v1\nfieldPath: metadata.namespace\nimage: tomcat\nname: tomcat\nvolumeMounts:\n- mountPath: /log\nname: datalog\nsubPathExpr: $(NAMESPACE)/$(POD_NAME)\nvolumes:\n- hostPath:\npath: /data/log\ntype: \"\"\nname: datalog\nCaution
\u8bf7\u786e\u4fddPod\u6302\u8f7d\u7684\u65e5\u5fd7\u8def\u5f84\u4e0d\u4f1a\u88ab\u591a\u4e2aPod\u5171\u4eab\u3002\u6bd4\u5982\u4e00\u4e2aDeployment\u4e0b\u7684\u4e24\u4e2aPod\uff0c\u4f7f\u7528\u672a\u914d\u7f6esubPathExpr\u7684hostPath\u6302\u8f7d\u65e5\u5fd7\u76ee\u5f55\uff0c\u5982\u679c\u4e24\u4e2aPod\u8c03\u5ea6\u5728\u76f8\u540c\u7684\u8282\u70b9\uff0c\u4f1a\u6253\u5370\u5230\u76f8\u540c\u7684\u65e5\u5fd7\u76ee\u5f55\u548c\u6587\u4ef6\u4e2d\uff0c\u6b64\u65f6\u53ef\u80fd\u5f15\u8d77\u91c7\u96c6\u5f02\u5e38\u3002
\u6211\u4eec\u9996\u5148\u521b\u5efa\u4ee5\u4e0a\u793a\u4f8b\u7684tomcat Deployment\u4f5c\u4e3a\u6211\u4eec\u88ab\u91c7\u96c6\u65e5\u5fd7\u7684\u5bf9\u8c61\u3002
kubectl apply -f tomcat-emptydir.yml\n\u63a5\u4e0b\u6765\u5c06\u6f14\u793a\u91c7\u96c6\u4e0a\u9762\u793a\u4f8b\u7684tomcat\u65e5\u5fd7\uff0c\u5e76\u53d1\u9001\u81f3Elasticsearch\u3002 \u5982\u679c\u4f60\u5e0c\u671b\u91c7\u96c6\u5176\u4ed6\u7684\u5bb9\u5668\u65e5\u5fd7\uff0c\u6216\u8005\u53d1\u9001\u81f3\u5176\u4ed6\u7684\u540e\u7aef\uff0c\u53ea\u9700\u8981\u4fee\u6539\u5176\u4e2d\u7684\u914d\u7f6e\u5373\u53ef\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#elasticsearchkibana","title":"\u90e8\u7f72Elasticsearch\u548cKibana\uff08\u53ef\u9009\uff09","text":"\u7531\u4e8e\u672c\u6587\u6f14\u793a\u53d1\u9001\u81f3Elasticsearch\uff0c\u6240\u4ee5\u8fd9\u91cc\u6211\u4eec\u90e8\u7f72\u4e00\u4e0bElasticsearch\u548cKibana\u3002 \u5982\u679c\u4f60\u7684\u73af\u5883\u4e2d\u5df2\u7ecf\u6709Elasticsearch\u548cKibana\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u5ffd\u7565\u3002 \u5982\u679c\u5e0c\u671b\u81ea\u5df1\u642d\u5efa\u4e00\u5957Elasticsearch\u548cKibana\u8bd5\u7528\uff0c\u8fd9\u91cc\u5efa\u8bae\uff1a
\u5982\u679c\u672c\u5730\u6ca1\u6709helm\u5ba2\u6237\u7aef\uff0c\u9700\u8981\u4e0b\u8f7dhelm\u3002
\u4f7f\u7528\u5982\u4e0b\u547d\u4ee4:
helm repo add elastic https://helm.elastic.co\nhelm install elasticsearch elastic/elasticsearch --set replicas=1\nhelm install kibana elastic/kibana\n\u901a\u8fc7\uff1a
kubectl port-forward service/kibana-kibana 5601:http\n\u53ef\u4ee5\u76f4\u63a5\u5728\u6d4f\u89c8\u5668\u8f93\u5165localhost:5601\uff0c\u8bbf\u95eeKibana\u9875\u9762\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#_4","title":"\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7","text":""},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#sink","title":"\u521b\u5efasink","text":"\u4e3a\u4e86\u8868\u660e\u6211\u4eec\u5373\u5c06\u91c7\u96c6\u65e5\u5fd7\u53d1\u9001\u5230\u7684Elasticsearch\uff0c\u9700\u8981\u914d\u7f6e\u5bf9\u5e94\u7684Sink\u3002 \u8fd9\u91cc\u6709\u4e24\u79cd\u65b9\u5f0f\uff1a
\u5982\u679c\u6574\u4e2a\u96c6\u7fa4\u53ea\u4f1a\u6709\u4e00\u4e2a\u5b58\u50a8\u540e\u7aef\uff0c\u6211\u4eec\u53ef\u4ee5\u5728\u5168\u5c40\u7684\u914d\u7f6e\u6587\u4ef6configMap\u91cc\uff0c\u914d\u7f6edefaults\u53c2\u6570\uff0c\u5177\u4f53\u53ef\u53c2\u8003\u3002
\u4f7f\u7528Sink CRD\uff0c\u5e76\u5728logConfig\u4e2d\u5f15\u7528\u3002\u8fd9\u79cd\u65b9\u5f0f\u53ef\u4ee5\u6269\u5c55\u4e3a\u591a\u4e2a\u540e\u7aef\uff0c\u4e0d\u540c\u7684logConfig\u53ef\u4ee5\u914d\u7f6e\u4f7f\u7528\u4e0d\u540c\u7684\u540e\u7aef\u5b58\u50a8\uff0c\u5927\u591a\u6570\u60c5\u51b5\u4e0b\uff0c\u6211\u4eec\u5efa\u8bae\u4f7f\u7528\u8be5\u65b9\u5f0f\u3002
\u521b\u5efa\u4e00\u4e2aSink\u5982\u4e0b\uff0cSink\u4e3a\u96c6\u7fa4\u7ea7\u522bCRD\u3002\u53ef\u4ee5\u5728spec.sink\u4e2d\u4fee\u6539\u4e3a\u5176\u4ed6\u914d\u7f6e\u3002
Example
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: Sink\nmetadata:\nname: default\nspec:\nsink: |\ntype: elasticsearch\nindex: \"loggie\"\nhosts: [\"elasticsearch-master.default.svc:9200\"]\nEOF\n\u53ef\u4ee5\u901a\u8fc7kubectl get sink default\u67e5\u770b\u662f\u5426\u521b\u5efa\u6210\u529f\u3002
\u548cSink\u4e00\u6837\uff0c\u914d\u7f6einterceptor\u6709\u4e24\u79cd\u65b9\u5f0f\uff1a
\u5728\u5168\u5c40\u914d\u7f6e\u6587\u4ef6configMap\u4e2d\uff0c\u914d\u7f6edefaults\u53c2\u6570\u3002\u6b64\u65f6\u610f\u5473\u7740\u6240\u6709\u7684Loggie Agent\u90fd\u4f7f\u7528\u8be5default interceptors\u914d\u7f6e\u3002
\u4f7f\u7528Interceptor CRD\uff0c\u5e76\u5728logConfig\u4e2d\u88ab\u5f15\u7528\u3002\u8fd9\u79cd\u65b9\u5f0f\u4f1a\u66f4\u52a0\u7075\u6d3b\uff0c\u9002\u5408\u6709\u4e00\u5b9a\u8fd0\u7ef4\u548c\u529f\u80fd\u9700\u6c42\u7684\u573a\u666f\u3002
Loggie\u76ee\u524d\u9ed8\u8ba4\u5185\u7f6e\u4e86metric\uff08\u76d1\u63a7\u6307\u6807\u53d1\u9001\uff09\u3001 maxbytes\uff08\u6700\u5927event\u5b57\u8282\u6570\u9650\u5236\uff09\u3001 retry\uff08\u91cd\u8bd5\uff09\u4e09\u4e2ainterceptors\u3002\u5373\u4f7f\u6211\u4eec\u6ca1\u6709\u4f7f\u7528\u4ee5\u4e0a\u4e24\u79cd\u65b9\u5f0f\u914d\u7f6e\uff0c\u4ecd\u7136\u4f1a\u81ea\u52a8\u4f7f\u7528\u8fd93\u4e2a\u9ed8\u8ba4interceptor\uff0c\u6240\u4ee5interceptor\u5176\u5b9e\u662f\u53ef\u9009\u9879\u3002
Caution
\u521b\u5efa\u4e00\u4e2ainterceptor\u793a\u4f8b\u5982\u4e0b\uff1a
Example
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: Interceptor\nmetadata:\nname: default\nspec:\ninterceptors: |\n- type: rateLimit\nqps: 90000\nEOF\n\u8fd9\u91cc\u6211\u4eec\u521b\u5efa\u4e86\u4e00\u4e2arateLimit interceptor\uff0c\u53ef\u7528\u4e8e\u91c7\u96c6\u53d1\u9001\u65e5\u5fd7\u7684\u9650\u6d41\u3002 \u53ef\u4ee5\u901a\u8fc7kubectl get interceptor default\u6216\u8005kubectl get icp default\u67e5\u770b\u662f\u5426\u521b\u5efa\u6210\u529f\u3002
\u521b\u5efa\u5b8csink\u548cinterceptor\u540e\uff0c\u6700\u91cd\u8981\u7684\u4fbf\u662f\u521b\u5efalogConfig\uff0c\u8868\u660e\u6211\u4eec\u5e0c\u671b\u91c7\u96c6\u54ea\u4e9bPod\u7684\u65e5\u5fd7\uff0c\u4ee5\u53ca\u91c7\u96c6Pod\u7684\u54ea\u4e9b\u65e5\u5fd7\u3002
logConfig\u91ccspec.selector\u90e8\u5206\u8868\u793a\u65e5\u5fd7\u914d\u7f6e\u7684\u4e0b\u53d1\u8303\u56f4\uff0c\u5bf9\u91c7\u96c6Pod\u65e5\u5fd7\u6765\u8bf4\uff0c\u9700\u8981\u914d\u7f6etype: pod\uff0c\u5e76\u4e14\u4f7f\u7528labelSelector\u9009\u62e9\u6307\u5b9a\u7684Pods\uff0c\u5373\u91c7\u96c6\u8fd9\u4e9bPods\u7684\u65e5\u5fd7\u3002 \u793a\u4f8b\u5982\u4e0b\uff1a
spec:\nselector:\ntype: pod\nlabelSelector:\napp: tomcat\n\u53e6\u5916\uff0c\u6211\u4eec\u9700\u8981\u5728pipeline.sources\u4e2d\u914d\u7f6efile source\uff0c\u5373\u91c7\u96c6Pods\u7684\u54ea\u4e9b\u65e5\u5fd7\u3002
pipeline:\nsources: |\n- type: file\nname: mylog\npaths:\n- stdout\n- /var/log/*.log\nstdout\uff0c\u8868\u793a\u91c7\u96c6\u5bb9\u5668\u7684\u6807\u51c6\u8f93\u51fa\u8def\u5f84\u3002type: pod\u65f6\uff0cpipeline.name\u4f1a\u88abloggie\u81ea\u52a8\u6309\u7167${namespace}-${logConfigName}\u7684\u683c\u5f0f\u751f\u6210\uff0c\u6240\u4ee5\u8fd9\u91cc\u65e0\u9700\u586b\u5199\u3002 ${podName}-${containerName}-${sourceName}\u683c\u5f0f\u751f\u6210\u3002 \u6700\u540e\uff0c\u6211\u4eec\u4f7f\u7528sinkRef\u548cinterceptorRef\u5f15\u7528\u4e0a\u9762\u521b\u5efa\u7684sink\u548cinterceptor\u3002 \u6700\u7ec8\u521b\u5efa\u4e00\u4e2alogConfig\u793a\u4f8b\u5982\u4e0b\uff1a
Example
cat << EOF | kubectl apply -f -\napiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\nname: tomcat\nnamespace: default\nspec:\nselector:\ntype: pod\nlabelSelector:\napp: tomcat\npipeline:\nsources: |\n- type: file\nname: common\npaths:\n- stdout\n- /usr/local/tomcat/logs/*.log\nsinkRef: default\ninterceptorRef: default\nEOF\n\u6211\u4eec\u53ef\u4ee5\u901a\u8fc7kubectl get logconfig tomcat\u6216\u8005kubectl get lgc tomcat\u6765\u67e5\u770b\u662f\u5426\u521b\u5efa\u6210\u529f\u3002 \u540c\u65f6\u53ef\u4ee5\u901a\u8fc7kubectl describe lgc tomcat\u67e5\u770blogConfig\u7684events\uff0c\u7528\u4e8e\u6392\u67e5\u72b6\u6001\u3002 \u5982\u679c\u6709\u4ee5\u4e0b\u7c7b\u4f3c\u7684events\uff0c\u8bf4\u660e\u914d\u7f6e\u5df2\u7ecf\u4e0b\u53d1\u6210\u529f\u3002
Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal syncSuccess 55s loggie/kind-control-plane Sync type pod [tomcat-684c698b66-hvztn] success\n\u6211\u4eec\u8fd8\u53ef\u4ee5\u901a\u8fc7kubectl -nloggie logs -f ${loggie-name}\u6765\u67e5\u770b\u6307\u5b9a\u8282\u70b9Loggie\u7684\u65e5\u5fd7\u6765\u5224\u65ad\u65e5\u5fd7\u91c7\u96c6\u60c5\u51b5\u3002
\u53d1\u9001\u6210\u529f\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u5728Kibana\u4e0a\u67e5\u8be2\u5230\u91c7\u96c6\u5230\u7684\u65e5\u5fd7\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-container-logs/#_5","title":"\u81ea\u52a8\u89e3\u6790\u5bb9\u5668\u6807\u51c6\u8f93\u51fa\u539f\u59cb\u65e5\u5fd7","text":"\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6211\u4eec\u91c7\u96c6\u5230\u7684\u6807\u51c6\u8f93\u51fa\u5e76\u4e0d\u662f\u6253\u5370\u7684\u65e5\u5fd7\u5185\u5bb9\uff0c\u800c\u662f\u88ab\u5bb9\u5668\u8fd0\u884c\u65f6\u589e\u52a0\u4e86\u4e00\u5c42\u5c01\u88c5\u3002
\u4f8b\u5982docker\u7684\u6807\u51c6\u8f93\u51fa\u4e3ajson\u5f62\u5f0f\uff1a
{\"log\":\"I0610 08:29:07.698664 Waiting for caches to sync\\n\", \"stream\":\"stderr\", \"time:\"2021-06-10T08:29:07.698731204Z\"}\ncontainerd\u7684\u6807\u51c6\u8f93\u51fa\u5f62\u5f0f\u7c7b\u4f3c\u5982\u4e0b\uff1a
2021-12-01T03:13:58.298476921Z stderr F INFO [main] Starting service [Catalina]\n2021-12-01T03:13:58.298476921Z stderr F\u4e3a\u8fd0\u884c\u65f6\u589e\u52a0\u7684\u524d\u7f00\u5185\u5bb9\uff0c\u540e\u9762\u5219\u4e3a\u539f\u59cb\u7684\u65e5\u5fd7\u3002 \u7279\u522b\u662f\u5982\u679c\u6211\u4eec\u914d\u7f6e\u4e86\u65e5\u5fd7\u91c7\u96c6\u591a\u884c\u914d\u7f6e\uff0c\u56e0\u4e3a\u91c7\u96c6\u5230\u7684\u65e5\u5fd7\u5185\u5bb9\u548c\u4e1a\u52a1\u8f93\u51fa\u7684\u65e5\u5fd7\u4e0d\u4e00\u81f4\uff0c\u4f1a\u5bfc\u81f4\u91c7\u96c6\u6807\u51c6\u8f93\u51fa\u65e5\u5fd7\u5339\u914d\u591a\u884c\u6709\u95ee\u9898\u3002
\u6240\u4ee5\uff0cLoggie\u63d0\u4f9b\u4e86\u4e00\u952e\u5f00\u5173\u914d\u7f6e\uff0c\u5728\u7cfb\u7edf\u914d\u7f6e\u4e2d\uff0c\u5c06parseStdout\u53c2\u6570\u8bbe\u7f6e\u4e3atrue\u5373\u53ef\u3002
parseStdout
config:\nloggie:\ndiscovery:\nenabled: true\nkubernetes:\nparseStdout: true\nLoggie\u4f1a\u5728\u6e32\u67d3LogConfig\u7684\u65f6\u5019\u81ea\u52a8\u589e\u52a0source codec\u89e3\u6790\u51fa\u539f\u59cb\u7684\u4e1a\u52a1\u65e5\u5fd7\u3002
\u9700\u8981\u6ce8\u610f\u7684\u662f\uff1a
\u4ec5\u5f53LogConfig\u914d\u7f6e\u91cc\u7684paths\u5355\u72ec\u4e3astdout\u65f6\u6709\u6548\uff1a \u3010\u6709\u6548\u3011
sources: |\n- type: file\nname: common\npaths:\n- stdout\n sources: |\n- type: file\nname: common\npaths:\n- stdout\n- /usr/local/tomcat/logs/*.log\n sources: |\n- type: file\nname: stdout\npaths:\n- stdout\n- type: file\nname: tomcat\npaths:\n- /usr/local/tomcat/logs/*.log\n\u76ee\u524d\u53ea\u4f1a\u4fdd\u7559\u539f\u59cb\u7684\u65e5\u5fd7\u5185\u5bb9\u5230body\u4e2d\uff0c\u5176\u4f59\u8fd0\u884c\u65f6\u9644\u52a0\u7684\u5b57\u6bb5\u4f1a\u88ab\u4e22\u5f03\u3002
\u867d\u7136\u6211\u4eec\u5efa\u8bae\u4f7f\u7528\u6302\u8f7dvolume(emptyDir/hostPath+subPathExpr)\u7684\u65b9\u5f0f\u5c06\u65e5\u5fd7\u6587\u4ef6\u6302\u51fa\u7ed9Loggie\u91c7\u96c6\uff0c\u4f46\u4ecd\u7136\u5b58\u5728\u5f88\u591a\u60c5\u51b5\u6211\u4eec\u6ca1\u529e\u6cd5\u5c06\u4e1a\u52a1\u7684Pod\u7edf\u4e00\u6302\u8f7d\u51fa\u65e5\u5fd7\u8def\u5f84\u3002 \u6bd4\u5982\u4e00\u4e9b\u57fa\u7840\u7ec4\u4ef6\u65e0\u6cd5\u914d\u7f6e\u72ec\u7acb\u7684\u65e5\u5fd7volume\uff0c\u6216\u8005\u4e1a\u52a1\u5355\u7eaf\u7684\u4e0d\u613f\u610f\u6539\u90e8\u7f72\u914d\u7f6e\u3002
Loggie\u63d0\u4f9b\u4e86\u65e0\u9700\u6302\u8f7d\u5373\u53ef\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u7684\u80fd\u529b\uff0c\u53ef\u81ea\u52a8\u8bc6\u522b\u5e76\u91c7\u96c6\u5bb9\u5668root filesystem\u91cc\u7684\u65e5\u5fd7\u6587\u4ef6\u3002
\u4f60\u53ea\u9700\u8981\u5c06\u90e8\u7f72\u7684helm chart\u4e2dvalues.yml\u91cc\u7684\u914d\u7f6erootFsCollectionEnabled\u8bbe\u7f6e\u4e3atrue\uff0c \u540c\u65f6\u586b\u4e0a\u5b9e\u9645\u73af\u5883\u7684\u5bb9\u5668\u8fd0\u884c\u65f6\uff08docker/containerd\uff09\uff0c\u5982\u4e0b\u6240\u793a\uff1a
rootFsCollectionEnabled
config:\nloggie:\ndiscovery:\nenabled: true\nkubernetes:\ncontainerRuntime: containerd\nrootFsCollectionEnabled: false\n\u4fee\u6539\u5b8c\u6210\u540e\uff0c\u91cd\u65b0helm upgrade\u5373\u53ef\u3002 helm\u6a21\u7248\u4f1a\u81ea\u52a8\u6e32\u67d3\u589e\u52a0\u989d\u5916\u7684\u4e00\u4e9b\u6302\u8f7d\u8def\u5f84\u548c\u914d\u7f6e\uff0c\u5982\u679c\u4f60\u4ece\u4f4e\u7248\u672c\u5347\u7ea7\uff0c\u9700\u8981\u989d\u5916\u4fee\u6539\u90e8\u7f72\u7684Daemonset yaml\u3002 \u5177\u4f53\u539f\u7406\u8bf7\u53c2\u8003issues #208\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-node-logs/","title":"\u4f7f\u7528Loggie\u91c7\u96c6Node\u8282\u70b9\u65e5\u5fd7","text":"\u5728Kubernetes\u96c6\u7fa4\u4e2d\uff0c\u9664\u4e86\u91c7\u96c6Pod\u91cc\u7684\u65e5\u5fd7\uff0c\u8fd8\u53ef\u80fd\u6709\u91c7\u96c6Node\u8282\u70b9\u4e0a\u7684\u4e00\u4e9b\u8bf8\u5982kubelet\u65e5\u5fd7\u3001\u7cfb\u7edf\u65e5\u5fd7\u7b49\u9700\u6c42\u3002
"},{"location":"user-guide/use-in-kubernetes/collect-node-logs/#_1","title":"\u914d\u7f6e\u8bf4\u660e","text":"\u548c\u91c7\u96c6\u5bb9\u5668\u4e0d\u540c\u7684\u662f\uff0c\u8282\u70b9\u65e5\u5fd7\u91c7\u96c6\u9700\u4f7f\u7528\u96c6\u7fa4\u7ea7\u522b\u7684ClusterLogConfig\uff0cselector\u4f7f\u7528type: node\uff0c\u5e76\u4e14\u586b\u5199nodeSelector\u7528\u4e8e\u9009\u62e9\u4e0b\u53d1\u914d\u7f6e\u5230\u54ea\u4e9b\u8282\u70b9\uff0c\u540c\u65f6\u9700\u786e\u4fddNode\u4e0a\u5305\u542b\u8fd9\u4e9blabels\u3002
\u793a\u4f8b\u5982\u4e0b\uff1a
Example
apiVersion: loggie.io/v1beta1\nkind: ClusterLogConfig\nmetadata:\nname: varlog\nspec:\nselector:\ntype: node\nnodeSelector:\nnodepool: demo\npipeline:\nsources: |\n- type: file\nname: varlog\npaths:\n- /var/log/*.log\nsinkRef: default\ninterceptorRef: default\n\u53e6\u5916\u5e94\u6ce8\u610f\u7684\u662f\uff0c\u5982\u679c\u9700\u8981\u91c7\u96c6Node\u8282\u70b9\u4e0a\u67d0\u8def\u5f84\u7684\u65e5\u5fd7\uff0c\u9700\u8981Loggie\u540c\u6837\u6302\u8f7d\u76f8\u540c\u7684\u8def\u5f84\uff0c\u5426\u5219\u7531\u4e8e\u5bb9\u5668\u9694\u79bb\u6027Loggie\u65e0\u6cd5\u83b7\u53d6\u5230\u8282\u70b9\u7684\u65e5\u5fd7\u3002 \u6bd4\u5982\u91c7\u96c6Node\u4e0a/var/log/\u8def\u5f84\u4e0b\u7684\u65e5\u5fd7\uff0c\u9700\u8981Loggie Agent\u589e\u52a0\u6302\u8f7d\u8be5\u8def\u5f84\u3002
Cite
\u76f8\u6bd4\u4f20\u7edf\u7684\u4e3b\u673a\u65e5\u5fd7\u91c7\u96c6\uff0c\u5728Kubernetes\u96c6\u7fa4\u4e2d\uff0c\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u6709\u4e00\u4e9b\u5dee\u5f02\uff0c\u4f7f\u7528\u65b9\u5f0f\u4e0a\u4e5f\u6709\u6240\u533a\u522b\u3002 \u8fd9\u91cc\u6211\u4eec\u7f57\u5217\u4e86\u4e00\u4e9b\u5e38\u89c4\u7684\u90e8\u7f72\u548c\u4f7f\u7528\u65b9\u5f0f\u4ee5\u4f9b\u53c2\u8003\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#1","title":"1. \u4ece\u4e3b\u673a\u5230\u5bb9\u5668","text":"\u5728\u4f20\u7edf\u7684\u4f7f\u7528\u865a\u62df\u673a/\u4e91\u4e3b\u673a/\u7269\u7406\u673a\u7684\u65f6\u4ee3\uff0c\u4e1a\u52a1\u8fdb\u7a0b\u90e8\u7f72\u5728\u56fa\u5b9a\u7684\u8282\u70b9\u4e0a\uff0c\u4e1a\u52a1\u65e5\u5fd7\u76f4\u63a5\u8f93\u51fa\u5230\u5bbf\u4e3b\u673a\u4e0a\uff0c\u8fd0\u7ef4\u53ea\u9700\u8981\u624b\u52a8\u6216\u8005\u4f7f\u7528\u81ea\u52a8\u5316\u5de5\u5177\u628a\u65e5\u5fd7\u91c7\u96c6Agent\u90e8\u7f72\u5728\u8282\u70b9\u4e0a\uff0c\u52a0\u4e00\u4e0bAgent\u7684\u914d\u7f6e\uff0c\u5c31\u53ef\u4ee5\u5f00\u59cb\u91c7\u96c6\u65e5\u5fd7\u4e86\u3002
\u800c\u5728Kubernetes\u73af\u5883\u4e2d\uff0c\u60c5\u51b5\u5c31\u6ca1\u8fd9\u4e48\u7b80\u5355\u4e86\uff1a
\u4ee5\u4e0a\u90fd\u662f\u6709\u522b\u4e8e\u4f20\u7edf\u65e5\u5fd7\u91c7\u96c6\u914d\u7f6e\u65b9\u5f0f\u7684\u9700\u6c42\u548c\u75db\u70b9\uff0c\u7a76\u5176\u539f\u56e0\uff0c\u8fd8\u662f\u56e0\u4e3a\u4f20\u7edf\u7684\u65b9\u5f0f\u8131\u79bb\u4e86Kubernetes\uff0c\u65e0\u6cd5\u611f\u77e5Kubernetes\uff0c\u65e0\u6cd5\u548cKubernetes\u96c6\u6210\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#2-kubernetes","title":"2. \u5728Kubernetes\u4e0b\u7684\u65e5\u5fd7\u5f62\u6001","text":"\u4e3a\u4e86\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0c\u6211\u4eec\u5148\u6765\u770b\u4e00\u4e0b\u5e02\u9762\u4e0a\u4e00\u822c\u90fd\u6709\u54ea\u4e9b\u89e3\u51b3\u65b9\u6848\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#21","title":"2.1 \u91c7\u96c6\u7684\u65e5\u5fd7\u7c7b\u578b","text":"\u9996\u5148\uff0c\u9700\u8981\u63d0\u53ca\u7684\u662f\uff0c\u5728\u4e91\u539f\u751f\u768412\u8981\u7d20\u91cc\uff0c\u63a8\u8350\u4e1a\u52a1\u5bb9\u5668\u5c06\u65e5\u5fd7\u8f93\u51fa\u5230stdout\u4e2d\uff0c\u800c\u4e0d\u662f\u91c7\u7528\u6253\u5370\u65e5\u5fd7\u6587\u4ef6\u7684\u65b9\u5f0f\u3002\u5f53\u7136\uff0c\u5b9e\u9645\u60c5\u51b5\u662f\uff0c\u6211\u4eec\u5f88\u96be\u8fd9\u4e48\u505a\uff0c\u539f\u56e0\u5927\u6982\u6709\uff1a
\u6240\u4ee5\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6211\u4eec\u9700\u8981\u540c\u65f6\u91c7\u96c6\uff1a
\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0cAgent\u6709\u4e24\u79cd\u90e8\u7f72\u65b9\u5f0f\uff1a
\u4e24\u79cd\u90e8\u7f72\u65b9\u5f0f\u7684\u4f18\u52a3\u90fd\u663e\u800c\u6613\u89c1\uff1a
Tip
\u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u4f18\u5148\u4f7f\u7528DaemonSet\u7684\u65b9\u5f0f\u91c7\u96c6\u65e5\u5fd7\uff0c\u5982\u679c\u5355\u4e2aPod\u65e5\u5fd7\u91cf\u7279\u522b\u5927\uff0c\u8d85\u8fc7\u4e00\u822cAgent\u53d1\u9001\u541e\u5410\u91cf\uff0c\u53ef\u4ee5\u5355\u72ec\u5bf9\u8be5Pod\u4f7f\u7528Sidecar\u7684\u65b9\u5f0f\u91c7\u96c6\u65e5\u5fd7\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#23","title":"2.3 \u91c7\u96c6\u65b9\u5f0f","text":""},{"location":"user-guide/use-in-kubernetes/general-usage/#daemonset-stdout","title":"DaemonSet + Stdout","text":"\u5982\u679c\u4f7f\u7528\u5bb9\u5668\u8fd0\u884c\u65f6\u7684\u662fdocker\uff0c\u6b63\u5e38\u60c5\u51b5\u4e0b\u6211\u4eec\u53ef\u4ee5\u5728\u8282\u70b9\u7684docker\u8def\u5f84\u4e2d\u627e\u5230\u5bb9\u5668\u7684stdout\u7684\u65e5\u5fd7\uff0c\u9ed8\u8ba4\u4e3a/var/lib/docker/containers/{containerId}/{containerId}-json.log\u3002
\u5728Kubernetes 1.14\u7248\u672c\u4e4b\u524d\uff0ckubelet\u4f1a\u5728/var/log/pods/<podUID>/<containerName>/<num>.log\u5efa\u7acb\u4e00\u4e2a\u8f6f\u94fe\u63a5\u5230stdout\u6587\u4ef6\u4e2d\u3002
\u7c7b\u4f3c\u5982\u4e0b\u6240\u793a\uff1a
root@master0:/var/log/pods# tree .\n|-- 6687e53201c01e3fad31e7d72fbb92a6\n| `-- kube-apiserver\n| |-- 865.log -> /var/lib/docker/containers/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03-json.log\n| `-- 866.log -> /var/lib/docker/containers/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662-json.log\n|-- a1083c6d-3b12-11ea-9af1-fa163e28f309\n| `-- kube-proxy\n| |-- 3.log -> /var/lib/docker/containers/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e-json.log\n| `-- 4.log -> /var/lib/docker/containers/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207-json.log\n\u5728Kubernetes 1.14\u7248\u672c\u4e4b\u540e\uff0c\u6539\u6210\u4e86/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/<num>.log\u7684\u5f62\u5f0f\u3002
root@master-0:/var/log/pods# tree .\n|-- kube-system_kube-apiserver-kind-control-plane_bd1c21fe1f0ef615e0b5e41299f1be61\n| `-- kube-apiserver\n| `-- 0.log\n|-- kube-system_kube-proxy-gcrfq_f07260b8-6055-4c19-9491-4a825579528f\n| `-- kube-proxy\n| `-- 0.log\n`-- loggie_loggie-csd4g_f1cc32e9-1002-4e64-bd58-fc6094394e06\n `-- loggie\n `-- 0.log\n\u6240\u4ee5\uff0c\u5bf9\u4e8eAgent\u91c7\u96c6\u6807\u51c6\u8f93\u51fa\u65e5\u5fd7\u6765\u8bf4\uff0c\u4e5f\u5c31\u662f\u91c7\u96c6\u8282\u70b9\u4e0a\u7684\u8fd9\u4e9b\u65e5\u5fd7\u6587\u4ef6\u3002
\u4e00\u79cd\u7b80\u5355\u7c97\u66b4\u7684\u91c7\u96c6\u65b9\u5f0f\u662f\uff0c\u4f7f\u7528DaemonSet\u90e8\u7f72\u65e5\u5fd7Agent\uff0c\u6302\u8f7d/var/log/pods\u76ee\u5f55\uff0cAgent\u7684\u914d\u7f6e\u6587\u4ef6\u4f7f\u7528\u7c7b\u4f3c/var/log/pod/*/*.log\u53bb\u901a\u914d\u65e5\u5fd7\u6587\u4ef6\uff0c\u91c7\u96c6\u8282\u70b9\u4e0a\u6240\u6709\u7684\u5bb9\u5668\u6807\u51c6\u8f93\u51fa\u3002
\u4f46\u662f\u8fd9\u6837\u7684\u5c40\u9650\u5728\u4e8e\uff1a
\u5f53\u7136\u73b0\u5728\u7684\u4e00\u4e9b\u65e5\u5fd7Agent\u6bd4\u5982Filebeat/Fluentd\u90fd\u9488\u5bf9\u6027\u7684\u505a\u4e86\u652f\u6301\uff0c\u6bd4\u5982\u53ef\u4ee5\u5c06namespace/pod\u7b49\u4fe1\u606f\u6ce8\u5165\u65e5\u5fd7\u4e2d\uff0c\u4f46\u4ecd\u7136\u6ca1\u6709\u89e3\u51b3\u5927\u90e8\u5206\u7684\u95ee\u9898\u3002
\u6240\u4ee5\uff0c\u8fd9\u79cd\u65b9\u5f0f\u53ea\u9002\u5408\u7b80\u5355\u7684\u4e1a\u52a1\u573a\u666f\uff0c\u540e\u7eed\u4e5f\u96be\u4ee5\u6ee1\u8db3\u5176\u4ed6\u66f4\u591a\u7684\u65e5\u5fd7\u9700\u6c42\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#daemonset","title":"DaemonSet + \u65e5\u5fd7\u6587\u4ef6","text":"\u5982\u679cPod\u91cc\u4e0d\u4ec5\u4ec5\u662f\u8f93\u51fastdout\uff0c\u8fd8\u5305\u62ec\u65e5\u5fd7\u6587\u4ef6\uff0c\u5c31\u9700\u8981\u8003\u8651\u5230\u6302\u8f7d\u65e5\u5fd7\u6587\u4ef6\u5230\u8282\u70b9\u4e0a\uff0c\u540c\u65f6\u91c7\u7528DaemonSet\u90e8\u7f72\u7684Agent\u4e5f\u9700\u8981\u6302\u8f7d\u76f8\u540c\u7684\u76ee\u5f55\uff0c\u5426\u5219\u91c7\u7528\u5bb9\u5668\u5316\u90e8\u7f72\u7684Agent\u65e0\u6cd5\u67e5\u770b\u5230\u76f8\u5e94\u7684\u6587\u4ef6\uff0c\u66f4\u65e0\u6cd5\u91c7\u96c6\u3002
\u4e1a\u52a1Pod\u6302\u8f7d\u65e5\u5fd7\u8def\u5f84\u7684\u65b9\u5f0f\u6709\u4ee5\u4e0b\u51e0\u79cd\uff1a
(1) emtpyDir
emtpyDir\u7684\u751f\u547d\u5468\u671f\u8ddf\u968fPod\uff0cPod\u9500\u6bc1\u540e\u5176\u4e2d\u5b58\u50a8\u7684\u65e5\u5fd7\u4e5f\u4f1a\u6d88\u5931\u3002
\u4f7f\u7528emptyDir\u6302\u8f7d\u7684\u65e5\u5fd7\u6587\u4ef6\uff0c\u4e00\u822c\u5728\u8282\u70b9\u7684\u8def\u5f84\u5982\u4e0b\uff1a
/var/lib/kubelet/pods/${pod.UID}/volumes/kubernetes.io~empty-dir/${volumeName}\n(2) hostPath
\u751f\u547d\u5468\u671f\u548cPod\u65e0\u5173\uff0cPod\u8fc1\u79fb\u6216\u8005\u9500\u6bc1\uff0c\u65e5\u5fd7\u6587\u4ef6\u8fd8\u4fdd\u7559\u5728\u73b0\u6709\u78c1\u76d8\u4e0a\u3002
\u4e3a\u4e86\u89e3\u51b3\u9694\u79bb\u6027\uff0c\u907f\u514d\u591a\u4e2aPod\u6253\u5370\u65e5\u5fd7\u5230\u76f8\u540c\u7684\u8def\u5f84\u548c\u6587\u4ef6\u4e2d\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 subPathExpr \u5b57\u6bb5\u4ece Downward API \u73af\u5883\u53d8\u91cf\u6784\u9020 subPath \u76ee\u5f55\u540d\u3002 \u8be5 VolumeSubpathEnvExpansion \u529f\u80fd\u4ece Kubernetes1.15 \u5f00\u59cb\u9ed8\u8ba4\u5f00\u542f\uff0c\u57281.17 GA\u3002\u53ef\u53c2\u8003 feature-gates \u548c using-subpath-expanded-environment\u3002
\u4f7f\u7528subPathExpr\u7684\u793a\u4f8b\u5982\u4e0b\u6240\u793a\uff1a
apiVersion: apps/v1\nkind: Deployment\nmetadata:\nlabels:\napp: nginx\nname: nginx\nnamespace: default\nspec:\nselector:\nmatchLabels:\napp: nginx\ntemplate:\nmetadata:\nlabels:\napp: nginx\nspec:\ncontainers:\n- env:\n- name: POD_NAME\nvalueFrom:\nfieldRef:\napiVersion: v1\nfieldPath: metadata.name\n- name: NAMESPACE\nvalueFrom:\nfieldRef:\napiVersion: v1\nfieldPath: metadata.namespace\nimage: nginx\nname: nginx\nresources: {}\nvolumeMounts:\n- mountPath: /data/log\nname: datalog\nsubPathExpr: $(NAMESPACE)/$(POD_NAME)\nvolumes:\n- hostPath:\npath: /data/log\ntype: \"\"\nname: datalog\n\u5728\u521b\u5efa\u5b8c\u4e4b\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u5728\u6240\u5728\u8282\u70b9\u7684/data/log\u4e0b\u53d1\u73b0\u7c7b\u4f3c\u7684\u76ee\u5f55\u7ed3\u6784\uff1a
.\n`-- default\n |-- nginx-888dc6778-krfqr\n `-- nginx-888dc6778-sw8vd\n/data/log/$(NAMESPACE)/$(POD_NAME)\u8def\u5f84\u4e0b\u3002 (3) Pv Pv\u7684\u8bbf\u95ee\u6a21\u5f0f\u5305\u62ec\uff1a
\u5bf9\u4e8e\u5927\u90e8\u5206\u7684\u4e1a\u52a1\u6765\u8bf4\uff0c\u90fd\u662fDeployment\u65e0\u72b6\u6001\u90e8\u7f72\uff0c\u9700\u8981\u6302\u8f7d\u540c\u4e00\u4e2aPv\u5171\u4eab\uff1b\u5bf9\u4e8e\u4e00\u4e9b\u4e2d\u95f4\u4ef6\u7b49\u6709\u72b6\u6001\u670d\u52a1\uff0c\u4e00\u822c\u4f1a\u4f7f\u7528StatefulSet\u90e8\u7f72\uff0c\u6bcf\u4e2aPod\u4f1a\u4f7f\u7528\u72ec\u7acb\u7684Pv\u3002
\u867d\u7136\u540c\u6837\u53ef\u4ee5\u5728Node\u4e0a\u627e\u5230\u4f7f\u7528Pv\u6302\u8f7d\u7684\u5bf9\u5e94\u65e5\u5fd7\u6587\u4ef6\uff0c\u4f46\u662fPv\u6839\u636e\u4e0d\u540c\u7684\u5e95\u5c42\u5b9e\u73b0\uff0c\u5728Node\u4e0a\u7684\u8def\u5f84\u4f1a\u6709\u4e00\u5b9a\u7684\u533a\u522b\u3002
\u76ee\u524d\u5e02\u9762\u4e0a\u5927\u90e8\u5206\u65e5\u5fd7Agent\u5747\u5bf9\u8fd9\u4e9b\u6302\u8f7d\u65b9\u5f0f\u6ca1\u6709\u611f\u77e5\uff0c\u6240\u4ee5\u4f60\u80fd\u505a\u7684\u548c\u4e0a\u9762\u4f7f\u7528stdout\u7684\u65b9\u5f0f\u7c7b\u4f3c\uff0c\u4e5f\u5c31\u662f\u7b80\u5355\u7c97\u66b4\u7684\u8ba9Agent\u5c06\u8def\u5f84\u90fd\u6302\u8f7d\uff0c\u4f7f\u7528\u901a\u914d\u7684\u65b9\u5f0f\u91c7\u96c6\u6240\u6709\u7684\u65e5\u5fd7\uff0c\u4f7f\u7528\u4e0a\u7684\u5c40\u9650\u548cstdout\u7684\u65b9\u5f0f\u540c\u6837\u4e00\u81f4\u3002
\u53e6\u5916\uff0c\u9274\u4e8e\u4e00\u4e9bAgent\u5bf9\u91c7\u96c6docker stdout\u6709\u4e00\u5b9a\u7684\u652f\u6301\uff0c\u6240\u4ee5\u8fd8\u5b58\u5728\u4e00\u4e9b\u4f7f\u7528\u4e0a\u53d8\u79cd\uff0c\u6bd4\u5982\u5229\u7528webhook\u6ce8\u5165\u4e00\u4e2asidecar\uff0c\u8bfb\u53d6Pod\u91cc\u7684\u65e5\u5fd7\u6587\u4ef6\uff0c\u8f6c\u6362\u6210sidecar\u7684stdout\uff0c\u7136\u540e\u91c7\u96c6sidecar\u7684stdout\u65e5\u5fd7\uff0c\u8fd9\u91cc\u4e0d\u518d\u8be6\u8ff0\u3002
\uff084\uff09\u4e0d\u6302\u8f7d
\u5f88\u591a\u60c5\u51b5\u4e0b\uff0c\u7528\u6237\u90fd\u758f\u4e8e\u6302\u8f7d\u65e5\u5fd7\u8def\u5f84volume\uff0c\u6216\u8005\u5bf9\u63a5\u4e00\u4e2a\u5df2\u7ecf\u5b58\u5728\u7684\u7cfb\u7edf\uff0c\u5f80\u5f80\u4e1a\u52a1\u65b9\u4e0d\u613f\u610f\u66f4\u6539\uff0c\u8fd9\u4e2a\u65f6\u5019\u80fd\u91c7\u96c6\u5bb9\u5668\u91cc\u7684\u65e5\u5fd7\u6587\u4ef6\u5417\uff1f
Loggie\u5c1d\u8bd5\u89e3\u51b3\u4e86\u8fd9\u4e2a\u95ee\u9898\uff0c\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6e\u5f00\u542f\uff0c\u4e0d\u8fc7\u8fd9\u79cd\u65b9\u5f0f\u4ecd\u7136\u9700\u8981\u66f4\u591a\u957f\u671f\u751f\u4ea7\u5b9e\u8df5\u7684\u68c0\u9a8c\u3002\u6b22\u8fce\u8bd5\u7528\uff0c\u5177\u4f53\u5f00\u5173\u8bf7\u53c2\u8003\u4e0b\u4e00\u8282\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#sidecar-stdout","title":"Sidecar + Stdout/\u65e5\u5fd7\u6587\u4ef6","text":"\u5982\u679c\u9700\u8981\u91c7\u7528sidecar\u7684\u65b9\u5f0f\uff0c\u9700\u8981\u5c06\u65e5\u5fd7Agent\u540c\u65f6\u6302\u8f7d\u76f8\u540c\u7684\u65e5\u5fd7\u8def\u5f84\uff0cAgent\u7684\u914d\u7f6e\u6587\u4ef6\u4e5f\u4e00\u822c\u4f7f\u7528ConfigMap\u6302\u8f7d\u51fa\u6765\u3002
\u4f7f\u7528Sidecar\u7684\u90e8\u7f72\u65b9\u5f0f\u548cDaemonSet\u7684\u65b9\u5f0f\u4f18\u52a3\u5bf9\u6bd4\u8bf7\u53c2\u8003\u4e0a\u6587\u3002
\u603b\u7ed3\uff1a \u5927\u90e8\u5206\u4e3b\u6d41\u7684\u5f00\u6e90Agent\uff0c\u53ea\u5bf9\u5bb9\u5668Stdout\u6709\u90e8\u5206\u652f\u6301\uff0c\u6bd4\u5982\u652f\u6301\u91c7\u96c6\u7684\u65f6\u5019\u7edf\u4e00\u52a0\u4e0a\u4e00\u4e9bK8s\u76f8\u5173\u5143\u4fe1\u606f\uff0c\u4f46\u4e0d\u652f\u6301\u5355\u72ec\u9488\u5bf9\u67d0\u4e9bPod\u8fdb\u884c\u914d\u7f6e\uff0c\u800c\u4e14\u5747\u6ca1\u6709\u5bf9\u5bb9\u5668\u4e2d\u7684\u65e5\u5fd7\u6587\u4ef6\u91c7\u96c6\u6709\u5f88\u597d\u7684\u652f\u6301\u3002 \u8fd9\u6837\u5bfc\u81f4\u5728\u5b9e\u9645\u590d\u6742\u7684\u5404\u7c7b\u4e1a\u52a1\u573a\u666f\u4e2d\uff0c\u6ca1\u529e\u6cd5\u6ee1\u8db3\u9700\u6c42\u3002
"},{"location":"user-guide/use-in-kubernetes/general-usage/#loggie","title":"Loggie\u5982\u4f55\u89e3\u51b3\u8fd9\u4e9b\u95ee\u9898\uff1f","text":"\u76f8\u4fe1\u4ee5\u4e0a\u5217\u51fa\u7684\u95ee\u9898\u5927\u5bb6\u90fd\u9047\u5230\u6216\u8005\u601d\u8003\u8fc7\uff0c\u90a3Loggie\u662f\u5982\u4f55\u89e3\u51b3\u8fd9\u4e9b\u95ee\u9898\u7684\u5462\uff1f \u8bf7\u770b\u4e0b\u4e00\u8282\u5982\u4f55 \u4f7f\u7528Loggie\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff1f
"},{"location":"user-guide/use-in-kubernetes/kube-event-source/","title":"\u91c7\u96c6Kubernetes Events","text":"\u9664\u4e86\u4f7f\u7528Logconfig\u91c7\u96c6\u65e5\u5fd7\u5916\uff0cLoggie\u540c\u6837\u53ef\u4ee5\u901a\u8fc7CRD\u914d\u7f6e\u4efb\u610f\u7684source/sink/interceptor\uff0c\u672c\u8d28\u4e0aLoggie\u5c31\u662f\u4e00\u4e2a\u652f\u6301\u591aPipeline\u7684\u6570\u636e\u6d41\uff0c\u96c6\u6210\u4e86\u901a\u7528\u7684\u8bf8\u5982\u961f\u5217\u91cd\u8bd5\u3001\u6570\u636e\u5904\u7406\u3001\u914d\u7f6e\u4e0b\u53d1\u3001\u76d1\u63a7\u62a5\u8b66\u7b49\u7b49\u529f\u80fd\uff0c\u51cf\u5c11\u4e86\u7c7b\u4f3c\u9700\u6c42\u7684\u7814\u53d1\u6210\u672c\u3002\u91c7\u96c6Kubernetes\u7684Events\u5c31\u662f\u5176\u4e2d\u4e00\u4e2a\u5f88\u597d\u7684\u4f8b\u5b50\u3002
Kubernetes Events\u662f\u7531Kubernetes\u672c\u8eab\u7ec4\u4ef6\u548c\u4e00\u4e9b\u63a7\u5236\u5668\u4ea7\u751f\u7684\u4e8b\u4ef6\uff0c\u6211\u4eec\u5e38\u7528\u7684kubectl describe\u547d\u4ee4\u5c31\u53ef\u4ee5\u67e5\u770b\u5173\u8054\u8d44\u6e90\u7684\u4e8b\u4ef6\u4fe1\u606f\uff0c\u91c7\u96c6\u8bb0\u5f55\u8fd9\u4e9b\u4e8b\u4ef6\u53ef\u4ee5\u5e2e\u52a9\u6211\u4eec\u56de\u6eaf\u3001\u6392\u67e5\u3001\u5ba1\u8ba1\u3001\u603b\u7ed3\u95ee\u9898\uff0c\u66f4\u597d\u7684\u4e86\u89e3Kubernetes\u96c6\u7fa4\u5185\u90e8\u72b6\u6001\u3002
\u548cLoggie\u4e2d\u8f6c\u673a\u7c7b\u4f3c\uff0c\u6211\u4eec\u53ef\u4ee5\u5355\u72ec\u90e8\u7f72Aggregator\u96c6\u7fa4\u6216\u8005\u590d\u7528\u73b0\u6709\u7684\u4e2d\u8f6c\u673a\u96c6\u7fa4\u3002
"},{"location":"user-guide/use-in-kubernetes/kube-event-source/#_2","title":"\u914d\u7f6e\u793a\u4f8b","text":"\u914d\u7f6ekubeEvents source\uff0c\u5e76\u4e14\u4f7f\u7528type: cluster\u4e0b\u53d1\u914d\u7f6e\u5230Aggregator\u96c6\u7fa4\u5373\u53ef\u3002
Config
apiVersion: loggie.io/v1beta1\nkind: ClusterLogConfig\nmetadata:\nname: kubeevent\nspec:\nselector:\ntype: cluster\ncluster: aggregator\npipeline:\nsources: |\n- type: kubeEvent\nname: event\nsinkRef: dev\n\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u4e0d\u7ba1\u662f\u53d1\u9001\u7ed9Elasticsearch\u8fd8\u662f\u5176\u4ed6\u7684sink\uff0c\u8f93\u51fa\u7684\u662f\u7c7b\u4f3c\u5982\u4e0b\u683c\u5f0f\u7684\u6570\u636e:
event
{\n\"body\": \"{\\\"metadata\\\":{\\\"name\\\":\\\"loggie-aggregator.16c277f8fc4ff0d0\\\",\\\"namespace\\\":\\\"loggie-aggregator\\\",\\\"uid\\\":\\\"084cea27-cd4a-4ce4-97ef-12e70f37880e\\\",\\\"resourceVersion\\\":\\\"2975193\\\",\\\"creationTimestamp\\\":\\\"2021-12-20T12:58:45Z\\\",\\\"managedFields\\\":[{\\\"manager\\\":\\\"kube-controller-manager\\\",\\\"operation\\\":\\\"Update\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"time\\\":\\\"2021-12-20T12:58:45Z\\\",\\\"fieldsType\\\":\\\"FieldsV1\\\",\\\"fieldsV1\\\":{\\\"f:count\\\":{},\\\"f:firstTimestamp\\\":{},\\\"f:involvedObject\\\":{\\\"f:apiVersion\\\":{},\\\"f:kind\\\":{},\\\"f:name\\\":{},\\\"f:namespace\\\":{},\\\"f:resourceVersion\\\":{},\\\"f:uid\\\":{}},\\\"f:lastTimestamp\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:source\\\":{\\\"f:component\\\":{}},\\\"f:type\\\":{}}}]},\\\"involvedObject\\\":{\\\"kind\\\":\\\"DaemonSet\\\",\\\"namespace\\\":\\\"loggie-aggregator\\\",\\\"name\\\":\\\"loggie-aggregator\\\",\\\"uid\\\":\\\"7cdf4792-815d-4eba-8a81-d60131ad1fc4\\\",\\\"apiVersion\\\":\\\"apps/v1\\\",\\\"resourceVersion\\\":\\\"2975170\\\"},\\\"reason\\\":\\\"SuccessfulCreate\\\",\\\"message\\\":\\\"Created pod: loggie-aggregator-pbkjk\\\",\\\"source\\\":{\\\"component\\\":\\\"daemonset-controller\\\"},\\\"firstTimestamp\\\":\\\"2021-12-20T12:58:45Z\\\",\\\"lastTimestamp\\\":\\\"2021-12-20T12:58:45Z\\\",\\\"count\\\":1,\\\"type\\\":\\\"Normal\\\",\\\"eventTime\\\":null,\\\"reportingComponent\\\":\\\"\\\",\\\"reportingInstance\\\":\\\"\\\"}\",\n\"systemPipelineName\": \"default/kubeevent/\",\n\"systemSourceName\": \"event\"\n}\n\u4e3a\u4e86\u65b9\u4fbf\u5206\u6790\u5c55\u793a\uff0c\u6211\u4eec\u53ef\u4ee5\u6dfb\u52a0\u4e00\u4e9binterceptor\u5c06\u91c7\u96c6\u5230\u7684events\u6570\u636ejson decode\u3002
\u914d\u7f6e\u793a\u4f8b\u5982\u4e0b\uff0c\u5177\u4f53\u8bf7\u53c2\u8003\u65e5\u5fd7\u5207\u5206\u4e0e\u5904\u7406\u3002
Config
interceptorapiVersion: loggie.io/v1beta1\nkind: Interceptor\nmetadata:\nname: jsondecode\nspec:\ninterceptors: |\n- type: normalize\nname: json\nprocessors:\n- jsonDecode: ~\n- drop:\ntargets: [\"body\"]\napiVersion: loggie.io/v1beta1\nkind: ClusterLogConfig\nmetadata:\nname: kubeevent\nspec:\nselector:\ntype: cluster\ncluster: aggregator\npipeline:\nsources: |\n- type: kubeEvent\nname: event\ninterceptorRef: jsondecode\nsinkRef: dev\n\u7ecf\u8fc7normalize interceptor\u91ccjsonDecode\u540e\u7684\u6570\u636e\u5982\u4e0b\u6240\u793a\uff1a
event
{\n\"metadata\": {\n\"name\": \"loggie-aggregator.16c277f8fc4ff0d0\",\n\"namespace\": \"loggie-aggregator\",\n\"uid\": \"084cea27-cd4a-4ce4-97ef-12e70f37880e\",\n\"resourceVersion\": \"2975193\",\n\"creationTimestamp\": \"2021-12-20T12:58:45Z\",\n\"managedFields\": [\n{\n\"fieldsType\": \"FieldsV1\",\n\"fieldsV1\": {\n\"f:type\": {\n\n},\n\"f:count\": {\n\n},\n\"f:firstTimestamp\": {\n\n},\n\"f:involvedObject\": {\n\"f:apiVersion\": {\n\n},\n\"f:kind\": {\n\n},\n\"f:name\": {\n\n},\n\"f:namespace\": {\n\n},\n\"f:resourceVersion\": {\n\n},\n\"f:uid\": {\n\n}\n},\n\"f:lastTimestamp\": {\n\n},\n\"f:message\": {\n\n},\n\"f:reason\": {\n\n},\n\"f:source\": {\n\"f:component\": {\n\n}\n}\n},\n\"manager\": \"kube-controller-manager\",\n\"operation\": \"Update\",\n\"apiVersion\": \"v1\",\n\"time\": \"2021-12-20T12:58:45Z\"\n}\n]\n},\n\"reportingComponent\": \"\",\n\"type\": \"Normal\",\n\"message\": \"Created pod: loggie-aggregator-pbkjk\",\n\"reason\": \"SuccessfulCreate\",\n\"reportingInstance\": \"\",\n\"source\": {\n\"component\": \"daemonset-controller\"\n},\n\"count\": 1,\n\"lastTimestamp\": \"2021-12-20T12:58:45Z\",\n\"firstTimestamp\": \"2021-12-20T12:58:45Z\",\n\"eventTime\": null,\n\"involvedObject\": {\n\"kind\": \"DaemonSet\",\n\"namespace\": \"loggie-aggregator\",\n\"name\": \"loggie-aggregator\",\n\"uid\": \"7cdf4792-815d-4eba-8a81-d60131ad1fc4\",\n\"apiVersion\": \"apps/v1\",\n\"resourceVersion\": \"2975170\"\n},\n}\n\u5982\u679c\u89c9\u5f97\u6570\u636e\u5b57\u6bb5\u592a\u591a\u6216\u8005\u683c\u5f0f\u4e0d\u7b26\u5408\u9700\u6c42\uff0c\u8fd8\u53ef\u4ee5\u914d\u7f6enormalize interceptor\u8fdb\u884c\u4fee\u6539\u3002
"},{"location":"user-guide/use-in-kubernetes/sidecar/","title":"Loggie Sidecar\u65b9\u5f0f\u91c7\u96c6\u65e5\u5fd7","text":"\u867d\u7136\u9ed8\u8ba4\u4e0d\u63a8\u8350\u4f7f\u7528Sidecar\u65b9\u5f0f\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0c\u4f46\u662f\u5728\u67d0\u4e9b\u53d7\u9650\u573a\u666f\u4e0b\u53ea\u80fd\u9009\u62e9\u4f7f\u7528Sidecar\u7684\u65b9\u5f0f\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\uff0c\u8fd9\u91cc\u6211\u4eec\u7ed9\u51fa\u4e00\u4e2a\u53c2\u8003\u7684\u793a\u4f8b\u3002
"},{"location":"user-guide/use-in-kubernetes/sidecar/#_1","title":"\u603b\u4f53\u601d\u8def","text":"\u5982\u4e0b\u56fe\u6240\u793a\uff1a
Loggie\u548c\u4e1a\u52a1container\u90e8\u7f72\u5728\u540c\u4e00\u4e2aPod\u91cc\uff0c\u540c\u65f6\u9700\u8981\u6302\u8f7d\u76f8\u540c\u7684\u65e5\u5fd7\u6587\u4ef6volume\uff0c\u53e6\u5916Loggie\u7684\u914d\u7f6e\u53ef\u901a\u8fc7configMap\u7684\u65b9\u5f0f\u6302\u8f7d\u5230\u5bb9\u5668\u4e2d\uff0cLoggie\u6839\u636e\u63d0\u4f9b\u7684configMap\u914d\u7f6e\u6587\u4ef6\uff0c\u91c7\u96c6\u5bb9\u5668\u7684\u65e5\u5fd7\uff0c\u5e76\u53d1\u9001\u5230\u540e\u7aef\u3002
"},{"location":"user-guide/use-in-kubernetes/sidecar/#loggie-sidecar_1","title":"\u6ce8\u5165Loggie sidecar","text":"apiVersion: apps/v1\nkind: Deployment\nmetadata:\nlabels:\napp: tomcat\nname: tomcat\nnamespace: default\nspec:\nreplicas: 1\nselector:\nmatchLabels:\napp: tomcat\ntemplate:\nmetadata:\nlabels:\napp: tomcat\nspec:\ncontainers:\n- name: tomcat\nimage: tomcat\nvolumeMounts:\n- mountPath: /usr/local/tomcat/logs\nname: log\n\n- name: loggie\nargs:\n- -config.system=/opt/loggie/loggie.yml\n- -config.pipeline=/opt/loggie/pipeline.yml\nimage: loggieio/loggie:main\nvolumeMounts:\n# loggie\u548c\u4e1a\u52a1container\u6302\u8f7d\u76f8\u540c\u7684log volume\n- mountPath: /usr/local/tomcat/logs\nname: log\n# \u6302\u8f7d\u65e5\u5fd7\u914d\u7f6econfigMap\n- mountPath: /opt/loggie\nname: loggie-config\n# \u6302\u8f7dloggie\u81ea\u8eab\u6301\u4e45\u5316\u7684\u6570\u636e\n- mountPath: /data\nname: registry\n\nvolumes:\n- emptyDir: {}\nname: log\n- emptyDir: {}\nname: registry\n- name: loggie-config\nconfigMap:\nname: tomcat-loggie-config\n\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\nname: tomcat-loggie-config\nnamespace: default\ndata:\nloggie.yml: |\nloggie:\nreload:\nenabled: true\nperiod: 10s\nmonitor:\nlogger:\nperiod: 30s\nenabled: true\nlisteners:\nfilesource: ~\nfilewatcher: ~\nreload: ~\nsink: ~\nhttp:\nenabled: true\nport: 9196\npipeline.yml: |\npipelines:\n- name: \"tomcat\"\nsources:\n- type: \"file\"\nname: \"tomcatlog\"\npaths:\n- \"/usr/local/tomcat/logs/*.log\"\nfields:\nnamespace: default\ndeployment: tomcat\nfieldsFromEnv:\npodname: HOSTNAME\nsink:\ntype: \"dev\"\nprintEvents: true\ncodec:\npretty: true\n\u53ef\u4ee5\u53c2\u8003\u4ee5\u4e0a\u7684\u90e8\u7f72\u65b9\u5f0f\u901a\u8fc7loggie sidecar\u91c7\u96c6\u5bb9\u5668\u65e5\u5fd7\u3002 \u9700\u8981\u6ce8\u610f\u7684\u662f\uff1a
Tips
env\u547d\u4ee4\u67e5\u770b\u3002 Info
Loggie\u540e\u7eed\u4f1a\u652f\u6301\u81ea\u52a8Sidecar\u6ce8\u5165\u548c\u901a\u8fc7LogConfig\u81ea\u52a8\u751f\u6210ConfigMap\u6302\u8f7d\u7684\u65b9\u5f0f\uff0c\u4ece\u800c\u8fbe\u5230\u548c\u4f7f\u7528DaemonSet\u4e00\u81f4\u7684\u4f53\u9a8c\u3002
"}]} \ No newline at end of file diff --git a/main/sitemap.xml b/main/sitemap.xml new file mode 100644 index 0000000..0f8724e --- /dev/null +++ b/main/sitemap.xml @@ -0,0 +1,3 @@ + +Loggie支持多个Pipeline，每个Pipeline都基于简单直观的source->interceptor->sink的架构。 +这样设计带来的好处有：
+多Pipeline设计，减少互相干扰。比如我们可以将重要的业务日志放在一个Pipeline中，其他的不重要的日志配置为另外的Pipeline，不重要的日志配置变动、发生下游堵塞时，不会影响重要日志的采集和发送。
+在一些场景下，我们可能会将不同类型的服务混合部署在一个节点上，很可能他们的日志会发送到不同的Kafka集群中，如果只有一个全局的输出源，需要在节点上部署两个Agent，如果使用Loggie则只需要使用不同的Pipeline即可，每个Pipeline配置不同的Sink，减少部署成本。
+我们甚至可以采集相同的日志，发送到不同的后端输出源，根据实际需求灵活配置。
本质上source->interceptor->sink架构是一个数据流式的设计，不同类型的source/interceptor/sink的排列组合，可以满足日志的不同需求， +Loggie并没有将interceptor更细化的分类成比如Filter/Formater等类型，interceptor承担了除了source读取，sink发送之外的大部分工作，只需要配置不同的interceptor就可以拥有中转、过滤、解析、切分、日志报警等能力。 +于是，Loggie可以：
+在部署和维护层面带来的好处是：
+如果之前采用常规的ELK架构，使用Filebeat采集日志、Logstash中转和解析日志，由于Filebeat和Logstash是两个不同的语言栈，带来排查问题和扩展开发成本均较高。
+改用Loggie后我们无需维护两个项目，甚至如果没有中转机的需求，可以选择在Agent端配置日志解析。
另外，目前针对日志报警，开源的方案一般为使用elastAlert，但是elastAlert无法直接对接AlertManager，并且在高可用等方面存在问题，同时会强依赖Elasticsearch。所以如果使用Loggie，可以无需引入额外的组件，直接使用Loggie来检测异常日志并接入报警。
+Loggie基于微内核的架构，所有的source/interceptor/sink/queue都被抽象成component，只需要在代码中实现Golang接口，即可方便的研发一个component。
+如果Loggie在某些场景下，无法满足你的需求，可以尝试写一个自己的component。
+比如需要Loggie转换成特定的日志格式，可以写一个interceptor去处理；需要Loggie将采集的日志发送至尚未支持的服务，可以写一个sink。
+当然，Loggie使用Golang编写，所以你目前需要用Golang来写component。Golang和Java或者C/C++相比，在性能和研发效率上有一个折中，更适合类似日志Agent的场景。
如果你尝试过在Kubernetes环境下采集日志，应该遇到过这些问题：
+如果你使用Loggie，你可以：
+在实际的生产环境中，日志Agent本身的稳定性很重要，同时不影响业务也很重要。
+Loggie可配置限流interceptor，在日志量太大时，可以避免发送日志数据占据了太多网络带宽。
+Loggie有合理的文件句柄处理机制，避免fd被占用的各种异常场景导致节点不稳定。
另外，Loggie结合我们在各种环境中遇到的各种问题，针对性的检测暴露出相应的指标。
+比如指标支持采集和发送延迟检测。比如针对文件size增长太快，或者文件size太大等场景，支持该类metric上报。
同时Loggie支持原生Prometheus metric，可避免额外部署exporter带来的部署成本和资源消耗。Loggie还提供了完善的Grafana监控图表，可以方便快速接入使用。
+Loggie基于Golang编写，在代码层面我们有很多优化，在较少资源占用的同时，还可提供强大的吞吐性能。
+ + + + + + +++为什么我们会选择研发Loggie，当时的背景和原因是什么？
+
在研发Loggie之前，我们的日志服务采集端都是使用Filebeat，当时为什么选择Filebeat呢？
+Filebeat是Elastic公司的产品，主打轻量级，用于替换原有Logstash去实现日志采集的工作，相比他们自家基于JRuby语言的Logstash，Filebeat确实是相对轻量、资源占用少。和其他的开源日志采集Agent相比，基于Golang开发的Filebeat，也有很多优势。
+例如，相比基于Java的Flume，性能更好，资源占用更少；相比基于Ruby的Fluentd，性能更好，二次开发更方便；相比基于C的Fluentd-bit，功能更完善，开发更友好。
+所以，总体来说，Filebeat是一个相对比较平衡的日志采集Agent，这也是我们当初选择Filebeat作为默认日志采集Agent的原因。
但是随着我们对Filebeat更加深度的使用，在公司集团内部实践和外部客户的交付中，也碰到了一些问题。
+由于Filebeat设计当初设计时，为了区分于Logstash，突出轻量级，牺牲了很多可扩展性的设计。
+最明显的就是，Filebeat只有一个Queue和一个Output。这样也导致了：
由于所有的服务日志都会发送到全局唯一的Queue里，导致服务日志数据混在一起，在异常场景发生时，无法有隔离性的保障。
+比如Filebeat全局的Queue堆积，会导致节点的所有服务日志均无法发送，如果我们对不同服务的日志的级别和要求不一样，不管重要还是不重要的日志都会受到影响。
有一些场景下，我们可能需要将不同服务的不同类型日志发送至不同的后端，但Filebeat无法使用同一个Agent去发送到不同的Kafka集群，只能在节点上部署多个Agent，导致维护和资源成本上升。
+相比Logstash/Flume等，Filebeat并非使用类似的input->queue->output的灵活多个pipeline设计，在对于日志数据的处理/过滤/增强上，依赖的是Filebeat有限的一些processor，可扩展性不足。
+同时Filebeat也无法作为中转聚合使用，在使用场景下大大受限，需要额外引入其他组件。另外类似日志报警等场景，Filebeat也无法满足。
+我们也尝试过定制化开发，但Filebeat本身的架构设计上难以实现更多的扩展能力，并且长期会带来升级与上游社区代码同步的问题。
Filebeat的metrics比较有限，很多时候我们想要排查诸如常见的日志是否有采集、采集的日志是否完整、发送是否有延迟等等排障场景，Filebeat没有提供相应的功能，十分影响线上的问题排查效率。而且Filebeat未提供Prometheus格式的监控指标，需要额外注入exporter。
+虽然Filebeat性能尚可，但是在我们的实际使用时，遇到日志场景复杂、日志量大的情况时，存在吞吐量的瓶颈，无法满足实时性的需求。
+Fluentd/Fluent-bit
+Fluentd基于Ruby性能一般，单线程；Fluent-bit基于C，对于我们的技术栈来说，Ruby和C的维护和二次开发成本比较大。
+Logstash
+Logstash性能较差，基于JRuby的资源占用和损耗都比较大。
+Flume
+资源占用较大，性能一般，之前有内部部门使用过Flume，对比的压测结论证实确实不如Filebeat。
+最重要的是，目前所有的开源Agent，均没有对K8s有很好的原生支持，个别支持的也只能采集stdout的日志。 +正是由于目前开源的Agent存在一些问题，不能满足长期的需求，所以我们决定开始自研。
+整体的目标： +高性能、资源占用低、高可用、稳定性强、可扩展性强、更适合云原生。
+性能与资源： +性能相同的情况下，CPU比社区Filebeat大大降低，吞吐量上限要远高于Filebeat。
+高可用、稳定性强： +资源隔离，作为基础设施一定要稳定可靠，同时默认支持大量监控指标，对常见的运维类问题有良好的支撑，减少运维负担。
+可扩展性：
+整体设计上，方便用户扩展，实现过滤、路由、编码等能力。比如可以很快速的写一个处理逻辑，就可以进行数据处理。
总结一下，我们理想中的日志Agent是一个：
+| + | Loggie | +Filebeat | +Fluentd | +Logstash | +Flume | +
|---|---|---|---|---|---|
| 开发语言 | +Golang | +Golang | +Ruby | +JRuby | +Java | +
| 多Pipeline | +支持 | +单队列 | +单队列 | +支持 | +支持 | +
| 多输出源 | +支持 | +不支持，仅一个Output | +配置copy | +支持 | +支持 | +
| 中转机 | +支持 | +不支持 | +支持 | +支持 | +支持 | +
| 日志报警 | +支持 | +不支持 | +不支持 | +不支持 | +不支持 | +
| Kubernetes容器日志采集 | +支持容器的stdout和容器内部日志文件 | +只支持容器stdout | +只支持容器stdout | +不支持 | +不支持 | +
| 配置下发 | +Kubernetes下可通过CRD配置，主机场景配置中心陆续支持中 | +手动配置 | +手动配置 | +手动配置 | +手动配置 | +
| 监控 | +原生支持Prometheus metrics，同时可配置单独输出指标日志文件、发送metrics等方式 | +API接口暴露，接入Prometheus需使用额外的exporter | +支持API和Prometheus metrics | +需使用额外的exporter | +需使用额外的exporter | +
| 资源占用 | +低 | +低 | +一般 | +较高 | +较高 | +
测试环境：
+测试目的：
+Filebeat和Loggie的性能对比
+测试思路：
+Filebeat和Loggie，均采集日志发送至Kafka，观察相应的资源占用和发送吞吐量
+测试详情：
+单文件自动生成5000000行日志，每行内容如下所示：
+[13/May/2021:10:20:29 +0800] 0.015 10.200.170.107 "GET /static/3tJHS3Ubrf.html?activity_channel_id=22=1_00000&fromMiniapp=1&miniapp_uuid=uEd93lG2eG8Qj5fRXuiJwNt4bmiylkmg HTTP/1.1" 200 138957 "110.183.45.54, 10.200.151.37" act.you.163.com "" "Mozilla/5.0 (Linux; Android 8.1.0; PADM00Build/O11019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 XWEB/568 MMWEBSDK/190102 Mobile Safari/537.36 MMWEBID/6881 MicroMessenger/7.0.3.1400(0x2700033B) Process/appbrand0 NetType/WIFI Language/zh_CN miniProgram" "" [127.0.0.1:8990] [0.014] [] [] immsg={"st":1553307293614,"sb":138963,"rc":200,"cf":{"sr":1},"if":"default","ut":14,"sv":"static","pd":"activity","qb":764}
+配置Filebeat和Loggie采集日志，并发送至Kafka某个Topic，不使用客户端压缩，Kafka Topic配置Partition为3。
+在保证Agent规格资源充足的情况下，修改采集的文件个数、发送客户端并发度（配置Filebeat worker和Loggie parallelism)，观察各自的CPU、Memory和Pod网卡发送速率。
+测试得到如下数据：
+| Agent | +文件大小 | +日志文件数 | +发送并发度 | +CPU | +MEM (rss) | +网卡发包速率 | +
|---|---|---|---|---|---|---|
| Filebeat | +3.2G | +1 | +3 | +7.5~8.5c | +63.8MiB | +75.9MiB/s | +
| Filebeat | +3.2G | +1 | +8 | +10c | +65MiB | +70MiB/s | +
| Filebeat | +3.2G | +10 | +8 | +11c | +65MiB | +80MiB/s | +
| + | + | + | + | + | + | + |
| Loggie | +3.2G | +1 | +3 | +2.1c | +60MiB | +120MiB/s | +
| Loggie | +3.2G | +1 | +8 | +2.4c | +68.7MiB | +120MiB/s | +
| Loggie | +3.2G | +10 | +8 | +3.5c | +70MiB | +210MiB/s | +
测试结论：
+相同压测条件和场景下：
+Loggie和Filebeat消耗的CPU相比，大概仅为后者的¼，同时发送吞吐量为后者的1.6～2.6倍。
+Memory相当，均处于较低的水准。
+Filebeat的极限吞吐量存在瓶颈，80MB/s后很难提升，而Loggie则可以达到200MiB/s以上。
+Loggie详细的设计如下图所示：
+
数据链路:
+控制链路:
+本质上Loggie是一个数据传输流式设计，所以我们可以灵活的使用Loggie的Pipeline。从使用形态上可分为：
+除了日志采集，很多涉及到数据传输、转换的场景，都可以考虑使用Loggie，即使Loggie没有你想要的组件，你也可以快速开发一个source、sink或interceptor，同时复用Loggie的很多能力，避免重复的开发工作，比如：
+Info
+了解Loggie内部数据格式的设计，能帮助我们配置合适的日志处理和日志格式转换
+在Loggie内部的日志数据，包括：
+如果以上的格式不满足需求，可以参考：
+对于原始日志数据的切分与处理，请使用 normalize interceptor，请参考：
+Loggie可以部署为Agent，同时支持独立部署，进行聚合、转发和处理。
+使用中转机架构的方式一般有多种，常见的有：
+Agent -> Aggregator: Agent直接发送至Aggregator，Aggregator再发送至后端存储。 Agent -> MQ -> Aggregator: Agent发送至消息队列，比如Kafka，然后Aggregator再消费Kafka消息发送至后端。 是否引入Kafka等消息队列，主要取决于自身的场景需求和数据的量级。
+部署Agent
+部署为Aggregator类型时，请务必在Kubernetes配置中指定cluster集群名称。
采集日志的LogConfig配置无区别，只需要修改sink发送至Loggie Aggregator或者Kafka即可。
+被采集的容器创建以及匹配的LogConfig，请参考Loggie采集容器日志。
+这里我们将其中的sink修改为以下示例：
+Example
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: aggregator
+spec:
+ sink: |
+ type: grpc
+ host: "loggie-aggregator.loggie-aggregator:6066"
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: kafka
+spec:
+ sink: |
+ type: kafka
+ brokers: ["127.0.0.1:6400"]
+ topic: "log-${fields.topic}"
+配置LogConfig下发至Aggregator本质上和Agent侧无区别，只需要修改其中selector部分。
类似如下：
+Example
+apiVersion: loggie.io/v1beta1
+kind: ClusterLogConfig
+metadata:
+ name: aggre
+spec:
+ selector:
+ type: cluster
+ cluster: aggregator
+ pipeline:
+ sources: |
+ - type: grpc
+ name: rec1
+ port: 6066
+ sinkRef: dev
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: dev
+spec:
+ sink: |
+ type: dev
+ printEvents: true
+ codec:
+ type: json
+ pretty: true
+type: cluster表示选择下发配置到cluster指定的Loggie集群，即我们刚部署的中转机集群，如果不填写会将配置指定到默认的Agent集群，导致无法生效。
这里的source为Grpc，接收Agent Grpc sink发出的数据，然后转发至自身sinkRef指定的sink中。这里我们创建了一个dev sink用于查看中转机输出的数据。
+通过kubectl -nloggie-aggregator logs -f <podName> --tail=200命令。
+可以在中转机节点上查看到类似如下日志：
events
+2021-12-20 09:58:50 INF go/src/loggie.io/loggie/pkg/sink/dev/sink.go:98 > event: {
+ "body": "14-Dec-2021 06:19:58.306 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [141] milliseconds",
+ "fields": {
+ "podname": "tomcat-684c698b66-gkrfs",
+ "containername": "tomcat",
+ "logconfig": "tomcat",
+ "namespace": "default",
+ "nodename": "kind-control-plane"
+ },
+}
+在日志被发送至下游服务时，并发度可以被控制。但起码有以下两点会对实际情况造成影响：下游服务器是否有压力，数据发送是否有压力。这两点是实时变化的，一个定值很难长期满足实际需求。
+自适应sink流量控制功能可以：
+根据下游数据响应的实际情况，自动调整下游数据发送并行数，要做到尽量发挥下游服务端的性能，且不影响其性能。
+在上游数据收集被阻塞时，适当调整下游数据发送速度，缓解上游阻塞。
+配置细节可参考sink concurrency。
+核心是模仿tcp流量控制，并根据loggie自身做调整。
+快速启动阶段
+快速启动结束
+平稳阶段
+其他细节
+下游额外搭建了本地服务器，可以自由调整rtt值，用来模拟网络波动。
+加入了随机的失败返回，概率可以设定。
+阻塞情况用大幅增大rtt情况来模拟，由于rttT的不是固定的值，所以当rtt稳定后，不会对判断逻辑有影响。
+Config
+concurrency:
+ enable: true
+ goroutine:
+ initThreshold: 16
+ maxGoroutine: 30
+ unstableTolerate: 3
+ channelLenOfCap: 0.4
+ rtt:
+ blockJudgeThreshold: 120%
+ newRttWeigh: 0.5
+ ratio:
+ multi: 2
+ linear: 2
+ linearWhenBlocked: 4
+ duration:
+ unstable: 15
+ stable: 30
+模拟下游服务没有返回错误情况，仅调整rtt，测试算法对网络延迟的响应。
+
解读
+在案例一的基础上，模拟下游有错误返回的情况，概率设定为0.15%。
+
解读
+Info
+此功能在试验阶段，默认为关闭，欢迎交流讨论。
+++建议先了解Loggie内部日志数据schema设计。
+
Loggie部署在不同的环境中，如果需要在原始的日志数据里，增加一些元信息，同时兼容已有的格式，可以参考如下的办法。
+使用schema interceptor可以增加时间字段，以及pipelineName与sourceName字段。另外还可以对字段进行重命名，比如修改body为message。
+请参考schema interceptor。
由于大部分情况下，我们需要全局生效，而不是仅仅只在某个pipeline里添加该interceptor，所以建议在系统配置的defaults中添加schema interceptor， +这样可以避免每个pipeline均需配置该interceptor。
+loggie.yml
+loggie:
+ defaults:
+ interceptors:
+ - type: schema
+ name: global
+ order: 700
+ addMeta:
+ timestamp:
+ key: "@timestamp"
+ remap:
+ body:
+ key: message
+这里的name是为了增加标识，避免如果在pipeline中又新增schema interceptor会导致校验不通过。另外增加order字段为一个较小的值（默认为900)，这样default里的interceptor会优先于pipeline里定义的其他interceptor执行。
+tranformer提供了更丰富的功能，可以应对复杂日志的场景。
+具体请参考transformer interceptor。
如果我们在source上配置了一些自定义的fields。
+pipelines.yml
+pipelines:
+ - name: local
+ sources:
+ - type: file
+ name: demo
+ paths:
+ - /tmp/log/*.log
+ fields:
+ topic: "loggie"
+
+ sink:
+ type: dev
+ printEvents: true
+ codec:
+ pretty: true
+那么sink输出的为：
+{
+ "fields": {
+ "topic": "loggie",
+ },
+ "body": "01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]"
+}
+当然我们也可以配置fieldsUnderRoot: true，让fields里的key:value和body同一层级。
pipelines.yml
+pipelines:
+ - name: local
+ sources:
+ - type: file
+ fields:
+ topic: "loggie"
+ fieldsUnderRoot: true
+...
+{
+ "topic": "loggie",
+ "body": "01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]"
+}
+在我们使用file source时，可能希望自动在日志原始数据里，增加一些日志采集的状态，比如采集的文件名称、采集的文件offsest等，file source提供了一个addonMeta配置，可快速enable。
示例：添加如下addonMeta，并设置为true。
file source
+sources:
+- type: file
+ paths:
+ - /var/log/*.log
+ addonMeta: true
+此时，采集的event会变成类似如下：
+Example
+{
+ "body": "this is test",
+ "state": {
+ "pipeline": "local",
+ "source": "demo",
+ "filename": "/var/log/a.log",
+ "timestamp": "2006-01-02T15:04:05.000Z",
+ "offset": 1024,
+ "bytes": 4096,
+ "hostname": "node-1"
+ }
+}
+具体字段含义可参考file source
+在Kubernetes的场景中，采集的容器日志，为了在查询的时候，使用namespace/podName等信息进行检索，往往需要增加相关的元数据。
+我们可以在系统配置的discovery.kubernetes中，配置额外的k8s fields字段。
+可参考discovery。
+有一些Loggie系统内置的元信息，我们也希望发送给下游，这个时候，需要使用normalize interceptor中的addMeta processors。 +（需要注意的是，该操作会对采集传输性能有一定影响，正常情况下，并不建议使用该方式）
+pipelines.yml
+pipelines:
+ - name: local
+ sources:
+ - type: file
+ name: demo
+ paths:
+ - /tmp/log/*.log
+ fields:
+ topic: "loggie"
+ interceptors:
+ - type: normalize
+ processors:
+ - addMeta: ~
+
+ sink:
+ type: dev
+ printEvents: true
+ codec:
+ pretty: true
+配置了addMeta processor之后，默认会把所有的系统内置元信息输出。
+默认Json格式输出示例如下：
+Example
+{
+ "fields": {
+ "topic": "loggie"
+ },
+ "meta": {
+ "systemState": {
+ "nextOffset": 720,
+ "filename": "/tmp/log/a.log",
+ "collectTime": "2022-03-08T11:33:47.369813+08:00",
+ "contentBytes": 90,
+ "jobUid": "43772050-16777231",
+ "lineNumber": 8,
+ "offset": 630
+ },
+ "systemProductTime": "2022-03-08T11:33:47.370166+08:00",
+ "systemPipelineName": "local",
+ "systemSourceName": "demo"
+ },
+ "body": "01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]"
+}
+当然，我们可能会觉得这些数据太多了，或者想对字段进行修改。我们就可以使用transformer interceptor里的action进行操作。
+ + + + + + +++Loggie可使用transformer interceptor来进行日志的切分和处理，将日志数据进行结构化的提取，同时可以对提取后的字段进行处理。
+
+建议先了解Loggie内部日志数据schema设计。
最主要的是对日志进行切分解析提取和处理。
+比如以下日志：
+01-Dec-2021 03:13:58.298 INFO [main] Starting service [Catalina]
+我们可能会需要将其中的日期、日志级别解析出来，最终形成：
+{
+ "time": "01-Dec-2021 03:13:58.298",
+ "level": "INFO",
+ "message": "[main] Starting service [Catalina]"
+}
+这种结构化的数据，存储的时候便于过滤查询，或者根据日志里的时间来排序，而不是采集的时间戳，或者根据日志级别进行一些过滤，可以方便查询到ERROR级别的日志等等。
+当然不仅仅是像以上tomcat的运维类日志，还有诸如业务的一些订单等等日志，都有类似的需求和使用场景。
关于stdout日志的解析提取
+以下示例仅提供日志切分处理的参考思路，如果你需要提取容器标准输出的原始日志，请参考采集容器日志。
+日志切分处理在Loggie Agent端或者Loggie中转机侧均可，取决于我们是否需要中转机，以及希望日志处理这种CPU密集型的计算是分布在Agent上，由各个节点承担，还是希望在中转机集群中集中进行。
+下面以采集tomcat服务的access日志为例，展示如何对access日志进行字段切分。
+简单起见，示例使用CRD实例配置下发在Agent，同时使用dev sink直接输出处理结果展示。
+请参考
+配置logconfig如下所示：
+Example
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: tomcat
+ namespace: default
+spec:
+ selector:
+ labelSelector:
+ app: tomcat
+ type: pod
+ pipeline:
+ sources: |
+ - type: file
+ name: access
+ paths:
+ - /usr/local/tomcat/logs/localhost_access_log.*.txt
+
+ interceptors: |
+ - type: transformer
+ actions:
+ - action: regex(body)
+ pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
+
+ sink: |
+ type: dev
+ printEvents: true
+ codec:
+ type: json
+ pretty: true
+这里我们在transformer interceptors里，配置了regex action，针对access日志进行正则提取。
+原始的access日志大概如下所示： +
10.244.0.1 - - [31/Aug/2022:03:13:40 +0000] "GET / HTTP/1.1" 404 683
+经过transformer处理后，我们可以通过kubectl -nloggie logs -f <loggie-pod-name> --tail=100来查看输出的日志。
转换后的event示例如下： +
{
+ "status": "404",
+ "size": "683",
+ "fields": {
+ "logconfig": "tomcat",
+ "namespace": "test1",
+ "nodename": "kind-control-plane",
+ "podname": "tomcat-85c84988d8-frs4n",
+ "containername": "tomcat"
+ },
+ "ip": "10.244.0.1",
+ "id": "-",
+ "u": "-",
+ "time": "[31/Aug/2022:03:13:40 +0000]",
+ "url": "\"GET / HTTP/1.1\""
+}
+我们可以打造以Loggie为核心的云原生可扩展的全链路数据平台，Loggie支持使用不同的技术选型。
+
在不同的业务类型、不同的使用场景、不同的日志规模下，我们可能会采用不同的日志系统架构，架构不存在好坏，只有合不合适。一个简单的场景下，使用复杂的架构搭建出来的日志系统，大概会带来运维灾难。
+这里通过规模演进的视角，总结一下常见的日志系统架构，当然实际的技术选型及变种有很多，我们无法一一列出，相信你可以通过参考下文，搭建适合自己业务的架构。
+需要提前说明的是:
+每天的日志规模较小，比如只有几百G（预估500G以下）左右，日志的使用场景仅仅用于日常运维排查问题，可以采用Loggie直接发送至Elasticsearch集群的方式。
+架构图如下所示：
+
优点：
+缺点：
+变种：
+因为一直以来ELK架构的流行，Elasticsearch是最常用的日志存储。
+如果有其他服务对Elasticsearch的依赖，或者有Elasticsearch的运维经验，Elasticsearch是一个还不错的选择。
+但是，Elasticsearch对资源和运维有一定的要求，在某些轻量级和资源敏感的环境下，可以考虑：
在每天的日志量级稍大，比如在500G至1T的规模，架构和业务使用上有扩展性的考虑，可考虑引入Loggie中转集群。
+
优点：
+缺点：
+如果日志量较大，比如1T以上场景，对性能与稳定性要求比较高，可考虑使用Kafka等消息队列集群。
+
需要注意的是，Kafka本身并不能直接发送至后端，所以这里需要考虑如何将Kafka的数据实时导入到后端存储中。
+这时候，我们可以选择一些组件消费Kafka，发送至后端，比如Loggie/Logstash/Kafka connect/Flink等。
+但是Flink适合有自己的实时流平台或者运维能力的企业，否则可能引入更多运维成本。
优点：
+几十TB至PB级，相比上面大规模场景，集群数量多，机房架构复杂，可以根据以上架构增加更多灵活的扩展。
+比如：
+最终我们可以基于Loggie，搭建一套生产级别的全链路日志数据平台。
+
实际在落地一套完善的日志架构和平台，还需要考虑：
+不管你的服务部署在阿里云还是使用混合云的架构，Loggie同样支持采集日志数据至阿里云可观测统一存储SLS。
+而你只需要使用sls sink即可。
++阿里云官方文档中关于Loggie的使用请参考：使用Loggie上传日志
+
为了使用阿里云可观测统一存储SLS，我们需要先创建一个对应的Project。
+如下所示，在SLS页面中，点击创建Project，填入project名称和对应的地域，创建完之后同步创建相应的Logstore。
+
然后我们可以准备一下sls sink的配置，示例如下所示，具体可参考sls sink 配置参数：
sls sink
+ sink:
+ type: sls
+ name: demo
+ endpoint: cn-hangzhou.log.aliyuncs.com
+ accessKeyId: xxxx
+ accessKeySecret: xxxx
+ project: test
+ logstore: test1
+ topic: myservice
+参考主机部署的方式，我们可以修改pipeline.yml，加上sls sink的配置即可。
+Pipeline配置参考示例如下：
+sls pipeline
+pipelines:
+ - name: test
+ sources:
+ - type: file
+ name: demo
+ addonMeta: true
+ paths:
+ - /tmp/log/*.log
+ sink:
+ type: sls
+ endpoint: cn-hangzhou.log.aliyuncs.com
+ accessKeyId: ${accessKeyId}
+ accessKeySecret: ${accessKeySecret}
+ project: loggietest
+ logstore: demo1
+ topic: myservice
+和自建Kubernetes集群一样，Loggie同样可以部署在阿里云ACK的Kubernetes集群中。 +我们可以根据ACK提供的kubeconfig文件，使用kubectl或者helm来部署。可参考Kubernetes部署。
+请注意：
+discovery.kubernetes.rootFsCollectionEnabled设置为true。discovery.kubernetes.containerRuntime为匹配的Kubernetes集群的容器运行时，默认配置为containerd。部署完之后，和常规的Kubernetes集群下的使用姿势类似，同样可以使用ClusterLogConfig/LogConfig/Interceptor/Sink CR的方式。
+ACK提供了CRD的使用页面，我们可以直接在该页面中进行创建和修改。
+
采集某个服务的日志示例参考如下，只需使用sls sink即可。当然也可以使用单独创建一个sink CR存储sls sink配置，在LogConfig中通过sinkRef引用。
logconfig
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: tomcat
+ namespace: default
+spec:
+ selector:
+ labelSelector:
+ app: tomcat
+ type: pod
+
+ pipeline:
+ sources: |
+ - type: file
+ name: stdout
+ paths:
+ - stdout
+ - type: file
+ name: access
+ ignoreOlder: 3d
+ paths:
+ - /usr/local/tomcat/logs/*.log
+
+ sink: |
+ type: sls
+ endpoint: cn-hangzhou.log.aliyuncs.com
+ accessKeyId: ${accessKeyId}
+ accessKeySecret: ${accessKeySecret}
+ project: loggietest
+ logstore: demo1
+ topic: myservice
+我们可以在SLS具体的Project中查看采集的日志，如下图所示：
+
在本「用户指南」部分，我们主要介绍Loggie的主要功能和特性，同时展示在各种不同的场景下，如何使用Loggie满足各类需求。
+如果希望查询具体组件的使用与配置方式，请戳「组件配置」。
+在企业中，我们需要怎么去构建一套完整的日志系统？如何根据实际情况选型，其中又会碰到哪些问题？
+在不同的业务类型、不同的使用场景、不同的日志规模下，我们可以采用哪些日志系统架构？
请看「企业实战」。
+在我们对落地一套功能完善、架构完整的日志系统有初步了解后，想要知道：
+市面上开源的日志采集Agent已经很多了，我们为什么又选择研发Loggie呢？
+Loggie的诞生是为了解决什么问题呢？
+Loggie和其他的开源日志Agent区别是什么？
请看「架构与特性」。
+如何在Kubernetes集群下使用Loggie？
+请看「Kubernetes」。
如何配置各种不同的日志架构？如何接入现有的服务？如何配置日志的处理和切分？Loggie还提供了哪些有用又好用的功能？
+请看「最佳实践」。
如何配置整体的监控和报警，保证Loggie正常运行？如何监控是否采集到ERROR日志？
+请看「监控报警」。
Loggie的monitor eventbus被设计为发布和订阅模式，各个组件发送metrics到指定的topic中，由独立的listener来消费处理。
+比如file source会将采集日志的一些指标数据，发送至filesource topic，由filesource listener来消费，filesource listener会将数据聚合计算之后，打印至日志，暴露出Prometheus指标。
组件和topic以及listener之间是松耦合关系，比如file source还会定时将全量匹配的日志文件指标发送至filewatcher topic，filewatcher listener会处理和暴露指标。
monitor eventbus配置在全局的系统配置中，示例如下：
+Config
+loggie:
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ queue: ~
+ sink: ~
+ http:
+ enabled: true
+ port: 9196
+其中logger控制所有的metrics指标的日志打印，会在period时间间隔将配置listeners产生的metrics聚合打印在Loggie日志中，便于回溯和排查问题。
listeners则用于配置相关的listener是否开启。
Prometheus格式metrics默认通过http.port端口暴露在/metrics。可以通过curl <podIp>:9196/metrics来查看当前的metrics指标。
目前有以下的listener，主要包含：
+可以使用环境已有的Prometheus或者Grafana，如果需要新部署，可参考：https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack。
+使用Helm部署: +
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+helm install prometheus prometheus-community/kube-prometheus-stack -nprometheus --create-namespace
+Note
+由于众所周知的原因，你的环境里可能无法下载其中的某些k8s.gcr.io镜像，可以考虑下载chart包替换后再重新部署。
确认Pod正常running后，可以根据实际环境访问grafana。通过代理访问grafana的方式可参考：
+kubectl -nprometheus port-forward --address 0.0.0.0 service/prometheus-grafana 8181:80
+Grafana用户名和密码可在prometheus-grafana sercret中，通过base64 -d来查看。
在部署了Loggie的Kubernetes集群中，需要创建如下的ServiceMonitor来让Prometheus采集Loggie Agent指标。
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+ labels:
+ app: loggie
+ release: prometheus
+ name: loggie-agent
+ namespace: prometheus
+spec:
+ namespaceSelector:
+ matchNames:
+ - loggie
+ endpoints:
+ - port: monitor
+ selector:
+ matchLabels:
+ app: loggie
+ instance: loggie
+同时，我们需要在Grafana中添加install工程中的json来展示Loggie的监控控制台。
+Note
+Kubernetes版本和Grafana版本不同，可能导致图表展示不兼容，需要根据情况进行修改。
+导入的Grafana图表目前包含监控大盘和节点Agent的监控图表，如下图所示：
+
+
+
除了Loggie本身的报警，业务日志本身的监控报警也是一个常用的功能，比如在日志中包含了ERROR日志，可以发送报警，这种报警会更贴近业务本身，是基于metrics报警的一种很好的补充。
+有以下两种方式可以选择：
+采集链路不需要独立部署Loggie，但是由于在采集的数据链路上进行匹配，理论上会对传输性能造成一定影响，但胜在方便简单。
+logAlert interceptor用于在日志传输的时候检测异常日志，异常日志会被封装成报警的事件发送至monitor eventbus的logAlert topic，由logAlert listener来消费。logAlert listener支持发送至任意http后端（可以多个）。
+发送体根据自定义模板进行渲染，若模板未定义，则会发送原始数据。在配置模板前，可以先观察原始数据（设置debug模式启动），再进行模板配置，原始数据可能会根据pipeline配置被其他interceptor改动而与示例不同。
配置新增logAlert listener发送告警配置。用于在检测到匹配的日志后，将日志报警发送至诸如alertManager等后端，详细配置可参考logAlert listener。
全局Config文件
+loggie:
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ logAlert:
+ addr: ["http://127.0.0.1:8080/loggie"]
+ bufferSize: 100
+ batchTimeout: 10s
+ batchSize: 10
+ linelimit: 10
+ template: |
+ {
+ "alerts":
+ [
+ {{$first := true}}
+ {{range .Alerts}}
+ {{if $first}}{{$first = false}}{{else}},{{end}}
+ {
+ "labels": {
+ "topic": "{{.fields.topic}}"
+ },
+ "annotations": {
+ "message": "\nNew alert: \nbody:\n{{range .body}}{{.}}\n{{end}}\ncontainerid: {{._meta.pipelineName}}\nsource: {{._meta.sourceName}}\ncontainername: {{.fields.containername}}\nlogconfig: {{.fields.logconfig}}\nname: {{.fields.name}}\nnamespace: {{.fields.namespace}}\nnodename: {{.fields.nodename}}\npodname: {{.fields.podname}}\nfilename: {{.state.filename}}\n",
+ "reason": "{{.reason}}"
+ },
+ "startsAt": "{{._meta.timestamp}}",
+ "endsAt": "{{._meta.timestamp}}"
+ }
+ {{end}}
+ ],
+ {{$first := true}}
+ {{range .Alerts}}
+ {{if $first}}{{$first = false}}{{else}}
+ "commonLabels": {
+ "module": "{{._additions.module}}",
+ "alertname": "{{._additions.alertname}}",
+ "cluster": "{{._additions.cluster}}"
+ }
+ {{end}}
+ {{end}}
+ }
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ queue: ~
+ sink: ~
+ http:
+ enabled: true
+ port: 9196
+上面的template模版表示发送的告警内容格式，使用go template格式。可参考GO Template，或者请自行搜索go template使用教程。
+其中可使用类似{{._meta.timestamp}}等形式动态渲染原始alert数据中的字段。
alert字段解释：
+字段 |
+是否内置 |
+含义 |
+
|---|---|---|
| _meta | +是 | +alert元数据 | +
| _meta.pipelineName | ++ | 表示pipeline名称 | +
| _meta.sourceName | ++ | 表示source名称 | +
| _meta.timestamp | ++ | 表示日志时间戳 | +
| body | +是 | +logBody | +
| reason | +是 | +匹配成功原因 | +
| fields | +否 | +field字段，由其余配置添加 | +
| state | +否 | +采集信息，需要在file source配置addonMeta: true |
+
| _additions | +否 | +由配置指定 | +
原始alert数据为一个json，其中Alerts为固定的key。
原始alert数据示例
+ {
+ "Alerts": [
+ {
+ "_meta": {
+ "pipelineName": "default/spring",
+ "sourceName": "loggie-source-756fd6bb94-4skqv/loggie-alert/common",
+ "timestamp": "2022-10-28T13:12:30.528824+08:00"
+ },
+ "body": [
+ "2022-10-28 01:48:07.093 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/]. [dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause",
+ "",
+ "java.lang.ArithmeticException: / by zero"
+ ],
+ "fields": {
+ "containerid": "0dc5f07983bfdf7709ee4fce752679983c4184e94c70dab5fe6df5843d5cbb68",
+ "containername": "loggie-alert",
+ "logconfig": "spring",
+ "name": "loggie-source",
+ "namespace": "default",
+ "nodename": "docker-desktop",
+ "podname": "loggie-source-756fd6bb94-4skqv",
+ "topic": "loggie"
+ },
+ "reason": "matches some rules",
+ "state": {
+ "bytes": 6913,
+ "filename": "/var/log/pods/ default_loggie-source-756fd6bb94-4skqv_9da3e440-e749-4930-8e4d-41e0d5b66417/ loggie-alert/1.log",
+ "hostname": "docker-desktop",
+ "offset": 3836,
+ "pipeline": "default/spring",
+ "source": "loggie-source-756fd6bb94-4skqv/loggie-alert/common",
+ "timestamp": "2022-10-28T13:12:30.527Z"
+ },
+ "_additions": {
+ "namespace": "default",
+ "cluster": "local",
+ "alertname": "loggie-test",
+ "module": "loggie"
+ }
+ }
+ ]
+ }
+Pipeline中增加logAlert interceptor用于在采集的时候检测日志并匹配日志报警规则，可在ClusterLogConfig/LogConfig中引用。其中additions为给alert额外添加的字段，会放入alert原始数据的_addtions字段中，可用做模板渲染。
建议先使用debug模式（-log.level=debug）观察原始alert数据格式，再配置模板进行渲染，原始数据会受到其他配置的影响，这里仅展示一个示例。
详细配置可参考logAlert interceptor。
+Config
+apiVersion: loggie.io/v1beta1
+kind: Interceptor
+metadata:
+ name: logalert
+spec:
+ interceptors: |
+ - type: logAlert
+ matcher:
+ contains: ["ERROR"]
+ additions:
+ module: "loggie"
+ alertname: "loggie-test"
+ cluster: "local"
+匹配到日志告警规则之后，告警后端可接收到类似的数据如下所示：
+Example
+{
+"alerts": [
+ {
+ "labels": {
+ "topic": "loggie"
+ },
+ "annotations": {
+ "message": "\nNew alert: \nbody:\n2022-10-28 01:48:07.093 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\n\njava.lang.ArithmeticException: / by zero\ncontainerid: 0dc5f07983bfdf7709ee4fce752679983c4184e94c70dab5fe6df5843d5cbb68\nsource: loggie-source-756fd6bb94-4skqv/loggie-alert/common\ncontainername: loggie-alert\nlogconfig: spring\nname: loggie-source\nnamespace: default\nnodename: docker-desktop\npodname: loggie-source-756fd6bb94-4skqv\nfilename: /var/log/pods/default_loggie-source-756fd6bb94-4skqv_9da3e440-e749-4930-8e4d-41e0d5b66417/loggie-alert/1.log\n",
+ "reason": "matches some rules"
+ },
+ "startsAt": "2022-10-28T13:12:30.527Z",
+ "endsAt": "2022-10-28T13:12:30.527Z"
+ }
+],
+"commonLabels": {
+ "module": "loggie",
+ "alertname": "loggie-test",
+ "cluster": "local"
+ }
+}
+Loggie配置source采集日志，经过logAlert interceptor匹配时，可配置sendOnlyMatched仅将匹配成功的日志发送至alertWebhook sink，匹配失败的日志看作正常日志被忽略。建议在使用alertWebhook sink时，同时开启logAlert interceptor, 设置sendOnlyMatched为true搭配使用。
配置新增alertWebhook sink。详细配置可参考alertWebhook Sink。
Config
+ sink:
+ type: alertWebhook
+ addr: http://localhost:8080/loggie
+ linelimit: 10
+ template: |
+ {
+ "alerts":
+ [
+ {{$first := true}}
+ {{range .Alerts}}
+ {{if $first}}{{$first = false}}{{else}},{{end}}
+ {
+ "labels": {
+ "topic": "{{.fields.topic}}"
+ },
+ "annotations": {
+ "message": "\nNew alert: \nbody:\n{{range .body}}{{.}}\n {{end}}\ncontainerid: {{._meta.pipelineName}}\nsource: {{. _meta.sourceName}}\ncontainername: {{.fields. containername}}\nlogconfig: {{.fields.logconfig}}\nname: {{.fields.name}}\nnamespace: {{.fields.namespace}} \nnodename: {{.fields.nodename}}\npodname: {{.fields. podname}}\nfilename: {{.state.filename}}\n",
+ "reason": "{{.reason}}"
+ },
+ "startsAt": "{{._meta.timestamp}}",
+ "endsAt": "{{._meta.timestamp}}"
+ }
+ {{end}}
+ ],
+ {{$first := true}}
+ {{range .Alerts}}
+ {{if $first}}{{$first = false}}{{else}}
+ "commonLabels": {
+ "namespace": "{{._additions.namespace}}",
+ "module": "{{._additions.module}}",
+ "alertname": "{{._additions.alertname}}",
+ "cluster": "{{._additions.cluster}}"
+ }
+ {{end}}
+ {{end}}
+ }
+logAlert Interceptor配置和接收方收到的报警与采集链路检测报警类似。
为什么我的日志没有采集？
+日志采集中，最为关键核心的问题是，日志有没有被采集到，为什么我配置的日志没有发送过去？
+下面提供了核心的排查思路和手段供参考。
+另外，最重要的是，在环境里配置Loggie的Prometheus监控和Grafana图表，可以快速发现问题。
了解实现机制是排障的基础：
+（针对非Kubernetes的主机场景，只是少了LogConfig CRD配置下发的步骤，其余类似）
+在Kubernetes的场景下，目前Loggie提供了一个基于terminal的可交互式dashboard，可以更好的帮助我们便捷的排查问题。
+进入dashboard
+找到任意一个Loggie Pod +
kubectl -nloggie get po -owide
+进入其中一个Loggie Pod +
kubectl -nloggie exec -it ${podName} bash
+运行dashboard +
./loggie inspect
+使用terminal
+terminal首页展示示例如下所示：
+
具体请参考使用说明视频。
+Tips
+Loggie terminal功能仅从v1.4版本开始提供，如果从低版本升级，需增加clusterrole配置，请参考这里。
+排查问题关键先要确定是哪一步出现了问题。
+查看我们要排查的日志采集任务LogConfig/ClusterLogConfig的Events事件：
+kubectl -n ${namespace} describe lgc ${name}
+如果没有events，则可能为：
+ Pod Label未匹配：labelSelector指定的label未和我们期望的Pod匹配。通过以下命令查看
+ kubectl -n ${namespace} get po -owide -l ${labels}
+kubectl -n ns1 get po -owide -l app=tomcat,service=web
+ 来判断一下是否有匹配的Pod。如果没有类似sync success的events，可根据events同时结合Loggie日志排查问题：
+ 配置问题或者Loggie异常：kubectl -n ${loggie-namespace} logs -f ${loggie-pod-name} —-tail=${N}
+kubectl -nloggie logs -f loggie-5x6vf --tail=100。
+查看对应节点的Loggie日志，根据日志情况进行处理。 常见的异常有：
+/var/log/*.log。最快速的做法是，在需要采集的业务的Pod里，执行ls <path>，因为ls也是使用的glob表达式去匹配日志文件。另外也需额外注意是否配置了ignoreOlder/excludeFiles等参数，忽略或者排除了我们希望采集的日志文件。
找到logConfig匹配的pod所在节点的日志Agent¶根据在logConfig的labelSelector找到一个匹配的业务pod： +
kubectl -n ${namespace} get po -owide -l ${labels}
+${node-name}，然后通过：
+kubectl -n ${loggie-namespace} get po -owide |grep ${node-name}
+检查是否有异常，如果有异常，则需根据异常日志再做分析判断 +
kubectl -n ${loggie-namespace} logs -f ${loggie-pod-name} —-tail=${N}
+调用自动排障接口
+针对对应的Agent，调用help接口： +
curl <ip>:9196/api/v1/help
+接口返回里最开头包括了一个使用提示： +
--------- Usage: -----------------------
+|--- view details: /api/v1/help?detail=<module>, module is one of: all/pipeline/log
+|--- query by pipeline name: /api/v1/help?pipeline=<name>
+|--- query by source name: /api/v1/help?source=<name>
+curl <ip>:9196/api/v1/help?detail=all来查询所有的细节详情，也可以使用根据pipeline或者source的名称来搜索。
+目前返回主要包括两部分：
+正常情况下，该接口覆盖了以下操作得到的内容，无需继续进行以下操作，下文步骤仅供参考。
+查看节点Loggie渲染生成的配置
+进入到容器中： +
kubectl -n ${loggie-namespace} exec -it ${loggie-pod-name} bash
+ls /opt/loggie/pipeline/
+curl ${loggie-pod-ip}:9196/api/v1/reload/config
+确认日志采集配置
+cat /opt/loggie/pipeline/
+查看日志采集持久化状态
+Loggie会记录每个日志文件的采集状态，这样即使Loggie重启后，也可以继续保持上一次的采集进度，避免重新采集日志文件。 +可以通过调用接口来查看： +
curl ${loggie-pod-ip}:9196/api/v1/source/file/registry?format=text | grep XXX
+ {
+ "id": 85,
+ "pipelineName": "default/tomcat",
+ "sourceName": "tomcat-7d64c4f6c9-cm8jm/tomcat/common",
+ "filename": "/var/lib/kubelet/pods/9397b8be-8927-44ba-8b94-73e5a4459377/volumes/kubernetes.io~empty-dir/log/catalina.2022-06-02.log",
+ "jobUid": "3670030-65025",
+ "offset": 4960,
+ "collectTime": "2022-06-06 12:44:12.861",
+ "version": "0.0.1"
+ },
+filename为Loggie转换后的实际节点上的日志路径，jobUid组成为文件的inode-deviceId，offset为sink发送成功接收到ack后的offset。stat ${filename}
+stat
+ File: /var/lib/kubelet/pods/9397b8be-8927-44ba-8b94-73e5a4459377/volumes/kubernetes.io~empty-dir/log/catalina.2022-06-02.log
+ Size: 4960 Blocks: 16 IO Block: 4096 regular file
+Device: fe01h/65025d Inode: 3670030 Links: 1
+Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
+Access: 2022-06-06 12:44:12.859236003 +0000
+Modify: 2022-06-02 08:54:33.177240007 +0000
+Change: 2022-06-02 08:54:33.177240007 +0000
+可通过比较size和offset来判断采集进度。如果size=offset，则说明已经文件已经全部采集并发送成功。
+ + + + + + +现象
+可能在使用了比如unix sock source的时候，Loggie启动后crash，并有如下的日志：
+fatal error: unexpected signal during runtime execution
+[signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f59b4528360]
+
+runtime stack:
+runtime.throw({0x213b1a1, 0x7f59b423b640})
+ /usr/local/go/src/runtime/panic.go:1198 +0x71
+runtime.sigpanic()
+ /usr/local/go/src/runtime/signal_unix.go:719 +0x396
+
+goroutine 86 [syscall]:
+runtime.cgocall(0x1a39d30, 0xc000510d90)
+ /usr/local/go/src/runtime/cgocall.go:156 +0x5c fp=0xc000510d68 sp=0xc000510d30 pc=0x40565c
+net._C2func_getaddrinfo(0xc00030bc40, 0x0, 0xc000724fc0, 0xc0005bd990)
+ _cgo_gotypes.go:91 +0x56 fp=0xc000510d90 sp=0xc000510d68 pc=0x5c7bb6
+net.cgoLookupIPCNAME.func1({0xc00030bc40, 0xc0004d29c0, 0x4}, 0xc00030bb80, 0xc000510e50)
+ /usr/local/go/src/net/cgo_unix.go:163 +0x9f fp=0xc000510de8 sp=0xc000510d90 pc=0x5c98ff
+net.cgoLookupIPCNAME({0x20f751c, 0x3}, {0xc00030bb80, 0xc00022b5e0})
+ /usr/local/go/src/net/cgo_unix.go:163 +0x16d fp=0xc000510f38 sp=0xc000510de8 pc=0x5c914d
+net.cgoIPLookup(0x358aed0, {0x20f751c, 0xc00030bbc0}, {0xc00030bb80, 0xc000510fb8})
+ /usr/local/go/src/net/cgo_unix.go:220 +0x3b fp=0xc000510fa8 sp=0xc000510f38 pc=0x5c99bb
+net.cgoLookupIP·dwrap·25()
+ /usr/local/go/src/net/cgo_unix.go:230 +0x36 fp=0xc000510fe0 sp=0xc000510fa8 pc=0x5c9e36
+runtime.goexit()
+ /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000510fe8 sp=0xc000510fe0 pc=0x46ae81
+created by net.cgoLookupIP
+ /usr/local/go/src/net/cgo_unix.go:230 +0x125
+原因
+具体原因请参考：go net。
+Name Resolution
+
+The method for resolving domain names, whether indirectly with functions like Dial
+or directly with functions like LookupHost and LookupAddr, varies by operating system.
+
+On Unix systems, the resolver has two options for resolving names.
+It can use a pure Go resolver that sends DNS requests directly to the servers
+listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls C
+library routines such as getaddrinfo and getnameinfo.
+
+By default the pure Go resolver is used, because a blocked DNS request consumes
+only a goroutine, while a blocked C call consumes an operating system thread.
+When cgo is available, the cgo-based resolver is used instead under a variety of
+conditions: on systems that do not let programs make direct DNS requests (OS X),
+when the LOCALDOMAIN environment variable is present (even if empty),
+when the RES_OPTIONS or HOSTALIASES environment variable is non-empty,
+when the ASR_CONFIG environment variable is non-empty (OpenBSD only),
+when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the
+Go resolver does not implement, and when the name being looked up ends in .local
+or is an mDNS name.
+
+The resolver decision can be overridden by setting the netdns value of the
+GODEBUG environment variable (see package runtime) to go or cgo, as in:
+
+ export GODEBUG=netdns=go # force pure Go resolver
+ export GODEBUG=netdns=cgo # force cgo resolver
+
+The decision can also be forced while building the Go source tree
+by setting the netgo or netcgo build tag.
+解决办法
+在Loggie 部署脚本里增加环境变量：
+ env:
+ - name: GODEBUG
+ value: netdns=go
+阅读本文之前，建议先参考Kubernetes下日志采集问题。
+由于Kubernetes良好的可扩展性，Kubernetes设计了一种自定义资源CRD的概念，用户可以自己定义CRD表示自己的期望状态，并借助一些framework开发Controller，使用Controller将我们的期望变成现实。
+基于这个思路，一个服务需要采集哪些日志，需要什么样的日志配置，是用户的期望，而这就需要我们开发一个日志采集的Controller去实现。
+所以，用户只需要在我们定义的CRD LogConfig中，填写需要采集哪些Pods的日志，在Pod中的日志路径是什么。
+核心架构如下图所示：
+
Loggie会感知到Pod和CRD的事件，进行配置的动态更新。同时，Loggie可以根据日志文件路径挂载的Volume，找到相应在节点的文件进行采集。另外还可以根据配置，自动将Pod上的Env/Annotation/Label加入到日志里作为元信息。
+同时相比粗暴的所有节点挂载相同路径进行通配采集的方式，也解决了没法针对单个服务精细化配置、采集无关日志的问题。
+当然带来的好处不仅仅这些，Loggie在动态配置下发、监控指标等方面都可以基于Kubernetes进行相应的适配和支持。
+Loggie目前有以下几种CRD：
+使用CRD的流程架构如下所示：
+
整体的部署架构是什么？
+本文仅关注采集端，如果需要使用部署Loggie Aggregator，请参考Loggie中转机。
+在采集容器日志之前，请确保已经在Kubernetes中部署了Loggie DaemonSet。如何在Kubernetes中部署Loggie？
+Info
+我们推荐使用DaemonSet的方式采集容器日志，后续Loggie计划支持自动注入Loggie Sidecar的方式采集日志。RoadMap
+业务Pod使用什么方式挂载日志？
+建议如果对日志可能性的丢失不敏感，比如运维类的日志，可以使用emptyDir。如果日志比较重要，不允许丢失，请使用hostPath，并在volumeMount配置subPathExpr实现路径隔离。
Example
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ labels:
+ app: tomcat
+ name: tomcat
+ namespace: default
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: tomcat
+ template:
+ metadata:
+ labels:
+ app: tomcat
+ spec:
+ containers:
+ - name: tomcat
+ image: tomcat
+ volumeMounts:
+ - mountPath: /usr/local/tomcat/logs
+ name: log
+ volumes:
+ - emptyDir: {}
+ name: log
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ labels:
+ app: tomcat
+ name: tomcat
+ namespace: default
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: tomcat
+ template:
+ metadata:
+ labels:
+ app: tomcat
+ spec:
+ containers:
+ - env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ apiVersion: v1
+ fieldPath: metadata.name
+ - name: NAMESPACE
+ valueFrom:
+ fieldRef:
+ apiVersion: v1
+ fieldPath: metadata.namespace
+ image: tomcat
+ name: tomcat
+ volumeMounts:
+ - mountPath: /log
+ name: datalog
+ subPathExpr: $(NAMESPACE)/$(POD_NAME)
+ volumes:
+ - hostPath:
+ path: /data/log
+ type: ""
+ name: datalog
+Caution
+请确保Pod挂载的日志路径不会被多个Pod共享。比如一个Deployment下的两个Pod，使用未配置subPathExpr的hostPath挂载日志目录，如果两个Pod调度在相同的节点，会打印到相同的日志目录和文件中，此时可能引起采集异常。
+我们首先创建以上示例的tomcat Deployment作为我们被采集日志的对象。 +
kubectl apply -f tomcat-emptydir.yml
+接下来将演示采集上面示例的tomcat日志，并发送至Elasticsearch。
+如果你希望采集其他的容器日志，或者发送至其他的后端，只需要修改其中的配置即可。
由于本文演示发送至Elasticsearch，所以这里我们部署一下Elasticsearch和Kibana。
+如果你的环境中已经有Elasticsearch和Kibana，这一步可以忽略。
+如果希望自己搭建一套Elasticsearch和Kibana试用，这里建议：
如果本地没有helm客户端，需要下载helm。
+使用如下命令: +
helm repo add elastic https://helm.elastic.co
+helm install elasticsearch elastic/elasticsearch --set replicas=1
+helm install kibana elastic/kibana
+通过：
+kubectl port-forward service/kibana-kibana 5601:http
+可以直接在浏览器输入localhost:5601，访问Kibana页面。
+为了表明我们即将采集日志发送到的Elasticsearch，需要配置对应的Sink。
+这里有两种方式：
如果整个集群只会有一个存储后端，我们可以在全局的配置文件configMap里，配置defaults参数，具体可参考。
+使用Sink CRD，并在logConfig中引用。这种方式可以扩展为多个后端，不同的logConfig可以配置使用不同的后端存储，大多数情况下，我们建议使用该方式。
+创建一个Sink如下，Sink为集群级别CRD。可以在spec.sink中修改为其他配置。
+Example
+cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: Sink
+metadata:
+ name: default
+spec:
+ sink: |
+ type: elasticsearch
+ index: "loggie"
+ hosts: ["elasticsearch-master.default.svc:9200"]
+EOF
+可以通过kubectl get sink default查看是否创建成功。
和Sink一样，配置interceptor有两种方式：
+在全局配置文件configMap中，配置defaults参数。此时意味着所有的Loggie Agent都使用该default interceptors配置。
+使用Interceptor CRD，并在logConfig中被引用。这种方式会更加灵活，适合有一定运维和功能需求的场景。
+Loggie目前默认内置了metric（监控指标发送）、 maxbytes（最大event字节数限制）、 retry（重试）三个interceptors。即使我们没有使用以上两种方式配置，仍然会自动使用这3个默认interceptor，所以interceptor其实是可选项。
Caution
+创建一个interceptor示例如下：
+Example
+cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: Interceptor
+metadata:
+ name: default
+spec:
+ interceptors: |
+ - type: rateLimit
+ qps: 90000
+EOF
+这里我们创建了一个rateLimit interceptor，可用于采集发送日志的限流。
+可以通过kubectl get interceptor default或者kubectl get icp default查看是否创建成功。
创建完sink和interceptor后，最重要的便是创建logConfig，表明我们希望采集哪些Pod的日志，以及采集Pod的哪些日志。
+logConfig里spec.selector部分表示日志配置的下发范围，对采集Pod日志来说，需要配置type: pod，并且使用labelSelector选择指定的Pods，即采集这些Pods的日志。
+示例如下：
+
spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: tomcat
+另外，我们需要在pipeline.sources中配置file source，即采集Pods的哪些日志。
+
pipeline:
+ sources: |
+ - type: file
+ name: mylog
+ paths:
+ - stdout
+ - /var/log/*.log
+stdout，表示采集容器的标准输出路径。type: pod时，pipeline.name会被loggie自动按照${namespace}-${logConfigName}的格式生成，所以这里无需填写。 ${podName}-${containerName}-${sourceName}格式生成。 最后，我们使用sinkRef和interceptorRef引用上面创建的sink和interceptor。
+最终创建一个logConfig示例如下：
Example
+cat << EOF | kubectl apply -f -
+apiVersion: loggie.io/v1beta1
+kind: LogConfig
+metadata:
+ name: tomcat
+ namespace: default
+spec:
+ selector:
+ type: pod
+ labelSelector:
+ app: tomcat
+ pipeline:
+ sources: |
+ - type: file
+ name: common
+ paths:
+ - stdout
+ - /usr/local/tomcat/logs/*.log
+ sinkRef: default
+ interceptorRef: default
+EOF
+我们可以通过kubectl get logconfig tomcat或者kubectl get lgc tomcat来查看是否创建成功。
+同时可以通过kubectl describe lgc tomcat查看logConfig的events，用于排查状态。
+如果有以下类似的events，说明配置已经下发成功。
+
Events:
+ Type Reason Age From Message
+ ---- ------ ---- ---- -------
+ Normal syncSuccess 55s loggie/kind-control-plane Sync type pod [tomcat-684c698b66-hvztn] success
+我们还可以通过kubectl -nloggie logs -f ${loggie-name}来查看指定节点Loggie的日志来判断日志采集情况。
发送成功后，我们可以在Kibana上查询到采集到的日志。
+正常情况下，我们采集到的标准输出并不是打印的日志内容，而是被容器运行时增加了一层封装。
+例如docker的标准输出为json形式： +
{"log":"I0610 08:29:07.698664 Waiting for caches to sync\n", "stream":"stderr", "time:"2021-06-10T08:29:07.698731204Z"}
+containerd的标准输出形式类似如下： +
2021-12-01T03:13:58.298476921Z stderr F INFO [main] Starting service [Catalina]
+2021-12-01T03:13:58.298476921Z stderr F为运行时增加的前缀内容，后面则为原始的日志。
+特别是如果我们配置了日志采集多行配置，因为采集到的日志内容和业务输出的日志不一致，会导致采集标准输出日志匹配多行有问题。
+所以，Loggie提供了一键开关配置，在系统配置中，将parseStdout参数设置为true即可。
parseStdout
+config:
+ loggie:
+ discovery:
+ enabled: true
+ kubernetes:
+ parseStdout: true
+Loggie会在渲染LogConfig的时候自动增加source codec解析出原始的业务日志。
+需要注意的是：
+仅当LogConfig配置里的paths单独为stdout时有效：
+ 【有效】
+
sources: |
+ - type: file
+ name: common
+ paths:
+ - stdout
+ sources: |
+ - type: file
+ name: common
+ paths:
+ - stdout
+ - /usr/local/tomcat/logs/*.log
+ sources: |
+ - type: file
+ name: stdout
+ paths:
+ - stdout
+ - type: file
+ name: tomcat
+ paths:
+ - /usr/local/tomcat/logs/*.log
+目前只会保留原始的日志内容到body中，其余运行时附加的字段会被丢弃。
虽然我们建议使用挂载volume(emptyDir/hostPath+subPathExpr)的方式将日志文件挂出给Loggie采集，但仍然存在很多情况我们没办法将业务的Pod统一挂载出日志路径。
+比如一些基础组件无法配置独立的日志volume，或者业务单纯的不愿意改部署配置。
Loggie提供了无需挂载即可采集容器日志的能力，可自动识别并采集容器root filesystem里的日志文件。
+你只需要将部署的helm chart中values.yml里的配置rootFsCollectionEnabled设置为true，
+同时填上实际环境的容器运行时（docker/containerd），如下所示：
rootFsCollectionEnabled
+config:
+ loggie:
+ discovery:
+ enabled: true
+ kubernetes:
+ containerRuntime: containerd
+ rootFsCollectionEnabled: false
+修改完成后，重新helm upgrade即可。
+helm模版会自动渲染增加额外的一些挂载路径和配置，如果你从低版本升级，需要额外修改部署的Daemonset yaml。
+具体原理请参考issues #208。
在Kubernetes集群中，除了采集Pod里的日志，还可能有采集Node节点上的一些诸如kubelet日志、系统日志等需求。
+和采集容器不同的是，节点日志采集需使用集群级别的ClusterLogConfig，selector使用type: node，并且填写nodeSelector用于选择下发配置到哪些节点，同时需确保Node上包含这些labels。
示例如下：
+Example
+apiVersion: loggie.io/v1beta1
+kind: ClusterLogConfig
+metadata:
+ name: varlog
+spec:
+ selector:
+ type: node
+ nodeSelector:
+ nodepool: demo
+ pipeline:
+ sources: |
+ - type: file
+ name: varlog
+ paths:
+ - /var/log/*.log
+ sinkRef: default
+ interceptorRef: default
+另外应注意的是，如果需要采集Node节点上某路径的日志，需要Loggie同样挂载相同的路径，否则由于容器隔离性Loggie无法获取到节点的日志。
+比如采集Node上/var/log/路径下的日志，需要Loggie Agent增加挂载该路径。
Cite
+相比传统的主机日志采集，在Kubernetes集群中，采集容器日志有一些差异，使用方式上也有所区别。 +这里我们罗列了一些常规的部署和使用方式以供参考。
+在传统的使用虚拟机/云主机/物理机的时代，业务进程部署在固定的节点上，业务日志直接输出到宿主机上，运维只需要手动或者使用自动化工具把日志采集Agent部署在节点上，加一下Agent的配置，就可以开始采集日志了。
+而在Kubernetes环境中，情况就没这么简单了：
+以上都是有别于传统日志采集配置方式的需求和痛点，究其原因，还是因为传统的方式脱离了Kubernetes，无法感知Kubernetes，无法和Kubernetes集成。
+为了采集容器日志，我们先来看一下市面上一般都有哪些解决方案。
+首先，需要提及的是，在云原生的12要素里，推荐业务容器将日志输出到stdout中，而不是采用打印日志文件的方式。当然，实际情况是，我们很难这么做，原因大概有：
+所以正常情况下，我们需要同时采集：
+采集容器日志，Agent有两种部署方式：
+两种部署方式的优劣都显而易见：
+Tip
+正常情况下，优先使用DaemonSet的方式采集日志，如果单个Pod日志量特别大，超过一般Agent发送吞吐量，可以单独对该Pod使用Sidecar的方式采集日志。
+如果使用容器运行时的是docker，正常情况下我们可以在节点的docker路径中找到容器的stdout的日志，默认为/var/lib/docker/containers/{containerId}/{containerId}-json.log。
在Kubernetes 1.14版本之前，kubelet会在/var/log/pods/<podUID>/<containerName>/<num>.log建立一个软链接到stdout文件中。
类似如下所示：
+
root@master0:/var/log/pods# tree .
+|-- 6687e53201c01e3fad31e7d72fbb92a6
+| `-- kube-apiserver
+| |-- 865.log -> /var/lib/docker/containers/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03-json.log
+| `-- 866.log -> /var/lib/docker/containers/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662-json.log
+|-- a1083c6d-3b12-11ea-9af1-fa163e28f309
+| `-- kube-proxy
+| |-- 3.log -> /var/lib/docker/containers/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e-json.log
+| `-- 4.log -> /var/lib/docker/containers/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207-json.log
+在Kubernetes 1.14版本之后，改成了/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/<num>.log的形式。
root@master-0:/var/log/pods# tree .
+|-- kube-system_kube-apiserver-kind-control-plane_bd1c21fe1f0ef615e0b5e41299f1be61
+| `-- kube-apiserver
+| `-- 0.log
+|-- kube-system_kube-proxy-gcrfq_f07260b8-6055-4c19-9491-4a825579528f
+| `-- kube-proxy
+| `-- 0.log
+`-- loggie_loggie-csd4g_f1cc32e9-1002-4e64-bd58-fc6094394e06
+ `-- loggie
+ `-- 0.log
+所以，对于Agent采集标准输出日志来说，也就是采集节点上的这些日志文件。
+一种简单粗暴的采集方式是，使用DaemonSet部署日志Agent，挂载/var/log/pods目录，Agent的配置文件使用类似/var/log/pod/*/*.log去通配日志文件，采集节点上所有的容器标准输出。
但是这样的局限在于：
+当然现在的一些日志Agent比如Filebeat/Fluentd都针对性的做了支持，比如可以将namespace/pod等信息注入日志中，但仍然没有解决大部分的问题。
+所以，这种方式只适合简单的业务场景，后续也难以满足其他更多的日志需求。
+如果Pod里不仅仅是输出stdout，还包括日志文件，就需要考虑到挂载日志文件到节点上，同时采用DaemonSet部署的Agent也需要挂载相同的目录，否则采用容器化部署的Agent无法查看到相应的文件，更无法采集。
+业务Pod挂载日志路径的方式有以下几种：
+(1) emtpyDir
+emtpyDir的生命周期跟随Pod，Pod销毁后其中存储的日志也会消失。
+使用emptyDir挂载的日志文件，一般在节点的路径如下： +
/var/lib/kubelet/pods/${pod.UID}/volumes/kubernetes.io~empty-dir/${volumeName}
+(2) hostPath
+生命周期和Pod无关，Pod迁移或者销毁，日志文件还保留在现有磁盘上。
+为了解决隔离性，避免多个Pod打印日志到相同的路径和文件中，我们需要使用 subPathExpr 字段从 Downward API 环境变量构造 subPath 目录名。 +该 VolumeSubpathEnvExpansion 功能从 Kubernetes1.15 开始默认开启，在1.17 GA。可参考 feature-gates 和 using-subpath-expanded-environment。
+使用subPathExpr的示例如下所示： +
apiVersion: apps/v1
+kind: Deployment
+metadata:
+ labels:
+ app: nginx
+ name: nginx
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ app: nginx
+ template:
+ metadata:
+ labels:
+ app: nginx
+ spec:
+ containers:
+ - env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ apiVersion: v1
+ fieldPath: metadata.name
+ - name: NAMESPACE
+ valueFrom:
+ fieldRef:
+ apiVersion: v1
+ fieldPath: metadata.namespace
+ image: nginx
+ name: nginx
+ resources: {}
+ volumeMounts:
+ - mountPath: /data/log
+ name: datalog
+ subPathExpr: $(NAMESPACE)/$(POD_NAME)
+ volumes:
+ - hostPath:
+ path: /data/log
+ type: ""
+ name: datalog
+在创建完之后，我们可以在所在节点的/data/log下发现类似的目录结构：
+
.
+`-- default
+ |-- nginx-888dc6778-krfqr
+ `-- nginx-888dc6778-sw8vd
+/data/log/$(NAMESPACE)/$(POD_NAME)路径下。
+(3) Pv
+Pv的访问模式包括：
对于大部分的业务来说，都是Deployment无状态部署，需要挂载同一个Pv共享；对于一些中间件等有状态服务，一般会使用StatefulSet部署，每个Pod会使用独立的Pv。
+虽然同样可以在Node上找到使用Pv挂载的对应日志文件，但是Pv根据不同的底层实现，在Node上的路径会有一定的区别。
+目前市面上大部分日志Agent均对这些挂载方式没有感知，所以你能做的和上面使用stdout的方式类似，也就是简单粗暴的让Agent将路径都挂载，使用通配的方式采集所有的日志，使用上的局限和stdout的方式同样一致。
+
另外，鉴于一些Agent对采集docker stdout有一定的支持，所以还存在一些使用上变种，比如利用webhook注入一个sidecar，读取Pod里的日志文件，转换成sidecar的stdout，然后采集sidecar的stdout日志，这里不再详述。
+（4）不挂载
+很多情况下，用户都疏于挂载日志路径volume，或者对接一个已经存在的系统，往往业务方不愿意更改，这个时候能采集容器里的日志文件吗？
+Loggie尝试解决了这个问题，可以通过配置开启，不过这种方式仍然需要更多长期生产实践的检验。欢迎试用，具体开关请参考下一节。
+如果需要采用sidecar的方式，需要将日志Agent同时挂载相同的日志路径，Agent的配置文件也一般使用ConfigMap挂载出来。
+使用Sidecar的部署方式和DaemonSet的方式优劣对比请参考上文。
+
总结：
+大部分主流的开源Agent，只对容器Stdout有部分支持，比如支持采集的时候统一加上一些K8s相关元信息，但不支持单独针对某些Pod进行配置，而且均没有对容器中的日志文件采集有很好的支持。
+这样导致在实际复杂的各类业务场景中，没办法满足需求。
相信以上列出的问题大家都遇到或者思考过，那Loggie是如何解决这些问题的呢？
+请看下一节如何 使用Loggie采集容器日志？
除了使用Logconfig采集日志外，Loggie同样可以通过CRD配置任意的source/sink/interceptor，本质上Loggie就是一个支持多Pipeline的数据流，集成了通用的诸如队列重试、数据处理、配置下发、监控报警等等功能，减少了类似需求的研发成本。采集Kubernetes的Events就是其中一个很好的例子。
+Kubernetes Events是由Kubernetes本身组件和一些控制器产生的事件，我们常用的kubectl describe命令就可以查看关联资源的事件信息，采集记录这些事件可以帮助我们回溯、排查、审计、总结问题，更好的了解Kubernetes集群内部状态。
和Loggie中转机类似，我们可以单独部署Aggregator集群或者复用现有的中转机集群。
+配置kubeEvents source，并且使用type: cluster下发配置到Aggregator集群即可。
Config
+apiVersion: loggie.io/v1beta1
+kind: ClusterLogConfig
+metadata:
+ name: kubeevent
+spec:
+ selector:
+ type: cluster
+ cluster: aggregator
+ pipeline:
+ sources: |
+ - type: kubeEvent
+ name: event
+ sinkRef: dev
+默认情况下，不管是发送给Elasticsearch还是其他的sink，输出的是类似如下格式的数据:
+event
+{
+"body": "{\"metadata\":{\"name\":\"loggie-aggregator.16c277f8fc4ff0d0\",\"namespace\":\"loggie-aggregator\",\"uid\":\"084cea27-cd4a-4ce4-97ef-12e70f37880e\",\"resourceVersion\":\"2975193\",\"creationTimestamp\":\"2021-12-20T12:58:45Z\",\"managedFields\":[{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2021-12-20T12:58:45Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:count\":{},\"f:firstTimestamp\":{},\"f:involvedObject\":{\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:lastTimestamp\":{},\"f:message\":{},\"f:reason\":{},\"f:source\":{\"f:component\":{}},\"f:type\":{}}}]},\"involvedObject\":{\"kind\":\"DaemonSet\",\"namespace\":\"loggie-aggregator\",\"name\":\"loggie-aggregator\",\"uid\":\"7cdf4792-815d-4eba-8a81-d60131ad1fc4\",\"apiVersion\":\"apps/v1\",\"resourceVersion\":\"2975170\"},\"reason\":\"SuccessfulCreate\",\"message\":\"Created pod: loggie-aggregator-pbkjk\",\"source\":{\"component\":\"daemonset-controller\"},\"firstTimestamp\":\"2021-12-20T12:58:45Z\",\"lastTimestamp\":\"2021-12-20T12:58:45Z\",\"count\":1,\"type\":\"Normal\",\"eventTime\":null,\"reportingComponent\":\"\",\"reportingInstance\":\"\"}",
+"systemPipelineName": "default/kubeevent/",
+"systemSourceName": "event"
+}
+为了方便分析展示，我们可以添加一些interceptor将采集到的events数据json decode。
+配置示例如下，具体请参考日志切分与处理。
+Config
+apiVersion: loggie.io/v1beta1
+kind: Interceptor
+metadata:
+ name: jsondecode
+spec:
+ interceptors: |
+ - type: normalize
+ name: json
+ processors:
+ - jsonDecode: ~
+ - drop:
+ targets: ["body"]
+apiVersion: loggie.io/v1beta1
+kind: ClusterLogConfig
+metadata:
+ name: kubeevent
+spec:
+ selector:
+ type: cluster
+ cluster: aggregator
+ pipeline:
+ sources: |
+ - type: kubeEvent
+ name: event
+ interceptorRef: jsondecode
+ sinkRef: dev
+经过normalize interceptor里jsonDecode后的数据如下所示：
+event
+{
+"metadata": {
+ "name": "loggie-aggregator.16c277f8fc4ff0d0",
+ "namespace": "loggie-aggregator",
+ "uid": "084cea27-cd4a-4ce4-97ef-12e70f37880e",
+ "resourceVersion": "2975193",
+ "creationTimestamp": "2021-12-20T12:58:45Z",
+ "managedFields": [
+ {
+ "fieldsType": "FieldsV1",
+ "fieldsV1": {
+ "f:type": {
+
+ },
+ "f:count": {
+
+ },
+ "f:firstTimestamp": {
+
+ },
+ "f:involvedObject": {
+ "f:apiVersion": {
+
+ },
+ "f:kind": {
+
+ },
+ "f:name": {
+
+ },
+ "f:namespace": {
+
+ },
+ "f:resourceVersion": {
+
+ },
+ "f:uid": {
+
+ }
+ },
+ "f:lastTimestamp": {
+
+ },
+ "f:message": {
+
+ },
+ "f:reason": {
+
+ },
+ "f:source": {
+ "f:component": {
+
+ }
+ }
+ },
+ "manager": "kube-controller-manager",
+ "operation": "Update",
+ "apiVersion": "v1",
+ "time": "2021-12-20T12:58:45Z"
+ }
+ ]
+},
+"reportingComponent": "",
+"type": "Normal",
+"message": "Created pod: loggie-aggregator-pbkjk",
+"reason": "SuccessfulCreate",
+"reportingInstance": "",
+"source": {
+ "component": "daemonset-controller"
+},
+"count": 1,
+"lastTimestamp": "2021-12-20T12:58:45Z",
+"firstTimestamp": "2021-12-20T12:58:45Z",
+"eventTime": null,
+"involvedObject": {
+ "kind": "DaemonSet",
+ "namespace": "loggie-aggregator",
+ "name": "loggie-aggregator",
+ "uid": "7cdf4792-815d-4eba-8a81-d60131ad1fc4",
+ "apiVersion": "apps/v1",
+ "resourceVersion": "2975170"
+},
+}
+如果觉得数据字段太多或者格式不符合需求，还可以配置normalize interceptor进行修改。
+ + + + + + +++虽然默认不推荐使用Sidecar方式采集容器日志，但是在某些受限场景下只能选择使用Sidecar的方式采集容器日志，这里我们给出一个参考的示例。
+
如下图所示：
+
Loggie和业务container部署在同一个Pod里，同时需要挂载相同的日志文件volume，另外Loggie的配置可通过configMap的方式挂载到容器中，Loggie根据提供的configMap配置文件，采集容器的日志，并发送到后端。
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ labels:
+ app: tomcat
+ name: tomcat
+ namespace: default
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: tomcat
+ template:
+ metadata:
+ labels:
+ app: tomcat
+ spec:
+ containers:
+ - name: tomcat
+ image: tomcat
+ volumeMounts:
+ - mountPath: /usr/local/tomcat/logs
+ name: log
+
+ - name: loggie
+ args:
+ - -config.system=/opt/loggie/loggie.yml
+ - -config.pipeline=/opt/loggie/pipeline.yml
+ image: loggieio/loggie:main
+ volumeMounts:
+ # loggie和业务container挂载相同的log volume
+ - mountPath: /usr/local/tomcat/logs
+ name: log
+ # 挂载日志配置configMap
+ - mountPath: /opt/loggie
+ name: loggie-config
+ # 挂载loggie自身持久化的数据
+ - mountPath: /data
+ name: registry
+
+ volumes:
+ - emptyDir: {}
+ name: log
+ - emptyDir: {}
+ name: registry
+ - name: loggie-config
+ configMap:
+ name: tomcat-loggie-config
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: tomcat-loggie-config
+ namespace: default
+data:
+ loggie.yml: |
+ loggie:
+ reload:
+ enabled: true
+ period: 10s
+ monitor:
+ logger:
+ period: 30s
+ enabled: true
+ listeners:
+ filesource: ~
+ filewatcher: ~
+ reload: ~
+ sink: ~
+ http:
+ enabled: true
+ port: 9196
+ pipeline.yml: |
+ pipelines:
+ - name: "tomcat"
+ sources:
+ - type: "file"
+ name: "tomcatlog"
+ paths:
+ - "/usr/local/tomcat/logs/*.log"
+ fields:
+ namespace: default
+ deployment: tomcat
+ fieldsFromEnv:
+ podname: HOSTNAME
+ sink:
+ type: "dev"
+ printEvents: true
+ codec:
+ pretty: true
+可以参考以上的部署方式通过loggie sidecar采集容器日志。
+需要注意的是：
Tips
+env命令查看。 Info
+Loggie后续会支持自动Sidecar注入和通过LogConfig自动生成ConfigMap挂载的方式，从而达到和使用DaemonSet一致的体验。
+