diff --git a/docs/reference/pipelines/sink/franzkafka.md b/docs/reference/pipelines/sink/franzkafka.md index 0e35743..511bea5 100644 --- a/docs/reference/pipelines/sink/franzkafka.md +++ b/docs/reference/pipelines/sink/franzkafka.md @@ -5,12 +5,26 @@ !!! example - ```yaml - sink: - type: franzKafka - brokers: ["127.0.0.1:6400"] - topic: "log-${fields.topic}" - ``` + === "简单" + ```yaml + sink: + type: franzKafka + brokers: ["127.0.0.1:6400"] + topic: "log-${fields.topic}" + ``` + + === "SASL认证" + ```yaml + sink: + type: kafka + brokers: ["127.0.0.1:6400"] + topic: "demo" + sasl: + enabled: true + mechanism: SCRAM-SHA-512 + username: *** + password: *** + ``` ## brokers @@ -48,6 +62,50 @@ ``` 可配置`topic: ${fields.topic}`，同样也会发送到topic "loggie"。 +## ifRenderTopicFailed + +| `字段` | `类型` | `是否必填` | `默认值` | `含义` | +| ---------- | ----------- | ----------- | --------- | -------- | +| ifRenderTopicFailed | | 非必填 | | 如果使用动态的规则渲染topic，比如`topic: ${fields.topic}`，有可能渲染失败（比如日志没有fields.topic字段），下面配置指示失败后的动作。 | +| ifRenderTopicFailed.dropEvent | | 非必填 | true | 默认为丢弃 | +| ifRenderTopicFailed.ignoreError | | 非必填 | | 忽略错误日志，请注意，这里只是不打印错误日志 | +| ifRenderTopicFailed.defaultTopic | | 非必填 | | 发送至该默认的topic，配置后dropEvent不生效 | + +!!! example + + === "1" + + ```yaml + sink: + type: kafka + brokers: ["127.0.0.1:6400"] + topic: "log-${fields.topic}" + ifRenderTopicFailed: + dropEvent: true + ``` + + === "2" + + ```yaml + sink: + type: kafka + brokers: ["127.0.0.1:6400"] + topic: "log-${fields.topic}" + ifRenderTopicFailed: + ignoreError: true + defaultTopic: default + ``` + +## ignoreUnknownTopicOrPartition + +| `字段` | `类型` | `是否必填` | `默认值` | `含义` | +| ---------- | ----------- | ----------- | --------- | -------- | +| ignoreUnknownTopicOrPartition | | 非必填 | | 用于当发送的topic不存在时，忽略Kafka返回UNKNOWN_TOPIC_OR_PARTITION的报错 | + +- 这种情况一般发生在使用动态渲染的topic，但是环境里的Kafka关闭了自动创建topic，导致无法发送至渲染出来的topic。默认情况Loggie会一直不停的重试，从而无法发送新的日志。 +- 开启ignoreUnknownTopicOrPartition后，会直接丢弃发送的日志，避免影响其他正常包含已存在topic的日志发送。 +- 请注意和上面`ifRenderTopicFailed`的区别，`ifRenderTopicFailed`是动态渲染不出来topic或者渲染的为空值，而`ignoreUnknownTopicOrPartition`则是渲染成功，但是topic在Kafka中实际不存在。 + ## balance @@ -122,31 +180,30 @@ | ---------- |--------| ----------- | ------ |-------------------------------------| | security | string | 非必填 | | java格式的安全认证内容，可以自动转化成为franz-go适配的格式 | -案例: -``` -pipelines: - - name: local - sources: - - type: file - name: demo - paths: - - /tmp/log/*.log - sink: - type: franzKafka - brokers: - - "hadoop74.axrzpt.com:9092" - topic: loggie - writeTimeout: 5s - sasl: - gssapi: - kerberosConfigPath: /etc/krb5-conf/krb5.conf - security: - security.protocol: "SASL_PLAINTEXT" - sasl.mechanism: "GSSAPI" - sasl.jaas.config: "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true keyTab=\"/shylock/kerberos/zork.keytab\" principal=\"zork@AXRZPT.COM\";" - sasl.kerberos.service.name: "kafka" - -``` +!!! example + ``` + pipelines: + - name: local + sources: + - type: file + name: demo + paths: + - /tmp/log/*.log + sink: + type: franzKafka + brokers: + - "hadoop74.axrzpt.com:9092" + topic: loggie + writeTimeout: 5s + sasl: + gssapi: + kerberosConfigPath: /etc/krb5-conf/krb5.conf + security: + security.protocol: "SASL_PLAINTEXT" + sasl.mechanism: "GSSAPI" + sasl.jaas.config: "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true keyTab=\"/shylock/kerberos/zork.keytab\" principal=\"zork@AXRZPT.COM\";" + sasl.kerberos.service.name: "kafka" + ``` Kubernetes 挂载keytab二进制证书，请参考[官方文档](https://kubernetes.io/zh-cn/docs/concepts/configuration/secret/)。 diff --git a/docs/reference/pipelines/sink/kafka.md b/docs/reference/pipelines/sink/kafka.md index 163a8db..9bd20cb 100644 --- a/docs/reference/pipelines/sink/kafka.md +++ b/docs/reference/pipelines/sink/kafka.md @@ -4,12 +4,27 @@ !!! example - ```yaml - sink: - type: kafka - brokers: ["127.0.0.1:6400"] - topic: "log-${fields.topic}" - ``` + === "简单" + ```yaml + sink: + type: kafka + brokers: ["127.0.0.1:6400"] + topic: "log-${fields.topic}" + ``` + + === "SASL认证" + ```yaml + sink: + type: kafka + brokers: ["127.0.0.1:6400"] + topic: "demo" + sasl: + type: scram + userName: *** + password: *** + algorithm: sha512 + ``` + ## brokers @@ -87,9 +102,9 @@ | ---------- | ----------- | ----------- | --------- | -------- | | ignoreUnknownTopicOrPartition | | 非必填 | | 用于当发送的topic不存在时，忽略Kafka返回UNKNOWN_TOPIC_OR_PARTITION的报错 | -这种情况一般发生在使用动态渲染的topic，但是环境里的Kafka关闭了自动创建topic，导致无法发送至渲染出来的topic。默认情况Loggie会一直不停的重试，从而无法发送新的日志。 -开启ignoreUnknownTopicOrPartition后，这种情况下会直接丢弃发送的日志，避免影响其他正常包含已存在topic的日志发送。 -请注意和上面`ifRenderTopicFailed`的区别，`ifRenderTopicFailed`是动态渲染不出来topic或者渲染的为空值，而`ignoreUnknownTopicOrPartition`则是渲染成功，但是topic在Kafka中实际不存在。 +- 这种情况一般发生在使用动态渲染的topic，但是环境里的Kafka关闭了自动创建topic，导致无法发送至渲染出来的topic。默认情况Loggie会一直不停的重试，从而无法发送新的日志。 +- 开启ignoreUnknownTopicOrPartition后，会直接丢弃发送的日志，避免影响其他正常包含已存在topic的日志发送。 +- 请注意和上面`ifRenderTopicFailed`的区别，`ifRenderTopicFailed`是动态渲染不出来topic或者渲染的为空值，而`ignoreUnknownTopicOrPartition`则是渲染成功，但是topic在Kafka中实际不存在。 ## balance diff --git a/mkdocs.yml b/mkdocs.yml index f3c9884..b58548f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -82,7 +82,8 @@ markdown_extensions: - name: mermaid class: mermaid-experimental format: !!python/name:pymdownx.superfences.fence_code_format - - pymdownx.tabbed + - pymdownx.tabbed: + alternate_style: true - pymdownx.tasklist: custom_checkbox: true - pymdownx.tilde