open-telemetry
/
opentelemetry-collector
mirror of https://github.com/open-telemetry/opentelemetry-collector.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
opentelemetry-collector/processor/batchprocessor
History
Nathan Dias aa450a0bf2
all: remove OtlpProtoSize in favor of Sizer interface (#3818) 
* all: remove OtlpProtoSize in favor of Sizer interface

* add {Metrics|Traces|Logs}Sizer, fix test commenting

* move size tests to pb_test, fix metrics+logs parameter names

* uncommented batch_processor tests + adjusted them to use *Sizer

* I forgot BenchmarkTraceSizeBytes :(

* Add docs for NewProtobof*Sizer + update docs for NewProtobuf*Marshaler

* cast *Marshaler to *Sizer for now

* add casts to batch_processor_test
 		2021-08-17 11:10:33 -07:00
..
testdata Add new clean nop components and use them in config tests (#2655) 2021-03-10 10:45:18 -08:00
README.md Make the batch processor limit data points rather than metrics. (#3141) 2021-06-02 10:52:24 -04:00
batch_processor.go all: remove OtlpProtoSize in favor of Sizer interface (#3818) 2021-08-17 11:10:33 -07:00
batch_processor_test.go all: remove OtlpProtoSize in favor of Sizer interface (#3818) 2021-08-17 11:10:33 -07:00
config.go Validate that batch config max size is greater than send size (#3126) 2021-05-07 12:19:34 -07:00
config_test.go Rename configtest.LoadConfigFile to configtest.LoadConfigAndValidate (#3306) 2021-05-28 11:48:26 -04:00
factory.go Standarize Settings, Params and Parameters in Processors (#3181) 2021-06-02 06:49:04 -07:00
factory_test.go Add componenttest.NewNop*CreateSettings to simplify tests (#3375) 2021-06-08 10:49:35 -04:00
metrics.go Remove obsreport.ProcessorMetricViews, use BuildProcessorCustomMetricName where needed (#3316) 2021-05-26 16:05:51 -07:00
metrics_test.go Remove level from all the MetricViews calls (#2149) 2020-11-16 23:11:27 -05:00
splitlogs.go Deprecate Resize() from pdata slice APIs (#3573) 2021-07-12 19:01:24 -07:00
splitlogs_test.go Deprecate Resize() from pdata slice APIs (#3573) 2021-07-12 19:01:24 -07:00
splitmetrics.go update initSumIntMetric and initGaugeIntMetric (#3718) 2021-07-26 16:27:07 -07:00
splitmetrics_test.go Deprecate Resize() from pdata slice APIs (#3573) 2021-07-12 19:01:24 -07:00
splittraces.go Move pdata in the new package (#3483) 2021-06-25 01:17:31 +03:00
splittraces_test.go Deprecate Resize() from pdata slice APIs (#3573) 2021-07-12 19:01:24 -07:00

README.md

Batch Processor

Supported pipeline types: metric, traces, logs

The batch processor accepts spans, metrics, or logs and places them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. This processor supports both size and time based batching.

It is highly recommended to configure the batch processor on every collector. The batch processor should be defined in the pipeline after the memory_limiter as well as any sampling processors. This is because batching should happen after any data drops such as sampling.

Please refer to config.go for the config spec.

The following configuration options can be modified:

  • send_batch_size (default = 8192): Number of spans, metric data points, or log records after which a batch will be sent regardless of the timeout.
  • timeout (default = 200ms): Time duration after which a batch will be sent regardless of size.
  • send_batch_max_size (default = 0): The upper limit of the batch size. 0 means no upper limit of the batch size. This property ensures that larger batches are split into smaller units. It must be greater or equal to send_batch_size.

Examples:

processors:
  batch:
  batch/2:
    send_batch_size: 10000
    timeout: 10s

Refer to config.yaml for detailed examples on using the processor.