- Instrument batch processor with OpenTelemetry Go.
- Also fixed a small typo on the bucket definition of the batch_send_size_bytes view of OpenCensus.
The main motivation is because we want to be able to extend these Marshaler/Unmarshaler implementation with possible options.
Because of that we have 2 options: 1. add Option to each New func, OR 2. expose the structs so that later users can configure different options inside these structs, similar with `jsonpb.Marshaler`.
I implemented the version 2 since it is simpler, and less code, and also common in the industry.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Split all `pdata` related code by type and move it from `model` to the new module `pdata`.
- `model/pdata` and `model/otlp` are moved to `pdata/plog`, `pdata/pmetric` and `pdata/ptrace`.
- `model/otlpgrpc` is moved to `pdata/plogotlp`, `pdata/pmetricotlp` and `pdata/ptraceotlp`.
Now all the API in `model` except for `model/semconv` is deprecated.
* chore: adds porto and fixes vanity imports.
* chore: fixes target overriding.
* chore: fixes install of porto.
* chore: includes porto as a tool.
* chore: upgrades porto to check internals.
* chore: rebase and update vanity import.
* chore: removes unnecessary space.
* chore: rollsback vanity import in generated files.
* all: remove OtlpProtoSize in favor of Sizer interface
* add {Metrics|Traces|Logs}Sizer, fix test commenting
* move size tests to pb_test, fix metrics+logs parameter names
* uncommented batch_processor tests + adjusted them to use *Sizer
* I forgot BenchmarkTraceSizeBytes :(
* Add docs for NewProtobof*Sizer + update docs for NewProtobuf*Marshaler
* cast *Marshaler to *Sizer for now
* add casts to batch_processor_test
The new module go.opentelemetry.io/collector/model will be created later when the internal data will
also be moved.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
**Link to tracking Issue:** #2754
This change introduces a BenchmarkBatchMetricProcessor that stress tests batching logic.
Results before:
`BenchmarkBatchMetricProcessor-12 20000 80614 ns/op`
Results after the change:
`BenchmarkBatchMetricProcessor-12 20000 96184 ns/op`
* The reordering could happen if during processing of an item other items were added to the newItems channel. This is fixed by avoiding to re-add the left items to the channel.
* The logic of spliting was wrong by forcing sendBatchSize instead of sendBatchMaxSize when split was called.
* Fix logic if a very large message is received > 2x sendBatchMaxSize to call export multiple times instead of once and reset the timer.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
I added a Shutdown() test that does basic verification of the behavior of the
Shutdown() function. More verifications can be added later.
The test revealed a bug in batchprocessor Shutdown() function which would
not wait until all pending data was drained.
The newItem channel cannot be closed ever since Shutdown is not
guaranteed to be called after all Consume* calls to the same processor.
Even after Shutdown is finished, there is not hard guarantee that
nothing will ever call Consume*, so we need to just leave the channel
open until GC will clean it up.
Need to re-generate the new internal metrics, but it is too much code to change,
so will move this to dataold (was internal anyway, so no public breaking change).
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
New internal data structure allows to batch together traces from different resources. This simplify batching a lot. This commit removes most of the functionality for handling many batches per node and uses only one batch with one ticker.
Bogdan Drutu
Gustavo Paiva
Corbin Phelps
Nathan Vērzemnieks
Dmitrii Anoshin
José Carlos Chávez
Nathan Dias
kisieland
Patryk Matyjasek
gregoryfranklin
Tigran Najaryan
Daniel Dyla
Ben Keith
Jay Camp
Pavol Loffay
Juraci Paixão Kröhling
Constance Caramanolis
Bruno Garcia