This PR also solves one more issues with the new interface, which is the
ability to record multiple observations for a single async metric using
one callback.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
<!-- Issue number if applicable -->
Drops metrics that depend on the metrics level:
- Batch processor metric
- otelarrow metrics (see open-telemetry/otel-arrow/issues/280 for
limitation).
- internal/otelarrow/netstats metrics. I did not implement
a25f058256/internal/otelarrow/netstats/netstats.go (L133-L136)
since `LevelNone` drops all metrics.
This attemps to unblock #11601 by hardcoding the metrics here since
there is a small number of them. Once we do #11754 we can move this back
to the individual components
#### Link to tracking issue
Updates #11061
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The single-shard batcher is currently created and started in the start
function. If the consume function is called before the start function,
sb.single will be nil and it will panic. I'm not sure if that can happen
in the Otel collector but it might happen in distributions that have a
different runtime (such as Alloy - Grafana's distribution).
I think that conceptually, it's better to create the shard in the
constructor and start it in the start function.
PS: I did not create an entry in the changelog because the initial
[PR](https://github.com/open-telemetry/opentelemetry-collector/pull/11314)
for this change didn't but let me know if I should add one.
Credits also to @thampiotr for this fix
#### Description
There are two implementations of the `batcher` interface with several
inconsistencies. Notably, the single-shard case would start a goroutine
before `Start()` is called, which can be confusing when the goroutine
leak checker notices it. This makes the single-shard batcher wait until
Start() to create its single shard. A confusing field name `batcher` in
this case becomes `single`, and a new field is added for use in start
named `processor` to create the new shard.
For the multi-shard batcher, `start()` does nothing, but this structure
confusingly embeds the batchProcessor. Rename the field `processor` for
consistency with the single-shard case.
#### Link to tracking issue
Part of #11308.
#### Description
Split the internal `batcher.export()` interface into three methods. This
is a refactoring that was applied in
https://github.com/open-telemetry/otel-arrow/tree/main/collector/processor/concurrentbatchprocessor
and is being back-ported as part of #11308. The reason this refactoring
is needed is that the parent context of the export() request will be
manipulated in common code (vs signal-specific code) for tracing
support.
#### Link to tracking issue
Part of #11308
#### Testing
Existing tests cover this.
---------
Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
Implement the fixes proposed in #7508. Otherwise, using a batchprocessor with Config{} passes validation but acts badly at runtime there is both a zero-duration timer and the call to sendItems never returns, constantly sending empty batches.
- Instrument batch processor with OpenTelemetry Go.
- Also fixed a small typo on the bucket definition of the batch_send_size_bytes view of OpenCensus.
The main motivation is because we want to be able to extend these Marshaler/Unmarshaler implementation with possible options.
Because of that we have 2 options: 1. add Option to each New func, OR 2. expose the structs so that later users can configure different options inside these structs, similar with `jsonpb.Marshaler`.
I implemented the version 2 since it is simpler, and less code, and also common in the industry.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Split all `pdata` related code by type and move it from `model` to the new module `pdata`.
- `model/pdata` and `model/otlp` are moved to `pdata/plog`, `pdata/pmetric` and `pdata/ptrace`.
- `model/otlpgrpc` is moved to `pdata/plogotlp`, `pdata/pmetricotlp` and `pdata/ptraceotlp`.
Now all the API in `model` except for `model/semconv` is deprecated.
* chore: adds porto and fixes vanity imports.
* chore: fixes target overriding.
* chore: fixes install of porto.
* chore: includes porto as a tool.
* chore: upgrades porto to check internals.
* chore: rebase and update vanity import.
* chore: removes unnecessary space.
* chore: rollsback vanity import in generated files.
* all: remove OtlpProtoSize in favor of Sizer interface
* add {Metrics|Traces|Logs}Sizer, fix test commenting
* move size tests to pb_test, fix metrics+logs parameter names
* uncommented batch_processor tests + adjusted them to use *Sizer
* I forgot BenchmarkTraceSizeBytes :(
* Add docs for NewProtobof*Sizer + update docs for NewProtobuf*Marshaler
* cast *Marshaler to *Sizer for now
* add casts to batch_processor_test
The new module go.opentelemetry.io/collector/model will be created later when the internal data will
also be moved.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
**Link to tracking Issue:** #2754
This change introduces a BenchmarkBatchMetricProcessor that stress tests batching logic.
Results before:
`BenchmarkBatchMetricProcessor-12 20000 80614 ns/op`
Results after the change:
`BenchmarkBatchMetricProcessor-12 20000 96184 ns/op`
* The reordering could happen if during processing of an item other items were added to the newItems channel. This is fixed by avoiding to re-add the left items to the channel.
* The logic of spliting was wrong by forcing sendBatchSize instead of sendBatchMaxSize when split was called.
* Fix logic if a very large message is received > 2x sendBatchMaxSize to call export multiple times instead of once and reset the timer.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
I added a Shutdown() test that does basic verification of the behavior of the
Shutdown() function. More verifications can be added later.
The test revealed a bug in batchprocessor Shutdown() function which would
not wait until all pending data was drained.
The newItem channel cannot be closed ever since Shutdown is not
guaranteed to be called after all Consume* calls to the same processor.
Even after Shutdown is finished, there is not hard guarantee that
nothing will ever call Consume*, so we need to just leave the channel
open until GC will clean it up.
Bogdan Drutu
Pablo Baeyens
William Dumont
Nathan Baulch
Joshua MacDonald
Alex Boten
Alex Boten
Daniel Jaglowski
Gustavo Paiva
Corbin Phelps
Nathan Vērzemnieks
Dmitrii Anoshin
José Carlos Chávez
Nathan Dias
kisieland
Patryk Matyjasek
gregoryfranklin
Tigran Najaryan
Daniel Dyla
Ben Keith