The main motivation is because we want to be able to extend these Marshaler/Unmarshaler implementation with possible options.
Because of that we have 2 options: 1. add Option to each New func, OR 2. expose the structs so that later users can configure different options inside these structs, similar with `jsonpb.Marshaler`.
I implemented the version 2 since it is simpler, and less code, and also common in the industry.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Split all `pdata` related code by type and move it from `model` to the new module `pdata`.
- `model/pdata` and `model/otlp` are moved to `pdata/plog`, `pdata/pmetric` and `pdata/ptrace`.
- `model/otlpgrpc` is moved to `pdata/plogotlp`, `pdata/pmetricotlp` and `pdata/ptraceotlp`.
Now all the API in `model` except for `model/semconv` is deprecated.
* Deprecation of pdata methods following OTLP v0.15.0
Adding deprecation methods for the following pdata methods:
- InstrumentationLibrary is now Scope
- InstrumentationLibrarySpans is now ScopeSpans
- InstrumentationLibraryLogs is now ScopeLogs
- InstrumentationLibraryMetrics is now ScopeMetrics
* update changelog
* add deprecated method on resource spans/metrics/logs
* add InstrumentationLibrary deprecation method
- Removed all deprecated metric messages and corresponding conversion code and tests.
- Removed deprecated_code from span status and corresponding conversions and tests.
- Renamed logs field to log_records in InstrumentationLibraryLogs message.
The batch size was 100 and the number of datapoints being generated was also 100 so at times, the batch processor would be sending the data before the timeout was reached, which is what this particular test is trying to test. To fix this, I'm increasing the batch send size to 101.
This PR is to rename `config.NewID` to `config.NewComponentID` and `config.NewIDFromString` to `config.NewComponentIDFromString` for naming consistency.
**Link to tracking Issue:**
Fixes#4105
* all: remove OtlpProtoSize in favor of Sizer interface
* add {Metrics|Traces|Logs}Sizer, fix test commenting
* move size tests to pb_test, fix metrics+logs parameter names
* uncommented batch_processor tests + adjusted them to use *Sizer
* I forgot BenchmarkTraceSizeBytes :(
* Add docs for NewProtobof*Sizer + update docs for NewProtobuf*Marshaler
* cast *Marshaler to *Sizer for now
* add casts to batch_processor_test
The new module go.opentelemetry.io/collector/model will be created later when the internal data will
also be moved.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
**Link to tracking Issue:** #2754
This change introduces a BenchmarkBatchMetricProcessor that stress tests batching logic.
Results before:
`BenchmarkBatchMetricProcessor-12 20000 80614 ns/op`
Results after the change:
`BenchmarkBatchMetricProcessor-12 20000 96184 ns/op`
* The reordering could happen if during processing of an item other items were added to the newItems channel. This is fixed by avoiding to re-add the left items to the channel.
* The logic of spliting was wrong by forcing sendBatchSize instead of sendBatchMaxSize when split was called.
* Fix logic if a very large message is received > 2x sendBatchMaxSize to call export multiple times instead of once and reset the timer.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Error checking on unit tests
* Add back license text
* Retrigger CI
* Wrap defer calls with `func`
Arguments are evaluated immediately on defer calls
* Update config/configauth/oidc_server_test.go
* Use t.Cleanup
* Change ioutil.ReadAll by resp.Body.Close()
* Retrigger CI (flaky tests)
I added a Shutdown() test that does basic verification of the behavior of the
Shutdown() function. More verifications can be added later.
The test revealed a bug in batchprocessor Shutdown() function which would
not wait until all pending data was drained.
Currently for slices of pointers, pdata exposes elements that are pointers to the pointer in the slice (double pointers in the internal implementation). Because of this users have to deal with possible nil values, have to initialize elements (in some cases) etc, but in reality the elements in the slice cannot be nil, so this just adds extra unnecessary complexity. This is possible because:
* Gogo proto (and protobuf) will not unmarshal any nil element in a slice;
* Our APIs to add elements/remove elements from the slice will guarantee that we never have a nil element in the slice between [0, len-1];
This is an important change because will allow us to change the internal representation (use slice of pointers or non-pointers) without affecting the public API.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
This is a preparation for enabling per component telemetry level:
* Views are global and they must always be installed; they are no-op if nobody records data;
* Views are global and they must always contain all the labels; In case of measurements being recorded without one of the labels (for perfomance reasons)
the extra label defined does not have impact, because all the timeseries will have that label as "null" (missing).
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Remove legacy metrics, they were marked as legacy for ~12 months
Fixes https://github.com/open-telemetry/opentelemetry-collector/issues/1082
This PR removes:
- Legacy receiver/exporter metrics:
- otelcol/receiver/received_spans
- otelcol/receiver/dropped_spans
- otelcol/receiver/received_timeseries
- otelcol/receiver/dropped_timeseries
- otelcol/exporter/received_spans
- otelcol/exporter/dropped_spans
- otelcol/exporter/received_timeseries
- otelcol/exporter/dropped_timeseries
- otelcol/exporter/received_logs
- otelcol/exporter/dropped_logs
- For processors remove the legacy metrics (new metrics have the same data, different names):
- e.g. "spans_dropped" -> "processor/spans_dropped"
- e.g. "batch_send_size_bytes" -> "processor/batch/batch_send_size_bytes"
All the new metrics were enabled when using the --new-metrics flag. The PR also removes two flags:
- "--new-metrics"
- "--legacy-metrics"
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Update changelog
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Fix comments from review
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Need to re-generate the new internal metrics, but it is too much code to change,
so will move this to dataold (was internal anyway, so no public breaking change).
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
The order in which Consume* and Shutdown get handled for batch processor is inconsistent when called back to back. This results in the number of items getting passed through to be inconsistent. To unblock tests and maintain coverage, the tests only check if some data got through and the batching size is 1.
Issue #1070 has been opened to properly fix the test and ensure the total number of items sent is accounted for after shutdown has been invoked.
**Link to tracking Issue:** Fixes issue #1066 and #1069
**Testing:** Run each of the test 100 times and no test failures.
Bogdan Drutu
Dmitrii Anoshin
Corbin Phelps
Nathan Vērzemnieks
Alex Boten
Tigran Najaryan
Pablo Baeyens
James
Nathan Dias
kisieland
Patryk Matyjasek
Ben Keith
Jay Camp
Pavol Loffay
Constance Caramanolis