This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
In an upcoming PR, I change Type to be an interface. This means the zero
value of Type will be nil - which will cause this test to fail.
Initializing ID instead of relying on the zero value fixes this
<!-- Issue number if applicable -->
#### Link to tracking issue
related to
https://github.com/open-telemetry/opentelemetry-collector/issues/9429
<!--Please delete paragraphs that you did not use before submitting.-->
In preparation for
https://github.com/open-telemetry/opentelemetry-collector/pull/10069
Peeling this set of changes from #9584 as a separate PR.
These changes allow a reduction of issues when working with ARM64, which
seems to fail on some of the resource locking used in those tests.
**Description:**
- Adds `component.MustNewType` to create a type. This function panics if
the type has invalid characters. Add similar functions
`component.MustNewID` and `component.MustNewIDWithName`.
- Adds `component.Type.String` to recover the string
- Use `component.MustNewType`, `component.MustNewID`,
`component.MustNewIDWithName` and `component.Type.String` everywhere in
this codebase. To do this I changed `component.Type` into an opaque
struct and checked for compile-time errors.
Some notes:
1. All components currently on core and contrib follow this rule. This
is still breaking for other components.
2. A future PR will change this into a struct, to actually validate this
(right now you can just do `component.Type("anything")` to bypass
validation). I want to do this in two steps to avoid breaking contrib
tests: we first introduce this function, and after that we change into a
struct.
**Link to tracking Issue:** Updates #9208
**Description:**
Following
https://github.com/open-telemetry/opentelemetry-collector/issues/8632,
this change introduces memory limiter as an extension. This allows
us to place the component to reject incoming connections due to limited
memory, providing better protection from running out of memory.
missing feature: receiver fairness. issue where a receiver hogs all the
resource can happen.
**Link to tracking Issue:** <Issue number if applicable>
https://github.com/open-telemetry/opentelemetry-collector/issues/8632
---------
Co-authored-by: Dmitry Anoshin <anoshindx@gmail.com>
These deprecated methods/structs have been moved to processorhelper:
- `obsreport.BuildProcessorCustomMetricName` ->
`processorhelper.BuildCustomMetricName`
- `obsreport.Processor` -> `processorhelper.ObsReport`
- `obsreport.ProcessorSettings` -> `processorhelper.ObsReportSettings`
- `obsreport.NewProcessor` -> `processorhelper.NewObsReport`
Same as the change for the exporter, no functional changes just moving
code over to the
new location.
~Follows
https://github.com/open-telemetry/opentelemetry-collector/pull/8493~
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
Contributes to https://github.com/open-telemetry/opentelemetry-collector/issues/1084
- Clarify what the memory limiter does.
- Set expectations from receivers, how they are supposed to react
when the memory limiter refuses the data.
- Add a test that demonstrates that memory limiter does not lose data
if the receiver and exporter behave according to the contract.
All receivers must adhere to this contract. See for example
an issue opened against filelog receiver:
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/20511
Note that there are no functional changes to the memory limiter.
Future work: one additional thing we can do is implement a backoff
logic in the memory limiter. When in memory limited mode the processor
can introduce pauses before it returns from the ConsumeLogs/Traces/Metrics
call. This will allow to slow down the inflow of data into the Collector
and give time for the pipeline to clear and memory usage to return to the
normal. This needs to be explored further.
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Main motivation is to ensure that Settings are passed (to enable obsreport usage which requires settings to be passed) as well as the context.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
On top of the errorlint errors, also changes `fmt.Errorf("string literal")` with `errors.New("string literal")`.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Split all `pdata` related code by type and move it from `model` to the new module `pdata`.
- `model/pdata` and `model/otlp` are moved to `pdata/plog`, `pdata/pmetric` and `pdata/ptrace`.
- `model/otlpgrpc` is moved to `pdata/plogotlp`, `pdata/pmetricotlp` and `pdata/ptraceotlp`.
Now all the API in `model` except for `model/semconv` is deprecated.
Signed-off-by: Dani Louca <dlouca@splunk.com>
**Description:**
This change 062e64f1d7 caused the memory limiter to "only" start the `checkMemLimits` routine for the ml instance used by the traces processor .
In other words, metrics and logs processor will NOT [drop/refuse](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/memorylimiter.go#L206) data and will pass them down to the next consumer regardless of the current memory pressure as their instance of ml->forcingDrop will not be set.
The simplest solution, is to call start for each processor (metrics, logs, traces) , but this will not be efficient as we'll be running 3 instances of `checkMemLimits`, ie: multiple GC .
But at the same we need to allow multiple instances, with different configs, example: `memory_limiter/another` and `memory_limiter`
````
extensions:
memory_ballast:
size_mib: 4
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 2s
limit_mib: 10
memory_limiter/another:
check_interval: 1s
limit_mib: 100
exporters:
logging:
logLevel: info
service:
telemetry:
logs:
level: "info"
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
metrics/default:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
traces:
receivers: [otlp]
processors: [memory_limiter/another]
exporters: [logging]
extensions: [memory_ballast]
````
The fix adds a global map to keep track of the different instance and add ~~sync once~~ mutex for the start and shutdown call, so only the first processor can launch the `checkMemLimits` routine and the last one to call `shutdown` to take it down.
If shutdown was called and no `checkMemLimits` has started, then we'll return an error message; unit tests were updated to handle this.
**Testing:**
Tested with above config and using splunk otel instance with valid data.
Made sure only a single `checkMemLimits` is running when there is a single config for memory-limiter and more than one when we have multiple.
I also verified that under memory pressure, when we pass the soft limit, all data types, traces, logs and metrics are getting dropped.
One we agree on this solution, I will look into adding more unit test to validate the change
* obsreport: wire up ProcessorCreateSettings for processors using obsreport
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Add ToProcessorCreateSettings() to obsreporttest settings
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Fix lint issue
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
This PR is to rename `config.NewID` to `config.NewComponentID` and `config.NewIDFromString` to `config.NewComponentIDFromString` for naming consistency.
**Link to tracking Issue:**
Fixes#4105
Matthieu MOREL
Daniel Jaglowski
Tyler Helmuth
Alex Boten
Ankit Patel
Antoine Toulme
Pablo Baeyens
splunkertimn
Alex Boten
Dmitrii Anoshin
Tigran Najaryan
Kshitij Patil
Bogdan Drutu
Gustavo Paiva
gustavo.paiva
Dani Louca
Anthony Mirabella
James