Discussed offline in relation to #12812
Introduction of this gate had some unintended side effects, such as
removing attributes from loggers.
---------
Co-authored-by: Jade Guiton <jade.guiton@datadoghq.com>
#### Description
Fork of #12384 to showcase how component attributes can be injected into
scope attributes instead of log/metric/span attributes. See that PR for
more context.
To see the diff from the previous PR, filter changes starting from the
"Prototype using scope attributes" commit.
#### Link to tracking issue
Resolves#12217
Also incidentally resolves#12213 and resolves#12117
#### Testing
I updated the existing tests to check for scope attributes, and did some
manual testing with a debug exporter to check that the scope attributes
are added/removed properly.
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
#### Description
Add profiles support to the memorylimiter processor.
#### Link to tracking issue
Fixes#12453
Signed-off-by: Israel Blancas <iblancasa@gmail.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Add `component.Type` parameter to `NewNopSettings` And deprecate
`NewNopSettingsWithType` cc @mx-psi
<!-- Issue number if applicable -->
#### Link to tracking issue
Relevant to #12305
<!--Describe what testing was performed and which tests were added.-->
#### Testing
Updated
<!--Describe the documentation added.-->
#### Documentation
Added
<!--Please delete paragraphs that you did not use before submitting.-->
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
<!-- Issue number if applicable -->
Creates `NewNopSettingsWithType` function on test modules and deprecates
the `NewNopSettings` functions on those modules. Replace all usages of
`NewNopSettings` with `NewNopSettingsWithType`.
Part of #12305 but applied to all component kinds.
#### Link to tracking issue
Updates #12221
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Moves `LoggerWithout` to an internal package so that we can keep on
experimenting on it while mark component 1.0 on v0.121.0.
This does not need a changelog since the change has not been released
<!-- Issue number if applicable -->
#### Link to tracking issue
Updates #12217
Implements the logger described in
https://github.com/open-telemetry/opentelemetry-collector/issues/12217
Alternative to #12057Resolves#11814
`component/componentattribute`:
- Initializes new module
- Defines constants for component telemetry attribute keys
- Defines a `zapcore.Core` which can remove attributes from the root
logger
`service`:
- Rebases component instantiation on attribute sets
- Internal constructors for attribute sets for each component type
- Constructs loggers from `componentattribute`
`otlpreceiver`:
- Uses `componentattribute` to remove `otelcol.signal` attribute from
logger
`memorylimiter`:
- Uses `componentattribute` to remove `otelcol.signal`,
`otelcol.pipeline.id` and `otelcol.component.id` attributes from logger
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
**Description:**
Following
https://github.com/open-telemetry/opentelemetry-collector/issues/8632,
this change introduces memory limiter as an extension. This allows
us to place the component to reject incoming connections due to limited
memory, providing better protection from running out of memory.
missing feature: receiver fairness. issue where a receiver hogs all the
resource can happen.
**Link to tracking Issue:** <Issue number if applicable>
https://github.com/open-telemetry/opentelemetry-collector/issues/8632
---------
Co-authored-by: Dmitry Anoshin <anoshindx@gmail.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
On top of the errorlint errors, also changes `fmt.Errorf("string literal")` with `errors.New("string literal")`.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Signed-off-by: Dani Louca <dlouca@splunk.com>
**Description:**
This change 062e64f1d7 caused the memory limiter to "only" start the `checkMemLimits` routine for the ml instance used by the traces processor .
In other words, metrics and logs processor will NOT [drop/refuse](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/memorylimiter.go#L206) data and will pass them down to the next consumer regardless of the current memory pressure as their instance of ml->forcingDrop will not be set.
The simplest solution, is to call start for each processor (metrics, logs, traces) , but this will not be efficient as we'll be running 3 instances of `checkMemLimits`, ie: multiple GC .
But at the same we need to allow multiple instances, with different configs, example: `memory_limiter/another` and `memory_limiter`
````
extensions:
memory_ballast:
size_mib: 4
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 2s
limit_mib: 10
memory_limiter/another:
check_interval: 1s
limit_mib: 100
exporters:
logging:
logLevel: info
service:
telemetry:
logs:
level: "info"
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
metrics/default:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
traces:
receivers: [otlp]
processors: [memory_limiter/another]
exporters: [logging]
extensions: [memory_ballast]
````
The fix adds a global map to keep track of the different instance and add ~~sync once~~ mutex for the start and shutdown call, so only the first processor can launch the `checkMemLimits` routine and the last one to call `shutdown` to take it down.
If shutdown was called and no `checkMemLimits` has started, then we'll return an error message; unit tests were updated to handle this.
**Testing:**
Tested with above config and using splunk otel instance with valid data.
Made sure only a single `checkMemLimits` is running when there is a single config for memory-limiter and more than one when we have multiple.
I also verified that under memory pressure, when we pass the soft limit, all data types, traces, logs and metrics are getting dropped.
One we agree on this solution, I will look into adding more unit test to validate the change
Daniel Jaglowski
Jade Guiton
Israel Blancas
Chao Weng
Pablo Baeyens
Bogdan Drutu
Matthieu MOREL
Alex Boten
splunkertimn
Dmitrii Anoshin
Alex Boten
Dani Louca