#### Description
Fork of #12384 to showcase how component attributes can be injected into
scope attributes instead of log/metric/span attributes. See that PR for
more context.
To see the diff from the previous PR, filter changes starting from the
"Prototype using scope attributes" commit.
#### Link to tracking issue
Resolves#12217
Also incidentally resolves#12213 and resolves#12117
#### Testing
I updated the existing tests to check for scope attributes, and did some
manual testing with a debug exporter to check that the scope attributes
are added/removed properly.
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
#### Description
Add profiles support to the memorylimiter processor.
#### Link to tracking issue
Fixes#12453
Signed-off-by: Israel Blancas <iblancasa@gmail.com>
When servers are in a high memory usage mode, the amount of times GC is
called creates a high CPU usage which combined with a high ingest rate
limits capability to offload existing queued data.
Configuring the minimum interval even for hard limit, allows the system
to spend some CPU cycles between GCs to offload old data from
queues/batch processor. The most amount of data I've seen accumulated
are blocked in the batch processor queue (incoming channel) not in the
exporter queue.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Moves `LoggerWithout` to an internal package so that we can keep on
experimenting on it while mark component 1.0 on v0.121.0.
This does not need a changelog since the change has not been released
<!-- Issue number if applicable -->
#### Link to tracking issue
Updates #12217
Implements the logger described in
https://github.com/open-telemetry/opentelemetry-collector/issues/12217
Alternative to #12057Resolves#11814
`component/componentattribute`:
- Initializes new module
- Defines constants for component telemetry attribute keys
- Defines a `zapcore.Core` which can remove attributes from the root
logger
`service`:
- Rebases component instantiation on attribute sets
- Internal constructors for attribute sets for each component type
- Constructs loggers from `componentattribute`
`otlpreceiver`:
- Uses `componentattribute` to remove `otelcol.signal` attribute from
logger
`memorylimiter`:
- Uses `componentattribute` to remove `otelcol.signal`,
`otelcol.pipeline.id` and `otelcol.component.id` attributes from logger
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
**Description:**
Following
https://github.com/open-telemetry/opentelemetry-collector/issues/8632,
this change introduces memory limiter as an extension. This allows
us to place the component to reject incoming connections due to limited
memory, providing better protection from running out of memory.
missing feature: receiver fairness. issue where a receiver hogs all the
resource can happen.
**Link to tracking Issue:** <Issue number if applicable>
https://github.com/open-telemetry/opentelemetry-collector/issues/8632
---------
Co-authored-by: Dmitry Anoshin <anoshindx@gmail.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Main motivation is to ensure that Settings are passed (to enable obsreport usage which requires settings to be passed) as well as the context.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Signed-off-by: Dani Louca <dlouca@splunk.com>
**Description:**
This change 062e64f1d7 caused the memory limiter to "only" start the `checkMemLimits` routine for the ml instance used by the traces processor .
In other words, metrics and logs processor will NOT [drop/refuse](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/memorylimiter.go#L206) data and will pass them down to the next consumer regardless of the current memory pressure as their instance of ml->forcingDrop will not be set.
The simplest solution, is to call start for each processor (metrics, logs, traces) , but this will not be efficient as we'll be running 3 instances of `checkMemLimits`, ie: multiple GC .
But at the same we need to allow multiple instances, with different configs, example: `memory_limiter/another` and `memory_limiter`
````
extensions:
memory_ballast:
size_mib: 4
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 2s
limit_mib: 10
memory_limiter/another:
check_interval: 1s
limit_mib: 100
exporters:
logging:
logLevel: info
service:
telemetry:
logs:
level: "info"
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
metrics/default:
receivers: [otlp]
processors: [memory_limiter]
exporters: [logging]
traces:
receivers: [otlp]
processors: [memory_limiter/another]
exporters: [logging]
extensions: [memory_ballast]
````
The fix adds a global map to keep track of the different instance and add ~~sync once~~ mutex for the start and shutdown call, so only the first processor can launch the `checkMemLimits` routine and the last one to call `shutdown` to take it down.
If shutdown was called and no `checkMemLimits` has started, then we'll return an error message; unit tests were updated to handle this.
**Testing:**
Tested with above config and using splunk otel instance with valid data.
Made sure only a single `checkMemLimits` is running when there is a single config for memory-limiter and more than one when we have multiple.
I also verified that under memory pressure, when we pass the soft limit, all data types, traces, logs and metrics are getting dropped.
One we agree on this solution, I will look into adding more unit test to validate the change
* chore: adds porto and fixes vanity imports.
* chore: fixes target overriding.
* chore: fixes install of porto.
* chore: includes porto as a tool.
* chore: upgrades porto to check internals.
* chore: rebase and update vanity import.
* chore: removes unnecessary space.
* chore: rollsback vanity import in generated files.
* obsreport: wire up ProcessorCreateSettings for processors using obsreport
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Add ToProcessorCreateSettings() to obsreporttest settings
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Fix lint issue
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
This PR is to rename `config.NewID` to `config.NewComponentID` and `config.NewIDFromString` to `config.NewComponentIDFromString` for naming consistency.
**Link to tracking Issue:**
Fixes#4105
Jade Guiton
Israel Blancas
Bogdan Drutu
Pablo Baeyens
Daniel Jaglowski
Alex Boten
splunkertimn
Alex Boten
Dani Louca
José Carlos Chávez
Anthony Mirabella
James