#### Description
Fork of #12384 to showcase how component attributes can be injected into
scope attributes instead of log/metric/span attributes. See that PR for
more context.
To see the diff from the previous PR, filter changes starting from the
"Prototype using scope attributes" commit.
#### Link to tracking issue
Resolves#12217
Also incidentally resolves#12213 and resolves#12117
#### Testing
I updated the existing tests to check for scope attributes, and did some
manual testing with a debug exporter to check that the scope attributes
are added/removed properly.
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Moves `LoggerWithout` to an internal package so that we can keep on
experimenting on it while mark component 1.0 on v0.121.0.
This does not need a changelog since the change has not been released
<!-- Issue number if applicable -->
#### Link to tracking issue
Updates #12217
Implements the logger described in
https://github.com/open-telemetry/opentelemetry-collector/issues/12217
Alternative to #12057Resolves#11814
`component/componentattribute`:
- Initializes new module
- Defines constants for component telemetry attribute keys
- Defines a `zapcore.Core` which can remove attributes from the root
logger
`service`:
- Rebases component instantiation on attribute sets
- Internal constructors for attribute sets for each component type
- Constructs loggers from `componentattribute`
`otlpreceiver`:
- Uses `componentattribute` to remove `otelcol.signal` attribute from
logger
`memorylimiter`:
- Uses `componentattribute` to remove `otelcol.signal`,
`otelcol.pipeline.id` and `otelcol.component.id` attributes from logger
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
<!-- Issue number if applicable -->
Moves `extension.Settings.ModuleInfo` to a `service.Host.GetModuleInfo`
method.
This field probably fits better on `extension.Settings` or even in
`component.BuildInfo`, but since its contents are a bit in flux per
discussions like #12283, I would like to move it here to not stabilize
it alongside the rest of `extension`.
The field is not used in any public code on Github, so I think it won't
have a huge impact to remove it in one go, happy to do this in two steps
if preferred.
One relevant difference is that this information is no longer available
at extension build time, but rather after the `Start` method is called.
Another relevant difference is that this is now available for all
component kinds, not just extensions. This could be worked around (we
could pass a different host depending on the component kind) but I don't
see the use right now.
#### Link to tracking issue
Updates #12283
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
<!-- Issue number if applicable -->
Moves extension optional interfaces to a new module,
`extensioncapabilities`.
The intent is to be able to evolve these interfaces independently from
the core interfaces on core.
These are used by at most one component/project, so I think they are not
mature yet for 1.0
---------
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
This moves the connector builder out of the `connector` package, and
into `service/internal/builders`.
There's no real reason for this struct to be public (folks shouldn't
call it), and making it private will allow us to add profiling support
to it.
<!-- Issue number if applicable -->
#### Link to tracking issue
https://github.com/open-telemetry/opentelemetry-collector/pull/10375#pullrequestreview-2144929463
While this is not technically required for the profiles work (there is
no notion of signals in extensions), this PR is here to keep things
consistent.
#### Description
Paired with @djaglowski on adding a new struct, `ModuleInfo`, to
`extension.Settings`. The goal is to make the collector's component go
modules available to extensions that may want to use that information
(in particular the OpAMP extension).
We didn't want to modify the `extension` package but adding this
information to the Settings struct seems to be the most painless way to
make this information available. No function signatures need to be
updated and existing extensions can just ignore the new field on the
Settings struct unless they want the information.
#### Link to tracking issue
Fixes#10876
#### Testing
Updated existing tests.
---------
Co-authored-by: Dan Jaglowski <jaglows3@gmail.com>
#### Description
This PR makes component.InstanceID immutable. Previously it was a struct
with all fields exported. Technically this is a breaking change, but the
only thing using the InstanceID is the in-progress
healthcheckv2extension.
<!-- Issue number if applicable -->
#### Link to tracking issue
Fixes#10494
<!--Describe what testing was performed and which tests were added.-->
#### Testing
units
<!--Describe the documentation added.-->
#### Documentation
code comments
<!--Please delete paragraphs that you did not use before submitting.-->
---------
Co-authored-by: Antoine Toulme <antoine@toulme.name>
Co-authored-by: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com>
Co-authored-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
#### Description
This PR removes `ReportStatus` from `component.TelemetrySettings` and
instead expects components to check if their `component.Host` implements
a new `componentstatus.Reporter` interface.
<!-- Issue number if applicable -->
#### Link to tracking issue
Related to
https://github.com/open-telemetry/opentelemetry-collector/pull/10725
Related to
https://github.com/open-telemetry/opentelemetry-collector/pull/10413
<!--Describe what testing was performed and which tests were added.-->
#### Testing
unit tests and a sharedinstance e2e test.
The contrib tests will fail because this is a breaking change. If we
merge this I and @mwear can commit to updating contrib before the next
release.
---------
Co-authored-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
#### Description
Reorganizes service to not require `servicetelemetry.TelemetrySettings`
and instead depend directly on `component.TelemetrySettings`
Whether or not we move forward with
https://github.com/open-telemetry/opentelemetry-collector/pull/10725 I
think this is a useful change for service.
#### Testing
Unit tests
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Improved logging of component start errors, so the log lines appear
where they chronologically happen, rather than at the end. They also use
the component logger, which makes them look nicer.
Output before change:
```text
2024-06-19T14:15:30.822+0200 info service@v0.103.0/service.go:115 Setting up own telemetry...
2024-06-19T14:15:30.822+0200 info service@v0.103.0/telemetry.go:96 Serving metrics {"address": ":8888", "level": "Normal"}
2024-06-19T14:15:30.823+0200 info service@v0.103.0/service.go:182 Starting otelcorecol... {"Version": "0.103.0-dev", "NumCPU": 32}
2024-06-19T14:15:30.823+0200 info extensions/extensions.go:35 Starting extensions...
2024-06-19T14:15:30.823+0200 info service@v0.103.0/service.go:245 Starting shutdown...
2024-06-19T14:15:30.823+0200 info extensions/extensions.go:61 Stopping extensions...
2024-06-19T14:15:30.823+0200 info service@v0.103.0/service.go:259 Shutdown complete.
Error: cannot start pipelines: failed to resolve authenticator "oauth2client": authenticator not found
2024/06/19 14:15:30 collector server run finished with error: cannot start pipelines: failed to resolve authenticator "oauth2client": authenticator not found
```
Output after change:
```
2024-06-19T14:25:18.324+0200 info service@v0.103.0/service.go:115 Setting up own telemetry...
2024-06-19T14:25:18.325+0200 info service@v0.103.0/telemetry.go:96 Serving metrics {"address": ":8888", "level": "Normal"}
2024-06-19T14:25:18.326+0200 info service@v0.103.0/service.go:182 Starting otelcorecol... {"Version": "0.103.0-dev", "NumCPU": 32}
2024-06-19T14:25:18.326+0200 info extensions/extensions.go:35 Starting extensions...
2024-06-19T14:25:18.326+0200 error graph/graph.go:428 Failed to start component {"error": "failed to resolve authenticator \"oauth2client\": authenticator not found", "type": "Exporter", "id": "otlphttp"}
2024-06-19T14:25:18.326+0200 info service@v0.103.0/service.go:245 Starting shutdown...
2024-06-19T14:25:18.326+0200 info extensions/extensions.go:61 Stopping extensions...
2024-06-19T14:25:18.326+0200 info service@v0.103.0/service.go:259 Shutdown complete.
Error: cannot start pipelines: failed to resolve authenticator "oauth2client": authenticator not found
2024/06/19 14:25:18 collector server run finished with error: cannot start pipelines: failed to resolve authenticator "oauth2client": authenticator not found
```
We've added the following line:
```
2024-06-19T14:25:18.326+0200 error graph/graph.go:428 Failed to start component {"error": "failed to resolve authenticator \"oauth2client\": authenticator not found", "type": "Exporter", "id": "otlphttp"}
```
For extensions, the new line looks like so:
```
2024-06-25T16:59:18.513+0200 error extensions/extensions.go:54 Failed to start extension {"kind": "extension", "name": "memory_limiter", "error": "failed to get memory limit"}
```
<!-- Issue number if applicable -->
#### Link to tracking issue
Fixes#7078
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Remove `hostWrapper`. This internal class is only used for extensions to
wrap a specific logger, but since we have deprecated `ReportFatalError`,
we should no longer expect this function to be called anyway.
This is part of the continued component status reporting effort.
Currently we have automated status reporting for the following component
lifecycle events: `Starting`, `Stopping`, `Stopped` as well as
definitive errors that occur in the starting or stopping process (e.g.
as determined by an error return value). This leaves the responsibility
to the component to report runtime status after start and before stop.
We'd like to be able to extend the automatic status reporting to report
`StatusOK` if `Start` completes without an error. One complication with
this approach is that some components spawn async work (via goroutines)
that, depending on the Go scheduler, can report status before `Start`
returns. As such, we cannot assume a nil return value from `Start` means
the component has started properly. The solution is to detect if the
component has already reported status when start returns, if it has, we
will use the component-reported status and will not automatically report
status. If it hasn't, and `Start` returns without an error, we can
report `StatusOK`. Any subsequent reports from the component (async or
otherwise) will transition the component status accordingly.
The tl;dr is that we cannot control the execution of async code, that's
up to the Go scheduler, but we can handle the race, report the status
based on the execution, and not clobber status reported from within the
component during the startup process. That said, for components with
async starts, you may see a `StatusOK` before the component-reported
status, or just the component-reported status depending on the actual
execution of the code. In both cases, the end status will be same.
The work in this PR will allow us to simplify #8684 and #8788 and
ultimately choose which direction we want to go for runtime status
reporting.
**Link to tracking Issue:** #7682
**Testing:** units / manual
---------
Co-authored-by: Alex Boten <aboten@lightstep.com>
**Description**:
Enforce order of start and shutdown of extensions according to their
internally declared dependencies
**Link to tracking Issue**:
Resolves#8732
**Motivation**:
This is an alternative approach to #8733 which uses declaration order in
the config to start extensions. That approach (a) enforces order when
it's not always necessary to enforce, and (b) exposes unnecessary
complexity to the user by making them responsible for the order.
This PR instead derives the desired order of extensions based on the
dependencies they declare by implementing a `DependentExtension`
interface. That means that extensions that must depend on others can
expose this interface and be guaranteed to start after their
dependencies, while other extensions can be started in arbitrary order
(same as happens today because of iterating over a map).
The extensions that have dependencies have two options to expose them:
1. if the dependency is always static (e.g. `jaeger_query` extension
depending on `jaeger_storage` as in the OP), the extension can express
this statically as well, by returning a predefined ID of the dependent
extension
2. in cases where dependencies are dynamic, the extension can read the
names of the dependencies from its configuration.
The 2nd scenario is illustrated by the following configuration. Here
each complex extension knows that it needs dependencies that implement
`storage` and `encoding` interfaces (both existing APIs in collector &
contrib), but does not know statically which instances of those, the
actual names are supplied by the user in the configuration.
```yaml
extensions:
complex_extension_1:
storage: filestorage
encoding: otlpencoding
complex_extension_2:
storage: dbstorage
encoding: jsonencoding
filestorage:
...
dbstorage:
...
otlpencoding:
jsonencoding:
```
**Changes**:
* Introduce `DependentExtension` optional interface
* Change `Extensions` constructor to derive the required order using a
directed graph (similar to pipelines)
* Inherited from #8733 - use new ordered list of IDs to
start/stop/notify extensions in the desired order (previously a map was
used to iterate over, which resulted in random order).
* Tests
**Testing**:
Unit tests
---------
Signed-off-by: Yuri Shkuro <github@ysh.us>
Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com>
This PR introduces component status reporting. There have been several
attempts to introduce this functionality previously, with the most
recent being: #6560.
This PR was orignally based off of #6560, but has evolved based on the
feedback received and some additional enhancements to improve the ease
of use of the `ReportComponentStatus` API.
In earlier discussions (see
https://github.com/open-telemetry/opentelemetry-collector/pull/8169#issuecomment-1668367246)
we decided to model status as a finite state machine with the following
statuses: `Starting`, `OK`, `RecoverableError`, `PermanentError`,
`FatalError`. `Stopping`, and `Stopped`. A benefit of this design is
that `StatusWatcher`s will be notified on changes in status rather than
on potentially repetitive reports of the same status.
With the additional statuses and modeling them using a finite state
machine, there are more statuses to report. Rather than having each
component be responsible for reporting all of the statuses, I automated
status reporting where possible. A component's status will automatically
be set to `Starting` at startup. If the components `Start` returns an
error, the status will automatically be set to `PermanentError`. A
component is expected to report `StatusOK` when it has successfully
started (if it has successfully started) and from there can report
changes in status as it runs. It will likely be a common scenario for
components to transition between `StatusOK` and `StatusRecoverableError`
during their lifetime. In extenuating circumstances they can transition
into terminal states of `PermanentError` and `FatalError` (where a fatal
error initiates collector shutdown). Additionally, during component
Shutdown statuses are automatically reported where possible. A
component's status is set to `Stopping` when Shutdown is initially
called, if Shutdown returns an error, the status will be set to
`PermanentError` if it does not return an error, the status is set to
`Stopped`.
In #6560 ReportComponentStatus was implemented on the `Host` interface.
I found that few components use the Host interface, and none of them
save a handle to it (to be used outside of the `start` method). I found
that many components keep a handle to the `TelemetrySettings` that they
are initialized with, and this seemed like a more natural, convenient
place for the `ReportComponentStatus` API. I'm ultimately flexible on
where this method resides, but feel that `TelemetrySettings` a more user
friendly place for it.
Regardless of where the `ReportComponentStatus` method resides (Host or
TelemetrySettings), there is a difference in the method signature for
the API based on whether it is used from the service or from a
component. As the service is not bound to a specific component, it needs
to take the `instanceID` of a component as a parameter, whereas the
component version of the method already knows the `instanceID`. In #6560
this led to having both `component.Host` and `servicehost.Host` versions
of the Host interface to be used at the component or service levels. In
this version, we have the same for TelemetrySettings. There is a
`component.TelemetrySettings` and a `servicetelemetry.Settings` with the
only difference being the method signature of `ReportComponentStatus`.
Lastly, this PR sets up the machinery for report component status, and
allows extensions to be `StatusWatcher`s, but it does not introduce any
`StatusWatcher`s. We expect the OpAMP extension to be a `StatusWatcher`
and use data from this system as part of its AgentHealth message (the
message is currently being extended to accommodate more component level
details). We also expect there to be a non-OpAMP `StatusWatcher`
implementation, likely via the HealthCheck extension (or something
similiar).
**Link to tracking Issue:** #7682
cc: @tigrannajaryan @djaglowski @evan-bradley
---------
Co-authored-by: Tigran Najaryan <tnajaryan@splunk.com>
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
Co-authored-by: Alex Boten <aboten@lightstep.com>
This was deprecated many versions ago, should have been removed but i
suspect was missed due to typo.
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
**Description:**
This adds an optional `ConfigWatcher` interface that extensions can
implement if they want to be notified of the effective configuration
that is used by the Collector.
I don't feel very strongly about any of the decisions I made in this PR,
so I am open to input if we would like to take a different approach
anywhere. I will leave some comments to explain the decisions I made.
**Link to tracking Issue:**
Closes
https://github.com/open-telemetry/opentelemetry-collector/issues/6596
**Testing:**
I've made minimal unit test changes, but I expect to write more tests
once there is consensus on the direction for implementing this
functionality. I have done some manual testing to show that an extension
can get a YAML representation of the effective config using two YAML
input files.
---------
Co-authored-by: Evan Bradley <evan-bradley@users.noreply.github.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Jade Guiton
Pablo Baeyens
Daniel Jaglowski
Tyler Helmuth
Bogdan Drutu
Damien Mathieu
Dakota Paasman
Matthew Wear
Mikołaj Świątek
Alex Boten
Antoine Toulme
Yuri Shkuro
Alex Boten
Evan Bradley