Add `data_type` attribute to the internal otelcol_exporter_queue_size
metric to report the type of data being processed.
All other metrics have the data type reported as part of their names. We
could've done the same for queue metrics, but that would introduce a
significant breaking change. We want to avoid that until we have all the
metrics standardized with OpenTelemetry semantic conventions.
Fixes
https://github.com/open-telemetry/opentelemetry-collector/issues/9943
This ensures the consistency for folks emitting metrics w/ OTLP until
the OTEP to specify pipeline telemetry is completed.
Waiting on
https://github.com/open-telemetry/opentelemetry-collector/pull/9775
before moving this forward
Fixes#9315
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
#### Link to tracking issue
Resolves#10353
#### Testing
Added equivalent testing to other processor metrics (accepted, refused,
dropped).
#### Documentation
Metric documentation is autogenerated.
#### Open Question
My initial implementation includes a breaking change to
`componenttest.TestTelemetry` which is public facing API. If we want to
avoid an immediate breaking change in this test package, I would propose
the following, which I can submit in a prerequisite PR:
1. Deprecate all `TestTelemetry.Check*` methods.
2. Replace with more granular `TestTelemetry.CheckOneSpecificMetric`
methods.
**Description:**
Remove `host.ReportFatalError`. It has been deprecated since 0.87.0.
**Link to tracking Issue:**
#6344
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Co-authored-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
**Description:**
- Adds `component.MustNewType` to create a type. This function panics if
the type has invalid characters. Add similar functions
`component.MustNewID` and `component.MustNewIDWithName`.
- Adds `component.Type.String` to recover the string
- Use `component.MustNewType`, `component.MustNewID`,
`component.MustNewIDWithName` and `component.Type.String` everywhere in
this codebase. To do this I changed `component.Type` into an opaque
struct and checked for compile-time errors.
Some notes:
1. All components currently on core and contrib follow this rule. This
is still breaking for other components.
2. A future PR will change this into a struct, to actually validate this
(right now you can just do `component.Type("anything")` to bypass
validation). I want to do this in two steps to avoid breaking contrib
tests: we first introduce this function, and after that we change into a
struct.
**Link to tracking Issue:** Updates #9208
**Description:**
It was suggested that we add the [goleak
package](https://pkg.go.dev/go.uber.org/goleak) to every test in core.
This change adds `goleak` to every package that is succeeds with goleak.
There a number that are not successful, the full list of which I've
posted in the bug.
I generated these files using a shell script to copy a template
`main_test.go` file into each package, then modified the package name.
I'm sure there was a better way to automate this, but it worked well
enough at this point. Here's the script:
```
cp ./main_test.go $1
PACKAGE_NAME=$(basename $1)
sed -i '' -e "s|package component|package $PACKAGE_NAME|g"
$1/main_test.go
pushd . && cd $1 && go mod tidy
go test -v .
```
Usage example:
```
$ ./add_leak_test reciever/scrapererror
```
**Link to tracking Issue:**
#9165
**Testing:**
All added tests are passing, but there are a number failing. Note that there's no sign of `goleak` running until it fails.
This PR introduces component status reporting. There have been several
attempts to introduce this functionality previously, with the most
recent being: #6560.
This PR was orignally based off of #6560, but has evolved based on the
feedback received and some additional enhancements to improve the ease
of use of the `ReportComponentStatus` API.
In earlier discussions (see
https://github.com/open-telemetry/opentelemetry-collector/pull/8169#issuecomment-1668367246)
we decided to model status as a finite state machine with the following
statuses: `Starting`, `OK`, `RecoverableError`, `PermanentError`,
`FatalError`. `Stopping`, and `Stopped`. A benefit of this design is
that `StatusWatcher`s will be notified on changes in status rather than
on potentially repetitive reports of the same status.
With the additional statuses and modeling them using a finite state
machine, there are more statuses to report. Rather than having each
component be responsible for reporting all of the statuses, I automated
status reporting where possible. A component's status will automatically
be set to `Starting` at startup. If the components `Start` returns an
error, the status will automatically be set to `PermanentError`. A
component is expected to report `StatusOK` when it has successfully
started (if it has successfully started) and from there can report
changes in status as it runs. It will likely be a common scenario for
components to transition between `StatusOK` and `StatusRecoverableError`
during their lifetime. In extenuating circumstances they can transition
into terminal states of `PermanentError` and `FatalError` (where a fatal
error initiates collector shutdown). Additionally, during component
Shutdown statuses are automatically reported where possible. A
component's status is set to `Stopping` when Shutdown is initially
called, if Shutdown returns an error, the status will be set to
`PermanentError` if it does not return an error, the status is set to
`Stopped`.
In #6560 ReportComponentStatus was implemented on the `Host` interface.
I found that few components use the Host interface, and none of them
save a handle to it (to be used outside of the `start` method). I found
that many components keep a handle to the `TelemetrySettings` that they
are initialized with, and this seemed like a more natural, convenient
place for the `ReportComponentStatus` API. I'm ultimately flexible on
where this method resides, but feel that `TelemetrySettings` a more user
friendly place for it.
Regardless of where the `ReportComponentStatus` method resides (Host or
TelemetrySettings), there is a difference in the method signature for
the API based on whether it is used from the service or from a
component. As the service is not bound to a specific component, it needs
to take the `instanceID` of a component as a parameter, whereas the
component version of the method already knows the `instanceID`. In #6560
this led to having both `component.Host` and `servicehost.Host` versions
of the Host interface to be used at the component or service levels. In
this version, we have the same for TelemetrySettings. There is a
`component.TelemetrySettings` and a `servicetelemetry.Settings` with the
only difference being the method signature of `ReportComponentStatus`.
Lastly, this PR sets up the machinery for report component status, and
allows extensions to be `StatusWatcher`s, but it does not introduce any
`StatusWatcher`s. We expect the OpAMP extension to be a `StatusWatcher`
and use data from this system as part of its AgentHealth message (the
message is currently being extended to accommodate more component level
details). We also expect there to be a non-OpAMP `StatusWatcher`
implementation, likely via the HealthCheck extension (or something
similiar).
**Link to tracking Issue:** #7682
cc: @tigrannajaryan @djaglowski @evan-bradley
---------
Co-authored-by: Tigran Najaryan <tnajaryan@splunk.com>
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
Co-authored-by: Alex Boten <aboten@lightstep.com>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
On top of the errorlint errors, also changes `fmt.Errorf("string literal")` with `errors.New("string literal")`.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Split all `pdata` related code by type and move it from `model` to the new module `pdata`.
- `model/pdata` and `model/otlp` are moved to `pdata/plog`, `pdata/pmetric` and `pdata/ptrace`.
- `model/otlpgrpc` is moved to `pdata/plogotlp`, `pdata/pmetricotlp` and `pdata/ptraceotlp`.
Now all the API in `model` except for `model/semconv` is deprecated.
Bump go.opentelemetry.io/contrib/zpages from 0.29.0 to 0.30.0
Bump google.golang.org/protobuf from 1.27.1 to 1.28.0 in /model
Bump go.opentelemetry.io/otel/metric from 0.27.0 to 0.28.0
Bump go.opentelemetry.io/otel/sdk from 1.5.0 to 1.6.1
Bump go.opentelemetry.io/otel/sdk/metric from 0.27.0 to 0.28.0
Bump go.opentelemetry.io/otel/exporters/prometheus from 0.27.0 to 0.28.0
Bump github.com/ory/go-acc from 0.2.7 to 0.2.8 in /internal/tools
* consolidate flags for configuring telemetry
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Enable configuring metrics via service config
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Make components take MetricsLevel from TelemetrySettings
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Fix lint errors
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Remove configuration for metrics prefix and adding instance ID
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Make entire Collector available to telemetry initialization, use it to set metrics prefix to buildInfo.Command
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* fix metrics prefix tests
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Fix lint errors
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* config/telemetry: parseLevel() no longer needs to be exported
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* config/telemetry: remove intanceID and prefix flags
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Apply PR feedback
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* address PR feedback
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Avoid linter complaining about use of deprecated functions
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Update CHANGELOG
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* Add internalinterface package
This adds a general InternalInterface to make an interface impossible
to implement.
* Use internalinterface.InternalInterface for Processor as a test
* Add Changelog note
* need to use -> must use
* Add small unit test
Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>
* chore: adds porto and fixes vanity imports.
* chore: fixes target overriding.
* chore: fixes install of porto.
* chore: includes porto as a tool.
* chore: upgrades porto to check internals.
* chore: rebase and update vanity import.
* chore: removes unnecessary space.
* chore: rollsback vanity import in generated files.
This PR is to rename `config.NewID` to `config.NewComponentID` and `config.NewIDFromString` to `config.NewComponentIDFromString` for naming consistency.
**Link to tracking Issue:**
Fixes#4105
The new module go.opentelemetry.io/collector/model will be created later when the internal data will
also be moved.
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Check testbed errors
* Check some more errors
* Empty commit to retrigger CI
* Address comments on hostmetricsreceiver
* Rewrite some errchecking code
* Inline ifs
* Inline ifs and change log.Fatalf + Error() calls by log.Fatal
* Made changes to componenttest package
* Changes to nop_exporter_test
* Reverted the changes in exporter and nop_exporter
* Updated comments for the Factory
* reverted usage of singletons
* Allow adding new signals in ProcessorFactory by forcing everyone to embed BaseProcessorFactory
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Update component/processor.go
Co-authored-by: Jay Camp <jay.r.camp@gmail.com>
Co-authored-by: Jay Camp <jay.r.camp@gmail.com>
I added a Shutdown() test that does basic verification of the behavior of the
Shutdown() function. More verifications can be added later.
The test revealed a bug in batchprocessor Shutdown() function which would
not wait until all pending data was drained.
* Merge componentdocs into componenttest
componentdocs is used for testing reasons, merging the docs validation code into the test package helps other developers to discover the existence of it.
Rename VerifyComponentDocumentation for CheckDocs to help verbosity and have consistency with the other checker functions in the module.
Signed-off-by: JBD <jbd@google.com>
* Fix lint error
We previously used string in the codebase to represent component type
(as it is known to factories and referred to in the config).
Now we have configmodels.Type alias that provides stronger compile-time
type checks.
This is a breaking change for components but fixing the components is easy.
Resolves: https://github.com/open-telemetry/opentelemetry-collector/issues/762
Dmitrii Anoshin
Alex Boten
Daniel Jaglowski
dockercui
Tyler Helmuth
Alex Boten
Antoine Toulme
Pablo Baeyens
Curtis Robert
Matthew Wear
Jacob Aronoff
Bogdan Drutu
Haleygo
Anthony Mirabella
José Carlos Chávez
James
Carlos Alberto Cortez
Patryk Matyjasek
Dhruv Vora
Pablo Baeyens
Min Xia
Min Xia
Tigran Najaryan
James Bebbington
Jay Camp
Ben Keith
JBD
Tomasz Piekarczyk
Akash Suresh
Bruno Garcia