<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
This is part of #11131, splitting the exporterhelper module with a new
`internal` submodule that holds everything shared by the signals.
This is so a new `exporterhelperprofiles` module can make use of the
shared structs.
#### Description
This PR renames `TimeoutSettings` and `QueueSettings` (as well as the
corresponding `NewDefault` functions) to `TimeoutConfig` and
`QueueConfig`, for naming consistency reasons.
The previous struct/function names are kept as deprecated aliases of the
new ones for now.
#### Link to tracking issue
Updates #6767 (edit: reworded by @mx-psi)
#### Testing
No behavior changes were made, so no additional testing should be
necessary.
The references to the aforementioned structs/functions in existing tests
have been renamed as well. This means the deprecated function aliases
are not tested, lowering the coverage a bit.
#### Documentation
The new functions are documented identically to the previous ones.
Add `data_type` attribute to the internal otelcol_exporter_queue_size
metric to report the type of data being processed.
All other metrics have the data type reported as part of their names. We
could've done the same for queue metrics, but that would introduce a
significant breaking change. We want to avoid that until we have all the
metrics standardized with OpenTelemetry semantic conventions.
Fixes
https://github.com/open-telemetry/opentelemetry-collector/issues/9943
This was introduced in go 1.21 and removes the need for a custom
implementation of the context.
Fixes#9049
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
This uses mdatagen to generate the queue metrics. This will allow users
to see the metric in the documentation for exporter helper
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
~Follows
https://github.com/open-telemetry/opentelemetry-collector/pull/10333~
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
This change introduces new experimental batching functionality to the
exporter helper. The batch sender is fully concurrent and synchronous.
It's set after the queue sender, which, if enabled, introduces the
asynchronous behavior and ensures no data loss with the permanent queue.
Follow-up TODO list:
- Add pre-built merge funcs for pdata
- Handle partial errors
- A missing part compared to the batch processor is the ability to shard
the batches by context value.
Updates
https://github.com/open-telemetry/opentelemetry-collector/issues/8122
This change makes the logging of the exported error failures cleaner.
1. Ensure an error message is logged every time and only once when data
is dropped/rejected due to export failure.
2. Update the wording. Specifically, don't use "dropped" term when an
error is reported back to the pipeline. If there is no queue configured,
the exporter doesn't drop data by itself but rather rejects it. Keep the
"dropped" wording for failures after the enabled queue.
3. Properly report any error reported by a queue. For example, a
persistent storage error must be reported as a storage error, not as
"queue overflow".
Fixes
https://github.com/open-telemetry/opentelemetry-collector/issues/9219
Introduce an option to limit the queue size by the number of items
instead of number of requests. This is preliminary step for having the
exporter helper v2 with a batcher sender placed after the queue sender.
Otherwise, it'll be hard for the users to estimate the queue size based
on the number of requests without batch processor in front of it.
This change doesn't effect the existing functionality and the items
based queue limiting cannot be utilized yet.
Updates
https://github.com/open-telemetry/opentelemetry-collector/issues/8122
Alternative to
https://github.com/open-telemetry/opentelemetry-collector/pull/9147
The current re-enqueuing behavior is not obvious and cannot be
configured. It takes place only for persistent queue and only if
`retry_on_failure::enabled=true` even if `retry_on_failure` is a setting
for a different backoff retry strategy.
This change removes the re-enqueuing behavior in favor of the
`retry_on_failure` option. Consider increasing
`retry_on_failure::max_elapsed_time` to reduce chances of data loss.
Resolves
https://github.com/open-telemetry/opentelemetry-collector/issues/8382
This change unblocks adding the `enqueue_on_failure` option to the queue
sender by removing the requeue behavior on the shutdown. If we don't
remove requeue on shutdown, it's possible to run into a situation
described in
https://github.com/open-telemetry/opentelemetry-collector/issues/7388.
After the recent refactoring, the chance of running into it is pretty
small, but it's still possible.
The only reason to requeue on shutdown is to make sure there is no data
loss with the persistent queue enabled. The persistent queue captures
all the inflight requests in the persistent storage anyway, so there is
no reason to requeue an inflight request. The only downside is it
potentially can cause sending duplicate data on the collector restart in
case of a partially failed request during shutdown.
Another option would be to rework the memory queue to never close the
channel but still ensure draining.
Use returned error instead to simplify the senders feedback loop. This
change preserves the behavior. Re-enqueueing of the temporary failures
depends on the enabled retry sender. This will be changed in the next
step when re-queueing becomes a configurable option
Pass request+context through the senders pipeline similar to what we do
in the collector's pipeline. Use a helper QueueRequest struct for
passing requests through queues
This changes also moves the experimental Request interface to a separate
package.
This change ensures the queue_size and queue_capacity metrics are
available when using OpenTelemetry for the collector's internal metrics.
Fixes#8682
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
**Description:** Validate number of consumers in exporterhelper
This is a very small bug fix that does not even deserve a changelog
entry. The collector will not work if a queue with a number of consumers
<= 0 is used, so we should fail loudly instead of just hanging not
draining the data from the queue.
**Testing:** UTs were added.
_Please delete paragraphs that you did not use before submitting._
Jade Guiton