<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
<!-- Issue number if applicable -->
#### Link to tracking issue
Fixes #
<!--Describe what testing was performed and which tests were added.-->
#### Testing
<!--Describe the documentation added.-->
#### Documentation
<!--Please delete paragraphs that you did not use before submitting.-->
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
This PR changes `mergeBatchFunc` and `mergeBatchSplit` function as a
member function of `batchRequest`.
<!-- Issue number if applicable -->
#### Link to tracking issue
<!--Describe what testing was performed and which tests were added.-->
#### Testing
<!--Describe the documentation added.-->
#### Documentation
<!--Please delete paragraphs that you did not use before submitting.-->
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
This is part of #11131, splitting the exporterhelper module with a new
`internal` submodule that holds everything shared by the signals.
This is so a new `exporterhelperprofiles` module can make use of the
shared structs.
#### Description
This PR renames `TimeoutSettings` and `QueueSettings` (as well as the
corresponding `NewDefault` functions) to `TimeoutConfig` and
`QueueConfig`, for naming consistency reasons.
The previous struct/function names are kept as deprecated aliases of the
new ones for now.
#### Link to tracking issue
Updates #6767 (edit: reworded by @mx-psi)
#### Testing
No behavior changes were made, so no additional testing should be
necessary.
The references to the aforementioned structs/functions in existing tests
have been renamed as well. This means the deprecated function aliases
are not tested, lowering the coverage a bit.
#### Documentation
The new functions are documented identically to the previous ones.
#### Description
This PR changes initialization of `batchSender` and `queueSender` to
AFTER configuration. That way we get to access `queueConfig` and
`batcherConfig` in the same place.
Context: This is some pre-work for changing queue->batch from a pushing
model to a pulling model. We will be initialization a
`queueBatchSender(queueConfig, batcherConfig)` if both queue and batcher
are enabled and initialize `batchSender(batchConfig)` if only batcher is
enabled. This change enables us to achieve the goal without changing
config API.
#### Link to tracking issue
https://github.com/open-telemetry/opentelemetry-collector/issues/10368
#### Testing
Ran `opentelemetry-collector$ make` to make sure all tests still pass.
Co-authored-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
This is leftover from the refactor to make that func private to the
package. Removing it as all the callers can use newExporter directly.
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Add `data_type` attribute to the internal otelcol_exporter_queue_size
metric to report the type of data being processed.
All other metrics have the data type reported as part of their names. We
could've done the same for queue metrics, but that would introduce a
significant breaking change. We want to avoid that until we have all the
metrics standardized with OpenTelemetry semantic conventions.
Fixes
https://github.com/open-telemetry/opentelemetry-collector/issues/9943
#### Description
- Correctly keep track of bs.activeRequests, which denotes the number of
send waiting for next sender in chain to return. This is already done
before this PR, but in a way that is vulnerable to unfavorable goroutine
scheduling
- Decrease bs.activeRequests by the number of requests blocked by an
activeBatch at once, so that it workarounds the "bug" mentioned in (1).
#### Link to tracking issue
Fixes#9952
---------
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
This uses mdatagen to generate the queue metrics. This will allow users
to see the metric in the documentation for exporter helper
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
This deprecates CreateSettings in favour of Settings.
NewNopCreateSettings is also being deprecated in favour of
NewNopSettings
Part of #9428
~Follows
https://github.com/open-telemetry/opentelemetry-collector/pull/10333~
---------
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Introduce default batching functionality based on the internal data type
(pdata). This makes the exporter batching capability available to the
regular exporter helpers without using custom requests.
Updates #8122
This change introduces new experimental batching functionality to the
exporter helper. The batch sender is fully concurrent and synchronous.
It's set after the queue sender, which, if enabled, introduces the
asynchronous behavior and ensures no data loss with the permanent queue.
Follow-up TODO list:
- Add pre-built merge funcs for pdata
- Handle partial errors
- A missing part compared to the batch processor is the ability to shard
the batches by context value.
Updates
https://github.com/open-telemetry/opentelemetry-collector/issues/8122
This change makes the logging of the exported error failures cleaner.
1. Ensure an error message is logged every time and only once when data
is dropped/rejected due to export failure.
2. Update the wording. Specifically, don't use "dropped" term when an
error is reported back to the pipeline. If there is no queue configured,
the exporter doesn't drop data by itself but rather rejects it. Keep the
"dropped" wording for failures after the enabled queue.
3. Properly report any error reported by a queue. For example, a
persistent storage error must be reported as a storage error, not as
"queue overflow".
Fixes
https://github.com/open-telemetry/opentelemetry-collector/issues/9219
The current re-enqueuing behavior is not obvious and cannot be
configured. It takes place only for persistent queue and only if
`retry_on_failure::enabled=true` even if `retry_on_failure` is a setting
for a different backoff retry strategy.
This change removes the re-enqueuing behavior in favor of the
`retry_on_failure` option. Consider increasing
`retry_on_failure::max_elapsed_time` to reduce chances of data loss.
Resolves
https://github.com/open-telemetry/opentelemetry-collector/issues/8382
Use returned error instead to simplify the senders feedback loop. This
change preserves the behavior. Re-enqueueing of the temporary failures
depends on the enabled retry sender. This will be changed in the next
step when re-queueing becomes a configurable option
Pass request+context through the senders pipeline similar to what we do
in the collector's pipeline. Use a helper QueueRequest struct for
passing requests through queues
This changes also moves the experimental Request interface to a separate
package.
These metrics were only exporter either via OC or via the prometheus
exporter. Fixes#8673
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
The functionality has been moved into exporter helper. This is part of
#8492
There is no functional changes, just:
- code moved from `obsreport` to `exporterhelper`
- tests moved
- updated references to `obsreport.Exporter`,
`obsreport.ExporterSettings`, `obsreport.NewExporter`
- added aliases and temporary methods to replace ^^^
- removed duplicated Exporter name in new code location:
- `obsreport.Exporter` -> `exporterhelper.Exporter`
- `obsreport.ExporterSettings` -> `exporterhelper.Settings`
- `obsreport.NewExporter` -> `exporterhelper.New`
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
Separate all the parts of the baseExporter into an explicit chain of
request senders. It makes it easier to follow the data flow and add
additional senders.
This change also removes the baseSettings, because keeping the settings
is not needed anymore. All the options update the internal senders in
place.
This change also removes confusing error messages like "Exporting
failed. Dropping data. Try enabling sending_queue to survive temporary
failures" when the Queue is not even available in the exporter
(WithQueue option is not applied) which means users don't
`sending_queue` config option. Now, such messages are only shown when
the sending_queue (or retry_on_failure) is available, but not enabled by
the user.
Make the queue initialization process consistent for both queue types.
Instead of having different workflows for memory and persistent queues,
this change breaks the initialization of both into two generic steps:
1. Queue factory: `NewBoundedMemoryQueue`, `NewPersistentQueue`
2. Start method: `queue.Start(context.Context, component.Host,
QueueSettings)`
This change:
- reduces coupling between `queuedRetrySender` and the queues;
- allows future refactoring of `queuedRetrySender`;
- allows future extraction of the queue package from the exporterhelper;
- makes it possible to have `WithRequestQueue` option for the new
exporter helper API as drafted in
https://github.com/open-telemetry/opentelemetry-collector/pull/8275.
Introduce a new exporter helper that operates over client-provided
requests instead of pdata. The helper user now has to provide
`Converter` - an interface with a function implementing translation of
pdata Metrics/Traces/Logs into a user-defined `Request`. `Request` is an
interface with only one required function `Export`.
It opens a door for moving batching to the exporter, where batches will
be built from client data format, instead of pdata. The batches can be
properly sized by custom request size, which can be different from OTLP.
The same custom request sizing will be applied to the sending queue. It
will also improve the performance of the sending queue retries for
non-OTLP exporters, they don't need to translate pdata on every retry.
This is an implementation alternative to
https://github.com/open-telemetry/opentelemetry-collector/pull/7874 as
suggested in
https://github.com/open-telemetry/opentelemetry-collector/pull/7874#discussion_r1278444371
Tracking Issue:
https://github.com/open-telemetry/opentelemetry-collector/issues/8122
---------
Co-authored-by: Alex Boten <alex@boten.ca>
* [chore] use license shortform
To remain consistent w/ contrib repo, see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/22052
Signed-off-by: Alex Boten <aboten@lightstep.com>
* make goporto
Signed-off-by: Alex Boten <aboten@lightstep.com>
---------
Signed-off-by: Alex Boten <aboten@lightstep.com>
The main reason is to remove the circular dependency between the config (including sub-packages) and component. Here is the current state:
* component depends on config
* config/sub-package[grpc, http, etc.] depends on config & component
Because of this "circular" dependency, we cannot split for example "config" into its own module, only if all the other config sub-packages are also split.
Signed-off-by: Bogdan <bogdandrutu@gmail.com>
Bogdan Drutu
Sindy Li
Jade Guiton
Damien Mathieu
Alex Boten
Antoine Toulme
Dmitrii Anoshin
Carson Ip
Pablo Baeyens
David Ashpole
Alex Boten
Antonio Jimenez
Gustavo Paiva