Rename OpenTelemetry Service to Collector in docs and comments (#354)

This begins implementation of https://github.com/open-telemetry/opentelemetry-service/issues/352
This commit is contained in:
Tigran Najaryan 2019-09-19 11:46:11 -04:00 committed by GitHub
parent bc5b299383
commit 7609eaa71b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 75 additions and 75 deletions

View File

@ -1,6 +1,6 @@
#####################################################
#
# List of approvers for OpenTelemetry Service
# List of approvers for OpenTelemetry Collector
#
#####################################################
#

View File

@ -1,6 +1,6 @@
# OpenTelemetry Service
# OpenTelemetry Collector
*IMPORTANT:* This is a pre-released version of the OpenTelemetry Service.
*IMPORTANT:* This is a pre-released version of the OpenTelemetry Collector.
For now, please use the [OpenCensus Service](https://github.com/census-instrumentation/opencensus-service).
[![Build Status][travis-image]][travis-url]
@ -25,7 +25,7 @@ For now, please use the [OpenCensus Service](https://github.com/census-instrumen
## Introduction
The OpenTelemetry Service can collect traces and metrics from processes
The OpenTelemetry Collector can collect traces and metrics from processes
instrumented by OpenTelemetry or other monitoring/tracing libraries (Jaeger,
Prometheus, etc.), handles aggregation and smart sampling, and export traces
and metrics to one or more monitoring/tracing backends.
@ -40,14 +40,14 @@ not an ideal at an incident time. In addition, currently users need to decide
which service backend they want to export to, before they distribute their
binary instrumented by OpenTelemetry.
The OpenTelemetry Service is trying to eliminate these requirements. With the
OpenTelemetry Service, users do not need to redeploy or restart their
The OpenTelemetry Collector is trying to eliminate these requirements. With the
OpenTelemetry Collector, users do not need to redeploy or restart their
applications as long as it has the OpenTelemetry exporter. All they need to do
is just configure and deploy the OpenTelemetry Service separately. The
OpenTelemetry Service will then automatically collect traces and metrics and
is just configure and deploy the OpenTelemetry Collector separately. The
OpenTelemetry Collector will then automatically collect traces and metrics and
export to any backend of users' choice.
Currently the OpenTelemetry Service consists of a single binary and two
Currently the OpenTelemetry Collector consists of a single binary and two
deployment methods:
1. Agent running with the application or on the same host as the application
@ -55,13 +55,13 @@ deployment methods:
For the detailed design specs, please see [design.md](docs/design.md).
For OpenTelemetry Service performance specs, please see [performance.md](docs/performance.md).
For OpenTelemetry Collector performance specs, please see [performance.md](docs/performance.md).
For the future vision of OpenTelemetry Service please see [vision.md](docs/vision.md).
For the future vision of OpenTelemetry Collector please see [vision.md](docs/vision.md).
## <a name="deploy"></a>Deployment
The OpenTelemetry Service can be deployed in a variety of different ways
The OpenTelemetry Collector can be deployed in a variety of different ways
depending on requirements. The Agent can be deployed with the application
either as a separate process, as a sidecar, or via a Kubernetes daemonset. The
Collector is deployed as a separate application as either a Docker container,
@ -123,8 +123,8 @@ exporting will resume.
## <a name="config"></a>Configuration
The OpenTelemetry Service (both the Agent and Collector) is configured via a
YAML file. In general, at least one enabled receiver and one enabled exporter
The OpenTelemetry Collector is configured via a YAML file.
In general, at least one enabled receiver and one enabled exporter
needs to be configured.
*Note* This documentation is still in progress. For any questions, please reach out in the
@ -145,7 +145,7 @@ pipelines:
### <a name="config-receivers"></a>Receivers
A receiver is how data gets into OpenTelemetry Service. One or more receivers
A receiver is how data gets into OpenTelemetry Collector. One or more receivers
must be configured.
A basic example of all available receivers is provided below. For detailed
@ -177,7 +177,7 @@ receivers:
An exporter is how you send data to one or more backends/destinations. One or
more exporters can be configured. By default, no exporters are configured on
the OpenTelemetry Service (either the Agent or Collector).
the OpenTelemetry Collector.
A basic example of all available exporters is provided below. For detailed
exporter configuration, please see the [exporter
@ -257,7 +257,7 @@ zpages:
**TODO** Remove this once processors have been documented since that handles
these features now.
The OpenTelemetry Service also takes some global configurations that modify its
The OpenTelemetry Collector also takes some global configurations that modify its
behavior for all receivers / exporters. This configuration is typically applied
on the Collector, but could also be added to the Agent.
@ -291,7 +291,7 @@ global:
### <a name="sampling"></a>Sampling
Sampling can also be configured on the OpenTelemetry Service. Both head-based and
Sampling can also be configured on the OpenTelemetry Collector. Both head-based and
tail-based sampling are supported. Either the Agent or the Collector may enable
head-based sampling. Tail sampling must be configured on the Collector as it
requires all spans for a given trace to make a sampling decision.
@ -352,7 +352,7 @@ sampling:
> It is recommended that you use the latest [release](https://github.com/open-telemetry/opentelemetry-service/releases).
The OpenTelemetry Service can be run directly from sources, binary, or a Docker
The OpenTelemetry Collector can be run directly from sources, binary, or a Docker
image. If you are planning to run from sources or build on your machine start
by cloning the repo using `go get -d
github.com/open-telemetry/opentelemetry-service`.
@ -385,7 +385,7 @@ $ docker run \
It can be configured via command-line or config file:
```
OpenTelemetry Service
OpenTelemetry Collector
Usage:
otelsvc [flags]

View File

@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Program otelsvc is the Open Telemetry Service that collects stats
// Program otelsvc is the OpenTelemetry Collector that collects stats
// and traces and exports to a configured backend.
package main

View File

@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Program otelsvc is the Open Telemetry Service that collects stats
// Program otelsvc is the OpenTelemetry Collector that collects stats
// and traces and exports to a configured backend.
package defaults

View File

@ -1,19 +1,19 @@
# OpenTelemetry Service Architecture
# OpenTelemetry Collector Architecture
This document describes the architecture design and implementation of
OpenTelemetry Service.
OpenTelemetry Collector.
## Summary
OpenTelemetry Service is an executable that allows to receive telemetry data, optionally transform it and send the data further.
OpenTelemetry Collector is an executable that allows to receive telemetry data, optionally transform it and send the data further.
The Service supports several popular open-source protocols for telemetry data receiving and sending as well as offering a pluggable architecture for adding more protocols.
The Collector supports several popular open-source protocols for telemetry data receiving and sending as well as offering a pluggable architecture for adding more protocols.
Data receiving, transformation and sending is done using Pipelines. The Service can be configured to have one or more Pipelines. Each Pipeline includes a set of Receivers that receive the data, a series of optional Processors that get the data from receivers and transform it and a set of Exporters which get the data from the Processors and send it further outside the Service. The same receiver can feed data to multiple Pipelines and multiple pipelines can feed data into the same Exporter.
Data receiving, transformation and sending is done using Pipelines. The Collector can be configured to have one or more Pipelines. Each Pipeline includes a set of Receivers that receive the data, a series of optional Processors that get the data from receivers and transform it and a set of Exporters which get the data from the Processors and send it further outside the Collector. The same receiver can feed data to multiple Pipelines and multiple pipelines can feed data into the same Exporter.
## Pipelines
Pipeline defines a path the data follows in the Service starting from reception, then further processing or modification and finally exiting the Service via exporters.
Pipeline defines a path the data follows in the Collector starting from reception, then further processing or modification and finally exiting the Collector via exporters.
Pipelines can operate on 2 telemetry data types: traces and metrics. The data type is a property of the pipeline defined by its configuration. Receivers, exporters and processors used in a pipeline must support the particular data type otherwise `ErrDataTypeIsNotSupported` will be reported when the configuration is loaded. A pipeline can be depicted the following way:
@ -21,7 +21,7 @@ Pipelines can operate on 2 telemetry data types: traces and metrics. The data ty
There can be one or more receivers in a pipeline. Data from all receivers is pushed to the first processor, which performs a processing on it and then pushes it to the next processor (or it may drop the data, e.g. if it is a “sampling” processor) and so on until the last processor in the pipeline pushes the data to the exporters. Each exporter gets a copy of each data element. The last processor uses a `FanOutConnector` to fan out the data to multiple exporters.
The pipeline is constructed during Service startup based on pipeline definition in the config file.
The pipeline is constructed during Collector startup based on pipeline definition in the config file.
A pipeline configuration typically looks like this:
@ -59,11 +59,11 @@ pipelines:
In the above example “opencensus” receiver will send the same data to pipeline “traces” and to pipeline “traces/2”. (Note: the configuration uses composite key names in the form of `type[/name]` as defined in this [this document](https://docs.google.com/document/d/1GWOzV0H0RTN1adiwo7fTmkjfCATDDFGuOB4jp3ldCc8/edit#)).
When the Service loads this config the result will look like this (part of processors and exporters are omitted from the diagram for brevity):
When the Collector loads this config the result will look like this (part of processors and exporters are omitted from the diagram for brevity):
![Receivers](images/design-receivers.png)
Important: when the same receiver is referenced in more than one pipeline the Service will create only one receiver instance at runtime that will send the data to `FanOutConnector` which in turn will send the data to the first processor of each pipeline. The data propagation from receiver to `FanOutConnector` and then to processors is via synchronous function call. This means that if one processor blocks the call the other pipelines that are attached to this receiver will be blocked from receiving the same data and the receiver itself will stop processing and forwarding newly received data.
Important: when the same receiver is referenced in more than one pipeline the Collector will create only one receiver instance at runtime that will send the data to `FanOutConnector` which in turn will send the data to the first processor of each pipeline. The data propagation from receiver to `FanOutConnector` and then to processors is via synchronous function call. This means that if one processor blocks the call the other pipelines that are attached to this receiver will be blocked from receiving the same data and the receiver itself will stop processing and forwarding newly received data.
### Exporters
@ -99,7 +99,7 @@ pipelines:
exporters: [jaeger]
```
In the above example “jaeger” exporter will get data from pipeline “traces” and from pipeline “traces/2”. When the Service loads this config the result will look like this (part of processors and receivers are omitted from the diagram for brevity):
In the above example “jaeger” exporter will get data from pipeline “traces” and from pipeline “traces/2”. When the Collector loads this config the result will look like this (part of processors and receivers are omitted from the diagram for brevity):
![Exporters](images/design-exporters.png)
@ -109,7 +109,7 @@ A pipeline can contain sequentially connected processors. The first processor ge
The traces pipeline must have at least one processor. Metrics pipeline does not require processors since we currently do not have any implemented metrics processors yet.
Processors can transform the data before forwarding it (i.e. add or remove attributes from spans), they can drop the data simply by deciding not to forward it (this is for example how “sampling” processor works), they can also generate new data (this is how for example how a “persistent-queue” processor can work after Service restarts by reading previously saved data from a local file and forwarding it on the pipeline).
Processors can transform the data before forwarding it (i.e. add or remove attributes from spans), they can drop the data simply by deciding not to forward it (this is for example how “sampling” processor works), they can also generate new data (this is how for example how a “persistent-queue” processor can work after Collector restarts by reading previously saved data from a local file and forwarding it on the pipeline).
The same name of the processor can be referenced in the “processors” key of multiple pipelines. In this case the same configuration will be used for each of these processors however each pipeline will always gets its own instance of the processor. Each of these processors will have its own state, the processors are never shared between pipelines. For example if “queued-retry” processor is used several pipelines each pipeline will have its own queue (although the queues will be configured exactly the same way if the reference the same key in the config file). As an example, given the following config:
@ -131,7 +131,7 @@ pipelines:
exporters: [opencensus]
```
When the Service loads this config the result will look like this:
When the Collector loads this config the result will look like this:
![Processors](images/design-processors.png)
@ -159,7 +159,7 @@ drawbacks, for example:
correct credentials\monitored resources), and users may be reluctant to
“pollute” their code with OpenTelemetry.
To resolve the issues above, you can run OpenTelemetry Service as an Agent.
To resolve the issues above, you can run OpenTelemetry Collector as an Agent.
The Agent runs as a daemon in the VM/container and can be deployed independent
of Library. Once Agent is deployed and running, it should be able to retrieve
spans/stats/metrics from Library, export them to other backends. We MAY also
@ -176,9 +176,9 @@ accept spans/stats/metrics from other tracing/monitoring libraries, such as
Zipkin, Prometheus, etc. This is done by adding specific receivers. See
[Receivers](#receivers) for details.
## <a name="opentelemetry-collector"></a>Running as a Collector
## <a name="opentelemetry-collector"></a>Running as a Standalone Collector
The OpenTelemetry Service can run as a standalone Collector instance and receives spans
The OpenTelemetry Collector can run as a Standalone instance and receives spans
and metrics exported by one or more Agents or Libraries, or by
tasks/agents that emit in one of the supported protocols. The Collector is
configured to send data to the configured exporter(s). The following figure
@ -199,7 +199,7 @@ TODO: move this section somewhere else since this document is intended to descri
OpenCensus Protocol uses a bi-directional gRPC
stream. Sender should initiate the connection, since theres only one
dedicated port for Agent, while there could be multiple instrumented processes. By default, the Service is available on port 55678.
dedicated port for Agent, while there could be multiple instrumented processes. By default, the Collector is available on port 55678.
#### <a name="agent-protocol-workflow"></a>Protocol Workflow
@ -207,24 +207,24 @@ dedicated port for Agent, while there could be multiple instrumented processes.
streams.
2. As the first message in each stream, Sender must send its identifier. Each
identifier should uniquely identify Sender within the VM/container. If
there is no identifier in the first message, Service should drop the whole
there is no identifier in the first message, Collector should drop the whole
message and return an error to the client. In addition, the first message
MAY contain additional data (such as `Span`s). As long as it has a valid
identifier associated, Service should handle the data properly, as if they
identifier associated, Collector should handle the data properly, as if they
were sent in a subsequent message. Identifier is no longer needed once the
streams are established.
3. On Sender side, if connection to Service failed, Sender should retry
3. On Sender side, if connection to Collector failed, Sender should retry
indefintely if possible, subject to available/configured memory buffer size.
(Reason: consider environments where the running applications are already
instrumented with OpenTelemetry Library but Service is not deployed yet.
Sometime in the future, we can simply roll out the Service to those
environments and Library would automatically connect to Service with
instrumented with OpenTelemetry Library but Collector is not deployed yet.
Sometime in the future, we can simply roll out the Collector to those
environments and Library would automatically connect to Collector with
indefinite retries. Zero changes are required to the applications.)
Depending on the language and implementation, retry can be done in either
background or a daemon thread. Retry should be performed at a fixed
frequency (rather than exponential backoff) to have a deterministic expected
connect time.
4. On Service side, if an established stream were disconnected, the identifier of
4. On Collector side, if an established stream were disconnected, the identifier of
the corresponding Sender would be considered expired. Sender needs to
start a new connection with a unique identifier (MAY be different than the
previous one).

View File

@ -1,7 +1,7 @@
## Action Plan for Bootstraping from OpenCensus
### Goals
We need to bootstrap the OpenTelemetry Service using the existing OpenCensus Service codebase. We agreed to split the Service codebase into 2 parts: core and contrib. This bootstrapping is a good opportunity to do the splitting by only including in the OpenTelemetry Service core the minimum number of receivers and exporters and moving the rest of functionality to a contrib package (most vendor-specific code).
We need to bootstrap the OpenTelemetry Collector using the existing OpenCensus Service codebase. We agreed to split the Service codebase into 2 parts: core and contrib. This bootstrapping is a good opportunity to do the splitting by only including in the OpenTelemetry Collector core the minimum number of receivers and exporters and moving the rest of functionality to a contrib package (most vendor-specific code).
The contrib package and vendor-specific receivers and exporters will continue to be available and there is no intent to retire it. The intent is to have a clear decoupling in the codebase that facilitates independent contribution of new components in the future, allows to easily create customized versions of a Service and makes it clear that core contributors will be responsible for maintenance of the core while vendor-specific components will be maintained by corresponding vendors (note: this does not exclude dual participation at all - some developers will likely work for vendors and will also be core maintainers).
@ -42,6 +42,6 @@ Make sure commit history is preserved.
(Note: alternatively `contrib` can be a directory in the main repo - this is still open for discussion).
[ ] Provide OpenCensus-to-OpenTelemetry Service migration guidelines for end-users who want to migrate. This will include recommendations on configuration file migration. We will also consider the possibility to support old configuration format in the new binary.
[ ] Provide OpenCensus-to-OpenTelemetry Collector migration guidelines for end-users who want to migrate. This will include recommendations on configuration file migration. We will also consider the possibility to support old configuration format in the new binary.
This approach allows us to have significant progress towards 2 stated goals in our [vision document](../blob/master/docs/VISION.md): unify the codebase for agent and collector and make the service more extensible.

View File

@ -1,12 +1,12 @@
# OpenTelemetry Service Observability
# OpenTelemetry Collector Observability
## Goal
The goal of this document is to have a comprehensive description of observability of the Service and changes needed to achieve observability part of our [vision](vision.md).
The goal of this document is to have a comprehensive description of observability of the Collector and changes needed to achieve observability part of our [vision](vision.md).
## What Needs Observation
The following elements of the Service need to be observable.
The following elements of the Collector need to be observable.
### Current Values
@ -56,9 +56,9 @@ Note: some of the current values and rates may be calculated as derivatives of c
We want to generate the following events (log and/or send as a trace with additional data):
- Service started/stopped.
- Collector started/stopped.
- Service reconfigured (if we support on-the-fly reconfiguration).
- Collector reconfigured (if we support on-the-fly reconfiguration).
- Begin dropping due to throttling (include throttling reason, e.g. local saturation, downstream saturation, downstream unavailable, etc).
@ -78,14 +78,14 @@ The service should collect host resource metrics in addition to service's own pr
## How We Expose Metrics/Traces
Service configuration must allow specifying the target for own metrics/traces (which can be different from the target of collected data). The metrics and traces must be clearly tagged to indicate that they are services own metrics (to avoid conflating with collected data in the backend).
Collector configuration must allow specifying the target for own metrics/traces (which can be different from the target of collected data). The metrics and traces must be clearly tagged to indicate that they are services own metrics (to avoid conflating with collected data in the backend).
### Impact
We need to be able to assess the impact of these observability improvements on the core performance of the Service.
We need to be able to assess the impact of these observability improvements on the core performance of the Collector.
### Configurable Level of Observability
Some of the metrics/traces can be high volume and may not be desirable to always observe. We should consider adding an observability verboseness “level” that allows configuring the Service to send more or less observability data (or even finer granularity to allow turning on/off specific metrics).
Some of the metrics/traces can be high volume and may not be desirable to always observe. We should consider adding an observability verboseness “level” that allows configuring the Collector to send more or less observability data (or even finer granularity to allow turning on/off specific metrics).
The default level of observability must be defined in a way that has insignificant performance impact on the service.

View File

@ -1,7 +1,7 @@
# OpenTelemetry Service Performance
# OpenTelemetry Collector Performance
The performance numbers that follow were generated using version 0.1.3 of the
OpenTelemetry Service, are applicable primarily to the OpenTelemetry Collector and
OpenTelemetry Collector, are applicable primarily to the OpenTelemetry Collector and
are measured only for traces. In the future, more configurations will be tested.
Note with the OpenTelemetry Agent you can expect as good if not better performance

View File

@ -1,4 +1,4 @@
# OpenTelemetry Service: Extensions
# OpenTelemetry Collector: Extensions
Besides the pipeline elements (receivers, processors, and exporters) the OTelSvc
uses various service extensions (e.g.: healthcheck, z-pages, etc).
@ -29,9 +29,9 @@ supported at this moment, but this design doesnt prevent such extensions in t
future[^1].
## Service State and Extensions
## Collector State and Extensions
The diagram below shows the basic state transitions of the OpenTelemetry Service
The diagram below shows the basic state transitions of the OpenTelemetry Collector
and how it will interact with the service extensions.
![ServiceLifeCycle](images/design-service-lifecycle.png)
@ -97,7 +97,7 @@ The interface defined below is the minimum required for
extensions in use on the service:
```go
// ServiceExtension is the interface for objects hosted by the OpenTelemetry Service that
// ServiceExtension is the interface for objects hosted by the OpenTelemetry Collector that
// don't participate directly on data pipelines but provide some functionality
// to the service, examples: health check endpoint, z-pages, etc.
type ServiceExtension interface {
@ -112,7 +112,7 @@ type ServiceExtension interface {
}
// PipelineWatcher is an extra interface for ServiceExtension hosted by the OpenTelemetry
// Service that is to be implemented by extensions interested in changes to pipeline
// Collector that is to be implemented by extensions interested in changes to pipeline
// states. Typically this will be used by extensions that change their behavior if data is
// being ingested or not, e.g.: a k8s readiness probe.
type PipelineWatcher interface {

View File

@ -1,6 +1,6 @@
# OpenTelemetry Service Long-term Vision
# OpenTelemetry Collector Long-term Vision
The following are high-level items that define our long-term vision for OpenTelemetry Service, what we aspire to achieve. This vision is our daily guidance when we design new features and make changes to the Service.
The following are high-level items that define our long-term vision for OpenTelemetry Collector, what we aspire to achieve. This vision is our daily guidance when we design new features and make changes to the Collector.
This is a living document that is expected to evolve over time.

View File

@ -1,12 +1,12 @@
# OpenTelemetry Service Demo
# OpenTelemetry Collector Demo
*IMPORTANT:* This is a pre-released version of the OpenTelemetry Service.
*IMPORTANT:* This is a pre-released version of the OpenTelemetry Collector.
Typical flow of tracing and metrics data with OpenTelemetry Service: tracing and metrics data initially received by OpenTelemetry Agent
Typical flow of tracing and metrics data with OpenTelemetry Collector: tracing and metrics data initially received by OpenTelemetry Agent
and then sent to OpenTelemetry Collector using OC data format. The OpenTelemetry Collector then sends the data to the
backend, in this demo Jaeger, Zipkin, and Prometheus.
This demo uses `docker-compose` and runs against locally built docker images of OpenTelemetry Service. In
This demo uses `docker-compose` and runs against locally built docker images of OpenTelemetry Collector. In
order to build the docker images use the commands below from the root of the repo:
```shell

View File

@ -1,6 +1,6 @@
# Exporters
Below is the list of exporters directly supported by the OpenTelemetry Service.
Below is the list of exporters directly supported by the OpenTelemetry Collector.
* [Jaeger](#jaeger)
* [Logging](#logging)

View File

@ -23,7 +23,7 @@ service:
## <a name="health-check"></a>Health Check
Health Check extension enables an HTTP url that can be probed to check the
status of the the OpenTelemetry Service. The only configuration setting is the
status of the the OpenTelemetry Collector. The only configuration setting is the
port in which the endpoint is going to be available, the default port is 13133.
This extension can be used as kubernetes liveness and readiness probe.

View File

@ -26,7 +26,7 @@ type Host interface {
ReportFatalError(err error)
}
// ServiceExtension is the interface for objects hosted by the OpenTelemetry Service that
// ServiceExtension is the interface for objects hosted by the OpenTelemetry Collector that
// don't participate directly on data pipelines but provide some functionality
// to the service, examples: health check endpoint, z-pages, etc.
type ServiceExtension interface {

View File

@ -3,7 +3,7 @@ reach out in the [OpenTelemetry Gitter](https://gitter.im/open-telemetry/opentel
or refer to the [issues page](https://github.com/open-telemetry/opentelemetry-service/issues).
# Receivers
A receiver is how data gets into OpenTelemetry Service. Generally, a receiver
A receiver is how data gets into OpenTelemetry Collector. Generally, a receiver
accepts data in a specified format and can support traces and/or metrics. The
format of the traces and metrics supported are receiver specific.

View File

@ -13,7 +13,7 @@
// limitations under the License.
// Package service handles the command-line, configuration, and runs the
// OpenTelemetry Service.
// OpenTelemetry Collector.
package service
import (
@ -295,7 +295,7 @@ func (app *Application) executeUnified() {
func (app *Application) StartUnified() error {
rootCmd := &cobra.Command{
Use: "otelsvc",
Long: "OpenTelemetry Service",
Long: "OpenTelemetry Collector",
Run: func(cmd *cobra.Command, args []string) {
app.init()
app.executeUnified()

View File

@ -1,3 +1,3 @@
# OpenTelemetry Service Testbed
# OpenTelemetry Collector Testbed
Testbed is a controlled environment and tools for conducting performance tests for the Agent, including reproducible short-term benchmarks,long-running stability tests and maximum load stress tests.