* Add config-based attributes to all spans passing through collector
There is a usecase where all spans going through the collector should be
tagged with some additional attribute information (e.g. env, pipeline).
This solves for that usecase.
Testing Done: unit tests
This adds a new cors_allowed_origins YAML config parameter to the opencensus receiver config block, which, if specified will allow HTTP clients to write to the trace grpc-gateway using CORS with the specified list of origins (wildcards can be used to match all origins or arbitrary origin substrings).
This will help with development of opencensus-web.
Added a Prometheus metrics exporter that allows
us to export metrics.
Its configuration is available under YAML field option
```yaml
exporters
prometheus:
<key>: <value>
```
where <key> could be any of
* address -- the address on which to run the Prometheus
scrape endpoint /metrics on
* const_labels -- labels that will be applied to each
metric that is exported
* namespace -- optionally if defined records metrics
under a specific namespace, leave it blank to preserve
the names of the proxied metrics (if that's the case)
This change uses the Prometheus metrics exporter
at github.com/orijtech/prometheus-go-metrics-exporter
which will be handed over to the OpenCensus community
whenever a repository is created under the
"census-ecosystem" organization.
Fixes#292
* Adds Batch Queue of Ids (for tail-sampling)
Adding one of the types used to implement tail-based sampling.
This type will be used to create the batches that are periodically
processed while running tail-based sampling. The idea is that
receivers will put ids concurrently in a batch while from time
to time a batch will be taken for sampling policy evaluation.
The pipeline of batches behaves in FIFO fashion and the owner of the type
is in charge of closing each batch by calling the proper method (which returns the
first batch in the pipe while also starting a new batch).
* PR Feedback
Trying to avoid confusion with the docs by using pipeline instead of queue.
* Better package name
From id_batch_queue to id_batcher (all related renames: files, folders, etc).
* Add basic telemetry to collector
This change adds a Prometheus endpoint using OC to provide metrics
to the collector. There are settings to control the level and port
in which the metrics are exposed. The main metrics added for now
are in the queued exporters.
* Check for view registration errors
* PR Feedback
* Add dynamic support for VERSION
As we complete milestones we plan to cut versions. Today, this requires
editing two separate files and committing them. Instead, default to an
undefined version and allow the version to be passed as part of
build_binaries or make.
* Default to version being latest
* Update receiver/jaeger.New to take in a Configuration
that now allows specifying the Jaeger agent UDP ports
* Separated out startCollector and startAgent helpers
for the Jaeger receiver
* Agent now runs on UDP ports:
+ Thrift Compact: by default 6831
+ Thrift Binary: by default 6832
+ Agent server: by default 5778
* Skipping Jaeger Binary Thrift test
Due to a nest of versions of Thrift-Go, and generated by various
Thrift compilers, we get an error
"Bad version in ReadMessageBegin"
so for now disable the Jaeger Binary Thrift test.
* Enabling queued processor for jaeger
Uses the processor queue to send data to Jaeger Collectors.
This allows deeper retry queues than what can be done directly with
the Jaeger exporter and also allows better handling of service information
for Jaeger backends.
* Merge fix and minor corrections
* PR Feedback: add queue info about defaults to README.md
* Fix one default that was incorrect.
* Update lint (this was needed since we are using latest lint and it ends updating go.mod and go.sum)
For now before any OpenCensus Metrics exporter is created, just
hook it up. This will allow applications such as the OpenCensus-PHP
stats backend to send content.
Fixes#216
Consolidated config related utilities into internal/config
This change introduces the consolidated changes. A followup
will be to also edit cmd/occollector for parity of use with
the now consolidated/extracted functions.
Updates #210
Updates #211
Moved the server starting logic to the receiver and updated
New to
New(address string) (*ZipkinReceiver, error)
instead of
New(spansink.Sink) (*ZipkinReceiver, error)
This change ensures that after invoking New all the user
has to do is then just invoke:
zr.StartTraceReception(context.Context, spansink.Sink) error
and then for cleanup
defer zr.StopTraceReception(context.Context)
Fixes#212
Fix TraceExporter's signature from:
ExportSpanData(ctx context.Context, node *commonpb.Node, spandata ...*trace.SpanData) error
to
ExportSpans(ctx context.Context, node *commonpb.Node, spans ...*tracepb.Span) error
as the original signature leaked the intention of reusing OpenCensus-Go
exporters, but really that is an implementation detail left to the
exporters.
Fixes#207
* Start adding pipeline to collector
This is the basis to do the work for actually implementing OC collector
in the short run. At first enable receivers:
1. OC Receiver
2. Jaeger
3. Zipkin
The queue processor is added but not connected to exporters by default,
although there are options to do so.
As previously indicated before there is some duplication of code
with the agent, this should be reduced as we implement the interfaces and
project coalesce around certain patterns.
The queue is missing its metrics and those will be added soon.
* Only allocate TraceID and SpanID iff they are non-nil lest we'll
consume 8/16 bytes for every span which becomes quite expensive
* Also retrieve "status.code" and "status.message" as the Jaeger exporter
from OpenCensus sends those
* Avoid creating an attributes map if attributes don't exist and
let go of the `make(map[<K>]<V>, size_hint)` idiom as it prevents
us from being able to detect if values were added to the attributes map
* Added in-code conversion tests to ensure fidelity preservation
* Translate "message" in logs/tags to a description value
Noticed while rolling out the Jaeger receiver/interceptor.
Fixes#187Fixes#188Fixes#189Fixes#190
Can now intercept Zipkin Protobuf spans that are
sent to a Zipkin server.
The ZipkinInterceptor multiplexes on the "Content-Type"
header, checking if the value set is "application/x-protobuf"
otherwise fallback to try parsing as JSON.
* go.mod: use Zipkin-Go which features proto parsing
* Add the "X-B3-Flags" reference from Zipkin resources
Fixes#154
Spawned https://github.com/openzipkin/zipkin-go/issues/87
Waiting on https://github.com/openzipkin/zipkin-go/pull/88
These changes are a follow up to comments in a previous PR. It changes
the normalization process to compare JSON strings and remove some
extra functions by replacing them with an if/else statement and local
variables.
This change introduces our own copy of a modified Zipkin Trace
exporter from OpenCensus-Go. This modification takes into
account the fact that our Zipkin interceptor "intercepts"/proxies
spans while the OpenCensus-Go Zipkin exporter assumes a static
world in which there is only one localEndpoint and no remoteEndpoint.
However, localEndpoint and remoteEndpoint are per-span fields
hence the necessity for us to ensure that users of this project
can have an almost transparent proxying and processing of Zipkin spans.
This change also features an end-to-end test that takes intercepted
Zipkin spans e.g. from a Zipkin Java exporting application, intercepts
them to the AgentCore and then on export we ensure that our output
has all the endpoints "localEndpoint", "remoteEndpoint" as well
as an equivalence of all the other fields.
We also now use the updated Zipkin-Go client after
https://github.com/openzipkin/zipkin-go/issues/85
was fixed with
https://github.com/openzipkin/zipkin-go/pull/86
The updated Zipkin-Go no longer serializes empty ipv4
addresses as 0.0.0.0 so remove those from our tests
Fixed as per https://github.com/openzipkin/zipkin-go/pull/86Fixes#132
Supersedes https://github.com/census-instrumentation/opencensus-go/issues/959
Supersedes https://github.com/census-instrumentation/opencensus-go/issues/960
Spawned https://github.com/openzipkin/zipkin-go/issues/85
* interceptor: add zipkin interceptor
* Zipkin interceptor multiplexing + running on 9411
* Can receive spans from diverse sources and successfully
intercept traces from various serviceNames and IPv* combinations.
Verified by examining the data on the Zipkin UI itself by
resending to a server running on a different port.
* End-to-end serialization tests where we intercept JSON
from a wild Zipkin reporter, transform it into trace-proto
then to OpenCensus-Go SpanData and then export it with the
OpenCensus-Go Zipkin exporter, inspect the JSON and compare
it with the final/expected JSON data.
Fixes#24
* interceptor/zipkin: add remoteEndpoint to Node.Attributes
Zipkin's RemoteEndpoint field is necessary when later on
building the dependency graph for Zipkin services.
This change ensures that that field is added as an attribute
to the Node with keys whose prefix is:
"zipkin.remoteEndpoint." e.g.
- "zipkin.remoteEndpoint.ipv4"
- "zipkin.remoteEndpoint.serviceName"
- "zipkin.remoteEndpoint.port"
* Fail if Zipkin interceptor and exporter run on same address
Fail spectacularly if the Zipkin interceptor and the Zipkin
exporter will all be run on the same address.
This is because this is a self DOS vector as intercepted spans
will be processed by the interceptor, then exported out by
the exporter but instead those exported spans will then be
sent back to the interceptor and this goes on indefinitely
and will just consume memory and resources.
To perform this check:
a) Compare HostPorts of interceptor vs exporter
b) If say "127.0.0.1", "localhost", "" for host and the ports
match
c) Resolve the IPs for the hosts
if any of those steps match return a logical conflict error
which will be thrown later.
The crash will look something like this
```shell
2018/10/23 18:54:41 Configuration logical error: ZipkinInterceptor address ("127.0.0.1:9411") aka (127.0.0.1 on port 9411)
is the same as the interceptor address ("localhost:9411") aka (127.0.0.1 on port 9411)
```
We can now detect any of these clashing configurations:
```yaml
interceptors:
zipkin:
address: "127.0.0.1:9411"
exporters:
zipkin:
endpoint: "http://localhost:9411/api/v2/spans"
```
```yaml
interceptors:
zipkin:
address: ":9411"
exporters:
zipkin:
endpoint: "http://localhost:9411/api/v2/spans"
```
```yaml
interceptors:
zipkin:
address: "localhost:9411"
exporters:
zipkin:
endpoint: "http://localhost:9411/api/v2/spans"
```
```yaml
interceptors:
zipkin:
address: "localhost:9411"
exporters:
zipkin:
endpoint: "http://:9411/api/v2/spans"
```
```yaml
interceptors:
zipkin:
address: "localhost:9411"
exporters:
zipkin:
endpoint: "http://127.0.0.1:9411/api/v2/spans"
```
and in a case where IP resolution helped, "10.0.0.147" is my local
address and I sneakishly tried to get the interceptor on "localhost"
which clashes with my local address as per:
```yaml
interceptors:
zipkin:
address: "localhost:9411"
exporters:
zipkin:
endpoint: "http://10.0.0.147:9411/api/v2/spans"
```
we caught it
```shell
2018/10/23 19:01:01 Configuration logical error: ZipkinInterceptor address ("10.0.0.147:9411") aka (10.0.0.147 on port 9411)
is the same as the interceptor address ("localhost:9411") aka (10.0.0.147 on port 9411)
```
* Go fmt with Go1.10 to match TravisCI
Go1.11's gofmt seems to be more conservating with
aligning literal assignments while Go1.10 seems more
extravagant. This subtle difference was causing TravisCI
failures since TravisCI uses Go1.10.
* interceptor/zipkin: handle compressed HTTP bodies
Some clients such as Zipkin-Java send HTTP bodies
with JSON that's been compressed as "gzip".
This change handles "Content-Encoding" values of:
* "gzip"
* "deflate", "zlib"
and decompresses them accordingly
* interceptor/zipkin: tests use anagram signature
A test that checked for raffled output before used an
"xorChecksum" but that's not really always unique.
The better way is to use an anagram counter, then
create a signature from the serialized map of counted
runes.
* interceptor/zipkin: address review feedback
* Package rename to "zipkinterceptor" --> "zipkininterceptor"
* Fix error names
* Address feedback from review
* move "zipkinRoute" const to the top to make it easily
discoverable and not be buried deep in code
* set status of zipkin interceptor if we encounter an error
while parsing spans. We set the span status to: StatusInvalidArgument
and the error message
* *zipkin: address feedback from review
* Add a comment ot exporter/exporterparse.zipkinExporter
that the mutex "mu" protects all the constituent fields
* Fix and ensure that node uniqueness is clear, with the
proper check from a map lookup but also rename the variable
to store nodes to "uniqueNodes" to make for better code readability
* README.md + default config.yaml: update interceptors docs and information
Added a section in the README.md on the various interceptors
and how to change the Zipkin interceptor address
* sync.Mutex instead of sync.RWMutex for zipkinExporter.mu
Move the common helper TimeToTimestamp to internal
instead of duplicating its functionality allover
the repository. It is needed for implementing other
interceptors.
Added a `version` command to the CLI which
will print out the information leading to when
and how the binary was built. The fields printed
out are:
* Version: The version of ocagent itself
* Goversion: The version of Go that compiled this binary
* OS: The OS it was built for (runtime.GOOS)
* Architecture: The architecture of the system
* GitHash: The hash of the ocagent commit
For example:
```shell
Version 0.0.1
GitHash 7ec3ea8
Goversion devel +62e5215a2a Thu Oct 4 04:08:08 2018 +0000
OS darwin
Architecture amd64
```
this will aid in:
* easy bug diagnoses and fixes
* plugin management whereby the exporter plugins MUST
match the version of Go that ocagent was built with inorder
for them to be application binary compatible
* reproducible builds
Also added a Makefile with a couple of directives
that allow for cross-compilation too.
Switched the flag parser from the inflexible "flag"
package in favour of "github.com/spf13/cobra".
Fixes#59
Trace instrument the section that performs batch uploading
to the spanreceiver. The created span has a parent link
from the initiating RPC's span. Added tests to ensure that
these conditions always hold.
Updates #63
Instrument interceptor/opencensus with stats to record
the number of received spans. This is accomplished by
a helper to process spans that sends over spans even
if the number of spans is 0. Also record with tag_key
"opencensus_interceptor" whose value is the name
of the respective interceptor.
The test to ensure that 0 length spans are also added
is currently disabled because:
* Issue https://github.com/census-instrumentation/opencensus-go/issues/862
is not yet implemented which requests that the OpenCensus-Go
stats worker provide a method Flush to flush all data. Without
it, metrics will contain stale data from previous tests.
Add tests to lock-in this behavior.
Also replace grpc.NewServer with a stats enabled server helper call.
This enables tracing and metrics for each gRPC server by replacing
naked usages of grpc.NewServer with the new helper function:
internal.GRPCServerWithObservabilityEnabled
Updates #63
This commit contains an initial prototype for the exporter daemon
that can discovered via and endpoint file on the host.
The exporter looks for the endpoint file to see the availability
of the daemon and exports if it is running.
Exporter daemon will allow OpenCensus users to export
without having to link a vendor-specific exporter in their final
binaries. We are expecting the prototype exporter is going to
be implemented in every language and registered by default.
Updates https://github.com/census-instrumentation/opencensus-specs/issues/72.
Steven Karis