Pior to this change, DDSpanBuilder.withTag behaves different than DDSpan.setTag when decorators are triggered.
Specifically builder.withTag(a, val).withTag(b, val) can behave differently than span.setTag(a, val); span.setTag(b, val)
This change makes a small step towards determinism by changing the HashMap inside DDSpanBuilder into a LinkedHashMap. That guarantees the tags from withTags translated to the same sequence of setTag calls.
NOTE: Even with this change, there are inconsistencies when tags are removed or set to an empty value.
Tested on docker. Although "/proc/self/cgroup" file is not empty, "/proc/self/cgroup".toFile().length() returns 0
as does Files.size("/proc/self/cgroup").
Improving the reliability of the slow sender test.
Still needs work, but to make this truly reliable, I'd need to add to the DDAgentWriter API. I'll probably do that, but I'm trying to start with sticking to the existing API.
To prevent unbounded memory consumption, restricting the size of the sender queue. Also, lowering the size of the Disruptor queue.
Unfortunately, our choice of a ScheduledExecutorService makes this a bit difficult, since ScheduledExecutorService doesn't allow us to supply the queue.
A bigger change is in-order but for now, this change restricts the queue size by introducing a Semaphore around the ScheduledExecutorService.
In effort to making testing easier, I introduced Monitor.onFlush. This is used in the new slow response test which attempts to simulate a situation where the sending queue would back up.
Adding a happy & comm failure test case for StatsDMonitor.
This checks some but not all of the counters but also makes sure the exceptional path branches are exercised.
To make this test possible, I added an additional constructor to StatsDMonitor that takes a StatsDClient.
Adding StatsDMonitor to DDAgentWriter
Modified Writer.createAgentWriter to read the health Monitor configs and construct StatsDMonitor when health metrics are enabled
Updated DDTracerTest to check for use of appropriate Monitor class depending on the Config
Introducing a Monitor for DDAgentWriter
Monitor can be used to track critical lifecycle events in the publishing of spans. This change only contains a NoopMonitor, but a StatsD variant will be added in later commit.
The DDAgentWriter has been altered to take a Monitor instance at construction for the purposes of testing the Monitor.
DDAgentWriter now includes tests to count spans published and sent -- including new tests for multi-threaded writing and slow agent response scenarios.
Changing the DDApi send methods to return a new Response object instead of a boolean result.
The Response object allows additional information to be propagated to the DDAgentWriter that is needed for helath monitoring.
Beyond success / fail represented by DDApi.Response.success, this includes...
- communication exception to the agent
- status code returned by the agent
- JSON parsing exception caused by bad payload returned by the agent
DDApiTest-s have also been updated to check response.success() but also to check response.status() as appropriate.
It is a linear operation that can take a long time when many spans are involved.
Using an atomic integer to track the size allows it to be a constant operation.
Laplie Anderson
Tyler Benson
Laplie Anderson
Douglas Q Hawkins
dougqh
Cecile terpin
Zachary Groves
ZStriker19
Nikolay Martynov